Please ask any questions for the competition organisers here.
", "messages": [ { "id": 3054381, "postDate": "2024-11-24T16:19:12.953Z", "content": "Hi,
\nThank you for the great challenge. I noticed that the submission system accepts prediction probabilities. Due to the nature of AUROC, when someone submits prediction probabilities instead of binary outputs it will get higher scores. In the previous question, you mentioned that there would be other evaluation metrics for the top scoring teams. My question is, will the submissions with prediction probabilities be eliminated from the leaderboard before you select the top scoring teams?
\nThanks!
I noticed that the evaluation metric for the competition is the average AUROC score, but the submission format requests binary labels for each diagnosis. Shouldn't we be submitting probability scores instead of binary labels if AUROC is the evaluation metric? I would appreciate any clarification on this. Thank you!
", "rawMarkdown": "I noticed that the evaluation metric for the competition is the average AUROC score, but the submission format requests binary labels for each diagnosis. Shouldn't we be submitting probability scores instead of binary labels if AUROC is the evaluation metric? I would appreciate any clarification on this. Thank you!", "votes": 2, "replies": [ { "id": 3027198, "postDate": "2024-10-24T15:07:11.313Z", "content": "Thanks for the question and for your interest in the challenge! The submission should be in the format provided (binary output labels) as we want to know what diagnoses your model identifies. There are many ways to assess the results of challenges and after the challenge has completed we will explore different metrics with the top scoring teams. For the purpose of the challenge we are happy with the metric as described. Many thanks.
", "rawMarkdown": "Thanks for the question and for your interest in the challenge! The submission should be in the format provided (binary output labels) as we want to know what diagnoses your model identifies. There are many ways to assess the results of challenges and after the challenge has completed we will explore different metrics with the top scoring teams. For the purpose of the challenge we are happy with the metric as described. Many thanks.", "replies": [ { "id": 3038238, "postDate": "2024-11-06T18:25:44.437Z", "content": "Sorry, could I ask for further clarification on what the metric described is? The AUROC of a model is defined with respect to probabilities, not predicted labels. I suppose that labels can still be fed into an AUROC function into Python or R, although internally the validation metric would interpret them as 0 or 1 probabilities, and the final metric would just be a weighted average of the sensitivity and specificity of the labels. Is this what the AUROC is here? Apologies if I am not understanding this correctly.
", "rawMarkdown": "Sorry, could I ask for further clarification on what the metric described is? The AUROC of a model is defined with respect to probabilities, not predicted labels. I suppose that labels can still be fed into an AUROC function into Python or R, although internally the validation metric would interpret them as 0 or 1 probabilities, and the final metric would just be a weighted average of the sensitivity and specificity of the labels. Is this what the AUROC is here? Apologies if I am not understanding this correctly.", "votes": 1 } ] } ] }, { "id": 2999425, "postDate": "2024-09-26T16:25:13.203Z", "content": "Please ask any questions for the competition organisers here.
", "rawMarkdown": "Please ask any questions for the competition organisers here.", "votes": 2 }, { "id": 3027201, "postDate": "2024-10-24T15:08:53.247Z", "content": "Many thanks to everyone who has signed up for the challenge so far! We are very excited to see so many of you have indicated your interest and we look forward to seeing your submissions.
", "rawMarkdown": "Many thanks to everyone who has signed up for the challenge so far! We are very excited to see so many of you have indicated your interest and we look forward to seeing your submissions." } ], "comments": [ { "id": 3054381, "author_name": "oğuzhan büyüksolak", "author_url": "", "post_date": "2024-11-24T16:19:12.953000", "content": "Hi,
\nThank you for the great challenge. I noticed that the submission system accepts prediction probabilities. Due to the nature of AUROC, when someone submits prediction probabilities instead of binary outputs it will get higher scores. In the previous question, you mentioned that there would be other evaluation metrics for the top scoring teams. My question is, will the submissions with prediction probabilities be eliminated from the leaderboard before you select the top scoring teams?
\nThanks!
I noticed that the evaluation metric for the competition is the average AUROC score, but the submission format requests binary labels for each diagnosis. Shouldn't we be submitting probability scores instead of binary labels if AUROC is the evaluation metric? I would appreciate any clarification on this. Thank you!
", "votes": 2, "replies": [ { "id": 3027198, "author_name": "BHF Data Science Centre", "author_url": "", "post_date": "2024-10-24T15:07:11.313000", "content": "Thanks for the question and for your interest in the challenge! The submission should be in the format provided (binary output labels) as we want to know what diagnoses your model identifies. There are many ways to assess the results of challenges and after the challenge has completed we will explore different metrics with the top scoring teams. For the purpose of the challenge we are happy with the metric as described. Many thanks.
", "votes": 0, "replies": [ { "id": 3038238, "author_name": "Jose Benitez-Aurioles", "author_url": "", "post_date": "2024-11-06T18:25:44.437000", "content": "Sorry, could I ask for further clarification on what the metric described is? The AUROC of a model is defined with respect to probabilities, not predicted labels. I suppose that labels can still be fed into an AUROC function into Python or R, although internally the validation metric would interpret them as 0 or 1 probabilities, and the final metric would just be a weighted average of the sensitivity and specificity of the labels. Is this what the AUROC is here? Apologies if I am not understanding this correctly.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3027201, "author_name": "BHF Data Science Centre", "author_url": "", "post_date": "2024-10-24T15:08:53.247000", "content": "Many thanks to everyone who has signed up for the challenge so far! We are very excited to see so many of you have indicated your interest and we look forward to seeing your submissions.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3054381": "Hi,\nThank you for the great challenge. I noticed that the submission system accepts prediction probabilities. Due to the nature of AUROC, when someone submits prediction probabilities instead of binary outputs it will get higher scores. In the previous question, you mentioned that there would be other evaluation metrics for the top scoring teams. My question is, will the submissions with prediction probabilities be eliminated from the leaderboard before you select the top scoring teams?\nThanks!", "3024904": "I noticed that the evaluation metric for the competition is the average AUROC score, but the submission format requests binary labels for each diagnosis. Shouldn't we be submitting probability scores instead of binary labels if AUROC is the evaluation metric? I would appreciate any clarification on this. Thank you!", "2999425": "Please ask any questions for the competition organisers here.", "3027201": "Many thanks to everyone who has signed up for the challenge so far! We are very excited to see so many of you have indicated your interest and we look forward to seeing your submissions." } }