![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F954117%2Ff2dac8be83d08fc602dd7eb6fe33272c%2Froc_curve.png?generation=1667081140909072&alt=media\")
The evaluation metric, roc_auc_score, is literally the area under the curve (AUC), and is a common metric in machine learning. Still, its value is not very intuitive for me compared to, say, accuracy.
I find it useful to interpret it as a detected fraction in this competition:
\n$$\\mathrm{roc\\_auc\\_score} \\approx \\frac{1 + f_\\mathrm{detected}}{2}$$
\nFor example, score 0.7 means 40% of the signal (positive data) are roughly speaking “detected” and the rest 60% signal are indistinguishable from noise-only (negative) data.
\nThe simplified view is that f_detected fraction of the signal are \"detected\" with sufficient signal-to-noise ratio and the rest are indistinguishable from noise. The fraction is computed within the signal (y_true = 1) and negatives (y_true = 0) are not counted.
f_detected = sum((y_pred > 0.5) && (y_true == 1)) / sum(y_true == 1) [*1]\n
The orange line is the approximate ROC curve:
\nThe roc_auc_socre is also the faction of correctly ordered pairs:
\nmean(y_pred[i] < y_pred[j])
for all pairs with negative sample i and positive sample j.
\nWith the toy \"detected\" and \"undetected\" model:
\nThe faction of correctly ordered pair becomes:
\n$$
\n1 \\times f_\\mathrm{detected} + \\frac{1}{2} \\times \\left(1 - f_\\mathrm{detected}\\right) = \\frac{1 + f_\\mathrm{detected}}{2}
\n$$
which is the same roc_auc_score as above.
\n[*1] The line between detected and undetected is never obvious in reality and I take a threshold 0.5 here; this is an intuitive toy model not an exact formula.
", "messages": [ { "id": 2009355, "postDate": "2022-10-29T22:44:48.243Z", "content": "The evaluation metric, roc_auc_score, is literally the area under the curve (AUC), and is a common metric in machine learning. Still, its value is not very intuitive for me compared to, say, accuracy.
I find it useful to interpret it as a detected fraction in this competition:
\n$$\\mathrm{roc\\_auc\\_score} \\approx \\frac{1 + f_\\mathrm{detected}}{2}$$
\nFor example, score 0.7 means 40% of the signal (positive data) are roughly speaking “detected” and the rest 60% signal are indistinguishable from noise-only (negative) data.
\nThe simplified view is that f_detected fraction of the signal are \"detected\" with sufficient signal-to-noise ratio and the rest are indistinguishable from noise. The fraction is computed within the signal (y_true = 1) and negatives (y_true = 0) are not counted.
f_detected = sum((y_pred > 0.5) && (y_true == 1)) / sum(y_true == 1) [*1]\n
The orange line is the approximate ROC curve:
\nThe roc_auc_socre is also the faction of correctly ordered pairs:
\nmean(y_pred[i] < y_pred[j])
for all pairs with negative sample i and positive sample j.
\nWith the toy \"detected\" and \"undetected\" model:
\nThe faction of correctly ordered pair becomes:
\n$$
\n1 \\times f_\\mathrm{detected} + \\frac{1}{2} \\times \\left(1 - f_\\mathrm{detected}\\right) = \\frac{1 + f_\\mathrm{detected}}{2}
\n$$
which is the same roc_auc_score as above.
\n[*1] The line between detected and undetected is never obvious in reality and I take a threshold 0.5 here; this is an intuitive toy model not an exact formula.
", "rawMarkdown": "The evaluation metric, `roc_auc_score`, is literally the area under the curve (AUC), and is a common metric in machine learning. Still, its value is not very intuitive for me compared to, say, accuracy.\n\nI find it useful to interpret it as a *detected fraction* in this competition:\n\n$$\\mathrm{roc\\\\_auc\\\\_score} \\approx \\frac{1 + f_\\mathrm{detected}}{2}$$\n\nFor example, score 0.7 means 40% of the signal (positive data) are roughly speaking “detected” and the rest 60% signal are indistinguishable from noise-only (negative) data.\n\n## Toy ROC curve\n\nThe simplified view is that `f_detected` fraction of the signal are \"detected\" with sufficient signal-to-noise ratio and the rest are indistinguishable from noise. The fraction is computed within the signal (`y_true = 1`) and negatives (`y_true = 0`) are not counted.\n\n```python\nf_detected = sum((y_pred > 0.5) && (y_true == 1)) / sum(y_true == 1) [*1]\n```\n\n\n\nThe orange line is the approximate ROC curve:\n\n1. Starting from a large threshold ~ 1, all detected samples are marked as true positives with no false positives, giving an intercept at (0, f_detected).\n2. Lowering the threshold, undetected positives and negatives join positive predictions randomly with a constant rate, drawing the straight line toward (1, 1).\n3. The area under the line is (1 + f_detected) / 2.\n\n## ROC AUC as ordered fraction\n\nThe roc_auc_socre is also the faction of correctly ordered pairs:\n\n`mean(y_pred[i] < y_pred[j])`\n\n for all pairs with negative sample i and positive sample j.\n\nWith the toy \"detected\" and \"undetected\" model:\n- Detected signals are perfectly ordered: 1.\n- Undetected signals are randomly ordered, i.e., 1/2 are correctly ordered.\n\nThe faction of correctly ordered pair becomes:\n\n$$\n1 \\times f_\\mathrm{detected} + \\frac{1}{2} \\times \\left(1 - f_\\mathrm{detected}\\right) = \\frac{1 + f_\\mathrm{detected}}{2}\n$$\n\n\nwhich is the same roc_auc_score as above. \n\n[*1] The line between detected and undetected is never obvious in reality and I take a threshold 0.5 here; this is an intuitive toy model not an exact formula.\n", "votes": 21 }, { "id": 2009757, "postDate": "2022-10-30T09:38:29.747Z", "content": "This is a great post that really helped with my understanding. Great job again Jun!
", "rawMarkdown": "This is a great post that really helped with my understanding. Great job again Jun!", "votes": 1 }, { "id": 2009404, "postDate": "2022-10-30T00:38:10.323Z", "content": "A very interesting post. You can imagine something like that while you observe the data. There is not a binary class at all, in fact i would say there is a three class instead: Possitive, Negative, and Ambiguous. Thank you very much for sharing that, Jun :)
", "rawMarkdown": "A very interesting post. You can imagine something like that while you observe the data. There is not a binary class at all, in fact i would say there is a three class instead: Possitive, Negative, and Ambiguous. Thank you very much for sharing that, Jun :)", "votes": 1, "replies": [ { "id": 2009419, "postDate": "2022-10-30T01:18:55.703Z", "content": "I have neglected an ambiguous population in the negatives that looks like positives. Maybe there is such contribution in the test score 👍
", "rawMarkdown": "I have neglected an ambiguous population in the negatives that looks like positives. Maybe there is such contribution in the test score 👍", "votes": 2 } ] }, { "id": 2015911, "postDate": "2022-11-03T16:11:16.607Z", "content": "Interesting post. Thanks for sharing!
", "rawMarkdown": "Interesting post. Thanks for sharing!" } ], "comments": [ { "id": 2009757, "author_name": "aspiring", "author_url": "", "post_date": "2022-10-30T09:38:29.747000", "content": "This is a great post that really helped with my understanding. Great job again Jun!
", "votes": 1, "replies": [] }, { "id": 2009404, "author_name": "Zollkron", "author_url": "", "post_date": "2022-10-30T00:38:10.323000", "content": "A very interesting post. You can imagine something like that while you observe the data. There is not a binary class at all, in fact i would say there is a three class instead: Possitive, Negative, and Ambiguous. Thank you very much for sharing that, Jun :)
", "votes": 1, "replies": [ { "id": 2009419, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2022-10-30T01:18:55.703000", "content": "I have neglected an ambiguous population in the negatives that looks like positives. Maybe there is such contribution in the test score 👍
", "votes": 2, "replies": [] } ] }, { "id": 2015911, "author_name": "Zoya Naseer Hashmi", "author_url": "", "post_date": "2022-11-03T16:11:16.607000", "content": "Interesting post. Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2009355": "The evaluation metric, `roc_auc_score`, is literally the area under the curve (AUC), and is a common metric in machine learning. Still, its value is not very intuitive for me compared to, say, accuracy.\n\nI find it useful to interpret it as a *detected fraction* in this competition:\n\n$$\\mathrm{roc\\\\_auc\\\\_score} \\approx \\frac{1 + f_\\mathrm{detected}}{2}$$\n\nFor example, score 0.7 means 40% of the signal (positive data) are roughly speaking “detected” and the rest 60% signal are indistinguishable from noise-only (negative) data.\n\n## Toy ROC curve\n\nThe simplified view is that `f_detected` fraction of the signal are \"detected\" with sufficient signal-to-noise ratio and the rest are indistinguishable from noise. The fraction is computed within the signal (`y_true = 1`) and negatives (`y_true = 0`) are not counted.\n\n```python\nf_detected = sum((y_pred > 0.5) && (y_true == 1)) / sum(y_true == 1) [*1]\n```\n\n\n\nThe orange line is the approximate ROC curve:\n\n1. Starting from a large threshold ~ 1, all detected samples are marked as true positives with no false positives, giving an intercept at (0, f_detected).\n2. Lowering the threshold, undetected positives and negatives join positive predictions randomly with a constant rate, drawing the straight line toward (1, 1).\n3. The area under the line is (1 + f_detected) / 2.\n\n## ROC AUC as ordered fraction\n\nThe roc_auc_socre is also the faction of correctly ordered pairs:\n\n`mean(y_pred[i] < y_pred[j])`\n\n for all pairs with negative sample i and positive sample j.\n\nWith the toy \"detected\" and \"undetected\" model:\n- Detected signals are perfectly ordered: 1.\n- Undetected signals are randomly ordered, i.e., 1/2 are correctly ordered.\n\nThe faction of correctly ordered pair becomes:\n\n$$\n1 \\times f_\\mathrm{detected} + \\frac{1}{2} \\times \\left(1 - f_\\mathrm{detected}\\right) = \\frac{1 + f_\\mathrm{detected}}{2}\n$$\n\n\nwhich is the same roc_auc_score as above. \n\n[*1] The line between detected and undetected is never obvious in reality and I take a threshold 0.5 here; this is an intuitive toy model not an exact formula.\n", "2009757": "This is a great post that really helped with my understanding. Great job again Jun!", "2009404": "A very interesting post. You can imagine something like that while you observe the data. There is not a binary class at all, in fact i would say there is a three class instead: Possitive, Negative, and Ambiguous. Thank you very much for sharing that, Jun :)", "2015911": "Interesting post. Thanks for sharing!" } }