\n","metadata":{}},{"cell_type":"markdown","source":"Furthermore, if you see my notebook example, you can predict wrongly, but still have an AUC of 1. One last thing is about the predictions ordering, there is no rule that your predictions must SORT IN DESCENDING ORDER, for example: This will give you an AUC score of 1, even though it may not seem to predict everything correctly. Because the below order gives rise to the best AUC, which is 1 in this case, and hence this will give you 1 as well. If you switch a few numbers inside y_pred you will notice it can still stay at 1. However, if you reverse the list order, then you will get an AUC of 0 (the opposite of the best).\n\n```python\ny_true = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]\ny_pred = [0.99999, 0.98, 0.97, 0.96, 0.95, 0.94, 0.68139, 0.50961, 0.48880, 0.44951]\nfull_score_example = sklearn.metrics.roc_auc_score(y_true, y_pred) \nprint(full_score_example) -> 1\n```","metadata":{}},{"cell_type":"code","source":"y_true = [1, 0, 1, 1, 0]\ny_pred = [0.5, 0.25, 0.2, 0.3, 0.1]\ny_pred_same_rank = [100, 25, 20, 30, 10]\n\n\nfpr_rank_1, tpr_rank_1, threshold_rank_1 = sklearn.metrics.roc_curve(y_true, y_pred, drop_intermediate=True)\nfpr_rank_2, tpr_rank_2, threshold_rank_2 = sklearn.metrics.roc_curve(y_true, y_pred_same_rank, drop_intermediate=True)\nroc_rank_1 = sklearn.metrics.roc_auc_score(y_true=y_true, y_score=y_pred, average=\"macro\", sample_weight=None, max_fpr=None) # 0.833\nroc_rank_2 = sklearn.metrics.roc_auc_score(y_true=y_true, y_score=y_pred_same_rank, average=\"macro\", sample_weight=None, max_fpr=None) # 0.833","metadata":{"execution":{"iopub.status.busy":"2021-09-20T10:37:06.433901Z","iopub.execute_input":"2021-09-20T10:37:06.434283Z","iopub.status.idle":"2021-09-20T10:37:06.445281Z","shell.execute_reply.started":"2021-09-20T10:37:06.434246Z","shell.execute_reply":"2021-09-20T10:37:06.444479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.figure(figsize =[10,9])\nplt.title('Receiver Operating Characteristic')\nplt.plot(fpr_rank_1, tpr_rank_1, 'b', label = 'AUC = %0.2f' % roc_rank_1)\nplt.plot(fpr_rank_2, tpr_rank_2, 'b', label = 'AUC = %0.2f' % roc_rank_2)\nplt.legend(loc = 'lower right')\nplt.plot([0, 1], [0, 1],'r--')\nplt.xlim([0, 1])\nplt.ylim([0, 1])\nplt.ylabel('True Positive Rate')\nplt.xlabel('False Positive Rate')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-09-20T10:37:06.530754Z","iopub.execute_input":"2021-09-20T10:37:06.531209Z","iopub.status.idle":"2021-09-20T10:37:06.822847Z","shell.execute_reply.started":"2021-09-20T10:37:06.531165Z","shell.execute_reply":"2021-09-20T10:37:06.82187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ROC as C-Statistic\n\nROC can be interpreted as [c-statistics](https://stats.stackexchange.com/questions/193138/roc-curve-drawbacks).\nROC AUC has the property that it coincides with the $c$ statistic. The $c$ statistic measures the probability that a positive example is ranked higher than a negative example. In this sense, the ROC AUC answers the question of how well the model discriminates between the two classes.\n\nA model with high discrimination is not necessarily well calibrated. Suppose a logistic regression model predicts probabilities of 0.52 for positives and 0.51 for negatives (imagine 10 ground truth where 6 is positive and 4 is negative, then the author meant the associated probabilities with each of these ground truth is 0.52 and 0.51 respectively, for the positive and negative classes). This model has an AUC of 1 (recall you need not predict everything correctly to get an AUC of 1) but the probabilities aren't helpful in the sense of identifying which purported positives are highest-risk. Because all of the positives are assigned the same posterior probability, they can't be differentiated.\n\nMoreover, a well-calibrated model will have its maximum ROC AUC fixed by the ratio of positives to negatives in the data. This means that a model which has some very desirable probabilities (i.e. its posterior probabilities match the true probability) has a cap on its performance, and therefore an **uncalibrated** model could \"dominate\" in terms of ROC AUC.\n\nROC AUC doesn't tell you anything about the costs of different kinds of errors. For example, if you're trying to detect fraud, a 10,000 dollar purchase of uncertain provenance represents a larger potential loss than a 10 dollar purchase. But ROC AUC would treat both events as if they have the same weight -- obviously any reasonable model should be able to distinguish between these two types of error.\n\nROC AUC also tends to be dominated by the \"high FPR\" points. **Depending on the application, these points may be the least relevant.** Consider the case where the model is used to refer high-risk transactions to experts who will conduct further vetting. There may only be enough humans to assess 50 transactions per unit time; since the most highly-ranked transactions occur on the \"left hand\" size of the ROC curve by definition, this is also the region with the lowest area. So by looking at the whole AUC, you're optimistically biasing your results upwards, i.e. ROC AUC is buoyed by the observations \"to the right\" of the actual set of observations which humans will vet. (Illustration is simple. Draw a vertical line at FPR<0.5 on any ROC curve. The area to left is higher for all such vertical lines.) To avoid this, some people use partial ROC AUC, which has its own host of problems, chief among them that software implementations tend to assume that you're interested in truncation *at some value of FPR.* But in the case that you care about the top $n$ transactions, this approach is obviously wrong because the top $n$ transactions will happen at different FPR values for different classifiers. Standardization of partial AUC (to preserve the property that AUC < 0.5 is worse than random, 1 is perfect, 0 is worthless) incurs further difficulties.\n\nThe ROC curve itself is of little interest. \"Dominating\" classifiers can be assessed by AUC. Stochastic equivalence can be assessed by tests of equivalence of ranks. Prof. Harrell's comment drives at a consistent theme of his work, which is that the real question diagnostics should answer is one of risk assessment and utility optimization. Examining ROC AUC tends to encourage selection of truncation points, which should be avoided because it only provides partial information to decision makers.\n\nAlternative measures of performance (e.g. log-likelihood) characterize the calibration of the model and proper scoring rules generally have the quality that they encourage honest forecasts.","metadata":{}},{"cell_type":"markdown","source":"### Pros and Cons of AUROC\n\nBefore we go ham on the [Drawbacks of AUROC](https://stats.stackexchange.com/questions/193138/roc-curve-drawbacks), we first try to think the following:\n\n\n\nOn the other hand, if you're not really interested in how the model performs on the negative class, but just want to make sure every positive prediction is correct (precision), and that you get as many of the positives predicted as positives as possible (recall), then you should choose PRAUC (more with it later). For example, for detecting cancer, you don't care how many of the negative predictions are correct, you want to make sure all the positive predictions are correct, and that you don't miss any. (In fact, in this case missing a cancer would be worse than a false positive so you'd want to put more weight towards recall.)\n
\n\nThe following [on why AUC can be misleading](https://stats.stackexchange.com/questions/360017/when-is-an-auc-score-misleadingly-high/360040#360040)\nOne possible reason you can get high AUROC with what some might consider a mediocre prediction is if you have imbalanced data (in favor of the \"zero\" prediction), high recall, and low precision. That is, you're predicting most of the ones at the higher end of your prediction probabilities, but most of the outcomes at the higher end of your prediction probabilities are still zero. This is because the ROC score still gets most of its \"lift\" at the early part of the plot, i.e., for only a small fraction of the zero-predictions. \n\nFor example, if 5% of the test set are \"ones\" and all of the ones appear in the top 10% of your predictions, then your AUC will be at least 18/19 because, after 18/19 of the zeroes are predicted, already 100% of the ones were predicted. Even if the top 5% are all zeroes.\n\nWhether this is a \"bad\" prediction depends on your priorities. If you think that false negatives are terrible and false positives are tolerable, then this prediction is okay. But if it's the opposite, then this prediction is pretty bad.\n
\n\nBelow is the \"phenomenon\" that AUC of 1 but the models look bad. Notice that even though we have a perfect AUROC score of 1, the model is not at all confident with the predictions in the sense that we cannot pin point any two positive labels and say that one of them is of higher probability than the other. If you are not convinced, the below code illustrates the point and the plot shows you.\n