{"cells":[{"metadata":{},"cell_type":"markdown","source":"##### This is a copy of v5 of[@kozodoi's](https://kaggle.com/kozodoi) Label Consistency [notebook](https://www.kaggle.com/kozodoi/checking-the-label-consistency-requirements). This copy has been tested by the challenge hosts (thanks to [@jeffrudie](https://kaggle.com/jeffrudie) ) and will be used by the hosts on the top 10 private leaderboard submissions to verify compliance with the logical consistency of labels requirements after the end of the contest."},{"metadata":{},"cell_type":"markdown","source":"# OVERVIEW\n\nOne of the unique rules of this competition is a special requirement regarding the label hierarchy consistency. We predict nine exam-level and one image-level label, where some of the labels are conflicting and must adhere to a specific hirearchy displayed on the image below.\n\n![hierarchy](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F115173%2Fa2a5ee66b5799274141dd547cc3ea466%2FPE%20figure.jpg?generation=1599575183749576&alt=media)\n\nAccording to the data description page: \n\n> Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page. The metric intends to heavily penalize submissions which mis-predict in this manner, however due to the complexity of predictions at both image and study levels and as an extra precaution, the host will verify that prospective winners have not made conflicting label predictions. The requirements which submissions will be held to are [specified by the host in this post](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473).\n\nThe goal of this notebook is to develop a code that checks for label consistency in a submission file to make sure it adheres to the competition rules. Please note that the code has not been fully verified with the organizers yet and can be interpreting some of the rules incorrectly. I will be happy to correct any inconsistencies you might find if you point me to them in the comments section :)"},{"metadata":{},"cell_type":"markdown","source":"# VERSION HISTORY\n\n- v1: first version of the notebook\n- v2: corrected handling of `prediction == 0.5` for some labels as noted by [@anthracene](https://www.kaggle.com/anthracene) in the comments\n- v3: wrapped consistency checks into `check_consitency()` function\n- v4: text adjustments (no code changes)\n- v5: added rule 1d (see the discussion)"},{"metadata":{},"cell_type":"markdown","source":"# CHECKING CONSISTENCY\n\n## PREPARATION"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# LIBRARIES\nimport pandas as pd\nimport numpy as np","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# IMPORT DATA\ntrain = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/train.csv')\ntest  = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's import one of the public kernel submission files to check it for the label consistency requirements. As an example, I am using the `submission.csv` file produced by [@seraphwedd18](https://www.kaggle.com/seraphwedd18) in [this kernel](https://www.kaggle.com/seraphwedd18/pe-detection-with-keras-model-creation/output?scriptVersionId=42782514&select=submission.csv). "},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# IMPORT EXAMPLE SUBMISSION\nsub = pd.read_csv('../input/pe-detection-with-keras-model-creation/submission.csv')\nsub.shape\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SUBMISSION TRANSFORMATION"},{"metadata":{},"cell_type":"markdown","source":"First, we transform submission from the long to the wide format similar to the `train.csv` file to make it easier to check for rules. This involves two steps:\n1. Reshaping exam-level predictions to wide format.\n2. Merging image-level predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# EXAM LEVEL\nfor i in test['StudyInstanceUID'].unique():\n\n    df_tmp = sub.loc[sub.id.str.contains(i, regex = False)].reset_index(drop = True)\n    df_tmp['StudyInstanceUID'] = df_tmp['id'].str.split('_').str[0]\n    df_tmp['label_type']       = df_tmp['id'].str.split('_').str[1:].apply(lambda x: '_'.join(x))\n    del df_tmp['id']\n    \n    if i == test['StudyInstanceUID'].unique()[0]:\n        df = df_tmp.copy()\n    else:\n        df = pd.concat([df, df_tmp], axis = 0)\n        \ndf_exam = df.pivot(index = 'StudyInstanceUID', columns = 'label_type', values = 'label')\ndf_exam.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# IMAGE LEVEL\ndf_image = sub.loc[sub.id.isin(test.SOPInstanceUID)].reset_index(drop = True)\ndf_image = df_image.merge(test, how = 'left', left_on = 'id', right_on = 'SOPInstanceUID')\ndf_image.rename(columns = {\"label\": \"pe_present_on_image\"}, inplace = True)\ndel df_image['id']\ndf_image.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# MERGER\ndf = df_exam.merge(df_image, how = 'left', on = 'StudyInstanceUID')\nids    = ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']\nlabels = [c for c in df.columns if c not in ids]\ndf = df[ids + labels]\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FORMALIZING CONSISTENCY RULES\n\nThe rules below represent my understanding of the label consistency requirements outlined by [@anthracene](https://www.kaggle.com/anthracene) in [this discussion topic](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473). I encourage you to read the topic before inspecting the rules below.\n\nWe implement the following two groups of rules on the exam level. The first group specifies conflicting characteristics of PE if it is detected on any of the images in the exam. The second group makes sure that if there are no images with detected PE in the exam, we do not predict any of the PE charactersitcs to be present.\n\n1. If there is at least one image per `StudyInstanceUID` with `pe_present_on_image` > 0.5, then:\n    - either `rv_lv_ratio_lt_1` or `rv_lv_ratio_gte_1` must have p > 0.5; both cannot have p > 0.5.\n    - at least one of `central_pe`, `rightsided_pe` and `leftsided_pe` must have p > 0.5; multiple having p > 0.5 is allowed.\n    - `acute_and_chronic_pe` and `chronic_pe`: only one of them can have p > 0.5; neither having p > 0.5 is allowed.\n2. If there are no images per `StudyInstanceUID` with `pe_present_on_image` > 0.5, then:\n    - either `indeterminate` or `negative_exam_for_pe` must have p > 0.5; both cannot have p > 0.5.\n    - all positive-related labels: `rv_lv_ratio_lt_1`, `rv_lv_ratio_gte_1`, `central_pe`, `rightsided_pe`, `leftsided_pe`, `acute_and_chronic_pe` and `chronic_pe` must have p < 0.5."},{"metadata":{},"cell_type":"markdown","source":"## CHECKING CONSISTENCY RULES\n\nLet's start by checking if there is at least one image predicted as positive (`pe_present_on_image > 0.5`) in an exam and splitting our submission data into positive and negative exams."},{"metadata":{"trusted":true},"cell_type":"code","source":"# SPLIT NEGATIVE AND POSITIVE EXAMS\n\ndf['positive_images_in_exam'] = df['StudyInstanceUID'].map(df.groupby(['StudyInstanceUID']).pe_present_on_image.max())\n\ndf_pos = df.loc[df.positive_images_in_exam >  0.5]\ndf_neg = df.loc[df.positive_images_in_exam <= 0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can check label consistency rules separately for positive and negative exams. We will identify rows that do not satisfy any of the requirements and merge them into a data frame representing the inconsistent predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECKING CONSISTENCY OF POSITIVE EXAM LABELS\n\nrule1a = df_pos.loc[((df_pos.rv_lv_ratio_lt_1  >  0.5)  & \n                     (df_pos.rv_lv_ratio_gte_1 >  0.5)) | \n                    ((df_pos.rv_lv_ratio_lt_1  <= 0.5)  & \n                     (df_pos.rv_lv_ratio_gte_1 <= 0.5))].reset_index(drop = True)\nrule1a['broken_rule'] = '1a'\n\nrule1b = df_pos.loc[(df_pos.central_pe    <= 0.5) & \n                    (df_pos.rightsided_pe <= 0.5) & \n                    (df_pos.leftsided_pe  <= 0.5)].reset_index(drop = True)\nrule1b['broken_rule'] = '1b'\n\nrule1c = df_pos.loc[(df_pos.acute_and_chronic_pe > 0.5) & \n                    (df_pos.chronic_pe           > 0.5)].reset_index(drop = True)\nrule1c['broken_rule'] = '1c'\n\nrule1d = df_pos.loc[(df_pos.indeterminate        > 0.5) | \n                    (df_pos.negative_exam_for_pe > 0.5)].reset_index(drop = True)\nrule1d['broken_rule'] = '1d'\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECKING CONSISTENCY OF NEGATIVE EXAM LABELS\n\nrule2a = df_neg.loc[((df_neg.indeterminate        >  0.5)  & \n                     (df_neg.negative_exam_for_pe >  0.5)) | \n                    ((df_neg.indeterminate        <= 0.5)  & \n                     (df_neg.negative_exam_for_pe <= 0.5))].reset_index(drop = True)\nrule2a['broken_rule'] = '2a'\n\nrule2b = df_neg.loc[(df_neg.rv_lv_ratio_lt_1     > 0.5) | \n                    (df_neg.rv_lv_ratio_gte_1    > 0.5) |\n                    (df_neg.central_pe           > 0.5) | \n                    (df_neg.rightsided_pe        > 0.5) | \n                    (df_neg.leftsided_pe         > 0.5) |\n                    (df_neg.acute_and_chronic_pe > 0.5) | \n                    (df_neg.chronic_pe           > 0.5)].reset_index(drop = True)\nrule2b['broken_rule'] = '2b'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# MERGING INCONSISTENT PREDICTIONS\nerrors = pd.concat([rule1a, rule1b, rule1c, rule2a, rule2b], axis = 0)\nerrors['broken_rule'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, the submission file has 26081 rows that do not satisfy the rule labeled as 2b. This means that, although the exam does not have any images with predicted PE (i.e., p(`pe_present_on_image`) > 0.5), some of the positive-related labels describing the characteristics of PE have p > 0.5. In my understanding, these inconsistencies should be fixed in order not to be disqualified from the leaderboard."},{"metadata":{},"cell_type":"markdown","source":"# WRAPPING IN A FUNCTION\n\nThe function below wraps the previous code blocks into a function that can be applied to a submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_consistency(sub, test):\n    \n    '''\n    Checks label consistency and returns the errors\n    \n    Args:\n    sub   = submission dataframe (pandas)\n    test  = test.csv dataframe (pandas)\n    '''\n    \n    # EXAM LEVEL\n    for i in test['StudyInstanceUID'].unique():\n        df_tmp = sub.loc[sub.id.str.contains(i, regex = False)].reset_index(drop = True)\n        df_tmp['StudyInstanceUID'] = df_tmp['id'].str.split('_').str[0]\n        df_tmp['label_type']       = df_tmp['id'].str.split('_').str[1:].apply(lambda x: '_'.join(x))\n        del df_tmp['id']\n        if i == test['StudyInstanceUID'].unique()[0]:\n            df = df_tmp.copy()\n        else:\n            df = pd.concat([df, df_tmp], axis = 0)\n    df_exam = df.pivot(index = 'StudyInstanceUID', columns = 'label_type', values = 'label')\n    \n    # IMAGE LEVEL\n    df_image = sub.loc[sub.id.isin(test.SOPInstanceUID)].reset_index(drop = True)\n    df_image = df_image.merge(test, how = 'left', left_on = 'id', right_on = 'SOPInstanceUID')\n    df_image.rename(columns = {\"label\": \"pe_present_on_image\"}, inplace = True)\n    del df_image['id']\n    \n    # MERGER\n    df = df_exam.merge(df_image, how = 'left', on = 'StudyInstanceUID')\n    ids    = ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']\n    labels = [c for c in df.columns if c not in ids]\n    df = df[ids + labels]\n    \n    # SPLIT NEGATIVE AND POSITIVE EXAMS\n    df['positive_images_in_exam'] = df['StudyInstanceUID'].map(df.groupby(['StudyInstanceUID']).pe_present_on_image.max())\n    df_pos = df.loc[df.positive_images_in_exam >  0.5]\n    df_neg = df.loc[df.positive_images_in_exam <= 0.5]\n    \n    # CHECKING CONSISTENCY OF POSITIVE EXAM LABELS\n    rule1a = df_pos.loc[((df_pos.rv_lv_ratio_lt_1  >  0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 >  0.5)) | \n                        ((df_pos.rv_lv_ratio_lt_1  <= 0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 <= 0.5))].reset_index(drop = True)\n    rule1a['broken_rule'] = '1a'\n    rule1b = df_pos.loc[(df_pos.central_pe    <= 0.5) & \n                        (df_pos.rightsided_pe <= 0.5) & \n                        (df_pos.leftsided_pe  <= 0.5)].reset_index(drop = True)\n    rule1b['broken_rule'] = '1b'\n    rule1c = df_pos.loc[(df_pos.acute_and_chronic_pe > 0.5) & \n                        (df_pos.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule1c['broken_rule'] = '1c'\n    rule1d = df_pos.loc[(df_pos.indeterminate        > 0.5) | \n                        (df_pos.negative_exam_for_pe > 0.5)].reset_index(drop = True)\n    rule1d['broken_rule'] = '1d'\n\n    # CHECKING CONSISTENCY OF NEGATIVE EXAM LABELS\n    rule2a = df_neg.loc[((df_neg.indeterminate        >  0.5)  & \n                         (df_neg.negative_exam_for_pe >  0.5)) | \n                        ((df_neg.indeterminate        <= 0.5)  & \n                         (df_neg.negative_exam_for_pe <= 0.5))].reset_index(drop = True)\n    rule2a['broken_rule'] = '2a'\n    rule2b = df_neg.loc[(df_neg.rv_lv_ratio_lt_1     > 0.5) | \n                        (df_neg.rv_lv_ratio_gte_1    > 0.5) |\n                        (df_neg.central_pe           > 0.5) | \n                        (df_neg.rightsided_pe        > 0.5) | \n                        (df_neg.leftsided_pe         > 0.5) |\n                        (df_neg.acute_and_chronic_pe > 0.5) | \n                        (df_neg.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule2b['broken_rule'] = '2b'\n    \n    # MERGING INCONSISTENT PREDICTIONS\n    errors = pd.concat([rule1a, rule1b, rule1c, rule1d, rule2a, rule2b], axis = 0)\n    \n    # OUTPUT\n    print('Found', len(errors), 'inconsistent predictions')\n    return errors\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECK\nerrors = check_consistency(sub, test)\nerrors.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}