{"cells":[{"metadata":{},"cell_type":"markdown","source":"# OVERVIEW\n\nIn this notebook, we will check if there is any predictive power in the meta-features that can be extracted from the DCM images. We will write a function to extract a few meta-features and pixel-based features taking advantage of the parallelization and build a cross-validation based LightGBM pipeline to predict one image-level and nine exam-level labels."},{"metadata":{},"cell_type":"markdown","source":"# PREPARATIONS"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"##### PACKAGES\n\nimport os\nimport numpy as np\nimport pandas as pd\n\nimport pydicom as dcm\nfrom tqdm import tqdm\n\nimport cv2\nimport vtk\nfrom vtk.util import numpy_support\n\nfrom joblib import Parallel, delayed\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import log_loss\nfrom sklearn.preprocessing import LabelEncoder\nimport lightgbm as lgb\n\nimport seaborn as sns\nsns.set(style = 'dark')\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EXTRACT META-FEATURES\n\nEach DCM image contains meta-data in addition to the pixel arrays. We can look at the meta-data using the following code as an example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"##### EXAMPLE META-DATA\n\nimage = dcm.dcmread('/kaggle/input/rsna-str-pulmonary-embolism-detection/test/e0b02c539fb7/9ca8bffc6624/a95a398ceaf9.dcm')\nimage","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's write a function to extract meta-features given the image path. We will also use windowing to extract stats on the pixel values from differen image windows."},{"metadata":{"trusted":true},"cell_type":"code","source":"##### META-FEATURES EXTRACTOR\n\ndef window(img, WL = 50, WW = 350):\n    upper, lower = WL + WW//2, WL - WW//2\n    X = np.clip(img.copy(), lower, upper)\n    X = X - np.min(X)\n    X = X / np.max(X)\n    return X\n\ndef extract_meta_feats(img):\n\n    img_id = img.split('/')[-1].replace('.dcm', '')\n    image  = dcm.dcmread(img)\n    \n    \n    ### META-FEATURES\n    \n    pixelspacing      = image.PixelSpacing[0]\n    slice_thicknesses = image.SliceThickness\n    kvp               = image.KVP\n    table_height      = image.TableHeight\n    x_ray             = image.XRayTubeCurrent\n    exposure          = image.Exposure\n    modality          = image.Modality\n    rot_direction     = image.RotationDirection \n    instance_number   = image.InstanceNumber\n    \n    \n    ### PIXEL-BASED FEATURES\n    \n    reader = vtk.vtkDICOMImageReader()\n    reader.SetFileName(img)\n    reader.Update()\n    _extent = reader.GetDataExtent()\n    ConstPixelDims = [_extent[1]-_extent[0]+1, _extent[3]-_extent[2]+1, _extent[5]-_extent[4]+1]\n\n    ConstPixelSpacing = reader.GetPixelSpacing()\n    imageData  = reader.GetOutput()\n    pointData  = imageData.GetPointData()\n    arrayData  = pointData.GetArray(0)\n    ArrayDicom = numpy_support.vtk_to_numpy(arrayData)\n    ArrayDicom = ArrayDicom.reshape(ConstPixelDims, order = 'F')\n    ArrayDicom = cv2.resize(ArrayDicom, (512,512))\n\n    img = ArrayDicom.astype(np.int16)\n    img[img <= -1000] = 0\n\n    intercept = reader.GetRescaleOffset()\n    slope     = reader.GetRescaleSlope()\n    if slope != 1:\n        img = slope * img.astype(np.float64)\n        img = img.astype(np.int16)\n    img += np.int16(intercept)\n\n    hu_min  = np.min(img)\n    hu_mean = np.mean(img)\n    hu_max  = np.max(img)\n    hu_std =  np.std(img)\n    \n    \n    ### WINDOW-BASED FEATURES\n    \n    img_lung = window(img, WL = -600, WW = 1500)\n    img_medi = window(img, WL = 40,   WW = 400)\n    img_pesp = window(img, WL = 100,  WW = 700)\n    \n    lung_mean = np.mean(img_lung)\n    lung_std  = np.std(img_lung)\n    \n    medi_mean = np.mean(img_medi)\n    medi_std = np.std(img_medi)\n    \n    pesp_mean = np.mean(img_pesp)\n    pesp_std  = np.std(img_pesp)\n    \n    \n    return [img_id, \n            pixelspacing, slice_thicknesses, kvp, table_height, x_ray, exposure, modality, rot_direction, instance_number,\n            hu_min, hu_mean, hu_max, hu_std,\n            lung_mean, lung_std, medi_mean, medi_std, pesp_mean, pesp_std,\n           ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the data set is very large (moree than 1.7 million images), we need to parallelize feature extraction. Below, we construct a list of all test images and extract their meta-features using `Parallel`. To speed up the process, we extract meta-features from training images [in a separate notebook](https://www.kaggle.com/kozodoi/extract-meta-features-from-training-images) using the same function and import them here."},{"metadata":{"trusted":true},"cell_type":"code","source":"##### IMAGE PATH FOR TEST IMAGES\n\nim_path = []\ntest_path = '../input/rsna-str-pulmonary-embolism-detection/test/'\nfor i in tqdm(os.listdir(test_path)): \n    for j in os.listdir(test_path + i):\n        for k in os.listdir(test_path + i + '/' + j):\n            x = test_path + i + '/' + j + '/' + k\n            im_path.append(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### EXTRACT META-FEATURES\n\nresults = Parallel(n_jobs = -1, verbose = 1)(map(delayed(extract_meta_feats), im_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### CONSTRUCT TEST DATAFRAME\n\ntest_meta = pd.DataFrame(results, columns = ['SOPInstanceUID', \n                                             'pixelspacing',\n                                             'slice_thicknesses',\n                                             'kvp',\n                                             'table_height',\n                                             'x_ray_tube_current',\n                                             'exposure',\n                                             'modality',\n                                             'rotation_direction',\n                                             'instance_number',\n                                             'hu_min',\n                                             'hu_mean',\n                                             'hu_max',\n                                             'hu_std',\n                                             'lung_mean',\n                                             'lung_std',\n                                             'medi_mean',\n                                             'medi_std',\n                                             'pesp_mean',\n                                             'pesp_std'])\ntest = pd.read_csv('/kaggle/input/rsna-str-pulmonary-embolism-detection/test.csv')\ntest = test.merge(test_meta, how = 'left', on = 'SOPInstanceUID')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### IMPORT SAVED TRAIN DATA FEATURES\n\ntrain      = pd.read_csv('/kaggle/input/rsna-str-pulmonary-embolism-detection/train.csv')\ntrain_meta = pd.read_csv('/kaggle/input/extract-meta-features-from-training-images/train_meta.csv')\ntrain      = train.merge(train_meta, how = 'left', on = 'SOPInstanceUID')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### ADDITIONAL FEATURES\n\ntrain['n_images_per_study'] = train['StudyInstanceUID'].map(train.groupby(['StudyInstanceUID']).SOPInstanceUID.nunique())\ntest['n_images_per_study']  = test['StudyInstanceUID'].map(test.groupby(['StudyInstanceUID']).SOPInstanceUID.nunique())\n\ntrain['relative_instance_number'] = train['instance_number'] / train['n_images_per_study']\ntest['relative_instance_number']  = test['instance_number'] / test['n_images_per_study']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODELING: IMAGE-LEVEL LABELS\n\nWe can now start modeling! First, let's tackle the image-level label: `pe_present_on_image`. We will create `X` and `y` data sets and set up a 5-fold cross-validation. We are using a LightGBM classifier with a binary log-loss objective function."},{"metadata":{"trusted":true},"cell_type":"code","source":"##### SEPARATE X AND Y\n\nfeatures = ['pixelspacing', 'slice_thicknesses', 'kvp', 'table_height', 'x_ray_tube_current', 'exposure', \n            'n_images_per_study', 'instance_number', 'relative_instance_number',\n            'hu_min', 'hu_mean', 'hu_max', 'hu_std',\n            'lung_mean', 'lung_std', 'medi_mean', 'medi_std', 'pesp_mean', 'pesp_std']\nlabel    = 'pe_present_on_image'\n\nX = train[features + ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']]\ny = train[label]\nprint(X.shape, y.shape)\n\ntest[label] = np.nan","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### MODELING PARAMS\n\n# random seed\nseed = 23\n\n# rounds and options\nstop_rounds = 100\nverbose     = 200\n\n# LGB parameters\nlgb_params = {\n    'objective':        'binary',\n    'metrics':          'logloss',\n    'n_estimators':     10000,\n    'learning_rate':    0.01,\n    'bagging_fraction': 0.8,\n    'feature_fraction': 0.8,\n    'lambda_l1':        0.1,\n    'lambda_l2':        0.1,\n    'scale_pos_weiht':  1,\n    'silent':           True,\n    'verbosity':        -1,\n    'nthread' :         -1,\n    'random_state':     seed,\n}\n\n# partitioning\n'''\nAdapted from https://www.kaggle.com/khyeh0719/stratified-validation-strategy\n'''\nfolds  = 10\nxfolds = pd.read_csv('/kaggle/input/stratified-validation-strategy/rsna_train_splits_fold_20.csv')\ntrain  = train.merge(xfolds[['StudyInstanceUID', 'fold']], how = 'left', on = 'StudyInstanceUID')\ntrain.loc[train.fold >= folds, 'fold'] = train.loc[train.fold >= folds, 'fold'] - 10\ntrain.fold.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### MODELING\n\n# placeholders\noof         = np.zeros(len(X))\ntest_preds  = np.zeros(len(test))\nimportances = pd.DataFrame()\n\n# cross-validation\nfor fold in range(folds):\n    \n    # display info\n    print('-' * 30)\n    print('FOLD {:d}/{:d}'.format(fold + 1, folds))    \n    print('-' * 30)\n    \n    # fold idx\n    trn_idx = train.loc[train.fold != fold].index\n    val_idx = train.loc[train.fold == fold].index\n     \n    # extract samples\n    X_train, y_train = X.iloc[trn_idx][features], y.iloc[trn_idx]\n    X_valid, y_valid = X.iloc[val_idx][features], y.iloc[val_idx]\n    \n    # modeling\n    clf = lgb.LGBMClassifier(**lgb_params) \n    clf = clf.fit(X_train, y_train, \n                  eval_set              = [(X_train, y_train), (X_valid, y_valid)],\n                  eval_metric           = 'logloss',\n                  early_stopping_rounds = stop_rounds,\n                  verbose               = verbose)\n    \n    # prediction\n    oof[val_idx] = clf.predict_proba(X_valid)[:,1]\n    test_preds  += clf.predict_proba(test[features])[:,1] / folds\n    \n    # feature importance\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df['Feature']    = features\n    fold_importance_df['Importance'] = clf.feature_importances_\n    fold_importance_df['Fold']       = fold + 1\n    importances = pd.concat([importances, fold_importance_df], axis = 0)\n    print('-' * 30)\n    print('')\n\n# print performance\nprint('-' * 30)\nprint('OOF LOG-LOSS: {:.6f}'.format(log_loss(y, oof)))\nprint('-' * 30)\n\n# store test preds\ntest[label] = test_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### FEATURE IMPORTANCE\n\ncols = importances[['Feature', 'Importance']].groupby('Feature').mean().sort_values(by = 'Importance', ascending = False).index\nimportance = importances.loc[importances.Feature.isin(cols)]\n\nfig = plt.figure(figsize = (8, 5))\nsns.barplot(x = 'Importance', y = 'Feature', data = importance.sort_values(by = 'Importance', ascending = False), ci = 0)\nplt.tight_layout()\nplt.savefig('varimp_' + label + '.pdf')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODELING: EXAM-LEVEL LABELS\n\nNow, we also have nine exam-level labels. Let's perform modeling for each of the labels separately. It is important to aggregate the data by exam first to perform modeling on the exam level. We will use a similar pipeline as with the image-level label. We can also add OOF image-level predictions of `pe_present_on_image` as a feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"##### ADD PE PREDICTION AS FEATURE\n\ntrain['pred_pe_present_on_image'] = oof\ntest['pred_pe_present_on_image']  = test_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### FEATURES AND LABELS\n\nfeatures = ['pixelspacing', 'slice_thicknesses', 'kvp', 'table_height', \n            'x_ray_tube_current', 'exposure', 'n_images_per_study',\n            'pred_pe_present_on_image', 'hu_min', 'hu_mean', 'hu_max', 'hu_std',\n            'lung_mean', 'lung_std', 'medi_mean', 'medi_std', 'pesp_mean', 'pesp_std']\n\nlabels   = ['negative_exam_for_pe',\n            'rv_lv_ratio_gte_1',\n            'rv_lv_ratio_lt_1',\n            'leftsided_pe',\n            'chronic_pe',\n            'rightsided_pe',\n            'acute_and_chronic_pe',\n            'central_pe',\n            'indeterminate']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### MODELING PARAMS\n\n# random seed\nseed = 23\n\n# rounds and options\nstop_rounds = 200\nverbose     = False\n\n# LGB parameters\nlgb_params = {\n    'objective':        'binary',\n    'metrics':          'logloss',\n    'n_estimators':     10000,\n    'learning_rate':    0.01,\n    'bagging_fraction': 0.8,\n    'feature_fraction': 0.8,\n    'lambda_l1':        0.1,\n    'lambda_l2':        0.1,\n    'scale_pos_weiht':  1,\n    'silent':           True,\n    'verbosity':        -1,\n    'nthread' :         -1,\n    'random_state':     seed,\n}\n\n# partitioning\nfolds = 10\nskf   = StratifiedKFold(n_splits     = folds, \n                        shuffle      = True, \n                        random_state = seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### MODELING\n\n# placeholders\noof = np.zeros((train['StudyInstanceUID'].nunique(), len(labels)))\n\n# modeling loop\nfor label in labels:\n    \n    # display info\n    print('-' * 30)\n    print('LABEL = {}'.format(label))    \n    print('-' * 30)\n    \n    ### SEPARATE X AND Y AND TRANSFORM DATA \n    \n    X_exam = train.groupby('StudyInstanceUID')[features].agg(['mean', 'min', 'max', 'std']).reset_index(drop = True)\n    X_exam.columns = ['_'.join(col).strip() for col in X_exam.columns.values]\n    X_exam = X_exam.sort_index()\n\n    test_exam = test.groupby('StudyInstanceUID')[features].agg(['mean', 'min', 'max', 'std']).reset_index(drop = False)\n    test_exam.columns = ['_'.join(col).strip() for col in test_exam.columns.values]\n    test_exam.rename({'StudyInstanceUID_': 'StudyInstanceUID'}, axis = 1, inplace = True)    \n    test_exam = test_exam.sort_index()\n\n    y_exam = train.groupby('StudyInstanceUID')[label].agg('mean').reset_index(drop = True)\n\n    # placeholders\n    test_preds  = np.zeros(len(test_exam))\n    importances = pd.DataFrame()\n\n    # cross-validation\n    for fold, (trn_idx, val_idx) in enumerate(skf.split(X_exam, y_exam)):\n\n        # display info\n        print('- FOLD {:d}/{:d}...'.format(fold + 1, folds))    \n\n        # extract samples\n        X_train, y_train = X_exam.iloc[trn_idx], y_exam.iloc[trn_idx]\n        X_valid, y_valid = X_exam.iloc[val_idx], y_exam.iloc[val_idx]\n\n        # modeling\n        clf = lgb.LGBMClassifier(**lgb_params) \n        clf = clf.fit(X_train, y_train, \n                      eval_set              = [(X_train, y_train), (X_valid, y_valid)],\n                      eval_metric           = 'logloss',\n                      early_stopping_rounds = stop_rounds,\n                      verbose               = verbose)\n\n        # prediction\n        oof[val_idx, labels.index(label)] = clf.predict_proba(X_valid)[:,1]\n        test_preds += clf.predict_proba(test_exam[X_train.columns])[:,1] / folds\n        \n        # feature importance\n        fold_importance_df = pd.DataFrame()\n        fold_importance_df['Feature']    = X_train.columns\n        fold_importance_df['Importance'] = clf.feature_importances_\n        fold_importance_df['Fold']       = fold + 1\n        importances = pd.concat([importances, fold_importance_df], axis = 0)\n\n    # print performance\n    print('- OOF LOG-LOSS: {:.6f}'.format(log_loss(y_exam, oof[:, labels.index(label)])))\n    print('-' * 30)\n    print('')\n    \n    # feature importance\n    cols = importances[['Feature', 'Importance']].groupby('Feature').mean().sort_values(by = 'Importance', ascending = False).index\n    importance = importances.loc[importances.Feature.isin(cols)]\n    fig = plt.figure(figsize = (8, 5))\n    sns.barplot(x = 'Importance', y = 'Feature', data = importance.sort_values(by = 'Importance', ascending = False), ci = 0)\n    plt.savefig('varimp_' + label + '.pdf')\n    \n    # store test preds\n    test_exam[label] = test_preds\n    test = test.merge(test_exam[['StudyInstanceUID', label]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PERFORMANCE EVALUATION\n\nLet's evaluate performance of our predictions in terms of the competition metric: weighted log-loss over ten labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"########## CONSTRUCT OOF DF\n\noof_df = pd.DataFrame(oof)\noof_df.columns = labels\noof_df.insert(0, 'StudyInstanceUID', train.groupby('StudyInstanceUID').agg('mean').reset_index(drop = False)['StudyInstanceUID'])\noof_df = oof_df.merge(train[['StudyInstanceUID' ,'SeriesInstanceUID', 'SOPInstanceUID', 'pred_pe_present_on_image']],\n                      on  = 'StudyInstanceUID',\n                      how = 'right')\noof_df = oof_df.rename(columns = {'pred_pe_present_on_image': 'pe_present_on_image'})\noof_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"########## COMPETITION METRIC\n\n'''\nAdapted from:\n- https://www.kaggle.com/khyeh0719/0929-updated-rsna-competition-metric\n- https://www.kaggle.com/kingstying/rsna-ped-check-metric\n'''\n\ndef competition_score(df_probs, df):\n    \n    def cross_entropy(targets, predictions, epsilon = 1e-12):\n        predictions = np.clip(predictions, epsilon, 1. - epsilon)\n        ce = -(targets*np.log(predictions) + (1.-targets)*np.log(1.-predictions))\n        return ce\n    \n    qi = df[['pe_present_on_image', 'StudyInstanceUID']].groupby('StudyInstanceUID').mean().reset_index()\n    qi.columns = ['StudyInstanceUID', 'qi']\n    \n    cols_label = ['pe_present_on_image',\n                  'negative_exam_for_pe', 'indeterminate', 'chronic_pe',\n                  'acute_and_chronic_pe', 'central_pe', 'leftsided_pe',\n                  'rightsided_pe', 'rv_lv_ratio_gte_1', 'rv_lv_ratio_lt_1']\n    \n    weights = [0.07361963,\n               0.0736196319, 0.09202453988, 0.1042944785,\n               0.1042944785, 0.1877300613, 0.06257668712,\n               0.06257668712, 0.2346625767, 0.0782208589]\n    \n    assert (df['SOPInstanceUID'] == df_probs['SOPInstanceUID']).all(), f'SOPInstanceUID not match!'\n    \n    target_exam = df[['StudyInstanceUID']       + cols_label[1:]].groupby('StudyInstanceUID').mean()\n    probs_exam  = df_probs[['StudyInstanceUID'] + cols_label[1:]].groupby('StudyInstanceUID').mean()\n    \n    score_exam = []\n    epsilon    = 1e-12\n    \n    for col,w in zip(cols_label[1:],weights[1:]):\n        score = log_loss(target_exam[col].values, np.clip(probs_exam[col].values, epsilon, 1. - epsilon)) * w \n        score = score * target_exam.shape[0] \n        score_exam.append(score)\n        \n    score_exam = np.sum(score_exam)\n    \n    df_probs = pd.merge(df_probs, qi, on = 'StudyInstanceUID', how = 'inner')\n    df_probs['target-pe_present_on_image'] = df['pe_present_on_image']\n    \n\n    df_probs['score_img'] = \\\n    df_probs[['target-pe_present_on_image', 'pe_present_on_image', 'qi']].apply(lambda x:cross_entropy(x[0],x[1])*x[2]*weights[0], axis=1)\n    \n    score_img = df_probs['score_img'].sum()\n    \n    total_score   = score_exam + score_img\n    total_weights = np.sum(weights[1:])*df.StudyInstanceUID.nunique() + np.sum(weights[0]*df_probs['qi'].values)\n    \n    return total_score / total_weights","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"########## CHECK PERFORMANCE\n\nprint('Competition score: {:.6f}'.format(competition_score(oof_df, train)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREPARE SUBMISSION\n\nWe can now prepare a submission file with our predictions. To do that, we need to transform predictions to the required format. This is done in the code snippets below."},{"metadata":{"trusted":true},"cell_type":"code","source":"##### CHECK PREDICTIONS\n\nsub_df = test.copy()\nsub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### EXAM-LEVEL PREDS\n\nexam_label_names = ['negative_exam_for_pe',\n                    'rv_lv_ratio_gte_1',\n                    'rv_lv_ratio_lt_1',\n                    'leftsided_pe',\n                    'chronic_pe',\n                    'rightsided_pe',\n                    'acute_and_chronic_pe',\n                    'central_pe',\n                    'indeterminate']\n\n# list of test ids\ntest_exam_ids = []\nfor v in test.StudyInstanceUID:\n    if v not in test_exam_ids:\n        test_exam_ids.append(v)\n        \n# placeholders\nids    = []\nlabels = []\n\n# aggregate exam-level preds\nfor label in tqdm(exam_label_names):\n    for key in test_exam_ids:\n        tmp_sub = sub_df.loc[sub_df.StudyInstanceUID == key]\n        ids.append('_'.join([key, label]))\n        labels.append(np.max(tmp_sub[label]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### IMAGE-LEVEL PREDS\n\nids    += sub_df.SOPInstanceUID.tolist()\nlabels += sub_df['pe_present_on_image'].tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### SUBMISSION FILE\n\nsub = pd.DataFrame({'id':ids, 'label': labels})\nsub.to_csv('submission.csv', index = False)\nsub.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CHECK LABEL CONSISTENCY\n\nFinally, let's check label consistency using a function from [this notebook](https://www.kaggle.com/kozodoi/checking-the-label-consistency-requirements)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_consistency(sub, test):\n    \n    '''\n    Checks label consistency and returns the errors\n    \n    Args:\n    sub   = submission dataframe (pandas)\n    test  = test.csv dataframe (pandas)\n    '''\n    \n    # EXAM LEVEL\n    for i in test['StudyInstanceUID'].unique():\n        df_tmp = sub.loc[sub.id.str.contains(i, regex = False)].reset_index(drop = True)\n        df_tmp['StudyInstanceUID'] = df_tmp['id'].str.split('_').str[0]\n        df_tmp['label_type']       = df_tmp['id'].str.split('_').str[1:].apply(lambda x: '_'.join(x))\n        del df_tmp['id']\n        if i == test['StudyInstanceUID'].unique()[0]:\n            df = df_tmp.copy()\n        else:\n            df = pd.concat([df, df_tmp], axis = 0)\n    df_exam = df.pivot(index = 'StudyInstanceUID', columns = 'label_type', values = 'label')\n    \n    # IMAGE LEVEL\n    df_image = sub.loc[sub.id.isin(test.SOPInstanceUID)].reset_index(drop = True)\n    df_image = df_image.merge(test, how = 'left', left_on = 'id', right_on = 'SOPInstanceUID')\n    df_image.rename(columns = {\"label\": \"pe_present_on_image\"}, inplace = True)\n    del df_image['id']\n    \n    # MERGER\n    df = df_exam.merge(df_image, how = 'left', on = 'StudyInstanceUID')\n    ids    = ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']\n    labels = [c for c in df.columns if c not in ids]\n    df = df[ids + labels]\n    \n    # SPLIT NEGATIVE AND POSITIVE EXAMS\n    df['positive_images_in_exam'] = df['StudyInstanceUID'].map(df.groupby(['StudyInstanceUID']).pe_present_on_image.max())\n    df_pos = df.loc[df.positive_images_in_exam >  0.5]\n    df_neg = df.loc[df.positive_images_in_exam <= 0.5]\n    \n    # CHECKING CONSISTENCY OF POSITIVE EXAM LABELS\n    rule1a = df_pos.loc[((df_pos.rv_lv_ratio_lt_1  >  0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 >  0.5)) | \n                        ((df_pos.rv_lv_ratio_lt_1  <= 0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 <= 0.5))].reset_index(drop = True)\n    rule1a['broken_rule'] = '1a'\n    rule1b = df_pos.loc[(df_pos.central_pe    <= 0.5) & \n                        (df_pos.rightsided_pe <= 0.5) & \n                        (df_pos.leftsided_pe  <= 0.5)].reset_index(drop = True)\n    rule1b['broken_rule'] = '1b'\n    rule1c = df_pos.loc[(df_pos.acute_and_chronic_pe > 0.5) & \n                        (df_pos.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule1c['broken_rule'] = '1c'\n    rule1d = df_pos.loc[(df_pos.indeterminate        > 0.5) | \n                        (df_pos.negative_exam_for_pe > 0.5)].reset_index(drop = True)\n    rule1d['broken_rule'] = '1d'\n\n    # CHECKING CONSISTENCY OF NEGATIVE EXAM LABELS\n    rule2a = df_neg.loc[((df_neg.indeterminate        >  0.5)  & \n                         (df_neg.negative_exam_for_pe >  0.5)) | \n                        ((df_neg.indeterminate        <= 0.5)  & \n                         (df_neg.negative_exam_for_pe <= 0.5))].reset_index(drop = True)\n    rule2a['broken_rule'] = '2a'\n    rule2b = df_neg.loc[(df_neg.rv_lv_ratio_lt_1     > 0.5) | \n                        (df_neg.rv_lv_ratio_gte_1    > 0.5) |\n                        (df_neg.central_pe           > 0.5) | \n                        (df_neg.rightsided_pe        > 0.5) | \n                        (df_neg.leftsided_pe         > 0.5) |\n                        (df_neg.acute_and_chronic_pe > 0.5) | \n                        (df_neg.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule2b['broken_rule'] = '2b'\n    \n    # MERGING INCONSISTENT PREDICTIONS\n    errors = pd.concat([rule1a, rule1b, rule1c, rule1d, rule2a, rule2b], axis = 0)\n    \n    # OUTPUT\n    print('Found', len(errors), 'inconsistent predictions')\n    return errors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##### CHECK\n\ntest_df = pd.read_csv('/kaggle/input/rsna-str-pulmonary-embolism-detection/test.csv')\nerrors  = check_consistency(sub, test_df)\nerrors.broken_rule.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For now, we will ingore the label inconsistencies and submit the predictions as is. \n\nJudging by the submission score, there is not much signal in the meta-data, as a simple mean baseline ouperforms the above code. Still, it was useful to check if there is any correlation in there. I hope you will find this notebook useful!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}