{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\nIn this kernel, I'm going to demonstrate how to build a stratified validation splits while doing some preliminary EDA on both train and test set"},{"metadata":{},"cell_type":"markdown","source":"# EDA and Observations"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/train.csv')\ntest = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/test.csv')\n\nprint('train.shape', train.shape, 'test.shape', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('StudyInstanceUID')['SeriesInstanceUID'].nunique().max(), test.groupby('StudyInstanceUID')['SeriesInstanceUID'].nunique().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> The StudyInstanceUID and SeriesInstanceUID is 1-1 mapping in both train and test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.intersect1d(train.StudyInstanceUID.unique(), test.StudyInstanceUID.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> No repeated StudyInstanceUID between train and test"},{"metadata":{},"cell_type":"markdown","source":"### **From the above 2 observations, it is important to split train and validation set based on StudyInstanceUID\\SeriesInstanceUID to simulate train-test split**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_num_per_patient = train.groupby('StudyInstanceUID')['SOPInstanceUID'].nunique()\ntest_image_num_per_patient = test.groupby('StudyInstanceUID')['SOPInstanceUID'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_num_per_patient.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_image_num_per_patient.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.title('image_num_per_patient')\nplt.hist(train_image_num_per_patient, bins=100, label='train', density=True)\nplt.hist(test_image_num_per_patient, bins=100, label='test', density=True)\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> As we could see, the number of images per patient in train and test are pretty close, except some slight right shift in train set (comparing the 25%-50%-75% from the describe() method)"},{"metadata":{},"cell_type":"markdown","source":"### **From the above, it is better to also do *stratified split* based on the *image number per patient* to simulate train-test split**"},{"metadata":{},"cell_type":"markdown","source":"### We will do validation splits based on\n1. Patient: Same patient should be in the same validation split\n2. Number of image per patient: Distribution should be similar across all validation splits"},{"metadata":{},"cell_type":"markdown","source":"# Create Stratified Validation Splits"},{"metadata":{"trusted":true},"cell_type":"code","source":"FOLD_NUM = 20\ntarget_cols = [c for i, c in enumerate(train.columns) if i > 2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# build summary of image num and target variables for each patient\ntrain_per_patient_char = pd.DataFrame(index=train_image_num_per_patient.index, columns=['image_per_patient'], data=train_image_num_per_patient.values.copy())\nfor t in target_cols:\n    train_per_patient_char[t] = train_per_patient_char.index.map(train.groupby('StudyInstanceUID')[t].mean())\n\ntrain_per_patient_char.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> only **pe_present_on_image** is image level, that's why only it is the only patient-level value with floating number after averaging"},{"metadata":{"trusted":true},"cell_type":"code","source":"# make image_per_patient and pe_present_on_image into bins\nbin_counts = [40] #, 20]\ndigitize_cols = ['image_per_patient'] #, 'pe_present_on_image']\nnon_digitize_cols = [c for c in train_per_patient_char.columns if c not in digitize_cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, c in enumerate(digitize_cols):\n    bin_count = bin_counts[i]\n    percentiles = np.percentile(train_per_patient_char[c], q=np.arange(bin_count)/bin_count*100.)\n    #print(percentiles)\n    print(train_per_patient_char[c].value_counts())\n    train_per_patient_char[c+'_digitize'] = np.digitize(train_per_patient_char[c], percentiles, right=False)\n    print(train_per_patient_char[c+'_digitize'].value_counts())\n    plt.hist(train_per_patient_char[c+'_digitize'], bins=bin_count)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_per_patient_char['key'] = train_per_patient_char[digitize_cols[0]+'_digitize'].apply(str)\nfor c in digitize_cols[1:]:\n    train_per_patient_char['key'] = train_per_patient_char['key']+'_'+train_per_patient_char[c+'_digitize'].apply(str)\n\ntrain_per_patient_char['key'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nfolds = FOLD_NUM\nkfolder = StratifiedKFold(n_splits=folds, shuffle=True, random_state=719)\nval_indices = [val_indices for _, val_indices in kfolder.split(train_per_patient_char['key'], train_per_patient_char['key'])]\n\ntrain_per_patient_char['fold'] = -1\nfor i, vi in enumerate(val_indices):\n    patients = train_per_patient_char.index[vi]\n    train_per_patient_char.loc[patients, 'fold'] = i\ntrain_per_patient_char['fold'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check each fold for the distribution of the number of images per patients\nfor col in digitize_cols:\n    fig, axs = plt.subplots(nrows=4, ncols=int(np.floor(folds/4)), constrained_layout=False, sharex=True, sharey=True)\n    fig.set_figheight(10)\n    fig.set_figwidth(20)\n    axs = axs.flat\n    for i, vi in enumerate(val_indices):\n        patients = train_per_patient_char.index[vi]\n        axs[i].set_title(col+' fold_'+str(i))\n        axs[i].hist(train_per_patient_char.loc[patients, col], bins=20, range=(train_per_patient_char[col].min(), train_per_patient_char[col].max()))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check each fold for the target distribution\nfor col in non_digitize_cols:\n    fig, axs = plt.subplots(nrows=4, ncols=int(np.floor(folds/4)), constrained_layout=False, sharex=True, sharey=True)\n    fig.set_figheight(10)\n    fig.set_figwidth(20)\n    axs = axs.flat\n    for i, vi in enumerate(val_indices):\n        patients = train_per_patient_char.index[vi]\n        axs[i].set_title(col+' fold_'+str(i))\n        axs[i].hist(train_per_patient_char.loc[patients, col], bins=20, range=(train_per_patient_char[col].min(), train_per_patient_char[col].max()))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_per_patient_char.to_csv('rsna_train_splits_fold_{}.csv'.format(FOLD_NUM))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Each fold looks similar in the distribution of the number of images per patient now"},{"metadata":{},"cell_type":"markdown","source":"**Further usage of this kernel:**\n* You could use the output csv directly to do patient level subsampling (ex. select fold=1-5 to do 5 fold cross-validation) \n* You could modify FOLD_NUM above to create different number of stratified folds yourself \n* You could modify bin_counts+digitize_cols above to digitize columns with designated bin counts, which will be futher incorporated into the new \"key\" to do the stratified validation splits"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}