{"cells":[{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"**Data Exploration**\n\nBefore going furthur into machine learning step, let's explore our dataset.\nI believe this EDA will help us plan out how to clean up data, preprocess image and design good neural network for good classification."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nimport pydicom\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"PATH = '/kaggle/input/rsna-intracranial-hemorrhage-detection/'\nraw_train = pd.read_csv(PATH + 'stage_1_train.csv')\nraw_train.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"null_num = pd.isnull(raw_train.Label).values.any()\nna_num = pd.isna(raw_train.Label).values.any()\nprint('isNull:', null_num)\nprint('isNA:', null_num)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you seen from following table, our raw data consists of 4045572 rows of 2 columns of ID and label.\n\nThere is no null or n/a cell.\n\nUmmm ... However, image subtypes are packed image id in the first column. Let's break down this column and convert this dataframe to a form that easier to analyze."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = raw_train.copy()\ntrain['Subtype'] = train['ID'].apply(lambda st: st.split('_')[2])\ntrain['ID'] = train['ID'].apply(lambda st: \"ID_\" + st.split('_')[1])\ntrain = train[['ID','Subtype','Label']]\ntrain.head(6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we get dataframe that enough to visualize the distribution of label in each subtype.\nBased on the following bar plot, it seems this dataset is 2-class classification (yes/no classification ) of 5 subtypes + unidentified (any) subtype.\n\nThe number of negative images (label = 0)in each subtype is obviously higher than the number of possitive images especially epidural subtype. It is worth noting that number of possitive images in each subtype mean there is negative to the other subtypes. One image can have than one subtypes. In other words, the number of images which pos sitive to one subtype is very small. Is these positive to negative images ratios good for training our model? Should we pass all of these data to our model?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.set(\"notebook\",\n font_scale=2.0, \n rc={'axes.facecolor':'white', \n 'figure.facecolor':'white',\n \"grid.linewidth\": 1,\n 'grid.color': 'gray'})\n\nplt.figure(figsize = (18,6),dpi=250)\nax = sns.countplot(x=\"Subtype\", hue=\"Label\", data=train, palette=\"ocean\", saturation=1.0)\nax.set_xticklabels(ax.get_xticklabels(), rotation=30)\nax.legend(title='Label', bbox_to_anchor=(0.5, 0.0, 0.65, 0.9))\nax.set_xlabel('Subtype',fontdict={'fontweight':'bold'})\nax.set_ylabel('Count',fontdict={'fontweight':'bold'})\nfor p in ax.patches:\n ax.annotate(p.get_height(), (p.get_x()+0.05, p.get_height()+20000), fontsize=14)\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the sake of demonstration, I try to break down images in each subtypes to 4 groups including \n* positype to subtype of interest/ negative to other subtypes (+/+)\n* positype to subtype of interest/ positive to other subtypes (+/-)\n* negative to subtype of interest/ positive to other subtypes (-/+)\n* negative to subtype of interest/ positive to other subtypes (-/-).\n\nThen I will show the percentage of each group with some images of *positive to subtype of interest/ negative to other subtypes* ."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reshape dataframe\ntrain_pivot = train.drop_duplicates().pivot(index='ID', columns = 'Subtype', values = 'Label').reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# list subtypes name to array list\nsubtype=train_pivot.select_dtypes(include=int).columns.tolist()\nsubtype.remove('any')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count number of subtype positive in each image\ntrain_pivot['subtype_num'] = train_pivot[subtype].sum(axis=1, skipna=True)\n# show head of new dataframe\ntrain_pivot.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read number of images\ndata_num = len(train_pivot)\nprint('Total number of ct image = ', data_num)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# separate negative to subtype of interest/ positive to other subtypes (all negative)and create new dataframe\nall_negative = train_pivot.loc[train_pivot['subtype_num']==0].reset_index(drop=True)\nall_negative['group'] = 'This subtype : - , Others subtypes: -'\nall_negative.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# separate dataframe of each subtype\ngbl = globals()\nfor i in subtype:\n # separate positive to subtype of interest/ positive to other subtypes\n gbl[i+'_positive_others_positive']=train_pivot.loc[(train_pivot[i]==1)&(train_pivot['subtype_num']>1)].reset_index(drop=True)\n gbl[i+'_positive_others_positive']['group'] = 'This subtype : + , Others subtypes: +'\n # separate negative to subtype of interest/ positive to other subtypes\n gbl[i+'_positive_others_negative']=train_pivot.loc[(train_pivot[i]==1)&(train_pivot['subtype_num']==1)].reset_index(drop=True)\n gbl[i+'_positive_others_negative']['group'] = 'This subtype : + , Others subtypes: -'\n # separate positive to subtype of interest/ negative to other subtypes\n gbl[i+'_negative_others_positive']=train_pivot.loc[(train_pivot[i]==0)&(train_pivot['subtype_num']>=1)].reset_index(drop=True)\n gbl[i+'_negative_others_positive']['group'] = 'This subtype : - , Others subtypes: +'\n # merge 4 groups into one dataframe\n gbl[i] = pd.concat([gbl[i+'_positive_others_positive'],\n gbl[i+'_positive_others_negative'], \n gbl[i+'_negative_others_positive'],\n all_negative])\n # rename dataframe\n gbl[i].name = i","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is an example of epidural dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"epidural","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, I will extract some images of +/+ group. I use Pydicom to read information from ct file, dicom file.\n\nThanks Richard McKinley (https://www.kaggle.com/omission/eda-view-dicom-images-with-correct-windowing) and Aldo Von Wangenheim (https://www.researchgate.net/post/Deep_Learning_What_is_the_best_way_to_to_feed_dicom_files_into_object_detection_algorithm). I learn from them that the raw pixel array of these ct images are scaled data. That 's why images without processing look similar, most of them have gray circle with low resolution of a brain inside. We have to read slope and intercep of each image, then use them to transform those scaled pixels to Hounsfield units.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# read information of window from dicom file\ndef read_value(dicom, tag):\n value = dicom[tag].value\n if type(value) == pydicom.multival.MultiValue:\n return int(value[0])\n else:\n return int(value)\n \ndef read_window(dicom):\n tags = [('0028','1050'), #window center tag\n ('0028','1051'), #window width tag\n ('0028','1052'), #intercept tag\n ('0028','1053')] #slope tag\n return [read_value(dicom, x) for x in tags]\n\n# use window properties to convert scaled pixel array to Hounsfield unit\ndef apply_linear_transform(pixel_array, center, width, intercept, slope):\n HU = (pixel_array*slope) + intercept\n HU_max = center + (width//2)\n HU_min = center - (width//2)\n HU[HU>HU_max] = HU_max \n HU[HU