{"cells":[{"metadata":{},"cell_type":"markdown","source":"This tutorial is for helping you kickstart this competition using fast.ai V1 library. In case you are not aware of this library please have a look at it here. It is really awesome and you can get good results really fast using this library.   "},{"metadata":{},"cell_type":"markdown","source":"## Import packages"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom # for reading .dcm images \nimport os\nimport numpy\nfrom matplotlib import pyplot, cm\nfrom fastai.vision import *\nimport fastai\nfrom IPython.display import FileLink","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Code for Reading DCM images"},{"metadata":{},"cell_type":"markdown","source":"Note that fast.ai library v1 does not natively support reading .dcm images. Therefore, I have made some modifications to the open image function to help us read .dcm images. Also some rescaling is required for these .dcm images, so I have incorporated that also in the code. You can read more about it here http://www.idlcoyote.com/fileio_tips/hounsfield.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"# function to open dcm images\ndef open_dcm_image(fn:PathOrStr, div:bool=True, convert_mode:str='L', cls:type=Image,\n        after_open:Callable=None)->Image:\n    \"Return `Image` object created from image in file `fn`.\"\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\", UserWarning) # EXIF warning from TiffPlugin\n        #x= PIL.Image.open(fn).convert(convert_mode) # previous code\n        # code added for opening dcm images\n        dicom_file = pydicom.dcmread(str(fn))\n        arr = dicom_file.pixel_array.copy() \n        arr = arr * int(dicom_file.RescaleSlope) + int(dicom_file.RescaleIntercept) \n        level = 40; window = 80\n        arr = np.clip(arr, level - window // 2, level + window // 2)\n        x = PIL.Image.fromarray(arr).convert(convert_mode)\n    if after_open: x = after_open(x)\n    x = pil2tensor(x,np.float32)\n    if div: x.div_(255)\n    return cls(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# modifying open_image function of fast.ai\nfastai.vision.data.open_image = open_dcm_image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparation of Input Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"dirpath = \"../input/rsna-intracranial-hemorrhage-detection/\"\n!ls ../input/rsna-intracranial-hemorrhage-detection","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read train Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# read train csv that has filename and the corresponding labels \ndf_train = pd.read_csv('../input/rsna-intracranial-hemorrhage-detection/stage_1_train.csv')\ndf_train['fn'] = df_train.ID.apply(lambda x: '_'.join(x.split('_')[:2]) + '.dcm')\ndf_train['label'] = df_train.ID.apply(lambda x: x.split('_')[-1])\ndf_train.head()\n# it was pointed out in the discussion that there is one file that is corrupred\n# removing the corrupted file\ndf_train = df_train[df_train.fn != 'ID_000039fa0.dcm']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display images\nopen_dcm_image( dirpath+\"stage_1_train_images/\"+df_train.fn.values[5],convert_mode= 'L').show(cmap= 'gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_train.shape)\ndf_train.drop_duplicates(inplace = True)\nprint(df_train.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a typical multilabel classification problem. In the train set csv, the data is arranged such that for each image and label (type of haemorrage) combination we have tag that particular type is present or not in image. I will rearrange this data to have all the labels presnt for that image in one row instead of multiple. This is particularly done so that we can load the data using fastai functions."},{"metadata":{"trusted":true},"cell_type":"code","source":"pivot = df_train.pivot(index='fn', columns='label', values='Label')\npivot.reset_index(inplace=True)\n# chcek if there are only two types of values in any\nassert pivot[pivot['any'] == 0].shape[0] + pivot[pivot['any'] == 1].shape[0] == pivot.shape[0] \npivot['any'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = pivot['any'] == 0\npivot['None'] = \"\"\npivot.loc[mask, 'None'] = 'None'\n\nlabel_cols = ['any', 'epidural', 'intraparenchymal', 'intraventricular','subarachnoid', 'subdural', 'None'] \nfor col in label_cols:\n    print(col, end= \", \")\n    pivot[col] = pivot[col].replace({0:\"\", 1:col})\n    \npivot['MultiLabel'] = pivot[label_cols].apply(lambda x: \" \".join((' '.join(x)).split()), axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read Test data"},{"metadata":{},"cell_type":"markdown","source":"similarly we have read the test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv('../input/rsna-intracranial-hemorrhage-detection/stage_1_sample_submission.csv')\ndf_test['fn'] = df_test.ID.apply(lambda x: '_'.join(x.split('_')[:2]) + '.dcm')\ndf_test['label'] = df_test.ID.apply(lambda x: x.split('_')[-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pivot_test = df_test.pivot(index='fn', columns='label', values='Label')\npivot_test.reset_index(inplace=True)\npivot_test['MultiLabel'] = \" \"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Image databunch"},{"metadata":{},"cell_type":"markdown","source":"Now we will load the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get_transforms is for data augmentation, however I am not using this as of now\n#tfms = get_transforms(do_flip = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/\"\nnp.random.seed(42)\ndata_train = (ImageList\n.from_df(path=path,df= pivot[['fn', 'MultiLabel']])\n.split_by_rand_pct()\n.label_from_df(cols=1, label_delim = \" \")\n.transform(size=(128,128))\n.databunch()\n.normalize(imagenet_stats))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/rsna-intracranial-hemorrhage-detection/stage_1_test_images/\"\ndata_test = (ImageList\n.from_df(path=path,df= pivot_test[['fn', 'MultiLabel']])\n.split_by_rand_pct(valid_pct= 0)\n.label_from_df(cols=1, label_delim = \" \")\n.transform(size=(128,128))\n.databunch()\n.normalize(imagenet_stats))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert len(data_test.train_ds.y) == pivot_test.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3, figsize=(12,9))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train model"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/output/'\nacc_02 = partial(accuracy_thresh, thresh=0.2)\nf_score = partial(fbeta, thresh=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner( data_train, models.resnet34, path = path, metrics = [acc_02, f_score] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 3e-3\nlearn.freeze()\nlearn.fit_one_cycle(2, slice(lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('stage-1')\nlearn.load('stage-1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(1, slice(3e-5, 3e-4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Get Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the classed from the learner class\n# This will give us the order in which predictions were given by the model\ndata_classes = learn.data.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading test dataset \nlearn.data = data_test\n# get prediction from the model\ny_predict, _ = learn.get_preds(ds_type=DatasetType.Fix)\n# check we got predictions for all test files\nassert len(y_predict) == pivot_test.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.data.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pivot_test['MultiLabel']  = y_predict\n# Now loading the prediction from multilabel column to repsective labels\nfor i, col in enumerate(data_classes):\n    print(col, end= \", \")\n    pivot_test[col] = pivot_test['MultiLabel'].apply(lambda x: x[i].numpy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now rearranging the predictions to the competitons format\ncols_to_consider = [col for col in pivot_test.columns if not col in ['None', 'MultiLabel']]\nprint(cols_to_consider)\ndf_temp = pd.melt(pivot_test[cols_to_consider], id_vars= ['fn'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_temp['ID'] = df_temp['fn'].apply(lambda x: x[:-4])\ndf_temp['ID'] = df_temp[['ID', 'label']].apply(lambda x: \"_\".join(x), axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert len(set(df_test.ID.unique()).intersection(set(df_temp.ID.unique()))) == df_test.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_temp.rename(columns={'value':'Label'}, inplace = True)\ndf_temp[['ID', 'Label']].to_csv(\"submission6.gz\", compression = 'gzip' , index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# this will generate a link using which you can download your predictions\nFileLink(\"submission6.gz\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}