{"cells":[{"metadata":{},"cell_type":"markdown","source":"We had some problems with fastaiv2, and therefore shifted back to the original version of fastai. Imports below."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\nimport os\nimport numpy\nimport pandas as pd\nfrom matplotlib import pyplot, cm\nfrom fastai.vision import *\nimport fastai","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Read in the data. I will be working with the original data."},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '../input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection'\nTRAIN_IMAGES_DIR = DATA_DIR + '/stage_2_train/'\nTRAIN_CSV_DIR = DATA_DIR + '/stage_2_train.csv'\nTEST_IMAGES_DIR = DATA_DIR + '/stage_2_test/'\nTEST_CSV_DIR = DATA_DIR + '/stage_2_sample_submission.csv'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at what we begin with."},{"metadata":{"trusted":true},"cell_type":"code","source":"initial_df = pd.read_csv(TRAIN_CSV_DIR)\ninitial_table = initial_df.copy()\nprint(initial_table.shape)\ninitial_table.head(n=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that this table has an ID for each image and the various ICH types, and what the label is corresponding to this. We want to clean this up a bit. Explanation is commented out."},{"metadata":{"trusted":true},"cell_type":"code","source":"#gets rid of the ID at the beginning, now we have the name of the sample and the ICH type together as label\nnew1 = initial_table[\"ID\"].str.split(\"_\", n = 1, expand = True)\n#further splits between name of sample and ICH type\nnew2 = new1[1].str.split(\"_\", n = 1, expand = True)\n\n#add values in the new2 table to the initial_table we were working with\ninitial_table['Image_ID'] = new2[0]\ninitial_table['Sub_type'] = new2[1]\ninitial_table.head(n=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#now extract the image IDs, we have the column set up so we can extract these\nimage_ids = initial_table.Image_ID.unique()\n#make an empty list for each unique image id\ntypelabel = [\"\" for _ in range(len(image_ids))]\n#manipulate to a dataframe with each row being a array with the id and the list of labels, not filled yet\ncleaned_df = pd.DataFrame(np.array([image_ids, typelabel]).transpose(), columns=[\"case\", \"label\"])\ncleaned_df.head(n=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#make an object type dictionary for each image id\nlbls = {i : \"\" for i in image_ids}\n#reduces initial_table to only be values where label is 1, ie there is an ICH\ninitial_table = initial_table[initial_table.Label == 1]\n#reduces initial_table, gets rid of cases where any = 1, which is redundant, but we could also edit this\ninitial_table = initial_table[initial_table.Sub_type != \"any\"]\n\n#fill the lbls dictionary object with the ICH if there is one, a blank if there isnt\ni = 0\nfor name, group in initial_table.groupby(\"Image_ID\"):\n    lbls[name] = \" \".join(group.Sub_type)\n    i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#now fill the labels column in the cleaned df to be blank if no ICH, have the type if there is\ncleaned_df = pd.DataFrame(np.array([list(lbls.keys()), list(lbls.values())]).transpose(), \n                          columns=[\"case\", \"label\"])\ncleaned_df.head(n=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some functions for viewing images, getting dicom data, extracting slope and intercept. We originally developed these when we were considering training using the whole dataset without the fastai framework. Citation for the idea of gradient sigmoid windowing: https://www.kaggle.com/reppic/gradient-sigmoid-windowing. Not all these functions may be needed."},{"metadata":{"trusted":true},"cell_type":"code","source":"#loads 5 random images of the different types and 5 images without ICHs\ndef load_random_images():\n    image_names = [list(train[train[h_type] == 1].sample(1)['filename'])[0] for h_type in hem_types]\n    image_names += list(train[train['any'] == 0].sample(5)['filename'])\n    return [pydicom.read_file(os.path.join(TRAIN_IMG_PATH, img_name)) for img_name in image_names]\n\n#function to view the images we sampled\ndef view_images(images):\n    width = 5\n    height = 2\n    fig, axs = plt.subplots(height, width, figsize=(15,5))\n    \n    for im in range(0, height * width):\n        image = images[im]\n        i = im // width\n        j = im % width\n        axs[i,j].imshow(image, cmap=plt.cm.bone) \n        axs[i,j].axis('off')\n        title = hem_types[im] if im < len(hem_types) else 'normal'\n        axs[i,j].set_title(title)\n\n    plt.show()\n\ndef get_first_of_dicom_field_as_int(x):\n       #get x[0] as in int is x is a 'pydicom.multival.MultiValue', otherwise get int(x)\n    if type(x) == pydicom.multival.MultiValue:\n        return int(x[0])\n    else:\n        return int(x)\n\n#gets the window center, width, intercept from dicom data    \ndef get_windowing(data):\n    dicom_fields = [data[('0028','1050')].value, #window center\n                    data[('0028','1051')].value, #window width\n                    data[('0028','1052')].value, #intercept\n                    data[('0028','1053')].value] #slope\n    return [get_first_of_dicom_field_as_int(x) for x in dicom_fields]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the windowing function we developed after a lot of reading of how ICH windowing is done. The default function is one that sets all the values above the window to the max value of the window, and the values below the window to the min value of the window. If exclusive = true, all values above the window will be set to the min value of the window instead. See the equation below, x and b are slope and intercept, WW = window width, WL = window center. From: https://arxiv.org/pdf/1812.00572.pdf."},{"metadata":{"trusted":true},"cell_type":"code","source":"#what is the purpose of rescale??? Setting to false for now\ndef window_image_matteo(img, window_center = None, window_width = None, rescale = False, exclusive = False):\n    if window_center is None and window_width is None:\n        window_center, window_width , intercept, slope = get_windowing(img)\n    else:\n        _, _, intercept, slope = get_windowing(img)\n    #img = img.pixel_array\n    img = (img*slope +intercept)\n    img_min = window_center - window_width//2\n    img_max = window_center + window_width//2\n    img[img<img_min] = img_min\n    if exclusive:\n        img[img>img_max] = img_min\n    else:\n        img[img>img_max] = img_max\n\n    if rescale:\n        # Extra rescaling to 0-1, not in the original notebook\n        img = (img - img_min) / (img_max - img_min)\n    \n    return img\n\ndef window_image(img, window_center,window_width, intercept, slope):\n\n    img = (img*slope +intercept)\n    img_min = window_center - window_width//2\n    img_max = window_center + window_width//2\n    img[img<img_min] = img_min\n    img[img>img_max] = img_max\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#we need IDs in our dataframe to match up with the image names, so lets convert them to be such\ncleaned_df.case = \"ID_\" + cleaned_df.case + \".dcm\"\ncleaned_df.head(n=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#this windows the image that is opened as needed to be passed into the training framework\ndef new_open_image(path, convert_mode=None, after_open=None):\n    dcm = pydicom.dcmread(str(path))\n    window_center, window_width, intercept, slope = get_windowing(dcm)\n    im = window_image(dcm.pixel_array, window_center, window_width, intercept, slope)\n    im = np.stack((im,)*3, axis=-1)\n    im -= im.min()\n    im_max = im.max()\n    if im_max != 0: im = im / im.max()\n    x = Image(pil2tensor(im, dtype=np.float32))\n    #if div: x.div_(2048)  # ??\n    return x\n\nvision.data.open_image = new_open_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#sampling a 50/50 split of 5,000 total images, half have labels, half do not\ndf_train = pd.concat([cleaned_df[cleaned_df.label == \"\"][:2500], cleaned_df[cleaned_df.label != \"\"][:2500]])\n\ndf_train.head(n=5)\ndf_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#this will make a list of images based on the subsetted training dataframe, type: fastai.vision.data.ImageList\nim_list = ImageList.from_df(df_train, path=TRAIN_IMAGES_DIR)\n#prepares a list of test names, and takes test images based on that\ntest_fnames = pd.DataFrame(\"ID_\" + pd.read_csv(TEST_CSV_DIR)[\"ID\"].str.split(\"_\", n=2, expand = True)[1].unique() + \".dcm\")\ntest_im_list = ImageList.from_df(test_fnames, TEST_IMAGES_DIR)\n\ntfms = get_transforms(do_flip=False)\nbs = 128\ndata = (im_list.split_by_rand_pct(0.2)\n               .label_from_df(label_delim=\" \")\n               .transform(tfms, size=512)\n               .add_test(test_im_list)\n               .databunch(bs=bs, num_workers=0)\n               .normalize())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, models.resnet18)\n\nmodels_path = Path(\"/kaggle/working/models\")\nif not models_path.exists(): models_path.mkdir()\n    \nlearn.model_dir = models_path\nlearn.metrics = [accuracy_thresh]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#learn.fit_one_cycle(1)\n#learn.recorder.plot()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":1}