{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Embolism Detection EDA"},{"metadata":{},"cell_type":"markdown","source":"This is an EDA (exploratory data analysis) for the newly launched RNSA **Embolism Detection** competition on Kaggle that we're going to be working on today. An **embolism** is caused when your arteries are blocked off in your lung, preventing blood flow and stopping your lung from getting the oxygen it needs to carry out respiration. It is the most fatal cardiovascular disease in the United States of America (60,000 to 100,000 deaths per annum)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np, pandas as pd, pydicom as dcm\nimport keras\nimport tensorflow as tf\nimport matplotlib.pyplot as plt, seaborn as sns\nimport os, glob\ntr = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/train.csv\")\nte = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/test.csv\")\nTRAIN_PATH = \"../input/rsna-str-pulmonary-embolism-detection/train/\"\nfiles = glob.glob('../input/rsna-str-pulmonary-embolism-detection/train/*/*/*.dcm')\ndef dicom_to_image(filename):\n    im = dcm.dcmread(filename)\n    img = im.pixel_array\n    img[img == -2000] = 0\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Starting the EDA"},{"metadata":{},"cell_type":"markdown","source":"First of all, let's do a small preliminary check to our data. Take a look at how many scans we have per folder (how many CT slices we have per patient, in essence). Kaggle says this dataset is almost 1 TB, so training would be quite the conundrum on this full dataset."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print('Total patients {}'.format(len(os.listdir(TRAIN_PATH))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So it seems like we have an abundance of patients and an (over?)abundance of training data that we can utilize to feed our EfficientNets. Let's take a look at one of our DICOM files, just to take a brief look."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.axis('off')\nplt.imshow(-dcm.dcmread(\"../input/rsna-str-pulmonary-embolism-detection/train/0003b3d648eb/d2b2960c2bbf/00ac73cfc372.dcm\").pixel_array);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that the colors in the image above have been inverted by me for maximum visibility. This data format seems close to the images in the Data Science Bowl 2017 and the recent OSIC competition, and not too close to other lung-based competitions (the SIIM-ACR Pneumothorax Competition used larger images and was a segmentation task)."},{"metadata":{},"cell_type":"markdown","source":"# Meta-exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"tr.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now this competition is a little (or a lot depending on your point of view) odd because of the submission scoring and the weighted log loss. In essence, our submission.csv file should have a number of rows equal to $n_{img} + (n_{studies} * n_{labels})$ where our $n$ basically serves as an indicator of \"number of\" and our metric is weighted on the basis of there being more important labels than others. They are giving the most importance with their weighted metric to \"Central PE\". Now what is this central PE you might ask?\n"},{"metadata":{},"cell_type":"markdown","source":"These are the principal data fields given by Kaggle on the competition's data page:\n```\n+ StudyInstanceUID - unique ID for each study (exam) in the data.\n+ SeriesInstanceUID - unique ID for each series within the study.\n+ SOPInstanceUID - unique ID for each image within the study (and data).\n+ pe_present_on_image - image-level, notes whether any form of PE is present on the image.\n+ negative_exam_for_pe - exam-level, whether there are any images in the study that have PE present.\n+ qa_motion - informational, indicates whether radiologists noted an issue with motion in the study.\n+ qa_contrast - informational, indicates whether radiologists noted an issue with contrast in the study.\n+ flow_artifact - informational\n+ rv_lv_ratio_gte_1 - exam-level, indicates whether the RV/LV ratio present in the study is >= 1\n+ rv_lv_ratio_lt_1 - exam-level, indicates whether the RV/LV ratio present in the study is < 1\n+ leftsided_pe - exam-level, indicates that there is PE present on the left side of the images in the study\n+ chronic_pe - exam-level, indicates that the PE in the study is chronic\n+ true_filling_defect_not_pe - informational, indicates a defect that is NOT PE\n+ rightsided_pe - exam-level, indicates that there is PE present on the right side of the images in the study\n+ acute_and_chronic_pe - exam-level, indicates that the PE present in the study is both acute AND chronic\n+ central_pe - exam-level, indicates that there is PE present in the center of the images in the study\n+ indeterminate -exam-level, indicates that while the study is not negative for PE, an ultimate set of exam-level labels could not be created, due to QA issues\n```"},{"metadata":{},"cell_type":"markdown","source":"Now, with the following data, it's possible to see that there are defects in a lung that are not a pulmonary embolism and are perhaps the effects of another pulmonary disease."},{"metadata":{},"cell_type":"markdown","source":"# Image-based analysis"},{"metadata":{},"cell_type":"markdown","source":"Time to get what all those who came here were looking for - the images."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f, plots = plt.subplots(4, 5, sharex='col', sharey='row', figsize=(10, 8))\nfor i in range(20):\n    plots[i // 5, i % 5].axis('off')\n    plots[i // 5, i % 5].imshow(dicom_to_image(np.random.choice(files[:1000])), cmap=plt.cm.bone) # last 1k images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images all look roughly similar to the Data Science Bowl and to the OSIC competition's images, which is helpful as we can apply existing preprocessing techniques(see \"Full Preprocessing Tutorial\" and \"Pulmonary Fibrosis Preprocessing\".)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}