{"cells":[{"metadata":{},"cell_type":"markdown","source":"Competition Overview:\n\n> If every breath is strained and painful, it could be a serious and potentially life-threatening condition. A pulmonary embolism (PE) is caused by an artery blockage in the lung. It is time consuming to confirm a PE and prone to overdiagnosis. Machine learning could help to more accurately identify PE cases, which would make management and treatment more effective for patients.\nCurrently, CT pulmonary angiography (CTPA), is the most common type of medical imaging to evaluate patients with suspected PE. These CT scans consist of hundreds of images that require detailed review to identify clots within the pulmonary arteries. As the use of imaging continues to grow, constraints of radiologists’ time may contribute to delayed diagnosis.\nThe Radiological Society of North America (RSNA®) has teamed up with the Society of Thoracic Radiology (STR) to help improve the use of machine learning in the diagnosis of PE.\nIn this competition, you’ll detect and classify PE cases. In particular, you'll use chest CTPA images (grouped together as studies) and your data science skills to enable more accurate identification of PE. If successful, you'll help reduce human delays and errors in detection and treatment.\nWith 60,000-100,000 PE deaths annually in the United States, it is among the most fatal cardiovascular diseases. Timely and accurate diagnosis will help these patients receive better care and may also improve outcomes.\n\n\n![Image](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F603584%2F9a3aac7e7ac865f134201cc2a5cd52f3%2Fkaggle_header3.png?generation=1599585319459400&alt=media)"},{"metadata":{},"cell_type":"markdown","source":"> Let's now explore the data! :)"},{"metadata":{},"cell_type":"markdown","source":"## Importing essential Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nsns.set(style = \"darkgrid\")\nimport pydicom as dcm\nimport matplotlib.cm as cm\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading and Understanding the files"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/train.csv\")\ndf_test = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/test.csv\")\ndf_sub = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Let's now talk a little bit about train.csv file."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head().T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Shape of train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> It seems 3 features are object and all other are int. Let's see the cardinality of remaining features."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_cols = ['pe_present_on_image', 'negative_exam_for_pe', 'qa_motion',\n       'qa_contrast', 'flow_artifact', 'rv_lv_ratio_gte_1', 'rv_lv_ratio_lt_1',\n       'leftsided_pe', 'chronic_pe', 'true_filling_defect_not_pe',\n       'rightsided_pe', 'acute_and_chronic_pe', 'central_pe', 'indeterminate']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count = 0\nfor col in train_cols:\n    if len(df_train[col].value_counts())==2:\n        count += 1\nif count==len(train_cols):\n    print(\"All the features other than UID's are Binary features.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Okay, so now, We know that All of the remaining features from training file are binary features."},{"metadata":{},"cell_type":"markdown","source":"> Let's now plot the two categories of all features side by side and see the count of values."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_grid(cols = train_cols):\n    fig=plt.figure(figsize=(8, 22))\n    columns = 2\n    rows = 7\n    for i in range(1, columns*rows + 1):\n        col = cols[i-1]\n        fig.add_subplot(rows, columns, i)\n        df_train[col].value_counts().plot(kind = \"bar\", color = \"Purple\", alpha = 0.4)\n        count_0 = df_train[col].value_counts()[0]\n        count_1 = df_train[col].value_counts()[1]\n        ratio = count_0/count_1\n        plt.xlabel(f\"Feature name: {col}\\n Count 0: {count_0}\\n Count 1: {count_1}\\n Ratio(0:1): {ratio:.1f}:1\")\n    plt.tight_layout()\n    plt.show()\n\nplot_grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> So with this plot, we can now see that the classes are not that balanced in between all those features.\n> Many of them have count of \"0\" to be very high in comparision to \"1\" and vice versa. "},{"metadata":{},"cell_type":"markdown","source":"> Let's see the total number of unique values in \"StudyInstanceUID\", \"SeriesInstanceUID\" and \"SOPInstanceUID\"."},{"metadata":{},"cell_type":"markdown","source":"> Let's now see the correlation between all the binary features available in the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"corr_mat = df_train[train_cols].corr()\nmask = np.triu(np.ones_like(corr_mat, dtype=bool))\nf, ax = plt.subplots(figsize=(14, 12))\nsns.heatmap(corr_mat, mask = mask, cmap = \"summer\", annot = True, vmax = 0.3, square = False, linewidths = 0.5, center = 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Almost all of the features seems to be containing unique information which is good for us!"},{"metadata":{"trusted":true},"cell_type":"code","source":"def details_first_three(df = df_train):\n    print(f\"Number of unique entries in StudyInstanceUID: {len(df.StudyInstanceUID.value_counts())}\")\n    print(f\"Number of unique entries in SeriesInstanceUID: {len(df.SeriesInstanceUID.value_counts())}\")\n    print(f\"Number of unique entries in SOPInstanceUID: {len(df.SOPInstanceUID.value_counts())}\")\ndetails_first_three()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Now, we can see that the number of uniques entries for both Study and Series Instance UID is same."},{"metadata":{},"cell_type":"markdown","source":"> Also, we know from the competition's data overview that SOPInstanceUID is a Unique Identifier for a image and to verify it, we can see that the number of unique SOP values is same as length of train data."},{"metadata":{},"cell_type":"markdown","source":"> Now, Let's talk a little about test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Shape of test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Unique values for first three features."},{"metadata":{"trusted":true},"cell_type":"code","source":"details_first_three(df_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Seems like the case for Study and Series Instance UID is same here as train."},{"metadata":{},"cell_type":"markdown","source":"> Let's now see what Submission files looks like and what do we need to predict?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Seems like we need to predict some \"label\" for the entries from test file."},{"metadata":{},"cell_type":"markdown","source":"> Shape of Submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What do we predict?"},{"metadata":{},"cell_type":"markdown","source":"> Now, we know by the Overview page of this competition that : \n\n> *In this competition we are predicting a number of labels, at both the image and study level. Note that some labels are logically mutually exclusive.*"},{"metadata":{},"cell_type":"markdown","source":"> Let's now check what these labels really are."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub[\"label_features\"] = df_sub.id.apply(lambda x: \"_\".join(x.split(\"_\")[1:]))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"df_sub.label_features[df_sub.label_features == \"\"] = \"UID\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub.label_features.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Now, we can see that we need to predict values for the above mentioned labels other than the UID itself.\n\nActually, this verifies the information provided by Jebastin Nadar in a discussion that:\n\n> *Each study has multiple images. We have to predict labels for images as well as studies.*\n    \n> *Image level - predict for each image i.e SOPInstanceUID*\n  \n> *Labels to predict : pe_present_on_image*\n    \n> *Study level - predict for each study i.e StudyInstanceUID*\n    \n> *Labels to predict : negative_exam_for_pe , indeterminate, rv_lv_ratio_gte_1, rv_lv_ratio_lt_1, leftsided_pe, rightsided_pe, central_pe, chronic_pe, acute_and_chronic_pe*\n"},{"metadata":{},"cell_type":"markdown","source":"## Visualizing some of the Scans"},{"metadata":{},"cell_type":"markdown","source":"> Now, finally, let's see some of the images provided to us in train and test folders."},{"metadata":{},"cell_type":"markdown","source":"> Let's pick some random image addresses."},{"metadata":{"trusted":true},"cell_type":"code","source":"img_addr = [\"../input/rsna-str-pulmonary-embolism-detection/train/0003b3d648eb/d2b2960c2bbf/00ac73cfc372.dcm\", \n           \"../input/rsna-str-pulmonary-embolism-detection/train/005df0f53614/5e0e0d0b7a65/081c2fa491a1.dcm\",\n           \"../input/rsna-str-pulmonary-embolism-detection/train/0072baad76be/d555455a1dc2/096497b1da4e.dcm\", \n           \"../input/rsna-str-pulmonary-embolism-detection/train/00d4f4409f0c/38a51605b9ab/079e029c0d1a.dcm\", \n           \"../input/rsna-str-pulmonary-embolism-detection/test/00e7015490cb/291c07d4a7c0/09c25538116c.dcm\", \n            \"../input/rsna-str-pulmonary-embolism-detection/test/0227030d6278/599fccda6e2b/0c247bfd9c27.dcm\", \n           \"../input/rsna-str-pulmonary-embolism-detection/train/00102474a2db/c1a6d49ce580/06ce8f7a39ae.dcm\", \n           \"../input/rsna-str-pulmonary-embolism-detection/train/00102474a2db/c1a6d49ce580/0fd29873e8e4.dcm\",\n           \"../input/rsna-str-pulmonary-embolism-detection/train/00617c9fe236/16ed05bf3395/01d00e27c5ac.dcm\", \n            \"../input/rsna-str-pulmonary-embolism-detection/test/08115e1b649d/f69e3f9c7067/10ba32beefb2.dcm\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_image_grid(addresses = img_addr):\n    fig=plt.figure(figsize=(12, 12))\n    columns = 5\n    rows = 2\n    for i in range(1, columns*rows + 1):\n        addr = addresses[i-1]\n        fig.add_subplot(rows, columns, i)\n        plt.imshow(dcm.dcmread(addr).pixel_array)\n        plt.axis(\"off\")\n    plt.tight_layout()\n    plt.show()\n\nplot_image_grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bonus: DICOM Metadata!"},{"metadata":{},"cell_type":"markdown","source":"* Bonus for reaching this far!\n\nNote: We can fetch the metadata details from the dcm image by using the following function (dicom_metadata)."},{"metadata":{"trusted":true},"cell_type":"code","source":"dicom_atts = [\"SpecificCharacterSet\",\"ImageType\",\"SOPInstanceUID\",\"Modality\",\"Manufacturer\", \"ManufacturerModelName\",\"PatientName\",\"PatientID\",\n             \"PatientSex\",\"DeidentificationMethod\",\"BodyPartExamined\",\"SliceThickness\", \"KVP\",\"SpacingBetweenSlices\",\"DistanceSourceToDetector\",\n              \"DistanceSourceToPatient\",\"GantryDetectorTilt\", \"TableHeight\",\"RotationDirection\",\"XRayTubeCurrent\",\"GeneratorPower\",\n              \"FocalSpots\",\"ConvolutionKernel\",\"PatientPosition\",\"RevolutionTime\", \"SingleCollimationWidth\",\"TotalCollimationWidth\",\"TableSpeed\",\"TableFeedPerRotation\",\n              \"SpiralPitchFactor\", \"StudyInstanceUID\",\"SeriesInstanceUID\",\"StudyID\",\"InstanceNumber\",\"PatientOrientation\",\n              \"ImagePositionPatient\",\"ImageOrientationPatient\",\"FrameOfReferenceUID\",\"PositionReferenceIndicator\",\n              \"SliceLocation\",\"SamplesPerPixel\",\"PhotometricInterpretation\", \"Rows\",\"Columns\",\"PixelSpacing\",\"BitsAllocated\",\"BitsStored\",\"HighBit\",\n              \"PixelRepresentation\",\"PixelPaddingValue\",\"WindowCenter\",\"WindowWidth\",\"RescaleIntercept\", \"RescaleSlope\",\"RescaleType\"]\n\nlist_attributes = [\"ImageType\",\"ImagePositionPatient\",\"ImageOrientationPatient\",\"PixelSpacing\"]\n\ndef dicom_metadata(folder_path):\n    files = os.listdir(folder_path)\n    patient_id = folder_path.split('/')[-1]\n    \n    ## Each row is an image file:\n    base_data = {'Patient': [patient_id]*len(files), 'File': files}\n    patient_df = pd.DataFrame(data=base_data)\n    \n    ## Add Columns by looping through DICOM attributes for each image file:\n    slices = [dcm.dcmread(folder_path + '/' + s) for s in files] \n    for d in dicom_atts:\n        attribute_i = []\n        for s in slices:\n            try:\n                attribute_i.append(s[d].value)\n            except:\n                attribute_i.append(np.nan)\n        patient_df[d] = attribute_i\n        \n    ## Store min pixel value for each image file \n    attribute_min_pixel = []\n    for s in slices:\n        try:\n            mp = np.min(s.pixel_array.astype(np.int16).flatten())\n        except:\n            mp = np.nan\n        attribute_min_pixel.append(mp)\n    patient_df[\"MinPixelValue\"] = attribute_min_pixel\n  \n    return patient_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> For example: let's fetch the metadata for images from \"test/00268ff88746/75d23269adbd\" directory."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = dicom_metadata(\"../input/rsna-str-pulmonary-embolism-detection/test/00268ff88746/75d23269adbd\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Let's now see the data!"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Amazing, Isn't it? :)"},{"metadata":{},"cell_type":"markdown","source":"> Let's now see some of the values from this data too!"},{"metadata":{},"cell_type":"markdown","source":"* We can see that SOPInstanceUID is available here in the metadata. So, now, it'll be very easy for us to join the dataframes if we want to do so.\n* Also, \"Patient\" is nothing but the SeriesInstanceUID.\n* We conclude that StudyInstanceUID is also in the metadata."},{"metadata":{},"cell_type":"markdown","source":"> Now, there are a total of 58 columns in the metadata, many features being not of any use, some are helpful too.\n> Let's see the resolution of most of the scans available."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"CT Scan resolution is: {df.Rows.value_counts().index[0]}x{df.Columns.value_counts().index[0]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del df_train, df_test, df_sub, df, count, dicom_atts, list_attributes, img_addr, corr_mat, mask, train_cols\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you like the work, I will make sure to update this over time. Please leave your comments down below in case of any suggestions.\n\nThanks for reading and Good Luck with the competition! :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}