{"cells":[{"metadata":{},"cell_type":"markdown","source":"## What is pulmonary embolism\nPulmonary embolism occurs when a clump of material, most often a blood clot, gets wedged into an artery in your lungs. These blood clots most commonly come from the deep veins of your legs, a condition known as deep vein thrombosis (DVT)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('<center><iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/8UnPPZlnfbk\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></center>')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What is RSNA STR Pulmonary Embolism Detection Competition?\n- In this competition you have to classify Pulmonary Embolism cases in chest CT scans.\n- This competition is inference-only, meaning that your submitted kernels will not have access to the training set.\n- Also note that the private test set is approximately 3x larger than the public test set (230GB vs. 70GB), so ensure that your kernels have enough time to finish their re-run. The training set includes 7279 studies, the public set 650, and the private set has 1517."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport pydicom as dicom\nimport matplotlib.pyplot as plt\nfrom os import listdir,mkdir\nimport plotly.express as px\nimport seaborn as sns\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"basepath = \"../input/rsna-str-pulmonary-embolism-detection/\"\nlistdir(basepath)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read train and test data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/rsna-str-pulmonary-embolism-detection/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/rsna-str-pulmonary-embolism-detection/test.csv\")\nsub = pd.read_csv(\"/kaggle/input/rsna-str-pulmonary-embolism-detection/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training Data Size\")\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Test Data Size\")\ntest.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Lets understand the training dataset\n\n- StudyInstanceUID: unique ID for each study (exam) in the data.\n- SeriesInstanceUID: unique ID for each series within the study.\n- SOPInstanceUID: unique ID for each image within the study (and data).\n- pe_present_on_image: image-level, notes whether any form of PE(Pulmonary Embolism) is present on the image. Possible value:0 or 1. Zero means there is no sign of PE in the image and one stands for PE presents in the image.\n- negative_exam_for_pe: exam-level, whether there are any images in the study that have PE present. Possible value: 0 or 1. Zero means there is no image with PE present in the exam and one stands for there is at least one or more images in the study with PE.\n- qa_motion: informational, indicates whether radiologists noted an issue with motion in the study. This field is informational that means no prediction required for this fields. \n- qa_contrast: informational, indicates whether radiologists noted an issue with contrast in the study.\n- flow_artifact: informational\n\nAbout RV/LV : Assessment of right ventricular strain using computed tomography (CT) in patients with pulmonary embolism (PE) has often relied on the ratio between the diameter of the right and left ventricles (RV/LV) in axial slices. The RV/LV ratio, measured in this manner, may be an unreliable marker of strain, due in part to inconsistencies in where measurements are taken and the complex three-dimensional nature of the RV. \n\n- rv_lv_ratio_gte_1: exam-level, indicates whether the RV/LV ratio present in the study is >= 1. Possible value: 0 or 1. One stands for RV/LV >=1 otherwise zero.\n- rv_lv_ratio_lt_1: exam-level, indicates whether the RV/LV ratio present in the study is < 1. Possible value: 0 or 1. One stands for RV/LV < 1 otherwise zero.\n- leftsided_pe: exam-level, indicates that there is PE present on the left side of the images in the study. Possible value 0 or 1. One stands for PE present on the left side of the images, zero otherwise.\n- chronic_pe: exam-level, indicates that the PE in the study is chronic. Possible value 0 or 1. One stands for PE in the study is chronic_pe, zero otherwise.\n- true_filling_defect_not_pe: informational, indicates a defect that is NOT PE. Possible value 0 or 1. One stands for there is defect but that is not a PE.\n- rightsided_pe: exam-level, indicates that there is PE present on the right side of the images in the study. Possible value 0 or 1. One stands for there is PE present on the right side, zero otherwise.\n- acute_and_chronic_pe: exam-level, indicates that the PE present in the study is both acute AND chronic. Possible value 0 or 1. One stands for PE present in the study is both acute and chronic otherwise zero.\n- central_pe: exam-level, indicates that there is PE present in the center of the images in the study. Possible value 0 or 1. One stands for PE present in the center of the images in the study and zero otherwise.\n- indeterminate: exam-level, indicates that while the study is not negative for PE, an ultimate set of exam-level labels could not be created, due to QA issues. Possible value 0 or 1. One stands for there is some issues(Motion or Contrast issue) in the CT images for which radiologist can't determine PE present or not."},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Lets understand the test dataset\nIn the test dataset, the unique ids for each study ,series and individual images are given and you have to predict the probability for the bilow labels value:\n- Negative for PE\n- Indeterminate\n- Chronic\n- Acute & Chronic\n- Central PE\n- Left PE\t\n- Right PE\t\n- RV/LV Ratio >= 1\n- RV/LV Ratio < 1\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Check missing value in train data')\ntrain.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Chack missing value in test data')\ntest.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**So there is no missing value in train and test data set**"},{"metadata":{},"cell_type":"markdown","source":"**Check Non Zero value for each column in training set**"},{"metadata":{"trusted":true},"cell_type":"code","source":"x = train.drop(['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID'], axis=1).sum(axis=0).sort_values().reset_index()\nx.columns = ['column', 'nonzero_records']\n\nfig = px.bar(\n    x, \n    x='nonzero_records', \n    y='column', \n    orientation='h', \n    title='Columns and non zero samples', \n    height=800, \n    width=800\n)\n\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dicom processsing and example "},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_scan(path):\n    slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]\n    slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_lungwin(img, hu=[-1200., 600.]):\n    lungwin = np.array(hu)\n    newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n    newimg[newimg < 0] = 0\n    newimg[newimg > 1] = 1\n    newimg = (newimg * 255).astype('uint8')\n    return newimg\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pixels_hu(slices):\n    image = np.stack([s.pixel_array for s in slices])\n    # Convert to int16 (from sometimes int16), \n    # should be possible as values should always be low enough (<32k)\n    image = image.astype(np.int16)\n\n    # Set outside-of-scan pixels to 0\n    # The intercept is usually -1024, so air is approximately 0\n    image[image == -2000] = 0\n    \n    # Convert to Hounsfield units (HU)\n    for slice_number in range(len(slices)):\n        \n        intercept = slices[slice_number].RescaleIntercept\n        slope = slices[slice_number].RescaleSlope\n        \n        if slope != 1:\n            image[slice_number] = slope * image[slice_number].astype(np.float64)\n            image[slice_number] = image[slice_number].astype(np.int16)\n            \n        image[slice_number] += np.int16(intercept)\n    \n    return np.array(image, dtype=np.int16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scans = load_scan('../input/rsna-str-pulmonary-embolism-detection/train/0003b3d648eb/d2b2960c2bbf/')\nscan_array = set_lungwin(get_pixels_hu(scans))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"import matplotlib.animation as animation\n\nfig = plt.figure()\n\nims = []\nfor image in scan_array:\n    im = plt.imshow(image, animated=True, cmap=\"Greys\")\n    plt.axis(\"off\")\n    ims.append([im])\n\nani = animation.ArtistAnimation(fig, ims, interval=100, blit=False,\n                                repeat_delay=1000)\nHTML(ani.to_jshtml())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_to_hu(slices):\n    images = np.stack([file.pixel_array for file in slices])\n    images = images.astype(np.int16)\n\n    # convert ouside pixel-values to air:\n    # I'm using <= -1000 to be sure that other defaults are captured as well\n    images[images <= -1000] = 0\n    \n    # convert to HU\n    for n in range(len(slices)):\n        \n        intercept = slices[n].RescaleIntercept\n        slope = slices[n].RescaleSlope\n        \n        if slope != 1:\n            images[n] = slope * images[n].astype(np.float64)\n            images[n] = images[n].astype(np.int16)\n            \n        images[n] += np.int16(intercept)\n    \n    return np.array(images, dtype=np.int16)\n\ndef load_slice(path):\n    slices = [dicom.read_file(path + '/' + s) for s in listdir(path)]\n    slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"first_patient = load_slice('../input/rsna-str-pulmonary-embolism-detection/train/0003b3d648eb/d2b2960c2bbf')\nfirst_patient_pixels = transform_to_hu(first_patient)\n\ndef sample_stack(stack, rows=6, cols=6, start_with=0, show_every=5):\n    fig,ax = plt.subplots(rows,cols,figsize=[18,20])\n    for i in range(rows*cols):\n        ind = start_with + i*show_every\n        ax[int(i/rows),int(i % rows)].set_title(f'slice {ind}')\n        ax[int(i/rows),int(i % rows)].imshow(stack[ind],cmap='bone')\n        ax[int(i/rows),int(i % rows)].axis('off')\n    plt.show()\n\nsample_stack(first_patient_pixels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Light EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = train.copy()\ncols.drop(['StudyInstanceUID','SeriesInstanceUID','SOPInstanceUID'],axis=1,inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr = cols.corr()\nmask = np.zeros_like(corr)\nmask[np.triu_indices_from(mask)] = True\nwith sns.axes_style(\"white\"):\n    f, ax = plt.subplots(figsize=(12, 12))\n    ax = sns.heatmap(corr,mask=mask,square=True,linewidths=.8,cmap=\"summer\",annot=True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Acknowledgement\n\n- [Pulmonary Embolism Dicom preprocessing & EDA](https://www.kaggle.com/nitindatta/pulmonary-embolism-dicom-preprocessing-eda)\n- [Pulmonary Embolism Detection EDA](https://www.kaggle.com/isaienkov/pulmonary-embolism-detection-eda)\n- [Pulmonary embolism: The route to recovery](https://www.youtube.com/watch?v=8UnPPZlnfbk&ab_channel=naturevideo)\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}