{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

RSNA 2022 Cervical Spine Fracture Detection

\n

Understanding the Dataset

\n

(as a beginner)

\n @shreydan\n
\n\n### Please make sure to checkout the credits and references at the end!\n___","metadata":{}},{"cell_type":"markdown","source":"# Installations & Imports\n___\n### from [@andradaolteanu](https://www.kaggle.com/andradaolteanu), [@ipythonx](https://www.kaggle.com/datasets/ipythonx/for-pydicom)","metadata":{}},{"cell_type":"markdown","source":"### Online","metadata":{}},{"cell_type":"code","source":"# !pip install -qU \"python-gdcm\" pydicom pylibjpeg \"opencv-python-headless\"","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-08-31T05:18:40.80908Z","iopub.execute_input":"2022-08-31T05:18:40.80949Z","iopub.status.idle":"2022-08-31T05:18:40.815319Z","shell.execute_reply.started":"2022-08-31T05:18:40.809459Z","shell.execute_reply":"2022-08-31T05:18:40.813697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Offline","metadata":{}},{"cell_type":"code","source":"!pip install -qU ../input/for-pydicom/python_gdcm-3.0.14-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ../input/for-pydicom/pylibjpeg-1.4.0-py3-none-any.whl --find-links frozen_packages --no-index","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:40.817373Z","iopub.execute_input":"2022-08-31T05:18:40.817903Z","iopub.status.idle":"2022-08-31T05:18:54.644837Z","shell.execute_reply.started":"2022-08-31T05:18:40.817857Z","shell.execute_reply":"2022-08-31T05:18:54.643145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot, plot\nfrom tqdm.notebook import tqdm\nfrom pathlib import Path\nfrom collections import Counter\n\ntqdm.pandas()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-31T06:28:04.893522Z","iopub.execute_input":"2022-08-31T06:28:04.894083Z","iopub.status.idle":"2022-08-31T06:28:04.902525Z","shell.execute_reply.started":"2022-08-31T06:28:04.894045Z","shell.execute_reply":"2022-08-31T06:28:04.901472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for dcm and nii\nimport pydicom\nimport nibabel as nib\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.173202Z","iopub.execute_input":"2022-08-31T05:18:56.173546Z","iopub.status.idle":"2022-08-31T05:18:56.50242Z","shell.execute_reply.started":"2022-08-31T05:18:56.173506Z","shell.execute_reply":"2022-08-31T05:18:56.501099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **The Vertebral Column**\n### **[Source: https://www.cancer.gov](https://www.cancer.gov/publications/dictionaries/cancer-terms/def/vertebral-column)**\n___\nThe spine is made up of bones, muscles, tendons, nerves, and other tissues that reach from the base of the skull near the spinal cord (clivus) to the coccyx (tailbone). **The vertebrae (back bones) of the spine include the cervical spine (C1-C7)**, thoracic spine (T1-T12), lumbar spine (L1-L5), sacral spine (S1-S5), and the tailbone. Each vertebra is separated by a disc. The vertebrae surround and protect the spinal cord.\n\n![](https://upload.wikimedia.org/wikipedia/commons/thumb/5/54/Gray_111_-_Vertebral_column-coloured.png/174px-Gray_111_-_Vertebral_column-coloured.png)\n\n[Image Source](https://upload.wikimedia.org/wikipedia/commons/thumb/5/54/Gray_111_-_Vertebral_column-coloured.png/174px-Gray_111_-_Vertebral_column-coloured.png)\n\n# **Learning More About the Cervical Spine**\n### **[Source: https://my.clevelandclinic.org/](https://my.clevelandclinic.org/health/articles/22278-cervical-spine)**\n___\n## What is the cervical spine?\nYour cervical spine — the neck area of your spine — consists of seven stacked bones called vertebrae. \n> Marked C1 to C7 in the above image\n\n## What does the cervical spine do?\n- Protecting your spinal cord.\n- Supporting your head and allowing movement.\n- Providing a safe passageway for vertebral arteries.\n\n## Cervical Spine Fracture\n\nA fracture to the bones of your spine can result from compression (often from minor trauma in a person with osteoporosis) or be a burst fracture (vertebra that’s crushed in all directions) or a fracture-dislocation (mostly from vehicle accidents or falls from heights).\n\n## Tests and Imaging\n\n**Computed tomography (CT) scan:** This scan uses X-rays and computers to produce images that are very thin “slices” of the area under examination. A CT scan can show the shape and size of your spinal canal, its contents and the bone around it. It helps diagnose bone spurs, osteophytes, bone fusion and bone destruction from infection or tumor.\n\n___","metadata":{}},{"cell_type":"markdown","source":"# **Competition Goal**\n___\n\nThe goal of this competition is to **identify fractures in CT scans of the cervical spine** (neck) at both the level of a single vertebrae and the entire patient.\n\n*Quickly detecting and determining the location of any vertebral fractures is essential to prevent neurologic deterioration and paralysis after trauma.*","metadata":{}},{"cell_type":"code","source":"paths = {\n 'train_df': Path('../input/rsna-2022-cervical-spine-fracture-detection/train.csv'),\n 'train_bbox': Path('../input/rsna-2022-cervical-spine-fracture-detection/train_bounding_boxes.csv'),\n 'train_images': Path('../input/rsna-2022-cervical-spine-fracture-detection/train_images'),\n 'train_nifti_segments': Path('../input/rsna-2022-cervical-spine-fracture-detection/segmentations'),\n 'test_df': Path('../input/rsna-2022-cervical-spine-fracture-detection/test.csv'),\n 'test_images': Path('../input/rsna-2022-cervical-spine-fracture-detection/test_images')\n}","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.505094Z","iopub.execute_input":"2022-08-31T05:18:56.505453Z","iopub.status.idle":"2022-08-31T05:18:56.512106Z","shell.execute_reply.started":"2022-08-31T05:18:56.505423Z","shell.execute_reply":"2022-08-31T05:18:56.510784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Preparing Main DataFrame**\n___","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(paths['train_df'])\ntest_df = pd.read_csv(paths['test_df'])","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.513651Z","iopub.execute_input":"2022-08-31T05:18:56.514054Z","iopub.status.idle":"2022-08-31T05:18:56.542483Z","shell.execute_reply.started":"2022-08-31T05:18:56.514021Z","shell.execute_reply":"2022-08-31T05:18:56.541564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Column Information\n___\n\n- `StudyInstanceUID` - The study ID. There is one unique study ID for each patient scan.\n- `patient_overall` - One of the target columns. The patient level outcome, i.e. if any of the vertebrae are fractured.\n- `C[1-7]` - The other target columns. Whether the given vertebrae is fractured. See this diagram for the real location of each vertbrae in the spine.\n","metadata":{}},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.543599Z","iopub.execute_input":"2022-08-31T05:18:56.54459Z","iopub.status.idle":"2022-08-31T05:18:56.571796Z","shell.execute_reply.started":"2022-08-31T05:18:56.544556Z","shell.execute_reply":"2022-08-31T05:18:56.570871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ignoring a specific StudyInstanceUID\nDiscussion Link: [The scan 1.2.826.0.1.3680043.20574 does not include a full cervical spine and should be ignored. Thanks to @kretes for catching this.](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/344862)","metadata":{}},{"cell_type":"code","source":"train_df[train_df['StudyInstanceUID'] == '1.2.826.0.1.3680043.20574']","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.573586Z","iopub.execute_input":"2022-08-31T05:18:56.57404Z","iopub.status.idle":"2022-08-31T05:18:56.592555Z","shell.execute_reply.started":"2022-08-31T05:18:56.573996Z","shell.execute_reply":"2022-08-31T05:18:56.591664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.drop(index=1135, inplace=True)\ntrain_df.reset_index(drop=True, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.593983Z","iopub.execute_input":"2022-08-31T05:18:56.594352Z","iopub.status.idle":"2022-08-31T05:18:56.60468Z","shell.execute_reply.started":"2022-08-31T05:18:56.594321Z","shell.execute_reply":"2022-08-31T05:18:56.603557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.606428Z","iopub.execute_input":"2022-08-31T05:18:56.607145Z","iopub.status.idle":"2022-08-31T05:18:56.625018Z","shell.execute_reply.started":"2022-08-31T05:18:56.607071Z","shell.execute_reply":"2022-08-31T05:18:56.623666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## adding `total_fractures` column to indicate how many cervical vertebrae are fractured","metadata":{}},{"cell_type":"code","source":"train_df['total_fractures'] = train_df.loc[:,[f\"C{i}\" for i in range(1,8)]].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.626552Z","iopub.execute_input":"2022-08-31T05:18:56.626893Z","iopub.status.idle":"2022-08-31T05:18:56.638003Z","shell.execute_reply.started":"2022-08-31T05:18:56.626864Z","shell.execute_reply":"2022-08-31T05:18:56.637032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.639343Z","iopub.execute_input":"2022-08-31T05:18:56.640022Z","iopub.status.idle":"2022-08-31T05:18:56.659021Z","shell.execute_reply.started":"2022-08-31T05:18:56.63998Z","shell.execute_reply":"2022-08-31T05:18:56.658205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.660457Z","iopub.execute_input":"2022-08-31T05:18:56.660806Z","iopub.status.idle":"2022-08-31T05:18:56.671871Z","shell.execute_reply.started":"2022-08-31T05:18:56.660776Z","shell.execute_reply":"2022-08-31T05:18:56.670531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Now in total, We've 2018 study IDs**","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"# **Understanding CT Scans**\n\n...A CT scan generates images that can be reformatted in multiple planes. It can even generate three-dimensional images...[read more](https://www.radiologyinfo.org/en/info/bodyct)\n\nThis allows multiple planes of observation, the most common one being axial.\n\n![CT Scan Planes](https://www.ipfradiologyrounds.com/_images/image-reconstruction-planes.png)\n\n[Image Source](https://www.ipfradiologyrounds.com/hrct-primer/image-reconstruction/)\n\n### The data in `train_images` and `test_images` is segmented as per the axial plane.","metadata":{}},{"cell_type":"markdown","source":"## Adding segmentations path of each patient to the dataframe","metadata":{}},{"cell_type":"code","source":"train_df['segment_path'] = train_df['StudyInstanceUID'].map(lambda x: paths['train_images']/x)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.677384Z","iopub.execute_input":"2022-08-31T05:18:56.678331Z","iopub.status.idle":"2022-08-31T05:18:56.696737Z","shell.execute_reply.started":"2022-08-31T05:18:56.678286Z","shell.execute_reply":"2022-08-31T05:18:56.69546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:18:56.698729Z","iopub.execute_input":"2022-08-31T05:18:56.699239Z","iopub.status.idle":"2022-08-31T05:18:56.716043Z","shell.execute_reply.started":"2022-08-31T05:18:56.699161Z","shell.execute_reply":"2022-08-31T05:18:56.714704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Each patient has different number of slices, adding `num_slices` to each patient","metadata":{}},{"cell_type":"code","source":"def num_slices(path):\n slices = list(path.glob('*'))\n return len(slices)\n\ntrain_df['num_slices'] = train_df['segment_path'].progress_map(num_slices)\ntrain_df['num_slices'] = train_df['num_slices'].astype('int')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:51:02.306229Z","iopub.execute_input":"2022-08-31T05:51:02.306696Z","iopub.status.idle":"2022-08-31T05:51:10.103446Z","shell.execute_reply.started":"2022-08-31T05:51:02.306637Z","shell.execute_reply":"2022-08-31T05:51:10.101711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:51:13.360229Z","iopub.execute_input":"2022-08-31T05:51:13.361132Z","iopub.status.idle":"2022-08-31T05:51:13.383957Z","shell.execute_reply.started":"2022-08-31T05:51:13.36107Z","shell.execute_reply":"2022-08-31T05:51:13.381975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:08.861954Z","iopub.execute_input":"2022-08-31T05:21:08.863432Z","iopub.status.idle":"2022-08-31T05:21:08.880326Z","shell.execute_reply.started":"2022-08-31T05:21:08.863383Z","shell.execute_reply":"2022-08-31T05:21:08.87922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___\n# **Some basic EDA**\n___","metadata":{}},{"cell_type":"code","source":"fig = px.pie(train_df['patient_overall'].value_counts().reset_index(), \n names=['not_fractured','fractured'], # [0,1]\n values='patient_overall',\n color_discrete_sequence = px.colors.qualitative.Pastel,\n title='Patient Fracture Overall Distribution'\n ).update_traces(textinfo='label+percent')\niplot(fig)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-31T05:21:08.881884Z","iopub.execute_input":"2022-08-31T05:21:08.882616Z","iopub.status.idle":"2022-08-31T05:21:10.045074Z","shell.execute_reply.started":"2022-08-31T05:21:08.882572Z","shell.execute_reply":"2022-08-31T05:21:10.043953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fractures = train_df.loc[:,[f\"C{i}\" for i in range(1,8)]].sum().reset_index()\nfractures['not_fractured'] = fractures[0].map(lambda x:len(train_df)-x)\nfractures.rename(columns={'index':'vertebra',0:'fractured'}, inplace=True)\nfig = px.bar(fractures, \n x='vertebra', \n y=['fractured','not_fractured'],\n title='Fractures as per Vertebra',\n color_discrete_sequence = ['#e0465f','#afde2c'],\n )\niplot(fig)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-31T05:21:10.046899Z","iopub.execute_input":"2022-08-31T05:21:10.047249Z","iopub.status.idle":"2022-08-31T05:21:10.186923Z","shell.execute_reply.started":"2022-08-31T05:21:10.047219Z","shell.execute_reply":"2022-08-31T05:21:10.185874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.bar(train_df['total_fractures'].value_counts().reset_index(), \n x='index', \n y='total_fractures',\n title='No. of fractures amongst patients',\n text='total_fractures',\n color_discrete_sequence = ['#d82cde'],\n )\niplot(fig)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-31T05:21:10.188426Z","iopub.execute_input":"2022-08-31T05:21:10.188894Z","iopub.status.idle":"2022-08-31T05:21:10.273868Z","shell.execute_reply.started":"2022-08-31T05:21:10.188851Z","shell.execute_reply":"2022-08-31T05:21:10.272519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(train_df, x=\"num_slices\")\niplot(fig)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.275502Z","iopub.execute_input":"2022-08-31T05:21:10.276227Z","iopub.status.idle":"2022-08-31T05:21:10.379378Z","shell.execute_reply.started":"2022-08-31T05:21:10.276167Z","shell.execute_reply":"2022-08-31T05:21:10.378248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"# **Understanding the different medical-imaging file formats**\n\n- `.dcm` - [DICOM](https://fileinfo.com/extension/dcm) file: A DCM file is an image file saved in the Digital Imaging and Communications in Medicine (DICOM) image format. It stores a medical image, such as a CT scan or ultrasound, and may also include patient information to pair the image with the patient. \n\n- `.nii` - [NIFTI file format](https://brainder.org/2012/09/23/the-nifti-file-format/) - [A nibabel image](https://nipy.org/nibabel/nibabel_images.html) object is the association of three things:\n - an N-D array containing the image data;\n - a (4, 4) affine matrix mapping array coordinates to coordinates in some RAS+ world coordinate space (Coordinate systems and affines);\n - image metadata in the form of a header.\n","metadata":{}},{"cell_type":"markdown","source":"___\n\n# **Loading DICOM and NIFTI files**\n### from [@andradaolteanu](https://www.kaggle.com/andradaolteanu) and [@harshitsheoran](https://www.kaggle.com/harshitsheoran)","metadata":{}},{"cell_type":"markdown","source":"## **Let's try opening a random `.dcm` and a `.nii` file**","metadata":{}},{"cell_type":"markdown","source":"## **.dcm**","metadata":{}},{"cell_type":"code","source":"random_dcm_file = list(train_df['segment_path'][123].glob('*'))[0]\nprint(random_dcm_file)\nrandom_dcm_file = pydicom.dcmread(random_dcm_file)\nprint(random_dcm_file)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.38097Z","iopub.execute_input":"2022-08-31T05:21:10.382296Z","iopub.status.idle":"2022-08-31T05:21:10.409064Z","shell.execute_reply.started":"2022-08-31T05:21:10.382245Z","shell.execute_reply":"2022-08-31T05:21:10.407835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# getting metadata with .get(key)\nprint(\"Instance Number:\",random_dcm_file.get('InstanceNumber'))\nprint(\"Rows x Columns:\", random_dcm_file.get(\"Rows\"), random_dcm_file.get(\"Columns\"))\nprint(\"Image Position (Patient):\", random_dcm_file.get(\"ImagePositionPatient\"))","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.410782Z","iopub.execute_input":"2022-08-31T05:21:10.411465Z","iopub.status.idle":"2022-08-31T05:21:10.416964Z","shell.execute_reply.started":"2022-08-31T05:21:10.411431Z","shell.execute_reply":"2022-08-31T05:21:10.41598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## from the above metadata:\n- patient ID = patient Name\n- `Slice Thickness` = 1.0 mm [might not be same for all patients]\n- `Instance Number` = slice number\n- `Image Position (Patient)`: gives an array of values [x-axis, y-axis, z-axis], here z-axis denotes the position in the sagittal plane\n\n`apply_voi_lut()` applies a windowing function to the image - [read more here](https://dicom.innolitics.com/ciods/ct-image/voi-lut/00281056)","metadata":{}},{"cell_type":"code","source":"dcm_image = apply_voi_lut(random_dcm_file.pixel_array, random_dcm_file)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.418654Z","iopub.execute_input":"2022-08-31T05:21:10.419311Z","iopub.status.idle":"2022-08-31T05:21:10.438084Z","shell.execute_reply.started":"2022-08-31T05:21:10.419277Z","shell.execute_reply":"2022-08-31T05:21:10.437067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(dcm_image, cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.440054Z","iopub.execute_input":"2022-08-31T05:21:10.44089Z","iopub.status.idle":"2022-08-31T05:21:10.747585Z","shell.execute_reply.started":"2022-08-31T05:21:10.440845Z","shell.execute_reply":"2022-08-31T05:21:10.746275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___\n\n## **.nii**","metadata":{}},{"cell_type":"code","source":"random_nii_file = list(paths['train_nifti_segments'].glob('*'))[10]\nprint(random_nii_file)\n\ndef open_nii_file(path):\n f = nib.load(path)\n segmentations = f.get_fdata()[:, ::-1, ::-1].transpose(2, 1, 0)\n return segmentations\n\nnii_segments = open_nii_file(random_nii_file)\nprint(nii_segments.shape, \"=> (num_slices, height, width)\")\nplt.imshow(nii_segments[123,:,:],cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:10.749438Z","iopub.execute_input":"2022-08-31T05:21:10.750507Z","iopub.status.idle":"2022-08-31T05:21:12.155105Z","shell.execute_reply.started":"2022-08-31T05:21:10.750456Z","shell.execute_reply":"2022-08-31T05:21:12.15425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A caveat of axial plane is that we are not aware of which cervical vertebra we are looking at, the above `nii_segments` array can be reused to get the vertebra index using `np.unique()`","metadata":{}},{"cell_type":"code","source":"# np.unique(nii_segments[slice_number])\nprint(\"[background, *, vertebra]\")\nprint(\"here 4. denotes that the slice is a part of C4 vertebra\")\nnp.unique(nii_segments[100])","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:12.156495Z","iopub.execute_input":"2022-08-31T05:21:12.157665Z","iopub.status.idle":"2022-08-31T05:21:12.17277Z","shell.execute_reply.started":"2022-08-31T05:21:12.157628Z","shell.execute_reply":"2022-08-31T05:21:12.171556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"## not all study UIDs have nii segments, for the ones we do have, adding their path to df","metadata":{}},{"cell_type":"code","source":"f\"We only have {len(list(paths['train_nifti_segments'].glob('*')))} nii files / {len(train_df)} studies\"","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:12.174454Z","iopub.execute_input":"2022-08-31T05:21:12.174991Z","iopub.status.idle":"2022-08-31T05:21:12.183771Z","shell.execute_reply.started":"2022-08-31T05:21:12.174938Z","shell.execute_reply":"2022-08-31T05:21:12.182488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def add_nii_segment_path(uid):\n base_path = paths['train_nifti_segments']\n # path if exists else None\n path = base_path/(uid+'.nii')\n if path.exists():\n return path\n return None\n\ntrain_df['nii_segments_path'] = train_df['StudyInstanceUID'].map(add_nii_segment_path)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:12.184946Z","iopub.execute_input":"2022-08-31T05:21:12.185299Z","iopub.status.idle":"2022-08-31T05:21:13.240235Z","shell.execute_reply.started":"2022-08-31T05:21:12.185266Z","shell.execute_reply":"2022-08-31T05:21:13.238661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:13.241815Z","iopub.execute_input":"2022-08-31T05:21:13.242196Z","iopub.status.idle":"2022-08-31T05:21:13.259711Z","shell.execute_reply.started":"2022-08-31T05:21:13.242142Z","shell.execute_reply":"2022-08-31T05:21:13.258516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"code","source":"def get_dcm_images(path):\n paths = list(path.glob('*'))\n paths.sort(key=lambda x:int(x.stem)) # sort based on slice index which is the filename: index.dcm\n data = [pydicom.dcmread(f) for f in paths]\n images = [apply_voi_lut(dcm.pixel_array, dcm) for dcm in data]\n return images","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:13.261156Z","iopub.execute_input":"2022-08-31T05:21:13.261549Z","iopub.status.idle":"2022-08-31T05:21:13.268727Z","shell.execute_reply.started":"2022-08-31T05:21:13.261502Z","shell.execute_reply":"2022-08-31T05:21:13.267594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_nii_segments(path):\n f = nib.load(path)\n segmentations = f.get_fdata()[:, ::-1, ::-1].transpose(2, 1, 0)\n return segmentations","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:13.270246Z","iopub.execute_input":"2022-08-31T05:21:13.271339Z","iopub.status.idle":"2022-08-31T05:21:13.280503Z","shell.execute_reply.started":"2022-08-31T05:21:13.271295Z","shell.execute_reply":"2022-08-31T05:21:13.279537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_idx = 99\ndcm_images = get_dcm_images(train_df['segment_path'][sample_idx])\nnii_segments = get_nii_segments(train_df['nii_segments_path'][sample_idx])\nprint((len(dcm_images), *dcm_images[0].shape), nii_segments.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:03:03.87401Z","iopub.execute_input":"2022-08-31T06:03:03.874517Z","iopub.status.idle":"2022-08-31T06:03:06.071781Z","shell.execute_reply.started":"2022-08-31T06:03:03.874478Z","shell.execute_reply":"2022-08-31T06:03:06.069788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_slice(idx, dcm_images=dcm_images, nii_segments=nii_segments):\n fig, (ax1, ax2) = plt.subplots(1, 2)\n ax1.axis('off'); ax2.axis('off')\n fig.suptitle(f'Slice {idx}')\n ax1.imshow(dcm_images[idx], cmap='bone')\n ax2.imshow(nii_segments[idx,:,:], cmap='bone')\n\nfor i in range(123,128):\n plot_slice(i)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:03:06.975037Z","iopub.execute_input":"2022-08-31T06:03:06.976672Z","iopub.status.idle":"2022-08-31T06:03:08.461384Z","shell.execute_reply.started":"2022-08-31T06:03:06.976602Z","shell.execute_reply":"2022-08-31T06:03:08.459127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___\n\n## all UIDs which have segmentations:","metadata":{}},{"cell_type":"code","source":"uids_with_segments = train_df[train_df['nii_segments_path'].notnull()]\nuids_with_segments.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:43:28.66105Z","iopub.execute_input":"2022-08-31T05:43:28.661572Z","iopub.status.idle":"2022-08-31T05:43:28.680992Z","shell.execute_reply.started":"2022-08-31T05:43:28.661534Z","shell.execute_reply":"2022-08-31T05:43:28.679609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___\n\n# **some scans maybe in reverse order**\n\n[A Segmentation is in reverse order by @itsuki9180](https://www.kaggle.com/code/itsuki9180/a-segmentation-is-in-reverse-order/comments) and [based on this comment by @abebe9849](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/348658#1918410)\n\n#### based on [this discussion](https://www.kaggle.com/code/itsuki9180/a-segmentation-is-in-reverse-order/comments#1919812), for the z_pos value in ImagePositionPatient. If z_pos of last slice is greater than z_pos of first slice, then we've to reverse the order since positive is towards the head of the patient\n___","metadata":{}},{"cell_type":"code","source":"def check_reverse_required(path):\n paths = list(path.glob('*'))\n paths.sort(key=lambda x:int(x.stem))\n z_first = pydicom.dcmread(paths[0]).get(\"ImagePositionPatient\")[-1]\n z_last = pydicom.dcmread(paths[-1]).get(\"ImagePositionPatient\")[-1]\n if z_last < z_first:\n return False\n return True","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:33:32.231502Z","iopub.execute_input":"2022-08-31T05:33:32.231984Z","iopub.status.idle":"2022-08-31T05:33:32.239561Z","shell.execute_reply.started":"2022-08-31T05:33:32.231944Z","shell.execute_reply":"2022-08-31T05:33:32.238071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checks = train_df['segment_path'].progress_map(check_reverse_required)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:17:39.136795Z","iopub.execute_input":"2022-08-31T06:17:39.137465Z","iopub.status.idle":"2022-08-31T06:20:41.509599Z","shell.execute_reply.started":"2022-08-31T06:17:39.137409Z","shell.execute_reply":"2022-08-31T06:20:41.507033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['reverse_required'] = checks","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:21:03.619149Z","iopub.execute_input":"2022-08-31T06:21:03.619651Z","iopub.status.idle":"2022-08-31T06:21:03.62771Z","shell.execute_reply.started":"2022-08-31T06:21:03.619609Z","shell.execute_reply":"2022-08-31T06:21:03.626144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Counter(checks)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:20:41.512967Z","iopub.execute_input":"2022-08-31T06:20:41.514047Z","iopub.status.idle":"2022-08-31T06:20:41.523697Z","shell.execute_reply.started":"2022-08-31T06:20:41.513977Z","shell.execute_reply":"2022-08-31T06:20:41.522561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"indices_where_reverse_required = [i for i,req in (checks.reset_index()).values if req is True]\nprint(indices_where_reverse_required)","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:21:33.099303Z","iopub.execute_input":"2022-08-31T06:21:33.099769Z","iopub.status.idle":"2022-08-31T06:21:33.113745Z","shell.execute_reply.started":"2022-08-31T06:21:33.099733Z","shell.execute_reply":"2022-08-31T06:21:33.111726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___\n\n# **Generating Sagittal View**\n### based on [this comment by @harshitsheoran](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612#1891123) which gives a decently usable saggital view","metadata":{}},{"cell_type":"code","source":"print(train_df.loc[41,:])\ndcm_images = get_dcm_images(train_df['segment_path'][41]) # 0 fractures\ndcm_np = np.array(dcm_images)\nif train_df['reverse_required'][41] == True:\n dcm_np = dcm_np[::-1]\nsaggital_view = dcm_np[:,:,256]\nplt.imshow(saggital_view, cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:22:51.892261Z","iopub.execute_input":"2022-08-31T06:22:51.895447Z","iopub.status.idle":"2022-08-31T06:22:57.268499Z","shell.execute_reply.started":"2022-08-31T06:22:51.895355Z","shell.execute_reply":"2022-08-31T06:22:57.266621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.loc[60,:])\ndcm_images = get_dcm_images(train_df['segment_path'][60]) # 4 fractures\ndcm_np = np.array(dcm_images)\nif train_df['reverse_required'][60] == True:\n dcm_np = dcm_np[::-1]\nsaggital_view = dcm_np[:,:,256]\nplt.imshow(saggital_view, cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:23:10.889817Z","iopub.execute_input":"2022-08-31T06:23:10.890359Z","iopub.status.idle":"2022-08-31T06:23:34.414966Z","shell.execute_reply.started":"2022-08-31T06:23:10.89031Z","shell.execute_reply":"2022-08-31T06:23:34.413701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.loc[4,:])\ndcm_images = get_dcm_images(train_df['segment_path'][4]) # 4 fractures\ndcm_np = np.array(dcm_images)\nif train_df['reverse_required'][4] == True:\n dcm_np = dcm_np[::-1]\nsaggital_view = dcm_np[:,:,256]\nplt.imshow(saggital_view, cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:24:46.049678Z","iopub.execute_input":"2022-08-31T06:24:46.051543Z","iopub.status.idle":"2022-08-31T06:24:47.969154Z","shell.execute_reply.started":"2022-08-31T06:24:46.051471Z","shell.execute_reply":"2022-08-31T06:24:47.967042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## You can run this to get all the saggital views\n","metadata":{}},{"cell_type":"code","source":"!mkdir saggital_views\n\nsaggital_path = Path('./saggital_views')\nfor uid, path, rev in tqdm(train_df.loc[:,['StudyInstanceUID','segment_path','reverse_required']].values):\n dcm_images = get_dcm_images(path)\n dcm_np = np.array(dcm_images)\n if rev == True:\n dcm_np = dcm_np[::-1]\n saggital_view = dcm_np[:,:,256]\n plt.imsave(saggital_path/(uid+'.png'),saggital_view,cmap='bone')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T05:21:51.416988Z","iopub.execute_input":"2022-08-31T05:21:51.417498Z","iopub.status.idle":"2022-08-31T05:21:51.423451Z","shell.execute_reply.started":"2022-08-31T05:21:51.417452Z","shell.execute_reply":"2022-08-31T05:21:51.422249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Saving","metadata":{}},{"cell_type":"code","source":"train_df.to_csv('train_new.csv')\nuids_with_segments.to_csv('train_with_segments.csv')","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:26:42.353294Z","iopub.execute_input":"2022-08-31T06:26:42.353864Z","iopub.status.idle":"2022-08-31T06:26:42.387135Z","shell.execute_reply.started":"2022-08-31T06:26:42.353819Z","shell.execute_reply":"2022-08-31T06:26:42.385838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"# **Animation**\n___","metadata":{}},{"cell_type":"code","source":"sample_index = train_df.iloc[99,:]\ndcm_images = get_dcm_images(sample_index['segment_path'])\nif sample_index['reverse_required'] == True:\n dcm_images.sort(reverse=True)\nsegments = get_nii_segments(sample_index['nii_segments_path'])","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:44:15.654027Z","iopub.execute_input":"2022-08-31T06:44:15.654703Z","iopub.status.idle":"2022-08-31T06:44:17.651365Z","shell.execute_reply.started":"2022-08-31T06:44:15.654648Z","shell.execute_reply":"2022-08-31T06:44:17.647598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib import animation, rc\nrc('animation', html='jshtml')\n\nfig, [ax1,ax2] = plt.subplots(1,2)\nax1.axis('off')\nax2.axis('off')\nimages = []\nfor i in tqdm(range(len(dcm_images))):\n im1 = ax1.imshow(dcm_images[i], animated=True, cmap='bone')\n im2 = ax2.imshow(segments[i,:,:], animated=True, cmap='bone')\n if i==0:\n ax1.imshow(dcm_images[i], cmap='bone')\n ax2.imshow(segments[i,:,:], cmap='bone')\n images.append([im1,im2])\n \n\nani = animation.ArtistAnimation(fig, images, interval=50, blit=True,\n repeat_delay=1000)\nplt.close()\nani","metadata":{"execution":{"iopub.status.busy":"2022-08-31T06:53:22.937602Z","iopub.execute_input":"2022-08-31T06:53:22.938101Z","iopub.status.idle":"2022-08-31T06:54:19.329521Z","shell.execute_reply.started":"2022-08-31T06:53:22.938065Z","shell.execute_reply":"2022-08-31T06:54:19.325618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"# **References**\n\n> This community is incredible, a few days ago I had no idea what DICOM files were but today I have a basic understanding of how medical imaging works. I went through many notebooks and discussions to understand how everyone was handling the dataset.\n\n## These are some references I used:\nIf I missed anything here, they should be in the notebook in the above markdown cells somewhere\n\n- **Discussions**\n - [Explaining Data and Submission in detail](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612) by [@harshitsheoran](https://www.kaggle.com/harshitsheoran)\n > this was very helpful. @harshitsheoran explained everything perfectly which gave me the confidence to try out this dataset\n - [Order of Slices](https://www.kaggle.com/code/itsuki9180/a-segmentation-is-in-reverse-order/comments#1919812) by [@solverworld](https://www.kaggle.com/solverworld)\n - [Sorting the DICOM files](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/348658#1918410) by [@abebe9849](https://www.kaggle.com/abebe9849)\n - [Generating Saggital Views](https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612#1891123) by [@harshitsheoran](https://www.kaggle.com/harshitsheoran)\n\n- **Notebooks**\n - [rsna-fracture-detection-dicom-images-explore](https://www.kaggle.com/code/andradaolteanu/rsna-fracture-detection-dicom-images-explore) by [@andradaolteanu](https://www.kaggle.com/andradaolteanu)\n - [rsna-fracture-detection-in-depth-eda](https://www.kaggle.com/code/samuelcortinhas/rsna-fracture-detection-in-depth-eda) by [@samuelcortinhas](https://www.kaggle.com/samuelcortinhas)\n - [spine-fracture-eda-loading-dicom-3d-browse](https://www.kaggle.com/code/jirkaborovec/spine-fracture-eda-loading-dicom-3d-browse) by [@jirkaborovec](https://www.kaggle.com/jirkaborovec)\n - [a-segmentation-is-in-reverse-order](https://www.kaggle.com/code/itsuki9180/a-segmentation-is-in-reverse-order) by [@itsuki9180](https://www.kaggle.com/itsuki9180)\n \n\n\n\n## Thank you to all the users for their work and help. It was fun to try this out :)\n","metadata":{}}]}