The original 18,000 images were in DICOM (Digital Imaging and Communications in Medicine) format and they consumed memory space of 191.82 GB. These images were preprocessed and converted into jpg format, consequently resulting in data set of 1.6 GB."},{"metadata":{},"cell_type":"markdown","source":"### CSV file\n\n
Apart from the dicom image files, the data comes with train.csv file. There are 8 columns in CSV file and each column contains the following information,\n\n
image_id - unique image identifier\n
class_name - the name of the class of detected object (or \"No finding\")\n
class_id - the ID of the class of detected object\n
rad_id - the ID of the radiologist that made the observation\n
x_min - minimum X coordinate of the object's bounding box\n
y_min - minimum Y coordinate of the object's bounding box\n
x_max - maximum X coordinate of the object's bounding box\n
y_max - maximum Y coordinate of the object's bounding box\n \n
There are 67,914 rows. Each row contains information about one bounding box annotated by one radiologist in a single image.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport pandas as pd\n\ntrain_csv_df = pd.read_csv(\"/kaggle/input/vinbigdata-chest-xray-abnormalities-detection/train.csv\")\ntrain_csv_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Abnormalities\n\n
These are the 14 types of thoracic (chest) abnormalities that we need to classify and localize. If no abnormality is found in the image, we will classify it as \"No finding\" class.\n \n
0 - Aortic enlargement\n
1 - Atelectasis\n
2 - Calcification\n
3 - Cardiomegaly\n
4 - Consolidation\n
5 - ILD (Interstitial Lung Disease)\n
6 - Infiltration\n
7 - Lung Opacity\n
8 - Nodule/Mass\n
9 - Other lesion\n
10 - Pleural effusion\n
11 - Pleural thickening\n
12 - Pneumothorax\n
13 - Pulmonary fibrosis\n
14 - No finding (absence of any of 14 diseases listed above)\n \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Distribution of 15 different abnormalities in given 15,000 train set images\n\nprint(f\"\\n... Total number of unique classes = {train_csv_df['class_name'].nunique()} ...\\n\")\n\n# Count of each class is given below \ntrain_csv_df['class_name'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
From plot below, it can be observed that we are dealing with highly imbalanced data set. Almost half of the annotations marked in the data set are comprised of \"No finding\" class."},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly\nimport plotly.express as px\n\nfig = px.histogram(train_csv_df, x=\"class_name\", color=\"class_name\",opacity=0.7,\n labels={\"class_name\":\"Abnormality\"},\n title=\"Annotations Per Class\",\n ).update_xaxes(categoryorder=\"total descending\")\nfig.update_layout(showlegend=False,\n xaxis_title=\"\",\n yaxis_title=\"Annotations Per Class\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculating percentage distribution of each class annotations\ndef percent_distribution(train_csv_df):\n # Get the count for each label\n label_counts = train_csv_df.class_name.value_counts()\n\n # Get total number of samples\n total_samples = len(train_csv_df)\n\n # Count the number of items in each class\n for i in range(len(label_counts)):\n label = label_counts.index[i]\n count = label_counts.values[i]\n percent = int((count / total_samples) * 10000) / 100\n print(\"{:<30s}: {} or {}%\".format(label, count, percent))\n\npercent_distribution(train_csv_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Annotations\n\n"},{"metadata":{},"cell_type":"markdown","source":"#### Annotations per Unique Image"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"\\n... Total number of unique images = {train_csv_df['image_id'].nunique()} ...\\n\")\n\n# Count of the number of annotations per unique image\ntrain_csv_df['image_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From plot below, it can be observed that the training images contain\n* atleast 3 annotations (1 distinct object annotation by 3 radiologists)\n* at most 57 annotations \n\n
The distribution overall is heavily skewed. The vast majority of images only have 3 annotations (~11,000 out of 15,000 images)"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(train_csv_df.image_id.value_counts(), \n log_y=True, color_discrete_sequence=['indianred'], opacity=0.7,\n labels={\"value\":\"Annotations Per Image\"},\n title=\"Distribution of Annotations per each unique CXR scan\" \\\n \"(Log Scale for Y-Axis)\",\n )\nfig.update_layout(showlegend=False,\n xaxis_title=\"Annotation Count\",\n yaxis_title=\"Unique Images Count\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at a sample image with only 3 annotations in the train.csv file."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_df[train_csv_df['image_id']=='000434271f63a053c4128a0ba6352c7f']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's take a look at a sample image with 57 annotations in the train.csv file."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_df[train_csv_df['image_id']=='03e6ecfa6f6fb33dfeac6ca4f9b459c9']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Unique Annotations per Unique Image\n\nThe goal of this is to determine the distribution of distinct diseases within the same patient. So for example, if a radiologist identifies 8 modules of same disease in an image, we count it as one annotation.\n\nFrom plot below, it can be observed that the training images contain\n* Images contain no more than 10 unique abnormalities (out of a possible 14)\n* The more unique abnormalities present in an image, the rarer it is. For example, out of 15,000 patients there are only 4 patients diagnosed with 10 distinct diseases \n* The vast majority of images only have 1 annotations (~11,000 out of 15,000 images)"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(train_csv_df.groupby('image_id')[\"class_name\"].unique().apply(lambda x: len(x)), \n log_y=True, color_discrete_sequence=['skyblue'], opacity=0.7,\n labels={\"value\":\"Number of Distinct Annotations\"},\n title=\"Distribution of Distinct Diseases per patient\" \\\n \"(Log Scale for Y-Axis)\",\n )\nfig.update_layout(showlegend=False,\n xaxis_title=\"Number of Distinct Annotations\",\n yaxis_title=\"Count of Patients\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Radiologists\n\nThe `rad_id` column indicates the the ID of the radiologist that made the observation. Remember, three radiologists will annotate a given image out of a pool of seventeen possible radiologists, where the radiologist ID is encoded from R1 to R17.\n\n#### Annotations per Radiologist\n\nFrom plot below, it can be observed that\n* 3 of the radiologists (R9, R10, & R8 in that order) are responsible for the vast majority of annotations (~60% of all annotations)\n* Among the other 14 radiologists there is some variation around the number of annotations made, however, these 14 radiologists all made between 3121 annotations and 812 annotations\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(train_csv_df, x=\"rad_id\", color=\"rad_id\",opacity=0.85,\n labels={\"rad_id\":\"Radiologist ID\"},\n title=\"Distribution of Annotations Per Radiologist\",\n ).update_xaxes(categoryorder=\"total descending\")\nfig.update_layout(showlegend=False,\n xaxis_title=\"Radiologist ID\",\n yaxis_title=\"Number of Annotations\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Do the radiologists agree or disagree when they independently review CXRs?\n\nWe will check for No findings class to check if all 3 radiologists agree or not. Thus we will aggregate the image_id by class 14 and check if the count is 3 or not."},{"metadata":{"trusted":true},"cell_type":"code","source":"is_normal_df = train_csv_df.groupby(\"image_id\")[\"class_id\"].agg(lambda s: (s == 14).sum()).reset_index().rename({\"class_id\": \"num_normal_annotations\"}, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"is_normal_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"is_normal_df[(is_normal_df[\"num_normal_annotations\"]!=3) & (is_normal_df[\"num_normal_annotations\"]!=0)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thus all the radioligists agree whether a CXR is healthy or abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"abnormality_0 = train_csv_df.groupby(\"image_id\")[\"class_id\"].agg(lambda s: (s == 0).sum()).reset_index().rename({\"class_id\": \"num_normal_annotations\"}, axis=1)\nlen(abnormality_0[(abnormality_0[\"num_normal_annotations\"]!=3) & (abnormality_0[\"num_normal_annotations\"]!=0)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But radiologists don't always agree on an abnormality"},{"metadata":{},"cell_type":"markdown","source":"#### Creating dictionary mappings\n\nHere we are creating the mappings between\n* `class_id` and `class_name`\n* `class_name` and `class_id`\n* `class_id` and class color"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nLABEL_COLORS = [px.colors.label_rgb(px.colors.convert_to_RGB_255(x)) for x in sns.color_palette(\"Spectral\", 15)]\n\n# Create dictionary mappings\nint_2_str = {i:train_csv_df[train_csv_df[\"class_id\"]==i].iloc[0][\"class_name\"] for i in range(15)}\nstr_2_int = {v:k for k,v in int_2_str.items()}\nint_2_clr = {str_2_int[k]:LABEL_COLORS[i] for i,k in enumerate(sorted(str_2_int.keys()))}\n\nprint(\"\\n... Dictionary Mapping Class Integer to Class String Representation [int_2_str]...\\n\")\ndisplay(int_2_str)\n\nprint(\"\\n... Dictionary Mapping Class String to Class Integer Representation [str_2_int]...\\n\")\ndisplay(str_2_int)\n\nprint(\"\\n... Dictionary Mapping Class Integer to Color Representation [str_2_clr]...\\n\")\ndisplay(int_2_clr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Annotations per Radiologists based on Class type \n\nHere we would like to identify if all 17 radiologists were able to see and annotate all 15 classes or not.\n\nFrom plot below, it can be observed that\n\n* Among the 17 radiologists, 7 of them (R1 through R7) have only ever annotated images as `No finding`\n* When compared to the 3 main radiologists (R8 through R10), the annotations made by other 7 radiologists are heavily skewed towards the `No finding` label\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure()\n\nfor i in range(15):\n fig.add_trace(go.Histogram(\n x=train_csv_df[train_csv_df[\"class_id\"]==i][\"rad_id\"],\n marker_color=int_2_clr[i],\n name=f\"{int_2_str[i]}\"\n ))\n\nfig.update_xaxes(categoryorder=\"total descending\")\nfig.update_layout(title=\"Distribution of Class Label Annotations by Radiologist\",\n barmode='stack',\n xaxis_title=\"Radiologist ID\",\n yaxis_title=\"Number of Annotations\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Bounding Box Coordinates"},{"metadata":{},"cell_type":"markdown","source":"The **`x_min`**, **`y_min`**, **`x_max`**, and **`y_max`** columns indicate the location of the annotated object bounding box, where the top-left corner is represented by the tuple (**`x_min`**, **`y_min`**) and the bottom-right corner is represented by the tuple (**`x_max`**, **`y_max`**).\n\nA value of **`NaN`** coincides with a label 14 (**`No finding`**) and means that there is nothing to annotate (healthy x-ray).
\nFor the purpose of examining these columns we will only be examining rows where the objects have been annotated with a bounding box
\n(i.e. All rows with a label of **`No finding`** will be discarded)
\n\nThe important thing to focus on will be identifying for each class the approximate range of locations the annotations are found in and the intensity of the locations within the heatmap.\n\n**From the heatmaps plotted below we can ascertain the following information**\n* Aortic Enlargement (CLASS-ID: 0)\n * Heatmap distribution is slightly oval (vertical) and is very tight and intense, located in the centre of the image (slight drift to the top-right).\n* Atelectasis (CLASS-ID: 1)\n * Heatmap distribution is lung shaped and relatively diffuse with a circular focus on the upper-left part of the left lung.\n* Calcification (CLASS-ID: 2)\n * Heatmap distribution is lung shaped and relatively diffuse with a oval (vertical) focus on the top-left edge of the right lung.\n* Cardiomegaly (CLASS-ID: 3)\n * Heatmap distribution is rectangular and is very tight and intense, located in the bottom-centre (to bottom-centre-right) of the image.\n* Consolidation (CLASS-ID: 4)\n * Heatmap distribution is lung shaped and relatively diffuse, the focus of the distribution covers the entire left lung.\n* ILD (CLASS-ID: 5)\n * Heatmap distribution is lung shaped and relatively diffuse, the focus leans a little towards the centre of the lungs.\n* Infiltration (CLASS-ID: 6)\n * Heatmap distribution is lung shaped and relatively diffuse, the focus of the distribution covers the entire left lung.\n* Lung Opacity (CLASS-ID: 7)\n * Heatmap distribution is lung shaped and relatively diffuse, the focus of the distribution covers the entire left lung.\n* Nodule/Mass (CLASS-ID: 8)\n * Heatmap distribution is lung shaped and relatively diffuse, the focus leans a little towards the centre of the lungs. (NOTE: The diffusion pattern looks patchy... probably due to smaller bounding boxes)\n* Other Lesion (CLASS-ID: 9)\n * Heatmap distribution is incredibly diffuse and covers most of the image, the focus is towards a vertical-strip in the centre of the image.\n* Pleural Effusion (CLASS-ID: 10)\n * Heatmap distribution is lung shaped (slightly more rectangular?) and relatively diffuse, the focus is towards the bottom of the lungs and although both lungs are covered, the left lung has a stronger focus.\n* Pleural Thickening (CLASS-ID: 11)\n * Heatmap distribution is vaguely lung shaped (patches near top and focus trails down exterior lung edge fading as it goes), the focus is towards the top of the lungs is oval (horizontal).\n* Pneumothorax (CLASS-ID: 12)\n * Heatmap distribution is lung shaped (more rectangular), the focus is on the entire left lung however the right lung has some diffuse coverage.\n* Pulmonary Fibrosis (CLASS-ID: 13)\n * Heatmap distribution is vaguely lung shaped (patches near top and focus trails down lung fading as it goes), the focus is towards the top of the lung and it is oval."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport tqdm\nfrom tqdm.notebook import tqdm\nimport pydicom\n\n\n# Get paths to images where bboxes exist `class_id!=14`\nbbox_df = train_csv_df[train_csv_df.class_id!=14].reset_index(drop=True)\nBBOX_PATHS = [\n os.path.join(TRAIN_DIR, name+\".dicom\") \\\n for name in bbox_df.image_id.unique()\n]\n\n# Initalize our map for image sizes\nsizes_of_images_w_bboxes = {}\n\n# ############################################################### #\n# ############## THIS STEP WILL TAKE 10 MINUTES ############## #\n# ############################################################### #\n#\n# Get the image sizes so we can resize the bboxes all based on a static size\n# so that we can generate a heatmap that is representative of the actual\n# locations of annotations\nfor path in tqdm(BBOX_PATHS, total=len(BBOX_PATHS)):\n dicom = pydicom.read_file(path)\n sizes_of_images_w_bboxes[path[:-6].rsplit(\"/\", 1)[1]] = \\\n (dicom.Rows, dicom.Columns)\n# ############################################################### #\n\n# Create new dataframe columns for the source image width and height\nbbox_df[\"img_height\"] = bbox_df[\"image_id\"].map(lambda x: sizes_of_images_w_bboxes[x][0])\nbbox_df[\"img_width\"] = bbox_df[\"image_id\"].map(lambda x: sizes_of_images_w_bboxes[x][1])\n\n# Create new dataframe columns for the bboxes that is a \n# percentage of the respective source image width and height\n# -- i.e. if x_min is 100 and the image width is 1000 than frac_x_min is 0.1\n# -- i.e. if y_max is 28 and the image height is 900 than frac_y_max is 0.031\n#\ndef create_fractional_bbox_coordinates(row):\n \"\"\" Function to return bbox coordiantes as fractions from DF row \"\"\"\n frac_x_min = row[\"x_min\"]/row[\"img_width\"]\n frac_x_max = row[\"x_max\"]/row[\"img_width\"]\n frac_y_min = row[\"y_min\"]/row[\"img_height\"]\n frac_y_max = row[\"y_max\"]/row[\"img_height\"]\n return frac_x_min, frac_x_max, frac_y_min, frac_y_max\n\n# This will allow us to pick a heat-map size and make sure that we can use\n# all of the bounding boxes and scale them appropriately\n# -- NOTE: We will most likely default the heatmap to the average\n# image shape so that there is as little distortion as possible\nbbox_df[\"frac_x_min\"], bbox_df[\"frac_x_max\"], bbox_df[\"frac_y_min\"], bbox_df[\"frac_y_max\"] = \\\n zip(*bbox_df.apply(create_fractional_bbox_coordinates, axis=1))\n\n# # Record some important values for later\nave_src_img_height = np.mean([size[0] for size in sizes_of_images_w_bboxes.values()], dtype=np.int32)\nave_src_img_width = np.mean([size[1] for size in sizes_of_images_w_bboxes.values()], dtype=np.int32)\n\n# # Preview the dataframe\nbbox_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib\n\n# DEFAULT\nHEATMAP_SIZE = (ave_src_img_height, ave_src_img_width, 14)\n\n# Initialize\nheatmap = np.zeros((HEATMAP_SIZE), dtype=np.int16)\nbbox_np = bbox_df[[\"class_id\", \"frac_x_min\", \"frac_x_max\", \"frac_y_min\", \"frac_y_max\"]].to_numpy()\nbbox_np[:, 1:3] *= ave_src_img_width\nbbox_np[:, 3:5] *= ave_src_img_height\nbbox_np = np.floor(bbox_np).astype(np.int16)\n\n# Color map stuff\ncustom_cmaps = [\n matplotlib.colors.LinearSegmentedColormap.from_list(\n colors=[(0.,0.,0.), c, (0.95,0.95,0.95)], \n name=f\"custom_{i}\") for i,c in enumerate(sns.color_palette(\"Spectral\", 15))\n]\ncustom_cmaps.pop(8) # Remove No-Finding\n\nfor row in tqdm(bbox_np, total=bbox_np.shape[0]):\n heatmap[row[3]:row[4]+1, row[1]:row[2]+1, row[0]] += 1\n \nfig = plt.figure(figsize=(20,25))\nplt.suptitle(\"Heatmaps Showing Bounding Box Placement\\n \", fontweight=\"bold\", fontsize=16)\nfor i in range(15):\n plt.subplot(4, 4, i+1)\n if i==0:\n plt.imshow(heatmap.mean(axis=-1), cmap=\"bone\")\n plt.title(f\"Average of All Classes\", fontweight=\"bold\")\n else:\n plt.imshow(heatmap[:, :, i-1], cmap=custom_cmaps[i-1])\n plt.title(f\"{int_2_str[i-1]} – ({i})\", fontweight=\"bold\")\n \n plt.axis(False)\nfig.tight_layout(rect=[0, 0.03, 1, 0.97])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"INVESTIGATE THE SIZES OF BOUNDING BOXES AND THE IMPACT OF CLASS\n\nAs we wish to examine the average, as well as the upper and lower limits for various class-based bounding box statistics, we will use a box plot to investigate. To make things easier to understand let us consider the following basic buckets.\n\nBounding Box Area - Median\n* Under 0.01 –– Smallest\n* 0.01 to 0.02 –– Small\n* 0.02 to 0.04 –– Medium\n* 0.04 to 0.06 –– Large\n* Above 0.06 –– Largest\n\nBounding Box Area - Quartile Range\n* Under 0.0075 –– Smallest\n* 0.0075 to 0.0125 –– Small\n* 0.0125 to 0.0250 –– Medium\n* 0.0250 to 0.0500 –– Large\n* Above 0.0500 –– Largest\n\n---\n\n**From the boxplot plotted below we can ascertain the following information**\n* Regarding Aortic Enlargement Box Plot (CLASS-ID: 0)\n * Median Value is Small ––– Quartile Range is Smallest\n* Regarding Atelectasis Box Plot (CLASS-ID: 1)\n * Median Value is Medium ––– Quartile Range is Large\n* Regarding Calcification Box Plot (CLASS-ID: 2)\n * Median Value is Smallest ––– Quartile Range is Medium\n* Regarding Cardiomegaly Box Plot (CLASS-ID: 3)\n * Median Value is Large ––– Quartile Range is Large\n* Regarding Consolidation Box Plot (CLASS-ID: 4)\n * Median Value is Medium ––– Quartile Range is Large\n* Regarding ILD Box Plot (CLASS-ID: 5)\n * Median Value is Largest ––– Quartile Range is Largest\n* Regarding Infiltration Box Plot (CLASS-ID: 6)\n * Median Value is Medium ––– Quartile Range is Large\n* Regarding Lung Opacity Box Plot (CLASS-ID: 7)\n * Median Value is Medium ––– Quartile Range is Large\n* Regarding Nodule/Mass Box Plot (CLASS-ID: 8)\n * Median Value is Smallest ––– Quartile Range is Smallest\n* Regarding Other Lesion Box Plot (CLASS-ID: 9)\n * Median Value is Small ––– Quartile Range is Large\n* Regarding Pleural Effusion Box Plot (CLASS-ID: 10)\n * Median Value is Smallest ––– Quartile Range is Large\n* Regarding Pleural Thickening Box Plot (CLASS-ID: 11)\n * Median Value is Smallest ––– Quartile Range is Smallest\n* Regarding Pneumothorax Box Plot (CLASS-ID: 12)\n * Median Value is Largest ––– Quartile Range is Largest\n* Regarding Pulmonary Fibrosis Box Plot (CLASS-ID: 13)\n * Median Value is Small ––– Quartile Range is Medium\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"LABEL_COLORS_WOUT_NO_FINDING = LABEL_COLORS[:8]+LABEL_COLORS[9:]\n\n# Update bbox dataframe for boxplots\nbbox_df[\"frac_bbox_area\"] = (bbox_df[\"frac_x_max\"]-bbox_df[\"frac_x_min\"])*(bbox_df[\"frac_y_max\"]-bbox_df[\"frac_y_min\"])\nbbox_df[\"class_id_as_str\"] = bbox_df[\"class_id\"].map(int_2_str)\ndisplay(bbox_df.head())\n\nfig = px.box(bbox_df.sort_values(by=\"class_id_as_str\"), x=\"class_id_as_str\", y=\"frac_bbox_area\", color=\"class_id_as_str\", \n color_discrete_sequence=LABEL_COLORS_WOUT_NO_FINDING, notched=True,\n labels={\"class_id_as_str\":\"Class Name\", \"frac_bbox_area\":\"BBox Area (%)\"},\n title=\"DISTRIBUTION OF BBOX AREAS AS % OF SOURCE IMAGE AREA \" \\\n \"(Some Upper Outliers Excluded For Better Visualization)\")\n\nfig.update_layout(showlegend=True,\n yaxis_range=[-0.025,0.4],\n legend_title_text=None,\n xaxis_title=\"\",\n yaxis_title=\"Bounding Box Area %\",\n )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### DICOM metadata"},{"metadata":{},"cell_type":"markdown","source":"So, what information is contained in a .dicom file? We'll use an image that has some abnormalities."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_df[train_csv_df['image_id']=='9a5094b2563a1ef3ff50dc5c7ff71345']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\n\ndicom_sample = pydicom.read_file('../input/vinbigdata-chest-xray-abnormalities-detection/train/9a5094b2563a1ef3ff50dc5c7ff71345.dicom')\ndicom_sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir(dicom_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We might be able to gain some insight from age, sex, rows and columns values of dicom meta data but the rest do not be seem to be very useful."},{"metadata":{"trusted":true},"cell_type":"code","source":"# list containing full paths to image_ids of all dicoms\nTRAIN_DICOM_PATHS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ############################################################### #\n# ############## THIS STEP WILL TAKE 30 MINUTES ############## #\n# ############################################################### #\n\n# converting dicom meta data to pandas dataframe\n\nimport re\nimport tqdm\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nimport pydicom\n\nprog = re.compile('^[A-Z]*')\n\ndef get_dcm_contents(file):\n dcm = pydicom.read_file(file) \n properties = [string for string in dir(dcm) if prog.match(string).group(0)!='']\n dict1 = {'file': file.replace('.dicom', '')} \n dict1.update( { what: dcm[what].value for what in properties if isinstance(dcm[what].value, (bytes, bytearray))!=True } )\n return dict1\n \ntrain_dicom_files = pd.DataFrame( [ get_dcm_contents(file) for file in tqdm(TRAIN_DICOM_PATHS) ] )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# saving df to csv file\ntrain_dicom_files.to_csv('train_dicom_metadata.csv', index=False)\n\ntrain_dicom_files.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# reading data from csv file\ntrain_dicom_csv = pd.read_csv('train_dicom_metadata.csv')\ntrain_dicom_csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking for missing values in dicom metadata\ntrain_dicom_csv.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From average aspect ratio of 0.877 for the X-rays we can say that the X-rays tends to be taller than wide."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\n# calculating mean aspect ratio of all images\ntrain_dicom_csv['Aspect Ratio'] = train_dicom_csv['Columns'] / train_dicom_csv['Rows']\nnp.mean(train_dicom_csv['Aspect Ratio'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Age data does not make much sense. For example age 000Y, Y, 238Y might either mean a missing age or truly an age < 1 years-old. Then there is 000D (66) which might represent the age in days."},{"metadata":{"trusted":true},"cell_type":"code","source":"# to display non-truncated output\npd.options.display.max_rows = 4000\n\ntrain_dicom_csv['PatientAge'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Out of 15,000 patients, we are certain that 3840 are Male and 3514 are Female whereas O might represent missing values."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dicom_csv['PatientSex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"References\n\nhttps://www.kaggle.com/bjoernholzhauer/eda-dicom-reading-vinbigdata-chest-x-ray\n
\nhttps://www.kaggle.com/dschettler8845/visual-in-depth-eda-vinbigdata-competition-data"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}