{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis of the Chest X-Ray Scans\n\nIn this dataset, we are given 18,000 X-Ray scans of the lungs and different radiologists have marked their findings from scans. The X-ray scans of lungs are given in DICOM format so we will have to convert them in np.array format for easy visualization and to use them as input for our object detection model.\n\n\n![](https://www.mxbowenppc.com/wp-content/uploads/2016/01/abnormal-xray-findings.jpg)\n<center><strong>Source: Google Images</strong></center>"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install webcolors\n!pip install gdown","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as ptc\nimport cv2\nimport random\nimport webcolors\nimport altair as alt\nfrom altair import X, Y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_dir = \"../input/vinbigdata-chest-xray-abnormalities-detection\"\n\ntrain_dir = os.path.join(base_dir, 'train')\ntest_dir = os.path.join(base_dir, 'test')\n\ntraining_images = [os.path.join(train_dir, x) for x in os.listdir(train_dir)]\ntest_images = [os.path.join(test_dir, x) for x in os.listdir(test_dir)]\n\nprint(\"There are \"+str(len(training_images))+\" training X-Ray scans\")\nprint(\"There are \"+str(len(test_images))+\" test X-Ray scans\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(base_dir, 'train.csv'))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"There are \"+str(train_df.shape[0])+\" rows in the dataset\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby(['class_name', 'class_id']).agg({'count'})['image_id'].sort_values(by='count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let us briefly look at the above defects/abnormalities\n\n- **Pneumothorax**:\n\nA collapsed lung.\nThis condition occurs when air leaks into the space between the lungs and chest wall. A blunt or penetrating chest injury, certain medical procedures or lung disease can cause a pneumothorax.\nSymptoms include shortness of breath. It is a rare medical condition.\n\n- **Atelectasis**:\n\nComplete or partial collapse of a lung or a section (lobe) of a lung.\nAnaesthesia's effect on the lungs causes almost everyone who undergoes surgery to have some atelectasis. Inhaled objects, asthma and other lung diseases and injuries can also cause atelectasis. It is very common.\n\n- **Consolidation**:\n\nA pulmonary consolidation is a region of normally compressible lung tissue that has filled with liquid instead of air. The condition is marked by induration (swelling or hardening of normally soft tissue) of a normally aerated lung.\n\n- **Calcification**:\n\nCalcification is the accumulation of calcium salts in a body tissue. It normally occurs in the formation of bone, but calcium can be deposited abnormally in soft tissue causing it to harden. The formation of calcified granulomas in the lungs is often due to infections. They’re often discovered when you undergo an imaging procedure such as an X-ray or CT scan.\n\n- **ILD**:\n\nA group of disorders that cause progressive scarring of lung tissue.\nInterstitial lung disease may be caused by long-term exposure to hazardous materials, such as asbestos or coal dust, or it can be caused by an auto-immune disease such as rheumatoid arthritis. Once lung scarring occurs, it's generally irreversible.\n\n- **Infiltration**:\n\nA pulmonary infiltrate is a substance denser than air, such as pus, blood, or protein, which lingers within the parenchyma of the lungs. Pulmonary infiltrates are associated with pneumonia, and tuberculosis. Pulmonary infiltrates can be observed on a chest radiograph.\n\n- **Pleural effusion**:\n\nA build-up of fluid between the tissues that line the lungs and the chest. Fluid can accumulate around the lungs due to poor pumping by the heart or by inflammation.\n\n- **Lung Opacity**:\n\nPulmonary opacification represents the result of a decrease in the ratio of gas to soft tissue (blood, lung parenchyma and stroma) in the lung. When reviewing an area of increased attenuation (opacification) on a chest radiograph or CT it is vital to determine where the opacification is.\n\n- **Nodule/Mass**:\n\nA lung nodule (or mass) is a small abnormal area that is sometimes found during a CT scan of the chest. These scans are done as a part of lung cancer screening, or to check the lungs if symptoms are present. Most lung nodules seen on CT scans are not cancer. They are more often the result of old infections, scar tissue, or other causes.\n\n- **Pulmonary fibrosis**:\n\nPulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly. As pulmonary fibrosis worsens, you become progressively more short of breath.\n\n- **Pleural thickening**:\n\nPleural thickening is a disease that can be caused by asbestos exposure. Asbestos fibers cause tissue in the lungs to scar, which leads to thickening of the pleural lining. Pleural thickening is incurable but treatable. Early pleural thickening has no symptoms, however. As more and more rigid pleural scarring forms around the lungs, it becomes harder for them to fully expand. Pleural thickening may be a symptom of pleural mesothelioma or lung cancer. \n\n- **Cardiomegaly**:\n\nAn enlarged heart, which is usually a sign of another condition.\nCardiomegaly is usually a sign of another condition such as a heart valve problem or heart disease. It may also signal a prior heart attack. It can also occur from bodily stress caused by pregnancy or certain infections.\n\n- **Aortic Enlargement**:\n\nThe aorta is the largest artery and it brings oxygenated blood to all parts of the body. If the walls of the aorta become weak, an enlargement can occur, which is known as an aortic aneurysm. Aneurysms can form in any section of the aorta, but are most common in the abdomen (abdominal aortic aneurysm) or the upper body (thoracic aortic aneurysm).\n\n- **Other Lesion**:\n\nOthers include all abnormalities that do not fall into any other category.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"alt.data_transformers.disable_max_rows()\ncolor = alt.Color('class_name:N') \nclick = alt.selection_multi(encodings=['color'])\nalt.Chart(train_df).mark_bar().encode(\n    x='count()',\n    y='class_name:N',\n    color=alt.condition(click, color, alt.value('lightgray')),\n    tooltip=['count()']\n).add_selection(click)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can clearly see that there is class imbalance in the above datatset. The number of data points with \"No finding\" class is way too higher than the other classes."},{"metadata":{"trusted":true},"cell_type":"code","source":"color = alt.Color('rad_id:N') \nclick = alt.selection_multi(encodings=['color'])\nalt.Chart(train_df).mark_bar().encode(\n    x='count()',\n    y='rad_id:N',\n    color=alt.condition(click, color, alt.value('lightgray')),\n    tooltip=['count()']\n).add_selection(click)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"alt.Chart(train_df).mark_bar().encode(\n    x='count()',\n    y='rad_id:N',\n    color='class_name:N',\n    order=alt.Order(\n      'count()',\n      sort='ascending'\n    ),\n    tooltip=['count()']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we may observe, some of the radiologists have only marked the X-Ray scans as 'No finding' which is intriguing and I guess it is one of the reasons why we need a second opinion when it comes to medical diagnosis as it concerns a human life. This is one of the biggest motivations to pursue this project."},{"metadata":{},"cell_type":"markdown","source":"**NOTE:** Using [raddar](https://www.kaggle.com/raddar) utility script for converting the DIACOM images to np.array"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n\ndef read_xray(path, voi_lut = True, fix_monochrome = True):\n    dicom = pydicom.read_file(path)\n    \n    # VOI LUT (if available by DICOM device) is used to transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n               \n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n        \n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n        \n    return data\n\ndef map_class2id(class_ids, class_names):\n    map_dict = {class_name:class_id for class_name, class_id in zip(class_names, class_ids)}\n    return map_dict\n\ndef map_id2class(class_ids, class_names):\n    map_dict = {class_id:class_name for class_name, class_id in zip(class_names, class_ids)}\n    return map_dict\n\ndef map_id2color(class_ids):\n    colors = [webcolors.hex_to_rgb(\"#\"+''.join([random.choice('0123456789ABCDEF') for j in range(6)])) for i in range(len(class_ids))]\n    map_dict = {class_id:tuple(color) for class_id, color in zip(class_ids, colors)}\n    return map_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_name_id = train_df.groupby(['class_name', 'class_id']).agg({'count'}).index\nclass_names = [x[0] for x in class_name_id]\nclass_ids = [x[1] for x in class_name_id]\n\nclass2id = map_class2id(class_ids, class_names)\nid2color = map_id2color(class_ids)\nid2class = map_id2class(class_ids, class_names)\n\nprint(\"Class to ID mapping \")\nprint(class2id)\nprint('--------------')\nprint(\"ID to RGB color mapping\")\nprint(id2color)\nprint('--------------')\nprint(\"ID to Class mapping \")\nprint(id2class)\nprint('--------------')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Bounding Boxes Visualized\n\nSince we will visualizing the bboxes of the X-Ray scans, we can discard the rows from the dataframe with \"No finding\" class as visualizing them will be useless. "},{"metadata":{"trusted":true},"cell_type":"code","source":"labelled_data = train_df[train_df.class_name!=\"No finding\"]\nprint(labelled_data.class_name.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_annotations(df, image_ids):\n    annot_dict = {}\n    for image in image_ids:\n        tmp_df = df[df['image_id']==image]\n        annot_dict[image] = []\n        for indx, row in tmp_df.iterrows():\n            bbox = [int(row['x_min']), int(row['y_min']), int(row['x_max']), int(row['y_max'])]\n            class_name = row['class_name']\n            color = id2color[class2id[class_name]]\n            annot_dict[image].append([bbox, class_name, color])\n    return annot_dict       ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_xray_class_id(labelled_data, iden, n_limit=5):\n    filter_df = labelled_data[labelled_data['class_id']==iden]\n    print(str(filter_df.shape[0])+\" rows found for class_id = \"+str(iden))\n    if filter_df.shape[0] < n_limit:\n        print('Number of images requested exceeds the total number of images!')\n        print('RE-INITIALIZING n_limit to '+str(filter_df.shape[0]))\n        n_limit = filter_df.shape[0]\n    subset_df = filter_df.sample(n=n_limit)\n    image_ids = subset_df.image_id.unique()\n    annotations_dict = get_annotations(filter_df, image_ids)\n    for image_id, data_ls in annotations_dict.items():\n        img_path = os.path.join(train_dir, image_id+\".dicom\")\n        img = read_xray(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)\n        for data in data_ls:\n            bbox = data[0]\n            name_class = data[1]\n            color = data[2]\n            # add color mask\n            sub_img = img[bbox[1]:bbox[3], bbox[0]:bbox[2]]\n            white_rect = np.uint8(np.ones(sub_img.shape, dtype=np.uint8) * color)\n            res = cv2.addWeighted(sub_img, 0.7, white_rect, 0.3, 1.0)\n            img[bbox[1]:bbox[3], bbox[0]:bbox[2]] = res\n            # add a rectangle and text\n            img = cv2.rectangle(img, (bbox[2], bbox[3]), (bbox[0], bbox[1]), color, 4)\n            img = cv2.putText(img, name_class, (bbox[0]+2, bbox[1]-15), cv2.FONT_HERSHEY_SIMPLEX, 3, color, 5)\n        plt.figure(figsize=(8, 14))\n        plt.imshow(img)\n        plt.title(image_id)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_xray_rad_id(labelled_data, iden, n_limit=5):\n    filter_df = labelled_data[labelled_data['rad_id']==iden]\n    print(str(filter_df.shape[0])+\" rows found for rad_id = \"+str(iden))\n    if filter_df.shape[0] < n_limit:\n        print('Number of images requested exceeds the total number of images!')\n        print('RE-INITIALIZING n_limit to '+str(filter_df.shape[0]))\n        n_limit = filter_df.shape[0]\n    subset_df = filter_df.sample(n=n_limit)\n    image_ids = subset_df.image_id.unique()\n    annotations_dict = get_annotations(filter_df, image_ids)\n    for image_id, data_ls in annotations_dict.items():\n        img_path = os.path.join(train_dir, image_id+\".dicom\")\n        img = read_xray(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)\n        for data in data_ls:\n            bbox = data[0]\n            name_class = data[1]\n            color = data[2]\n            # add color mask\n            sub_img = img[bbox[1]:bbox[3], bbox[0]:bbox[2]]\n            white_rect = np.uint8(np.ones(sub_img.shape, dtype=np.uint8) * color)\n            res = cv2.addWeighted(sub_img, 0.7, white_rect, 0.3, 1.0)\n            img[bbox[1]:bbox[3], bbox[0]:bbox[2]] = res\n            # add a rectangle and text\n            img = cv2.rectangle(img, (bbox[2], bbox[3]), (bbox[0], bbox[1]), color, 4)\n            img = cv2.putText(img, name_class, (bbox[0]+2, bbox[1]-15), cv2.FONT_HERSHEY_SIMPLEX, 3, color, 5)\n        plt.figure(figsize=(8, 14))\n        plt.imshow(img)\n        plt.title(image_id)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_xray_image_id(labelled_data, n_limit=5):\n    image_ids = labelled_data.image_id.unique()\n    image_ids = random.sample(list(image_ids), n_limit)\n    annotations_dict = get_annotations(labelled_data, image_ids)\n    for image_id, data_ls in annotations_dict.items():\n        img_path = os.path.join(train_dir, image_id+\".dicom\")\n        img = read_xray(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)\n        for data in data_ls:\n            bbox = data[0]\n            name_class = data[1]\n            color = data[2]\n            # add color mask\n            sub_img = img[bbox[1]:bbox[3], bbox[0]:bbox[2]]\n            white_rect = np.uint8(np.ones(sub_img.shape, dtype=np.uint8) * color)\n            res = cv2.addWeighted(sub_img, 0.7, white_rect, 0.3, 1.0)\n            img[bbox[1]:bbox[3], bbox[0]:bbox[2]] = res\n            # add a rectangle and text\n            img = cv2.rectangle(img, (bbox[2], bbox[3]), (bbox[0], bbox[1]), color, 4)\n            img = cv2.putText(img, name_class, (bbox[0]+2, bbox[1]-15), cv2.FONT_HERSHEY_SIMPLEX, 3, color, 5)\n        plt.figure(figsize=(8, 14))\n        plt.imshow(img)\n        plt.title(image_id)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_class_id(labelled_data, 7, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_class_id(labelled_data, 6, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_class_id(labelled_data, 10, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_rad_id(labelled_data, \"R9\", 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_rad_id(labelled_data, \"R10\", 3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_rad_id(labelled_data, \"R13\", 3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_xray_image_id(labelled_data, 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the X-Ray scans visualized above, we can see that there is a certain area of the X-Ray scan associated with each abnormality. So, can we approximate the location of an abnormality based on this data? Let us find out.\n\nWe shall be plotting the heatmaps of the bounding boxes to visualize the approximate locations of the abnormalities. To do that, we will be following the steps given below:\n1. Scale all the images to a common dimension (currently all the X-Ray images have different dimensions)\n2. Shift the bounding box co-ordinates according to the new dimension. To do this, we will have finding the scaling factors for the x and y co-ordinates."},{"metadata":{"trusted":true},"cell_type":"code","source":"labelled_images = labelled_data.image_id.unique()\nprint(len(labelled_images))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# In this cell, we saving the height & width of all the X-Ray scans in a dictionary with \"image_id\"\n# as the key. This is a very time-taking process as the number of images to be read is very huge.\n# It took me around 1.5 hours to run it, so I saved the dictionary as a JSON file and saved it on \n# GDrive. Using gdown library, we can download the json file and load the dictionary for next\n# computations.\n\n\n# from tqdm import tqdm\n\n# image_size = {}\n\n# for img_name in tqdm(labelled_images, total=len(labelled_images)):\n#     img_path = os.path.join(train_dir, img_name+'.dicom')\n#     xray_image = read_xray(img_path)\n#     h, w = xray_image.shape\n#     image_size[img_name] = [h, w]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gdown\nurl = 'https://drive.google.com/uc?id=1Ya4cmUPRzl-37K-N0LVPKii4ZqqUA4I4'\noutput = 'image_size.json'\ngdown.download(url, output, quiet=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nimage_size = {}\nwith open('image_size.json') as js:\n    image_size = json.load(js)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labelled_data[\"image_height\"] = labelled_data[\"image_id\"].map(lambda x: image_size[x][0])\nlabelled_data[\"image_width\"] = labelled_data[\"image_id\"].map(lambda x: image_size[x][1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labelled_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def scaling_ratio_calc(data):\n    scale_x_min = data['x_min'] / data['image_width']\n    scale_x_max = data['x_max'] / data['image_width']\n    scale_y_min = data['y_min'] / data['image_height']\n    scale_y_max = data['y_max'] / data['image_height']\n    return scale_x_min, scale_x_max, scale_y_min, scale_y_max","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labelled_data['scale_x_min'], labelled_data['scale_x_max'], labelled_data['scale_y_min'], labelled_data['scale_y_max'] = zip(*labelled_data.apply(scaling_ratio_calc, axis=1))\nlabelled_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom tqdm import tqdm\n\nIMAGE_HEIGHT = int(labelled_data['image_height'].mean())\nIMAGE_WIDTH = int(labelled_data['image_width'].mean())\nprint(\"Average image height is \", IMAGE_HEIGHT)\nprint(\"Average image width is \", IMAGE_WIDTH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"heatmap = np.zeros((IMAGE_HEIGHT, IMAGE_WIDTH, 14), dtype=np.int16)\nbbox_np = labelled_data[[\"class_id\", \"scale_x_min\", \"scale_x_max\", \"scale_y_min\", \"scale_y_max\"]].to_numpy()\nbbox_np[:, 1:3] *= IMAGE_HEIGHT\nbbox_np[:, 3:5] *= IMAGE_WIDTH\nbbox_np = bbox_np.astype(np.int16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for row in tqdm(bbox_np, total=bbox_np.shape[0]):\n    heatmap[row[3]:row[4]+1, row[1]:row[2]+1, row[0]] += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(14,22))\nfor i in range(14):\n    plt.subplot(4, 5, i+1)\n    plt.imshow(heatmap[:, :, i], cmap='magma')\n    plt.title(f\"{id2class[i]}\")\n    plt.axis(False) \nfig.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inferences from above heatmaps\n\n- For Aortic Enlargement, we can see circular/oval shaped heatmap located roughly where aorta is present i.e. slightly top-right from the center.\n\n- Cardiomegaly mainly concerns with an enlarged heart and looking at the concentration of bounding boxes, it makes sense.\n\n- Pleural Thickening as we know is thickening of the lungs' lining and in the heat map, we can see that the heatmap is roughly creating an outline of the lungs.\n\n- Since \"Other Lesion\" class does not particularly represent an abnormality, we can see it is highly dispersed with no clear pattern and it kind of makes sense.\n\n- For the rest of abnormalities, we can see that nearly entire lung is affected by it."},{"metadata":{},"cell_type":"markdown","source":"## Do you know that \"dicom\" images can store some metadata about the X-Ray or CT Scan?\n\nLet us see if we can exploit the metadata available with the X-Rays to understand more about the data patterns. First of all, we'll check what all information of user/patient is present in dicom metadata."},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pydicom.read_file(training_images[10])\ndata","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the above data, we can see two important field related to the patient that we can use in our study: \n- Patient's Sex \n- Patient's Age. \n\nSome of the possible exploration of the data can be to see if there is certain age-group particularly affected by an abnormality/disorder. We can also see if there is any disorder which is gender discriminant."},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata = []\n\nfor img_name in tqdm(labelled_images, total=len(labelled_images)):\n    img_path = os.path.join(train_dir, img_name+'.dicom')\n    data = pydicom.read_file(img_path)\n    if [0x0010, 0x0040] in data:\n        sex = data[0x0010, 0x0040].value\n    else:\n        sex = \"\"\n    if [0x0010, 0x1010] in data:\n        age = data[0x0010, 0x1010].value\n        if age == '':\n            age = \"0Y\"\n    else:\n        age = \"0Y\"\n    age = age.replace('Y', '')\n    # I saw some ages as '000D' which was incomprehensible. It could be an error in dataset creation.\n    # So I have added this check of \"isnumeric\"\n    if age.isnumeric():\n        age = int(age)\n    else:\n        age = 0\n    metadata.append([img_name, age, sex])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata_df = pd.DataFrame(metadata, columns=['Image_Name', 'Age', 'Sex'])\nmetadata_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata_df['Age'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Out of 4394 X-Ray scans, there is no \"age\" information present for 1824 patients."},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata_df['Sex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For Gender, we can see that there is no gender infomation available for around 80 patients.\n\nLet us eliminate the rows with age 0 and no gender information for visualization."},{"metadata":{"trusted":true},"cell_type":"code","source":"eda_age_df = metadata_df[(metadata_df['Age']!=0) & (metadata_df['Sex']!=\"\")]\nprint(eda_age_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.boxplot(y='Age', data=eda_age_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.boxplot(x='Sex', y='Age', data=eda_age_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Surprisingly, we see age > 200 which is impossible unless you are an \"incarnation\" of God. 😜\n\nLet us remove the datapoints with age > 100."},{"metadata":{"trusted":true},"cell_type":"code","source":"eda_age_df = eda_age_df[eda_age_df['Age'] <= 100]\nprint(eda_age_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.boxplot(y='Age', data=eda_age_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above boxplot, we can see that most of patients belong to 55-65 age range."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(25, 25))\nsns.displot(eda_age_df, x=\"Age\", hue=\"Sex\", kind=\"kde\", fill=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(labelled_data.shape)\ndf3 = pd.merge(labelled_data, eda_age_df, left_on=['image_id'], right_on = ['Image_Name'], how = 'left')\nprint(df3.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df3.sort_values(by=\"image_id\").head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df3.dropna(subset=['Image_Name'], inplace=True)\ndf3.sort_values(by=\"image_id\").head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df3.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for defect_class, class_id in class2id.items():\n    eda_defect = df3[df3['class_id'] == class_id]\n    if eda_defect.shape[0] > 0:\n        sns.displot(eda_defect, x=\"Age\", hue=\"Sex\", kind=\"kde\", fill=True)\n        plt.title(defect_class)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inference from density plots\n\n- Except for Pnuemothorax & ILD, the age distribution of patients for all the genders is roughly the same.\n\n- While for most of the abnormalities, the number of X-Ray scans belonging to M is more than F. Only Aortic Enlargement, Calcification & Cardiomegaly are cases where number of patients recorded as F are more than M."},{"metadata":{},"cell_type":"markdown","source":"### That would be all for this EDA Notebook. In case you find it useful, please don't forget to give it an upvote. It will encourage me to work harder.✌️ \n### Happy Learning!😄"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}