{"cells":[{"metadata":{},"cell_type":"markdown","source":"# IMPORTS"},{"metadata":{},"cell_type":"markdown","source":"[Reference](https://www.kaggle.com/dschettler8845/visual-in-depth-eda-vinbigdata-competition-data)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Machine Learning and Data Sciences library\nimport numpy as np\nimport pandas as pd\nimport scipy\n\n# Built-in \nfrom glob import glob\nimport IPython\nimport zipfile\nimport shutil\nimport tqdm\nimport os\n\n# Visualize library\nimport seaborn as sns\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport cv2\n\n# Presets\nFIG_FONT = dict(family='Helvetica, Arial', size=14, color='#7f7f7f')\n\n# Others\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\nprint('\\n...IMPORT COMPLETE...\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import color pallete 14 colors for 14 classes\n# https://seaborn.pydata.org/tutorial/color_palettes.html\n# https://matplotlib.org/3.3.3/tutorials/colors/colormaps.html\n\ncolor_palette = sns.color_palette(\"rainbow\", 14)\n\n# hex code format\ncolor_palette_hex = color_palette.as_hex()\n# (r,b,g) format\n# NOTE: make sure the elements in color tuple are int, not numpy.int\ncolor_palette_rbg = []\nfor color in color_palette:\n    color_rgb = tuple(int(c*255) for c in color)\n    color_palette_rbg.append(color_rgb)\n\nsns.palplot(color_palette_hex)\n# for easy to view, use special color for class \"No finding\"\ncolor_palette_hex.append(\"#fae1dd\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# NOTEBOOK SETUP"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define the root data directory\nDATA_DIR = \"../input/vinbigdata-chest-xray-abnormalities-detection\"\n\n# define the path to the train and test dicom folders\nTRAIN_DIR = os.path.join(DATA_DIR, \"train\")\nTEST_DIR = os.path.join(DATA_DIR, \"test\")\n\n# get all dicom file paths in train and test folders\nTRAIN_DICOM_PATHS = [\n    os.path.join(TRAIN_DIR, file_name) \\\n    for file_name in os.listdir(TRAIN_DIR)\n]\nTEST_DICOM_PATHS = [\n    os.path.join(TEST_DIR, file_name) \\\n    for file_name in os.listdir(TEST_DIR)\n]\n\nprint(f\"\\n...The number of training files is {len(TRAIN_DICOM_PATHS)}...\")\nprint(f\"...The number of testing files is {len(TEST_DICOM_PATHS)}...\")\n\n# define paths to csv files\nTRAIN_CSV = os.path.join(DATA_DIR, \"train.csv\")\nSS_CSV = os.path.join(DATA_DIR, \"sample_submission.csv\")\n\n# create dataframe from csv files\ntrain_df = pd.read_csv(TRAIN_CSV)\nss_df = pd.read_csv(SS_CSV)\n\nprint(\"\\nTRAIN DATAFRAME\\n\")\ndisplay(train_df.head())\n\nprint(\"\\nSAMPLE SUBMISSION DATAFRAME\\n\")\ndisplay(ss_df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EXPLORE CSV DATA"},{"metadata":{},"cell_type":"markdown","source":"1. **IMAGE_ID EXPLORATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\n\n# see number of annotations (bboxes) per an image\n# or number of image contain n annotations\nfig = px.histogram(train_df['image_id'].value_counts(), opacity=.7,\n                   log_y=True, color_discrete_sequence=['salmon'],\n                   labels={'value':'# annotations per image'},\n                   title=\"DISTRIBUTION OF NUMBER ANNOTATIONS PER IMAGE/ PATIENT\")\n\nfig.update_layout(showlegend=False,\n                 xaxis_title=\"# annotations\",\n                 yaxis_title=\"# unique images\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**From the histogram plotted above we can ascertain the following information:**\n* An image can be labeled by up to 3 radiologists, so there can be multiple bboxes for the same abnomalities\n* Annotations can be included \"No finding\"\n* Images contain at least 3 annotations (~11,000 images)\n* Images contain at most 57 annotations (1 image)"},{"metadata":{},"cell_type":"markdown","source":"2. **CLASS_ID/ CLASS_ID EXPLORATION**\n\nThe class_id column indicates the \"label (class_name) encoded as number\". We would rather work with a numeric labels representation. So we will create a dictionary which allow us to translate numeric labels back into their respective string lables, and vice versa."},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a list of class name class name corresponding to class_id order\n\n# get unique class name in df and convert to list\nclass_name = train_df[\"class_name\"].unique().tolist()\n# all class name are in alphabet order except \"No finding\"\nclass_name.sort()\n# move \"No finding\" to the end corresponding to class id 14\nclass_name.append(class_name.pop(class_name.index(\"No finding\")))\n\nprint(class_name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create dictionary mappings\nclassID_to_STR = {i:class_name[i] for i in range(15)}\nclassSTR_to_ID = {class_name[i]:i for i in range(15)}\n\ndisplay(classID_to_STR)\ndisplay(classSTR_to_ID)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# see number of annotations per class\nfig = px.bar(train_df[\"class_id\"].value_counts().sort_index(), opacity=.8,\n             log_y=True, color=class_name,\n             labels={\"value\":\"# annotation of class\"},\n             title=\"NUMBER OF ANNOTATIONS PER CLASS\")\n\nfig.update_layout(legend_title=None,\n                  xaxis_title=\"classes\",\n                  yaxis_title=\"# annotation\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"# non-log scale y axis version\nplt.figure(figsize=(10, 10))\nsns.countplot(x=\"class_id\", data=train_df)\nplt.title(\"Class ID Distribution\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. **RAD_ID EXPLORATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# see number of annotations were labeled by a radiologist\nfig = px.bar(train_df[\"rad_id\"].value_counts(), opacity=.8,\n             labels={\"value\":\"# annotation\", \"color\":\"rad ID\"}, \n             color=train_df[\"rad_id\"].value_counts().keys(),\n             title=\"DISTRIBUTION OF # ANNOTATIONS PER RADIOLOGIST\")\n\nfig.update_layout(legend_title=\"RADIOLOGIST ID\",\n                  xaxis_title=\"Radiologist ID\",\n                  yaxis_title=\"# annotation\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**From the histogram plotted below we can ascertain the following information**\n\n3 of the radiologists (R9, R10, & R8 in that order) are responsible for the vast majority of annotations (~40-50% of all annotations)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# see the expertise of radiologist\n# whether all 17 radiologists can label all 15 distinct objects?\n# or some radiologist just responsible for only 1 or 2 unique abnomalities?\n\n# create dataframe rad_id and annotations (class) they made respectively\ndata = train_df[[\"rad_id\", \"class_id\"]]\ndata = data.value_counts().to_frame().reset_index()\ndata = data.rename(columns= {0: 'count'})\n# sort class in order\ndata = data.sort_values(by=[\"class_id\"])\n# convert to string for showing as legend in graph\ndata[\"class_id\"] = data[\"class_id\"].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# see each radiologist labeled how many bbox for each class\nfig = px.bar(data, x=\"rad_id\", y=\"count\", color=\"class_id\",\n             color_discrete_sequence=color_palette_hex,\n             title=\"DISTRIBUTION OF # EACH CLASS ANNOTATION MADE BY EACH RADIOLOGIST\"\n             ).update_xaxes(categoryorder=\"total descending\")\n\nfig.update_layout(legend_title=\"CLASS ID\",\n                  xaxis_title=\"Radiologist ID\",\n                  yaxis_title=\"# annotation in each class\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# although very least, but some radiologist (except R8, R9, R10) \n# also labeled unique abnomalities\n# zoom on those radiologist\n\n# only plot other 14 radiologist except R8, R9, R10\ndata_zoom = data.drop(data[data[\"rad_id\"].isin([\"R8\", \"R9\", \"R10\"])].index)\n\nfig = px.bar(data_zoom, x=\"rad_id\", y=\"count\", color=\"class_id\",\n             color_discrete_sequence=color_palette_hex,\n             title=\"DISTRIBUTION OF # EACH CLASS ANNOTATION MADE BY EACH RADIOLOGIST\"\n             ).update_xaxes(categoryorder=\"total descending\")\n\nfig.update_layout(legend_title=\"CLASS ID\",\n                  yaxis_range=[0, 500],\n                  xaxis_title=\"Radiologist ID (except R8. R9. R10)\",\n                  yaxis_title=\"# annotation in each class\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**From the second histogram plotted below we can ascertain the following information**\n\n* Among the other 11 radiologists, 7 of them (R1 through R7) have only ever ()100% annotated images as No finding\n* The other 4 radiologists are also heavily skewed towards the No finding label when compared to the main 3 radiologists (R8 through R10). "},{"metadata":{},"cell_type":"markdown","source":"4. **BOUNDING BOX EXPLORATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# the bbox coordinates are represented by (xmin, ymin, xmax, ymax).\n# visualize the heatmaps to see distribution of each class bboxes\n# or the approximate range of locations that the annotations are found \n# and the intensity of the locations within the heatmap.\n\n# get paths to images have bboxes, ignore 'No finding' since they have no bboxes\nbbox_df = train_df[train_df[\"class_id\"] != 14].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# those dicom have different size in image\n# get image size so that we can resize the bboxes in same static size range\n# so that we can generate a heatmap that is representative of the actual\n# locations of annotations\nfrom tqdm import tqdm\n\n# initialize a dictionary with image name and size respectively\nimages_size_dict = {}\nunique_image_name = bbox_df[\"image_id\"].unique()\n\nfor fname in tqdm(unique_image_name, total=len(unique_image_name)):\n    path = os.path.join(TRAIN_DIR, fname+\".dicom\")\n    dicom = pydicom.read_file(path)\n    images_size_dict[fname] = (dicom.Columns, dicom.Rows)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create list of image width and height corresponding to all bbox in dataframe\n# not only unique images\nimage_width, image_height = [], []\nfor i in bbox_df[\"image_id\"]:\n    image_width.append(images_size_dict[i][0])\n    image_height.append(images_size_dict[i][1])\n\n# create 2 column width and height fo respective bbox\nbbox_df[\"img_width\"] = image_width\nbbox_df[\"img_height\"] = image_height\n\n# normalize bbox coordinates in range 0-1\nbbox_df[\"xmin_norm\"] = bbox_df[\"x_min\"] / bbox_df[\"img_width\"]\nbbox_df[\"ymin_norm\"] = bbox_df[\"y_min\"] / bbox_df[\"img_height\"]\nbbox_df[\"xmax_norm\"] = bbox_df[\"x_max\"] / bbox_df[\"img_width\"]\nbbox_df[\"ymax_norm\"] = bbox_df[\"y_max\"] / bbox_df[\"img_height\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bbox_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/craigmthomas/localization-of-findings\n\n# define heatmap size in ration 4:5 as x-ray ratio (width=400px, height=500px)\n# image format (width, height), but in nd array format (row=height, col=width)\nheatmap_size = (500, 400)\n\n# scale bbox coordinates based on heatmap size, because we will draw bbox in heatmap\nbbox_scale = pd.DataFrame()\nbbox_scale[\"class_id\"] = bbox_df[\"class_id\"]\nbbox_scale[\"xmin_norm\"] = bbox_df[\"xmin_norm\"] * heatmap_size[1]\nbbox_scale[\"ymin_norm\"] = bbox_df[\"ymin_norm\"] * heatmap_size[0]\nbbox_scale[\"xmax_norm\"] = bbox_df[\"xmax_norm\"] * heatmap_size[1]\nbbox_scale[\"ymax_norm\"] = bbox_df[\"ymax_norm\"] * heatmap_size[0]\n\nbbox_scale = bbox_scale.astype(int)\nbbox_scale.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define function draw all bboxes same class on same heatmap\ndef draw_bbox_on_heatmap(bboxes, class_id):\n    # initialize empty (full black) heatmap\n    heatmap = np.zeros((heatmap_size))\n    for _, row in bboxes[bboxes[\"class_id\"] == class_id].iterrows():\n        # draw white bboxes on black heatmap based on bboxes coordinate\n        # that mean multiple bboxes which same class_id will be drawn on same heatmap\n        heatmap[row[2]:row[4], row[1]:row[3]] += 1\n    return heatmap","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create subplots\nfig, axes = plt.subplots(nrows=5, ncols=3, sharey=True, sharex=True, \n                         gridspec_kw={'hspace': .1, 'wspace': 0}, figsize=(12, 26))\n\nfor i, ax in enumerate(axes.flatten()):\n    # display heatmap\n    ax.imshow(draw_bbox_on_heatmap(bbox_scale, i),\n              cmap=\"inferno\", interpolation='nearest')\n    # set title for heatmap\n    _ = ax.set_title(str(i) + ' - ' + class_name[i], size=12)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# see what % is the bbox area of each class over the image area?\n# area of each class bbox = ?% area of image\n# because 1 image can contain multiple bboxes for multiple distinct abnomalities\n# and they can be overlap together, % area of each class bbox \n# can show impact of that class\n\nbbox_df[\"bbox_area_norm\"] = (bbox_df[\"xmax_norm\"]-bbox_df[\"xmin_norm\"]) \\\n                          * (bbox_df[\"ymax_norm\"]-bbox_df[\"ymin_norm\"])\nbbox_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create custome legend function for visualize\n# swap class id to name\ndef customLegend(fig, nameSwap):\n    for i, data in enumerate(fig.data):\n        data[\"name\"] = nameSwap[i]\n    return fig","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.box(bbox_df.sort_values(by=[\"class_id\"]), x=\"class_id\", y=\"bbox_area_norm\",\n             color=\"class_id\", color_discrete_sequence=color_palette_hex,\n             title=\"DISTRIBUTION OF BBOX AREAS = % SOURCE IMAGE AREA\")\n\nfig.update_layout(legend_title=\"CLASS NAME\",\n                  yaxis_range=[-0.025,0.4],\n                  xaxis_title = \"Classes\",\n                  yaxis_title = \"bbox area %\")\n\nfig = customLegend(fig, class_name)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **IMAGE DATA**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert dicom to ndarray function\n# from https://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way\n\ndef dicom_to_array(path, voi_lut = True, fix_monochrome = True):\n    dicom = pydicom.dcmread(path)\n    \n    # VOI LUT (if available by DICOM device) is used to \n    # transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n    \n    # make sure the x-ray be visualized in monochorme1, if not x-ray may look inverted\n    if fix_monochrome and dicom.PhotometricInterpretation == 'MONOCHROME1':\n        # convert background to black (color value 0)\n        data = np.max(data) - data\n    \n    #data = data - np.min(data)\n    data = data / np.max(data)                   # normalize in 0-1 range\n    data = (data * 255).astype(np.uint8)         # convert in range RGB 0-255\n    \n    return data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a function receive list of images which already drawn bboxes and plot it\ndef plot_images(images, images_id,rows=2, cols=2):\n    #create subplots\n    fig, axs = plt.subplots(rows, cols, figsize=(16, 20), sharex='col', sharey='row',\n                            gridspec_kw={'hspace': .05, 'wspace': .05})\n    axs = axs.flatten()\n    for img, img_id, ax in zip(images, images_id, axs):\n        # resize all image to same size\n        img = cv2.resize(img, (400, 500))\n        ax.set_title(\"IMAGE ID - \" + img_id)\n        ax.imshow(img, cmap=\"gray\")\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a function draw all bboxes over respective images in given list of image\ndef draw_bboxes_over_images(images_id_list, draw_all_classes, **kwargs):\n    # initialize list of image will be visualized\n    IMAGES = []\n    \n    for image_id in images_id_list:\n        path = os.path.join(TRAIN_DIR, image_id+\".dicom\")\n        image = dicom_to_array(path)\n        # convert grayscale img (2D) to RBG image (3D)\n        image = cv2.cvtColor(image,cv2.COLOR_GRAY2RGB)\n\n        # get all bboxes of respective image_id and convert to ndarray (.values)\n        bboxes = bbox_df.loc[bbox_df[\"image_id\"] == image_id,\n                            [\"class_id\", \"x_min\", \"y_min\", \"x_max\", \"y_max\"]].astype(int)\n        # condition for draw all bboxes (keep all bboxes)\n        # or just draw a specific class_id bboxes (keep bboxes of specific class_id)\n        if draw_all_classes:\n            bboxes = bboxes.values\n        else:\n            # get optional argument kwargs\n            specific_id = kwargs.get('class_id', None)\n            bboxes = bboxes[bboxes[\"class_id\"] == specific_id].values\n\n        # draw all bboxes over image\n        for box in bboxes:\n            class_id = box[0]                            # get class_id \n            class_name = classID_to_STR[class_id]        # get class name from id\n            color = color_palette_rbg[class_id]          # get color corresponding id\n            \n            # draw overlay box (opacity = 0.1) for cooler :))\n            alpha = 0.9\n            overlay_box = image.copy()\n            overlay_box = cv2.rectangle(overlay_box,\n                                       (box[1], box[2]), (box[3], box[4]), color, -1)\n            image = cv2.addWeighted(image, alpha, overlay_box, 1-alpha, 1.0)\n            \n            #draw border and add label text\n            image = cv2.rectangle(image, (box[1], box[2]), (box[3], box[4]), color, 5)\n            image = cv2.putText(image, class_name, (box[1], box[2]-15), \n                                cv2.FONT_HERSHEY_SIMPLEX, 1.75, color, 6)\n        \n        IMAGES.append(image)\n    \n    return IMAGES","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\n\n# list of unique images which have bboxes\nIMAGES_ID = bbox_df[\"image_id\"].unique().tolist()\n\n# choose randome 4 images for visualization\nn = random.randint(0, len(IMAGES_ID))\nimages_id_list = IMAGES_ID[n:n+4]\n\nimages_list = draw_bboxes_over_images(images_id_list, draw_all_classes=True)\nplot_images(images_list, images_id_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **VISUALIZE EACH ABNOMALITY BBOX**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# creat a function get a list of 4 images id based on class_id\n# and plot them\ndef plot_specific_class_bboxes(class_id):\n    # get images unique id based on given class_id\n    IMAGES_ID = bbox_df[bbox_df[\"class_id\"] == class_id]\n    IMAGES_ID = IMAGES_ID[\"image_id\"].unique().tolist()\n    \n    # choose randome 4 images for visualization\n    n = random.randint(0, len(IMAGES_ID))\n    images_id_list = IMAGES_ID[n:n+4]\n    \n    # visualize list of images\n    images_list = draw_bboxes_over_images(images_id_list, draw_all_classes=False, \n                                          class_id=class_id)\n    plot_images(images_list, images_id_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"0. **AORTIC ENLARGMENT**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. **ATELECTASIS**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. **CALCIFICATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. **CARDIOMEGALY**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. **CONSOLIDATION** "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"5. **ILD**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"6. **INFILTRATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"7. **LUNG OPACITY** "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"8. **NODULE/MASS**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"9. **OTHER LESION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"10. **PLEURAL EFFUSION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"11. **PLEURAL THICKENING**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(11)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"12. **PNEUMOTHORAX**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(12)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"13. **PULMONARY FIBROSIS**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_specific_class_bboxes(13)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **FUSING BBOXES**\n[Reference](https://www.kaggle.com/sreevishnudamodaran/vinbigdata-fusing-bboxes-coco-dataset)"},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"# https://github.com/ZFTurbo/Weighted-Boxes-Fusion\n!pip install ensemble-boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import all bboxes fusing methods\nfrom ensemble_boxes import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create function receive a single image and np array of all bboxes\ndef draw_bboxes_over_single_image(image, bboxes):\n    # convert grayscale img (2D) to RBG image (3D)\n    image = cv2.cvtColor(image,cv2.COLOR_GRAY2RGB)\n        \n    # draw all bboxes over image\n    for box in bboxes:\n        class_id = box[0]                            # get class_id \n        class_name = classID_to_STR[class_id]        # get class name from id\n        color = color_palette_rbg[class_id]          # get color corresponding id\n\n        # draw overlay box (opacity = 0.1) for cooler :))\n        alpha = 0.9\n        overlay_box = image.copy()\n        overlay_box = cv2.rectangle(overlay_box, \n                                   (box[1], box[2]), (box[3], box[4]), color, -1)\n        image = cv2.addWeighted(image, alpha, overlay_box, 1-alpha, 1.0)\n\n        #draw border and add label text\n        image = cv2.rectangle(image, (box[1], box[2]), (box[3], box[4]), color, 5)\n        image = cv2.putText(image, class_name, (box[1], box[2]-15), \n                            cv2.FONT_HERSHEY_SIMPLEX, 1.75, color, 6)\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\n\n# create a function receive list of image_id and fusing method\n# for each image, we will plot 2 version: original and after apply fusing method\n# return a list of image contain both version of each image\ndef fusing_bboxes(images_id_list, method, iou_thr, skip_box_thr):\n    # initialize list of image will be visualized\n    IMAGES = []\n    IMAGES_ID = []\n\n    for image_id in images_id_list:\n        # get image info\n        path = os.path.join(TRAIN_DIR, image_id+\".dicom\")\n        image_array = dicom_to_array(path)\n        height, width = image_array.shape\n\n        # -------------------- ORIGINAL IMAGE --------------------\n        # get all bboxes of respective image_id and convert to ndarray (.values)\n        bboxes = bbox_df.loc[bbox_df[\"image_id\"] == image_id, \n                            [\"class_id\", \"x_min\", \"y_min\", \"x_max\", \"y_max\"]].astype(int)\n        bboxes = bboxes.values\n\n        # plot original image with original bboxes\n        image_before = image_array.copy()\n        image_before = draw_bboxes_over_single_image(image_before, bboxes)\n\n        IMAGES.append(image_before)\n        IMAGES_ID.append(image_id)\n        # -------------------- ORIGINAL IMAGE --------------------\n\n        # ------------------- FUSING BBOX IMAGE ------------------\n        # create a dictionary count number of bboxes of each class\n        count_class = Counter(bboxes[:,0].tolist())\n        print(count_class)\n\n        # initialize lists to store class_id and bboxes \n        # which don't have any overlap boxes\n        single_class = []\n        single_box = []\n        # initialize lists to store class_id and bboxes \n        # which have multiple overlap boxes\n        multi_classes = []\n        multi_boxes = []\n        list_scores = []\n\n        unique_classes = np.unique(bboxes[:,0])\n        for class_id in unique_classes:\n            # select all rows have same class_id (value in column 0)\n            specific_class_bboxes = bboxes[np.where(bboxes[:,0]==class_id)]\n            # get coordinates of bboxes, take all values except column 0 (class_id)\n            bboxes_of_class = specific_class_bboxes[:,1:]\n\n            if count_class[class_id] == 1:\n                single_class.append(class_id)\n                single_box.append(bboxes_of_class.tolist())\n            else:\n                # list the class_id, i.g. [3,3,3] or [10,10]\n                list_class = specific_class_bboxes[:,0].tolist()\n                # set confidence score=1 for all bboxes\n                list_scores.append([1 for i in list_class])\n                multi_classes.append(list_class)\n                # normalize bboxes coordinates to 0-1 for using ensemble-boxes package\n                # check data type to use as arguments in ensemble-boxes package\n                bboxes_of_class = bboxes_of_class / (width, height, width, height)\n                multi_boxes.append(bboxes_of_class.tolist())\n\n        # apply bboxes fusion method\n        if method == 'nms':\n            print(method)\n            boxes, scores, box_labels = nms(multi_boxes, list_scores, \n                                            multi_classes, weights=None, \n                                            iou_thr=iou_thr)\n        elif method == 'soft-nms':\n            print(method)\n            boxes, scores, box_labels = soft_nms(multi_boxes, list_scores, \n                                                 multi_classes, weights=None, \n                                                 iou_thr=iou_thr, thresh=skip_box_thr)\n        elif method == 'nmw':\n            print(method)\n            boxes, scores, box_labels = non_maximum_weighted(multi_boxes, list_scores, \n                                                             multi_classes, weights=None, \n                                                             iou_thr=iou_thr, \n                                                             skip_box_thr=skip_box_thr)\n        elif method == 'wbf':\n            print(method)\n            boxes, scores, box_labels = weighted_boxes_fusion(multi_boxes, list_scores,\n                                                              multi_classes, weights=None, \n                                                              iou_thr=iou_thr, \n                                                              skip_box_thr=skip_box_thr)\n\n        # resize bboxes to original size\n        boxes = boxes * (width, height, width, height)\n        # convert list of single box into np array and stack with bboxes after fusing\n        single_box = np.asarray(single_box).reshape((len(single_box),4))\n        boxes = np.row_stack((boxes, single_box))\n\n        # stack single class_id np array with class_id after fusing\n        single_class = np.asarray(single_class)\n        box_labels = np.concatenate([box_labels, single_class])\n\n        # stack label to respective bbox\n        boxes = np.column_stack((box_labels, boxes)).astype(int)\n\n        # plot image with bboxes after fusing\n        image_after = image_array.copy()\n        image_after = draw_bboxes_over_single_image(image_after, boxes)\n\n        IMAGES.append(image_after)\n        IMAGES_ID.append(image_id)\n        # ------------------- FUSING BBOX IMAGE ------------------\n        \n    return IMAGES, IMAGES_ID","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. **NON-MAXIMUN SUPPRESSION (NMS)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# list of unique images which have bboxes\nIMAGES_ID = bbox_df[\"image_id\"].unique().tolist()\n\n# choose randome 2 images for visualization\n# before & after fusing bboxes = 4 images\nn = random.randint(0, len(IMAGES_ID))\nimages_id_list = IMAGES_ID[n:n+2]\nprint(images_id_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fusing method: NMS\n# left column: before fusing\n# right column: after fusing\nimages, images_id = fusing_bboxes(images_id_list, method='nms', \n                                  iou_thr=.5, skip_box_thr=.0001)\nplot_images(images, images_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. **SOFT NON-MAXIMUN SUPPRESSION (SOFT-NMS)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# fusing method: SOFT-NMS\nimages, images_id = fusing_bboxes(images_id_list, method='soft-nms', \n                                  iou_thr=.5, skip_box_thr=.0001)\nplot_images(images, images_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. **NON-MAXIMUM WEIGHTED (NMW)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# fusing method: NMW\nimages, images_id = fusing_bboxes(images_id_list, method='nmw', \n                                  iou_thr=.5, skip_box_thr=.0001)\nplot_images(images, images_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. **WEIGHTED BBOXES FUSION (WBF)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# fusing method: WBF\nimages, images_id = fusing_bboxes(images_id_list, method='wbf', \n                                  iou_thr=.5, skip_box_thr=.0001)\nplot_images(images, images_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}