{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://www.futuretimeline.net/blog/images/1466-chest-xray-ai-technology.jpg)\n\n

VINBIGDATA - FUSING BBOXES + BUILDING COCO DATASET

\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"Overview\n\n

This notebook explores the different ways to select or fuse multiple bboxes of the same chest abnormality, annotated by several radiologists

\n\n

It also covers the conversion of custom bbox annotations to the COCO format for frameworks such as TensorFlow Object Detection API, Detectron2 etc

\n\n\n**A few key intricacies in this competetion includes the way the training data is provided. To state a few:** \n\n- **The abnormalities are labelled by multiple radiologists and there seems to be multiple bounding boxes for some abnormalities.**\n\n- **Another issue being that some dense abnormality/lesion area may contain multiple labels. The radiologists creates boxes and then they may assign many labels to a single bounding box. This was stated by one of the competetion hosts.**\n\nSo the challenge of this competetion includes handling these issues before or after model training.Some ways to handle this would be to use suppression, selection or fusion techniques below which are covered in this notebook:\n\n- **Non-maximum Suppression (NMS)**\n- **Soft-NMS**\n- **Non-maximum Weighted (NMW)**\n- **Weighted Bboxes Fusion (WBF)**\n\n\n\nThese are generally used after model scoring to get to a consensus of a bounding box where the abnormality is based on the confidence scores, weightage of different models (if ensembling is used) etc.\n\nHere, the challenge lies in the fact that we don't have confidence scores or metrics to assign weightage to the multiple annotations by radiologists. So all radiologists are treated equally and the suppression or fusion of bounding boxes has to be done with these factors out of the picture. This seems to show a behaviour by which the bounding boxes which appear alone, seems to get suppressed. This issue is handled in this notebook by separating out single bounding boxes before any technique is applied.\n\n**Finally, please feel free to suggest any other novel methods to address these issues. Hope everyone finds this useful!**\n\n\n\n
\n
\n\n\n

Check out the training notebook which uses the COCO dataset generated here. It explains the Installation, Data preparation, Training and Inference using the TF2 Object Detection API for EfficientDet models. It can be adapted to various other models with just 1-2 lines of code change.

\n\n\n

VBD EfficientDET TF2 Object Detection API⚡📈

\n\n\nTRAINING NOTEBOOK - https://www.kaggle.com/sreevishnudamodaran/vbd-efficientdet-tf2-object-detection-api\n\n
\n
\n\n

The COCO dataset with Fused Boxes generated as a part of this notebook is public. Please do check it out.

\n\n\n

VinBigData - Coco Dataset with WBF 3x Downscaled

\n\n\nDATASET LINK - https://www.kaggle.com/sreevishnudamodaran/vinbigdata-coco-dataset-with-wbf-3x-downscaled\n\n**Please note that only images with Chest Abnormalities are present in this dataset.**\n\n
\n
\n\n**Class Mapping in the Annotations File:**\n\n 0 - Aortic enlargement\n 1 - Atelectasis\n 2 - Calcification\n 3 - Cardiomegaly\n 4 - Consolidation\n 5 - ILD\n 6 - Infiltration\n 7 - Lung Opacity\n 8 - Nodule/Mass\n 9 - Other lesion\n 10 - Pleural effusion\n 11 - Pleural thickening\n 12 - Pneumothorax\n 13 - Pulmonary fibrosis\n \n \n### Citations:\n\n**Thanks to [raddar](http://https://www.kaggle.com/raddar) for creating the 3x Downsampled Images which is used here:\nhttps://www.kaggle.com/raddar/vinbigdata-competition-jpg-data-3x-downsampled**\n\n**[Weighted Boxes Fusion: ensembling boxes for object detection models paper](https://arxiv.org/abs/1910.13302)**\n\n\n
\n
\n\n[![Ask Me Anything !](https://img.shields.io/badge/Ask%20me-something-1abc9c.svg?style=flat-square&logo=kaggle)](https://www.kaggle.com/sreevishnudamodaran)\n
\n\n![Upvote!](https://img.shields.io/badge/Upvote-If%20you%20like%20my%20work-07b3c8?style=for-the-badge&logo=kaggle)"},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"!pip install ensemble-boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib import rcParams\nsns.set(rc={\"font.size\":9,\"axes.titlesize\":15,\"axes.labelsize\":9,\n \"axes.titlepad\":11, \"axes.labelpad\":9, \"legend.fontsize\":7,\n \"legend.title_fontsize\":7, 'axes.grid' : False})\nimport cv2\nimport json\nimport pandas as pd\nimport glob\nimport os.path as osp\nfrom path import Path\nimport datetime\nimport numpy as np\nfrom tqdm.auto import tqdm\nimport random\nimport shutil\nfrom sklearn.model_selection import train_test_split\n\nfrom ensemble_boxes import *\nimport warnings\nfrom collections import Counter","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the Annotation CSV"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_annotations = pd.read_csv(\"../input/vinbigdata-competition-jpg-data-3x-downsampled/train_downsampled.csv\")\ntrain_annotations.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Selecting Images with Abnormalities"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_annotations = train_annotations[train_annotations.class_id!=14]\ntrain_annotations['image_path'] = train_annotations['image_id'].map(lambda x:os.path.join('../input/vinbigdata-competition-jpg-data-3x-downsampled/train/train', str(x)+'.jpg'))\ntrain_annotations.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imagepaths = train_annotations['image_path'].unique()\nprint(\"Number of Images with abnormalities:\",len(imagepaths))\nanno_count = train_annotations.shape[0]\nprint(\"Number of Annotations with abnormalities:\", anno_count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_img(img, size=(18, 18), is_rgb=True, title=\"\", cmap='gray'):\n plt.figure(figsize=size)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\n\ndef plot_imgs(imgs, cols=2, size=10, is_rgb=True, title=\"\", cmap='gray', img_size=None):\n rows = len(imgs)//cols + 1\n fig = plt.figure(figsize=(cols*size, rows*size))\n for i, img in enumerate(imgs):\n if img_size is not None:\n img = cv2.resize(img, img_size)\n fig.add_subplot(rows, cols, i+1)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n \ndef draw_bbox(image, box, label, color): \n alpha = 0.1\n alpha_box = 0.4\n overlay_bbox = image.copy()\n overlay_text = image.copy()\n output = image.copy()\n\n text_width, text_height = cv2.getTextSize(label.upper(), cv2.FONT_HERSHEY_SIMPLEX, 0.6, 1)[0]\n cv2.rectangle(overlay_bbox, (box[0], box[1]), (box[2], box[3]),\n color, -1)\n cv2.addWeighted(overlay_bbox, alpha, output, 1 - alpha, 0, output)\n cv2.rectangle(overlay_text, (box[0], box[1]-7-text_height), (box[0]+text_width+2, box[1]),\n (0, 0, 0), -1)\n cv2.addWeighted(overlay_text, alpha_box, output, 1 - alpha_box, 0, output)\n cv2.rectangle(output, (box[0], box[1]), (box[2], box[3]),\n color, thickness)\n cv2.putText(output, label.upper(), (box[0], box[1]-5),\n cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 255, 255), 1, cv2.LINE_AA)\n return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = [\n \"__ignore__\",\n \"Aortic_enlargement\",\n \"Atelectasis\",\n \"Calcification\",\n \"Cardiomegaly\",\n \"Consolidation\",\n \"ILD\",\n \"Infiltration\",\n \"Lung_Opacity\",\n \"Nodule/Mass\",\n \"Other_lesion\",\n \"Pleural_effusion\",\n \"Pleural_thickening\",\n \"Pneumothorax\",\n \"Pulmonary_fibrosis\"\n ]\nviz_labels = labels[1:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize Original Bboxes"},{"metadata":{"trusted":true},"cell_type":"code","source":"# map label_id to specify color\n#label2color = [[random.randint(0,255) for i in range(3)] for class_id in viz_labels]\nlabel2color = [[59, 238, 119], [222, 21, 229], [94, 49, 164], [206, 221, 133], [117, 75, 3],\n [210, 224, 119], [211, 176, 166], [63, 7, 197], [102, 65, 77], [194, 134, 175],\n [209, 219, 50], [255, 44, 47], [89, 125, 149], [110, 27, 100]]\n\nthickness = 3\nimgs = []\n\nfor img_id, path in zip(train_annotations['image_id'][:6], train_annotations['image_path'][:6]):\n\n boxes = train_annotations.loc[train_annotations['image_id'] == img_id,\n ['x_min', 'y_min', 'x_max', 'y_max']].values\n img_labels = train_annotations.loc[train_annotations['image_id'] == img_id, ['class_id']].values.squeeze()\n \n img = cv2.imread(path)\n \n for label_id, box in zip(img_labels, boxes):\n color = label2color[label_id]\n img = draw_bbox(img, list(np.int_(box)), viz_labels[label_id], color)\n imgs.append(img)\n\nplot_imgs(imgs, size=9, cmap=None)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploring Techniques to Combine Bboxes\n## Non-maximum Suppression (NMS)"},{"metadata":{"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\n\nfor i, path in tqdm(enumerate(imagepaths[5:8])):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n\n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n print(\"Bboxes before nms:\\n\", boxes_viz)\n print(\"Labels before nms:\\n\", labels_viz)\n \n ## Visualize Original Bboxes\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n \n cls_ids = img_annotations['class_id'].unique().tolist()\n count_dict = Counter(img_annotations['class_id'].tolist())\n print(count_dict)\n\n for cid in cls_ids: \n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n\n weights.append(1)\n \n # Perform NMS\n boxes, scores, box_labels = nms(boxes_list, scores_list, labels_list, weights=weights,\n iou_thr=iou_thr)\n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n\n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n print(\"Bboxes after nms:\\n\", boxes)\n print(\"Labels after nms:\\n\", box_labels)\n \n ## Visualize Bboxes after operation\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n print()\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"Non-max Suppression\", va=\"top\", ha=\"center\", size=25)\nplt.savefig('nms.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Soft-NMS"},{"metadata":{"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\nsigma = 0.1\n\nfor i, path in tqdm(enumerate(imagepaths[5:8])):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n \n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n print(\"Bboxes before soft_nms:\\n\", boxes_viz)\n print(\"Labels before soft_nms:\\n\", labels_viz)\n \n ## Visualize Original Bboxes\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n \n cls_ids = img_annotations['class_id'].unique().tolist()\n count_dict = Counter(img_annotations['class_id'].tolist())\n print(count_dict)\n\n for cid in cls_ids: \n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n\n weights.append(1)\n \n \n # Perform Soft-NMS\n boxes, scores, box_labels = soft_nms(boxes_list, scores_list, labels_list, weights=weights,\n iou_thr=iou_thr, sigma=sigma, thresh=skip_box_thr)\n \n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n \n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n print(\"Bboxes after soft_nms:\\n\", boxes)\n print(\"Labels after soft_nms:\\n\", box_labels)\n \n ## Visualize Bboxes after operation\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n print()\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"Soft NMS\", va=\"top\", ha=\"center\", size=25)\nplt.savefig('snms.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Non-maximum Weighted"},{"metadata":{"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\n\nfor i, path in tqdm(enumerate(imagepaths[5:8])):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n\n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n print(\"Bboxes before non_maximum_weighted:\\n\", boxes_viz)\n print(\"Labels before non_maximum_weighted:\\n\", labels_viz)\n \n ## Visualize Original Bboxes\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n \n cls_ids = img_annotations['class_id'].unique().tolist()\n count_dict = Counter(img_annotations['class_id'].tolist())\n print(count_dict)\n\n for cid in cls_ids: \n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n\n weights.append(1)\n \n\n # Perform Non-maximum Weighted\n boxes, scores, box_labels = non_maximum_weighted(boxes_list, scores_list, labels_list,\n weights=weights, iou_thr=iou_thr,skip_box_thr=skip_box_thr)\n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n\n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n print(\"Bboxes after non_maximum_weighted:\\n\", boxes)\n print(\"Labels after non_maximum_weighted:\\n\", box_labels)\n \n ## Visualize Bboxes after operation\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n print()\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"Non-maximum Weighted\", va=\"top\", ha=\"center\", size=25)\nplt.savefig('nmw.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weighted boxes fusion (WBF)"},{"metadata":{"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\nsigma = 0.1\n\nfor i, path in tqdm(enumerate(imagepaths[5:8])):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n\n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n print(\"Bboxes before WBF:\\n\", boxes_viz)\n print(\"Labels before WBF:\\n\", labels_viz)\n \n ## Visualize Original Bboxes\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n \n cls_ids = img_annotations['class_id'].unique().tolist()\n count_dict = Counter(img_annotations['class_id'].tolist())\n print(count_dict)\n\n for cid in cls_ids: \n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n\n weights.append(1)\n \n\n # Perform WBF\n boxes, scores, box_labels= weighted_boxes_fusion(boxes_list, scores_list, labels_list, weights=weights,\n iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n \n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n\n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n print(\"Bboxes after WBF:\\n\", boxes)\n print(\"Labels after WBF:\\n\", box_labels)\n \n ## Visualize Bboxes after operation\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n print()\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"WBF\", va=\"top\", ha=\"center\", size=25)\nplt.savefig('wbf.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weighted Boxes Fusion seems to give gives better results comparing to others in this situation considering that we don't have the confidence/weights of the annotations done by different radiologists"},{"metadata":{},"cell_type":"markdown","source":"# Building COCO DATASET\n## Train & Validation Split"},{"metadata":{"trusted":true},"cell_type":"code","source":"random.seed(42)\n## 42 - The Answer to the Ultimate Question of Life\nrandom.shuffle(imagepaths)\ntrain_len = round(0.75*len(imagepaths))\ntrain_paths = imagepaths[:train_len]\nval_paths = imagepaths[train_len:]\n\nprint(\"Split Counts\\nTrain Images:\\t\\t{0}\\nVal Images:\\t\\t{1}\"\n .format(len(train_paths), len(val_paths)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## COCO Format Overview\n### Basic Syntax\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n\n```python\n{\n \"type\": \"instances\",\n \"images\": [\n {\n \"file_name\": \"\",\n \"height\": \"\",\n \"width\": \"\",\n \"id\": \"\"\n }\n# .\n# .\n# .\n \n ],\n\n \"categories\": [\n {\n \"supercategory\": \"none\",\n \"name\": \"\",\n \"id\": 0\n },\n {\n \"supercategory\": \"none\",\n \"name\": \"\",\n \"id\": 2\n }\n# .\n# .\n# .\n\n ],\n\n \"annotations\": [\n {\n \"id\": 1,\n \"bbox\": [\n \"\",\n \"\",\n \"\",\n \"\"\n ],\n \"image_id\": \"\",\n\n \"segmentation\": [\n \"\",\n \"\",\n \"\",\n \"\"\n# .\n# .\n# .\n\n ],\n \"ignore\": 0,\n \"area\": \"\",\n \"iscrowd\": 0,\n \"category_id\": \"\"\n },\n\n ],\n\"categories\": [\n {\n \"supercategory\": null,\n \"id\": \"\",\n \"name\": \"\"\n },\n# .\n# .\n# .\n\n]\n}\n```"},{"metadata":{},"cell_type":"markdown","source":"## Defining Structure"},{"metadata":{"trusted":true},"cell_type":"code","source":"now = datetime.datetime.now()\n\ndata = dict(\n info=dict(\n description=None,\n url=None,\n version=None,\n year=now.year,\n contributor=None,\n date_created=now.strftime('%Y-%m-%d %H:%M:%S.%f'),\n ),\n licenses=[dict(\n url=None,\n id=0,\n name=None,\n )],\n images=[\n # license, url, file_name, height, width, date_captured, id\n ],\n type='instances',\n annotations=[\n # segmentation, area, iscrowd, image_id, bbox, category_id, id\n ],\n categories=[\n # supercategory, id, name\n ],\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_name_to_id = {}\nfor i, each_label in enumerate(labels):\n class_id = i - 1 # starts with -1\n class_name = each_label\n if class_id == -1:\n assert class_name == '__ignore__'\n continue\n class_name_to_id[class_name] = class_id\n data['categories'].append(dict(\n supercategory=None,\n id=class_id,\n name=class_name,\n ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Dict Before Adding Annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating The Output Directories"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# !rm -r ./vinbigdata_coco_chest_xray","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_output_dir = \"./vinbigdata_coco_chest_xray/train_images\"\nval_output_dir = \"./vinbigdata_coco_chest_xray/val_images\"\n\nif not osp.exists(train_output_dir):\n os.makedirs(train_output_dir)\n print('Coco Train Image Directory:', train_output_dir)\n \nif not osp.exists(val_output_dir):\n os.makedirs(val_output_dir)\n print('Coco Val Image Directory:', val_output_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Coco Conversion Job\n## Prepare the Training Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"warnings.filterwarnings(\"ignore\", category=UserWarning)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setting the output annotations json file path\ntrain_out_file = './vinbigdata_coco_chest_xray/train_annotations.json'\n\ndata_train = data.copy()\ndata_train['images'] = []\ndata_train['annotations'] = []","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\n\nfor i, path in tqdm(enumerate(train_paths)):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n# print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n \n ## Copy Image \n shutil.copy2(path, train_output_dir)\n \n ## Add Images to annotation\n data_train['images'].append(dict(\n license=0,\n url=None,\n file_name=os.path.join('train_images', image_basename+'.jpg'),\n height=img_array.shape[0],\n width=img_array.shape[1],\n date_captured=None,\n id=i\n ))\n \n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n ## Visualize Original Bboxes every 500th\n if (i%500==0):\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n\n cls_ids = img_annotations['class_id'].unique().tolist()\n \n count_dict = Counter(img_annotations['class_id'].tolist())\n\n for cid in cls_ids:\n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n \n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n weights.append(1)\n \n ## Perform WBF\n boxes, scores, box_labels = weighted_boxes_fusion(boxes_list=boxes_list, scores_list=scores_list,\n labels_list=labels_list, weights=weights,\n iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n area = round((x_max-x_min)*(y_max-y_min),1)\n bbox =[\n round(x_min, 1),\n round(y_min, 1),\n round((x_max-x_min), 1),\n round((y_max-y_min), 1)\n ]\n \n data_train['annotations'].append(dict( id=len(data_train['annotations']), image_id=i,\n category_id=int(label), area=area, bbox=bbox,\n iscrowd=0))\n \n ## Visualize Bboxes after operation every 500th\n if (i%500==0):\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n\nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"WBF\", va=\"top\", ha=\"center\", size=25)\nplt.show()\n \nwith open(train_out_file, 'w') as f:\n json.dump(data_train, f, indent=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the Validation Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Setting the output annotations json file path\nval_out_file = './vinbigdata_coco_chest_xray/val_annotations.json'\n\ndata_val = data.copy()\ndata_val['images'] = []\ndata_val['annotations'] = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iou_thr = 0.5\nskip_box_thr = 0.0001\nviz_images = []\n\nfor i, path in tqdm(enumerate(val_paths)):\n img_array = cv2.imread(path)\n image_basename = Path(path).stem\n# print(f\"(\\'{image_basename}\\', \\'{path}\\')\")\n \n ## Copy Image \n shutil.copy2(path, val_output_dir)\n \n ## Add Images to annotation\n data_val['images'].append(dict(\n license=0,\n url=None,\n file_name=os.path.join('val_images', image_basename+'.jpg'),\n height=img_array.shape[0],\n width=img_array.shape[1],\n date_captured=None,\n id=i\n ))\n \n img_annotations = train_annotations[train_annotations.image_id==image_basename]\n boxes_viz = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_viz = img_annotations['class_id'].to_numpy().tolist()\n \n ## Visualize Original Bboxes every 500th\n if (i%500==0):\n img_before = img_array.copy()\n for box, label in zip(boxes_viz, labels_viz):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n \n boxes_list = []\n scores_list = []\n labels_list = []\n weights = []\n \n boxes_single = []\n labels_single = []\n\n cls_ids = img_annotations['class_id'].unique().tolist()\n \n count_dict = Counter(img_annotations['class_id'].tolist())\n for cid in cls_ids:\n ## Performing Fusing operation only for multiple bboxes with the same label\n if count_dict[cid]==1:\n labels_single.append(cid)\n boxes_single.append(img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().squeeze().tolist())\n\n else:\n cls_list =img_annotations[img_annotations.class_id==cid]['class_id'].tolist()\n labels_list.append(cls_list)\n bbox = img_annotations[img_annotations.class_id==cid][['x_min', 'y_min', 'x_max', 'y_max']].to_numpy()\n \n ## Normalizing Bbox by Image Width and Height\n bbox = bbox/(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n bbox = np.clip(bbox, 0, 1)\n boxes_list.append(bbox.tolist())\n scores_list.append(np.ones(len(cls_list)).tolist())\n weights.append(1)\n \n ## Perform WBF\n boxes, scores, box_labels = weighted_boxes_fusion(boxes_list=boxes_list, scores_list=scores_list,\n labels_list=labels_list, weights=weights,\n iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n \n boxes = boxes*(img_array.shape[1], img_array.shape[0], img_array.shape[1], img_array.shape[0])\n boxes = boxes.round(1).tolist()\n box_labels = box_labels.astype(int).tolist()\n boxes.extend(boxes_single)\n box_labels.extend(labels_single)\n \n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n area = round((x_max-x_min)*(y_max-y_min),1)\n bbox =[\n round(x_min, 1),\n round(y_min, 1),\n round((x_max-x_min), 1),\n round((y_max-y_min), 1)\n ]\n \n data_val['annotations'].append(dict( id=len(data_val['annotations']), image_id=i,\n category_id=int(label), area=area, bbox=bbox,\n iscrowd=0))\n \n ## Visualize Bboxes after operation\n if (i%500==0):\n img_after = img_array.copy()\n for box, label in zip(boxes, box_labels):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"WBF\", va=\"top\", ha=\"center\", size=25)\nplt.show()\n \nwith open(val_out_file, 'w') as f:\n json.dump(data_val, f, indent=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Verify Annotaions"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of Images in the Train Annotations File:\", len(data_train['images']))\nprint(\"Number of Bboxes in the Train Annotations File:\", len(data_train['annotations']))\n\nprint(\"Number of Images in the Val Annotations File:\", len(data_val['images']))\nprint(\"Number of Bboxes in the Val Annotations File:\", len(data_val['annotations']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Converted Example with 2 Images"},{"metadata":{},"cell_type":"markdown","source":"```python\n{\n \"info\": {\n \"description\": null,\n \"url\": null,\n \"version\": null,\n \"year\": 2021,\n \"contributor\": null,\n \"date_created\": \"2021-01-12 13:26:45.546865\"\n },\n \"licenses\": [\n {\n \"url\": null,\n \"id\": 0,\n \"name\": null\n }\n ],\n \"images\": [\n {\n \"license\": 0,\n \"url\": null,\n \"file_name\": \"val_images/a5ef63166c0cc0acf772a7f06ef54fac.jpg\",\n \"height\": 997,\n \"width\": 956,\n \"date_captured\": null,\n \"id\": 0\n },\n {\n \"license\": 0,\n \"url\": null,\n \"file_name\": \"val_images/02acf0e7d0932f9c6a06fd4bbe1f5d90.jpg\",\n \"height\": 1024,\n \"width\": 1024,\n \"date_captured\": null,\n \"id\": 1\n }\n ],\n \"type\": \"instances\",\n \"annotations\": [\n {\n \"id\": 0,\n \"image_id\": 0,\n \"category_id\": 0,\n \"area\": 15507.8,\n \"bbox\": [\n 496.3,\n 300.7,\n 106.0,\n 146.3\n ],\n \"iscrowd\": 0\n },\n {\n \"id\": 1,\n \"image_id\": 0,\n \"category_id\": 3,\n \"area\": 55841.5,\n \"bbox\": [\n 360.0,\n 567.5,\n 390.5,\n 143.0\n ],\n \"iscrowd\": 0\n },\n {\n \"id\": 2,\n \"image_id\": 0,\n \"category_id\": 7,\n \"area\": 9146.2,\n \"bbox\": [\n 345.5,\n 485.5,\n 67.5,\n 135.5\n ],\n \"iscrowd\": 0\n },\n {\n \"id\": 3,\n \"image_id\": 1,\n \"category_id\": 3,\n \"area\": 36659.7,\n \"bbox\": [\n 384.7,\n 468.0,\n 370.3,\n 99.0\n ],\n \"iscrowd\": 0\n },\n {\n \"id\": 4,\n \"image_id\": 1,\n \"category_id\": 0,\n \"area\": 12193.3,\n \"bbox\": [\n 535.0,\n 207.7,\n 99.7,\n 122.3\n ],\n \"iscrowd\": 0\n },\n {\n \"id\": 5,\n \"image_id\": 1,\n \"category_id\": 9,\n \"area\": 13837.0,\n \"bbox\": [\n 453.0,\n 217.0,\n 137.0,\n 101.0\n ],\n \"iscrowd\": 0\n }\n ],\n \"categories\": [\n {\n \"supercategory\": null,\n \"id\": 0,\n \"name\": \"Aortic_enlargement\"\n },\n {\n \"supercategory\": null,\n \"id\": 1,\n \"name\": \"Atelectasis\"\n },\n {\n \"supercategory\": null,\n \"id\": 2,\n \"name\": \"Calcification\"\n },\n {\n \"supercategory\": null,\n \"id\": 3,\n \"name\": \"Cardiomegaly\"\n },\n {\n \"supercategory\": null,\n \"id\": 4,\n \"name\": \"Consolidation\"\n },\n {\n \"supercategory\": null,\n \"id\": 5,\n \"name\": \"ILD\"\n },\n {\n \"supercategory\": null,\n \"id\": 6,\n \"name\": \"Infiltration\"\n },\n {\n \"supercategory\": null,\n \"id\": 7,\n \"name\": \"Lung_Opacity\"\n },\n {\n \"supercategory\": null,\n \"id\": 8,\n \"name\": \"Nodule/Mass\"\n },\n {\n \"supercategory\": null,\n \"id\": 9,\n \"name\": \"Other_lesion\"\n },\n {\n \"supercategory\": null,\n \"id\": 10,\n \"name\": \"Pleural_effusion\"\n },\n {\n \"supercategory\": null,\n \"id\": 11,\n \"name\": \"Pleural_thickening\"\n },\n {\n \"supercategory\": null,\n \"id\": 12,\n \"name\": \"Pneumothorax\"\n },\n {\n \"supercategory\": null,\n \"id\": 13,\n \"name\": \"Pulmonary_fibrosis\"\n }\n ]\n}\n```"},{"metadata":{},"cell_type":"markdown","source":"### Verify Number of Files "},{"metadata":{"trusted":true},"cell_type":"code","source":"!find ./vinbigdata_coco_chest_xray/val_images -type f | wc -l","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!find ./vinbigdata_coco_chest_xray/train_images -type f | wc -l","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Packaging Dataset into Zip for Upload"},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"%%bash\ncd ./vinbigdata_coco_chest_xray\nzip -r ../vinbigdata-coco-dataset-with-wbf-3x-downscaled.zip ./*","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"%%bash\nrm -r ./vinbigdata_coco_chest_xray\nls -ahl","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

THANK YOU! PLEASE UPVOTE

\n\n

HOPE IT WAS USEFUL

\n\n

Check out the Train Notebook below

\n\n\n\n

VBD EfficientDET TF2 Object Detection API⚡📈

\n\n

\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}