{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Bounding Box Fusion\nIn this competition, we are given annotated bounding boxes by 3 radiologists for each image in the training set. We will be scored on our predictions for the test set, which was annotated by the **consensus** of **5** radiologists.  From the paper:\n> Each scan in the training set was independently labeled by 3 radiologists, while each scan in the test set was labeled by the consensus of 5 radiologists.\n\nIt may be safe to assume that annotations from a consensus of 5 radiologists would be similar in number and shape to the annotations of only 1 radiologist. If so, it may make sense to transform the annotations in the training set to something approximating the output of only 1 radiologist.\n\n\n### Implications:\nIf each radiologist in the training set...\n* had an identical # of annotations with equal bounding boxes -> we could use the annotations from 1 radiologist for each image\n* had an identical # of annotations but slightly different bounding boxes -> we could merge the bounding boxes for each annotation\n* had a differing # of annotations, and thus different bounding boxes -> what to do?\n\nThe last scenario above is what we have. There are many options, which mostly depend on how to merge bounding boxes based on how much they overlap and/or assigning weight/confidence to each bounding box (aka we may assign 33% confidence to each annotation - which would boost overlapping boxes).  Or if we want to get fancy: start normalizing radiologists' annotations (perhaps one radiologist sees \"Nodules\" everywhere but others do not).\n\nAdditionally, as noted in our [disussion](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/211035), it seems prudent to assume that only 1 aortic enlargement and only 1 cardiomegaly can appear in any image (we only have one aortic valve and one heart (unless having a very very rare condition)).  The aortic enlargement annotations seem to be uniformly located around the aortic arch.\n\nI have written code below to combine annotations so that there is only 1 aortic enlargement and 1 cardiomegaly in each image.  The code also combines bounding boxes with an intersection over union of > .55.\n\nLet me know what you think!"},{"metadata":{},"cell_type":"markdown","source":"Inspiration and code from:\n\n[@backtracking](https://www.kaggle.com/backtracking): https://www.kaggle.com/backtracking/bounding-boxes-optimization\n\nWeighted-Boxes-Fusion: https://github.com/ZFTurbo/Weighted-Boxes-Fusion\n\nhttps://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install -U iterative-stratification\n!pip install -U datatable\n!pip install -U ensemble-boxes\nimport copy\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom random import randint\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nfrom PIL import Image\nfrom ensemble_boxes import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data\nvin_dir = '/kaggle/input/vinbigdata-chest-xray-abnormalities-detection'\nsizes_dir = '/kaggle/input/vindrimagesizes'\ntrain = pd.read_csv(f'{vin_dir}/train.csv')\ntrain_orig = copy.deepcopy(train)\nsizes = pd.read_csv(f'{sizes_dir}/sizes.csv')\nsizes.columns = ['image_id', 'width', 'height']\nsizes.image_id = sizes.image_id.str[:-4]\ntrain = train.set_index('image_id').join(sizes.set_index('image_id')).reset_index()\nclass_id_to_name_dict = train[['class_name', 'class_id']].drop_duplicates().iloc[:,[1,0]].set_index('class_id').to_dict()['class_name']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Translate our bounding boxes into those readable by ZFTurbo and merge\ndef get_fused_boxes(image_id, records, only_one=False):\n    boxes = records[['x_min', 'y_min', 'x_max', 'y_max']].values\n    pix_multiplier = pd.DataFrame([records.width,records.height,records.width,records.height]).T\n    boxes = [(boxes/(pix_multiplier)).values.tolist()]\n    labels = [records[\"class_id\"].tolist()]\n    scores = [[1]*len(records)]\n    weights = [1]\n\n    iou_thr = 0.55\n    skip_box_thr = 0\n    sigma = 0.1\n    # If we demand only one of the label per image, we set iou threshold to 0\n    if only_one:\n        boxes, scores, labels = weighted_boxes_fusion(boxes, scores, labels, weights=weights, iou_thr=0, skip_box_thr=skip_box_thr)\n    else:\n        boxes, scores, labels = weighted_boxes_fusion(boxes, scores, labels, weights=weights, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n    boxes = boxes * pix_multiplier.iloc[:len(boxes),:]\n    boxes.columns = ['x_min', 'y_min', 'x_max', 'y_max']\n    boxes['class_id'] = labels.astype(int)\n    boxes['image_id'] = image_id\n    return boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Merge annotations\ntrain_w_finding = train.loc[train.class_id != 14]\ntrain_w_finding_image_ids = list(train_w_finding.image_id.unique())\n\ntrain_fused = []\n# For each image with a finding:\nfor image_id in train_w_finding_image_ids:\n    records = copy.deepcopy(train_w_finding.loc[train_w_finding.image_id == image_id,:])\n    # make only one each of cardiomegaly and aortic enlargement boxes\n    for class_id in [0,3]:\n        idx = records.loc[records.class_id==class_id,:].index\n        if len(idx) > 1:\n            finding = records.loc[idx,:]\n            boxes = get_fused_boxes(image_id, finding, only_one=True)\n            train_fused.append(boxes)\n            records = records.drop(idx)\n    \n    boxes = get_fused_boxes(image_id, records)\n    train_fused.append(boxes)\n\ntrain_fused = pd.concat(train_fused)\ntrain_fused['class_name'] = train_fused.class_id.map(class_id_to_name_dict)\ntrain_fused = train_fused[['image_id','class_name', 'class_id', 'x_min', 'y_min', 'x_max', 'y_max']]\n\n# Add 'no findings' back in\nno_finding_idx = train.loc[train.class_id == 14].image_id.drop_duplicates().index\nno_finding_df = copy.deepcopy(train.iloc[no_finding_idx,:])\nno_finding_df[['x_min', 'y_min', 'x_max', 'y_max']] = [0,0,1,1]\nno_finding_df = no_finding_df[['image_id','class_name', 'class_id', 'x_min', 'y_min', 'x_max', 'y_max']]\ntrain_fused = pd.concat([no_finding_df,train_fused])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_xray(path, rescale_color=True):\n\n    dicom = pydicom.read_file(path)\n    with warnings.catch_warnings():\n        warnings.simplefilter('ignore')\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    if rescale_color:\n        data = data - np.min(data)\n        data = data / np.max(data)\n        data = (data * 255).astype(np.uint8)\n    return data\n\ndef plot_scale_compare(imageid, t_old, t_new, size=None):\n    img = read_xray('/kaggle/input/vinbigdata-chest-xray-abnormalities-detection/train/{}.dicom'.format(imageid))\n    if size is not None:\n        img_new = resize(img, size)\n    else:\n        img_new = img\n    \n    fig, (ax1, ax2) = plt.subplots(1, 2)\n    for t, img, ax in zip([t_old, t_new], [img, img_new], [ax1, ax2]): \n        infos = t[t['image_id'] == imageid].sort_values(by='class_id')\n        class_ids = infos['class_id'].unique()\n        label2color = {class_id:[randint(0,255)/255 for i in range(3)] for class_id in class_ids}\n        for index, row in infos.iterrows():\n            rec_min = (row['x_min'], row['y_min'])\n            color = label2color[row['class_id']]\n            rect = patches.Rectangle(rec_min,row['x_max']-row['x_min'],row['y_max']-row['y_min'],\n                                     linewidth=2,edgecolor= color,facecolor='none',label=row['class_name'])\n            ax.add_patch(rect)\n            ax.legend()\n        ax.imshow(img, 'gray')\n    fig.set_size_inches(22,16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualize.  Old on left, new on right\n\nif False: # Only show images with >2 cardiomegalies or aortic enlargements\n    t = train.loc[train.class_id.isin([0,3]),:].groupby('image_id').count().reset_index()\n    image_ids = list(t.loc[t.class_name>1,:].image_id.unique())\nelse:\n    image_ids = train.image_id.unique()\n\nfor image_id in image_ids[:20]:\n    plot_scale_compare(image_id, train, train_fused)    ","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}