{
  "id": 208468,
  "title": "Merge annotations using WBF",
  "url": "/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/208468",
  "author_name": "Peter",
  "post_date": "2021-01-03T15:45:05.167000",
  "votes": 45,
  "comment_count": 6,
  "views": 0,
  "content": "<p>In this competition, we have annotations from many radiologists. Often, they annotated the same issue, but with a slightly different target box. We need to handle these in some way. You can not simply average the boxes by <code>class_id</code>, because there are cases where for example the same class has multiple instances, one in the left, one in the right part of the lung.</p>\n<p>One way to solve this if we merge the boxes. </p>\n<p><em>I am not 100% sure that the models (FasterRCNN for example) handle these cases internally or not</em></p>\n<h2>Weighted Box Fusion</h2>\n<p><a href=\"https://github.com/ZFTurbo/Weighted-Boxes-Fusion\" target=\"_blank\">Weighted Box Fusion</a> is a small library, it is an implementation of several methods for ensembling boxes from object detection models.</p>\n<h2>Code</h2>\n<pre><code>import numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm\nfrom ensemble_boxes import *\n\n# ===============================\n# Default WBF config (you can change these)\niou_thr = 0.5\nskip_box_thr = 0.0001\nsigma = 0.1\n# ===============================\n\n# Loading the train DF\ndf = pd.read_csv(\"./../input/train.csv\")\ndf.fillna(0, inplace=True)\ndf.loc[df[\"class_id\"] == 14, ['x_max', 'y_max']] = 1.0\n\nresults = []\nimage_ids = df[\"image_id\"].unique()\n\nfor image_id in tqdm(image_ids, total=len(image_ids)):\n\n    # All annotations for the current image.\n    data = df[df[\"image_id\"] == image_id]\n    data = data.reset_index(drop=True)\n\n    annotations = {}\n    weights = []\n\n    # WBF expects the coordinates in 0-1 range.\n    max_value = data.iloc[:, 4:].values.max()\n    data.loc[:, [\"x_min\", \"y_min\", \"x_max\", \"y_max\"]] = data.iloc[:, 4:] / max_value\n\n    # Loop through all of the annotations\n    for idx, row in data.iterrows():\n\n        rad_id = row[\"rad_id\"]\n\n        if rad_id not in annotations:\n            annotations[rad_id] = {\n                \"boxes_list\": [],\n                \"scores_list\": [],\n                \"labels_list\": [],\n            }\n\n            # We consider all of the radiologists as equal.\n            weights.append(1.0)\n\n        annotations[rad_id][\"boxes_list\"].append([row[\"x_min\"], row[\"y_min\"], row[\"x_max\"], row[\"y_max\"]])\n        annotations[rad_id][\"scores_list\"].append(1.0)\n        annotations[rad_id][\"labels_list\"].append(row[\"class_id\"])\n\n    boxes_list = []\n    scores_list = []\n    labels_list = []\n\n    for annotator in annotations.keys():\n        boxes_list.append(annotations[annotator][\"boxes_list\"])\n        scores_list.append(annotations[annotator][\"scores_list\"])\n        labels_list.append(annotations[annotator][\"labels_list\"])\n\n    # Calculate WBF\n    boxes, scores, labels = weighted_boxes_fusion(\n        boxes_list,\n        scores_list,\n        labels_list,\n        weights=weights,\n        iou_thr=iou_thr,\n        skip_box_thr=skip_box_thr\n    )\n\n    for idx, box in enumerate(boxes):\n        results.append({\n            \"image_id\": image_id,\n            \"class_id\": int(labels[idx]),\n            \"rad_id\": \"wbf\",\n            \"x_min\": box[0] * max_value,\n            \"y_min\": box[1] * max_value,\n            \"x_max\": box[2] * max_value,\n            \"y_max\": box[3] * max_value,\n        })\n\nresults = pd.DataFrame(results)\n</code></pre>",
  "messages": [
    {
      "id": 1137007,
      "postDate": "2021-01-03T15:45:05.167Z",
      "content": "<p>In this competition, we have annotations from many radiologists. Often, they annotated the same issue, but with a slightly different target box. We need to handle these in some way. You can not simply average the boxes by <code>class_id</code>, because there are cases where for example the same class has multiple instances, one in the left, one in the right part of the lung.</p>\n<p>One way to solve this if we merge the boxes. </p>\n<p><em>I am not 100% sure that the models (FasterRCNN for example) handle these cases internally or not</em></p>\n<h2>Weighted Box Fusion</h2>\n<p><a href=\"https://github.com/ZFTurbo/Weighted-Boxes-Fusion\" target=\"_blank\">Weighted Box Fusion</a> is a small library, it is an implementation of several methods for ensembling boxes from object detection models.</p>\n<h2>Code</h2>\n<pre><code>import numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm\nfrom ensemble_boxes import *\n\n# ===============================\n# Default WBF config (you can change these)\niou_thr = 0.5\nskip_box_thr = 0.0001\nsigma = 0.1\n# ===============================\n\n# Loading the train DF\ndf = pd.read_csv(\"./../input/train.csv\")\ndf.fillna(0, inplace=True)\ndf.loc[df[\"class_id\"] == 14, ['x_max', 'y_max']] = 1.0\n\nresults = []\nimage_ids = df[\"image_id\"].unique()\n\nfor image_id in tqdm(image_ids, total=len(image_ids)):\n\n    # All annotations for the current image.\n    data = df[df[\"image_id\"] == image_id]\n    data = data.reset_index(drop=True)\n\n    annotations = {}\n    weights = []\n\n    # WBF expects the coordinates in 0-1 range.\n    max_value = data.iloc[:, 4:].values.max()\n    data.loc[:, [\"x_min\", \"y_min\", \"x_max\", \"y_max\"]] = data.iloc[:, 4:] / max_value\n\n    # Loop through all of the annotations\n    for idx, row in data.iterrows():\n\n        rad_id = row[\"rad_id\"]\n\n        if rad_id not in annotations:\n            annotations[rad_id] = {\n                \"boxes_list\": [],\n                \"scores_list\": [],\n                \"labels_list\": [],\n            }\n\n            # We consider all of the radiologists as equal.\n            weights.append(1.0)\n\n        annotations[rad_id][\"boxes_list\"].append([row[\"x_min\"], row[\"y_min\"], row[\"x_max\"], row[\"y_max\"]])\n        annotations[rad_id][\"scores_list\"].append(1.0)\n        annotations[rad_id][\"labels_list\"].append(row[\"class_id\"])\n\n    boxes_list = []\n    scores_list = []\n    labels_list = []\n\n    for annotator in annotations.keys():\n        boxes_list.append(annotations[annotator][\"boxes_list\"])\n        scores_list.append(annotations[annotator][\"scores_list\"])\n        labels_list.append(annotations[annotator][\"labels_list\"])\n\n    # Calculate WBF\n    boxes, scores, labels = weighted_boxes_fusion(\n        boxes_list,\n        scores_list,\n        labels_list,\n        weights=weights,\n        iou_thr=iou_thr,\n        skip_box_thr=skip_box_thr\n    )\n\n    for idx, box in enumerate(boxes):\n        results.append({\n            \"image_id\": image_id,\n            \"class_id\": int(labels[idx]),\n            \"rad_id\": \"wbf\",\n            \"x_min\": box[0] * max_value,\n            \"y_min\": box[1] * max_value,\n            \"x_max\": box[2] * max_value,\n            \"y_max\": box[3] * max_value,\n        })\n\nresults = pd.DataFrame(results)\n</code></pre>",
      "rawMarkdown": "In this competition, we have annotations from many radiologists. Often, they annotated the same issue, but with a slightly different target box. We need to handle these in some way. You can not simply average the boxes by `class_id`, because there are cases where for example the same class has multiple instances, one in the left, one in the right part of the lung.\n\nOne way to solve this if we merge the boxes. \n\n*I am not 100% sure that the models (FasterRCNN for example) handle these cases internally or not*\n\n## Weighted Box Fusion\n[Weighted Box Fusion](https://github.com/ZFTurbo/Weighted-Boxes-Fusion) is a small library, it is an implementation of several methods for ensembling boxes from object detection models.\n\n\n## Code\n```python\nimport numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm\nfrom ensemble_boxes import *\n\n# ===============================\n# Default WBF config (you can change these)\niou_thr = 0.5\nskip_box_thr = 0.0001\nsigma = 0.1\n# ===============================\n\n# Loading the train DF\ndf = pd.read_csv(\"./../input/train.csv\")\ndf.fillna(0, inplace=True)\ndf.loc[df[\"class_id\"] == 14, ['x_max', 'y_max']] = 1.0\n\nresults = []\nimage_ids = df[\"image_id\"].unique()\n\nfor image_id in tqdm(image_ids, total=len(image_ids)):\n    \n    # All annotations for the current image.\n    data = df[df[\"image_id\"] == image_id]\n    data = data.reset_index(drop=True)\n    \n    annotations = {}\n    weights = []\n    \n    # WBF expects the coordinates in 0-1 range.\n    max_value = data.iloc[:, 4:].values.max()\n    data.loc[:, [\"x_min\", \"y_min\", \"x_max\", \"y_max\"]] = data.iloc[:, 4:] / max_value\n\n    # Loop through all of the annotations\n    for idx, row in data.iterrows():\n        \n        rad_id = row[\"rad_id\"]\n        \n        if rad_id not in annotations:\n            annotations[rad_id] = {\n                \"boxes_list\": [],\n                \"scores_list\": [],\n                \"labels_list\": [],\n            }\n            \n            # We consider all of the radiologists as equal.\n            weights.append(1.0)\n            \n        annotations[rad_id][\"boxes_list\"].append([row[\"x_min\"], row[\"y_min\"], row[\"x_max\"], row[\"y_max\"]])\n        annotations[rad_id][\"scores_list\"].append(1.0)\n        annotations[rad_id][\"labels_list\"].append(row[\"class_id\"])\n        \n    boxes_list = []\n    scores_list = []\n    labels_list = []\n        \n    for annotator in annotations.keys():\n        boxes_list.append(annotations[annotator][\"boxes_list\"])\n        scores_list.append(annotations[annotator][\"scores_list\"])\n        labels_list.append(annotations[annotator][\"labels_list\"])\n        \n    # Calculate WBF\n    boxes, scores, labels = weighted_boxes_fusion(\n        boxes_list,\n        scores_list,\n        labels_list,\n        weights=weights,\n        iou_thr=iou_thr,\n        skip_box_thr=skip_box_thr\n    )\n    \n    for idx, box in enumerate(boxes):\n        results.append({\n            \"image_id\": image_id,\n            \"class_id\": int(labels[idx]),\n            \"rad_id\": \"wbf\",\n            \"x_min\": box[0] * max_value,\n            \"y_min\": box[1] * max_value,\n            \"x_max\": box[2] * max_value,\n            \"y_max\": box[3] * max_value,\n        })\n        \nresults = pd.DataFrame(results)\n```",
      "votes": 45
    },
    {
      "id": 1221755,
      "postDate": "2021-03-01T08:40:49.160Z",
      "content": "<p>I tried box fusion, CV improved, but lb decreased.</p>",
      "rawMarkdown": "I tried box fusion, CV improved, but lb decreased.",
      "replies": [
        {
          "id": 1222517,
          "postDate": "2021-03-01T21:19:26.180Z",
          "content": "<p>Did you try on training or on inference ?</p>",
          "rawMarkdown": "Did you try on training or on inference ?"
        },
        {
          "id": 1222644,
          "postDate": "2021-03-02T02:12:29.147Z",
          "content": "<p>both. some said they got imporved using box fusion. <br>\nMaybe I made some mistakes. I'll check it.</p>",
          "rawMarkdown": "both. some said they got imporved using box fusion. \nMaybe I made some mistakes. I'll check it."
        }
      ]
    },
    {
      "id": 1220338,
      "postDate": "2021-02-27T21:35:31.687Z",
      "content": "<p>Thank you for sharing useful boxes fusion technique! Did it improve the result for you and how much better than training on all boxes?</p>",
      "rawMarkdown": "Thank you for sharing useful boxes fusion technique! Did it improve the result for you and how much better than training on all boxes?"
    },
    {
      "id": 1517837,
      "postDate": "2021-09-20T05:46:04.850Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 1138800,
      "postDate": "2021-01-05T01:23:22.860Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1221755,
      "author_name": "Tian",
      "author_url": "",
      "post_date": "2021-03-01T08:40:49.160000",
      "content": "<p>I tried box fusion, CV improved, but lb decreased.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1222517,
          "author_name": "Rauf  Yagfarov",
          "author_url": "",
          "post_date": "2021-03-01T21:19:26.180000",
          "content": "<p>Did you try on training or on inference ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1222644,
          "author_name": "Tian",
          "author_url": "",
          "post_date": "2021-03-02T02:12:29.147000",
          "content": "<p>both. some said they got imporved using box fusion. <br>\nMaybe I made some mistakes. I'll check it.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1220338,
      "author_name": "Rauf  Yagfarov",
      "author_url": "",
      "post_date": "2021-02-27T21:35:31.687000",
      "content": "<p>Thank you for sharing useful boxes fusion technique! Did it improve the result for you and how much better than training on all boxes?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1517837,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-09-20T05:46:04.850000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1138800,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-01-05T01:23:22.860000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1137007": "In this competition, we have annotations from many radiologists. Often, they annotated the same issue, but with a slightly different target box. We need to handle these in some way. You can not simply average the boxes by `class_id`, because there are cases where for example the same class has multiple instances, one in the left, one in the right part of the lung.\n\nOne way to solve this if we merge the boxes. \n\n*I am not 100% sure that the models (FasterRCNN for example) handle these cases internally or not*\n\n## Weighted Box Fusion\n[Weighted Box Fusion](https://github.com/ZFTurbo/Weighted-Boxes-Fusion) is a small library, it is an implementation of several methods for ensembling boxes from object detection models.\n\n\n## Code\n```python\nimport numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm\nfrom ensemble_boxes import *\n\n# ===============================\n# Default WBF config (you can change these)\niou_thr = 0.5\nskip_box_thr = 0.0001\nsigma = 0.1\n# ===============================\n\n# Loading the train DF\ndf = pd.read_csv(\"./../input/train.csv\")\ndf.fillna(0, inplace=True)\ndf.loc[df[\"class_id\"] == 14, ['x_max', 'y_max']] = 1.0\n\nresults = []\nimage_ids = df[\"image_id\"].unique()\n\nfor image_id in tqdm(image_ids, total=len(image_ids)):\n    \n    # All annotations for the current image.\n    data = df[df[\"image_id\"] == image_id]\n    data = data.reset_index(drop=True)\n    \n    annotations = {}\n    weights = []\n    \n    # WBF expects the coordinates in 0-1 range.\n    max_value = data.iloc[:, 4:].values.max()\n    data.loc[:, [\"x_min\", \"y_min\", \"x_max\", \"y_max\"]] = data.iloc[:, 4:] / max_value\n\n    # Loop through all of the annotations\n    for idx, row in data.iterrows():\n        \n        rad_id = row[\"rad_id\"]\n        \n        if rad_id not in annotations:\n            annotations[rad_id] = {\n                \"boxes_list\": [],\n                \"scores_list\": [],\n                \"labels_list\": [],\n            }\n            \n            # We consider all of the radiologists as equal.\n            weights.append(1.0)\n            \n        annotations[rad_id][\"boxes_list\"].append([row[\"x_min\"], row[\"y_min\"], row[\"x_max\"], row[\"y_max\"]])\n        annotations[rad_id][\"scores_list\"].append(1.0)\n        annotations[rad_id][\"labels_list\"].append(row[\"class_id\"])\n        \n    boxes_list = []\n    scores_list = []\n    labels_list = []\n        \n    for annotator in annotations.keys():\n        boxes_list.append(annotations[annotator][\"boxes_list\"])\n        scores_list.append(annotations[annotator][\"scores_list\"])\n        labels_list.append(annotations[annotator][\"labels_list\"])\n        \n    # Calculate WBF\n    boxes, scores, labels = weighted_boxes_fusion(\n        boxes_list,\n        scores_list,\n        labels_list,\n        weights=weights,\n        iou_thr=iou_thr,\n        skip_box_thr=skip_box_thr\n    )\n    \n    for idx, box in enumerate(boxes):\n        results.append({\n            \"image_id\": image_id,\n            \"class_id\": int(labels[idx]),\n            \"rad_id\": \"wbf\",\n            \"x_min\": box[0] * max_value,\n            \"y_min\": box[1] * max_value,\n            \"x_max\": box[2] * max_value,\n            \"y_max\": box[3] * max_value,\n        })\n        \nresults = pd.DataFrame(results)\n```",
    "1221755": "I tried box fusion, CV improved, but lb decreased.",
    "1220338": "Thank you for sharing useful boxes fusion technique! Did it improve the result for you and how much better than training on all boxes?",
    "1517837": "",
    "1138800": ""
  }
}