{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":24800,"databundleVersionId":1831594,"sourceType":"competition"},{"sourceId":1801996,"sourceType":"datasetVersion","datasetId":1069999},{"sourceId":1810939,"sourceType":"datasetVersion","datasetId":1075804},{"sourceId":1950595,"sourceType":"datasetVersion","datasetId":1164135}],"dockerImageVersionId":30043,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# VinBigData detectron2 train\n\n\nThis competition is object detaction task to find a class and location of thoracic abnormalities from chest x-ray image (radiographs).\n\n`detectron2` is one of the famous pytorch object detection library\n - https://github.com/facebookresearch/detectron2\n\n\n - How to define & use `mapper` to add your customized augmentation.\n - Use validation data during training.\n - Define `Evaluator` and calculating competition metric, AP40.\n - Define `Hook` to calculate validation loss & plotting training/validation loss curve.\n","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents\n\n** [Dataset preparation](#dataset)**
\n** [Installation](#installation)**
\n** [Training method implementations](#train_method)**
\n** [Customizing detectron2 trainer](#custom_trainer) ** [Advanced topic, skip it first time]
\n** - [Mapper for augmentation](#mapper)**
\n** - [Evaluator](#evaluator)**
\n** - [Loss evaluation hook](#loss_hook)**
\n** [Loading Data](#load_data)**
\n** [Data Visualization](#data_vis)**
\n** [Training](#training)**
\n** [Visualize loss curve & competition metric AP40](#vis_loss)**
\n** [Visualization of augmentation by Mapper](#vis_aug)**
\n** [Next step](#next_step)**
","metadata":{}},{"cell_type":"markdown","source":"\n# Dataset preparation\n\nPreprocessing x-ray image format (dicom) into normal png image format is already done by @xhlulu in the below discussion:\n - [Multiple preprocessed datasets: 256/512/1024px, PNG and JPG, modified and original ratio](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/207955).\n\nHere I will just use the dataset [VinBigData Chest X-ray Resized PNG (256x256)](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-256x256) to skip the preprocessing and focus on modeling part.\n\nI also uploaded the original sized png images:\n - [vinbigdata-chest-xray-original-png](https://www.kaggle.com/corochann/vinbigdata-chest-xray-original-png) ([notebook](https://www.kaggle.com/corochann/preprocessing-image-original-size-lossless-png) on kaggle fails due to disk limit)\n\nPlease upvote the dataset as well!","metadata":{}},{"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:37:54.061958Z","iopub.execute_input":"2025-04-14T08:37:54.062206Z","iopub.status.idle":"2025-04-14T08:38:02.081303Z","shell.execute_reply.started":"2025-04-14T08:37:54.062163Z","shell.execute_reply":"2025-04-14T08:38:02.080575Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n# Installation\n\ndetectron2 is not pre-installed in this kaggle docker, so let's install it. \nWe can follow [installation instruction](https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md), we need to know CUDA and pytorch version to install correct `detectron2`.","metadata":{}},{"cell_type":"code","source":"!nvidia-smi","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:02.08414Z","iopub.execute_input":"2025-04-14T08:38:02.084527Z","iopub.status.idle":"2025-04-14T08:38:03.145598Z","shell.execute_reply.started":"2025-04-14T08:38:02.084479Z","shell.execute_reply":"2025-04-14T08:38:03.144802Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!nvcc --version","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:03.148683Z","iopub.execute_input":"2025-04-14T08:38:03.149081Z","iopub.status.idle":"2025-04-14T08:38:04.189104Z","shell.execute_reply.started":"2025-04-14T08:38:03.149033Z","shell.execute_reply":"2025-04-14T08:38:04.18838Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch\n\ntorch.__version__","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:04.190965Z","iopub.execute_input":"2025-04-14T08:38:04.191373Z","iopub.status.idle":"2025-04-14T08:38:05.327839Z","shell.execute_reply.started":"2025-04-14T08:38:04.191324Z","shell.execute_reply":"2025-04-14T08:38:05.326922Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It seems CUDA=10.2 and torch==1.7.0 is used in this kaggle docker image.\n\nSee [installation](https://detectron2.readthedocs.io/tutorials/install.html) for details.","metadata":{}},{"cell_type":"code","source":"!pip install detectron2 -f \\\n https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.7/index.html","metadata":{"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:05.328971Z","iopub.execute_input":"2025-04-14T08:38:05.329219Z","iopub.status.idle":"2025-04-14T08:38:31.746047Z","shell.execute_reply.started":"2025-04-14T08:38:05.32919Z","shell.execute_reply":"2025-04-14T08:38:31.745313Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n# Training method implementations\n\nBasically we don't need to implement neural network part, `detectron2` already implements famous architectures and provides its pre-trained weights. We can finetune these pre-trained architectures.\n\nThese models are summarized in [MODEL_ZOO.md](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md).\n\nIn this competition, we need object detection model, I will choose [R50-FPN](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml) for this kernel.","metadata":{}},{"cell_type":"markdown","source":"## Data preparation\n\n`detectron2` provides high-level API for training custom dataset.\n\nTo define custom dataset, we need to create **list of dict** (`dataset_dicts`) where each dict contains following:\n\n - file_name: file name of the image.\n - image_id: id of the image, index is used here.\n - height: height of the image.\n - width: width of the image.\n - annotation: This is the ground truth annotation data for object detection, which contains following\n - bbox: bounding box pixel location with shape (n_boxes, 4)\n - bbox_mode: `BoxMode.XYXY_ABS` is used here, meaning that absolute value of (xmin, ymin, xmax, ymax) annotation is used in the `bbox`.\n - category_id: class label id for each bounding box, with shape (n_boxes,)\n\n`get_vinbigdata_dicts` is for train dataset preparation and `get_vinbigdata_dicts_test` is for test dataset preparation.\n\nThis `dataset_dicts` contains the metadata for actual data fed into the neural network.
\nIt is loaded beforehand of the training **on memory**, so it should contain all the metadata (image filepath etc) to construct training dataset, but **should not contain heavy data**.
\n\nIn practice, loading all the taining image arrays are too heavy to be loaded on memory, so these are loaded inside `DataLoader` on-demand (This is done by mapper class in `detectron2`, as I will expain later).","metadata":{}},{"cell_type":"code","source":"import pickle\nfrom pathlib import Path\nfrom typing import Optional\n\nimport cv2\nimport numpy as np\nimport pandas as pd\nfrom detectron2.structures import BoxMode\nfrom tqdm import tqdm\n\n\ndef get_vinbigdata_dicts(\n imgdir: Path,\n train_df: pd.DataFrame,\n train_data_type: str = \"original\",\n use_cache: bool = True,\n debug: bool = True,\n target_indices: Optional[np.ndarray] = None,\n use_class14: bool = False,\n):\n debug_str = f\"_debug{int(debug)}\"\n train_data_type_str = f\"_{train_data_type}\"\n class14_str = f\"_14class{int(use_class14)}\"\n cache_path = Path(\".\") / f\"dataset_dicts_cache{train_data_type_str}{class14_str}{debug_str}.pkl\"\n if not use_cache or not cache_path.exists():\n print(\"Creating data...\")\n train_meta = pd.read_csv(imgdir / \"train_meta.csv\")\n if debug:\n train_meta = train_meta.iloc[:500] # For debug....\n\n # Load 1 image to get image size.\n image_id = train_meta.loc[0, \"image_id\"]\n image_path = str(imgdir / \"train\" / f\"{image_id}.png\")\n image = cv2.imread(image_path)\n resized_height, resized_width, ch = image.shape\n print(f\"image shape: {image.shape}\")\n\n dataset_dicts = []\n for index, train_meta_row in tqdm(train_meta.iterrows(), total=len(train_meta)):\n record = {}\n\n image_id, height, width = train_meta_row.values\n filename = str(imgdir / \"train\" / f\"{image_id}.png\")\n record[\"file_name\"] = filename\n record[\"image_id\"] = image_id\n record[\"height\"] = resized_height\n record[\"width\"] = resized_width\n objs = []\n for index2, row in train_df.query(\"image_id == @image_id\").iterrows():\n # print(row)\n # print(row[\"class_name\"])\n # class_name = row[\"class_name\"]\n class_id = row[\"class_id\"]\n if class_id == 14:\n # It is \"No finding\"\n if use_class14:\n # Use this No finding class with the bbox covering all image area.\n bbox_resized = [0, 0, resized_width, resized_height]\n obj = {\n \"bbox\": bbox_resized,\n \"bbox_mode\": BoxMode.XYXY_ABS,\n \"category_id\": class_id,\n }\n objs.append(obj)\n else:\n # This annotator does not find anything, skip.\n pass\n else:\n # bbox_original = [int(row[\"x_min\"]), int(row[\"y_min\"]), int(row[\"x_max\"]), int(row[\"y_max\"])]\n h_ratio = resized_height / height\n w_ratio = resized_width / width\n bbox_resized = [\n float(row[\"x_min\"]) * w_ratio,\n float(row[\"y_min\"]) * h_ratio,\n float(row[\"x_max\"]) * w_ratio,\n float(row[\"y_max\"]) * h_ratio,\n ]\n obj = {\n \"bbox\": bbox_resized,\n \"bbox_mode\": BoxMode.XYXY_ABS,\n \"category_id\": class_id,\n }\n objs.append(obj)\n record[\"annotations\"] = objs\n dataset_dicts.append(record)\n with open(cache_path, mode=\"wb\") as f:\n pickle.dump(dataset_dicts, f)\n\n print(f\"Load from cache {cache_path}\")\n with open(cache_path, mode=\"rb\") as f:\n dataset_dicts = pickle.load(f)\n if target_indices is not None:\n dataset_dicts = [dataset_dicts[i] for i in target_indices]\n return dataset_dicts\n\n\ndef get_vinbigdata_dicts_test(\n imgdir: Path, test_meta: pd.DataFrame, use_cache: bool = True, debug: bool = True,\n):\n debug_str = f\"_debug{int(debug)}\"\n cache_path = Path(\".\") / f\"dataset_dicts_cache_test{debug_str}.pkl\"\n if not use_cache or not cache_path.exists():\n print(\"Creating data...\")\n # test_meta = pd.read_csv(imgdir / \"test_meta.csv\")\n if debug:\n test_meta = test_meta.iloc[:500] # For debug....\n\n # Load 1 image to get image size.\n image_id = test_meta.loc[0, \"image_id\"]\n image_path = str(imgdir / \"test\" / f\"{image_id}.png\")\n image = cv2.imread(image_path)\n resized_height, resized_width, ch = image.shape\n print(f\"image shape: {image.shape}\")\n\n dataset_dicts = []\n for index, test_meta_row in tqdm(test_meta.iterrows(), total=len(test_meta)):\n record = {}\n\n image_id, height, width = test_meta_row.values\n filename = str(imgdir / \"test\" / f\"{image_id}.png\")\n record[\"file_name\"] = filename\n # record[\"image_id\"] = index\n record[\"image_id\"] = image_id\n record[\"height\"] = resized_height\n record[\"width\"] = resized_width\n # objs = []\n # record[\"annotations\"] = objs\n dataset_dicts.append(record)\n with open(cache_path, mode=\"wb\") as f:\n pickle.dump(dataset_dicts, f)\n\n print(f\"Load from cache {cache_path}\")\n with open(cache_path, mode=\"rb\") as f:\n dataset_dicts = pickle.load(f)\n return dataset_dicts\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:31.74777Z","iopub.execute_input":"2025-04-14T08:38:31.74811Z","iopub.status.idle":"2025-04-14T08:38:32.29804Z","shell.execute_reply.started":"2025-04-14T08:38:31.74803Z","shell.execute_reply":"2025-04-14T08:38:32.297251Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- utils ---\nfrom pathlib import Path\nfrom typing import Any, Union\n\nimport yaml\n\n\ndef save_yaml(filepath: Union[str, Path], content: Any, width: int = 120):\n with open(filepath, \"w\") as f:\n yaml.dump(content, f, width=width)\n\n\ndef load_yaml(filepath: Union[str, Path]) -> Any:\n with open(filepath, \"r\") as f:\n content = yaml.full_load(f)\n return content\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.299181Z","iopub.execute_input":"2025-04-14T08:38:32.299415Z","iopub.status.idle":"2025-04-14T08:38:32.305339Z","shell.execute_reply.started":"2025-04-14T08:38:32.299391Z","shell.execute_reply":"2025-04-14T08:38:32.304634Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- configs ---\nthing_classes = [\n \"Aortic enlargement\",\n \"Atelectasis\",\n \"Calcification\",\n \"Cardiomegaly\",\n \"Consolidation\",\n \"ILD\",\n \"Infiltration\",\n \"Lung Opacity\",\n \"Nodule/Mass\",\n \"Other lesion\",\n \"Pleural effusion\",\n \"Pleural thickening\",\n \"Pneumothorax\",\n \"Pulmonary fibrosis\"\n]\ncategory_name_to_id = {class_name: index for index, class_name in enumerate(thing_classes)}\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.306793Z","iopub.execute_input":"2025-04-14T08:38:32.307061Z","iopub.status.idle":"2025-04-14T08:38:32.318647Z","shell.execute_reply.started":"2025-04-14T08:38:32.307035Z","shell.execute_reply":"2025-04-14T08:38:32.317853Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\nYou can refer the [Detectron2 Beginner's Tutorial](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=QHnVupBBn9eR) Colab Notebook (or [version 7 of this kernel](https://www.kaggle.com/corochann/vinbigdata-detectron2-train?scriptVersionId=51628272)) for the simple usage of detectron2 how to train custom dataset. `DefaultTrainer` is used in the example which provides the starting point to train your model with custom dataset.\n\nIt is nice to start with, however I want to customize the training behavior more to improve the model's performance.\nWe can make own Trainer class (`MyTrainer` here) for this purpose, and override methods to provide customized behavior.","metadata":{}},{"cell_type":"markdown","source":"\n## Mapper for augmentation\n\n`Mapper` class is used inside pytorch `DataLoader`. It is responsible for converting `dataset_dicts` into actual data fed into the neural network, and we can insert augmentation process in this Mapper class.\n\n - Ref: [detectron2 docs \"Dataloader\"](https://detectron2.readthedocs.io/en/latest/tutorials/data_loading.html)\n\nI implemented `MyMapper` which uses augmentations implemented in `detectron2`, and `AlbumentationsMapper` which uses albumentations library augmentations.
\nI will demonstrate these augmentations later, so you can skip reading the code and please just jump to next.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nReferenced:\n - https://detectron2.readthedocs.io/en/latest/tutorials/data_loading.html\n - https://www.kaggle.com/dhiiyaur/detectron-2-compare-models-augmentation/#data\n\"\"\"\nimport copy\nimport logging\n\nimport detectron2.data.transforms as T\nimport torch\nfrom detectron2.data import detection_utils as utils\n\n\nclass MyMapper:\n \"\"\"Mapper which uses `detectron2.data.transforms` augmentations\"\"\"\n\n def __init__(self, cfg, is_train: bool = True):\n aug_kwargs = cfg.aug_kwargs\n aug_list = [\n # T.Resize((800, 800)),\n ]\n if is_train:\n aug_list.extend([getattr(T, name)(**kwargs) for name, kwargs in aug_kwargs.items()])\n self.augmentations = T.AugmentationList(aug_list)\n self.is_train = is_train\n\n mode = \"training\" if is_train else \"inference\"\n print(f\"[MyDatasetMapper] Augmentations used in {mode}: {self.augmentations}\")\n\n def __call__(self, dataset_dict):\n dataset_dict = copy.deepcopy(dataset_dict) # it will be modified by code below\n image = utils.read_image(dataset_dict[\"file_name\"], format=\"BGR\")\n\n aug_input = T.AugInput(image)\n transforms = self.augmentations(aug_input)\n image = aug_input.image\n\n # if not self.is_train:\n # # USER: Modify this if you want to keep them for some reason.\n # dataset_dict.pop(\"annotations\", None)\n # dataset_dict.pop(\"sem_seg_file_name\", None)\n # return dataset_dict\n\n image_shape = image.shape[:2] # h, w\n dataset_dict[\"image\"] = torch.as_tensor(image.transpose(2, 0, 1).astype(\"float32\"))\n annos = [\n utils.transform_instance_annotations(obj, transforms, image_shape)\n for obj in dataset_dict.pop(\"annotations\")\n if obj.get(\"iscrowd\", 0) == 0\n ]\n instances = utils.annotations_to_instances(annos, image_shape)\n dataset_dict[\"instances\"] = utils.filter_empty_instances(instances)\n return dataset_dict","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.319841Z","iopub.execute_input":"2025-04-14T08:38:32.320146Z","iopub.status.idle":"2025-04-14T08:38:32.400587Z","shell.execute_reply.started":"2025-04-14T08:38:32.320118Z","shell.execute_reply":"2025-04-14T08:38:32.399883Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\"\"\"\nReferenced:\n - https://detectron2.readthedocs.io/en/latest/tutorials/data_loading.html\n - https://www.kaggle.com/dhiiyaur/detectron-2-compare-models-augmentation/#data\n\"\"\"\nimport albumentations as A\nimport copy\nimport numpy as np\n\nimport torch\nfrom detectron2.data import detection_utils as utils\n\n\nclass AlbumentationsMapper:\n \"\"\"Mapper which uses `albumentations` augmentations\"\"\"\n def __init__(self, cfg, is_train: bool = True):\n aug_kwargs = cfg.aug_kwargs\n aug_list = [\n ]\n if is_train:\n aug_list.extend([getattr(A, name)(**kwargs) for name, kwargs in aug_kwargs.items()])\n self.transform = A.Compose(\n aug_list, bbox_params=A.BboxParams(format=\"pascal_voc\", label_fields=[\"category_ids\"])\n )\n self.is_train = is_train\n\n mode = \"training\" if is_train else \"inference\"\n print(f\"[AlbumentationsMapper] Augmentations used in {mode}: {self.transform}\")\n\n def __call__(self, dataset_dict):\n dataset_dict = copy.deepcopy(dataset_dict) # it will be modified by code below\n image = utils.read_image(dataset_dict[\"file_name\"], format=\"BGR\")\n\n # aug_input = T.AugInput(image)\n # transforms = self.augmentations(aug_input)\n # image = aug_input.image\n\n prev_anno = dataset_dict[\"annotations\"]\n bboxes = np.array([obj[\"bbox\"] for obj in prev_anno], dtype=np.float32)\n # category_id = np.array([obj[\"category_id\"] for obj in dataset_dict[\"annotations\"]], dtype=np.int64)\n category_id = np.arange(len(dataset_dict[\"annotations\"]))\n\n transformed = self.transform(image=image, bboxes=bboxes, category_ids=category_id)\n image = transformed[\"image\"]\n annos = []\n for i, j in enumerate(transformed[\"category_ids\"]):\n d = prev_anno[j]\n d[\"bbox\"] = transformed[\"bboxes\"][i]\n annos.append(d)\n dataset_dict.pop(\"annotations\", None) # Remove unnecessary field.\n\n # if not self.is_train:\n # # USER: Modify this if you want to keep them for some reason.\n # dataset_dict.pop(\"annotations\", None)\n # dataset_dict.pop(\"sem_seg_file_name\", None)\n # return dataset_dict\n\n image_shape = image.shape[:2] # h, w\n dataset_dict[\"image\"] = torch.as_tensor(image.transpose(2, 0, 1).astype(\"float32\"))\n instances = utils.annotations_to_instances(annos, image_shape)\n dataset_dict[\"instances\"] = utils.filter_empty_instances(instances)\n return dataset_dict\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.401762Z","iopub.execute_input":"2025-04-14T08:38:32.40201Z","iopub.status.idle":"2025-04-14T08:38:32.71555Z","shell.execute_reply.started":"2025-04-14T08:38:32.401986Z","shell.execute_reply":"2025-04-14T08:38:32.714876Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Evaluator\n\nTo evaluate validation dataset to calculate competition metric, we need `Evaluator`.\n\nFamouns dataset's evaluator is already implemented in `detectron2`.
\nFor example, many kinds of AP (Average Precision) is calculted in `COCOEvaluator`.
\n`COCOEvaluator` only calculates AP with IoU from 0.50 to 0.95, but we need AP with IoU 0.40.\n\nHere, I modified `COCOEvaluator` implementation to calculate AP with IoU 0.40 and replaced to show this value instead of AP with IoU 0.70.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nOriginal code from https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py\nJust modified to show AP@40\n\"\"\"\n# Copyright (c) Facebook, Inc. and its affiliates.\nimport contextlib\nimport copy\nimport io\nimport itertools\nimport json\nimport logging\nimport numpy as np\nimport os\nimport pickle\nfrom collections import OrderedDict\nimport pycocotools.mask as mask_util\nimport torch\nfrom pycocotools.coco import COCO\nfrom pycocotools.cocoeval import COCOeval\nfrom tabulate import tabulate\n\nimport detectron2.utils.comm as comm\nfrom detectron2.config import CfgNode\nfrom detectron2.data import MetadataCatalog\nfrom detectron2.data.datasets.coco import convert_to_coco_json\nfrom detectron2.evaluation.evaluator import DatasetEvaluator\nfrom detectron2.evaluation.fast_eval_api import COCOeval_opt\nfrom detectron2.structures import Boxes, BoxMode, pairwise_iou\nfrom detectron2.utils.file_io import PathManager\nfrom detectron2.utils.logger import create_small_table\n\n\ndef vin_summarize(self):\n '''\n Compute and display summary metrics for evaluation results.\n Note this functin can *only* be applied on the default parameter setting\n '''\n\n def _summarize(ap=1, iouThr=None, areaRng='all', maxDets=100):\n p = self.params\n iStr = ' {:<18} {} @[ IoU={:<9} | area={:>6s} | maxDets={:>3d} ] = {:0.3f}'\n titleStr = 'Average Precision' if ap == 1 else 'Average Recall'\n typeStr = '(AP)' if ap == 1 else '(AR)'\n iouStr = '{:0.2f}:{:0.2f}'.format(p.iouThrs[0], p.iouThrs[-1]) \\\n if iouThr is None else '{:0.2f}'.format(iouThr)\n\n aind = [i for i, aRng in enumerate(p.areaRngLbl) if aRng == areaRng]\n mind = [i for i, mDet in enumerate(p.maxDets) if mDet == maxDets]\n if ap == 1:\n # dimension of precision: [TxRxKxAxM]\n s = self.eval['precision']\n # IoU\n if iouThr is not None:\n t = np.where(iouThr == p.iouThrs)[0]\n s = s[t]\n s = s[:, :, :, aind, mind]\n else:\n # dimension of recall: [TxKxAxM]\n s = self.eval['recall']\n if iouThr is not None:\n t = np.where(iouThr == p.iouThrs)[0]\n s = s[t]\n s = s[:, :, aind, mind]\n if len(s[s > -1]) == 0:\n mean_s = -1\n else:\n mean_s = np.mean(s[s > -1])\n print(iStr.format(titleStr, typeStr, iouStr, areaRng, maxDets, mean_s))\n return mean_s\n\n def _summarizeDets():\n stats = np.zeros((12,))\n stats[0] = _summarize(1)\n stats[1] = _summarize(1, iouThr=.5, maxDets=self.params.maxDets[2])\n # stats[2] = _summarize(1, iouThr=.75, maxDets=self.params.maxDets[2])\n stats[2] = _summarize(1, iouThr=.4, maxDets=self.params.maxDets[2])\n stats[3] = _summarize(1, areaRng='small', maxDets=self.params.maxDets[2])\n stats[4] = _summarize(1, areaRng='medium', maxDets=self.params.maxDets[2])\n stats[5] = _summarize(1, areaRng='large', maxDets=self.params.maxDets[2])\n stats[6] = _summarize(0, maxDets=self.params.maxDets[0])\n stats[7] = _summarize(0, maxDets=self.params.maxDets[1])\n stats[8] = _summarize(0, maxDets=self.params.maxDets[2])\n stats[9] = _summarize(0, areaRng='small', maxDets=self.params.maxDets[2])\n stats[10] = _summarize(0, areaRng='medium', maxDets=self.params.maxDets[2])\n stats[11] = _summarize(0, areaRng='large', maxDets=self.params.maxDets[2])\n return stats\n\n def _summarizeKps():\n stats = np.zeros((10,))\n stats[0] = _summarize(1, maxDets=20)\n stats[1] = _summarize(1, maxDets=20, iouThr=.5)\n stats[2] = _summarize(1, maxDets=20, iouThr=.75)\n stats[3] = _summarize(1, maxDets=20, areaRng='medium')\n stats[4] = _summarize(1, maxDets=20, areaRng='large')\n stats[5] = _summarize(0, maxDets=20)\n stats[6] = _summarize(0, maxDets=20, iouThr=.5)\n stats[7] = _summarize(0, maxDets=20, iouThr=.75)\n stats[8] = _summarize(0, maxDets=20, areaRng='medium')\n stats[9] = _summarize(0, maxDets=20, areaRng='large')\n return stats\n\n if not self.eval:\n raise Exception('Please run accumulate() first')\n iouType = self.params.iouType\n if iouType == 'segm' or iouType == 'bbox':\n summarize = _summarizeDets\n elif iouType == 'keypoints':\n summarize = _summarizeKps\n self.stats = summarize()\n\n\nprint(\"HACKING: overriding COCOeval.summarize = vin_summarize...\")\nCOCOeval.summarize = vin_summarize\n\n\nclass VinbigdataEvaluator(DatasetEvaluator):\n \"\"\"\n Evaluate AR for object proposals, AP for instance detection/segmentation, AP\n for keypoint detection outputs using COCO's metrics.\n See http://cocodataset.org/#detection-eval and\n http://cocodataset.org/#keypoints-eval to understand its metrics.\n\n In addition to COCO, this evaluator is able to support any bounding box detection,\n instance segmentation, or keypoint detection dataset.\n \"\"\"\n\n def __init__(\n self,\n dataset_name,\n tasks=None,\n distributed=True,\n output_dir=None,\n *,\n use_fast_impl=True,\n kpt_oks_sigmas=(),\n ):\n \"\"\"\n Args:\n dataset_name (str): name of the dataset to be evaluated.\n It must have either the following corresponding metadata:\n\n \"json_file\": the path to the COCO format annotation\n\n Or it must be in detectron2's standard dataset format\n so it can be converted to COCO format automatically.\n tasks (tuple[str]): tasks that can be evaluated under the given\n configuration. A task is one of \"bbox\", \"segm\", \"keypoints\".\n By default, will infer this automatically from predictions.\n distributed (True): if True, will collect results from all ranks and run evaluation\n in the main process.\n Otherwise, will only evaluate the results in the current process.\n output_dir (str): optional, an output directory to dump all\n results predicted on the dataset. The dump contains two files:\n\n 1. \"instances_predictions.pth\" a file in torch serialization\n format that contains all the raw original predictions.\n 2. \"coco_instances_results.json\" a json file in COCO's result\n format.\n use_fast_impl (bool): use a fast but **unofficial** implementation to compute AP.\n Although the results should be very close to the official implementation in COCO\n API, it is still recommended to compute results with the official API for use in\n papers. The faster implementation also uses more RAM.\n kpt_oks_sigmas (list[float]): The sigmas used to calculate keypoint OKS.\n See http://cocodataset.org/#keypoints-eval\n When empty, it will use the defaults in COCO.\n Otherwise it should be the same length as ROI_KEYPOINT_HEAD.NUM_KEYPOINTS.\n \"\"\"\n self._logger = logging.getLogger(__name__)\n self._distributed = distributed\n self._output_dir = output_dir\n self._use_fast_impl = use_fast_impl\n\n if tasks is not None and isinstance(tasks, CfgNode):\n kpt_oks_sigmas = (\n tasks.TEST.KEYPOINT_OKS_SIGMAS if not kpt_oks_sigmas else kpt_oks_sigmas\n )\n self._logger.warn(\n \"COCO Evaluator instantiated using config, this is deprecated behavior.\"\n \" Please pass in explicit arguments instead.\"\n )\n self._tasks = None # Infering it from predictions should be better\n else:\n self._tasks = tasks\n\n self._cpu_device = torch.device(\"cpu\")\n\n self._metadata = MetadataCatalog.get(dataset_name)\n if not hasattr(self._metadata, \"json_file\"):\n self._logger.info(\n f\"'{dataset_name}' is not registered by `register_coco_instances`.\"\n \" Therefore trying to convert it to COCO format ...\"\n )\n\n cache_path = os.path.join(output_dir, f\"{dataset_name}_coco_format.json\")\n self._metadata.json_file = cache_path\n convert_to_coco_json(dataset_name, cache_path)\n\n json_file = PathManager.get_local_path(self._metadata.json_file)\n with contextlib.redirect_stdout(io.StringIO()):\n self._coco_api = COCO(json_file)\n\n # Test set json files do not contain annotations (evaluation must be\n # performed using the COCO evaluation server).\n self._do_evaluation = \"annotations\" in self._coco_api.dataset\n if self._do_evaluation:\n self._kpt_oks_sigmas = kpt_oks_sigmas\n\n def reset(self):\n self._predictions = []\n\n def process(self, inputs, outputs):\n \"\"\"\n Args:\n inputs: the inputs to a COCO model (e.g., GeneralizedRCNN).\n It is a list of dict. Each dict corresponds to an image and\n contains keys like \"height\", \"width\", \"file_name\", \"image_id\".\n outputs: the outputs of a COCO model. It is a list of dicts with key\n \"instances\" that contains :class:`Instances`.\n \"\"\"\n for input, output in zip(inputs, outputs):\n prediction = {\"image_id\": input[\"image_id\"]}\n\n if \"instances\" in output:\n instances = output[\"instances\"].to(self._cpu_device)\n prediction[\"instances\"] = instances_to_coco_json(instances, input[\"image_id\"])\n if \"proposals\" in output:\n prediction[\"proposals\"] = output[\"proposals\"].to(self._cpu_device)\n if len(prediction) > 1:\n self._predictions.append(prediction)\n\n def evaluate(self, img_ids=None):\n \"\"\"\n Args:\n img_ids: a list of image IDs to evaluate on. Default to None for the whole dataset\n \"\"\"\n if self._distributed:\n comm.synchronize()\n predictions = comm.gather(self._predictions, dst=0)\n predictions = list(itertools.chain(*predictions))\n\n if not comm.is_main_process():\n return {}\n else:\n predictions = self._predictions\n\n if len(predictions) == 0:\n self._logger.warning(\"[VinbigdataEvaluator] Did not receive valid predictions.\")\n return {}\n\n if self._output_dir:\n PathManager.mkdirs(self._output_dir)\n file_path = os.path.join(self._output_dir, \"instances_predictions.pth\")\n with PathManager.open(file_path, \"wb\") as f:\n torch.save(predictions, f)\n\n self._results = OrderedDict()\n if \"proposals\" in predictions[0]:\n self._eval_box_proposals(predictions)\n if \"instances\" in predictions[0]:\n self._eval_predictions(predictions, img_ids=img_ids)\n # Copy so the caller can do whatever with results\n return copy.deepcopy(self._results)\n\n def _tasks_from_predictions(self, predictions):\n \"\"\"\n Get COCO API \"tasks\" (i.e. iou_type) from COCO-format predictions.\n \"\"\"\n tasks = {\"bbox\"}\n for pred in predictions:\n if \"segmentation\" in pred:\n tasks.add(\"segm\")\n if \"keypoints\" in pred:\n tasks.add(\"keypoints\")\n return sorted(tasks)\n\n def _eval_predictions(self, predictions, img_ids=None):\n \"\"\"\n Evaluate predictions. Fill self._results with the metrics of the tasks.\n \"\"\"\n self._logger.info(\"Preparing results for COCO format ...\")\n coco_results = list(itertools.chain(*[x[\"instances\"] for x in predictions]))\n tasks = self._tasks or self._tasks_from_predictions(coco_results)\n\n # unmap the category ids for COCO\n if hasattr(self._metadata, \"thing_dataset_id_to_contiguous_id\"):\n dataset_id_to_contiguous_id = self._metadata.thing_dataset_id_to_contiguous_id\n all_contiguous_ids = list(dataset_id_to_contiguous_id.values())\n num_classes = len(all_contiguous_ids)\n assert min(all_contiguous_ids) == 0 and max(all_contiguous_ids) == num_classes - 1\n\n reverse_id_mapping = {v: k for k, v in dataset_id_to_contiguous_id.items()}\n for result in coco_results:\n category_id = result[\"category_id\"]\n assert category_id < num_classes, (\n f\"A prediction has class={category_id}, \"\n f\"but the dataset only has {num_classes} classes and \"\n f\"predicted class id should be in [0, {num_classes - 1}].\"\n )\n result[\"category_id\"] = reverse_id_mapping[category_id]\n\n if self._output_dir:\n file_path = os.path.join(self._output_dir, \"coco_instances_results.json\")\n self._logger.info(\"Saving results to {}\".format(file_path))\n with PathManager.open(file_path, \"w\") as f:\n f.write(json.dumps(coco_results))\n f.flush()\n\n if not self._do_evaluation:\n self._logger.info(\"Annotations are not available for evaluation.\")\n return\n\n self._logger.info(\n \"Evaluating predictions with {} COCO API...\".format(\n \"unofficial\" if self._use_fast_impl else \"official\"\n )\n )\n for task in sorted(tasks):\n coco_eval = (\n _evaluate_predictions_on_coco(\n self._coco_api,\n coco_results,\n task,\n kpt_oks_sigmas=self._kpt_oks_sigmas,\n use_fast_impl=self._use_fast_impl,\n img_ids=img_ids,\n )\n if len(coco_results) > 0\n else None # cocoapi does not handle empty results very well\n )\n\n res = self._derive_coco_results(\n coco_eval, task, class_names=self._metadata.get(\"thing_classes\")\n )\n self._results[task] = res\n\n def _eval_box_proposals(self, predictions):\n \"\"\"\n Evaluate the box proposals in predictions.\n Fill self._results with the metrics for \"box_proposals\" task.\n \"\"\"\n if self._output_dir:\n # Saving generated box proposals to file.\n # Predicted box_proposals are in XYXY_ABS mode.\n bbox_mode = BoxMode.XYXY_ABS.value\n ids, boxes, objectness_logits = [], [], []\n for prediction in predictions:\n ids.append(prediction[\"image_id\"])\n boxes.append(prediction[\"proposals\"].proposal_boxes.tensor.numpy())\n objectness_logits.append(prediction[\"proposals\"].objectness_logits.numpy())\n\n proposal_data = {\n \"boxes\": boxes,\n \"objectness_logits\": objectness_logits,\n \"ids\": ids,\n \"bbox_mode\": bbox_mode,\n }\n with PathManager.open(os.path.join(self._output_dir, \"box_proposals.pkl\"), \"wb\") as f:\n pickle.dump(proposal_data, f)\n\n if not self._do_evaluation:\n self._logger.info(\"Annotations are not available for evaluation.\")\n return\n\n self._logger.info(\"Evaluating bbox proposals ...\")\n res = {}\n areas = {\"all\": \"\", \"small\": \"s\", \"medium\": \"m\", \"large\": \"l\"}\n for limit in [100, 1000]:\n for area, suffix in areas.items():\n stats = _evaluate_box_proposals(predictions, self._coco_api, area=area, limit=limit)\n key = \"AR{}@{:d}\".format(suffix, limit)\n res[key] = float(stats[\"ar\"].item() * 100)\n self._logger.info(\"Proposal metrics: \\n\" + create_small_table(res))\n self._results[\"box_proposals\"] = res\n\n def _derive_coco_results(self, coco_eval, iou_type, class_names=None):\n \"\"\"\n Derive the desired score numbers from summarized COCOeval.\n\n Args:\n coco_eval (None or COCOEval): None represents no predictions from model.\n iou_type (str):\n class_names (None or list[str]): if provided, will use it to predict\n per-category AP.\n\n Returns:\n a dict of {metric name: score}\n \"\"\"\n\n metrics = {\n \"bbox\": [\"AP\", \"AP50\", \"AP75\", \"APs\", \"APm\", \"APl\"],\n \"segm\": [\"AP\", \"AP50\", \"AP75\", \"APs\", \"APm\", \"APl\"],\n \"keypoints\": [\"AP\", \"AP50\", \"AP75\", \"APm\", \"APl\"],\n }[iou_type]\n\n if coco_eval is None:\n self._logger.warn(\"No predictions from the model!\")\n return {metric: float(\"nan\") for metric in metrics}\n\n # the standard metrics\n results = {\n metric: float(coco_eval.stats[idx] * 100 if coco_eval.stats[idx] >= 0 else \"nan\")\n for idx, metric in enumerate(metrics)\n }\n self._logger.info(\n \"Evaluation results for {}: \\n\".format(iou_type) + create_small_table(results)\n )\n if not np.isfinite(sum(results.values())):\n self._logger.info(\"Some metrics cannot be computed and is shown as NaN.\")\n\n if class_names is None or len(class_names) <= 1:\n return results\n # Compute per-category AP\n # from https://github.com/facebookresearch/Detectron/blob/a6a835f5b8208c45d0dce217ce9bbda915f44df7/detectron/datasets/json_dataset_evaluator.py#L222-L252 # noqa\n precisions = coco_eval.eval[\"precision\"]\n # precision has dims (iou, recall, cls, area range, max dets)\n assert len(class_names) == precisions.shape[2]\n\n results_per_category = []\n for idx, name in enumerate(class_names):\n # area range index 0: all area ranges\n # max dets index -1: typically 100 per image\n precision = precisions[:, :, idx, 0, -1]\n precision = precision[precision > -1]\n ap = np.mean(precision) if precision.size else float(\"nan\")\n results_per_category.append((\"{}\".format(name), float(ap * 100)))\n\n # tabulate it\n N_COLS = min(6, len(results_per_category) * 2)\n results_flatten = list(itertools.chain(*results_per_category))\n results_2d = itertools.zip_longest(*[results_flatten[i::N_COLS] for i in range(N_COLS)])\n table = tabulate(\n results_2d,\n tablefmt=\"pipe\",\n floatfmt=\".3f\",\n headers=[\"category\", \"AP\"] * (N_COLS // 2),\n numalign=\"left\",\n )\n self._logger.info(\"Per-category {} AP: \\n\".format(iou_type) + table)\n\n results.update({\"AP-\" + name: ap for name, ap in results_per_category})\n return results\n\n\ndef instances_to_coco_json(instances, img_id):\n \"\"\"\n Dump an \"Instances\" object to a COCO-format json that's used for evaluation.\n\n Args:\n instances (Instances):\n img_id (int): the image id\n\n Returns:\n list[dict]: list of json annotations in COCO format.\n \"\"\"\n num_instance = len(instances)\n if num_instance == 0:\n return []\n\n boxes = instances.pred_boxes.tensor.numpy()\n boxes = BoxMode.convert(boxes, BoxMode.XYXY_ABS, BoxMode.XYWH_ABS)\n boxes = boxes.tolist()\n scores = instances.scores.tolist()\n classes = instances.pred_classes.tolist()\n\n has_mask = instances.has(\"pred_masks\")\n if has_mask:\n # use RLE to encode the masks, because they are too large and takes memory\n # since this evaluator stores outputs of the entire dataset\n rles = [\n mask_util.encode(np.array(mask[:, :, None], order=\"F\", dtype=\"uint8\"))[0]\n for mask in instances.pred_masks\n ]\n for rle in rles:\n # \"counts\" is an array encoded by mask_util as a byte-stream. Python3's\n # json writer which always produces strings cannot serialize a bytestream\n # unless you decode it. Thankfully, utf-8 works out (which is also what\n # the pycocotools/_mask.pyx does).\n rle[\"counts\"] = rle[\"counts\"].decode(\"utf-8\")\n\n has_keypoints = instances.has(\"pred_keypoints\")\n if has_keypoints:\n keypoints = instances.pred_keypoints\n\n results = []\n for k in range(num_instance):\n result = {\n \"image_id\": img_id,\n \"category_id\": classes[k],\n \"bbox\": boxes[k],\n \"score\": scores[k],\n }\n if has_mask:\n result[\"segmentation\"] = rles[k]\n if has_keypoints:\n # In COCO annotations,\n # keypoints coordinates are pixel indices.\n # However our predictions are floating point coordinates.\n # Therefore we subtract 0.5 to be consistent with the annotation format.\n # This is the inverse of data loading logic in `datasets/coco.py`.\n keypoints[k][:, :2] -= 0.5\n result[\"keypoints\"] = keypoints[k].flatten().tolist()\n results.append(result)\n return results\n\n\n# inspired from Detectron:\n# https://github.com/facebookresearch/Detectron/blob/a6a835f5b8208c45d0dce217ce9bbda915f44df7/detectron/datasets/json_dataset_evaluator.py#L255 # noqa\ndef _evaluate_box_proposals(dataset_predictions, coco_api, thresholds=None, area=\"all\", limit=None):\n \"\"\"\n Evaluate detection proposal recall metrics. This function is a much\n faster alternative to the official COCO API recall evaluation code. However,\n it produces slightly different results.\n \"\"\"\n # Record max overlap value for each gt box\n # Return vector of overlap values\n areas = {\n \"all\": 0,\n \"small\": 1,\n \"medium\": 2,\n \"large\": 3,\n \"96-128\": 4,\n \"128-256\": 5,\n \"256-512\": 6,\n \"512-inf\": 7,\n }\n area_ranges = [\n [0 ** 2, 1e5 ** 2], # all\n [0 ** 2, 32 ** 2], # small\n [32 ** 2, 96 ** 2], # medium\n [96 ** 2, 1e5 ** 2], # large\n [96 ** 2, 128 ** 2], # 96-128\n [128 ** 2, 256 ** 2], # 128-256\n [256 ** 2, 512 ** 2], # 256-512\n [512 ** 2, 1e5 ** 2],\n ] # 512-inf\n assert area in areas, \"Unknown area range: {}\".format(area)\n area_range = area_ranges[areas[area]]\n gt_overlaps = []\n num_pos = 0\n\n for prediction_dict in dataset_predictions:\n predictions = prediction_dict[\"proposals\"]\n\n # sort predictions in descending order\n # TODO maybe remove this and make it explicit in the documentation\n inds = predictions.objectness_logits.sort(descending=True)[1]\n predictions = predictions[inds]\n\n ann_ids = coco_api.getAnnIds(imgIds=prediction_dict[\"image_id\"])\n anno = coco_api.loadAnns(ann_ids)\n gt_boxes = [\n BoxMode.convert(obj[\"bbox\"], BoxMode.XYWH_ABS, BoxMode.XYXY_ABS)\n for obj in anno\n if obj[\"iscrowd\"] == 0\n ]\n gt_boxes = torch.as_tensor(gt_boxes).reshape(-1, 4) # guard against no boxes\n gt_boxes = Boxes(gt_boxes)\n gt_areas = torch.as_tensor([obj[\"area\"] for obj in anno if obj[\"iscrowd\"] == 0])\n\n if len(gt_boxes) == 0 or len(predictions) == 0:\n continue\n\n valid_gt_inds = (gt_areas >= area_range[0]) & (gt_areas <= area_range[1])\n gt_boxes = gt_boxes[valid_gt_inds]\n\n num_pos += len(gt_boxes)\n\n if len(gt_boxes) == 0:\n continue\n\n if limit is not None and len(predictions) > limit:\n predictions = predictions[:limit]\n\n overlaps = pairwise_iou(predictions.proposal_boxes, gt_boxes)\n\n _gt_overlaps = torch.zeros(len(gt_boxes))\n for j in range(min(len(predictions), len(gt_boxes))):\n # find which proposal box maximally covers each gt box\n # and get the iou amount of coverage for each gt box\n max_overlaps, argmax_overlaps = overlaps.max(dim=0)\n\n # find which gt box is 'best' covered (i.e. 'best' = most iou)\n gt_ovr, gt_ind = max_overlaps.max(dim=0)\n assert gt_ovr >= 0\n # find the proposal box that covers the best covered gt box\n box_ind = argmax_overlaps[gt_ind]\n # record the iou coverage of this gt box\n _gt_overlaps[j] = overlaps[box_ind, gt_ind]\n assert _gt_overlaps[j] == gt_ovr\n # mark the proposal box and the gt box as used\n overlaps[box_ind, :] = -1\n overlaps[:, gt_ind] = -1\n\n # append recorded iou coverage level\n gt_overlaps.append(_gt_overlaps)\n gt_overlaps = (\n torch.cat(gt_overlaps, dim=0) if len(gt_overlaps) else torch.zeros(0, dtype=torch.float32)\n )\n gt_overlaps, _ = torch.sort(gt_overlaps)\n\n if thresholds is None:\n step = 0.05\n # thresholds = torch.arange(0.5, 0.95 + 1e-5, step, dtype=torch.float32)\n thresholds = torch.arange(0.4, 0.95 + 1e-5, step, dtype=torch.float32)\n recalls = torch.zeros_like(thresholds)\n # compute recall for each iou threshold\n for i, t in enumerate(thresholds):\n recalls[i] = (gt_overlaps >= t).float().sum() / float(num_pos)\n # ar = 2 * np.trapz(recalls, thresholds)\n ar = recalls.mean()\n return {\n \"ar\": ar,\n \"recalls\": recalls,\n \"thresholds\": thresholds,\n \"gt_overlaps\": gt_overlaps,\n \"num_pos\": num_pos,\n }\n\n\ndef _evaluate_predictions_on_coco(\n coco_gt, coco_results, iou_type, kpt_oks_sigmas=None, use_fast_impl=True, img_ids=None\n):\n \"\"\"\n Evaluate the coco results using COCOEval API.\n \"\"\"\n assert len(coco_results) > 0\n\n if iou_type == \"segm\":\n coco_results = copy.deepcopy(coco_results)\n # When evaluating mask AP, if the results contain bbox, cocoapi will\n # use the box area as the area of the instance, instead of the mask area.\n # This leads to a different definition of small/medium/large.\n # We remove the bbox field to let mask AP use mask area.\n for c in coco_results:\n c.pop(\"bbox\", None)\n\n coco_dt = coco_gt.loadRes(coco_results)\n coco_eval = (COCOeval_opt if use_fast_impl else COCOeval)(coco_gt, coco_dt, iou_type)\n\n # HACKING: overwrite iouThrs to calc ious 0.4\n coco_eval.params.iouThrs = np.linspace(\n .4, 0.95, int(np.round((0.95 - .4) / .05)) + 1, endpoint=True)\n\n if img_ids is not None:\n coco_eval.params.imgIds = img_ids\n\n if iou_type == \"keypoints\":\n # Use the COCO default keypoint OKS sigmas unless overrides are specified\n if kpt_oks_sigmas:\n assert hasattr(coco_eval.params, \"kpt_oks_sigmas\"), \"pycocotools is too old!\"\n coco_eval.params.kpt_oks_sigmas = np.array(kpt_oks_sigmas)\n # COCOAPI requires every detection and every gt to have keypoints, so\n # we just take the first entry from both\n num_keypoints_dt = len(coco_results[0][\"keypoints\"]) // 3\n num_keypoints_gt = len(next(iter(coco_gt.anns.values()))[\"keypoints\"]) // 3\n num_keypoints_oks = len(coco_eval.params.kpt_oks_sigmas)\n assert num_keypoints_oks == num_keypoints_dt == num_keypoints_gt, (\n f\"[VinbigdataEvaluator] Prediction contain {num_keypoints_dt} keypoints. \"\n f\"Ground truth contains {num_keypoints_gt} keypoints. \"\n f\"The length of cfg.TEST.KEYPOINT_OKS_SIGMAS is {num_keypoints_oks}. \"\n \"They have to agree with each other. For meaning of OKS, please refer to \"\n \"http://cocodataset.org/#keypoints-eval.\"\n )\n\n coco_eval.evaluate()\n coco_eval.accumulate()\n coco_eval.summarize()\n\n return coco_eval\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.717473Z","iopub.execute_input":"2025-04-14T08:38:32.71783Z","iopub.status.idle":"2025-04-14T08:38:32.821337Z","shell.execute_reply.started":"2025-04-14T08:38:32.717794Z","shell.execute_reply":"2025-04-14T08:38:32.82042Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Loss evaluation hook\n\nWe implemented Evaluator and now we can calculate competition metric, however, validation loss is not calculated inside Evaluator. This is because model's evaluation is done in `model.eval()` mode and it outputs bounding box prediction but does not output `loss`.\n\nTo calculate validation loss, we need to call `model` with the training mode. This can be done by adding `Hook` which calculates the loss to the trainer.
\nTrainer has attribute `storage` and calculated metrics are summarized. Its content is saved to `metric.json` (jsonl format) during training.\n\nBelow `LossEvalHook` calculates validation loss in `_do_loss_eval` method, and `self.trainer.storage.put_scalars(validation_loss=mean_loss)` is called to put this validation loss to the `storage`, which will be saved to `metrics.json`.
\nNote that current implementation is not efficient in the sense that Evaluator's evaluation and `LossEvalHook`'s loss calculation run separately, even if both need a model forward calculation for same validation data.\n\n - Ref: [Training on Detectron2 with a Validation set, and plot loss on it to avoid overfitting](https://medium.com/@apofeniaco/training-on-detectron2-with-a-validation-set-and-plot-loss-on-it-to-avoid-overfitting-6449418fbf4e)","metadata":{}},{"cell_type":"code","source":"\"\"\"\nTo calculate & record validation loss\n\nOriginal code from https://medium.com/@apofeniaco/training-on-detectron2-with-a-validation-set-and-plot-loss-on-it-to-avoid-overfitting-6449418fbf4e\nby @apofeniaco\n\"\"\"\nimport numpy as np\nimport logging\n\nfrom detectron2.engine.hooks import HookBase\nfrom detectron2.utils.logger import log_every_n_seconds\nimport detectron2.utils.comm as comm\nimport torch\nimport time\nimport datetime\n\n\nclass LossEvalHook(HookBase):\n def __init__(self, eval_period, model, data_loader):\n self._model = model\n self._period = eval_period\n self._data_loader = data_loader\n\n def _do_loss_eval(self):\n # Copying inference_on_dataset from evaluator.py\n total = len(self._data_loader)\n num_warmup = min(5, total - 1)\n\n start_time = time.perf_counter()\n total_compute_time = 0\n losses = []\n for idx, inputs in enumerate(self._data_loader):\n if idx == num_warmup:\n start_time = time.perf_counter()\n total_compute_time = 0\n start_compute_time = time.perf_counter()\n if torch.cuda.is_available():\n torch.cuda.synchronize()\n total_compute_time += time.perf_counter() - start_compute_time\n iters_after_start = idx + 1 - num_warmup * int(idx >= num_warmup)\n seconds_per_img = total_compute_time / iters_after_start\n if idx >= num_warmup * 2 or seconds_per_img > 5:\n total_seconds_per_img = (time.perf_counter() - start_time) / iters_after_start\n eta = datetime.timedelta(seconds=int(total_seconds_per_img * (total - idx - 1)))\n log_every_n_seconds(\n logging.INFO,\n \"Loss on Validation done {}/{}. {:.4f} s / img. ETA={}\".format(\n idx + 1, total, seconds_per_img, str(eta)\n ),\n n=5,\n )\n loss_batch = self._get_loss(inputs)\n losses.append(loss_batch)\n mean_loss = np.mean(losses)\n # self.trainer.storage.put_scalar('validation_loss', mean_loss)\n comm.synchronize()\n\n # return losses\n return mean_loss\n\n def _get_loss(self, data):\n # How loss is calculated on train_loop\n metrics_dict = self._model(data)\n metrics_dict = {\n k: v.detach().cpu().item() if isinstance(v, torch.Tensor) else float(v)\n for k, v in metrics_dict.items()\n }\n total_losses_reduced = sum(loss for loss in metrics_dict.values())\n return total_losses_reduced\n\n def after_step(self):\n next_iter = int(self.trainer.iter) + 1\n is_final = next_iter == self.trainer.max_iter\n if is_final or (self._period > 0 and next_iter % self._period == 0):\n mean_loss = self._do_loss_eval()\n self.trainer.storage.put_scalars(validation_loss=mean_loss)\n print(\"validation do loss eval\", mean_loss)\n else:\n pass\n # self.trainer.storage.put_scalars(timetest=11)\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.822889Z","iopub.execute_input":"2025-04-14T08:38:32.82314Z","iopub.status.idle":"2025-04-14T08:38:32.893124Z","shell.execute_reply.started":"2025-04-14T08:38:32.823114Z","shell.execute_reply":"2025-04-14T08:38:32.892532Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now all the preparation has done!\n\n`MyTrainer` overwraps `build_evaluator` method of `DefaultTrainer` provided by `detectron2` to support validation dataset evaluation.\n\n\n1. `build_train_loader` & `build_test_loader`: \nThese class methods deine how to construct DataLoader for training data & validation data respectively.\nHere `AlbumentationMapper` is passed to construct DataLoader to insert customized augmentation process.\n\n2. `build_evaluator`:\nThis class method defines how to construct Evaluator. \nHere implemented `VinbigdataEvaluator` is constructed (we can also use `COCOEvaluator` here).\n\n3. `build_hooks`:\nThis method defines how to construct hooks. I insert `LossEvalHook` before evalutor to work well.","metadata":{}},{"cell_type":"code","source":"import os\n\nfrom detectron2.data import build_detection_test_loader, build_detection_train_loader\nfrom detectron2.engine import DefaultPredictor, DefaultTrainer, launch\n\n# from detectron2.evaluation import COCOEvaluator, PascalVOCDetectionEvaluator\n\n\nclass MyTrainer(DefaultTrainer):\n @classmethod\n def build_train_loader(cls, cfg, sampler=None):\n return build_detection_train_loader(\n cfg, mapper=AlbumentationsMapper(cfg, True), sampler=sampler\n )\n\n @classmethod\n def build_test_loader(cls, cfg, dataset_name):\n return build_detection_test_loader(\n cfg, dataset_name, mapper=AlbumentationsMapper(cfg, False)\n )\n\n @classmethod\n def build_evaluator(cls, cfg, dataset_name, output_folder=None):\n if output_folder is None:\n output_folder = os.path.join(cfg.OUTPUT_DIR, \"inference\")\n # return PascalVOCDetectionEvaluator(dataset_name) # not working\n # return COCOEvaluator(dataset_name, (\"bbox\",), False, output_dir=output_folder)\n return VinbigdataEvaluator(dataset_name, (\"bbox\",), False, output_dir=output_folder)\n\n def build_hooks(self):\n hooks = super(MyTrainer, self).build_hooks()\n cfg = self.cfg\n if len(cfg.DATASETS.TEST) > 0:\n loss_eval_hook = LossEvalHook(\n cfg.TEST.EVAL_PERIOD,\n self.model,\n MyTrainer.build_test_loader(cfg, cfg.DATASETS.TEST[0]),\n )\n hooks.insert(-1, loss_eval_hook)\n\n return hooks\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.89411Z","iopub.execute_input":"2025-04-14T08:38:32.894364Z","iopub.status.idle":"2025-04-14T08:38:32.902955Z","shell.execute_reply.started":"2025-04-14T08:38:32.89434Z","shell.execute_reply":"2025-04-14T08:38:32.902209Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## LR scheduling\n\nTo further customize learning rate scheduling you may override `build_lr_scheduler` class method to construct any pytorch LRScheduler.\n\nDefault `build_lr_schduler` method ([docs](https://detectron2.readthedocs.io/en/latest/_modules/detectron2/solver/build.html#build_lr_scheduler)) supports only 2 types of LR scheduling, WarmupMultiStepLR (default) & WarmupCosineLR. You can change which one to use by setting `cfg.SOLVER.LR_SCHEDULER_NAME` as you can see from the docs.","metadata":{}},{"cell_type":"markdown","source":"Now the methods are ready. main scripts starts from here.","metadata":{}},{"cell_type":"markdown","source":"\n# Loading Data\n\nThis `Flags` class is to manage experiments. I will tune these parameters through the competition to improve model's performance.","metadata":{}},{"cell_type":"code","source":"import argparse\nimport dataclasses\nimport json\nimport os\nimport pickle\nimport random\nimport sys\nfrom dataclasses import dataclass\nfrom distutils.util import strtobool\nfrom pathlib import Path\n\nimport cv2\nimport detectron2\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom detectron2 import model_zoo\nfrom detectron2.config import get_cfg\nfrom detectron2.data import DatasetCatalog, MetadataCatalog\nfrom detectron2.engine import DefaultPredictor, DefaultTrainer, launch\nfrom detectron2.evaluation import COCOEvaluator\nfrom detectron2.structures import BoxMode\nfrom detectron2.utils.logger import setup_logger\nfrom detectron2.utils.visualizer import Visualizer\nfrom tqdm import tqdm\n\nsetup_logger()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.904425Z","iopub.execute_input":"2025-04-14T08:38:32.904702Z","iopub.status.idle":"2025-04-14T08:38:32.922023Z","shell.execute_reply.started":"2025-04-14T08:38:32.904675Z","shell.execute_reply":"2025-04-14T08:38:32.921146Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- flags ---\nfrom dataclasses import dataclass, field\nfrom typing import Dict\n\n\n@dataclass\nclass Flags:\n # General\n debug: bool = True\n outdir: str = \"results/det\"\n\n # Data config\n # imgdir_name: str = \"vinbigdata-chest-xray-resized-png-512x512\"\n imgdir_name: str = \"vinbigdata\"\n split_mode: str = \"all_train\" # all_train or valid20\n seed: int = 111\n train_data_type: str = \"original\" # original or wbf\n use_class14: bool = False\n # Training config\n iter: int = 750 # 0.1e: 750 # 1e: 7500\n ims_per_batch: int = 2 # images per batch, this corresponds to \"total batch size\"\n num_workers: int = 4\n lr_scheduler_name: str = \"WarmupMultiStepLR\" # WarmupMultiStepLR (default) or WarmupCosineLR\n base_lr: float = 0.00025\n roi_batch_size_per_image: int = 512\n eval_period: int = 10000\n aug_kwargs: Dict = field(default_factory=lambda: {})\n\n def update(self, param_dict: Dict) -> \"Flags\":\n # Overwrite by `param_dict`\n for key, value in param_dict.items():\n if not hasattr(self, key):\n raise ValueError(f\"[ERROR] Unexpected key for flag = {key}\")\n setattr(self, key, value)\n return self","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.923425Z","iopub.execute_input":"2025-04-14T08:38:32.923685Z","iopub.status.idle":"2025-04-14T08:38:32.937702Z","shell.execute_reply.started":"2025-04-14T08:38:32.92366Z","shell.execute_reply":"2025-04-14T08:38:32.936839Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# flags_dict = {\n# \"debug\": True,\n# \"outdir\": \"results/debug\", \n# \"imgdir_name\": \"vinbigdata-chest-xray-resized-png-256x256\",\n# \"split_mode\": \"valid20\",\n# \"iter\": 100, # debug, small value should be set.\n# \"roi_batch_size_per_image\": 128, # faster, and good enough for this toy dataset (default: 512)\n# \"eval_period\": 20,\n# \"aug_kwargs\": {\n# \"HorizontalFlip\": {\"p\": 0.5},\n# \"ShiftScaleRotate\": {\"scale_limit\": 0.15, \"rotate_limit\": 10, \"p\": 0.5},\n# \"RandomBrightnessContrast\": {\"p\": 0.5}\n# }\n# }","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.938915Z","iopub.execute_input":"2025-04-14T08:38:32.939135Z","iopub.status.idle":"2025-04-14T08:38:32.947714Z","shell.execute_reply.started":"2025-04-14T08:38:32.939114Z","shell.execute_reply":"2025-04-14T08:38:32.94688Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"flags_dict = {\n \"debug\": False,\n \"outdir\": \"results/v9\", \n # \"imgdir_name\": \"vinbigdata-chest-xray-resized-png-512x512\",\n \"imgdir_name\": \"vinbigdata\",\n \"split_mode\": \"valid20\",\n \"iter\": 750, #10000,\n \"roi_batch_size_per_image\": 512,\n \"eval_period\": 1000,\n \"lr_scheduler_name\": \"WarmupCosineLR\",\n \"base_lr\": 0.001,\n \"num_workers\": 4,\n \"aug_kwargs\": {\n \"HorizontalFlip\": {\"p\": 0.5},\n \"ShiftScaleRotate\": {\"scale_limit\": 0.15, \"rotate_limit\": 10, \"p\": 0.5},\n \"RandomBrightnessContrast\": {\"p\": 0.5}\n }\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.951757Z","iopub.execute_input":"2025-04-14T08:38:32.951995Z","iopub.status.idle":"2025-04-14T08:38:32.96499Z","shell.execute_reply.started":"2025-04-14T08:38:32.951972Z","shell.execute_reply":"2025-04-14T08:38:32.964073Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# args = parse()\nprint(\"torch\", torch.__version__)\nflags = Flags().update(flags_dict)\nprint(\"flags\", flags)\ndebug = flags.debug\noutdir = Path(flags.outdir)\nos.makedirs(str(outdir), exist_ok=True)\nflags_dict = dataclasses.asdict(flags)\nsave_yaml(outdir / \"flags.yaml\", flags_dict)\n\n# --- Read data ---\ninputdir = Path(\"/kaggle/input\")\ndatadir = inputdir / \"vinbigdata-chest-xray-abnormalities-detection\"\nimgdir = inputdir / flags.imgdir_name\n\n# Read in the data CSV files\ntrain_df = pd.read_csv(datadir / \"train.csv\")\ntrain = train_df # alias\n# sample_submission = pd.read_csv(datadir / 'sample_submission.csv')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:32.966138Z","iopub.execute_input":"2025-04-14T08:38:32.966394Z","iopub.status.idle":"2025-04-14T08:38:33.134639Z","shell.execute_reply.started":"2025-04-14T08:38:32.966353Z","shell.execute_reply":"2025-04-14T08:38:33.133964Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_data_type = flags.train_data_type\nif flags.use_class14:\n thing_classes.append(\"No finding\")\n\nsplit_mode = flags.split_mode\nif split_mode == \"all_train\":\n DatasetCatalog.register(\n \"vinbigdata_train\",\n lambda: get_vinbigdata_dicts(\n imgdir, train_df, train_data_type, debug=debug, use_class14=flags.use_class14\n ),\n )\n MetadataCatalog.get(\"vinbigdata_train\").set(thing_classes=thing_classes)\nelif split_mode == \"valid20\":\n # To get number of data...\n n_dataset = len(\n get_vinbigdata_dicts(\n imgdir, train_df, train_data_type, debug=debug, use_class14=flags.use_class14\n )\n )\n n_train = int(n_dataset * 0.8)\n print(\"n_dataset\", n_dataset, \"n_train\", n_train)\n rs = np.random.RandomState(flags.seed)\n inds = rs.permutation(n_dataset)\n train_inds, valid_inds = inds[:n_train], inds[n_train:]\n DatasetCatalog.register(\n \"vinbigdata_train\",\n lambda: get_vinbigdata_dicts(\n imgdir,\n train_df,\n train_data_type,\n debug=debug,\n target_indices=train_inds,\n use_class14=flags.use_class14,\n ),\n )\n MetadataCatalog.get(\"vinbigdata_train\").set(thing_classes=thing_classes)\n DatasetCatalog.register(\n \"vinbigdata_valid\",\n lambda: get_vinbigdata_dicts(\n imgdir,\n train_df,\n train_data_type,\n debug=debug,\n target_indices=valid_inds,\n use_class14=flags.use_class14,\n ),\n )\n MetadataCatalog.get(\"vinbigdata_valid\").set(thing_classes=thing_classes)\nelse:\n raise ValueError(f\"[ERROR] Unexpected value split_mode={split_mode}\")\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:38:33.136122Z","iopub.execute_input":"2025-04-14T08:38:33.136494Z","iopub.status.idle":"2025-04-14T08:39:35.81305Z","shell.execute_reply.started":"2025-04-14T08:38:33.136457Z","shell.execute_reply":"2025-04-14T08:39:35.812132Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataset_dicts = get_vinbigdata_dicts(imgdir, train, debug=debug)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:39:35.81445Z","iopub.execute_input":"2025-04-14T08:39:35.814792Z","iopub.status.idle":"2025-04-14T08:39:35.864704Z","shell.execute_reply.started":"2025-04-14T08:39:35.814762Z","shell.execute_reply":"2025-04-14T08:39:35.863896Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n# Data Visualization\n\nIt's also very easy to visualize prepared training dataset with `detectron2`.
\nIt provides `Visualizer` class, we can use it to draw an image with bounding box as following.","metadata":{}},{"cell_type":"code","source":"# Visualize data...\nanomaly_image_ids = train.query(\"class_id != 14\")[\"image_id\"].unique()\ntrain_meta = pd.read_csv(imgdir/\"train_meta.csv\")\nanomaly_inds = np.argwhere(train_meta[\"image_id\"].isin(anomaly_image_ids).values)[:, 0]\n\nvinbigdata_metadata = MetadataCatalog.get(\"vinbigdata_train\")\n\ncols = 3\nrows = 3\nfig, axes = plt.subplots(rows, cols, figsize=(18, 18))\naxes = axes.flatten()\n\nfor index, anom_ind in enumerate(anomaly_inds[:cols * rows]):\n ax = axes[index]\n # print(anom_ind)\n d = dataset_dicts[anom_ind]\n img = cv2.imread(d[\"file_name\"])\n visualizer = Visualizer(img[:, :, ::-1], metadata=vinbigdata_metadata, scale=0.5)\n out = visualizer.draw_dataset_dict(d)\n # cv2_imshow(out.get_image()[:, :, ::-1])\n #cv2.imwrite(str(outdir / f\"vinbigdata{index}.jpg\"), out.get_image()[:, :, ::-1])\n ax.imshow(out.get_image()[:, :, ::-1])\n ax.set_title(f\"{anom_ind}: image_id {anomaly_image_ids[index]}\")","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:39:35.865679Z","iopub.execute_input":"2025-04-14T08:39:35.865933Z","iopub.status.idle":"2025-04-14T08:39:38.022175Z","shell.execute_reply.started":"2025-04-14T08:39:35.865894Z","shell.execute_reply":"2025-04-14T08:39:38.021199Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n# Training","metadata":{}},{"cell_type":"code","source":"from detectron2.config.config import CfgNode as CN\n\ncfg = get_cfg()\ncfg.aug_kwargs = CN(flags.aug_kwargs) # pass aug_kwargs to cfg\n\noriginal_output_dir = cfg.OUTPUT_DIR\ncfg.OUTPUT_DIR = str(outdir)\nprint(f\"cfg.OUTPUT_DIR {original_output_dir} -> {cfg.OUTPUT_DIR}\")\n\nconfig_name = \"COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml\"\ncfg.merge_from_file(model_zoo.get_config_file(config_name))\ncfg.DATASETS.TRAIN = (\"vinbigdata_train\",)\nif split_mode == \"all_train\":\n cfg.DATASETS.TEST = ()\nelse:\n cfg.DATASETS.TEST = (\"vinbigdata_valid\",)\n cfg.TEST.EVAL_PERIOD = flags.eval_period\n\ncfg.DATALOADER.NUM_WORKERS = flags.num_workers\n# Let training initialize from model zoo\ncfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(config_name)\ncfg.SOLVER.IMS_PER_BATCH = flags.ims_per_batch\ncfg.SOLVER.LR_SCHEDULER_NAME = flags.lr_scheduler_name\ncfg.SOLVER.BASE_LR = flags.base_lr # pick a good LR\ncfg.SOLVER.MAX_ITER = flags.iter\ncfg.SOLVER.CHECKPOINT_PERIOD = 100000 # Small value=Frequent save need a lot of storage.\ncfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = flags.roi_batch_size_per_image\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = len(thing_classes)\n# NOTE: this config means the number of classes,\n# but a few popular unofficial tutorials incorrect uses num_classes+1 here.\n\nos.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\n","metadata":{"trusted":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-04-14T08:39:38.02328Z","iopub.execute_input":"2025-04-14T08:39:38.023502Z","iopub.status.idle":"2025-04-14T08:39:38.046403Z","shell.execute_reply.started":"2025-04-14T08:39:38.023479Z","shell.execute_reply":"2025-04-14T08:39:38.045353Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainer = MyTrainer(cfg)\ntrainer.resume_or_load(resume=False)\ntrainer.train()","metadata":{"trusted":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-04-14T08:39:38.047871Z","iopub.execute_input":"2025-04-14T08:39:38.048172Z","iopub.status.idle":"2025-04-14T08:46:27.274369Z","shell.execute_reply.started":"2025-04-14T08:39:38.048125Z","shell.execute_reply":"2025-04-14T08:46:27.273452Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It's actually very easy to use multiple gpus for training.\n\nYou just need to wrap above training scripts by `main` method and use `launch` method provided by `detectron2`.\n\nPlease refer official example [train_net.py](https://github.com/facebookresearch/detectron2/blob/master/tools/train_net.py#L161) for details.","metadata":{}},{"cell_type":"markdown","source":"\n# Visualize loss curve & competition metric AP40\n\nAs I explained, the calculated metrics are saved in `metrics.json`. We can analyze/plot them to check how the training proceeded.","metadata":{}},{"cell_type":"code","source":"metrics_df = pd.read_json(outdir / \"metrics.json\", orient=\"records\", lines=True)\nmdf = metrics_df.sort_values(\"iteration\")\nmdf","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:46:27.276067Z","iopub.execute_input":"2025-04-14T08:46:27.276416Z","iopub.status.idle":"2025-04-14T08:46:27.47763Z","shell.execute_reply.started":"2025-04-14T08:46:27.276376Z","shell.execute_reply":"2025-04-14T08:46:27.476625Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 1. Loss curve\nfig, ax = plt.subplots()\n\nmdf1 = mdf[~mdf[\"total_loss\"].isna()]\nax.plot(mdf1[\"iteration\"], mdf1[\"total_loss\"], c=\"C0\", label=\"train\")\nif \"validation_loss\" in mdf.columns:\n mdf2 = mdf[~mdf[\"validation_loss\"].isna()]\n ax.plot(mdf2[\"iteration\"], mdf2[\"validation_loss\"], c=\"C1\", label=\"validation\")\n\n# ax.set_ylim([0, 0.5])\nax.legend()\nax.set_title(\"Loss curve\")\nplt.show()\nplt.savefig(outdir/\"loss.png\")","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:46:27.478849Z","iopub.execute_input":"2025-04-14T08:46:27.479137Z","iopub.status.idle":"2025-04-14T08:46:27.646Z","shell.execute_reply.started":"2025-04-14T08:46:27.479088Z","shell.execute_reply":"2025-04-14T08:46:27.645196Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots()\nmdf3 = mdf[~mdf[\"bbox/AP75\"].isna()]\nax.plot(mdf3[\"iteration\"], mdf3[\"bbox/AP75\"] / 100., c=\"C2\", label=\"validation\")\n\nax.legend()\nax.set_title(\"AP40\")\nplt.show()\nplt.savefig(outdir / \"AP40.png\")","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:46:27.647521Z","iopub.execute_input":"2025-04-14T08:46:27.647879Z","iopub.status.idle":"2025-04-14T08:46:27.803951Z","shell.execute_reply.started":"2025-04-14T08:46:27.647842Z","shell.execute_reply":"2025-04-14T08:46:27.802961Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots()\nmdf_bbox_class = mdf3.iloc[-1][[f\"bbox/AP-{col}\" for col in thing_classes]]\nmdf_bbox_class.plot(kind=\"bar\", ax=ax)\n_ = ax.set_title(\"AP by class\")","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:46:27.818064Z","iopub.execute_input":"2025-04-14T08:46:27.818504Z","iopub.status.idle":"2025-04-14T08:46:27.994961Z","shell.execute_reply.started":"2025-04-14T08:46:27.818463Z","shell.execute_reply":"2025-04-14T08:46:27.994139Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Our Evaluator calculaes AP by class, and it is easy to check which class is diffucult to train.\n\nIn my experiment, **\"Calcification\" seems to be the most difficult class to predict**.","metadata":{}},{"cell_type":"markdown","source":"\n# Visualization of augmentation by Mapper\n\nLet's check the behavior of Mapper method. Since mapper is used inside DataLoader, we can check its behavior by constucting DataLoader and visualize the data processed by the DataLoader.\n\nThe defined Trainer class has **class method** `build_train_loader`. We can construct train_loader purely from `cfg`, without instantiating `trainer` since it's class method.\n\nBelow code is to visualize the same data 4 times. You can check that augmentation is applied and every time the image looks different.\n\nNote that both `detectron2.data.transforms` & `albumentations` augmentations properly handles bounding box. Thus bounding box is adjusted when the image is scaled, rotated etc!\n\nAt first I was using `detectron2.data.transforms` with `MyMapper` class, it provides basic augmentations.
\nThen I noticed that we can use many augmentations in `albumentations`, so I implemented `AlbumentationsMapper` to support it.
\nHow many augmentations can be used in albumentations?
\nYou can see official github page, all [Pixel-level transforms](https://github.com/albumentations-team/albumentations#pixel-level-transforms) and [Spatial-level transforms](https://github.com/albumentations-team/albumentations#spatial-level-transforms) with \"BBoxes\" checked can be used. There are really many!!!","metadata":{}},{"cell_type":"code","source":"# Visualize data...\n# import matplotlib.pyplot as plt\nfrom detectron2.data.samplers import TrainingSampler\n\nn_images = 2\nn_aug = 4\n\nfig, axes = plt.subplots(n_images, n_aug, figsize=(16, 8))\n\n# Ref https://github.com/facebookresearch/detectron2/blob/22b70a8078eb09da38d0fefa130d0f537562bebc/tools/visualize_data.py#L79-L88\nfor i in range(n_aug):\n sampler = TrainingSampler(len(dataset_dicts), shuffle=False)\n train_vis_loader = MyTrainer.build_train_loader(\n cfg, sampler=sampler\n ) # For visualization...\n for batch in train_vis_loader:\n for j, per_image in enumerate(batch):\n ax = axes[j, i]\n\n img_arr = per_image[\"image\"].cpu().numpy().transpose((1, 2, 0))\n visualizer = Visualizer(\n img_arr[:, :, ::-1], metadata=vinbigdata_metadata, scale=1.0\n )\n target_fields = per_image[\"instances\"].get_fields()\n labels = [\n vinbigdata_metadata.thing_classes[i] for i in target_fields[\"gt_classes\"]\n ]\n out = visualizer.overlay_instances(\n labels=labels,\n boxes=target_fields.get(\"gt_boxes\", None),\n masks=target_fields.get(\"gt_masks\", None),\n keypoints=target_fields.get(\"gt_keypoints\", None),\n )\n # out = visualizer.draw_dataset_dict(per_image)\n\n img = out.get_image()[:, :, ::-1]\n filepath = str(outdir / f\"vinbigdata_{j}_aug{i}.jpg\")\n cv2.imwrite(filepath, img)\n print(f\"Visualization img {img_arr.shape} saved in {filepath}\")\n ax.imshow(img)\n ax.set_title(f\"image{j}, {i}-th aug\")\n break","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T08:46:27.996099Z","iopub.execute_input":"2025-04-14T08:46:27.996408Z","iopub.status.idle":"2025-04-14T08:46:33.018918Z","shell.execute_reply.started":"2025-04-14T08:46:27.996383Z","shell.execute_reply":"2025-04-14T08:46:33.018094Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"That's all\nThe competition data is not so many (15000 for all images, 4000 images after filtering \"No finding\" images).
","metadata":{}},{"cell_type":"markdown","source":"### Evaluate vinbigdata on train","metadata":{}},{"cell_type":"code","source":"import pickle\nfrom pathlib import Path\nimport cv2\nimport pandas as pd\nfrom tqdm import tqdm\n\ndef get_test_dicts_from_train(\n imgdir: Path,\n train_meta: pd.DataFrame,\n use_cache: bool = True,\n debug: bool = True,\n):\n \"\"\"\n Dùng train_meta.csv để tạo dataset cho inference (giả định dùng ảnh train làm test).\n Chú ý: Các file ảnh được load từ thư mục \"train\".\n \"\"\"\n debug_str = f\"_debug{int(debug)}\"\n cache_path = Path(\".\") / f\"dataset_dicts_test_from_train{debug_str}.pkl\"\n if not use_cache or not cache_path.exists():\n print(\"Creating test dataset (from train_meta)...\")\n # Nếu debug thì dùng một phần của train_meta\n if debug:\n train_meta = train_meta.iloc[:500]\n \n dataset_dicts = []\n # Duyệt từng dòng trong train_meta\n for index, row in tqdm(train_meta.iterrows(), total=len(train_meta)):\n record = {}\n # Giả sử train_meta có các cột: image_id, height, width\n image_id = row[\"image_id\"]\n filename = str(imgdir / \"train\" / f\"{image_id}.png\")\n # Bạn có thể load ảnh để lấy kích thước thực nếu cần,\n # hoặc sử dụng giá trị height, width có trong train_meta.\n image = cv2.imread(filename)\n if image is None:\n continue\n h, w = image.shape[:2]\n record[\"file_name\"] = filename\n record[\"image_id\"] = image_id\n record[\"height\"] = h\n record[\"width\"] = w\n # Ở đây không có annotation vì đây là tập test.\n dataset_dicts.append(record)\n with open(cache_path, \"wb\") as f:\n pickle.dump(dataset_dicts, f)\n else:\n print(f\"Load from cache {cache_path}\")\n with open(cache_path, \"rb\") as f:\n dataset_dicts = pickle.load(f)\n return dataset_dicts\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T09:10:34.194261Z","iopub.execute_input":"2025-04-14T09:10:34.194611Z","iopub.status.idle":"2025-04-14T09:10:34.204737Z","shell.execute_reply.started":"2025-04-14T09:10:34.194576Z","shell.execute_reply":"2025-04-14T09:10:34.20397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from detectron2.engine import DefaultPredictor\nimport matplotlib.pyplot as plt\n\npredictor = DefaultPredictor(cfg)\n\n# Giả sử imgdir là đường dẫn (Path) đến thư mục chứa dataset VinBigData.\n# Load train_meta.csv (vì dataset chỉ có file này)\nimgdir = Path(\"/kaggle/input/vinbigdata\") # cập nhật đường dẫn thật của bạn\ntrain_meta = pd.read_csv(imgdir / \"train_meta.csv\")\n\n# Lấy dataset test tạo ra từ train_meta.csv\ntest_dicts = get_test_dicts_from_train(imgdir, train_meta, use_cache=True, debug=False)\n\n# Chọn mẫu đầu tiên trong tập test\nsample_dict = test_dicts[0]\nimage_path = sample_dict[\"file_name\"]\nim = cv2.imread(image_path)\n\n# Chạy inference\noutputs = predictor(im)\ninstances = outputs[\"instances\"].to(\"cpu\")\n\n# Lấy thông tin dự đoán\npred_boxes = instances.pred_boxes if len(instances) > 0 else None\npred_scores = instances.scores if len(instances) > 0 else None\npred_classes = instances.pred_classes if len(instances) > 0 else None\n\n# Vẽ bounding box lên ảnh\nif pred_boxes is not None:\n sample_im = im.copy()\n for box, score, cls in zip(pred_boxes, pred_scores, pred_classes):\n x1, y1, x2, y2 = box.int().numpy()\n cv2.rectangle(sample_im, (x1, y1), (x2, y2), color=(0, 255, 0), thickness=2)\n label_text = f\"{cls.item()}:{score:.2f}\" # Nếu có tên lớp, bạn có thể map thêm\n cv2.putText(sample_im, label_text, (x1, y1 - 5), cv2.FONT_HERSHEY_SIMPLEX, \n 0.5, (0, 255, 0), 1)\n \n plt.figure(figsize=(10, 10))\n plt.axis(\"off\")\n plt.imshow(cv2.cvtColor(sample_im, cv2.COLOR_BGR2RGB))\n plt.show()\nelse:\n print(\"Không có dự đoán nào được tìm thấy.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T09:11:20.1106Z","iopub.execute_input":"2025-04-14T09:11:20.110909Z","iopub.status.idle":"2025-04-14T09:14:11.896016Z","shell.execute_reply.started":"2025-04-14T09:11:20.110882Z","shell.execute_reply":"2025-04-14T09:14:11.895229Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import cv2\nimport numpy as np\nfrom tqdm import tqdm\n\n# --- Các hàm hỗ trợ tính chỉ số ---\n\ndef compute_iou(box1, box2):\n \"\"\"\n Tính IoU giữa 2 box. Các box có định dạng [x1, y1, x2, y2].\n \"\"\"\n x1, y1, x2, y2 = box1\n x1g, y1g, x2g, y2g = box2\n xi1, yi1 = max(x1, x1g), max(y1, y1g)\n xi2, yi2 = min(x2, x2g), min(y2, y2g)\n inter_area = max(0, xi2 - xi1) * max(0, yi2 - yi1)\n box_area = (x2 - x1) * (y2 - y1)\n gt_area = (x2g - x1g) * (y2g - y1g)\n union_area = box_area + gt_area - inter_area\n return inter_area / union_area if union_area > 0 else 0\n\ndef calculate_precision_recall_ap(pred_boxes, gt_boxes, iou_threshold=0.5):\n \"\"\"\n pred_boxes: Danh sách các mục [image_id, class_id, confidence, x1, y1, x2, y2]\n gt_boxes: Danh sách các mục [image_id, class_id, x1, y1, x2, y2]\n \"\"\"\n pred_boxes.sort(key=lambda x: x[2], reverse=True)\n tp, fp = [], []\n matched_gt = set()\n\n for pred in pred_boxes:\n image_id, cls, score, *pred_box = pred\n found_match = False\n for i, gt in enumerate(gt_boxes):\n if gt[0] == image_id and gt[1] == cls and i not in matched_gt:\n iou = compute_iou(pred_box, gt[2:])\n if iou >= iou_threshold:\n found_match = True\n matched_gt.add(i)\n break\n if found_match:\n tp.append(1)\n fp.append(0)\n else:\n tp.append(0)\n fp.append(1)\n\n tp_cum = np.cumsum(tp)\n fp_cum = np.cumsum(fp)\n eps = 1e-6\n precisions = tp_cum / (tp_cum + fp_cum + eps)\n recalls = tp_cum / (len(gt_boxes) + eps)\n \n # Tính AP theo 11 điểm recall\n ap = 0.0\n for t in np.linspace(0, 1, 11):\n p = precisions[recalls >= t]\n ap += np.max(p) if p.size else 0\n ap /= 11.0\n return precisions, recalls, ap\n\ndef calculate_metrics_per_class(pred_boxes, gt_boxes, iou_threshold=0.5):\n \"\"\"\n Tính các chỉ số cho 1 class cụ thể: AP, precision, recall, F1 và mean IoU (chỉ tính trên TP).\n \"\"\"\n pred_boxes.sort(key=lambda x: x[2], reverse=True)\n tp, fp, used_gt_idx, iou_list = [], [], set(), []\n for pred in pred_boxes:\n image_id, cls, score, *pred_box = pred\n found = False\n for i, gt in enumerate(gt_boxes):\n if gt[0] == image_id and gt[1] == cls and i not in used_gt_idx:\n iou = compute_iou(pred_box, gt[2:])\n if iou >= iou_threshold:\n found = True\n used_gt_idx.add(i)\n iou_list.append(iou)\n break\n tp.append(1 if found else 0)\n fp.append(0 if found else 1)\n tp_cum = np.cumsum(tp)\n fp_cum = np.cumsum(fp)\n eps = 1e-6\n precision_curve = tp_cum / (tp_cum + fp_cum + eps)\n recall_curve = tp_cum / (len(gt_boxes) + eps)\n final_precision = precision_curve[-1] if len(precision_curve) else 0\n final_recall = recall_curve[-1] if len(recall_curve) else 0\n f1 = 2 * final_precision * final_recall / (final_precision + final_recall + eps)\n _, _, ap = calculate_precision_recall_ap(pred_boxes, gt_boxes, iou_threshold)\n mean_iou = np.mean(iou_list) if iou_list else 0\n return {\n \"AP\": ap,\n \"precision\": final_precision,\n \"recall\": final_recall,\n \"f1_score\": f1,\n \"mean_iou\": mean_iou,\n }\n\ndef compute_map(pred_boxes, gt_boxes, iou_thresh=0.5):\n \"\"\"\n Tính mAP theo từng lớp.\n pred_boxes: [image_id, class_id, confidence, x1, y1, x2, y2]\n gt_boxes: [image_id, class_id, x1, y1, x2, y2]\n \"\"\"\n classes = sorted(set([b[1] for b in gt_boxes] + [b[1] for b in pred_boxes]))\n ap_list = []\n metrics_per_class = {}\n for cls in classes:\n preds_cls = [b for b in pred_boxes if b[1] == cls]\n gts_cls = [b for b in gt_boxes if b[1] == cls]\n metrics = calculate_metrics_per_class(preds_cls, gts_cls, iou_threshold=iou_thresh)\n metrics_per_class[cls] = metrics\n ap_list.append(metrics[\"AP\"])\n mAP = np.mean(ap_list) if ap_list else 0\n return mAP, metrics_per_class\n\ndef evaluate_model(predictor, dataset_dicts, iou_thresh=0.5):\n \"\"\"\n predictor: instance của DefaultPredictor.\n dataset_dicts: danh sách các dict chứa thông tin \"file_name\", \"image_id\", và \"annotations\" (cho ảnh có annotation).\n \"\"\"\n pred_boxes = [] # [image_id, class_id, confidence, x1, y1, x2, y2]\n gt_boxes = [] # [image_id, class_id, x1, y1, x2, y2]\n\n for record in tqdm(dataset_dicts, desc=\"Evaluating validation set\"):\n image_id = record[\"image_id\"]\n im = cv2.imread(record[\"file_name\"])\n outputs = predictor(im)\n instances = outputs[\"instances\"].to(\"cpu\")\n \n # Dự đoán\n if len(instances) > 0:\n boxes = instances.pred_boxes.tensor.numpy()\n scores = instances.scores.numpy()\n classes = instances.pred_classes.numpy()\n for box, score, cls in zip(boxes, scores, classes):\n pred_boxes.append([image_id, int(cls), float(score), *box])\n \n # Ground truth (giả sử record có key \"annotations\" chứa danh sách các dict)\n for ann in record.get(\"annotations\", []):\n bbox = ann[\"bbox\"]\n cls = ann[\"category_id\"]\n gt_boxes.append([image_id, int(cls), *bbox])\n \n mAP, metrics_per_class = compute_map(pred_boxes, gt_boxes, iou_thresh)\n \n # Tính trung bình precision, recall, f1 và mean IoU trên toàn bộ lớp\n all_precisions = [metrics[\"precision\"] for metrics in metrics_per_class.values()]\n all_recalls = [metrics[\"recall\"] for metrics in metrics_per_class.values()]\n all_f1 = [metrics[\"f1_score\"] for metrics in metrics_per_class.values()]\n all_iou = [metrics[\"mean_iou\"] for metrics in metrics_per_class.values()]\n \n mean_precision = np.mean(all_precisions) if all_precisions else 0\n mean_recall = np.mean(all_recalls) if all_recalls else 0\n mean_f1 = np.mean(all_f1) if all_f1 else 0\n mean_iou = np.mean(all_iou) if all_iou else 0\n \n results = {\n \"mAP\": mAP,\n \"mean_precision\": mean_precision,\n \"mean_recall\": mean_recall,\n \"mean_f1_score\": mean_f1,\n \"mean_iou\": mean_iou,\n \"per_class\": metrics_per_class\n }\n return results\n\n# --- 4. Ví dụ sử dụng đánh giá mô hình ---\n# Giả sử bạn đã có dataset_dicts của validation (chứa annotation) từ hàm get_vinbigdata_dicts hoặc tương tự.\n# Ví dụ:\nvalid_dicts = get_vinbigdata_dicts(imgdir, train_df, train_data_type=\"original\", use_cache=True, debug=False)\n# (Lưu ý: valid_dicts cần chứa key \"annotations\" cho việc đánh giá)\n\nresults = evaluate_model(predictor, valid_dicts, iou_thresh=0.5)\n\nprint(\"=== Các chỉ số trung bình trên tập validation ===\")\nprint(f\"mAP@0.5: {results['mAP']:.4f}\")\nprint(f\"Precision: {results['mean_precision']:.4f}\")\nprint(f\"Recall: {results['mean_recall']:.4f}\")\nprint(f\"F1 Score: {results['mean_f1_score']:.4f}\")\nprint(f\"Mean IoU: {results['mean_iou']:.4f}\")\n\nprint(\"\\n=== Chỉ số theo từng class ===\")\nfor cls_id, metrics in results[\"per_class\"].items():\n print(f\"Class {cls_id}: \" + \", \".join(f\"{k}={v:.3f}\" for k, v in metrics.items()))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-14T09:35:06.326413Z","iopub.execute_input":"2025-04-14T09:35:06.326705Z","iopub.status.idle":"2025-04-14T10:17:51.441901Z","shell.execute_reply.started":"2025-04-14T09:35:06.326679Z","shell.execute_reply":"2025-04-14T10:17:51.441121Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}