{"cells":[{"metadata":{},"cell_type":"markdown","source":"# VinBigData detectron2 prediction\n\n\n**Following from the training kernel [VinBigData detectron2 train](https://www.kaggle.com/corochann/vinbigdata-detectron2-train), I will try prediction with the `detectron2` trained model**\n\n`detectron2` is one of the famous pytorch object detection library, I will introduce how to use this library to predict bounding boxes with the trained model.\n\n - https://github.com/facebookresearch/detectron2\n\n> Detectron2 is Facebook AI Research's next generation software system that implements state-of-the-art object detection algorithms. It is a ground-up rewrite of the previous version, Detectron, and it originates from maskrcnn-benchmark.\n![](https://user-images.githubusercontent.com/1381301/66535560-d3422200-eace-11e9-9123-5535d469db19.png)\n\n\n## Version history\n\n2021/1/22: Update to add 2-class filter in troduced in [VinBigData 🌟2 Class Filter🌟](https://www.kaggle.com/awsaf49/vinbigdata-2-class-filter) by @awsaf49 <br/>\nI also wrote kernel to train 2-class model: [📸VinBigData 2-class classifier complete pipeline](https://www.kaggle.com/corochann/vinbigdata-2-class-classifier-complete-pipeline)"},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n** [Prediction method implementations](#pred_method)** <br/>\n** [Prediction scripts](#pred_scripts)** <br/>\n** [Apply 2 class filter](#2class)** <br/>\n** [Other kernels](#ref)** <br/>\n\nSince first setup part is same with the training kernel, I skipped listing on ToC."},{"metadata":{},"cell_type":"markdown","source":"# Dataset preparation\n\nPreprocessing x-ray image format (dicom) into normal png image format is already done by @xhlulu in the below discussion:\n - [Multiple preprocessed datasets: 256/512/1024px, PNG and JPG, modified and original ratio](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/207955).\n\nHere I will just use the dataset [VinBigData Chest X-ray Resized PNG (256x256)](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-256x256) to skip the preprocessing and focus on modeling part. Please upvote the dataset as well!"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Installation\n\ndetectron2 is not pre-installed in this kaggle docker, so let's install it. \nWe can follow [installation instruction](https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md), we need to know CUDA and pytorch version to install correct `detectron2`."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!nvidia-smi","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!nvcc --version","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import torch\n\ntorch.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems CUDA=10.2 and torch==1.7.0 is used in this kaggle docker image.\n\nSee [installation](https://detectron2.readthedocs.io/tutorials/install.html) for details."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install detectron2 -f \\\n  https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.7/index.html","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"pred_method\"></a>\n# Prediction method implementations\n\nBasically we don't need to implement neural network part, `detectron2` already implements famous architectures and provides its pre-trained weights. We can finetune these pre-trained architectures.\n\nThese models are summarized in [MODEL_ZOO.md](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md).\n\nIn this competition, we need object detection model, I will choose [R50-FPN](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml) for this kernel."},{"metadata":{},"cell_type":"markdown","source":"## Data preparation\n\n`detectron2` provides high-level API for training custom dataset.\n\nTo define custom dataset, we need to create **list of dict** where each dict contains following:\n\n - file_name: file name of the image.\n - image_id: id of the image, index is used here.\n - height: height of the image.\n - width: width of the image.\n - annotation: This is the ground truth annotation data for object detection, which contains following\n     - bbox: bounding box pixel location with shape (n_boxes, 4)\n     - bbox_mode: `BoxMode.XYXY_ABS` is used here, meaning that absolute value of (xmin, ymin, xmax, ymax) annotation is used in the `bbox`.\n     - category_id: class label id for each bounding box, with shape (n_boxes,)\n\n`get_vinbigdata_dicts` is for train dataset preparation and `get_vinbigdata_dicts_test` is for test dataset preparation."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import pickle\nfrom pathlib import Path\n\nimport cv2\nimport pandas as pd\nfrom detectron2.structures import BoxMode\nfrom tqdm import tqdm\n\n\ndef get_vinbigdata_dicts(\n    imgdir: Path, train: pd.DataFrame, use_cache: bool = True, debug: bool = True,\n):\n    debug_str = f\"_debug{int(debug)}\"\n    cache_path = Path(\".\") / f\"dataset_dicts_cache{debug_str}.pkl\"\n    if not use_cache or not cache_path.exists():\n        print(\"Creating data...\")\n        train_meta = pd.read_csv(imgdir / \"train_meta.csv\")\n        if debug:\n            train_meta = train_meta.iloc[:500]  # For debug....\n\n        # Load 1 image to get image size.\n        image_id = train_meta.loc[0, \"image_id\"]\n        image_path = str(imgdir / \"train\" / f\"{image_id}.png\")\n        image = cv2.imread(image_path)\n        resized_height, resized_width, ch = image.shape\n        print(f\"image shape: {image.shape}\")\n\n        dataset_dicts = []\n        for index, train_meta_row in tqdm(train_meta.iterrows(), total=len(train_meta)):\n            record = {}\n\n            image_id, height, width = train_meta_row.values\n            filename = str(imgdir / \"train\" / f\"{image_id}.png\")\n            record[\"file_name\"] = filename\n            record[\"image_id\"] = index\n            record[\"height\"] = resized_height\n            record[\"width\"] = resized_width\n            objs = []\n            for index2, row in train.query(\"image_id == @image_id\").iterrows():\n                # print(row)\n                # print(row[\"class_name\"])\n                # class_name = row[\"class_name\"]\n                class_id = row[\"class_id\"]\n                if class_id == 14:\n                    # It is \"No finding\"\n                    # This annotator does not find anything, skip.\n                    pass\n                else:\n                    # bbox_original = [int(row[\"x_min\"]), int(row[\"y_min\"]), int(row[\"x_max\"]), int(row[\"y_max\"])]\n                    h_ratio = resized_height / height\n                    w_ratio = resized_width / width\n                    bbox_resized = [\n                        int(row[\"x_min\"]) * w_ratio,\n                        int(row[\"y_min\"]) * h_ratio,\n                        int(row[\"x_max\"]) * w_ratio,\n                        int(row[\"y_max\"]) * h_ratio,\n                    ]\n                    obj = {\n                        \"bbox\": bbox_resized,\n                        \"bbox_mode\": BoxMode.XYXY_ABS,\n                        \"category_id\": class_id,\n                    }\n                    objs.append(obj)\n            record[\"annotations\"] = objs\n            dataset_dicts.append(record)\n        with open(cache_path, mode=\"wb\") as f:\n            pickle.dump(dataset_dicts, f)\n\n    print(f\"Load from cache {cache_path}\")\n    with open(cache_path, mode=\"rb\") as f:\n        dataset_dicts = pickle.load(f)\n    return dataset_dicts\n\n\ndef get_vinbigdata_dicts_test(\n    imgdir: Path, test_meta: pd.DataFrame, use_cache: bool = True, debug: bool = True,\n):\n    debug_str = f\"_debug{int(debug)}\"\n    cache_path = Path(\".\") / f\"dataset_dicts_cache_test{debug_str}.pkl\"\n    if not use_cache or not cache_path.exists():\n        print(\"Creating data...\")\n        # test_meta = pd.read_csv(imgdir / \"test_meta.csv\")\n        if debug:\n            test_meta = test_meta.iloc[:500]  # For debug....\n\n        # Load 1 image to get image size.\n        image_id = test_meta.loc[0, \"image_id\"]\n        image_path = str(imgdir / \"test\" / f\"{image_id}.png\")\n        image = cv2.imread(image_path)\n        resized_height, resized_width, ch = image.shape\n        print(f\"image shape: {image.shape}\")\n\n        dataset_dicts = []\n        for index, test_meta_row in tqdm(test_meta.iterrows(), total=len(test_meta)):\n            record = {}\n\n            image_id, height, width = test_meta_row.values\n            filename = str(imgdir / \"test\" / f\"{image_id}.png\")\n            record[\"file_name\"] = filename\n            record[\"image_id\"] = index\n            record[\"height\"] = resized_height\n            record[\"width\"] = resized_width\n            # objs = []\n            # record[\"annotations\"] = objs\n            dataset_dicts.append(record)\n        with open(cache_path, mode=\"wb\") as f:\n            pickle.dump(dataset_dicts, f)\n\n    print(f\"Load from cache {cache_path}\")\n    with open(cache_path, mode=\"rb\") as f:\n        dataset_dicts = pickle.load(f)\n    return dataset_dicts\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Methods for prediction for this competition"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Methods for prediction for this competition\nfrom math import ceil\nfrom typing import Any, Dict, List\n\nimport cv2\nimport detectron2\nimport numpy as np\nfrom numpy import ndarray\nimport pandas as pd\nimport torch\nfrom detectron2 import model_zoo\nfrom detectron2.config import get_cfg\nfrom detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader\nfrom detectron2.engine import DefaultPredictor\nfrom detectron2.evaluation import COCOEvaluator, inference_on_dataset\nfrom detectron2.structures import BoxMode\nfrom detectron2.utils.logger import setup_logger\nfrom detectron2.utils.visualizer import ColorMode, Visualizer\nfrom tqdm import tqdm\n\n\ndef format_pred(labels: ndarray, boxes: ndarray, scores: ndarray) -> str:\n    pred_strings = []\n    for label, score, bbox in zip(labels, scores, boxes):\n        xmin, ymin, xmax, ymax = bbox.astype(np.int64)\n        pred_strings.append(f\"{label} {score} {xmin} {ymin} {xmax} {ymax}\")\n    return \" \".join(pred_strings)\n\n\ndef predict_batch(predictor: DefaultPredictor, im_list: List[ndarray]) -> List:\n    with torch.no_grad():  # https://github.com/sphinx-doc/sphinx/issues/4258\n        inputs_list = []\n        for original_image in im_list:\n            # Apply pre-processing to image.\n            if predictor.input_format == \"RGB\":\n                # whether the model expects BGR inputs or RGB\n                original_image = original_image[:, :, ::-1]\n            height, width = original_image.shape[:2]\n            image = predictor.aug.get_transform(original_image).apply_image(original_image)\n            image = torch.as_tensor(image.astype(\"float32\").transpose(2, 0, 1))\n            inputs = {\"image\": image, \"height\": height, \"width\": width}\n            inputs_list.append(inputs)\n        predictions = predictor.model(inputs_list)\n        return predictions\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# --- utils ---\nfrom pathlib import Path\nfrom typing import Any, Union\n\nimport yaml\n\n\ndef save_yaml(filepath: Union[str, Path], content: Any, width: int = 120):\n    with open(filepath, \"w\") as f:\n        yaml.dump(content, f, width=width)\n\n\ndef load_yaml(filepath: Union[str, Path]) -> Any:\n    with open(filepath, \"r\") as f:\n        content = yaml.full_load(f)\n    return content\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- configs ---\nthing_classes = [\n    \"Aortic enlargement\",\n    \"Atelectasis\",\n    \"Calcification\",\n    \"Cardiomegaly\",\n    \"Consolidation\",\n    \"ILD\",\n    \"Infiltration\",\n    \"Lung Opacity\",\n    \"Nodule/Mass\",\n    \"Other lesion\",\n    \"Pleural effusion\",\n    \"Pleural thickening\",\n    \"Pneumothorax\",\n    \"Pulmonary fibrosis\"\n]\ncategory_name_to_id = {class_name: index for index, class_name in enumerate(thing_classes)}\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This `Flags` class is to manage experiments. I will tune these parameters through the competition to improve model's performance."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# --- flags ---\nfrom dataclasses import dataclass\nfrom typing import Dict\n\n\n@dataclass\nclass Flags:\n    # General\n    debug: bool = True\n    outdir: str = \"results/det\"\n\n    # Data config\n    imgdir_name: str = \"vinbigdata-chest-xray-resized-png-256x256\"\n    # Training config\n    iter: int = 10000\n    ims_per_batch: int = 2  # images per batch, this corresponds to \"total batch size\"\n    num_workers: int = 4\n    base_lr: float = 0.00025\n    roi_batch_size_per_image: int = 512\n\n    def update(self, param_dict: Dict) -> \"Flags\":\n        # Overwrite by `param_dict`\n        for key, value in param_dict.items():\n            if not hasattr(self, key):\n                raise ValueError(f\"[ERROR] Unexpected key for flag = {key}\")\n            setattr(self, key, value)\n        return self\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"pred_scripts\"></a>\n# Prediction scripts\n\nNow the methods are ready. Main training scripts starts from here."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"flags_dict = {\n    \"debug\": False,\n    \"outdir\": \"results/debug\", \n    \"imgdir_name\": \"vinbigdata-chest-xray-resized-png-256x256\",\n    \"iter\": 100,  # debug, small value should be set.\n    \"roi_batch_size_per_image\": 128  # faster, and good enough for this toy dataset (default: 512)\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DatasetCatalog.register(\n    \"vinbigdata_test\", lambda: get_vinbigdata_dicts_test(imgdir, test_meta, debug=debug)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf /kaggle/working/dataset_dicts_cache_test_debug1.pkl","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inputdir = Path(\"/kaggle/input\")\ntraineddir = inputdir / \"/kaggle/input/vinbigdatadetectron22/results/v9\"\n\n# flags = Flags()\nflags: Flags = Flags().update(load_yaml(str(\"../input/vinbigdatadetectron2/results/debug/flags.yaml\")))\nprint(\"flags\", flags)\ndebug = False \n# flags_dict = dataclasses.asdict(flags)\noutdir = Path(flags.outdir)\nos.makedirs(str(outdir), exist_ok=True)\n\n# --- Read data ---\ndatadir = inputdir / \"vinbigdata-chest-xray-abnormalities-detection\"\nimgdir = inputdir / flags.imgdir_name\n\n# Read in the data CSV files\n# train = pd.read_csv(datadir / \"train.csv\")\ntest_meta = pd.read_csv(inputdir / \"vinbigdata-testmeta\" / \"test_meta.csv\")\nsample_submission = pd.read_csv(datadir / \"sample_submission.csv\")\n\ncfg = get_cfg()\noriginal_output_dir = cfg.OUTPUT_DIR\ncfg.OUTPUT_DIR = str(outdir)\nprint(f\"cfg.OUTPUT_DIR {original_output_dir} -> {cfg.OUTPUT_DIR}\")\n\ncfg.merge_from_file(model_zoo.get_config_file(\"COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml\"))\ncfg.DATASETS.TRAIN = (\"vinbigdata_train\",)\ncfg.DATASETS.TEST = ()\n# cfg.DATASETS.TEST = (\"vinbigdata_train\",)\n# cfg.TEST.EVAL_PERIOD = 50\ncfg.DATALOADER.NUM_WORKERS = 2\n# Let training initialize from model zoo\ncfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml\")\ncfg.SOLVER.IMS_PER_BATCH = 2\ncfg.SOLVER.BASE_LR = flags.base_lr  # pick a good LR\ncfg.SOLVER.MAX_ITER = flags.iter\ncfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = flags.roi_batch_size_per_image\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = len(thing_classes)\n# NOTE: this config means the number of classes, but a few popular unofficial tutorials incorrect uses num_classes+1 here.\n\n### --- Inference & Evaluation ---\n# Inference should use the config with parameters that are used in training\n# cfg now already contains everything we've set previously. We changed it a little bit for inference:\n# path to the model we just trained\n# cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, \"model_final.pth\")\ncfg.MODEL.WEIGHTS = str(traineddir / \"model_final.pth\")\nprint(\"Original thresh\", cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST)  # 0.05\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.3   # set a custom testing threshold\nprint(\"Changed  thresh\", cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST)  # 0.0\npredictor = DefaultPredictor(cfg)\n\n\nMetadataCatalog.get(\"vinbigdata_test\").set(thing_classes=thing_classes)\nmetadata = MetadataCatalog.get(\"vinbigdata_test\")\ndataset_dicts = get_vinbigdata_dicts_test(imgdir, test_meta, debug=debug)\n\n# if debug:\n#     dataset_dicts = dataset_dicts[:100]\nprint(len(dataset_dicts))\nresults_list = []\nindex = 0\nbatch_size = 4\n\nfor i in tqdm(range(ceil(len(dataset_dicts) / batch_size))):\n    inds = list(range(batch_size * i, min(batch_size * (i + 1), len(dataset_dicts))))\n    dataset_dicts_batch = [dataset_dicts[i] for i in inds]\n    im_list = [cv2.imread(d[\"file_name\"]) for d in dataset_dicts_batch]\n    outputs_list = predict_batch(predictor, im_list)\n\n    for im, outputs, d in zip(im_list, outputs_list, dataset_dicts_batch):\n        resized_height, resized_width, ch = im.shape\n        # outputs = predictor(im)\n        if index < 5:\n            # format is documented at https://detectron2.readthedocs.io/tutorials/models.html#model-output-format\n            v = Visualizer(\n                im[:, :, ::-1],\n                metadata=metadata,\n                scale=0.5,\n                instance_mode=ColorMode.IMAGE_BW\n                # remove the colors of unsegmented pixels. This option is only available for segmentation models\n            )\n            out = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n            cv2.imwrite(str(outdir / f\"pred_{index}.jpg\"), out.get_image()[:, :, ::-1])\n            plt.title(f\"index {index}\")\n            plt.imshow(out.get_image()[:, :, ::-1])\n\n        assert d[\"image_id\"] == index\n        image_id, dim0, dim1 = test_meta.iloc[index].values\n\n        instances = outputs[\"instances\"]\n        if len(instances) == 0:\n            # No finding, let's set 14 0 0 0 0.\n            result = {\"image_id\": image_id, \"PredictionString\": \"14 1.0 0 0 1 1\"}\n        else:\n            # Find some bbox...\n            # print(f\"index={index}, find {len(instances)} bbox.\")\n            fields: Dict[str, Any] = instances.get_fields()\n            pred_classes = fields[\"pred_classes\"]  # (n_boxes,)\n            pred_scores = fields[\"scores\"]\n            # shape (n_boxes, 4). (xmin, ymin, xmax, ymax)\n            pred_boxes = fields[\"pred_boxes\"].tensor\n\n            h_ratio = dim0 / resized_height\n            w_ratio = dim1 / resized_width\n            pred_boxes[:, [0, 2]] *= w_ratio\n            pred_boxes[:, [1, 3]] *= h_ratio\n\n            pred_classes_array = pred_classes.cpu().numpy()\n            pred_boxes_array = pred_boxes.cpu().numpy()\n            pred_scores_array = pred_scores.cpu().numpy()\n\n            result = {\n                \"image_id\": image_id,\n                \"PredictionString\": format_pred(\n                    pred_classes_array, pred_boxes_array, pred_scores_array\n                ),\n            }\n        results_list.append(result)\n        index += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I set `cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.0` to produce **all the detection box prediction even if confidence score is very low**.<br/>\nActually it affects a lot to score, since competition metric is AP (Average-Precision) which is calculated using the boxes with confidence score = 0~100%."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This submission includes only detection model's predictions\nsubmission_det = pd.DataFrame(results_list, columns=['image_id', 'PredictionString'])\nsubmission_det.to_csv(outdir/\"submission_det.csv\", index=False)\nsubmission_det","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2class\"></a>\n# Apply 2 class filter\n\nAs mentioned in [VinBigData 🌟2 Class Filter🌟](https://www.kaggle.com/awsaf49/vinbigdata-2-class-filter) by @awsaf49, applying 2-class filter improves LB score significantly. (Please upvote his kernel as well!)<br/>\nAlso, it is mentioned that we can submit **14 prob 1 1 0 0** where the `prob` is the normal probability in the discussion [[Scoring bug] Improve your LB score by 0.053, just adding \"14 1 0 0 1 1\"](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/211971)!\n\nHere, I will propose new post processing (similar to [this](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/211971#1157809) by @pestipeti):\n\nHere `p` is the **normal probability**.\n\n1. `p < low_threshold`                   -> Do nothing, Keep det prediction.\n2. `low_threshold <= p < high_threshold` -> Just \"Add\" Normal prediction, **keep** detection prediction.\n3. `high_threshold <= p`                 -> Replace with Normal prediction with normal score 1.0, **remove** all detection predictoin.\n\n\n[Note] I also wrote another kernel to train 2-class model: [📸VinBigData 2-class classifier complete pipeline](https://www.kaggle.com/corochann/vinbigdata-2-class-classifier-complete-pipeline) to train these 2-class classifier model!"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_2class = pd.read_csv(inputdir/\"vinbigdata-2class-prediction/2-cls test pred.csv\")\npred_2class","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NORMAL = \"14 1 0 0 1 1\"\nlow_threshold = 0.0\nhigh_threshold = 0.95\n\npred_det_df = submission_det  # You can load from another submission.csv here too.\nn_normal_before = len(pred_det_df.query(\"PredictionString == @NORMAL\"))\nmerged_df = pd.merge(pred_det_df, pred_2class, on=\"image_id\", how=\"left\")\n\n# 1. p < low_threshold                   -> \"Keep\": Do nothing, Keep det prediction.\n# 2. low_threshold <= p < high_threshold -> \"Add\": Just \"Add\" Normal prediction\n# 3. high_threshold <= p                 -> \"Replace\": Replace with Normal prediction\n\nif \"target\" in merged_df.columns:\n    merged_df[\"class0\"] = 1 - merged_df[\"target\"]\n\nc0, c1, c2 = 0, 0, 0\nfor i in range(len(merged_df)):\n    p0 = merged_df.loc[i, \"class0\"]\n    if p0 < low_threshold:\n        # Keep, do nothing.\n        c0 += 1\n    elif low_threshold <= p0 and p0 < high_threshold:\n        # Add, keep \"det\" preds and add normal pred.\n        merged_df.loc[i, \"PredictionString\"] += f\" 14 {p0} 0 0 1 1\"\n        c1 += 1\n    else:\n        # Replace, remove all \"det\" preds.\n        merged_df.loc[i, \"PredictionString\"] = NORMAL\n        c2 += 1\n\nn_normal_after = len(merged_df.query(\"PredictionString == @NORMAL\"))\nprint(\n    f\"n_normal: {n_normal_before} -> {n_normal_after} with threshold {low_threshold} & {high_threshold}\"\n)\nprint(f\"Keep {c0} Add {c1} Replace {c2}\")\nsubmission_filepath = str(outdir / \"submission.csv\")\nsubmission_df = merged_df[[\"image_id\", \"PredictionString\"]]\nsubmission_df.to_csv(submission_filepath, index=False)\nprint(f\"Saved to {submission_filepath}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In my experiment:\n - The baseline submission: score 0.141\n - Just replace by threshold ([version3](https://www.kaggle.com/corochann/vinbigdata-detectron2-prediction?scriptVersionId=52412540) ): score 0.206\n - This post process (combine replace & add): **0.221**\n\nSo the score improved by about **0.8, which is significant**!\n\nIn more detail, I tried to change several `low_threshold` value and lower `low_threshold` achieved better results. So it may be okay to set `low_threshold=0.0` which means **always add \"No finding\" prediction with the predicted probability.**<br/>\nI also noticed that setting `high_threshold` value less than 1 is important, which means we have a benefit to **remove abnormality predictions** for the images which is highly likely to be normal.<br/>\nThis may be because my [training kernel](https://www.kaggle.com/corochann/vinbigdata-detectron2-train) currently only uses abnormal image during training and model tend to produce more abnormal boxes. I'm now thinking that it's better to include normal images for training to learn where there is **no** abnormality.<br/>\nAlso, I think it's nice to try **including \"No finding\" class during detection training** (by adding virtual \"No finding\" boxes, or by adding global classifier together with the detection).\n\n\nThat's all!\nObject deteaction is rather complicated task among deep learning tasks, but it's easy to train SoTA models & predict using `detectron2`!!!\n\n<h3 style=\"color:red\">If this kernel helps you, please upvote to keep me motivated 😁<br>Thanks!</h3>"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"ref\"></a>\n# Other kernels\n\n[📸VinBigData detectron2 train](https://www.kaggle.com/corochann/vinbigdata-detectron2-train) kernel explains how to run object detection training, using `detectron2` library.\n\n[📸VinBigData 2-class classifier complete pipeline](https://www.kaggle.com/corochann/vinbigdata-2-class-classifier-complete-pipeline) kernel explains how to train 2 class classifier model for the prediction and submisssion for this competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}