{"cells":[{"metadata":{},"cell_type":"markdown","source":"The aim of the notebook is run inference on the test dataset using the faster-rcnn model trained for 12 epochs. This is just to show the demo of how to use the mmdet framework for test inference. The accuracy obtained from the submission file created is only 0.027. The mmdet framework can be used to easily train and test multiple models with different backbones, so it will be easier to create ensemble of models for final evaluation.\n"},{"metadata":{},"cell_type":"markdown","source":"# Training Notebook Link FasterRCNN\n\nhttps://www.kaggle.com/gauravsingh1/mmdet-pytorch-framework-training-fasterrcnn-base"},{"metadata":{},"cell_type":"markdown","source":"# Installing MMDET Framework"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport random\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check nvcc version\n!nvcc -V\n# Check GCC version\n!gcc --version","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nprint(\"this will take around 10 mins\")\n# install dependencies: (use cu101 because colab has CUDA 10.1)\n# !pip install -U torch==1.7.0+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html\n\n# install mmcv-full thus we could use CUDA operators\n!pip install mmcv-full","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf mmdetection\n!git clone --branch v2.7.0 https://github.com/open-mmlab/mmdetection.git\n%cd mmdetection\n\n!pip install -e .\n\n# install Pillow 7.0.0 back in order to avoid bug in colab\n!pip install Pillow==7.0.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Installation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check Pytorch installation\nimport torch, torchvision\nprint(torch.__version__, torch.cuda.is_available())\n\n# Check MMDetection installation\nimport mmdet\nprint(mmdet.__version__)\n\n# Check mmcv installation\nfrom mmcv.ops import get_compiling_cuda_version, get_compiler_version\nprint(get_compiling_cuda_version())\nprint(get_compiler_version())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Downloading Sample Checkpoints"},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir checkpoints\n!wget -c http://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth \\\n      -O checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing test anno file: Coco Format"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_anno = \"../../input/vinbigdata-1024-image-dataset/vinbigdata/test\"\n\nids = os.listdir(test_anno)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_infos = []\nfor i, _id in enumerate(ids):\n    if '.png' in _id:\n        img_infos.append({\n                    \"license\": 0,\n                    \"url\": 'null',\n                    \"file_name\": _id,\n                    \"height\": 1024,\n                    \"width\": 1024,\n                    \"date_captured\": 'null',\n                    \"id\": _id\n                })","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_infos[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Over here I am just replacing the contents of validation annotation file with the list of test images. The annotation key in the dictionary will be an empty list. Rest will be same for the test ann file."},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nval_anno = '../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_annotations.json'\n\nwith open(val_anno) as f:\n    dd = json.load(f)\n\ndd.keys()\ndd['annotations']=[]\ndd['images']\ndd['images'] = img_infos\nwith open('./test_ann.json', 'w') as outfile:\n    json.dump(dd, outfile)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_ids = os.listdir('../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_images')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_classes = (\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MMDET framework gives the option to pass the extra config options as a command line arguments or one can pass the extra config in an dict variable in the function. The function can then merge this dict on the base config."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\n_cfg_options = {\"dataset_type\" : 'CocoDataset',\n\"classes\" : '''(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")''',\n\"data.train.img_prefix\" : '../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/',\n\"data.train.classes\" : '''(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")''',\n\"data.train.ann_file\" : '../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/train_annotations.json',\n\"data.train.type\":'CocoDataset',\n\"data.val.img_prefix\" : '../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/',\n\"data.val.classes\" : '''(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")''',\n\"data.val.ann_file\" : '../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_annotations.json',\n\"data.val.type\" : 'CocoDataset',\n\"data.test.img_prefix\" :  '../../input/vinbigdata-1024-image-dataset/vinbigdata/test/',\n\"data.test.classes\" :  '''(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")''',\n\"data.test.ann_file\" :  './test_ann.json',\n\"data.test.type\": 'CocoDataset',   \n\"model.roi_head.bbox_head.num_classes\" : '14',\n\"evaluation.metric\" : 'bbox',\n\"work_dir\": \"../vinbig_output\",\n\"load_from\" : './checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth',\n\"total_epochs\" :'1'\n}\n\n\ncfg_op = \"\"\nfor k, v in _cfg_options.items():\n    cfg_op+=f\"{k}='{v}' \"\nprint(cfg_op)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_cfg_options","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_cfg_options['model.roi_head.bbox_head.num_classes']=14","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Adding extra config on BaseConfig"},{"metadata":{"trusted":true},"cell_type":"code","source":"from mmdet.apis import inference_detector, init_detector, show_result_pyplot\n# Choose to use a config and initialize the detector\nconfig = 'configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.py'\n# Setup a checkpoint file to load\ncheckpoint = '../../input/vinbig-mmdet-fasterrcnn-base12-epoch/vinbig_output/epoch_10.pth'\n# initialize the detector\nmodel = init_detector(config, checkpoint, device='cuda:0', cfg_options=_cfg_options)\nmodel.CLASSES = _classes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference On Single Image"},{"metadata":{},"cell_type":"markdown","source":"Feel free to change the id variable to see more predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"_id = random.randint(1,1098)\n# Use the detector to do inference\nimg = f'../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_images/{val_ids[_id]}'\nresult = inference_detector(model, img)\nshow_result_pyplot(model, img, result, score_thr=0.3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction Sample "},{"metadata":{},"cell_type":"markdown","source":"The output of the mmdet network is in the following form:"},{"metadata":{"trusted":true},"cell_type":"code","source":"result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.makedirs(\"../vinbig_output\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the model using the python command"},{"metadata":{"trusted":true},"cell_type":"code","source":"# !python tools/train.py ./configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.py --cfg-options dataset_type='CocoDataset' classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.train.img_prefix='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/' data.train.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.train.ann_file='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/train_annotations.json' data.train.type='CocoDataset' data.val.img_prefix='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/' data.val.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.val.ann_file='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_annotations.json' data.val.type='CocoDataset' data.test.img_prefix='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/' data.test.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.test.ann_file='./test_ann.json' data.test.type='CocoDataset' model.roi_head.bbox_head.num_classes='14' evaluation.metric='bbox' work_dir='../vinbig_output' load_from='./checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth' total_epochs='1'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predicting For All the Test Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"!python tools/test.py ./configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.py ../../input/vinbig-mmdet-fasterrcnn-base12-epoch/vinbig_output/epoch_10.pth --cfg-options dataset_type='CocoDataset' classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.train.img_prefix='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/' data.train.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.train.ann_file='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/train_annotations.json' data.train.type='CocoDataset' data.val.img_prefix='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/' data.val.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.val.ann_file='../../input/vinbigdata-coco-dataset-with-wbf-3x-downscaled/vinbigdata-coco-dataset-with-wbf-3x-downscaled/val_annotations.json' data.val.type='CocoDataset' data.test.img_prefix='../../input/vinbigdata-1024-image-dataset/vinbigdata/test/' data.test.classes='(\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\", \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\", \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\", \"Pulmonary_fibrosis\")' data.test.ann_file='./test_ann.json' data.test.type='CocoDataset' model.roi_head.bbox_head.num_classes='14' evaluation.metric='bbox' work_dir='../vinbig_output' load_from='./checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth' total_epochs='1' --out results.pkl\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plotting loss curve"},{"metadata":{"trusted":true},"cell_type":"code","source":"# !python tools/analyze_logs.py plot_curve ../../input/vinbig-mmdet-fasterrcnn-base12-epoch/vinbig_output/None.log.json --keys lr --legend run1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(\"./\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Running Inference on Test Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\n\nwith open('./results.pkl', 'rb') as f:\n    data = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport json\nwith open('./test_ann.json', 'rb') as f:\n    ann = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ntest_df = pd.read_csv('../../input/vinbigdata-original-image-dataset/vinbigdata/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file_ids = [file_name.get('file_name').split('.png')[0] for file_name in ann.get('images')]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ensemble_boxes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# WBF on Test Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"from ensemble_boxes import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_with_pred = zip(file_ids, data)\nsubmission_vals = []\nfor _id, preds in ann_with_pred:\n    boxes = []\n    scores = []\n    labels = []\n    width = test_df[test_df.image_id==_id]['width'].iloc[0]\n    height = test_df[test_df.image_id==_id]['height'].iloc[0]  \n    for i, pred in enumerate(preds):\n        if len(pred):\n            for p in pred:\n                box = p[:4]/1024\n                boxes.append(box)\n                score = p[4].astype(float)\n                scores.append(score)\n                labels.append(i)\n    boxes, scores, labels = weighted_boxes_fusion([boxes], [scores], [labels], iou_thr=0.4, skip_box_thr=0.4)\n    boxes[:, 0] = boxes[:, 0]*height\n    boxes[:, 2] = boxes[:, 2]*height\n    boxes[:, 1] = boxes[:, 1]*width\n    boxes[:, 3] = boxes[:, 3]*width\n    \n    scaled_boxes = boxes.astype(int)\n    labels = labels.astype(int)\n    _id_preds = []\n    if len(scaled_boxes):\n        for i in range(len(scaled_boxes)):\n            _id_preds.append(str(labels[i]))\n            _id_preds.append(str(scores[i].round(2)))\n            _id_preds.append(str(scaled_boxes[i][0]))\n            _id_preds.append(str(scaled_boxes[i][1]))\n            _id_preds.append(str(scaled_boxes[i][2]))\n            _id_preds.append(str(scaled_boxes[i][3]))\n        pred_str = \" \".join(_id_preds)\n    else:\n        pred_str = '14 1 0 0 1 1'\n    submission_vals.append([_id, pred_str])\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating Submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(submission_vals, columns = ['image_id','PredictionString'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you have reached till here, Thanks for your time!!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}