{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n\n\n
\n\n# VinBigData 2-class classifier complete pipeline\n\nThis competition is object detection task to find a class and location of thoracic abnormalities from chest x-ray image (radiographs).
\nI explained how to train detection model in the kernel [📸VinBigData detectron2 train](https://www.kaggle.com/corochann/vinbigdata-detectron2-train).\n\nHowever, it is mentioned that training 2 class classifier to understand which is the normal image is important to get high score.\n - Kernel: [VinBigData 🌟2 Class Filter🌟](https://www.kaggle.com/awsaf49/vinbigdata-2-class-filter)\n - Discussion: [[LB0.155] baseline solution](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/208837)\n\nHere, I will introduce complete **EDA, Training (with 5-fold cross validation) and Prediction pipeline** for training 2-class classifier.\n\nYou can learn the usage of following tools to accelerate deep learning tasks in computer vision!\n - [pytorch](https://github.com/pytorch/pytorch): Deep learning framework, it's popular among researchers for its flexible usage. no need to explain detail!\n - [albumentations](https://github.com/albumentations-team/albumentations): Image augmentation library, developed by famous kagglers!\n - [timm](https://github.com/rwightman/pytorch-image-models): pytorch-image-models, it provides a lot of popular SoTA CNN models with pretrained weights.\n - [pytorch ignite](https://github.com/pytorch/ignite): Traning/Evaluation abstraction framework on top of pytorch.\n - [pytorch pfn extras](https://github.com/pfnet/pytorch-pfn-extras): It is used to add more feature-rich functionality on Ignite.\n\n\n[UPDATE 2021/2/6] I added submission procedure using 2-class classification prediction result.\n\n\n[UPDATE 2021/2/10] I added mixup augmentation and label smoothing explanation.
\nAlso, further improvement using ChExpert dataset is mentioned at \"Next step\" section
","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents\n\n** [Dataset preparation](#dataset)**
\n** [Installation](#installation)**
\n** [EDA: distribution between normal & abnormal class](#eda)**
\n** [Image visualizaion & augmentation with albumentations](#aug)**
\n** [Defining CNN models](#model)**
\n** [Training utils](#trainutil)**
\n** [Training scripts](#trainscript)**
\n** [Prediction on validation & test dataset](#prediction)**
\n** [Next step](#nextstep)**
","metadata":{}},{"cell_type":"markdown","source":"\n# Dataset preparation\n\nPreprocessing x-ray image format (dicom) into normal png image format is already done by @xhlulu in the below discussion:\n - [Multiple preprocessed datasets: 256/512/1024px, PNG and JPG, modified and original ratio](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/207955).\n\nHere I will just use the dataset [VinBigData Chest X-ray Resized PNG (256x256)](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-256x256) to skip the preprocessing and focus on modeling part. Please upvote the dataset as well!","metadata":{}},{"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\nimport torch\n\n# --- setup ---\npd.set_option('max_columns', 50)\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Installation\n\ndetectron2 is not pre-installed in this kaggle docker, so let's install it. \nWe can follow [installation instruction](https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md), we need to know CUDA and pytorch version to install correct `detectron2`.","metadata":{}},{"cell_type":"code","source":"!pip install detectron2 -f \\\n https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.7/index.html\n!pip install pytorch-pfn-extras timm","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This `Flags` class summarizes all the configuratoin available during the training.\n\nAs I will show later, you can change various hyperparameters to experiment improving your models!","metadata":{}},{"cell_type":"code","source":"from typing import Any\nimport yaml\n\ndef save_yaml(filepath: str, content: Any, width: int = 120):\n with open(filepath, \"w\") as f:\n yaml.dump(content, f, width=width)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from dataclasses import dataclass, field\nfrom typing import Dict, Any, Tuple, Union, List\n\n\n@dataclass\nclass Flags:\n # General\n debug: bool = True\n outdir: str = \"results/det\"\n device: str = \"cuda:0\"\n\n # Data config\n imgdir_name: str = \"vinbigdata-chest-xray-resized-png-256x256\"\n # split_mode: str = \"all_train\" # all_train or valid20\n seed: int = 111\n target_fold: int = 0 # 0~4\n label_smoothing: float = 0.0\n # Model config\n model_name: str = \"resnet18\"\n model_mode: str = \"normal\" # normal, cnn_fixed supported\n # Training config\n epoch: int = 20\n batchsize: int = 8\n valid_batchsize: int = 16\n num_workers: int = 4\n snapshot_freq: int = 5\n ema_decay: float = 0.999 # negative value is to inactivate ema.\n scheduler_type: str = \"\"\n scheduler_kwargs: Dict[str, Any] = field(default_factory=lambda: {})\n scheduler_trigger: List[Union[int, str]] = field(default_factory=lambda: [1, \"iteration\"])\n aug_kwargs: Dict[str, Dict[str, Any]] = field(default_factory=lambda: {})\n mixup_prob: float = -1.0 # Apply mixup augmentation when positive value is set.\n\n def update(self, param_dict: Dict) -> \"Flags\":\n # Overwrite by `param_dict`\n for key, value in param_dict.items():\n if not hasattr(self, key):\n raise ValueError(f\"[ERROR] Unexpected key for flag = {key}\")\n setattr(self, key, value)\n return self","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"flags_dict = {\n \"debug\": True, # Change to True for fast debug run!\n \"outdir\": \"results/tmp_debug\",\n # Data\n \"imgdir_name\": \"vinbigdata-chest-xray-resized-png-256x256\",\n # Model\n \"model_name\": \"resnet18\",\n # Training\n \"num_workers\": 4,\n \"epoch\": 15,\n \"batchsize\": 8,\n \"scheduler_type\": \"CosineAnnealingWarmRestarts\",\n \"scheduler_kwargs\": {\"T_0\": 28125}, # 15000 * 15 epoch // (batchsize=8)\n \"scheduler_trigger\": [1, \"iteration\"],\n \"aug_kwargs\": {\n \"HorizontalFlip\": {\"p\": 0.5},\n \"ShiftScaleRotate\": {\"scale_limit\": 0.15, \"rotate_limit\": 10, \"p\": 0.5},\n \"RandomBrightnessContrast\": {\"p\": 0.5},\n \"CoarseDropout\": {\"max_holes\": 8, \"max_height\": 25, \"max_width\": 25, \"p\": 0.5},\n \"Blur\": {\"blur_limit\": [3, 7], \"p\": 0.5},\n \"Downscale\": {\"scale_min\": 0.25, \"scale_max\": 0.9, \"p\": 0.3},\n \"RandomGamma\": {\"gamma_limit\": [80, 120], \"p\": 0.6},\n }\n}","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import dataclasses\n\n# args = parse()\nprint(\"torch\", torch.__version__)\nflags = Flags().update(flags_dict)\nprint(\"flags\", flags)\ndebug = flags.debug\noutdir = Path(flags.outdir)\nos.makedirs(str(outdir), exist_ok=True)\nflags_dict = dataclasses.asdict(flags)\nsave_yaml(str(outdir / \"flags.yaml\"), flags_dict)\n\n# --- Read data ---\ninputdir = Path(\"/kaggle/input\")\ndatadir = inputdir / \"vinbigdata-chest-xray-abnormalities-detection\"\nimgdir = inputdir / flags.imgdir_name\n\n# Read in the data CSV files\ntrain = pd.read_csv(datadir / \"train.csv\")\n# sample_submission = pd.read_csv(datadir / 'sample_submission.csv')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# EDA: distribution between normal & abnormal class\n\nAt first, let's check how many normal class exist in the training data.\nIt is classified as \"class_name = No finding\" and \"class_id = 14\".\n\nHowever you need to be careful that 3 radiologists annotated for each image, so you can find 3 annotations as you can see below.","metadata":{}},{"cell_type":"code","source":"train.query(\"image_id == '50a418190bc3fb1ef1633bf9678929b3'\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So the question arises, is there an image that the 3 radiologists' opinions differ?\n\nLet's check number of \"No finding\" annotations for each image, if the opinions are in complete agreement the number of \"No finding\" annotations should be **0 -> Abnormal(all radiologists does not think this is normal)\" or \"1 -> Normal(all radiologists think this is normal)\"**.","metadata":{}},{"cell_type":"code","source":"is_normal_df = train.groupby(\"image_id\")[\"class_id\"].agg(lambda s: (s == 14).sum()).reset_index().rename({\"class_id\": \"num_normal_annotations\"}, axis=1)\nis_normal_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We could confirm that **always 3 radiologists opinions match** for normal - abnormal diagnosis.\n\n[Note] I noticed that it does not apply for the other classes. i.e., 3 radiologists opinions sometimes do not match for the other class of thoracic abnormalities.","metadata":{}},{"cell_type":"code","source":"num_normal_anno_counts = is_normal_df[\"num_normal_annotations\"].value_counts()\nnum_normal_anno_counts.plot(kind=\"bar\")\nplt.title(\"The number of 'No finding' annotations in each image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_normal_anno_counts_df = num_normal_anno_counts.reset_index()\nnum_normal_anno_counts_df[\"name\"] = num_normal_anno_counts_df[\"index\"].map({0: \"Abnormal\", 3: \"Normal\"})\nnum_normal_anno_counts_df","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So almost 70% of the data is actually \"Normal\" X-ray images.\n\nOnly 30% of the images need thoracic abnormality location detection.","metadata":{}},{"cell_type":"code","source":"px.pie(num_normal_anno_counts_df, values=\"num_normal_annotations\", names=\"name\", title=\"Normal/Abnormal ratio\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Image visualizaion & augmentation with albumentations\n\nWhen you train CNN models, image augmentation is important to avoid model to overfit.
\nI'll show examples to use Albumentations to run image augmentation very easily.
\nAt first, I will define pytorch Dataset class for this competition, which can be also used later in the training.","metadata":{}},{"cell_type":"code","source":"import pickle\nfrom pathlib import Path\nfrom typing import Optional\n\nimport cv2\nimport numpy as np\nimport pandas as pd\nfrom detectron2.structures import BoxMode\nfrom tqdm import tqdm\n\n\ndef get_vinbigdata_dicts(\n imgdir: Path,\n train_df: pd.DataFrame,\n train_data_type: str = \"original\",\n use_cache: bool = True,\n debug: bool = True,\n target_indices: Optional[np.ndarray] = None,\n):\n debug_str = f\"_debug{int(debug)}\"\n train_data_type_str = f\"_{train_data_type}\"\n cache_path = Path(\".\") / f\"dataset_dicts_cache{train_data_type_str}{debug_str}.pkl\"\n if not use_cache or not cache_path.exists():\n print(\"Creating data...\")\n train_meta = pd.read_csv(imgdir / \"train_meta.csv\")\n if debug:\n train_meta = train_meta.iloc[:500] # For debug....\n\n # Load 1 image to get image size.\n image_id = train_meta.loc[0, \"image_id\"]\n image_path = str(imgdir / \"train\" / f\"{image_id}.png\")\n image = cv2.imread(image_path)\n resized_height, resized_width, ch = image.shape\n print(f\"image shape: {image.shape}\")\n\n dataset_dicts = []\n for index, train_meta_row in tqdm(train_meta.iterrows(), total=len(train_meta)):\n record = {}\n\n image_id, height, width = train_meta_row.values\n filename = str(imgdir / \"train\" / f\"{image_id}.png\")\n record[\"file_name\"] = filename\n record[\"image_id\"] = image_id\n record[\"height\"] = resized_height\n record[\"width\"] = resized_width\n objs = []\n for index2, row in train_df.query(\"image_id == @image_id\").iterrows():\n # print(row)\n # print(row[\"class_name\"])\n # class_name = row[\"class_name\"]\n class_id = row[\"class_id\"]\n if class_id == 14:\n # It is \"No finding\"\n # This annotator does not find anything, skip.\n pass\n else:\n # bbox_original = [int(row[\"x_min\"]), int(row[\"y_min\"]), int(row[\"x_max\"]), int(row[\"y_max\"])]\n h_ratio = resized_height / height\n w_ratio = resized_width / width\n bbox_resized = [\n int(row[\"x_min\"]) * w_ratio,\n int(row[\"y_min\"]) * h_ratio,\n int(row[\"x_max\"]) * w_ratio,\n int(row[\"y_max\"]) * h_ratio,\n ]\n obj = {\n \"bbox\": bbox_resized,\n \"bbox_mode\": BoxMode.XYXY_ABS,\n \"category_id\": class_id,\n }\n objs.append(obj)\n record[\"annotations\"] = objs\n dataset_dicts.append(record)\n with open(cache_path, mode=\"wb\") as f:\n pickle.dump(dataset_dicts, f)\n\n print(f\"Load from cache {cache_path}\")\n with open(cache_path, mode=\"rb\") as f:\n dataset_dicts = pickle.load(f)\n if target_indices is not None:\n dataset_dicts = [dataset_dicts[i] for i in target_indices]\n return dataset_dicts\n\n\ndef get_vinbigdata_dicts_test(\n imgdir: Path, test_meta: pd.DataFrame, use_cache: bool = True, debug: bool = True,\n):\n debug_str = f\"_debug{int(debug)}\"\n cache_path = Path(\".\") / f\"dataset_dicts_cache_test{debug_str}.pkl\"\n if not use_cache or not cache_path.exists():\n print(\"Creating data...\")\n # test_meta = pd.read_csv(imgdir / \"test_meta.csv\")\n if debug:\n test_meta = test_meta.iloc[:500] # For debug....\n\n # Load 1 image to get image size.\n image_id = test_meta.loc[0, \"image_id\"]\n image_path = str(imgdir / \"test\" / f\"{image_id}.png\")\n image = cv2.imread(image_path)\n resized_height, resized_width, ch = image.shape\n print(f\"image shape: {image.shape}\")\n\n dataset_dicts = []\n for index, test_meta_row in tqdm(test_meta.iterrows(), total=len(test_meta)):\n record = {}\n\n image_id, height, width = test_meta_row.values\n filename = str(imgdir / \"test\" / f\"{image_id}.png\")\n record[\"file_name\"] = filename\n # record[\"image_id\"] = index\n record[\"image_id\"] = image_id\n record[\"height\"] = resized_height\n record[\"width\"] = resized_width\n # objs = []\n # record[\"annotations\"] = objs\n dataset_dicts.append(record)\n with open(cache_path, mode=\"wb\") as f:\n pickle.dump(dataset_dicts, f)\n\n print(f\"Load from cache {cache_path}\")\n with open(cache_path, mode=\"rb\") as f:\n dataset_dicts = pickle.load(f)\n return dataset_dicts\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nReferenced `chainer.dataset.DatasetMixin` to work with pytorch Dataset.\n\"\"\"\nimport numpy\nimport six\nimport torch\nfrom torch.utils.data.dataset import Dataset\n\n\nclass DatasetMixin(Dataset):\n\n def __init__(self, transform=None):\n self.transform = transform\n\n def __getitem__(self, index):\n \"\"\"Returns an example or a sequence of examples.\"\"\"\n if torch.is_tensor(index):\n index = index.tolist()\n if isinstance(index, slice):\n current, stop, step = index.indices(len(self))\n return [self.get_example_wrapper(i) for i in\n six.moves.range(current, stop, step)]\n elif isinstance(index, list) or isinstance(index, numpy.ndarray):\n return [self.get_example_wrapper(i) for i in index]\n else:\n return self.get_example_wrapper(index)\n\n def __len__(self):\n \"\"\"Returns the number of data points.\"\"\"\n raise NotImplementedError\n\n def get_example_wrapper(self, i):\n \"\"\"Wrapper of `get_example`, to apply `transform` if necessary\"\"\"\n example = self.get_example(i)\n if self.transform:\n example = self.transform(example)\n return example\n\n def get_example(self, i):\n \"\"\"Returns the i-th example.\n\n Implementations should override it. It should raise :class:`IndexError`\n if the index is invalid.\n\n Args:\n i (int): The index of the example.\n\n Returns:\n The i-th example.\n\n \"\"\"\n raise NotImplementedError\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[Update] \nHere I mixup augmentation in the dataset. It makes interpolation of 2 images, with the label is also modified according to the mix ratio. \nmixup augmentation is expecially useful when the number of data is limited.\nBecause it can make **combination of any 2 images**, instead of just using 1 image.\n\nI also added label smoothing feature. Sometimes it is difficult to learn label is 0, 1. \nLabel smoothing changes its label 0 -> 0.01 & 1 -> 0.99. By smoothing the label the loss surface becomes more \"soft\", and sometimes model can learn well.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport numpy as np\n\n\nclass VinbigdataTwoClassDataset(DatasetMixin):\n def __init__(self, dataset_dicts, image_transform=None, transform=None, train: bool = True,\n mixup_prob: float = -1.0, label_smoothing: float = 0.0):\n super(VinbigdataTwoClassDataset, self).__init__(transform=transform)\n self.dataset_dicts = dataset_dicts\n self.image_transform = image_transform\n self.train = train\n self.mixup_prob = mixup_prob\n self.label_smoothing = label_smoothing\n\n def _get_single_example(self, i):\n d = self.dataset_dicts[i]\n filename = d[\"file_name\"]\n\n img = cv2.imread(filename)\n if self.image_transform:\n img = self.image_transform(img)\n img = torch.tensor(np.transpose(img, (2, 0, 1)).astype(np.float32))\n\n if self.train:\n label = int(len(d[\"annotations\"]) > 0) # 0 normal, 1 abnormal\n if self.label_smoothing > 0:\n if label == 0:\n return img, float(label) + self.label_smoothing\n else:\n return img, float(label) - self.label_smoothing\n else:\n return img, float(label)\n else:\n # Only return img\n return img, None\n\n def get_example(self, i):\n img, label = self._get_single_example(i)\n if self.mixup_prob > 0. and np.random.uniform() < self.mixup_prob:\n j = np.random.randint(0, len(self.dataset_dicts))\n p = np.random.uniform()\n img2, label2 = self._get_single_example(j)\n img = img * p + img2 * (1 - p)\n if self.train:\n label = label * p + label2 * (1 - p)\n\n if self.train:\n label_logit = torch.tensor([1 - label, label], dtype=torch.float32)\n return img, label_logit\n else:\n # Only return img\n return img\n\n def __len__(self):\n return len(self.dataset_dicts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now creating the dataset is just easy as following:","metadata":{}},{"cell_type":"code","source":"dataset_dicts = get_vinbigdata_dicts(imgdir, train, debug=debug)\ndataset = VinbigdataTwoClassDataset(dataset_dicts)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can access each image and its label (0=Normal, 1=Abnormal) by just access `dataset` with index.","metadata":{}},{"cell_type":"code","source":"index = 0\nimg, label = dataset[index]\nplt.imshow(img.cpu().numpy().transpose((1, 2, 0)) / 255.)\nplt.title(f\"{index}-th image: label {label}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To run augmentation on this image, I will define `Transform` class which is applied each time the data is accessed.\n\nYou can refer [albumentations](https://github.com/albumentations-team/albumentations) page, that various kinds of augmentation is already implemented and can be used very easily!","metadata":{}},{"cell_type":"code","source":"import albumentations as A\n\n\nclass Transform:\n def __init__(\n self, hflip_prob: float = 0.5, ssr_prob: float = 0.5, random_bc_prob: float = 0.5\n ):\n self.transform = A.Compose(\n [\n A.HorizontalFlip(p=hflip_prob),\n A.ShiftScaleRotate(\n shift_limit=0.0625, scale_limit=0.1, rotate_limit=10, p=ssr_prob\n ),\n A.RandomBrightnessContrast(p=random_bc_prob),\n ]\n )\n\n def __call__(self, image):\n image = self.transform(image=image)[\"image\"]\n return image\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To use augmentation, you can just define dataset with the `Transform` function.","metadata":{}},{"cell_type":"code","source":"aug_dataset = VinbigdataTwoClassDataset(dataset_dicts, image_transform=Transform())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize, looks good.
\nYou can see each image looks different (rotated, brightness is different etc...) even if it is generated from the same image :)","metadata":{}},{"cell_type":"code","source":"index = 0\n\nn_images = 4\n\nfig, axes = plt.subplots(1, n_images, figsize=(16, 5))\nfor i in range(n_images):\n # Each time the data is accessed, the result is different due to random augmentation!\n img, label = aug_dataset[index]\n ax = axes[i]\n ax.imshow(img.cpu().numpy().transpose((1, 2, 0)) / 255.)\n ax.set_title(f\"{index}-th image: label {label}\")\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Extend to more general form**\n\nAugmentation is very important hyperparameter to improve model's performance, and you want to experiment with various configurations.
\nBelow updated `Transform` function is written to **support all the augmentations implemented in albumentations**.
\nYou can specify `aug_kwargs` from external configuration inside `flag` as follows:\n\n```\naug_kwargs:\n HorizontalFlip: {\"p\": 0.5}\n ShiftScaleRotate: {\"scale_limit\": 0.15, \"rotate_limit\": 10, \"p\": 0.5}\n RandomBrightnessContrast: {\"p\": 0.5}\n CoarseDropout: {\"max_holes\": 8, \"max_height\": 25, \"max_width\": 25, \"p\": 0.5}\n Blur: {\"blur_limit\": [3, 7], \"p\": 0.5}\n Downscale: {\"scale_min\": 0.25, \"scale_max\": 0.9, \"p\": 0.3}\n RandomGamma: {\"gamma_limit\": [80, 120], \"p\": 0.6}\n```","metadata":{}},{"cell_type":"code","source":"from typing import Dict\n\nimport albumentations as A\n\n\nclass Transform:\n def __init__(self, aug_kwargs: Dict):\n self.transform = A.Compose(\n [getattr(A, name)(**kwargs) for name, kwargs in aug_kwargs.items()]\n )\n\n def __call__(self, image):\n image = self.transform(image=image)[\"image\"]\n return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Defining CNN models\n\nRecently, several libraries of CNN-collection are available on public.\n\nI will use `timm` this time. You don't need to impelment deep CNN models by yourself, you can just re-use latest research results without hustle.
\nYou can focus on more about looking data and try experiment now.","metadata":{}},{"cell_type":"code","source":"from torch import nn\nfrom torch.nn import Linear\n\n\nclass CNNFixedPredictor(nn.Module):\n def __init__(self, cnn: nn.Module, num_classes: int = 2):\n super(CNNFixedPredictor, self).__init__()\n self.cnn = cnn\n self.lin = Linear(cnn.num_features, num_classes)\n print(\"cnn.num_features\", cnn.num_features)\n\n # We do not learn CNN parameters.\n # https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html\n for param in self.cnn.parameters():\n param.requires_grad = False\n\n def forward(self, x):\n feat = self.cnn(x)\n return self.lin(feat)\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import timm\n\n\ndef build_predictor(model_name: str, model_mode: str = \"normal\"):\n if model_mode == \"normal\":\n # normal configuration. train all parameters.\n return timm.create_model(model_name, pretrained=True, num_classes=2, in_chans=3)\n elif model_mode == \"cnn_fixed\":\n # normal configuration. train all parameters.\n # https://rwightman.github.io/pytorch-image-models/feature_extraction/\n timm_model = timm.create_model(model_name, pretrained=True, num_classes=0, in_chans=3)\n return CNNFixedPredictor(timm_model, num_classes=2)\n else:\n raise ValueError(f\"[ERROR] Unexpected value model_mode={model_mode}\")\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\n\n\ndef accuracy(y: torch.Tensor, t: torch.Tensor) -> torch.Tensor:\n \"\"\"Computes multi-class classification accuracy\"\"\"\n assert y.shape[:-1] == t.shape, f\"y {y.shape}, t {t.shape} is inconsistent.\"\n pred_label = torch.max(y.detach(), dim=-1)[1]\n count = t.nelement()\n correct = (pred_label == t).sum().float()\n acc = correct / count\n return acc\n\n\ndef accuracy_with_logits(y: torch.Tensor, t: torch.Tensor) -> torch.Tensor:\n \"\"\"Computes multi-class classification accuracy\"\"\"\n assert y.shape == t.shape\n gt_label = torch.max(t.detach(), dim=-1)[1]\n return accuracy(y, gt_label)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn.functional as F\n\n\ndef cross_entropy_with_logits(input, target, dim=-1):\n loss = torch.sum(- target * F.log_softmax(input, dim), dim)\n return loss.mean()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn.functional as F\nfrom torch import nn\nimport pytorch_pfn_extras as ppe\n\n\nclass Classifier(nn.Module):\n \"\"\"two class classfication\"\"\"\n\n def __init__(self, predictor, lossfun=cross_entropy_with_logits):\n super().__init__()\n self.predictor = predictor\n self.lossfun = lossfun\n self.prefix = \"\"\n\n def forward(self, image, targets):\n outputs = self.predictor(image)\n loss = self.lossfun(outputs, targets)\n metrics = {\n f\"{self.prefix}loss\": loss.item(),\n f\"{self.prefix}acc\": accuracy_with_logits(outputs, targets).item()\n }\n ppe.reporting.report(metrics, self)\n return loss, metrics\n\n def predict(self, data_loader):\n pred = self.predict_proba(data_loader)\n label = torch.argmax(pred, dim=1)\n return label\n\n def predict_proba(self, data_loader):\n device: torch.device = next(self.parameters()).device\n y_list = []\n self.eval()\n with torch.no_grad():\n for batch in data_loader:\n if isinstance(batch, (tuple, list)):\n # Assumes first argument is \"image\"\n batch = batch[0].to(device)\n else:\n batch = batch.to(device)\n y = self.predictor(batch)\n y = torch.softmax(y, dim=-1)\n y_list.append(y)\n pred = torch.cat(y_list)\n return pred\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What kind of models are supported in the `timm` library?","metadata":{}},{"cell_type":"code","source":"supported_models = timm.list_models()\nprint(f\"{len(supported_models)} models are supported in timm.\")\nprint(supported_models)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow more than 300 models are supported!
\nIt of course includes **resnet** related models, **efficientnet**, etc.
\nYou may wonder which model should be used?
\nI will go with `resnet18` as a baseline at first, and try using more deeper/latest models in the experiment.","metadata":{}},{"cell_type":"markdown","source":"\n# Training utils\n\nHere are training util methods. You can just copy these to use in other projects.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nFrom https://github.com/pfnet-research/kaggle-lyft-motion-prediction-4th-place-solution\n\"\"\"\nfrom logging import getLogger\n\nfrom torch import nn\n\n\nclass EMA(object):\n \"\"\"Exponential moving average of model parameters.\n\n Ref\n - https://github.com/tensorflow/addons/blob/v0.10.0/tensorflow_addons/optimizers/moving_average.py#L26-L103\n - https://anmoljoshi.com/Pytorch-Dicussions/\n\n Args:\n model (nn.Module): Model with parameters whose EMA will be kept.\n decay (float): Decay rate for exponential moving average.\n strict (bool): Apply strict check for `assign` & `resume`.\n use_dynamic_decay (bool): Dynamically change decay rate. If `True`, small decay rate is\n used at the beginning of training to move moving average faster.\n \"\"\" # NOQA\n\n def __init__(\n self,\n model: nn.Module,\n decay: float,\n strict: bool = True,\n use_dynamic_decay: bool = True,\n ):\n self.decay = decay\n self.model = model\n self.strict = strict\n self.use_dynamic_decay = use_dynamic_decay\n self.logger = getLogger(__name__)\n self.n_step = 0\n\n self.shadow = {}\n self.original = {}\n\n # Flag to manage which parameter is assigned.\n # When `False`, original model's parameter is used.\n # When `True` (`assign` method is called), `shadow` parameter (ema param) is used.\n self._assigned = False\n\n # Register model parameters\n for name, param in model.named_parameters():\n if param.requires_grad:\n self.shadow[name] = param.data.clone()\n\n def step(self):\n self.n_step += 1\n if self.use_dynamic_decay:\n _n_step = float(self.n_step)\n decay = min(self.decay, (1.0 + _n_step) / (10.0 + _n_step))\n else:\n decay = self.decay\n\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.shadow\n new_average = (1.0 - decay) * param.data + decay * self.shadow[name]\n self.shadow[name] = new_average.clone()\n\n # alias\n __call__ = step\n\n def assign(self):\n \"\"\"Assign exponential moving average of parameter values to the respective parameters.\"\"\"\n if self._assigned:\n if self.strict:\n raise ValueError(\"[ERROR] `assign` is called again before `resume`.\")\n else:\n self.logger.warning(\n \"`assign` is called again before `resume`.\"\n \"shadow parameter is already assigned, skip.\"\n )\n return\n\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.shadow\n self.original[name] = param.data.clone()\n param.data = self.shadow[name]\n self._assigned = True\n\n def resume(self):\n \"\"\"Restore original parameters to a model.\n\n That is, put back the values that were in each parameter at the last call to `assign`.\n \"\"\"\n if not self._assigned:\n if self.strict:\n raise ValueError(\"[ERROR] `resume` is called before `assign`.\")\n else:\n self.logger.warning(\"`resume` is called before `assign`, skip.\")\n return\n\n for name, param in self.model.named_parameters():\n if param.requires_grad:\n assert name in self.shadow\n param.data = self.original[name]\n self._assigned = False\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nFrom https://github.com/pfnet-research/kaggle-lyft-motion-prediction-4th-place-solution\n\"\"\"\nfrom typing import Mapping, Any\n\nfrom torch import optim\n\nfrom pytorch_pfn_extras.training.extension import Extension, PRIORITY_READER\nfrom pytorch_pfn_extras.training.manager import ExtensionsManager\n\n\nclass LRScheduler(Extension):\n \"\"\"A thin wrapper to resume the lr_scheduler\"\"\"\n\n trigger = 1, 'iteration'\n priority = PRIORITY_READER\n name = None\n\n def __init__(self, optimizer: optim.Optimizer, scheduler_type: str, scheduler_kwargs: Mapping[str, Any]) -> None:\n super().__init__()\n self.scheduler = getattr(optim.lr_scheduler, scheduler_type)(optimizer, **scheduler_kwargs)\n\n def __call__(self, manager: ExtensionsManager) -> None:\n self.scheduler.step()\n\n def state_dict(self) -> None:\n return self.scheduler.state_dict()\n\n def load_state_dict(self, to_load) -> None:\n self.scheduler.load_state_dict(to_load)\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from ignite.engine import Engine\n\n\ndef create_trainer(model, optimizer, device) -> Engine:\n model.to(device)\n\n def update_fn(engine, batch):\n model.train()\n optimizer.zero_grad()\n loss, metrics = model(*[elem.to(device) for elem in batch])\n loss.backward()\n optimizer.step()\n return metrics\n trainer = Engine(update_fn)\n return trainer\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Training scripts","metadata":{}},{"cell_type":"code","source":"import dataclasses\nimport os\nimport sys\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport pytorch_pfn_extras.training.extensions as E\nimport torch\nfrom ignite.engine import Events\nfrom pytorch_pfn_extras.training import IgniteExtensionsManager\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch import nn, optim\nfrom torch.utils.data.dataloader import DataLoader","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing data by 5-fold cross validation\n\nWhen we have few data, running stable evaluation is very important. \nWe can use cross validation to reduce validation error standard deviation.\n\nHere, I will use **[`StratifiedKFold`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html)** to keep the balance between normal/abnormal ratio same for the train & validation dataset.\n\nAccording to [this discussion](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/208837#1139712), using multi label stratified kfold https://github.com/trent-b/iterative-stratification may be more stable.","metadata":{}},{"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=flags.seed)\n# skf.get_n_splits(None, None)\ny = np.array([int(len(d[\"annotations\"]) > 0) for d in dataset_dicts])\nsplit_inds = list(skf.split(dataset_dicts, y))\ntrain_inds, valid_inds = split_inds[flags.target_fold] # 0th fold\ntrain_dataset = VinbigdataTwoClassDataset(\n [dataset_dicts[i] for i in train_inds],\n image_transform=Transform(flags.aug_kwargs),\n mixup_prob=flags.mixup_prob,\n label_smoothing=flags.label_smoothing,\n)\nvalid_dataset = VinbigdataTwoClassDataset([dataset_dicts[i] for i in valid_inds])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Write training code\n\npytorch-ignite & pytorch-pfn-extras are used here.\n\n - [pytorch/ignite](https://github.com/pytorch/ignite): It provides abstraction for writing training loop.\n - [pfnet/pytorch-pfn-extras](https://github.com/pfnet/pytorch-pfn-extras): It provides several \"extensions\" useful for training. Useful for **logging, printing, evaluating, saving the model, scheduling the learning rate** during training.\n \n**[Note] Why training abstraction library is used?**\n\nYou may feel understanding training abstraction code below is a bit unintuitive compared to writing \"raw\" training loop.
\nThe advantage of abstracting the code is that we can re-use implemented handler class for other training, other competition.
\nYou don't need to write code for saving models, logging training loss/metric, show progressbar etc.\nThese are done by provided util classes in `pytorch-pfn-extras` library!\n\nYou may refer my other kernel in previous competition too:\n - [Bengali: SEResNeXt training with pytorch](https://www.kaggle.com/corochann/bengali-seresnext-training-with-pytorch)\n - [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)","metadata":{}},{"cell_type":"code","source":"train_loader = DataLoader(\n train_dataset,\n batch_size=flags.batchsize,\n num_workers=flags.num_workers,\n shuffle=True,\n pin_memory=True,\n)\nvalid_loader = DataLoader(\n valid_dataset,\n batch_size=flags.valid_batchsize,\n num_workers=flags.num_workers,\n shuffle=False,\n pin_memory=True,\n)\n\ndevice = torch.device(flags.device)\n\npredictor = build_predictor(model_name=flags.model_name, model_mode=flags.model_mode)\nclassifier = Classifier(predictor)\nmodel = classifier\n# optimizer = optim.Adam(model.parameters(), lr=1e-3)\noptimizer = optim.Adam([param for param in model.parameters() if param.requires_grad], lr=1e-3)\n\n# Train setup\ntrainer = create_trainer(model, optimizer, device)\n\nema = EMA(predictor, decay=flags.ema_decay)\n\ndef eval_func(*batch):\n loss, metrics = model(*[elem.to(device) for elem in batch])\n # HACKING: report ema value with prefix.\n if flags.ema_decay > 0:\n classifier.prefix = \"ema_\"\n ema.assign()\n loss, metrics = model(*[elem.to(device) for elem in batch])\n ema.resume()\n classifier.prefix = \"\"\n\nvalid_evaluator = E.Evaluator(\n valid_loader, model, progress_bar=False, eval_func=eval_func, device=device\n)\n\n# log_trigger = (10 if debug else 1000, \"iteration\")\nlog_trigger = (1, \"epoch\")\nlog_report = E.LogReport(trigger=log_trigger)\nextensions = [\n log_report,\n E.ProgressBarNotebook(update_interval=10 if debug else 100), # Show progress bar during training\n E.PrintReportNotebook(), # Show \"log\" on jupyter notebook \n # E.ProgressBar(update_interval=10 if debug else 100), # Show progress bar during training\n # E.PrintReport(), # Print \"log\" to terminal\n E.FailOnNonNumber(), # Stop training when nan is detected.\n]\nepoch = flags.epoch\nmodels = {\"main\": model}\noptimizers = {\"main\": optimizer}\nmanager = IgniteExtensionsManager(\n trainer, models, optimizers, epoch, extensions=extensions, out_dir=str(outdir),\n)\n# Run evaluation for valid dataset in each epoch.\nmanager.extend(valid_evaluator)\n\n# Save predictor.pt every epoch\nmanager.extend(\n E.snapshot_object(predictor, \"predictor.pt\"), trigger=(flags.snapshot_freq, \"epoch\")\n)\n# Check & Save best validation predictor.pt every epoch\n# manager.extend(E.snapshot_object(predictor, \"best_predictor.pt\"),\n# trigger=MinValueTrigger(\"validation/module/nll\",\n# trigger=(flags.snapshot_freq, \"iteration\")))\n\n# --- lr scheduler ---\nif flags.scheduler_type != \"\":\n scheduler_type = flags.scheduler_type\n print(f\"using {scheduler_type} scheduler with kwargs {flags.scheduler_kwargs}\")\n manager.extend(\n LRScheduler(optimizer, scheduler_type, flags.scheduler_kwargs),\n trigger=flags.scheduler_trigger,\n )\n\nmanager.extend(E.observe_lr(optimizer=optimizer), trigger=log_trigger)\n\nif flags.ema_decay > 0:\n # Exponential moving average\n manager.extend(lambda manager: ema(), trigger=(1, \"iteration\"))\n\n def save_ema_model(manager):\n ema.assign()\n torch.save(predictor.state_dict(), outdir / \"predictor_ema.pt\")\n ema.resume()\n\n manager.extend(save_ema_model, trigger=(flags.snapshot_freq, \"epoch\"))\n\n_ = trainer.run(train_loader, max_epochs=epoch)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So what is happening in above training abstraction? Let's understand what each extension did.\n\n**Extensions** - Each role:\n - **`ProgressBar` (`ProgressBarNotebook`)**: Shows training progress in formatted style.\n - **`LogReport`**: Logging metrics reported by `ppe.reporter.report` (see `LyftMultiRegressor` for reporting point) method and save to **log** file. It automatically collects reported value in each iteration and saves the \"mean\" of reported value for regular frequency (for example every 1 epoch).\n - **`PrintReport` (`PrintReportNotebook`)**: Prints the value which `LogReport` collected in formatted style.\n - **`Evaluator`**: Evaluate on validation dataset.\n - **`snapshot_object`**: Saves the object. Here the `model` is saved in regular interval `flags.snapshot_freq`. Even you quit training using Ctrl+C without finishing all the epoch, the intermediate trained model is saved and you can use it for inference.\n - **`LRScheduler`**: You can insert learning rate scheduling with this extension, together with the regular interval call specified by `trigger`. Here cosine annealing is applied (configured by Flags) by calling `scheduler.step()` every iteration.\n - **`observe_lr`**: `LogReport` will check optimizer's learning rate using this extension. So you can follow how the learning rate changed through the training.\n\n\nSuch many functionalities can be \"added\" easily using extensions!","metadata":{}},{"cell_type":"markdown","source":"Also **Exponential Moving Average of model weights** is calculated by `EMA` class during training, together with showing its validation loss. We can usually obtrain more stable models with EMA.","metadata":{}},{"cell_type":"markdown","source":"You can obtrain training history results really easily by just accessing `LogReport` class, which is useful for managing a lot of experiments during kaggle competitions.","metadata":{}},{"cell_type":"code","source":"torch.save(predictor.state_dict(), outdir / \"predictor_last.pt\")\ndf = log_report.to_dataframe()\ndf.to_csv(outdir / \"log.csv\", index=False)\ndf","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Prediction on validation & test dataset","metadata":{}},{"cell_type":"code","source":"# --- Prediction ---\nprint(\"Training done! Start prediction...\")\n# valid data\nvalid_pred = classifier.predict_proba(valid_loader).cpu().numpy()\nvalid_pred_df = pd.DataFrame({\n \"image_id\": [dataset_dicts[i][\"image_id\"] for i in valid_inds],\n \"class0\": valid_pred[:, 0],\n \"class1\": valid_pred[:, 1]\n})\nvalid_pred_df.to_csv(outdir/\"valid_pred.csv\", index=False)\n\n# test data\ntest_meta = pd.read_csv(inputdir / \"vinbigdata-testmeta\" / \"test_meta.csv\")\ndataset_dicts_test = get_vinbigdata_dicts_test(imgdir, test_meta, debug=False)\ntest_dataset = VinbigdataTwoClassDataset(dataset_dicts_test, train=False)\ntest_loader = DataLoader(\n test_dataset,\n batch_size=flags.valid_batchsize,\n num_workers=flags.num_workers,\n shuffle=False,\n pin_memory=True,\n)\ntest_pred = classifier.predict_proba(test_loader).cpu().numpy()\ntest_pred_df = pd.DataFrame({\n \"image_id\": [d[\"image_id\"] for d in dataset_dicts_test],\n \"class0\": test_pred[:, 0],\n \"class1\": test_pred[:, 1]\n})\ntest_pred_df.to_csv(outdir/\"test_pred.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# --- Test dataset prediction result ---\ntest_pred_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.distplot(valid_pred_df[\"class0\"].values, color='green', label='valid pred')\nsns.distplot(test_pred_df[\"class0\"].values, color='orange', label='test pred')\nplt.title(\"Prediction results histogram\")\nplt.xlim([0., 1.])\nplt.legend()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Apply 2 class filter on detection prediction\n\nI will use detection prediction from the kernel:\n - [📸VinBigData detectron2 train](https://www.kaggle.com/corochann/vinbigdata-detectron2-train)\n - [📸VinBigData detectron2 prediction](https://www.kaggle.com/corochann/vinbigdata-detectron2-prediction)\n\nAnd 2class prediction is updated as dataset: [vinbigdata-2class-pred](https://www.kaggle.com/corochann/vinbigdata2classpred).\n\nAs mentioned in [VinBigData 🌟2 Class Filter🌟](https://www.kaggle.com/awsaf49/vinbigdata-2-class-filter) by @awsaf49, applying 2-class filter improves LB score significantly. (Please upvote his kernel as well!)
\nAlso, it is mentioned that we can submit **14 prob 1 1 0 0** where the `prob` is the normal probability in the discussion [[Scoring bug] Improve your LB score by 0.053, just adding \"14 1 0 0 1 1\"](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/211971)!\n\nHere, I will propose new post processing (similar to [this](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/211971#1157809) by @pestipeti):\n\nHere `p` is the **normal probability**.\n\n1. `p < low_threshold` -> Do nothing, Keep det prediction.\n2. `low_threshold <= p < high_threshold` -> Just \"Add\" Normal prediction, **keep** detection prediction.\n3. `high_threshold <= p` -> Replace with Normal prediction with normal score 1.0, **remove** all detection predictoin.\n\n\n[Note] I also wrote another kernel to train 2-class model: [📸VinBigData 2-class classifier complete pipeline](https://www.kaggle.com/corochann/vinbigdata-2-class-classifier-complete-pipeline) to train these 2-class classifier model!","metadata":{}},{"cell_type":"code","source":"pred_2class = pd.read_csv(inputdir/\"vinbigdata-2class-prediction/2-cls test pred.csv\")\n# pred_2class = pd.read_csv(inputdir/\"vinbigdata2classpred/test_pred.csv\")\npred_2class.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_2class = test_pred_df.copy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_2class.copy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NORMAL = \"14 1 0 0 1 1\"\nlow_threshold = 0.0\nhigh_threshold = 0.975\n\npred_det_df = pd.read_csv(inputdir/\"vinbigdata-detectron2-prediction/results/20210125_all_alb_aug_512_cos/submission.csv\") # You can load from another submission.csv here too.\nn_normal_before = len(pred_det_df.query(\"PredictionString == @NORMAL\"))\nmerged_df = pd.merge(pred_det_df, pred_2class, on=\"image_id\", how=\"left\")\n\n# 1. p < low_threshold -> \"Keep\": Do nothing, Keep det prediction.\n# 2. low_threshold <= p < high_threshold -> \"Add\": Just \"Add\" Normal prediction\n# 3. high_threshold <= p -> \"Replace\": Replace with Normal prediction\n\nif \"target\" in merged_df.columns:\n merged_df[\"class0\"] = 1 - merged_df[\"target\"]\n\nc0, c1, c2 = 0, 0, 0\nfor i in range(len(merged_df)):\n p0 = merged_df.loc[i, \"class0\"]\n if p0 < low_threshold:\n # Keep, do nothing.\n c0 += 1\n elif low_threshold <= p0 and p0 < high_threshold:\n # Add, keep \"det\" preds and add normal pred.\n merged_df.loc[i, \"PredictionString\"] += f\" 14 {p0} 0 0 1 1\"\n c1 += 1\n else:\n # Replace, remove all \"det\" preds.\n merged_df.loc[i, \"PredictionString\"] = NORMAL\n c2 += 1\n\nn_normal_after = len(merged_df.query(\"PredictionString == @NORMAL\"))\nprint(\n f\"n_normal: {n_normal_before} -> {n_normal_after} with threshold {low_threshold} & {high_threshold}\"\n)\nprint(f\"Keep {c0} Add {c1} Replace {c2}\")\nsubmission_filepath = str(outdir / \"submission.csv\")\nsubmission_df = merged_df[[\"image_id\", \"PredictionString\"]]\nsubmission_df.to_csv(submission_filepath, index=False)\nprint(f\"Saved to {submission_filepath}\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}