EfficientDet PyTorch Pipeline with CutMix + Mixup + KFold + Cosine Annealing
"},{"metadata":{},"cell_type":"markdown","source":"Overview\n\nThis notebook covers a PyTorch EfficientDet training and validation pipeline with novel augmentation, regularization and validation techniques such as Mixup and CutMix Augmentations, Cosine Annealing LR Scheduling and more...
\n\nI hope this helps in saving some time with some techniques for a good score. This notebook uses the latest effdet version - 0.2.3, by @rwightman with the new code changes, improvements and updates. Most kernals I found was still using the older effdet versions.
\n\nThe dataset used in this notebook for the offline installations of the latest EffDet package and its dependancies along with the WBF fused annotation csv file, is registered as a public dataset.
\n\n\n\n**EffDet 0.2.3 Latest + VinBigData WBF Fused**\n\nDATASET LINK - https://www.kaggle.com/sreevishnudamodaran/effdet-latestvinbigdata-wbf-fused\n \n\n\n### References:\n\n**Thanks to @rwightman for the awesome EfficientDet implementation. Do check it out https://github.com/rwightman/efficientdet-pytorch**\n\n**Thanks to @shonenkov, ultralytics (https://github.com/ultralytics/yolov5) and @nvnnghia for the base PyTorch pipeline, Mixup and CutMix implementations from which this notebook is adapted.**\n\n**Thanks to the original authors of the paper.** \n\nCutMix Paper: https://arxiv.org/abs/1905.04899\n\nMixUp Paper: https://arxiv.org/pdf/1710.09412.pdf\n\n**Please do check out their work**\n\n\n\n\n[](https://www.kaggle.com/sreevishnudamodaran)\n\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"Environment Setup\n\nPackage installations for pytorch 1.7.0 which is the current pytorch version in Kaggle kernals as of 21st Jan 2021.\n\nFor efficientdet-pytorch, there is a conflict/bug with Numpy 1.18+ and pycocotools 2.0, force install numpy <= 1.17.5 or ensure you install pycocotools >= 2.0.2. Installing pycocotools >= 2.0.2 here."},{"metadata":{},"cell_type":"markdown","source":" Online installations"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# !pip install pycocotools>=2.0.2\n# !pip install timm>=0.3.2\n# !pip install omegaconf>=2.0\n# !pip install ensemble-boxes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Offline installations"},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"## Install omegaconf - dependancy for effdet\n!pip install --no-deps ../input/effdet-latestvinbigdata-wbf-fused/omegaconf-2.0.6-py3-none-any.whl\n\n## Install timm & ensemble_boxes\n!pip install --no-deps ../input/effdet-latestvinbigdata-wbf-fused/timm-0.3.4-py3-none-any.whl\n!pip install --no-deps ../input/effdet-latestvinbigdata-wbf-fused/ensemble_boxes-1.0.4-py3-none-any.whl\n\n!cp -r ../input/effdet-latestvinbigdata-wbf-fused/pycocotools-2.0.2/ .\n!cd ./pycocotools-2.0.2 && python setup.py install\n!rm -r pycocotools-2.0.2\n\n!cp -r ../input/effdet-latestvinbigdata-wbf-fused/efficientdet-pytorch/ .\n!cd ./efficientdet-pytorch && python setup.py install\n!rm -r efficientdet-pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Update Environment & Release GPU Memory\n\nPlease note that the kernal will stop execution here when using 'Run All'. This is because of the exit command which is used to load refresh and reload the packages installed in the previous steps. So run up till this cell manually then, use 'Run After' with the next cell selected to run the rest in one go.\n\n\nThis applies to fresh environments or if packages in the environment got reset.\n\nThe below cell also helps in clearing GPU memory on crashes and error. When facing OOM error, just run the below cell manually and then run the rest in one go using 'Run After'.
\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Release GPU Memory.\nfrom numba import cuda as CU\ntry:\n device = CU.get_current_device()\n device.reset()\nexcept Exception as E:\n print(\"GPU not enabled. Nothing to clear and good to go.\")\n\n\n## Restart session to detect installed libraries\n!pip list | grep effdet\nimport os\nos._exit(00)\n## Check PyTorch Version","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Import Packages"},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\nimport torch\nimport os\nfrom datetime import datetime\nimport time\nimport random\nimport cv2\nimport pandas as pd\nimport numpy as np\nfrom tqdm.auto import tqdm\nfrom glob import glob\nimport warnings\nfrom collections import Counter\n\nfrom ensemble_boxes import weighted_boxes_fusion\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nfrom torch.utils.data.dataloader import default_collate\n\nfrom effdet import get_efficientdet_config, EfficientDet, DetBenchTrain\nfrom effdet.efficientdet import HeadNet\nfrom effdet import create_model, unwrap_bench, create_loader, create_dataset, create_evaluator, create_model_from_config\nfrom effdet.data import resolve_input_config, SkipSubset\nfrom effdet.anchors import Anchors, AnchorLabeler\nfrom timm.models import resume_checkpoint, load_checkpoint\nfrom timm.utils import *\nfrom timm.optim import create_optimizer\nfrom timm.scheduler import create_scheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seed Everything\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 42\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_annotations = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\ndf_annotations = df_annotations[df_annotations['class_id']!=14].reset_index(drop=True)\ndf_annotations['image_path'] = df_annotations['image_id'].map(lambda x:os.path.join('../input/vinbigdata-original-image-dataset/vinbigdata/train',\n str(x)+'.jpg'))\ndf_annotations.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_paths = df_annotations['image_path'].unique()\nprint(\"Number of Images with abnormalities:\",len(image_paths))\nanno_count = df_annotations.shape[0]\nprint(\"Number of Annotations with abnormalities:\", anno_count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Loading Fused Annotations From Dataset\n\nI have covered the bounding box fusion techniques in another notebook.\n\nhttps://www.kaggle.com/sreevishnudamodaran/vinbigdata-fusing-bboxes-coco-dataset\n\nDATASET USED: **EffDet 0.2.3 Latest + VinBigData WBF Fused**\n\nhttps://www.kaggle.com/sreevishnudamodaran/effdet-latestvinbigdata-wbf-fused\n\nReading the fused bbox annotations directly from the registered dataset. Please note that this dataset only has annotations of the images with bboxes. Along with the annotations, I have also included the latest EfficientDet-Pytorch and it's dependancies for offline installations."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_annotations_wbf = pd.read_csv('../input/effdet-latestvinbigdata-wbf-fused/train_wbf_original.csv', index_col='Unnamed: 0')\n\n#For Testing\n# df_annotations_wbf = df_annotations_wbf.head(18000)\n\ndf_annotations_wbf.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize Original vs Fused\n\nHelper Funtions"},{"metadata":{"trusted":true},"cell_type":"code","source":"label2color = [[59, 238, 119], [222, 21, 229], [94, 49, 164], [206, 221, 133], [117, 75, 3],\n [210, 224, 119], [211, 176, 166], [63, 7, 197], [102, 65, 77], [194, 134, 175],\n [209, 219, 50], [255, 44, 47], [89, 125, 149], [110, 27, 100]]\n\nviz_labels = [\"Aortic_enlargement\", \"Atelectasis\", \"Calcification\", \"Cardiomegaly\",\n \"Consolidation\", \"ILD\", \"Infiltration\", \"Lung_Opacity\", \"Nodule/Mass\",\n \"Other_lesion\", \"Pleural_effusion\", \"Pleural_thickening\", \"Pneumothorax\",\n \"Pulmonary_fibrosis\"]\n\ndef plot_img(img, size=(18, 18), is_rgb=True, title=\"\", cmap=None):\n plt.figure(figsize=size)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\n\ndef plot_imgs(imgs, cols=2, size=10, is_rgb=True, title=\"\", cmap=None, img_size=None):\n rows = len(imgs)//cols + 1\n fig = plt.figure(figsize=(cols*size, rows*size))\n for i, img in enumerate(imgs):\n if img_size is not None:\n img = cv2.resize(img, img_size)\n fig.add_subplot(rows, cols, i+1)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n return fig\n \ndef draw_bbox(image, box, label, color): \n alpha = 0.4\n alpha_font = 0.6\n thickness = 4\n font_size = 2.0\n font_weight = 2\n overlay_bbox = image.copy()\n overlay_text = image.copy()\n output = image.copy()\n\n text_width, text_height = cv2.getTextSize(label.upper(), cv2.FONT_HERSHEY_SIMPLEX, font_size, font_weight)[0]\n cv2.rectangle(overlay_bbox, (box[0], box[1]), (box[2], box[3]),\n color, -1)\n cv2.addWeighted(overlay_bbox, alpha, output, 1 - alpha, 0, output)\n cv2.rectangle(overlay_text, (box[0], box[1]-18-text_height), (box[0]+text_width+8, box[1]),\n (0, 0, 0), -1)\n cv2.addWeighted(overlay_text, alpha_font, output, 1 - alpha_font, 0, output)\n cv2.rectangle(output, (box[0], box[1]), (box[2], box[3]),\n color, thickness)\n cv2.putText(output, label.upper(), (box[0], box[1]-12),\n cv2.FONT_HERSHEY_SIMPLEX, font_size, (255, 255, 255), font_weight, cv2.LINE_AA)\n return output\n\ndef draw_bbox_small(image, box, label, color): \n alpha = 0.4\n alpha_text = 0.4\n thickness = 1\n font_size = 0.4\n overlay_bbox = image.copy()\n overlay_text = image.copy()\n output = image.copy()\n\n text_width, text_height = cv2.getTextSize(label.upper(), cv2.FONT_HERSHEY_SIMPLEX, font_size, thickness)[0]\n cv2.rectangle(overlay_bbox, (box[0], box[1]), (box[2], box[3]),\n color, -1)\n cv2.addWeighted(overlay_bbox, alpha, output, 1 - alpha, 0, output)\n cv2.rectangle(overlay_text, (box[0], box[1]-7-text_height), (box[0]+text_width+2, box[1]),\n (0, 0, 0), -1)\n cv2.addWeighted(overlay_text, alpha_text, output, 1 - alpha_text, 0, output)\n cv2.rectangle(output, (box[0], box[1]), (box[2], box[3]),\n color, thickness)\n cv2.putText(output, label.upper(), (box[0], box[1]-5),\n cv2.FONT_HERSHEY_SIMPLEX, font_size, (255, 255, 255), thickness, cv2.LINE_AA)\n return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"viz_images = []\n\nfor img_id in df_annotations_wbf['image_id'].unique()[:2]:\n img_path = df_annotations_wbf[df_annotations_wbf.image_id==img_id]['image_path'].iloc[0]\n# print(img_path)\n img_array = cv2.imread(img_path)\n\n img_annotations = df_annotations[df_annotations.image_id==img_id]\n boxes_actual = img_annotations[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n labels_actual = img_annotations['class_id'].to_numpy().tolist()\n \n img_annotations_wbf = df_annotations_wbf[df_annotations_wbf.image_id==img_id]\n boxes_wbf = img_annotations_wbf[['x_min', 'y_min', 'x_max', 'y_max']].to_numpy().tolist()\n box_labels_wbf = img_annotations_wbf['class_id'].to_numpy().tolist()\n \n print(\"Bboxes before WBF:\\n\", boxes_actual)\n print(\"Labels before WBF:\\n\", labels_actual)\n \n ## Visualize Original Bboxes\n img_before = img_array.copy()\n for box, label in zip(boxes_actual, labels_actual):\n x_min, y_min, x_max, y_max = (box[0], box[1], box[2], box[3])\n color = label2color[int(label)]\n img_before = draw_bbox(img_before, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_before)\n\n print(\"Bboxes after WBF:\\n\", boxes_wbf)\n print(\"Labels after WBF:\\n\", box_labels_wbf)\n \n ## Visualize Bboxes after operation\n img_after = img_array.copy()\n for box, label in zip(boxes_wbf, box_labels_wbf):\n color = label2color[int(label)]\n img_after = draw_bbox(img_after, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(img_after)\n print()\n \nplot_imgs(viz_images, cmap=None)\nplt.figtext(0.3, 0.9,\"Original Bboxes\", va=\"top\", ha=\"center\", size=25)\nplt.figtext(0.73, 0.9,\"WBF\", va=\"top\", ha=\"center\", size=25)\nplt.savefig('wbf.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentations & Transforms\n\nUsign the Albumentations package to create an augmentation pipeline for training and validation. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_transforms():\n return A.Compose(\n [\n ## RandomSizedCrop not working for some reason. I'll post a thread for this issue soon.\n ## Any help or suggestions are appreciated.\n# A.RandomSizedCrop(min_max_height=(300, 512), height=512, width=512, p=0.5),\n# A.RandomSizedCrop(min_max_height=(300, 1000), height=1000, width=1000, p=0.5),\n A.OneOf([\n A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, \n val_shift_limit=0.2, p=0.9),\n A.RandomBrightnessContrast(brightness_limit=0.2, \n contrast_limit=0.2, p=0.9),\n ],p=0.9),\n A.JpegCompression(quality_lower=85, quality_upper=95, p=0.2),\n A.OneOf([\n A.Blur(blur_limit=3, p=1.0),\n A.MedianBlur(blur_limit=3, p=1.0)\n ],p=0.1),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.RandomRotate90(p=0.5),\n A.Transpose(p=0.5),\n A.Resize(height=512, width=512, p=1),\n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n ToTensorV2(p=1.0)\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )\n\ndef get_valid_transforms():\n return A.Compose(\n [\n A.Resize(height=512, width=512, p=1.0),\n ToTensorV2(p=1.0),\n ], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0, \n min_visibility=0,\n label_fields=['labels']\n )\n )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Dataset Retrieval and Pre-processsing\n\nDefine the data loader class to retrive images, perform albumentation-based + custom CutMix and MixUp augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_ROOT_PATH = '../input/vinbigdata-original-image-dataset/vinbigdata/train'\n\nclass DatasetRetriever(Dataset):\n\n def __init__(self, marking, image_ids, transforms=None, test=False):\n super().__init__()\n self.image_ids = image_ids\n self.marking = marking\n self.transforms = transforms\n self.test = test\n \n def __getitem__(self, index: int):\n image_id = self.image_ids[index]\n \n image, boxes, labels = self.load_image_and_boxes(index)\n \n if self.test or random.random() > 0.33:\n image, boxes, labels = self.load_image_and_boxes(index)\n elif random.random() > 0.5:\n image, boxes, labels = self.load_cutmix_image_and_boxes(index)\n else:\n image, boxes, labels = self.load_mixup_image_and_boxes(index)\n \n ## To prevent ValueError: y_max is less than or equal to y_min for bbox from albumentations bbox_utils\n labels = np.array(labels, dtype=np.int).reshape(len(labels), 1)\n combined = np.hstack((boxes.astype(np.int), labels))\n combined = combined[np.logical_and(combined[:,2] > combined[:,0],\n combined[:,3] > combined[:,1])]\n boxes = combined[:, :4]\n labels = combined[:, 4].tolist()\n \n target = {}\n target['boxes'] = boxes\n target['labels'] = torch.tensor(labels)\n target['image_id'] = torch.tensor([index])\n if self.transforms:\n for i in range(10):\n sample = self.transforms(**{\n 'image': image,\n 'bboxes': target['boxes'],\n 'labels': labels\n })\n if len(sample['bboxes']) > 0:\n image = sample['image']\n target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]] ## ymin, xmin, ymax, xmax\n break\n \n ## Handling case where no valid bboxes are present\n if len(target['boxes'])==0 or i==9:\n return None\n else:\n ## Handling case where augmentation and tensor conversion yields no valid annotations\n try:\n assert torch.is_tensor(image), f\"Invalid image type:{type(image)}\"\n assert torch.is_tensor(target['boxes']), f\"Invalid target type:{type(target['boxes'])}\"\n except Exception as E:\n print(\"Image skipped:\", E)\n return None \n\n return image, target, image_id\n\n def __len__(self) -> int:\n return self.image_ids.shape[0]\n \n def load_image_and_boxes(self, index):\n image_id = self.image_ids[index]\n# print(f'{TRAIN_ROOT_PATH}/{image_id}.jpg')\n image = cv2.imread(f'{TRAIN_ROOT_PATH}/{image_id}.jpg', cv2.IMREAD_COLOR).copy()\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n records = self.marking[self.marking['image_id'] == image_id]\n boxes = records[['x_min', 'y_min', 'x_max', 'y_max']].values\n labels = records['class_id'].tolist()\n resize_transform = A.Compose([A.Resize(height=512, width=512, p=1.0)], \n p=1.0, \n bbox_params=A.BboxParams(\n format='pascal_voc',\n min_area=0.1, \n min_visibility=0.1,\n label_fields=['labels'])\n )\n\n resized = resize_transform(**{\n 'image': image,\n 'bboxes': boxes,\n 'labels': labels\n })\n\n resized_bboxes = np.vstack((list(bx) for bx in resized['bboxes']))\n return resized['image'], resized_bboxes, resized['labels']\n \n def load_mixup_image_and_boxes(self, index):\n image, boxes, labels = self.load_image_and_boxes(index)\n r_image, r_boxes, r_labels = self.load_image_and_boxes(random.randint(0, self.image_ids.shape[0] - 1))\n return (image+r_image)/2, np.vstack((boxes, r_boxes)).astype(np.int32), np.concatenate((labels, r_labels))\n\n def load_cutmix_image_and_boxes(self, index, imsize=512):\n \"\"\" \n This implementation of cutmix author: https://www.kaggle.com/nvnnghia \n Refactoring and adaptation: https://www.kaggle.com/shonenkov\n \"\"\"\n w, h = imsize, imsize\n s = imsize // 2\n \n xc, yc = [int(random.uniform(imsize * 0.25, imsize * 0.75)) for _ in range(2)] # center x, y\n indexes = [index] + [random.randint(0, self.image_ids.shape[0] - 1) for _ in range(3)]\n\n result_image = np.full((imsize, imsize, 3), 1, dtype=np.float32)\n result_boxes = []\n result_labels = np.array([], dtype=np.int)\n\n for i, index in enumerate(indexes):\n image, boxes, labels = self.load_image_and_boxes(index)\n if i == 0:\n x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc # xmin, ymin, xmax, ymax (large image)\n x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h # xmin, ymin, xmax, ymax (small image)\n elif i == 1: # top right\n x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\n x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\n elif i == 2: # bottom left\n x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\n x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, max(xc, w), min(y2a - y1a, h)\n elif i == 3: # bottom right\n x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\n x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\n result_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]\n padw = x1a - x1b\n padh = y1a - y1b\n\n boxes[:, 0] += padw\n boxes[:, 1] += padh\n boxes[:, 2] += padw\n boxes[:, 3] += padh\n\n result_boxes.append(boxes)\n result_labels = np.concatenate((result_labels, labels))\n\n result_boxes = np.concatenate(result_boxes, 0)\n np.clip(result_boxes[:, 0:], 0, 2 * s, out=result_boxes[:, 0:])\n result_boxes = result_boxes.astype(np.int32)\n index_to_use = np.where((result_boxes[:,2]-result_boxes[:,0])*(result_boxes[:,3]-result_boxes[:,1]) > 0)\n result_boxes = result_boxes[index_to_use]\n result_labels = result_labels[index_to_use]\n \n return result_image, result_boxes, result_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize Mosaic and MixUp Techniques\n\nMixUp\n\nMixUp is a technique which trains a neural network on convex combinations of pairs of examples and their labels imparting regularization to favor simple linear behavior in-between training examples. Experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. It also reduces the memorization of corrupt labels, increases the robustness of examples, and stabilizes the training process.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"warnings.filterwarnings(\"ignore\")\n\nviz_ids = df_annotations_wbf.sample(6).image_id.tolist()\nviz_dataset = DatasetRetriever(\n image_ids=np.array(viz_ids),\n marking=df_annotations_wbf,\n transforms=get_train_transforms(),\n test=False,\n )\nviz_images = []\nfor idx, im_id in enumerate(viz_ids):\n image, boxes, labels = viz_dataset.load_mixup_image_and_boxes(idx)\n image_viz = image.copy()\n# print(\"image_viz.shape\", image_viz.shape)\n for box, label in zip(boxes, labels):\n color = label2color[int(label)]\n# image_viz *= 255 \n# image_viz = image_viz.astype('uint8')\n image_viz = cv2.normalize(src=image_viz, dst=None, alpha=0, beta=255,\n norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8U)\n image_viz = draw_bbox_small(image_viz, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(image_viz)\n\nfig = plot_imgs(viz_images)\nfig.suptitle(\"MIXUP VISUALIZED\", x=0.125, y=0.91, ha='left',\n fontweight=100, fontfamily='Lato', size=36)\nplt.savefig('mixup.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"CutMix - Mosaic\n\nIn CutMix augmentation, patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. By making efficient use of training pixels and retaining the regularization effect of regional dropout, CutMix consistently outperforms the state-of-the-art augmentation strategies \n\nCutMix achieves lower validation errors. When the learning rates are low, the training without cutmix suffer from overfitting with increasing validation error. CutMix has experimentally shown a steady decrease in validation error with significantly less overfitting.\n\n**I have used the Mosaic CutMix implementaion which mixes four images and its bboxes randomly. Recent research and experiments show they yield the best performance.**\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"viz_ids = df_annotations_wbf.sample(6).image_id.tolist()\nviz_dataset = DatasetRetriever(\n image_ids=np.array(viz_ids),\n marking=df_annotations_wbf,\n transforms=get_train_transforms(),\n test=False,\n )\nviz_images = []\nfor idx, im_id in enumerate(viz_ids):\n image, boxes, labels = viz_dataset.load_cutmix_image_and_boxes(idx)\n image_viz = image.copy()\n# print(\"image_viz.shape\", image_viz.shape)\n for box, label in zip(boxes, labels):\n color = label2color[int(label)]\n# image_viz *= 255 \n# image_viz = image_viz.astype('uint8')\n image_viz = cv2.normalize(src=image_viz, dst=None, alpha=0, beta=255,\n norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8U)\n image_viz = draw_bbox_small(image_viz, list(np.int_(box)), viz_labels[label], color)\n viz_images.append(image_viz)\n\nfig = plot_imgs(viz_images)\nfig.suptitle(\"CUTMIX VISUALIZED\", x=0.125, y=0.91, ha='left',\n fontweight=100, fontfamily='Lato', size=36)\nplt.savefig('cutmix.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Helper Functions\n\nFunction to store Average and Current values"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Build Training and Validation Loops"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Fitter:\n def __init__(self, model, device, config):\n self.config = config\n self.epoch = 0\n\n self.base_dir = f'./{config.folder}'\n \n if not os.path.exists(self.base_dir):\n os.makedirs(self.base_dir)\n \n self.log_path = f'{self.base_dir}/log.txt'\n self.best_summary_loss = 10**5\n self.model = model\n self.device = device\n\n param_optimizer = list(self.model.named_parameters())\n no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n optimizer_grouped_parameters = [\n {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n ] \n\n self.optimizer = config.OptimizerClass(self.model.parameters(), lr=config.lr)\n self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\n self.log(f'Fitter prepared. Device is {self.device}')\n\n def fit(self, train_loader, validation_loader):\n history_dict = {}\n history_dict['epoch'] = []\n history_dict['train_loss'] = []\n history_dict['val_loss'] = []\n history_dict['train_lr'] = []\n \n for e in range(self.config.n_epochs):\n history_dict['epoch'].append(self.epoch)\n lr = self.optimizer.param_groups[0]['lr']\n timestamp = datetime.utcnow().isoformat()\n \n if self.config.verbose:\n self.log(f'\\n{timestamp}\\nLR: {lr}')\n\n t = time.time()\n summary_loss, loss_trend, lr_trend = self.train_epoch(train_loader)\n history_dict['train_loss'].append(loss_trend)\n history_dict['train_lr'].append(lr_trend)\n self.log(f'[RESULT]: Train. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n self.save(f'{self.base_dir}/last-checkpoint.bin')\n \n t = time.time()\n summary_loss, loss_trend = self.validation(validation_loader)\n history_dict['val_loss'].append(loss_trend)\n self.log(f'[RESULT]: Val. Epoch: {self.epoch}, summary_loss: {summary_loss.avg:.5f}, time: {(time.time() - t):.5f}')\n \n if summary_loss.avg < self.best_summary_loss:\n self.best_summary_loss = summary_loss.avg\n self.model.eval()\n self.save(f'{self.base_dir}/best-checkpoint-{str(self.epoch).zfill(3)}epoch.bin')\n \n try:\n os.remove(f)\n except:pass\n f = f'{self.base_dir}/best-checkpoint-{str(self.epoch).zfill(3)}epoch.bin'\n\n if self.config.validation_scheduler:\n self.scheduler.step(metrics=summary_loss.avg)\n\n self.epoch += 1\n return history_dict\n\n def train_epoch(self, train_loader):\n self.model.train()\n summary_loss = AverageMeter()\n t = time.time()\n loss_trend = []\n lr_trend = []\n for step, (images, targets, image_ids) in tqdm(enumerate(train_loader), total=len(train_loader)):\n if self.config.verbose:\n if step % self.config.verbose_step == 0:\n print(\n f'Train Step {step}/{len(train_loader)}, ' + \\\n f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n f'time: {(time.time() - t):.5f}', end='\\r'\n ) \n \n images = torch.stack(images)\n images = images.to(self.device).float()\n \n target_res = {}\n boxes = [target['boxes'].to(self.device).float() for target in targets]\n labels = [target['labels'].to(self.device).float() for target in targets]\n target_res['bbox'] = boxes\n target_res['cls'] = labels\n self.optimizer.zero_grad()\n output = self.model(images, target_res)\n\n loss = output['loss']\n loss.backward()\n summary_loss.update(loss.detach().item(), self.config.batch_size)\n self.optimizer.step()\n\n if self.config.step_scheduler:\n self.scheduler.step()\n\n \n lr = self.optimizer.param_groups[0]['lr']\n loss_trend.append(summary_loss.avg)\n lr_trend.append(lr)\n return summary_loss, loss_trend, lr_trend\n \n def validation(self, val_loader):\n self.model.eval()\n summary_loss = AverageMeter()\n t = time.time()\n loss_trend = []\n# lr_trend = []\n \n for step, (images, targets, image_ids) in tqdm(enumerate(val_loader), total=len(val_loader)):\n if self.config.verbose:\n if step % self.config.verbose_step == 0:\n print(\n f'Val Step {step}/{len(val_loader)}, ' + \\\n f'summary_loss: {summary_loss.avg:.5f}, ' + \\\n f'time: {(time.time() - t):.5f}', end='\\r'\n )\n with torch.no_grad():\n images = torch.stack(images)\n images = images.to(self.device).float()\n target_res = {}\n boxes = [target['boxes'].to(self.device).float() for target in targets]\n labels = [target['labels'].to(self.device).float() for target in targets]\n target_res['bbox'] = boxes\n target_res['cls'] = labels \n target_res[\"img_scale\"] = torch.tensor([1.0] * self.config.batch_size,\n dtype=torch.float).to(self.device)\n target_res[\"img_size\"] = torch.tensor([images[0].shape[-2:]] * self.config.batch_size,\n dtype=torch.float).to(self.device)\n \n output = self.model(images, target_res)\n \n loss = output['loss']\n summary_loss.update(loss.detach().item(), self.config.batch_size)\n\n loss_trend.append(summary_loss.avg)\n return summary_loss, loss_trend[-1]\n \n \n def save(self, path):\n self.model.eval()\n torch.save({\n 'model_state_dict': self.model.model.state_dict(),\n 'optimizer_state_dict': self.optimizer.state_dict(),\n 'scheduler_state_dict': self.scheduler.state_dict(),\n 'best_summary_loss': self.best_summary_loss,\n 'epoch': self.epoch,\n }, path)\n\n def load(self, path):\n checkpoint = torch.load(path)\n self.model.model.load_state_dict(checkpoint['model_state_dict'])\n self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n self.best_summary_loss = checkpoint['best_summary_loss']\n self.epoch = checkpoint['epoch'] + 1\n \n def log(self, message):\n if self.config.verbose:\n print(message)\n with open(self.log_path, 'a+') as logger:\n logger.write(f'{message}\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define Training Configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"class TrainGlobalConfig:\n def __init__(self):\n self.num_classes = 14\n self.num_workers = 2\n self.batch_size = 4 \n self.n_epochs = 2\n self.lr = 0.0002\n self.model_name = 'tf_efficientdet_d1'\n self.folder = 'training_job'\n self.verbose = True\n self.verbose_step = 1\n self.step_scheduler = True\n self.validation_scheduler = False\n self.n_img_count = len(df_annotations_wbf.image_id.unique())\n self.OptimizerClass = torch.optim.AdamW\n self.SchedulerClass = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts\n self.scheduler_params = dict(\n T_0=50,\n T_mult=1,\n eta_min=0.0001,\n last_epoch=-1,\n verbose=False\n )\n self.kfold = 3\n \n def reset(self):\n self.OptimizerClass = torch.optim.AdamW\n self.SchedulerClass = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts\n\ntrain_config = TrainGlobalConfig()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating Validation Folds"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import GroupKFold, train_test_split\n\ndf_annotations_wbf['fold'] = -1\ngroup_kfold = GroupKFold(n_splits = 3)\nfor fold, (train_index, val_index) in enumerate(group_kfold.split(df_annotations_wbf,\n groups=df_annotations_wbf.image_id.tolist())):\n df_annotations_wbf.loc[val_index, 'fold'] = fold\ndf_annotations_wbf.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cosine Annealing Warm Restarts LR Scheduler\n\nCosine Annealing is a type of learning rate schedule where the learning rate is decreased according to the cosine function to a minimum value before being increased rapidly again. The resetting of the learning rate acts like a simulated restart of the learning process and it also mimics a sequential model ensembling. This rapid increase of the learning rate may also cause the 'hopping' out of the weights of the model from local minima and it drives its way towards the global minimum.\n\nThe reuse of good weights of the model as a starting point of the restart is referred to as a \"warm restart\" in contrast to a \"cold restart\" where a new set of small random numbers may be used as a starting point.\n\nIt has been found empirically in numerous experiments that a cyclic LR schedule such as Cosine Annealing outperforms traditional learning rate schedulers in terms of faster and better convergence.\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport matplotlib.ticker as ticker\nfrom matplotlib import rcParams\nsns.set(rc={\"font.size\":18,\"axes.titlesize\":30,\"axes.labelsize\":18,\n \"axes.titlepad\":22, \"axes.labelpad\":18, \"legend.fontsize\":15,\n \"legend.title_fontsize\":15, \"figure.titlesize\":35})\n\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nscheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer,\n T_0=100,\n T_mult=1,\n eta_min=0.0001,\n last_epoch=-1,\n verbose = False)\n\nlrs = []\nsample_steps = train_config.n_img_count//train_config.n_epochs\nfor e in range(train_config.n_epochs):\n for i in range(sample_steps):\n scheduler.step()\n lrs.append(\n optimizer.param_groups[0][\"lr\"]\n )\nfig = plt.figure(figsize=(22,8))\nfig.suptitle(\"PROJECTED LR TREND - COSINE ANNEALING WARM RESTARTS\", x=0.125, y=1.00, ha='left',\n fontweight=100, fontfamily='Lato', size=34)\nplt.plot(lrs)\nplt.savefig('cosanneal.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initiate Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Training will resume if the checkpoint path is specified below\ncheckpoint_path = None\n\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else \"cpu\")\n\n## Filters out invalid return items from the Dataloader\n# def collate_fn(batch):\n# batch = list(filter(lambda x : x is not None, batch))\n# return default_collate(batch)\n\ndef collate_fn(batch):\n batch = list(filter(lambda x: x is not None, batch))\n \n return tuple(zip(*batch))\n\n# def collate_fn(batch):\n# return tuple(zip(*batch))\n\nfold_history = []\nfor val_fold in range(train_config.kfold):\n print(f'Fold {val_fold+1}/{train_config.kfold}')\n \n train_ids = df_annotations_wbf[df_annotations_wbf['fold'] != val_fold].image_id.unique()\n val_ids = df_annotations_wbf[df_annotations_wbf['fold'] == val_fold].image_id.unique()\n \n train_dataset = DatasetRetriever(\n image_ids=train_ids,\n marking=df_annotations_wbf,\n transforms=get_train_transforms(),\n test=False,\n )\n\n validation_dataset = DatasetRetriever(\n image_ids=val_ids,\n marking=df_annotations_wbf,\n transforms=get_valid_transforms(),\n test=True,\n )\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=train_config.batch_size,\n sampler=RandomSampler(train_dataset),\n pin_memory=False,\n drop_last=True,\n num_workers=train_config.num_workers,\n collate_fn=collate_fn,\n )\n val_loader = torch.utils.data.DataLoader(\n validation_dataset, \n batch_size=train_config.batch_size,\n num_workers=train_config.num_workers,\n shuffle=False,\n sampler=SequentialSampler(validation_dataset),\n pin_memory=False,\n collate_fn=collate_fn,\n )\n \n base_config = get_efficientdet_config(train_config.model_name)\n base_config.image_size = (512, 512)\n\n if(checkpoint_path):\n print(f'Resuming from checkpoint: {checkpoint_path}') \n model = create_model_from_config(base_config, bench_task='train', bench_labeler=True,\n num_classes=train_config.num_classes,\n pretrained=False)\n model.to(device)\n \n fitter = Fitter(model=model, device=device, config=train_config)\n fitter.load(checkpoint_path)\n \n else:\n model = create_model_from_config(base_config, bench_task='train', bench_labeler=True,\n pretrained=True,\n num_classes=train_config.num_classes)\n model.to(device)\n \n fitter = Fitter(model=model, device=device, config=train_config) \n \n model_config = model.config\n history_dict = fitter.fit(train_loader, val_loader)\n fold_history.append(history_dict)\n \n ## Reset Optimizer and LR Sch+)\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize Training Job & Model Metrics"},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_history_c = fold_history.copy()\n\ntrain_loss_all = []\nval_loss_all = []\n\nn_steps_fold = (train_config.n_img_count//train_config.n_epochs)//train_config.kfold\n\nfor fold, fold_dict in enumerate(fold_history_c):\n train_losses = [item for sublist in fold_dict['train_loss'] for item in sublist]\n val_losses = [item for item in fold_dict['val_loss']]\n train_lrs = [item for sublist in fold_dict['train_lr'] for item in sublist]\n train_loss_all.append(np.array(train_losses))\n \n val_losses = np.repeat(val_losses, n_steps_fold).tolist()\n val_loss_all.append(np.array(val_losses))\n \n fig = plt.figure(figsize=(22,8))\n fig.suptitle(f'FOLD{fold+1} - TRAIN LOSS & VAL LOSSES', x=0.125, y=1.00, ha='left',\n fontweight=100, fontfamily='Lato', size=36)\n plt.plot(train_losses, color='red', label='train_loss', linewidth=1)\n plt.plot(val_losses, color='green', label='val_loss', linewidth=1)\n plt.legend() \n plt.savefig(f'fold{fold+1}_loss_trend.png', bbox_inches='tight')\n plt.show()\n \n fig = plt.figure(figsize=(22,8))\n fig.suptitle(f'FOLD{fold+1} - LEARNING RATE TREND', x=0.125, y=1.00, ha='left',\n fontweight=100, fontfamily='Lato', size=36)\n plt.plot(train_lrs, color='blue', label='lr', linewidth=1)\n plt.legend() \n plt.savefig(f'fold{fold+1}_lr_trend.png', bbox_inches='tight')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.DataFrame([np.array(i) for i in train_loss_all]).transpose()\ntrain_df['average'] = train_df.mean(axis=1)\nval_df = pd.DataFrame([np.array(i) for i in val_loss_all]).transpose()\nval_df['average'] = val_df.mean(axis=1)\n\nfig = plt.figure(figsize=(22,8))\nfig.suptitle(f'AVERAGE TRAIN LOSS & VAL LOSSES', x=0.125, y=1.00, ha='left',\n fontweight=100, fontfamily='Lato', size=36)\nplt.plot(train_df['average'], color='red', label='train_loss', linewidth=1)\nplt.plot(val_df['average'], color='green', label='val_loss', linewidth=1)\nplt.legend() \nplt.savefig('avg_loss_trend.png', bbox_inches='tight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!zip -r training_job_full_2epochs_30_jan.zip ./training_job/*","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let me know if you have any suggestions or improvements.
\n\nTHANK YOU! PLEASE UPVOTE
\n\nHOPE IT WAS USEFUL
\n\nTuning & updation in progress.
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}