{"cells":[{"metadata":{"papermill":{"duration":0.022577,"end_time":"2021-02-17T09:48:38.649598","exception":false,"start_time":"2021-02-17T09:48:38.627021","status":"completed"},"tags":[]},"cell_type":"markdown","source":"<h1 style=\"background-color:powderblue;\"><center>Introduction to Faster RCNN with pytorch</center></h1>\n\nFaster R-CNN was originally published in NIPS 2015. After publication, it went through a couple of revisions which we’ll later discuss. As we mentioned in our previous blog post, Faster R-CNN is the third iteration of the R-CNN papers — which had Ross Girshick as author & co-author.\n\nEverything started with “Rich feature hierarchies for accurate object detection and semantic segmentation” (R-CNN) in 2014, which used an algorithm called Selective Search to propose possible regions of interest and a standard Convolutional Neural Network (CNN) to classify and adjust them. It quickly evolved into Fast R-CNN, published in early 2015, where a technique called Region of Interest Pooling allowed for sharing expensive computations and made the model much faster. Finally came Faster R-CNN, where the first fully differentiable model was proposed.\n\nhttps://www.alegion.com/faster-r-cnn - For further read\n\n<h3 style=\"background-color:red;\">Work in progress: Upvote and support</h3>\n\n## Ran on low epochs to save GPU time"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-02-17T09:48:38.697529Z","iopub.status.busy":"2021-02-17T09:48:38.696705Z","iopub.status.idle":"2021-02-17T09:48:38.709214Z","shell.execute_reply":"2021-02-17T09:48:38.709725Z"},"papermill":{"duration":0.038042,"end_time":"2021-02-17T09:48:38.710007","exception":false,"start_time":"2021-02-17T09:48:38.671965","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('<iframe src=https://arxiv.org/pdf/1506.01497.pdf width=600 height=650></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2021-02-17T09:48:38.760707Z","iopub.status.busy":"2021-02-17T09:48:38.760044Z","iopub.status.idle":"2021-02-17T09:48:43.14777Z","shell.execute_reply":"2021-02-17T09:48:43.147185Z"},"papermill":{"duration":4.416077,"end_time":"2021-02-17T09:48:43.147908","exception":false,"start_time":"2021-02-17T09:48:38.731831","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport cv2\nimport os\nimport re\nimport pydicom\nimport warnings\n\nfrom PIL import Image\n\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom albumentations.core.transforms_interface import ImageOnlyTransform\n\nimport torch\nimport torchvision\n\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\nfrom torchvision.models.detection.rpn import AnchorGenerator\n\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.utils.data.sampler import SequentialSampler\n\nfrom matplotlib import pyplot as plt\nimport random\npaddingSize= 0\n\nwarnings.filterwarnings(\"ignore\")\n\n\nDIR_INPUT = '/kaggle/input/vinbigdata-chest-xray-abnormalities-detection'\nDIR_TRAIN = f'{DIR_INPUT}/train'\nDIR_TEST = f'{DIR_INPUT}/test'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2021-02-17T09:48:43.203292Z","iopub.status.busy":"2021-02-17T09:48:43.202527Z","iopub.status.idle":"2021-02-17T09:48:43.552267Z","shell.execute_reply":"2021-02-17T09:48:43.552884Z"},"papermill":{"duration":0.38271,"end_time":"2021-02-17T09:48:43.553126","exception":false,"start_time":"2021-02-17T09:48:43.170416","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(f'{DIR_INPUT}/train.csv')\ntrain_df.fillna(0, inplace=True)\ntrain_df.loc[train_df[\"class_id\"] == 14, ['x_max', 'y_max']] = 1.0\n\n# FasterRCNN handles class_id==0 as the background.\ntrain_df[\"class_id\"] = train_df[\"class_id\"] + 1\ntrain_df.loc[train_df[\"class_id\"] == 15, [\"class_id\"]] = 0\n\nprint(\"df Shape: \"+str(train_df.shape))\nprint(\"No Of Classes: \"+str(train_df[\"class_id\"].nunique()))\ntrain_df.sort_values(by='image_id').head(10)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.028398,"end_time":"2021-02-17T09:48:43.610149","exception":false,"start_time":"2021-02-17T09:48:43.581751","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Challange Overview\n\nIn this competition, you’ll automatically localize and classify 14 types of thoracic abnormalities from chest radiographs. You'll work with a dataset consisting of 18,000 scans that have been annotated by experienced radiologists. You can train your model with 15,000 independently-labeled images and will be evaluated on a test set of 3,000 images. These annotations were collected via VinBigData's web-based platform, VinLab. Details on building the dataset can be found in our recent paper “VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations”.\n\n\n## Data Description\n\nIn this competition, we are classifying common thoracic lung diseases and localizing critical findings. This is an object detection and classification problem.\n\nFor each test image, you will be predicting a bounding box and class for all findings. If you predict that there are no findings, you should create a prediction of \"14 1 0 0 1 1\" (14 is the class ID for no finding, and this provides a one-pixel bounding box with a confidence of 1.0).\n\nThe images are in DICOM format, which means they contain additional data that might be useful for visualizing and classifying.\nDataset information\n\nThe dataset comprises 18,000 postero-anterior (PA) CXR scans in DICOM format, which were de-identified to protect patient privacy. All images were labeled by a panel of experienced radiologists for the presence of 14 critical radiographic findings as listed below:\n\n\n\n\n    0 - Aortic enlargement\n    1 - Atelectasis\n    2 - Calcification\n    3 - Cardiomegaly\n    4 - Consolidation\n    5 - ILD\n    6 - Infiltration\n    7 - Lung Opacity\n    8 - Nodule/Mass\n    9 - Other lesion\n    10 - Pleural effusion\n    11 - Pleural thickening\n    12 - Pneumothorax\n    13 - Pulmonary fibrosis\n"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:43.679429Z","iopub.status.busy":"2021-02-17T09:48:43.677253Z","iopub.status.idle":"2021-02-17T09:48:43.680317Z","shell.execute_reply":"2021-02-17T09:48:43.680897Z"},"papermill":{"duration":0.042215,"end_time":"2021-02-17T09:48:43.68109","exception":false,"start_time":"2021-02-17T09:48:43.638875","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def label_to_name(id):\n    id = int(id)\n    id = id-1\n    if id == 0:\n        return \"Aortic enlargement\"\n    if id == 1:\n        return \"Atelectasis\"\n    if id == 2:\n        return \"Calcification\"\n    if id == 3:\n        return \"Cardiomegaly\"\n    if id == 4:\n        return \"Consolidation\"\n    if id == 5:\n        return \"ILD\"\n    if id == 6:\n        return \"Infiltration\"\n    if id == 7:\n        return \"Lung Opacity\"\n    if id == 8:\n        return \"Nodule/Mass\"\n    if id == 9:\n        return \"Other lesion\"\n    if id == 10:\n        return \"Pleural effusion\"\n    if id == 11:\n        return \"Pleural thickening\"\n    if id == 12:\n        return \"Pneumothorax\"\n    if id == 13:\n        return \"Pulmonary fibrosis\"\n    else:\n        return str(id)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:43.742017Z","iopub.status.busy":"2021-02-17T09:48:43.741276Z","iopub.status.idle":"2021-02-17T09:48:43.773707Z","shell.execute_reply":"2021-02-17T09:48:43.773167Z"},"papermill":{"duration":0.065949,"end_time":"2021-02-17T09:48:43.773871","exception":false,"start_time":"2021-02-17T09:48:43.707922","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"image_ids = train_df['image_id'].unique()\nvalid_ids = image_ids[-10000:]# Tran and Validation Split \ntrain_ids = image_ids[:-10000]\n\n\nvalid_df = train_df[train_df['image_id'].isin(valid_ids)]\ntrain_df = train_df[train_df['image_id'].isin(train_ids)]\n\ntrain_df[\"class_id\"] = train_df[\"class_id\"].apply(lambda x: x+1)\nvalid_df[\"class_id\"] = valid_df[\"class_id\"].apply(lambda x: x+1)\n\ntrain_df.shape\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Clean\n\ntrain_df['area'] = (train_df['x_max'] - train_df['x_min']) * (train_df['y_max'] - train_df['y_min'])\nvalid_df['area'] = (valid_df['x_max'] - valid_df['x_min']) * (valid_df['y_max'] - valid_df['y_min'])\ntrain_df = train_df[train_df['area'] > 1]\nvalid_df = valid_df[valid_df['area'] > 1]\n\ntrain_df = train_df[(train_df['class_id'] > 1) & (train_df['class_id'] < 15)]\nvalid_df = valid_df[(valid_df['class_id'] > 1) & (valid_df['class_id'] < 15)]\n\n\ntrain_df = train_df.drop(['area'], axis = 1)\ntrain_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max(train_df['class_id'])","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:43.842184Z","iopub.status.busy":"2021-02-17T09:48:43.841287Z","iopub.status.idle":"2021-02-17T09:48:43.844602Z","shell.execute_reply":"2021-02-17T09:48:43.844014Z"},"papermill":{"duration":0.04688,"end_time":"2021-02-17T09:48:43.844747","exception":false,"start_time":"2021-02-17T09:48:43.797867","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Thanks -  https://www.kaggle.com/pestipeti/\nclass VinBigDataset(Dataset): #Class to load Training Data\n    \n    def __init__(self, dataframe, image_dir, transforms=None,stat = 'Train'):\n        super().__init__()\n        \n        self.image_ids = dataframe[\"image_id\"].unique()\n        self.df = dataframe\n        self.image_dir = image_dir\n        self.transforms = transforms\n        self.stat = stat\n        \n    def __getitem__(self, index):\n        if self.stat == 'Train':\n            \n            image_id = self.image_ids[index]\n            records = self.df[(self.df['image_id'] == image_id)]\n            records = records.reset_index(drop=True)\n\n            dicom = pydicom.dcmread(f\"{self.image_dir}/{image_id}.dicom\")\n\n            image = dicom.pixel_array\n\n            if \"PhotometricInterpretation\" in dicom:\n                if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n                    image = np.amax(image) - image\n\n            intercept = dicom.RescaleIntercept if \"RescaleIntercept\" in dicom else 0.0\n            slope = dicom.RescaleSlope if \"RescaleSlope\" in dicom else 1.0\n\n            if slope != 1:\n                image = slope * image.astype(np.float64)\n                image = image.astype(np.int16)\n\n        \n            image += np.int16(intercept)        \n\n            image = np.stack([image, image, image])\n            image = image.astype('float32')\n            image = image - image.min()\n            image = image / image.max()\n            image = image * 255.0\n            image = image.transpose(1,2,0)\n\n            if records.loc[0, \"class_id\"] == 0:\n                records = records.loc[[0], :]\n\n            boxes = records[['x_min', 'y_min', 'x_max', 'y_max']].values\n            area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n            area = torch.as_tensor(area, dtype=torch.float32)\n            labels = torch.tensor(records[\"class_id\"].values, dtype=torch.int64)\n\n            # suppose all instances are not crowd\n            iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)\n\n            target = {}\n            target['boxes'] = boxes\n            target['labels'] = labels\n            target['image_id'] = torch.tensor([index])\n            target['area'] = area\n            target['iscrowd'] = iscrowd\n\n            if self.transforms:\n                sample = {\n                    'image': image,\n                    'bboxes': target['boxes'],\n                    'labels': labels\n                }\n                sample = self.transforms(**sample)\n                image = sample['image']\n\n                target['boxes'] = torch.tensor(sample['bboxes'])\n\n            if target[\"boxes\"].shape[0] == 0:\n                # Albumentation cuts the target (class 14, 1x1px in the corner)\n                target[\"boxes\"] = torch.from_numpy(np.array([[0.0, 0.0, 1.0, 1.0]]))\n                target[\"area\"] = torch.tensor([1.0], dtype=torch.float32)\n                target[\"labels\"] = torch.tensor([0], dtype=torch.int64)\n\n            return image, target, image_ids\n        \n        else:\n                   \n            image_id = self.image_ids[index]\n            records = self.df[(self.df['image_id'] == image_id)]\n            records = records.reset_index(drop=True)\n\n            dicom = pydicom.dcmread(f\"{self.image_dir}/{image_id}.dicom\")\n\n            image = dicom.pixel_array\n\n            intercept = dicom.RescaleIntercept if \"RescaleIntercept\" in dicom else 0.0\n            slope = dicom.RescaleSlope if \"RescaleSlope\" in dicom else 1.0\n\n            if slope != 1:\n                image = slope * image.astype(np.float64)\n                image = image.astype(np.int16)\n\n            image += np.int16(intercept)        \n\n            image = np.stack([image, image, image])\n            image = image.astype('float32')\n            image = image - image.min()\n            image = image / image.max()\n            image = image * 255.0\n            image = image.transpose(1,2,0)\n\n            if self.transforms:\n                sample = {\n                    'image': image,\n                }\n                sample = self.transforms(**sample)\n                image = sample['image']\n\n            return image, image_id\n    \n    def __len__(self):\n        return self.image_ids.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom Image Augmentaion with Albumentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def dilation(img): # custom image processing function\n    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, tuple(np.random.randint(1, 6, 2)))\n    img = cv2.dilate(img, kernel, iterations=1)\n    return img\n\nclass Dilation(ImageOnlyTransform):\n    def apply(self, img, **params):\n        return dilation(img)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:43.9012Z","iopub.status.busy":"2021-02-17T09:48:43.900349Z","iopub.status.idle":"2021-02-17T09:48:43.9034Z","shell.execute_reply":"2021-02-17T09:48:43.902833Z"},"papermill":{"duration":0.034502,"end_time":"2021-02-17T09:48:43.903545","exception":false,"start_time":"2021-02-17T09:48:43.869043","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Albumentations\ndef get_train_transform():\n    return A.Compose([\n        A.Flip(0.5),\n        A.ShiftScaleRotate(scale_limit=0.1, rotate_limit=45, p=0.25),\n        A.LongestMaxSize(max_size=800, p=1.0),\n        Dilation(),\n        # FasterRCNN will normalize.\n        A.Normalize(mean=(0, 0, 0), std=(1, 1, 1), max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_valid_transform():\n    return A.Compose([\n        A.Normalize(mean=(0, 0, 0), std=(1, 1, 1), max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\ndef get_test_transform():\n    return A.Compose([\n        A.Normalize(mean=(0, 0, 0), std=(1, 1, 1), max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0)\n    ])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025268,"end_time":"2021-02-17T09:48:43.953282","exception":false,"start_time":"2021-02-17T09:48:43.928014","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Model\n\nLets import Faster RCNN from timm\n\n\n## Architecture\n\nThe architecture of Faster R-CNN is complex because it has several moving parts. We’ll start with a high level overview, and then go over the details for each of the components.\n\nIt all starts with an image, from which we want to obtain:\n\n    a list of bounding boxes.\n    a label assigned to each bounding box.\n    a probability for each label and bounding box.\n    \n![](https://tryolabs.com/blog/images/blog/post-images/2018-01-18-faster-rcnn/fasterrcnn-architecture.b9035cba.png)    \n\n\nThe input images are represented as Height×Width×Depth\\mathit{Height} \\times \\mathit{Width} \\times \\mathit{Depth}Height×Width×Depth tensors (multidimensional arrays), which are passed through a pre-trained CNN up until an intermediate layer, ending up with a convolutional feature map. We use this as a feature extractor for the next part.\n\nThis technique is very commonly used in the context of Transfer Learning, especially for training a classifier on a small dataset using the weights of a network trained on a bigger dataset. We’ll take a deeper look at this in the following sections.\n\nNext, we have what is called a Region Proposal Network (RPN, for short). Using the features that the CNN computed, it is used to find up to a predefined number of regions (bounding boxes), which may contain objects.\n\nProbably the hardest issue with using Deep Learning (DL) for object detection is generating a variable-length list of bounding boxes. When modeling deep neural networks, the last block is usually a fixed sized tensor output (except when using Recurrent Neural Networks, but that is for another post). For example, in image classification, the output is a (N,)(N,)(N,) shaped tensor, with NNN being the number of classes, where each scalar in location iii contains the probability of that image being labeli\\mathit{label}_ilabel​i​​.\n\nThe variable-length problem is solved in the RPN by using anchors: fixed sized reference bounding boxes which are placed uniformly throughout the original image. Instead of having to detect where objects are, we model the problem into two parts. For every anchor, we ask:\n\n    Does this anchor contain a relevant object?\n    How would we adjust this anchor to better fit the relevant object?\n\nThis is probably getting confusing, but fear not, we’ll dive into this below.\n\nAfter having a list of possible relevant objects and their locations in the original image, it becomes a more straightforward problem to solve. Using the features extracted by the CNN and the bounding boxes with relevant objects, we apply Region of Interest (RoI) Pooling and extract those features which would correspond to the relevant objects into a new tensor.\n\nFinally, comes the R-CNN module, which uses that information to:\n\n    Classify the content in the bounding box (or discard it, using “background” as a label).\n    Adjust the bounding box coordinates (so it better fits the object).\n\nObviously, some major bits of information are missing, but that’s basically the general idea of how Faster R-CNN works. Next, we’ll go over the details on both the architecture and loss/training for each of the components\n\n\nhttps://www.alegion.com/faster-r-cnn - For further read"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:44.010289Z","iopub.status.busy":"2021-02-17T09:48:44.009443Z","iopub.status.idle":"2021-02-17T09:48:51.275811Z","shell.execute_reply":"2021-02-17T09:48:51.276313Z"},"papermill":{"duration":7.298364,"end_time":"2021-02-17T09:48:51.27651","exception":false,"start_time":"2021-02-17T09:48:43.978146","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:51.330312Z","iopub.status.busy":"2021-02-17T09:48:51.329457Z","iopub.status.idle":"2021-02-17T09:48:51.333542Z","shell.execute_reply":"2021-02-17T09:48:51.332888Z"},"papermill":{"duration":0.032401,"end_time":"2021-02-17T09:48:51.333673","exception":false,"start_time":"2021-02-17T09:48:51.301272","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"num_classes = 15 # 14 Classes + 1 background\n\n# get number of input features for the classifier\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n\n# replace the pre-trained head with a new one\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:51.778495Z","iopub.status.busy":"2021-02-17T09:48:51.77756Z","iopub.status.idle":"2021-02-17T09:48:51.792018Z","shell.execute_reply":"2021-02-17T09:48:51.792559Z"},"papermill":{"duration":0.435385,"end_time":"2021-02-17T09:48:51.792726","exception":false,"start_time":"2021-02-17T09:48:51.357341","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\ntrain_dataset = VinBigDataset(train_df, DIR_TRAIN, get_train_transform())\nvalid_dataset = VinBigDataset(valid_df, DIR_TRAIN, get_valid_transform())\n\n\n# split the dataset in train and test set\nindices = torch.randperm(len(train_dataset)).tolist()\n# Create train and validate data loader\ntrain_data_loader = DataLoader(\n    train_dataset,\n    batch_size=8,\n    shuffle=True,\n    num_workers=4,\n    collate_fn=collate_fn\n)\n\nvalid_data_loader = DataLoader(\n    valid_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=4,\n    collate_fn=collate_fn\n)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:48:51.854455Z","iopub.status.busy":"2021-02-17T09:48:51.8537Z","iopub.status.idle":"2021-02-17T09:49:52.005304Z","shell.execute_reply":"2021-02-17T09:49:52.005891Z"},"papermill":{"duration":60.188545,"end_time":"2021-02-17T09:49:52.006104","exception":false,"start_time":"2021-02-17T09:48:51.817559","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Train dataset sample\nimages, targets, image_ids = next(iter(train_data_loader))\nimages = list(image.to(device) for image in images)\ntargets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\nfor number in random.sample([1,2,3],3):\n  boxes = targets[number]['boxes'].cpu().numpy().astype(np.int32)\n  img = images[number].permute(1,2,0).cpu().numpy()\n  labels= targets[number]['labels'].cpu().numpy().astype(np.int32)\n  fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n  for i in range(len(boxes)):\n      img = cv2.rectangle(img,(boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize),(255,0,0),2)\n      #print(le.inverse_transform([labels[i]-1])[0])\n      #print(label_to_name(labels[i]), (int(boxes[i][0]), int(boxes[i][1])))\n      img = cv2.putText(img, label_to_name(labels[i]), (int(boxes[i][0]), int(boxes[i][1])), cv2.FONT_HERSHEY_TRIPLEX,1, (255,0,0), 2, cv2.LINE_AA)\n\n  ax.set_axis_off()\n  ax.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:49:52.095256Z","iopub.status.busy":"2021-02-17T09:49:52.094226Z","iopub.status.idle":"2021-02-17T09:49:52.097496Z","shell.execute_reply":"2021-02-17T09:49:52.096813Z"},"papermill":{"duration":0.051248,"end_time":"2021-02-17T09:49:52.097653","exception":false,"start_time":"2021-02-17T09:49:52.046405","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class Averager:\n    def __init__(self):\n        self.current_total = 0.0\n        self.iterations = 0.0\n\n    def send(self, value):\n        self.current_total += value\n        self.iterations += 1\n\n    @property\n    def value(self):\n        if self.iterations == 0:\n            return 0\n        else:\n            return 1.0 * self.current_total / self.iterations\n\n    def reset(self):\n        self.current_total = 0.0\n        self.iterations = 0.0","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:49:52.189886Z","iopub.status.busy":"2021-02-17T09:49:52.189031Z","iopub.status.idle":"2021-02-17T09:49:52.259841Z","shell.execute_reply":"2021-02-17T09:49:52.259186Z"},"papermill":{"duration":0.120826,"end_time":"2021-02-17T09:49:52.259998","exception":false,"start_time":"2021-02-17T09:49:52.139172","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"model.to(device)\nparams = [p for p in model.parameters() if p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=4, gamma=0.1)\n\nnum_epochs =  2 #Low epoch to save GPU time","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T09:49:52.353657Z","iopub.status.busy":"2021-02-17T09:49:52.35258Z","iopub.status.idle":"2021-02-17T12:03:54.471141Z","shell.execute_reply":"2021-02-17T12:03:54.471682Z"},"papermill":{"duration":8042.170889,"end_time":"2021-02-17T12:03:54.471881","exception":false,"start_time":"2021-02-17T09:49:52.300992","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"loss_hist = Averager()\nitr = 1\nlossHistoryiter = []\nlossHistoryepoch = []\n\nimport time\nstart = time.time()\n\nfor epoch in range(num_epochs):\n    loss_hist.reset()\n    \n    for images, targets, image_ids in train_data_loader:\n        \n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n        loss_dict = model(images, targets)  \n        \n        losses = sum(loss for loss in loss_dict.values())\n        loss_value = losses.item()\n\n        loss_hist.send(loss_value)\n        lossHistoryiter.append(loss_value)\n        optimizer.zero_grad()\n        losses.backward()\n        optimizer.step()\n\n        if itr % 50 == 0:\n            print(f\"Iteration #{itr} loss: {loss_value}\")\n\n        itr += 1\n    \n    # update the learning rate\n    if lr_scheduler is not None:\n        lr_scheduler.step()\n    lossHistoryepoch.append(loss_hist.value)\n    print(f\"Epoch #{epoch} loss: {loss_hist.value}\")   \n    \nend = time.time()\nhours, rem = divmod(end-start, 3600)\nminutes, seconds = divmod(rem, 60)\nprint(\"Time taken to Train the model :{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T12:03:54.574532Z","iopub.status.busy":"2021-02-17T12:03:54.573838Z","iopub.status.idle":"2021-02-17T12:03:55.174996Z","shell.execute_reply":"2021-02-17T12:03:55.175561Z"},"papermill":{"duration":0.659383,"end_time":"2021-02-17T12:03:55.175739","exception":false,"start_time":"2021-02-17T12:03:54.516356","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import plotly.graph_objects as go\n\nx = [i for i in range(num_epochs)]\ny = lossHistoryepoch\nfig = go.Figure()\nfig.add_trace(go.Scatter(x=x,y=y,\n                    mode='lines',\n                    name='lines'))\n\nfig.update_layout(title='Loss vs Epochs',\n                   xaxis_title='Epochs',\n                   yaxis_title='Loss')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T12:03:55.277254Z","iopub.status.busy":"2021-02-17T12:03:55.276625Z","iopub.status.idle":"2021-02-17T12:03:55.29764Z","shell.execute_reply":"2021-02-17T12:03:55.296704Z"},"papermill":{"duration":0.075165,"end_time":"2021-02-17T12:03:55.297802","exception":false,"start_time":"2021-02-17T12:03:55.222637","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"DIR_TEST = f'{DIR_INPUT}/test'\ntest_df = pd.read_csv(f'{DIR_INPUT}/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels =  targets[1]['labels'].cpu().numpy()\nmodel.eval()\ncpu_device = torch.device(\"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T12:03:55.606564Z","iopub.status.busy":"2021-02-17T12:03:55.605122Z","iopub.status.idle":"2021-02-17T12:03:55.609242Z","shell.execute_reply":"2021-02-17T12:03:55.60871Z"},"papermill":{"duration":0.055186,"end_time":"2021-02-17T12:03:55.609373","exception":false,"start_time":"2021-02-17T12:03:55.554187","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"test_dataset = VinBigDataset(test_df, DIR_TEST, get_test_transform(),\"Test\")\n\ntest_data_loader = DataLoader(\n    test_dataset,\n    batch_size=8,\n    shuffle=False,\n    num_workers=1,\n    drop_last=False,\n    collate_fn=collate_fn\n)\n","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T12:03:55.706798Z","iopub.status.busy":"2021-02-17T12:03:55.706207Z","iopub.status.idle":"2021-02-17T12:03:55.710193Z","shell.execute_reply":"2021-02-17T12:03:55.709664Z"},"papermill":{"duration":0.055415,"end_time":"2021-02-17T12:03:55.71037","exception":false,"start_time":"2021-02-17T12:03:55.654955","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def format_prediction_string(labels, boxes, scores):\n    pred_strings = []\n    for j in zip(labels, scores, boxes):\n        pred_strings.append(\"{0} {1:.4f} {2} {3} {4} {5}\".format(\n            j[0], j[1], j[2][0], j[2][1], j[2][2], j[2][3]))\n\n    return \" \".join(pred_strings)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.046043,"end_time":"2021-02-17T12:03:55.802003","exception":false,"start_time":"2021-02-17T12:03:55.75596","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Sample Test Inputs"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T12:03:55.901809Z","iopub.status.busy":"2021-02-17T12:03:55.900548Z","iopub.status.idle":"2021-02-17T12:04:18.014059Z","shell.execute_reply":"2021-02-17T12:04:18.013553Z"},"papermill":{"duration":22.16632,"end_time":"2021-02-17T12:04:18.014231","exception":false,"start_time":"2021-02-17T12:03:55.847911","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Test dataset sample\nimages, image_ids = next(iter(test_data_loader))\nimages = list(image.to(device) for image in images)\n\nfor number in random.sample([1,2,3],3):\n  img = images[number].permute(1,2,0).cpu().numpy()\n  #labels= targets[number]['labels'].cpu().numpy().astype(np.int32)\n  fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n  ax.set_axis_off()\n  ax.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.0559,"end_time":"2021-02-17T13:10:19.598737","exception":false,"start_time":"2021-02-17T13:10:19.542837","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Sample Outputs"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T13:10:19.726227Z","iopub.status.busy":"2021-02-17T13:10:19.724896Z","iopub.status.idle":"2021-02-17T13:10:41.378092Z","shell.execute_reply":"2021-02-17T13:10:41.377525Z"},"papermill":{"duration":21.723567,"end_time":"2021-02-17T13:10:41.378242","exception":false,"start_time":"2021-02-17T13:10:19.654675","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"images, image_ids = next(iter(test_data_loader))\nimages = list(img.to(device) for img in images)\n\noutputs = model(images)\noutputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]\n\n\nboxes = outputs[0]['boxes'].cpu().detach().numpy().astype(np.int32)\nimg = images[0].permute(1,2,0).cpu().detach().numpy()\nlabels= outputs[0]['labels'].cpu().detach().numpy().astype(np.int32)\nscore = outputs[0]['scores']\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nimg = cv2.cvtColor(np.float32(img), cv2.COLOR_RGB2BGR)\nfor i in range(len(boxes)):\n  img = cv2.rectangle(img,(boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize),(255,0,0),20)\n  #print(le.inverse_transform([labels[i]-1])[0])\n  #print(label_to_name(labels[i]), (boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize))\n  img = cv2.putText(img, label_to_name(labels[i]), (int(boxes[i][0]), int(boxes[i][1])), cv2.FONT_HERSHEY_TRIPLEX,3, (255,0,0), 3, cv2.LINE_AA)\n\nax.set_axis_off()\nax.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T13:10:41.505156Z","iopub.status.busy":"2021-02-17T13:10:41.504333Z","iopub.status.idle":"2021-02-17T13:10:43.754295Z","shell.execute_reply":"2021-02-17T13:10:43.753252Z"},"papermill":{"duration":2.313757,"end_time":"2021-02-17T13:10:43.754436","exception":false,"start_time":"2021-02-17T13:10:41.440679","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"\nimages = list(img.to(device) for img in images)\n\noutputs = model(images)\noutputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]\n\n\nboxes = outputs[1]['boxes'].cpu().detach().numpy().astype(np.int32)\nimg = images[1].permute(1,2,0).cpu().detach().numpy()\nlabels= outputs[1]['labels'].cpu().detach().numpy().astype(np.int32)\nscore = outputs[1]['scores']\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nimg = cv2.cvtColor(np.float32(img), cv2.COLOR_RGB2BGR)\nfor i in range(len(boxes)):\n  img = cv2.rectangle(img,(boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize),(255,0,0),20)\n  #print(le.inverse_transform([labels[i]-1])[0])\n  #print(label_to_name(labels[i]), (boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize))\n  img = cv2.putText(img, label_to_name(labels[i]), (int(boxes[i][0]), int(boxes[i][1])), cv2.FONT_HERSHEY_TRIPLEX,3, (255,0,0), 3, cv2.LINE_AA)\n\n\nax.set_axis_off()\nax.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-17T13:10:43.900044Z","iopub.status.busy":"2021-02-17T13:10:43.899227Z","iopub.status.idle":"2021-02-17T13:10:45.765596Z","shell.execute_reply":"2021-02-17T13:10:45.764807Z"},"papermill":{"duration":1.942352,"end_time":"2021-02-17T13:10:45.765797","exception":false,"start_time":"2021-02-17T13:10:43.823445","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"\nimages = list(img.to(device) for img in images)\n\noutputs = model(images)\noutputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]\n\n\nboxes = outputs[2]['boxes'].cpu().detach().numpy().astype(np.int32)\nimg = images[2].permute(1,2,0).cpu().detach().numpy()\nlabels= outputs[2]['labels'].cpu().detach().numpy().astype(np.int32)\nscore = outputs[2]['scores']\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nimg = cv2.cvtColor(np.float32(img), cv2.COLOR_RGB2BGR)\nfor i in range(len(boxes)):\n  img = cv2.rectangle(img,(boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize),(255,0,0),20)\n  #print(le.inverse_transform([labels[i]-1])[0])\n  #print(label_to_name(labels[i]), (boxes[i][0]+paddingSize,boxes[i][1]+paddingSize),(boxes[i][2]+paddingSize,boxes[i][3]+paddingSize))\n  img = cv2.putText(img, label_to_name(labels[i]), (int(boxes[i][0]), int(boxes[i][1])), cv2.FONT_HERSHEY_TRIPLEX,3, (255,0,0), 3, cv2.LINE_AA)\n\nax.set_axis_off()\nax.imshow(img)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}