Meet the New Seresnext
\nWith ATTENTION!!!🔥
"},{"metadata":{"papermill":{"duration":0.018305,"end_time":"2021-02-18T22:27:17.279658","exception":false,"start_time":"2021-02-18T22:27:17.261353","status":"completed"},"tags":[]},"cell_type":"markdown","source":"We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how **ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks**. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 2.33x faster in compute time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision.\n\n*Bottleneck Transformers for Visual Recognition*: https://arxiv.org/abs/2101.11605\n\n"},{"metadata":{"papermill":{"duration":0.01836,"end_time":"2021-02-18T22:27:17.316545","exception":false,"start_time":"2021-02-18T22:27:17.298185","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Install Required Packages"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:17.359992Z","iopub.status.busy":"2021-02-18T22:27:17.359322Z","iopub.status.idle":"2021-02-18T22:27:27.797199Z","shell.execute_reply":"2021-02-18T22:27:27.796106Z"},"papermill":{"duration":10.461981,"end_time":"2021-02-18T22:27:27.797409","exception":false,"start_time":"2021-02-18T22:27:17.335428","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"!pip install timm adamp bottleneck-transformer-pytorch","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.023244,"end_time":"2021-02-18T22:27:27.845943","exception":false,"start_time":"2021-02-18T22:27:27.822699","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Import Packages"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:27.901074Z","iopub.status.busy":"2021-02-18T22:27:27.900263Z","iopub.status.idle":"2021-02-18T22:27:31.595537Z","shell.execute_reply":"2021-02-18T22:27:31.594784Z"},"papermill":{"duration":3.726334,"end_time":"2021-02-18T22:27:31.595668","exception":false,"start_time":"2021-02-18T22:27:27.869334","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"import os\nimport cv2\nimport copy\nimport time\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nimport torchvision\nfrom torchvision import models\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.cuda import amp\n\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.utils import class_weight\n\nfrom tqdm.notebook import tqdm\nfrom collections import defaultdict\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nimport timm\nfrom adamp import AdamP\nfrom bottleneck_transformer_pytorch import BottleStack","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:31.648463Z","iopub.status.busy":"2021-02-18T22:27:31.647227Z","iopub.status.idle":"2021-02-18T22:27:31.649648Z","shell.execute_reply":"2021-02-18T22:27:31.650076Z"},"papermill":{"duration":0.030631,"end_time":"2021-02-18T22:27:31.65021","exception":false,"start_time":"2021-02-18T22:27:31.619579","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"ROOT_DIR = \"../input/cassava-leaf-disease-classification\"\nTRAIN_DIR = \"../input/cassava-leaf-disease-classification/train_images\"\nTEST_DIR = \"../input/cassava-leaf-disease-classification/test_images\"","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.023599,"end_time":"2021-02-18T22:27:31.697203","exception":false,"start_time":"2021-02-18T22:27:31.673604","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Training Configuration"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.094972Z","iopub.status.busy":"2021-02-18T22:27:32.094214Z","iopub.status.idle":"2021-02-18T22:27:32.097432Z","shell.execute_reply":"2021-02-18T22:27:32.097001Z"},"papermill":{"duration":0.376769,"end_time":"2021-02-18T22:27:32.097558","exception":false,"start_time":"2021-02-18T22:27:31.720789","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"class CFG:\n model_name = 'seresnext50_32x4d'\n img_size = 512\n scheduler = 'CosineAnnealingLR'\n T_max = 10\n T_0 = 10\n lr = 1e-4\n min_lr = 1e-6\n batch_size = 12\n weight_decay = 1e-6\n seed = 42\n num_classes = 5\n num_epochs = 10\n n_fold = 5\n smoothing = 0.2\n device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.023577,"end_time":"2021-02-18T22:27:32.145277","exception":false,"start_time":"2021-02-18T22:27:32.1217","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Set Seed for Reproducibility"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.198752Z","iopub.status.busy":"2021-02-18T22:27:32.198221Z","iopub.status.idle":"2021-02-18T22:27:32.203806Z","shell.execute_reply":"2021-02-18T22:27:32.203378Z"},"papermill":{"duration":0.034634,"end_time":"2021-02-18T22:27:32.203923","exception":false,"start_time":"2021-02-18T22:27:32.169289","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def set_seed(seed = 42):\n '''Sets the seed of the entire notebook so results are the same every time we run.\n This is for REPRODUCIBILITY.'''\n np.random.seed(seed)\n random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n # When running on the CuDNN backend, two further options must be set\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n # Set a fixed value for the hash seed\n os.environ['PYTHONHASHSEED'] = str(seed)\n \nset_seed(CFG.seed)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.023566,"end_time":"2021-02-18T22:27:32.251241","exception":false,"start_time":"2021-02-18T22:27:32.227675","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Create Folds"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.304892Z","iopub.status.busy":"2021-02-18T22:27:32.304363Z","iopub.status.idle":"2021-02-18T22:27:32.345588Z","shell.execute_reply":"2021-02-18T22:27:32.345133Z"},"papermill":{"duration":0.070721,"end_time":"2021-02-18T22:27:32.345703","exception":false,"start_time":"2021-02-18T22:27:32.274982","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"df = pd.read_csv(f\"{ROOT_DIR}/train.csv\")\nskf = StratifiedKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor fold, ( _, val_) in enumerate(skf.split(X=df, y=df.label)):\n df.loc[val_ , \"kfold\"] = int(fold)\n \ndf['kfold'] = df['kfold'].astype(int)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025368,"end_time":"2021-02-18T22:27:32.398046","exception":false,"start_time":"2021-02-18T22:27:32.372678","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Dataset Class"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.450998Z","iopub.status.busy":"2021-02-18T22:27:32.450193Z","iopub.status.idle":"2021-02-18T22:27:32.460759Z","shell.execute_reply":"2021-02-18T22:27:32.460342Z"},"papermill":{"duration":0.03813,"end_time":"2021-02-18T22:27:32.460872","exception":false,"start_time":"2021-02-18T22:27:32.422742","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"class CassavaLeafDataset(nn.Module):\n def __init__(self, root_dir, df, transforms=None):\n self.root_dir = root_dir\n self.df = df\n self.transforms = transforms\n \n def __len__(self):\n return len(self.df)\n \n def __getitem__(self, index):\n img_path = os.path.join(self.root_dir, self.df.iloc[index, 0])\n img = cv2.imread(img_path)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n label = self.df.iloc[index, 1]\n \n if self.transforms:\n img = self.transforms(image=img)[\"image\"]\n \n return img, label","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025459,"end_time":"2021-02-18T22:27:32.513229","exception":false,"start_time":"2021-02-18T22:27:32.48777","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Augmentations & Transforms"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.575892Z","iopub.status.busy":"2021-02-18T22:27:32.575211Z","iopub.status.idle":"2021-02-18T22:27:32.578347Z","shell.execute_reply":"2021-02-18T22:27:32.577925Z"},"papermill":{"duration":0.039174,"end_time":"2021-02-18T22:27:32.578455","exception":false,"start_time":"2021-02-18T22:27:32.539281","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"data_transforms = {\n \"train\": A.Compose([\n A.RandomResizedCrop(CFG.img_size, CFG.img_size),\n A.Transpose(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.15, rotate_limit=60, p=0.5),\n A.HueSaturationValue(\n hue_shift_limit=0.2, \n sat_shift_limit=0.2, \n val_shift_limit=0.2, \n p=0.5\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n\n A.CoarseDropout(p=0.5),\n A.Cutout(p=0.5),\n ToTensorV2()], p=1.),\n \n \"valid\": A.Compose([\n A.Resize(600, 600),\n A.CenterCrop(CFG.img_size, CFG.img_size, p=1.),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n ToTensorV2()], p=1.)\n}","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025948,"end_time":"2021-02-18T22:27:32.630853","exception":false,"start_time":"2021-02-18T22:27:32.604905","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Loss Function\n\nRefer to this [Discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/211475) for this Loss Function"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.696573Z","iopub.status.busy":"2021-02-18T22:27:32.694795Z","iopub.status.idle":"2021-02-18T22:27:32.697229Z","shell.execute_reply":"2021-02-18T22:27:32.697625Z"},"papermill":{"duration":0.04175,"end_time":"2021-02-18T22:27:32.697775","exception":false,"start_time":"2021-02-18T22:27:32.656025","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"class TaylorSoftmax(nn.Module): # a new version of the old softmax, somthing new..?? \n def __init__(self, dim=1, n=2):\n super(TaylorSoftmax, self).__init__()\n assert n % 2 == 0\n self.dim = dim\n self.n = n\n\n def forward(self, x):\n fn = torch.ones_like(x)\n denor = 1.\n for i in range(1, self.n+1):\n denor *= i\n fn = fn + x.pow(i) / denor\n out = fn / fn.sum(dim=self.dim, keepdims=True)\n return out\n\nclass LabelSmoothingLoss(nn.Module): \n def __init__(self, classes=5, smoothing=0.0, dim=-1): \n super(LabelSmoothingLoss, self).__init__() \n self.confidence = 1.0 - smoothing \n self.smoothing = smoothing \n self.cls = classes \n self.dim = dim \n def forward(self, pred, target):\n with torch.no_grad():\n true_dist = torch.zeros_like(pred) \n true_dist.fill_(self.smoothing / (self.cls - 1)) \n true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence) \n return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))\n \n\nclass TaylorCrossEntropyLoss(nn.Module):\n def __init__(self, n=2, ignore_index=-1, reduction='mean', smoothing=0.2):\n super(TaylorCrossEntropyLoss, self).__init__()\n assert n % 2 == 0\n self.taylor_softmax = TaylorSoftmax(dim=1, n=n)\n self.reduction = reduction\n self.ignore_index = ignore_index\n self.lab_smooth = LabelSmoothingLoss(CFG.num_classes, smoothing=smoothing)\n\n def forward(self, logits, labels):\n log_probs = self.taylor_softmax(logits).log()\n loss = self.lab_smooth(log_probs, labels)\n return loss","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.023947,"end_time":"2021-02-18T22:27:32.745886","exception":false,"start_time":"2021-02-18T22:27:32.721939","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Training Function"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.808896Z","iopub.status.busy":"2021-02-18T22:27:32.807642Z","iopub.status.idle":"2021-02-18T22:27:32.810086Z","shell.execute_reply":"2021-02-18T22:27:32.810499Z"},"papermill":{"duration":0.040612,"end_time":"2021-02-18T22:27:32.810632","exception":false,"start_time":"2021-02-18T22:27:32.77002","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def train_model(model, criterion, optimizer, scheduler, num_epochs, dataloaders, dataset_sizes, device, fold):\n start = time.time()\n best_model_wts = copy.deepcopy(model.state_dict())\n best_acc = 0.0\n history = defaultdict(list)\n\n for epoch in range(1,num_epochs+1):\n print('Epoch {}/{}'.format(epoch, num_epochs))\n print('-' * 10)\n\n # Each epoch has a training and validation phase\n for phase in ['train','valid']:\n if(phase == 'train'):\n model.train() # Set model to training mode\n else:\n model.eval() # Set model to evaluation mode\n \n running_loss = 0.0\n running_corrects = 0.0\n \n # Iterate over data\n for inputs,labels in dataloaders[phase]:\n inputs = inputs.to(CFG.device)\n labels = labels.to(CFG.device)\n\n # forward\n # track history if only in train\n with torch.set_grad_enabled(phase == 'train'):\n outputs = model(inputs)\n _, preds = torch.max(outputs,1)\n loss = criterion(outputs, labels) # use this loss for any training statistics\n\n # backward + optimize only if in training phase\n if phase == 'train':\n # first forward-backward pass\n loss.backward()\n optimizer.first_step(zero_grad=True)\n \n # second forward-backward pass\n criterion(model(inputs), labels).backward()\n optimizer.second_step(zero_grad=True)\n\n\n running_loss += loss.item()*inputs.size(0)\n running_corrects += torch.sum(preds == labels.data).double().item()\n\n \n epoch_loss = running_loss/dataset_sizes[phase]\n epoch_acc = running_corrects/dataset_sizes[phase]\n\n history[phase + ' loss'].append(epoch_loss)\n history[phase + ' acc'].append(epoch_acc)\n\n if phase == 'train' and scheduler != None:\n scheduler.step()\n\n print('{} Loss: {:.4f} Acc: {:.4f}'.format(\n phase, epoch_loss, epoch_acc))\n \n # deep copy the model\n if phase=='valid' and epoch_acc >= best_acc:\n best_acc = epoch_acc\n best_model_wts = copy.deepcopy(model.state_dict())\n PATH = f\"Fold{fold}_{best_acc}_epoch{epoch}.bin\"\n torch.save(model.state_dict(), PATH)\n\n print()\n\n end = time.time()\n time_elapsed = end - start\n print('Training complete in {:.0f}h {:.0f}m {:.0f}s'.format(\n time_elapsed // 3600, (time_elapsed % 3600) // 60, (time_elapsed % 3600) % 60))\n print(\"Best Accuracy \",best_acc)\n\n # load best model weights\n model.load_state_dict(best_model_wts)\n return model, history","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:32.867402Z","iopub.status.busy":"2021-02-18T22:27:32.866756Z","iopub.status.idle":"2021-02-18T22:27:32.870179Z","shell.execute_reply":"2021-02-18T22:27:32.869753Z"},"papermill":{"duration":0.035347,"end_time":"2021-02-18T22:27:32.870289","exception":false,"start_time":"2021-02-18T22:27:32.834942","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def run_fold(model, criterion, optimizer, scheduler, device, fold, num_epochs=10):\n valid_df = df[df.kfold == fold]\n train_df = df[df.kfold != fold]\n \n train_data = CassavaLeafDataset(TRAIN_DIR, train_df, transforms=data_transforms[\"train\"])\n valid_data = CassavaLeafDataset(TRAIN_DIR, valid_df, transforms=data_transforms[\"valid\"])\n \n dataset_sizes = {\n 'train' : len(train_data),\n 'valid' : len(valid_data)\n }\n \n train_loader = DataLoader(dataset=train_data, batch_size=CFG.batch_size, num_workers=4, pin_memory=True, shuffle=True)\n valid_loader = DataLoader(dataset=valid_data, batch_size=CFG.batch_size, num_workers=4, pin_memory=True, shuffle=False)\n \n dataloaders = {\n 'train' : train_loader,\n 'valid' : valid_loader\n }\n\n model, history = train_model(model, criterion, optimizer, scheduler, num_epochs, dataloaders, dataset_sizes, device, fold)\n \n return model, history","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.024442,"end_time":"2021-02-18T22:27:32.919389","exception":false,"start_time":"2021-02-18T22:27:32.894947","status":"completed"},"tags":[]},"cell_type":"markdown","source":"SAM Optimizer"},{"metadata":{"papermill":{"duration":0.024223,"end_time":"2021-02-18T22:27:32.968402","exception":false,"start_time":"2021-02-18T22:27:32.944179","status":"completed"},"tags":[]},"cell_type":"markdown","source":"In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization -- including a generalization bound that we prove here -- we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that **SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels.**\n\n*Sharpness-Aware Minimization for Efficiently Improving\nGeneralization* : https://arxiv.org/abs/2010.01412\n\n"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:33.030251Z","iopub.status.busy":"2021-02-18T22:27:33.029581Z","iopub.status.idle":"2021-02-18T22:27:33.032692Z","shell.execute_reply":"2021-02-18T22:27:33.032244Z"},"papermill":{"duration":0.039303,"end_time":"2021-02-18T22:27:33.032815","exception":false,"start_time":"2021-02-18T22:27:32.993512","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"class SAM(torch.optim.Optimizer):\n def __init__(self, params, base_optimizer, rho=0.05, **kwargs):\n assert rho >= 0.0, f\"Invalid rho, should be non-negative: {rho}\"\n\n defaults = dict(rho=rho, **kwargs)\n super(SAM, self).__init__(params, defaults)\n\n self.base_optimizer = base_optimizer(self.param_groups, **kwargs)\n self.param_groups = self.base_optimizer.param_groups\n\n @torch.no_grad()\n def first_step(self, zero_grad=False):\n grad_norm = self._grad_norm()\n for group in self.param_groups:\n scale = group[\"rho\"] / (grad_norm + 1e-12)\n\n for p in group[\"params\"]:\n if p.grad is None: continue\n e_w = p.grad * scale.to(p)\n p.add_(e_w) # climb to the local maximum \"w + e(w)\"\n self.state[p][\"e_w\"] = e_w\n\n if zero_grad: self.zero_grad()\n\n @torch.no_grad()\n def second_step(self, zero_grad=False):\n for group in self.param_groups:\n for p in group[\"params\"]:\n if p.grad is None: continue\n p.sub_(self.state[p][\"e_w\"]) # get back to \"w\" from \"w + e(w)\"\n\n self.base_optimizer.step() # do the actual \"sharpness-aware\" update\n\n if zero_grad: self.zero_grad()\n\n @torch.no_grad()\n def step(self, closure=None):\n assert closure is not None, \"Sharpness Aware Minimization requires closure, but it was not provided\"\n closure = torch.enable_grad()(closure) # the closure should do a full forward-backward pass\n\n self.first_step(zero_grad=True)\n closure()\n self.second_step()\n\n def _grad_norm(self):\n shared_device = self.param_groups[0][\"params\"][0].device # put everything on the same device, in case of model parallelism\n norm = torch.norm(\n torch.stack([\n p.grad.norm(p=2).to(shared_device)\n for group in self.param_groups for p in group[\"params\"]\n if p.grad is not None\n ]),\n p=2\n )\n return norm","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.024303,"end_time":"2021-02-18T22:27:33.081454","exception":false,"start_time":"2021-02-18T22:27:33.057151","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Helper Function\n\nConverts the activation function for the entire network"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:33.136254Z","iopub.status.busy":"2021-02-18T22:27:33.135485Z","iopub.status.idle":"2021-02-18T22:27:33.139476Z","shell.execute_reply":"2021-02-18T22:27:33.138957Z"},"papermill":{"duration":0.033201,"end_time":"2021-02-18T22:27:33.139588","exception":false,"start_time":"2021-02-18T22:27:33.106387","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def convert_act_cls(model, layer_type_old, layer_type_new):\n conversion_count = 0\n for name, module in reversed(model._modules.items()):\n if len(list(module.children())) > 0:\n # recurse\n model._modules[name] = convert_act_cls(module, layer_type_old, layer_type_new)\n\n if type(module) == layer_type_old:\n layer_old = module\n layer_new = layer_type_new\n model._modules[name] = layer_new\n\n return model","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.024338,"end_time":"2021-02-18T22:27:33.188386","exception":false,"start_time":"2021-02-18T22:27:33.164048","status":"completed"},"tags":[]},"cell_type":"markdown","source":"BottleNeck Transformer"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:33.244502Z","iopub.status.busy":"2021-02-18T22:27:33.243962Z","iopub.status.idle":"2021-02-18T22:27:34.43407Z","shell.execute_reply":"2021-02-18T22:27:34.433528Z"},"papermill":{"duration":1.221037,"end_time":"2021-02-18T22:27:34.434215","exception":false,"start_time":"2021-02-18T22:27:33.213178","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"from bottleneck_transformer_pytorch import BottleStack\n\nbot_layer_1 = BottleStack(\n dim = 1024, # channels in\n fmap_size = 32, # feature map size\n dim_out = 2048, # channels out\n proj_factor = 4, # projection factor\n downsample = True, # downsample on first layer or not\n heads = 4, # number of heads\n dim_head = 128, # dimension per head, defaults to 128\n rel_pos_emb = True, # use relative positional embedding - uses absolute if False\n activation = nn.SiLU() # activation throughout the network\n)\n\nbot_layer_2 = BottleStack(\n dim = 2048, # channels in\n fmap_size = 16, # feature map size\n dim_out = 2048, # channels out\n proj_factor = 4, # projection factor\n downsample = True, # downsample on first layer or not\n heads = 4, # number of heads\n dim_head = 128, # dimension per head, defaults to 128\n rel_pos_emb = True, # use relative positional embedding - uses absolute if False\n activation = nn.SiLU() # activation throughout the network\n)\n\nbot_layer_3 = BottleStack(\n dim = 2048, # channels in\n fmap_size = 8, # feature map size\n dim_out = 2048, # channels out\n proj_factor = 4, # projection factor\n downsample = True, # downsample on first layer or not\n heads = 4, # number of heads\n dim_head = 128, # dimension per head, defaults to 128\n rel_pos_emb = True, # use relative positional embedding - uses absolute if False\n activation = nn.SiLU() # activation throughout the network\n)\n\nBotStackLayer = nn.Sequential(bot_layer_1,bot_layer_2,bot_layer_3)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025123,"end_time":"2021-02-18T22:27:34.485234","exception":false,"start_time":"2021-02-18T22:27:34.460111","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Create Model"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:34.543078Z","iopub.status.busy":"2021-02-18T22:27:34.542226Z","iopub.status.idle":"2021-02-18T22:27:43.135391Z","shell.execute_reply":"2021-02-18T22:27:43.136446Z"},"papermill":{"duration":8.625398,"end_time":"2021-02-18T22:27:43.136658","exception":false,"start_time":"2021-02-18T22:27:34.51126","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"model = timm.create_model(CFG.model_name, pretrained=True)\nnum_features = model.fc.in_features\nmodel.layer4 = BotStackLayer\nmodel.fc = nn.Linear(num_features, CFG.num_classes)\nmodel = convert_act_cls(model, nn.ReLU, nn.SiLU()) # convert ReLU activation to SiLU\nmodel.to(CFG.device);","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:43.223054Z","iopub.status.busy":"2021-02-18T22:27:43.222259Z","iopub.status.idle":"2021-02-18T22:27:43.225674Z","shell.execute_reply":"2021-02-18T22:27:43.226265Z"},"papermill":{"duration":0.050212,"end_time":"2021-02-18T22:27:43.226457","exception":false,"start_time":"2021-02-18T22:27:43.176245","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def fetch_scheduler(optimizer):\n if CFG.scheduler == 'CosineAnnealingLR':\n scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=CFG.T_max, eta_min=CFG.min_lr)\n elif CFG.scheduler == 'CosineAnnealingWarmRestarts':\n scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG.T_0, T_mult=1, eta_min=CFG.min_lr)\n elif CFG.scheduler == None:\n return None\n \n return scheduler","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.041417,"end_time":"2021-02-18T22:27:43.310879","exception":false,"start_time":"2021-02-18T22:27:43.269462","status":"completed"},"tags":[]},"cell_type":"markdown","source":"AdamP Optimizer"},{"metadata":{"papermill":{"duration":0.042018,"end_time":"2021-02-18T22:27:43.394215","exception":false,"start_time":"2021-02-18T22:27:43.352197","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Normalization techniques are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, stabilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, a phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is a crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-based GD (e.g. SGD or Adam) and (2) scale-invariant parameters. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in those benchmarks.\n\n*AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights*: https://arxiv.org/abs/2006.08217\n\n"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:43.487393Z","iopub.status.busy":"2021-02-18T22:27:43.486605Z","iopub.status.idle":"2021-02-18T22:27:43.494459Z","shell.execute_reply":"2021-02-18T22:27:43.495092Z"},"papermill":{"duration":0.059851,"end_time":"2021-02-18T22:27:43.495295","exception":false,"start_time":"2021-02-18T22:27:43.435444","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"base_optimizer = AdamP\noptimizer = SAM(model.parameters(), base_optimizer, lr=CFG.lr, weight_decay=CFG.weight_decay)\ncriterion = TaylorCrossEntropyLoss(n=2,smoothing=0.2)\nscheduler = fetch_scheduler(optimizer)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.041023,"end_time":"2021-02-18T22:27:43.577699","exception":false,"start_time":"2021-02-18T22:27:43.536676","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Run Fold: 0"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-18T22:27:43.667261Z","iopub.status.busy":"2021-02-18T22:27:43.66644Z","iopub.status.idle":"2021-02-19T04:54:44.528511Z","shell.execute_reply":"2021-02-19T04:54:44.529129Z"},"papermill":{"duration":23220.909783,"end_time":"2021-02-19T04:54:44.529331","exception":false,"start_time":"2021-02-18T22:27:43.619548","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"model, history = run_fold(model, criterion, optimizer, scheduler, device=CFG.device, fold=0, num_epochs=CFG.num_epochs)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.033319,"end_time":"2021-02-19T04:54:44.597162","exception":false,"start_time":"2021-02-19T04:54:44.563843","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Visualize Training & Validation Metrics"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-19T04:54:44.669798Z","iopub.status.busy":"2021-02-19T04:54:44.668073Z","iopub.status.idle":"2021-02-19T04:54:44.670468Z","shell.execute_reply":"2021-02-19T04:54:44.670893Z"},"papermill":{"duration":0.040898,"end_time":"2021-02-19T04:54:44.671013","exception":false,"start_time":"2021-02-19T04:54:44.630115","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"plt.style.use('fivethirtyeight')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.033392,"end_time":"2021-02-19T04:54:44.737511","exception":false,"start_time":"2021-02-19T04:54:44.704119","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Training Loss vs Validation Loss"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-19T04:54:44.823477Z","iopub.status.busy":"2021-02-19T04:54:44.822974Z","iopub.status.idle":"2021-02-19T04:54:45.035361Z","shell.execute_reply":"2021-02-19T04:54:45.035789Z"},"papermill":{"duration":0.265375,"end_time":"2021-02-19T04:54:45.03592","exception":false,"start_time":"2021-02-19T04:54:44.770545","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"fig = plt.figure(figsize=(22,8))\nplt.plot(history['train loss'], label='train loss')\nplt.plot(history['valid loss'], label='valid loss')\nplt.legend()\nplt.title('Loss Curve');","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.034923,"end_time":"2021-02-19T04:54:45.105961","exception":false,"start_time":"2021-02-19T04:54:45.071038","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Training Accuracy vs Validation Accuracy"},{"metadata":{"execution":{"iopub.execute_input":"2021-02-19T04:54:45.192911Z","iopub.status.busy":"2021-02-19T04:54:45.191881Z","iopub.status.idle":"2021-02-19T04:54:45.3582Z","shell.execute_reply":"2021-02-19T04:54:45.35861Z"},"papermill":{"duration":0.218146,"end_time":"2021-02-19T04:54:45.358757","exception":false,"start_time":"2021-02-19T04:54:45.140611","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"fig = plt.figure(figsize=(22,8))\nplt.plot(history['train acc'], label='train acc')\nplt.plot(history['valid acc'], label='valid acc')\nplt.legend()\nplt.title('Accuracy Curve');","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.063622,"end_time":"2021-02-19T04:54:45.499009","exception":false,"start_time":"2021-02-19T04:54:45.435387","status":"completed"},"tags":[]},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}