{"cells":[{"metadata":{},"cell_type":"markdown","source":"

Learnable Histograms: Statistical Context Features for Deep Neural Networks

\n\n---\n

ABSTRACT

\n

Statistical features, such as histogram, Bag-of-Words (BoW)\nand Fisher Vector, were commonly used with hand-crafted features in\nconventional classification methods, but attract less attention since the\npopularity of deep learning methods. In this paper, we propose a learnable histogram layer, which learns histogram features within deep neural\nnetworks in end-to-end training. Such a layer is able to back-propagate\n(BP) errors, learn optimal bin centers and bin widths, and be jointly\noptimized with other layers in deep networks during training. Two vision problems, semantic segmentation and object detection, are explored\nby integrating the learnable histogram layer into deep networks, which\nshow that the proposed layer could be well generalized to different applications. In-depth investigations are conducted to provide insights on\nthe newly introduced layer.\n

\n\n PAPER LINK \n"},{"metadata":{},"cell_type":"markdown","source":"

\n Context Features play a crucial role in many vision classification problems such as semantic segmentations, object detection, pose estimation, etc.

\n Context features could be mainly categorized into statistical and non-statistical\nones depending on whether they abandon the spatial orders of the context information. On the one hand, for most deep learning methods that gain increasing attention in recent years, non-statistical context features dominate

\n On the other hand, statistical context features were mostly used in conventional classification methods with hand-crafted features. Commonly used statistical features include histogram, Bag-of-Words (BoW), Fisher vector, \nSecond-order pooling, etc

\n In this paper, Histogram, a statistical feature of image vectors is represented in the form of convolutions of feature vectors. Unlike existing deep learning methods that treat statistical operations as a\nseparate module, this proposed histogram layer is able to back-propagate (BP)\nerrors and learn optimal bin centers and bin width during training. Such properties make it possible to be integrated into neural networks and end-to-end\ntrained. In this way, the appearance and statistical features in a neural network\ncould effectively adapt each other and thus lead to better classification accuracy\n

"},{"metadata":{},"cell_type":"markdown","source":"

\nMY QUEST\n

\n\n

\nWhen I first Started with CASSAVA IMAGE CLASSIFICATION competition, I came across some comprehensive and extensive EDA notebooks like this and this , where they try to cluster images and understand them using their pixel statistics. I was surprised by how much we can understand about images from their pixel values and histograms alone.

\n I then tried to find out if there are any ways where we can combine this statistical information and the Deep Learning models and get the best of both worlds. On my Exploration, I found this Amazing paper!

\n This paper provides us a method to combine these image statistics with CNN Feature vectors. I encourage you to read the paper since it explains the approach in a clear manner. The main approach involves making a LEARNABLE HISTOGRAM LAYER and using it in the model.

\n The Paper was originally made for OBJECT DETECTION and SEMANTIC SEGMENTATION, but I have tried use it in the classification task as well. The original layer is same in all the tasks so feel free to use it in your model. (After you upvote😉)\n

"},{"metadata":{},"cell_type":"markdown","source":"## IMPORTING LIBRARIES AND SETTINGS"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# installing timm\n!pip install -q timm\n\nimport os\nimport cv2\nimport copy\nimport time\nimport random\nimport pickle\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nimport torchvision\nfrom torchvision import models\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.cuda import amp\nfrom tqdm.notebook import tqdm\n\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.utils import class_weight\n\nfrom tqdm.notebook import tqdm\nfrom collections import defaultdict\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nimport timm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# defining paths\nROOT_DIR = \"../input/cassava-leaf-disease-classification\"\nTRAIN_DIR = \"../input/cassava-leaf-disease-classification/train_images\"\nTEST_DIR = \"../input/cassava-leaf-disease-classification/test_images\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CFG - Here we can change different options\nclass CFG:\n model_name = 'tf_efficientnet_b4_ns'\n img_size = 512\n scheduler = 'CosineAnnealingLR'\n T_max = 10\n T_0 = 10\n lr = 1e-5\n min_lr = 1e-7\n batch_size = 20\n weight_decay = 1e-6\n seed = 42\n num_classes = 5\n num_epochs = 3\n n_fold = 5\n smoothing = 0.2\n device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\") \n \n # THIS IS SOMETHING NEW. We will see where this is used later\n Binsize = 100\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SEEDING\n\ndef set_seed(seed = 42):\n '''Sets the seed of the entire notebook so results are the same every time we run.\n This is for REPRODUCIBILITY.'''\n np.random.seed(seed)\n random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n # When running on the CuDNN backend, two further options must be set\n torch.backends.cudnn.deterministic = True\n # Set a fixed value for the hash seed\n os.environ['PYTHONHASHSEED'] = str(seed)\n \nset_seed(CFG.seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DATASET "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.read_csv(f\"{ROOT_DIR}/train.csv\")\n\nskf = StratifiedKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor fold, ( _, val_) in enumerate(skf.split(X=df, y=df.label)):\n df.loc[val_ , \"kfold\"] = int(fold)\n \ndf['kfold'] = df['kfold'].astype(int)\n\nclass CassavaLeafDataset(nn.Module):\n def __init__(self, root_dir, df, transforms=None):\n self.root_dir = root_dir\n self.df = df\n self.transforms = transforms\n \n def __len__(self):\n return len(self.df)\n \n def __getitem__(self, index):\n img_path = os.path.join(self.root_dir, self.df.iloc[index, 0])\n img = cv2.imread(img_path)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n label = self.df.iloc[index, 1]\n \n if self.transforms:\n img = self.transforms(image=img)[\"image\"]\n \n return img, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## AUGMENTATIONS"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"data_transforms = {\n \"train\": A.Compose([\n A.RandomResizedCrop(CFG.img_size, CFG.img_size),\n A.Transpose(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.ShiftScaleRotate(p=0.5),\n A.HueSaturationValue(\n hue_shift_limit=0.2, \n sat_shift_limit=0.2, \n val_shift_limit=0.2, \n p=0.5\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n A.CoarseDropout(p=0.5),\n A.Cutout(p=0.5),\n ToTensorV2()], p=1.),\n \n \"valid\": A.Compose([\n A.CenterCrop(CFG.img_size, CFG.img_size, p=1.),\n A.Resize(CFG.img_size, CFG.img_size),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n ToTensorV2()], p=1.)\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DEFINING LOSS\n\n

\n Here I use Taylor Cross Entropy and Label Smoothing Combo loss. REFERENCE

"},{"metadata":{"trusted":true},"cell_type":"code","source":"# implementations reference - https://github.com/CoinCheung/pytorch-loss/blob/master/pytorch_loss/taylor_softmax.py\n# paper - https://www.ijcai.org/Proceedings/2020/0305.pdf\n\nclass TaylorSoftmax(nn.Module):\n\n def __init__(self, dim=1, n=2):\n super(TaylorSoftmax, self).__init__()\n assert n % 2 == 0\n self.dim = dim\n self.n = n\n\n def forward(self, x):\n \n fn = torch.ones_like(x)\n denor = 1.\n for i in range(1, self.n+1):\n denor *= i\n fn = fn + x.pow(i) / denor\n out = fn / fn.sum(dim=self.dim, keepdims=True)\n return out\n\nclass LabelSmoothingLoss(nn.Module):\n\n def __init__(self, classes, smoothing=0.0, dim=-1): \n super(LabelSmoothingLoss, self).__init__() \n self.confidence = 1.0 - smoothing \n self.smoothing = smoothing \n self.cls = classes \n self.dim = dim \n def forward(self, pred, target): \n \"\"\"Taylor Softmax and log are already applied on the logits\"\"\"\n #pred = pred.log_softmax(dim=self.dim) \n with torch.no_grad(): \n true_dist = torch.zeros_like(pred) \n true_dist.fill_(self.smoothing / (self.cls - 1)) \n true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence) \n return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))\n \n\nclass TaylorCrossEntropyLoss(nn.Module):\n\n def __init__(self, n=2, ignore_index=-1, reduction='mean', smoothing=0.2):\n super(TaylorCrossEntropyLoss, self).__init__()\n assert n % 2 == 0\n self.taylor_softmax = TaylorSoftmax(dim=1, n=n)\n self.reduction = reduction\n self.ignore_index = ignore_index\n self.lab_smooth = LabelSmoothingLoss(CFG.num_classes, smoothing=smoothing)\n\n def forward(self, logits, labels):\n\n log_probs = self.taylor_softmax(logits).log()\n #loss = F.nll_loss(log_probs, labels, reduction=self.reduction,\n # ignore_index=self.ignore_index)\n loss = self.lab_smooth(log_probs, labels)\n return loss\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TRAINING FUNCTIONS"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def train_model(model, criterion, optimizer, scheduler, num_epochs, dataloaders, dataset_sizes, device, fold):\n start = time.time()\n best_model_wts = copy.deepcopy(model.state_dict())\n best_acc = 0.0\n history = defaultdict(list)\n scaler = amp.GradScaler()\n\n for epoch in range(1,num_epochs+1):\n print('Epoch {}/{}'.format(epoch, num_epochs))\n print('-' * 10)\n\n # Each epoch has a training and validation phase\n for phase in ['train','valid']:\n if(phase == 'train'):\n model.train() # Set model to training mode\n else:\n model.eval() # Set model to evaluation mode\n \n running_loss = 0.0\n running_corrects = 0.0\n \n # Iterate over data\n for inputs,labels in tqdm(dataloaders[phase]):\n inputs = inputs.to(CFG.device)\n labels = labels.to(CFG.device)\n\n # zero the parameter gradients\n optimizer.zero_grad()\n\n # forward\n # track history if only in train\n with torch.set_grad_enabled(phase == 'train'):\n with amp.autocast():\n outputs = model(inputs)\n _, preds = torch.max(outputs,1)\n loss = criterion(outputs, labels)\n\n # backward + optimize only if in training phase\n if phase == 'train':\n scaler.scale(loss).backward()\n scaler.step(optimizer)\n scaler.update()\n\n\n running_loss += loss.item()*inputs.size(0)\n running_corrects += torch.sum(preds == labels.data).double().item()\n\n \n epoch_loss = running_loss/dataset_sizes[phase]\n epoch_acc = running_corrects/dataset_sizes[phase]\n\n history[phase + ' loss'].append(epoch_loss)\n history[phase + ' acc'].append(epoch_acc)\n\n if phase == 'train' and scheduler != None:\n scheduler.step()\n\n print('{} Loss: {:.4f} Acc: {:.4f}'.format(\n phase, epoch_loss, epoch_acc))\n \n # deep copy the model\n if phase=='valid' and epoch_acc >= best_acc:\n best_acc = epoch_acc\n best_model_wts = copy.deepcopy(model.state_dict())\n PATH = f\"Fold{fold}_{best_acc}_epoch{epoch}.bin\"\n torch.save(model.state_dict(), PATH)\n\n print()\n\n end = time.time()\n time_elapsed = end - start\n print('Training complete in {:.0f}h {:.0f}m {:.0f}s'.format(\n time_elapsed // 3600, (time_elapsed % 3600) // 60, (time_elapsed % 3600) % 60))\n print(\"Best Accuracy \",best_acc)\n\n # load best model weights\n model.load_state_dict(best_model_wts)\n return model, history, best_acc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def run_fold(model, criterion, optimizer, scheduler, device, fold, num_epochs=10):\n valid_df = df[df.kfold == fold]\n train_df = df[df.kfold != fold]\n \n train_data = CassavaLeafDataset(TRAIN_DIR, train_df, transforms=data_transforms[\"train\"])\n valid_data = CassavaLeafDataset(TRAIN_DIR, valid_df, transforms=data_transforms[\"valid\"])\n \n dataset_sizes = {\n 'train' : len(train_data),\n 'valid' : len(valid_data)\n }\n \n train_loader = DataLoader(dataset=train_data, batch_size=CFG.batch_size, num_workers=4, pin_memory=True, shuffle=True)\n valid_loader = DataLoader(dataset=valid_data, batch_size=CFG.batch_size, num_workers=4, pin_memory=True, shuffle=False)\n \n dataloaders = {\n 'train' : train_loader,\n 'valid' : valid_loader\n }\n\n model, history, best_acc = train_model(model, criterion, optimizer, scheduler, num_epochs, dataloaders, dataset_sizes, device, fold)\n \n return model, history, best_acc\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def fetch_scheduler(optimizer):\n if CFG.scheduler == 'CosineAnnealingLR':\n scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=CFG.T_max, eta_min=CFG.min_lr)\n elif CFG.scheduler == 'CosineAnnealingWarmRestarts':\n scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG.T_0, T_mult=1, eta_min=CFG.min_lr)\n elif CFG.scheduler == None:\n return None\n \n return scheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## A LOOK AT ORIGINAL MODEL\n\n

\n Lets take a look at the journey of our image when it does a pass through our EFFICIENTNET-B4-NS.

\n We will use the library TORCHSUMMARY \n

"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# EXPAND THE OUTPUT TO SEE THE FULL SUMMARY\n\n# Getting the model - standard code\nmodel = timm.create_model(CFG.model_name, pretrained=True)\nnum_features = model.classifier.in_features\nmodel.classifier = nn.Linear(num_features, CFG.num_classes)\n\n# installing torch summary\n!pip install -q torchsummary\n\n# import it\nfrom torchsummary import summary\n\n# use it!\nsummary(model, (3, 512,512), device='cpu')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

\n We will apply the HistNet Layer to our classification task by appending a feature matrix between 418 and 419 layers (Refer the above summary) \n

"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/images/arch.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## HISTOGRAM LAYER\n\n

\n The Histogram layer is implemented as a separate Module which will be used in our main Model\n

"},{"metadata":{"trusted":true},"cell_type":"code","source":"class HistModule(nn.Module):\n \n def __init__(self, in_shape=(1792, 16,16), B=6):\n \n super(HistModule, self).__init__()\n \n # making the feature vector shape (5,16,16)\n self.preconv = nn.Conv2d(1792, 5, kernel_size=(1,1))\n \n # CONV1\n \n # Filter for class 0\n self.conv1_0 = nn.Conv2d(5, B, kernel_size=(1,1))\n # initializing weight appropriately + freezing\n self.conv1_0.weight.data.fill_(0)\n self.conv1_0.weight.data[:, 0, :, :] = 1\n self.conv1_0.weight.data.requires_grad =False\n \n # Filter for class 1\n self.conv1_1 = nn.Conv2d(5, B, kernel_size=(1,1))\n # initializing weight appropriately + freezing\n self.conv1_1.weight.data.fill_(0)\n self.conv1_1.weight.data[:, 1, :, :] = 1\n self.conv1_1.weight.data.requires_grad =False\n \n # Filter for class 2\n self.conv1_2 = nn.Conv2d(5, B, kernel_size=(1,1))\n # initializing weight appropriately + freezing\n self.conv1_2.weight.data.fill_(0)\n self.conv1_2.weight.data[:, 2, :, :] = 1\n self.conv1_2.weight.data.requires_grad =False\n \n # Filter for class 3\n self.conv1_3 = nn.Conv2d(5, B, kernel_size=(1,1))\n # initializing weight appropriately + freezing\n self.conv1_3.weight.data.fill_(0)\n self.conv1_3.weight.data[:, 3, :, :] = 1\n self.conv1_3.weight.data.requires_grad =False\n \n # Filter for class 4\n self.conv1_4 = nn.Conv2d(5, B, kernel_size=(1,1))\n # initializing weight appropriately + freezing\n self.conv1_4.weight.data.fill_(0)\n self.conv1_4.weight.data[:, 4, :, :] = 1\n self.conv1_4.weight.data.requires_grad =False\n \n \n # CONV 2\n \n self.conv2 = nn.Conv2d(5*B, 5*B, kernel_size=(1,1))\n self.conv2.bias.data.fill_(1)\n self.conv2.bias.data.requires_grad = False\n \n \n def forward(self, x):\n # get to class size\n inp = self.preconv(x)\n \n # classwise convs\n cls0 = self.conv1_0(inp)\n cls1 = self.conv1_1(inp)\n cls2 = self.conv1_2(inp)\n cls3 = self.conv1_3(inp)\n cls4 = self.conv1_4(inp)\n \n # concatenate\n concat = torch.cat([cls0, cls1, cls2, cls3, cls4], 1)\n concat = torch.abs(concat)\n \n # conv2\n out = self.conv2(concat)\n out = F.relu(out)\n \n # final outshape = (1792+(5*binsize), 5, 5)\n finout = torch.cat([x, out], 1)\n \n return finout\n\n# check the output shape\nhs = HistModule()\nhs(torch.rand(1,1792, 16, 16)).shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MAKING THE FINAL MODEL\n\n

\n I have loaded up my original weights for the EFFNET B4 Model and fine tuned them.
\n All the weights are preloaded except the Histogram layers which will be learned.
\n The learning rate is kept low as we are only fine tuning the model.
\n

"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MyHistEffnet(nn.Module):\n \n def __init__(self, Binsize = 6):\n # initialize parent\n super(MyHistEffnet, self).__init__()\n \n # initialize bin size\n self.bins = Binsize\n \n # make model\n self.model = timm.create_model(CFG.model_name, pretrained=True)\n num_features = self.model.classifier.in_features\n self.model.classifier = nn.Linear(num_features, CFG.num_classes)\n \n \"\"\"\n We are not passing the image through the complete model, hence we don't need the output of the last two layers.\n Lets get rid of them!\n \"\"\"\n self.modified_model = nn.Sequential(*list(self.model.children())[:-2])\n \n # defining histogram layer\n self.histmod = HistModule(B=Binsize)\n \n # THE LAST TWO LAYERS\n self.global_pool = nn.AdaptiveAvgPool2d(output_size=1)\n self.fc = nn.Linear(1792+(5*self.bins), 5)\n \n \n def forward(self, x):\n \n # get the feature vector\n ftrvec = self.modified_model(x) # 1792, 16, 16\n \n # pass through the histmodule\n x = self.histmod(ftrvec) # 1792+30, 16, 16\n \n # global average pool\n x = self.global_pool(x) # 1792,1, 1\n x = x.view(-1, 1792+(5*self.bins))\n x = self.fc(x)\n \n return x\n\n# check the output shape\nmod = MyHistEffnet(Binsize=CFG.Binsize)\nmod(torch.rand((1,3,512,512))).shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

\nI am loading my fold0 weights which originally give 0.902 validation accuracy. Lets see if we can improve on that.
\n

"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The binsize defined in the config is used here\nHistModel = MyHistEffnet(Binsize=CFG.Binsize)\nHistModel.to(CFG.device)\n\n# remember to load the weights in the \"model\" only\nHistModel.model.load_state_dict(torch.load('../input/images/Fold0_weights.bin'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# defining optimizer, criterion, scheduler\noptimizer = optim.Adam(HistModel.parameters(), lr=CFG.lr, weight_decay=CFG.weight_decay, amsgrad=False)\ncriterion = TaylorCrossEntropyLoss(n=2, smoothing=0.2)\nscheduler = fetch_scheduler(optimizer) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## START THE TRAINING!"},{"metadata":{"trusted":true},"cell_type":"code","source":"model, history, ba = run_fold(HistModel, criterion, optimizer, \n scheduler, device=CFG.device, fold=0, num_epochs=CFG.num_epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

\n The final accuracy is 0.907 which is not much better than initial 0.902, But this was just the Baseline.\n
\n We have the Binsize to be tuned, We can follow a different fine tuning strategy or we can use a different model as a backbone.\n
\n The possibilites are endless!\n

"},{"metadata":{},"cell_type":"markdown","source":"

Thank you for making it till here. If the implementation and the code helps you consider UPVOTING!

"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}