{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n
\n

Visualizing predictions with GradCAM

\n \"GradCAM\n
\n

In this notebook, I show you how different models affect the importance given to certain parts of the image. We wil build a SE_ResNeXT that has a Squeeze and Excitation module.\n\nBut first let's study GradCAM a bit...

\n
\n

GradCAM stands for Gradient-weighted class activation map. It is a technique to visualize which parts of an image guide a CNN model towards a certain prediction. GradCAM, unlike CAM, uses the gradient information flowing into the last convolutional layer of the CNN to understand each neuron for a decision of interest. To obtain the class discriminative localization map of width u and height v for any class c, we first compute the gradient of the score for the class c, yc (before the softmax) with respect to feature maps Ak of a convolutional layer. These gradients flowing back are global average-pooled to obtain the neuron importance weights ak for the target class.

\n
\n

Sources:\n

\n
\n

\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n

Table of Contents

\n 1. Importing dependencies1\n 2. Config 2\n 3. Utils3\n 4. Dataset generator4 \n 5. Loading data and visualization5 \n 6. Building model6 \n 7. Training7\n 8. GradCAM visualization8 \n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

1. Importing dependencies

","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install --upgrade -q tensorflow\n!pip install -q image-classifiers\n!pip install -q tensorflow-addons","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy as sp\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import StratifiedKFold, train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.keras.utils import Sequence\nfrom tensorflow.keras.layers import GlobalAveragePooling2D, Dense, Dropout, Input, Flatten, BatchNormalization\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n\nimport tensorflow_addons as tfa\n\nfrom classification_models.tfkeras import Classifiers\n\nfrom tqdm.notebook import tqdm\n\nimport skimage.io\nimport cv2\nfrom PIL import Image\n\nfrom functools import partial\n\nimport os, gc, time, random\nfrom datetime import datetime\n\nfrom math import ceil\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport albumentations","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

2. Config

\n\n

First, let's define a config class where we can manage our hyperparameters. I always find it very useful to have a kind of dashboard where you can tune pretty every parameter of your workflow.

","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class config:\n seed = 2020\n batch_size = 16\n img_size = 64\n num_tiles = 16\n num_classes = 6\n num_splits = 5\n num_epochs = 4\n learning_rate = 3e-3\n num_workers = 1\n verbose = True\n backbone_train_path = '../input/prostate-cancer-grade-assessment/train_images/'\n backbone_test_path = '../input/prostate-cancer-grade-assessment/test_images/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3. Utils

\n\n

For reproducibility purposes, let's seed everything.

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n\nseed_everything(config.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_axis_max_min(array, axis=0):\n one_axis = list((array != 255).sum(axis=tuple([x for x in (0, 1, 2) if x != axis])))\n axis_min = next((i for i, x in enumerate(one_axis) if x), 0)\n axis_max = len(one_axis) - next((i for i, x in enumerate(one_axis[::-1]) if x), 0)\n return axis_min, axis_max","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4. Dataset generator

\n\n

Now it is time to define our custom ImageGenerator. In this kernel, we will use the techniques used by Vasilyi of concatenating tiles of the image.\nSource: https://www.kaggle.com/vgarshin/panda-keras-baseline

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class PANDAGenerator(Sequence):\n def __init__(self, df, config, mode='fit', apply_tfms=True, shuffle=True):\n super(PANDAGenerator, self).__init__()\n \n self.image_ids = df['image_id'].reset_index(drop=True).values\n self.labels = df['isup_grade'].reset_index(drop=True).values\n \n self.config = config\n self.shuffle = shuffle\n self.mode = mode\n \n self.apply_tfms = apply_tfms\n \n self.side = int(self.config.num_tiles ** 0.5)\n \n self.tfms = albumentations.Compose([\n albumentations.HorizontalFlip(p=0.5),\n albumentations.VerticalFlip(p=0.5),\n albumentations.ShiftScaleRotate(shift_limit=.1, scale_limit=.1, rotate_limit=20, p=0.5),\n ])\n \n self.on_epoch_end()\n \n def __len__(self):\n return int(np.floor(len(self.image_ids) / self.config.batch_size))\n \n def on_epoch_end(self):\n self.indexes = np.arange(len(self.image_ids))\n \n if self.shuffle:\n np.random.shuffle(self.indexes)\n \n def __getitem__(self, index):\n X = np.zeros((self.config.batch_size, self.side * self.config.img_size, \\\n self.side * self.config.img_size, 3), dtype=np.float32)\n \n imgs_batch = self.image_ids[index * self.config.batch_size : (index + 1) * self.config.batch_size]\n \n for i, img_name in enumerate(imgs_batch):\n img_path = '{}/{}.tiff'.format(self.config.backbone_train_path, img_name)\n img_patches = self.get_patches(img_path)\n X[i, ] = self.glue_to_one(img_patches)\n \n if self.mode == 'fit':\n y = np.zeros((self.config.batch_size, self.config.num_classes), dtype=np.float32)\n lbls_batch = self.labels[index * self.config.batch_size : (index + 1) * self.config.batch_size]\n \n for i in range(self.config.batch_size):\n y[i, lbls_batch[i]] = 1\n return X, y\n \n elif self.mode == 'predict':\n return X\n \n else:\n raise AttributeError('mode parameter error')\n \n def get_patches(self, img_path):\n num_patches = self.config.num_tiles\n p_size = self.config.img_size\n img = skimage.io.MultiImage(img_path)[-1] / 255\n \n if self.apply_tfms:\n img = self.tfms(image=img)['image'] \n \n pad0, pad1 = (p_size - img.shape[0] % p_size) % p_size, (p_size - img.shape[1] % p_size) % p_size\n \n img = np.pad(\n img,\n [\n [pad0 // 2, pad0 - pad0 // 2], \n [pad1 // 2, pad1 - pad1 // 2], \n [0, 0]\n ],\n constant_values=1\n )\n \n img = img.reshape(img.shape[0] // p_size, p_size, img.shape[1] // p_size, p_size, 3)\n img = img.transpose(0, 2, 1, 3, 4).reshape(-1, p_size, p_size, 3)\n \n if len(img) < num_patches:\n img = np.pad(\n img, \n [\n [0, num_patches - len(img)],\n [0, 0],\n [0, 0],\n [0, 0]\n ],\n constant_values=1\n )\n \n idxs = np.argsort(img.reshape(img.shape[0], -1).sum(-1))[:num_patches]\n return np.array(img[idxs])\n \n def glue_to_one(self, imgs_seq):\n img_glue = np.zeros((self.config.img_size * self.side, self.config.img_size * self.side, 3), dtype=np.float32)\n \n for i, ptch in enumerate(imgs_seq):\n x = i // self.side\n y = i % self.side\n img_glue[x * self.config.img_size : (x + 1) * self.config.img_size, \n y * self.config.img_size : (y + 1) * self.config.img_size, :] = ptch\n \n return img_glue","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

5. Loading data and visualization

\n\n

Now let's create our Data generator objects and visualize the data we feed our model.\n\nAnother tips: Always check what you give your model. It will prevent you from running into stupid mistakes that could take hours to debug.

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/prostate-cancer-grade-assessment/train.csv\", nrows=6000)\ndf = df.sample(frac=1, random_state=config.seed).reset_index(drop=True)\ntrain_df, valid_df = train_test_split(df, test_size=0.2, random_state=config.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = PANDAGenerator(\n df=train_df, \n config=config,\n mode='fit', \n apply_tfms=False,\n shuffle=True, \n)\n\nval_datagen = PANDAGenerator(\n df=valid_df, \n config=config,\n mode='fit', \n apply_tfms=False,\n shuffle=False, \n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Xt, yt = train_datagen.__getitem__(0)\n\nprint('Shape of X: ', Xt.shape)\nprint('Shape of y: ', yt.shape)\n\nfig, ax = plt.subplots(figsize=(15, 15), ncols=3)\n\nfor i in range(3):\n ax[i].imshow(Xt[i])\n ax[i].set_title('label {}'.format(np.argmax(yt[i, ])))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

6. Building model

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_seresnext():\n \n SEResNEXT50, _ = Classifiers.get('seresnext50')\n base_model = SEResNEXT50(input_shape=(config.img_size*int(config.num_tiles**0.5), \\\n config.img_size*int(config.num_tiles**0.5), 3), \\\n weights='imagenet', include_top=False)\n \n x = GlobalAveragePooling2D()(base_model.output)\n output = Dense(config.num_classes, activation='softmax')(x)\n model = Model(inputs=[base_model.input], outputs=[output])\n\n model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=config.learning_rate), \\\n metrics=[tfa.metrics.CohenKappa(weightage='quadratic', num_classes=6)])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model = build_seresnext()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

7. Training

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cb1 = ReduceLROnPlateau(monitor='val_loss', factor=0.3, patience=1, verbose=1, min_lr=1e-6)\ncb2 = ModelCheckpoint(\"best_seresnext50.h5\", monitor='val_loss', verbose=1, save_best_only=True, save_weights_only=True, mode='min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(\n train_datagen,\n validation_data=val_datagen,\n callbacks=[cb1, cb2],\n epochs=config.num_epochs,\n verbose=1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# \"Accuracy\"\nplt.plot(history.history['cohen_kappa'])\nplt.plot(history.history['val_cohen_kappa'])\nplt.title('model cohen kappa')\nplt.ylabel('cohen kappa')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()\n# \"Loss\"\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

8. Visualizing with GradCAM

","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('best_seresnext50.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_img_array(img_path):\n img = tf.keras.preprocessing.image.load_img(img_path, target_size=(config.img_size, config.img_size))\n array = tf.keras.preprocessing.image.img_to_array(img)\n array = np.expand_dims(array, axis=0) # Add one dimension to transform our array into a batch\n return array\n\ndef make_gradcam_heatmap(img_array, model, last_conv_layer_name, classifier_layer_names):\n last_conv_layer = model.get_layer(last_conv_layer_name)\n last_conv_layer_model = Model(model.inputs, last_conv_layer.output)\n \n classifier_input = Input(shape=last_conv_layer.output.shape[1:])\n x = classifier_input\n for layer_name in classifier_layer_names:\n x = model.get_layer(layer_name)(x)\n classifier_model = Model(classifier_input, x)\n \n with tf.GradientTape() as tape:\n last_conv_layer_output = last_conv_layer_model(img_array)\n tape.watch(last_conv_layer_output)\n \n preds = classifier_model(last_conv_layer_output)\n top_pred_index = tf.argmax(preds[0])\n top_class_channel = preds[:, top_pred_index]\n \n grads = tape.gradient(top_class_channel, last_conv_layer_output)\n pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n \n last_conv_layer_output = last_conv_layer_output.numpy()[0]\n pooled_grads = pooled_grads.numpy()\n for i in range(pooled_grads.shape[-1]):\n last_conv_layer_output[:, :, i] *= pooled_grads[i]\n \n heatmap = np.mean(last_conv_layer_output, axis=-1)\n heatmap = np.maximum(heatmap, 0) / np.max(heatmap)\n return heatmap\n\ndef create_superimposed_visualization(img, heatmap):\n heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))\n \n heatmap = np.uint8(255*heatmap)\n heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)\n superimposed_img = heatmap * 0.4 + img\n \n return superimposed_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We need to get the names of the last convolution layer of ResNet50\nlast_conv_layer_name = 'activation_80'\n\n# We also need the names of all the layers that are part of the model head\nclassifier_layer_names = [\n 'global_average_pooling2d_16',\n 'dense'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's take the first 5 images of the dataset\n\nfig, ax = plt.subplots(figsize=(15, 10), ncols=3, nrows=2)\n\nfor i in range(3):\n raw_image = Xt[i]\n\n image = np.expand_dims(raw_image, axis=0)\n\n heatmap = make_gradcam_heatmap(image, model, last_conv_layer_name, classifier_layer_names)\n superimposed_image = create_superimposed_visualization(raw_image, heatmap)\n\n ax[0][i].imshow(raw_image)\n ax[0][i].set_title('Original - label {}'.format(np.argmax(yt[i, ])))\n ax[1][i].imshow(superimposed_image)\n ax[1][i].set_title('GradCAM - label {}'.format(np.argmax(yt[i, ])))\n\nfig.suptitle('SE_ResNeXT_50')\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}