{"cells":[{"metadata":{},"cell_type":"markdown","source":"**INFERENCE KERNEL**\n\nThis is a starting point Inference Kernel for the Panda Challenge competition  \nIn this kernel there are used 2 model arhitecture types:   \n* first one is based on a pretrained densenet121 arhitecture followed by a dense layer (from 262144 outputs to 5 one which are our classes)   \n* second one is based on a pretrained se_resnet50 arhitecture followed by a dense layer (from 524288 outputs to 5 one which are our classes)\n\n**Usefull Information** \n\n* Each arhitecture was splited during training in 5 folds (here are present just the first folds from each arhitecture, the rest of them are curently in training, as soon as they will be ready, the kernel will be updated) \n* image size is 512x512. From my initial experiments working with 300x300 or smaller image has lead to weaker results\n* it was made a TTA enable flag which will enable TTA. During TTA mode there will be created 4 distinct image augmentation during inference and then  we will use the prediction mean (one inference of the normal image + one inference of the ROTATE_90_CLOCKWISE + one inference ROTATE_90_COUNTERCLOCKWISE + one inference ROTATE_180)\n* leaderboard score for this initial configuration: 0.65\n\n* The training kernel is online now: https://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds \n\n-------------------Version 17 updates--------------------------- \n\nModified ModelArhitectureV1 from densenet121 to se_resnext101_32x4d (better results in the same training conditions)\n\n\n\n-----------------Version 20 updates---------------------------- \n\nEnable TTA mode\n\n-----------------Version 20 updates---------------------------- \n\nDisable TTA mode  \nChanged trained fold1 on v1 arhitecture with a better training file (.pth)\n\n----------------Version 26 updates----------------------------  \n\nAdd fold 2 training file for ModelArhitectureV2\n\n\n-------------Version 30 updates-----------------------------  \n\nAdd fold 4 training file for ModelArhitectureV2 (they will keep going as soon the training is over, which is taking a while..)  \nNew score: 0.70"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.\n\n\n#install pretrained models library\n!pip install ../input/pretrainedmodels/pretrainedmodels-0.7.4/pretrainedmodels-0.7.4/ > /dev/null # no output","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom tqdm import tqdm_notebook as tqdm\nimport matplotlib.pyplot as plt\nimport cv2\nimport albumentations as A\nimport torch\nfrom skimage.transform import AffineTransform, warp\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport torch.nn.functional as F\nfrom torch.utils.data.dataloader import DataLoader\nfrom torchvision import datasets, models, transforms\nimport torch.nn as nn\nfrom torch.nn import functional as F\nimport pretrainedmodels\nimport matplotlib.pyplot as plt\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nimport skimage.io","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#Dataset for inference set\n#Image will be loaded -> it will be normalized to 0-1 interval ->  it be send to the transform augmentation class -> switch dimensions to (2,0,1) -> convert to tensor\n\nclass PandaDataset:\n    def __init__(self, df, transform=None,image_dir=\" \"):\n        self.image_ids=df.index.values\n        self.transform=transform\n        self.df=df\n        self.image_dir=image_dir\n    def __len__(self):\n        return len(self.image_ids)\n    \n    def __getitem__(self, index):\n\n        imgPath= f\"{self.image_dir}{self.image_ids[index]}.tiff\"\n        img = skimage.io.MultiImage(imgPath)\n        image = img[-1]\n        image = (255 - image).astype(np.float32) / 255.\n        if self.transform:\n            image,_ = self.transform(image)\n        image = np.transpose(image, (2, 0, 1)).astype(np.float32)\n\n        return torch.tensor(image, dtype=torch.float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#augmentation class for inference set\n#here we perform resize to (512,512) and normalize (mean=0.0692, std=0.2051)\n#also there is a flag in order to check if TTA should be taken into account and if so what type of augmentation\n\ndef Aug_resize(img,size):\n    return cv2.resize(img, size, interpolation=cv2.INTER_AREA )\n\n\nclass Transform:\n    def __init__(self, size=(512, 512),normalize=True, TTAconfig=0):\n        self.size=size\n        self.normalize=normalize\n        self.TTAconfig=TTAconfig\n        \n    def __call__(self, x):\n\n        # --- Augmentation ---\n\n        x = Aug_resize(x, size=self.size)\n\n        x = (x.astype(np.float32) - 0.0692) / 0.2051\n        \n        if (self.TTAconfig==1):\n            x = cv2.rotate(x, cv2.ROTATE_90_CLOCKWISE)\n            \n        if (self.TTAconfig==2):\n            x = cv2.rotate(x, cv2.ROTATE_90_COUNTERCLOCKWISE)\n            \n        if (self.TTAconfig==3):\n            x = cv2.rotate(x, cv2.ROTATE_180)\n        \n        return x, None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#model design for the inference of the model V1 based in densenet121\nclass ModelArhitectureV1(nn.Module):\n    def __init__(self,pretrainedModelArhitecture='se_resnext101_32x4d', pretrainedModelWeights=None):\n        super(ModelArhitectureV1, self).__init__()\n        self.base_model = pretrainedmodels.__dict__[pretrainedModelArhitecture](pretrained=pretrainedModelWeights).to(device)\n        self.final1_1 = nn.Linear(in_features=524288, out_features=6, bias=True).to(device)\n                \n    def forward(self,x):\n        self.do_pooling=False\n        h=self.base_model.features(x)\n        if self.do_pooling:\n            h = torch.sum(h, dim=(-1, -2))\n        else:\n            bs, ch, height, width = h.shape\n            h = h.view(bs, ch*height*width)\n\n        h1=self.final1_1(h)      \n        return h1\n\n#model design for the inference of the model V2 based in se_resnet50\nclass ModelArhitectureV2(nn.Module):\n    def __init__(self,pretrainedModelArhitecture='se_resnet50', pretrainedModelWeights=None):\n        super(ModelArhitectureV2, self).__init__()\n        self.base_model = pretrainedmodels.__dict__[pretrainedModelArhitecture](pretrained=pretrainedModelWeights).to(device)\n        self.final1_1 = nn.Linear(in_features=524288, out_features=6, bias=True).to(device)\n                \n    def forward(self,x):\n        self.do_pooling=False\n        h=self.base_model.features(x)\n        if self.do_pooling:\n            h = torch.sum(h, dim=(-1, -2))\n        else:\n            bs, ch, height, width = h.shape\n            h = h.view(bs, ch*height*width)\n\n        h1=self.final1_1(h)\n\n        \n        return h1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#general algorithm class made for inference \n#the output is a list with all the predictions after applying softmax to the vector, example [[0.2, 0.3, 0.2, 0.05, 0.25],[0.1, 0.4, 0.1, 0.15, 0.25], ...]\nclass PandaAlgorithm(nn.Module):\n    \n    def __init__(self, model):        \n        super(PandaAlgorithm, self).__init__()\n        self.model=model\n    def forward(self,x):\n\n        inputs = x.to(device)\n        outputs =self.model(inputs)\n\n        return outputs\n    \n    def getPredictions(self, dataloader):\n        predList=[]\n        with torch.no_grad():\n            for inputs in tqdm(dataloader):\n                rawPreds = self.forward(inputs)\n                predList.extend(F.softmax(rawPreds))\n        return predList\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# a manager function that takes a dict with the training params and performs inference\ndef doInference(train_args_dict):\n    if train_args_dict.get('model_arhitecture_base')=='v1':\n        arhitecture = ModelArhitectureV1()\n    else:\n        arhitecture = ModelArhitectureV2()\n        \n        \n    if (TTAmode):\n        transformTest0 = Transform(size = train_args_dict.get('image_size'),TTAconfig=0)\n        test_dataset0 = PandaDataset(df,transformTest0, image_dir)\n        test_loader0 = DataLoader(test_dataset0, batch_size=batch_size, shuffle=False)\n        \n        transformTest1 = Transform(size = train_args_dict.get('image_size'),TTAconfig=1)\n        test_dataset1 = PandaDataset(df,transformTest1, image_dir)\n        test_loader1 = DataLoader(test_dataset1, batch_size=batch_size, shuffle=False)\n        \n        transformTest2 = Transform(size = train_args_dict.get('image_size'),TTAconfig=2)\n        test_dataset2 = PandaDataset(df,transformTest2, image_dir)\n        test_loader2 = DataLoader(test_dataset2, batch_size=batch_size, shuffle=False)\n        \n        transformTest3 = Transform(size = train_args_dict.get('image_size'),TTAconfig=3)\n        test_dataset3 = PandaDataset(df,transformTest3, image_dir)\n        test_loader3 = DataLoader(test_dataset3, batch_size=batch_size, shuffle=False)\n\n        arhitecture.load_state_dict(torch.load(train_args_dict.get('model_path'),map_location='cuda:0'))\n        pandaAlgorithm = PandaAlgorithm(arhitecture)\n        \n        results0= pandaAlgorithm.getPredictions(dataloader=test_loader0)\n        results1= pandaAlgorithm.getPredictions(dataloader=test_loader1)\n        results2= pandaAlgorithm.getPredictions(dataloader=test_loader2)\n        results3= pandaAlgorithm.getPredictions(dataloader=test_loader3)\n        \n        results=[results0,results1,results2,results3]\n        return results\n    \n    else:\n        transformTest = Transform(size = train_args_dict.get('image_size'),TTAconfig=0)\n        test_dataset = PandaDataset(df,transformTest, image_dir)\n        test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n\n        arhitecture.load_state_dict(torch.load(train_args_dict.get('model_path'),map_location='cuda:0'))\n        pandaAlgorithm = PandaAlgorithm(arhitecture)\n        results= pandaAlgorithm.getPredictions(dataloader=test_loader)\n    \n    return [results]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#flag for enabling TTA\nTTAmode=False\n#cuda mode ON\ndevice='cuda:0'\ndevice = torch.device(device)\n#batch size for inference, it does not matter for accuracy, just for speed (watch out for Out Of Memory in case of increasing)\nbatch_size = 6\n#number of total models, here we have 2 models (fold 1 from each arhitecture)\nnumber_of_models=6\nmodel_preds_list = []\nsubmission = pd.read_csv('/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv')\n\nif os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n    submissionMode=True\nelse:\n    submissionMode=False\n    \n    \nif (submissionMode):    \n    image_dir= '/kaggle/input/prostate-cancer-grade-assessment/test_images/'\n\n    df = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/test.csv\").set_index(\"image_id\")\n\n\n    for i in range (number_of_models):\n        train_args_dict={}  \n\n        if (i==0):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model2fold0v1/model_fold0_epoch45_Qwk0.6675334622527956_v3Beta.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v2'                \n            })  \n\n        if (i==1):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model2fold1v0/model_fold1_epoch77_Qwk0.6080847762871844_v3Beta.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v2'                \n            })  \n\n        if (i==2):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model2fold2v0/model_fold2_epoch12_Qwk0.6049939916560642_v3Beta.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v2'                \n            })  \n          \n        if (i==3):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model2fold3v0/model_fold3_epoch26_Qwk0.6141516886726479_v3Beta.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v2'                \n            })  \n\n        if (i==4):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model2fold4v0/model_fold4_epoch21_Qwk0.57012076934546_v3Beta.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v2'                \n            })  \n\n        if (i==5):\n            train_args_dict.update({\n                'model_path': '/kaggle/input/model1fold0v0/model_fold0_epoch36_Qwk0.6627780473749447_v3Gamma.pth',                                    \n                'image_size': (512,512),\n                'model_arhitecture_base':'v1'                \n            })  \n\n        test_preds = doInference(train_args_dict)\n        \n        for i in range(len(test_preds)):\n            model_preds_list.append(test_preds[i])\n\n    \n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if (submissionMode):  \n    if TTAmode:\n        weightLists=[1/number_of_models] * number_of_models * 4\n    else:\n        weightLists = [1/number_of_models] * number_of_models\n    \n    results=[]\n    for i in range(len(model_preds_list[0])):\n        klist=[]\n        for j in range(len(model_preds_list[0][i])):\n            temp=0\n            for m in range(len(model_preds_list)):\n                temp+=(model_preds_list[m][i][j].item())*weightLists[m]\n            klist.append(temp)\n        results.append(klist)\n\n    predictions=[]    \n    for i in range(len(results)):\n         predictions.append(np.argmax(results[i]))\nelse:\n    predictions=[0,0,0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.isup_grade = predictions\nsubmission.isup_grade = submission['isup_grade'].astype(int)\nsubmission.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.isup_grade","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations:**\n* Feel free to use this kernel as a template for your models, it is made in such way so that anyone can simply upload their models, just change the dictionary with param and simply add new model class \n* When the next folds will be ready, they will be uploaded into this kernel\n* Soon it will be uploaded a kernel with the training and all training tricks used, until then here are several details https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/146332\n\nGoodluck to everybody \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}