{"cells":[{"metadata":{},"cell_type":"markdown","source":"# PANDA EfficientNet-B3 Baseline with 36 x tiles_256\n\n\n推論用のkernelは、https://www.kaggle.com/haqishen/panda-inference-w-36-tiles-256　を参考に作成する.\n\n\n## TL/DL\n\n* https://www.kaggle.com/iafoss/panda-16x128x128-tiles　を参考にtiling methodを利用\n    * Simply setting the `N = 36` and `sz=256` then extract from median resolution\n    * big imageのサイズ1単位をタイルとして（6, 6)->(6×256, 6×256)、各タイルのサイズは(256, 256)\n* Create 6x6 big image from 36 tiles\n* Efficientnet-B3を利用\n* 損失関数はBCE loss（多クラス多分類によく使われる）\n* BCE lossを使っているためラベル付けは以下のようにマルチクラスのように扱った.\n    * E.g.\n        * `label = [0,0,0,0,0]` means `isup_grade = 0`\n        * `label = [1,1,1,0,0]` means `isup_grade = 3`\n        * `label = [1,1,1,1,1]` means `isup_grade = 5`\n* Augmentationはタイルごと、合成した画像ごとどちらにも行う\n* 1epochごとにCosineAnnealingLRをかける?\n\n## MEMO\n\n学習が全て終わるまで10時間以上かかる\n\n## Update\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# FLOW\n\n1. 各種設定(データセットのパス通したりモジュールインポートしたり)\n2. ラベルの処理(StratifiedKFold)\n3. モデルの構築\n4. タイル化用の関数を定義\n5. Datasetの構築\n6. Augumentation\n7. 損失関数の設定, train, validation用の関数の定義\n9. Dataloaderの構築\n8. KFold用の関数の定義\n9. train\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **Let's coding!**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# DEBUG = False ですべてのデータで学習する\n\nDEBUG = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## pytorch-gradual-warmup-lr\n\n巨大なbatch sizeを扱うときにバッチサイズに比例させて学習率をあげると精度の悪化が見られないよと言う論文<br>\nAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour『https://arxiv.org/abs/1706.02677』 (参考：https://www.slideshare.net/iwiwi/nips17-86470238)\nを元に作られたモジュール","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install git+https://github.com/ildoonet/pytorch-gradual-warmup-lr.git","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"優しい誰かが作ってくれたPyTorch用Efficient netのpathを通す<br>\n元記事はhttps://www.kaggle.com/hmendonca/efficientnet-pytorch <br>\n元々記事https://github.com/lukemelas/EfficientNet-PyTorch","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\nsys.path = [\n    '../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master',\n] + sys.path","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import time\nimport skimage.io\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport PIL.Image\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.utils.data.sampler import SubsetRandomSampler, RandomSampler, SequentialSampler\nfrom warmup_scheduler import GradualWarmupScheduler\nfrom efficientnet_pytorch import model as enet\nimport albumentations\nfrom sklearn.model_selection import StratifiedKFold\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import cohen_kappa_score\nfrom tqdm import tqdm_notebook as tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Config","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/prostate-cancer-grade-assessment'\ndf_train = pd.read_csv(os.path.join(data_dir, 'train.csv'))\nimage_folder = os.path.join(data_dir, 'train_images')\n\nkernel_type = 'how_to_train_effnet_b0_to_get_LB_0.86'\n\nenet_type = 'efficientnet-b0'\nfold = 0\ntile_size = 256\nimage_size = 256\nn_tiles = 36\nbatch_size = 2\nnum_workers = 4\nout_dim = 5\ninit_lr = 3e-4\nwarmup_factor = 10\n\nwarmup_epo = 1\n# DEBUG = Trueのときは1epochだけ実験\nn_epochs = 1 if DEBUG else 30\ndf_train = df_train.sample(100).reset_index(drop=True) if DEBUG else df_train\n\n# 今回はGPUを使う\ndevice = torch.device('cuda')\n\nprint(image_folder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Folds\n\nラベルをKFoldで処理する.\n\n過学習防止、未知のデータへの汎用性を損なわないように**StratifiedKFold**(層別KFold).<br>\n**StratifiedKFold**はデータのラベルの分布を維持したままKFoldしてくれるやつ.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(5, shuffle=True, random_state=42)\n# 'fold'カラムを作成しとりあえず−1を入れておく\ndf_train['fold'] = -1\n# StratifiedKFoldによるラベルづけ（[4:1]で[train:test]）\nfor i, (train_idx, valid_idx) in enumerate(skf.split(df_train, df_train['isup_grade'])):\n    df_train.loc[valid_idx, 'fold'] = i\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model\n\nモデルを構築する.<br>\n今回は、Efficient Net b3の学習済みモデルを利用する.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# 今回はeffiicentnet-b3を使う\n\npretrained_model = {\n    'efficientnet-b3': '../input/efficientnet-pytorch/efficientnet-b3-c8376fa2.pth'\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"enet -> Identity() -> myfc","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# モデルの構築\n\nclass enetv2(nn.Module):\n    def __init__(self, backbone, out_dim):\n        '''\n        params:\n            backbone : Efficien Netの種類(今回はb3)\n            out_dim : 最終層のノード数(クラス数、今回は6)\n        '''\n        super(enetv2, self).__init__()\n        # 学習済みモデルのロード(Document参照)\n        self.enet = enet.EfficientNet.from_name(backbone)\n        self.enet.load_state_dict(torch.load(pretrained_model[backbone]))\n        # enet最終層の恒等関数\n        self.enet._fc = nn.Identity()\n        # 付け加える全結合層\n        # .enet._fc.in_featuresがenetの出力次元数\n        self.myfc = nn.Linear(self.enet._fc.in_features, out_dim)\n        \n\n    def extract(self, x):\n        return self.enet(x)\n\n    def forward(self, x):\n        x = self.extract(x)\n        x = self.myfc(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_tiles(img, mode=0):\n        result = []\n        h, w, c = img.shape\n        pad_h = (tile_size - h % tile_size) % tile_size + ((tile_size * mode) // 2)\n        pad_w = (tile_size - w % tile_size) % tile_size + ((tile_size * mode) // 2)\n\n        img2 = np.pad(img,[[pad_h // 2, pad_h - pad_h // 2], [pad_w // 2,pad_w - pad_w//2], [0,0]], constant_values=255)\n        img3 = img2.reshape(\n            img2.shape[0] // tile_size,\n            tile_size,\n            img2.shape[1] // tile_size,\n            tile_size,\n            3\n        )\n\n        img3 = img3.transpose(0,2,1,3,4).reshape(-1, tile_size, tile_size,3)\n        n_tiles_with_info = (img3.reshape(img3.shape[0],-1).sum(1) < tile_size ** 2 * 3 * 255).sum()\n        if len(img3) < n_tiles:\n            img3 = np.pad(img3,[[0,n_tiles-len(img3)],[0,0],[0,0],[0,0]], constant_values=255)\n        idxs = np.argsort(img3.reshape(img3.shape[0],-1).sum(-1))[:n_tiles]\n        img3 = img3[idxs]\n        for i in range(len(img3)):\n            result.append({'img':img3[i], 'idx':i})\n        return result, n_tiles_with_info >= n_tiles\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## get_tiles(img, mode=0) 関数について\n\n今関数は入力した元データをタイル化して新たな画像にするための前処理関数\n\n引数は、\n\n* img : 与えられた元データ（訓練データ）1枚(tiff画像がサイズごとに3枚入っているが2枚目の中解像度のものを想定している)\n* mode : デフォルトは0\n\n返り値は、\n\n* result :　numpy配列、\n\n\n### 関数の動きを見るためのデバッグ\n\n\n```python\n\nimport numpy as np\nimport skimage.io\n\nn_tiles = 36\ntile_size = 256\nmode = 0\n\nimg = skimage.io.MultiImage('../input/prostate-cancer-grade-assessment/train_images/0005f7aaab2800f6170c399693a96917.tiff')[1]\nprint('input img : ', img.shape) # (7360, 6912, 3)\nresult = []\n# h:高さ、w:幅、c:チャネル数(RGBなので３)\nh, w, c = img.shape\n# pad_h,wはどのくらいパディング用変数\npad_h = (tile_size - h % tile_size) % tile_size + ((tile_size * mode) // 2)\npad_w = (tile_size - w % tile_size) % tile_size + ((tile_size * mode) // 2)\n\nprint('padding : ', pad_h, pad_w) # 96 192\n\n# padding後の画像img2\n\nimg2 = np.pad(img,[[pad_h // 2, pad_h - pad_h // 2], [pad_w // 2,pad_w - pad_w//2], [0,0]], constant_values=255)\nprint('img2 shape : ', img2.shape) # (7456, 7104, 3)\n\nimg3 = img2.reshape(\n    img2.shape[0] // tile_size,\n    tile_size,\n    img2.shape[1] // tile_size,\n    tile_size,\n    3\n)\n\nprint('img3 shape : ', img3.shape)\n\n# PILの形式からTensor型の形式に変換\n# その後1枚の画像を(256, 256, 3)にreshape\n\nimg3 = img3.transpose(0,2,1,3,4).reshape(-1, tile_size, tile_size,3)\nprint('img2 reshape : ', img3.shape)\n\n\n# print((img3.reshape(img3.shape[0],-1)).sum(1))\n# print(tile_size ** 2 * 3 * 255)\n\n# print((img3.reshape(img3.shape[0],-1).sum(1) < tile_size ** 2 * 3 * 255))\n\n# 画像が存在する（真っ白ではない）タイルの枚数を数えている\nn_tiles_with_info = (img3.reshape(img3.shape[0],-1).sum(1) < tile_size ** 2 * 3 * 255).sum()\nprint('n_tiles_with_info : ', n_tiles_with_info)\n\n# 1枚の元画像からタイル化した枚数\nprint(len(img3))\n\n\nif len(img3) < n_tiles:\n    img3 = np.pad(img3,[[0,n_tiles-len(img3)],[0,0],[0,0],[0,0]], constant_values=255)\n    \n# タイル化した画像のうち白色背景が少ないものから順にソート(36枚分)\n# 白に近いほどピクセルレベルが255に近づく\nidxs = np.argsort(img3.reshape(img3.shape[0],-1).sum(-1))[:n_tiles]\nprint(idxs)\nprint('tail枚数 : ', len(idxs))\nimg3 = img3[idxs]\nprint(img3.shape)\n# for i in range(len(img3)):\n#     result.append({'img':img3[i], 'idx':i})\n# return result, n_tiles_with_info >= n_tiles\n\n\n```\n\n```output\ninput img :  (7360, 6912, 3)\npadding :  64 0\nimg2 shape :  (7424, 6912, 3)\nimg3 shape :  (29, 256, 27, 256, 3)\nimg2 reshape :  (783, 256, 256, 3)\nn_tiles_with_info :  58\n783\n[358 465 412 385 250 331 625 599 439 277 167 304 492 222 194 518 572 438\n 545 519 626 305 332 491 112 139 546 140 411 166 573 598 384 249 359  85]\ntail枚数 :  36\n(36, 256, 256, 3)\n```\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class PANDADataset(Dataset):\n    def __init__(self,\n                 df,\n                 image_size,\n                 n_tiles=n_tiles,\n                 tile_mode=0,\n                 rand=False,\n                 transform=None,\n                ):\n\n        self.df = df.reset_index(drop=True)\n        self.image_size = image_size\n        self.n_tiles = n_tiles\n        self.tile_mode = tile_mode\n        self.rand = rand\n        self.transform = transform\n\n    def __len__(self):\n        return self.df.shape[0]\n\n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        img_id = row.image_id\n        \n        tiff_file = os.path.join(image_folder, f'{img_id}.tiff')\n        image = skimage.io.MultiImage(tiff_file)[1]\n        tiles, OK = get_tiles(image, self.tile_mode)\n\n        if self.rand:\n            idxes = np.random.choice(list(range(self.n_tiles)), self.n_tiles, replace=False)\n        else:\n            idxes = list(range(self.n_tiles))\n\n        # n_row_tiles　今回は6×6=36画像なので6\n        n_row_tiles = int(np.sqrt(self.n_tiles))\n        # image_sizeは256を想定\n        images = np.zeros((image_size * n_row_tiles, image_size * n_row_tiles, 3))\n        # tilesの辞書からタイル画像を取ってきてbig imageとしてマッピング\n        for h in range(n_row_tiles):\n            for w in range(n_row_tiles):\n                i = h * n_row_tiles + w\n    \n                if len(tiles) > idxes[i]:\n                    this_img = tiles[idxes[i]]['img']\n                else:\n                    this_img = np.ones((self.image_size, self.image_size, 3)).astype(np.uint8) * 255\n                # 画像を黒色化している？\n                this_img = 255 - this_img\n                \n                # ここが前述の「タイルレベルでのAugumentaion」に想定する部分\n                # transformはAugumentaionをするオブジェクトを想定\n                if self.transform is not None:\n                    this_img = self.transform(image=this_img)['image']\n                h1 = h * image_size\n                w1 = w * image_size\n                # 各タイル画像をbig image(images)にマッピング\n                images[h1:h1+image_size, w1:w1+image_size] = this_img\n\n        if self.transform is not None:\n            images = self.transform(image=images)['image']\n        # numpy.float32に変換\n        images = images.astype(np.float32)\n        # 0~1の範囲に正規化\n        images /= 255\n        # (h, w, c)のPIL形式を（c, h, w）に変換しPyTorchで扱えるようにしている\n        images = images.transpose(2, 0, 1)\n\n        label = np.zeros(5).astype(np.float32)\n        # BCE Lossで扱うことを想定にしている（詳細は上記）\n        label[:row.isup_grade] = 1.\n        # tensor型にしたimageとtensor型にしたラベルをreturn\n        return torch.tensor(images), torch.tensor(label)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentations\n\nalbumentarionsを利用<br>\n詳しい説明は以下参照<br>\n[画像データ拡張ライブラリ ~ albumentations ~](https://qiita.com/Takayoshi_Makabe/items/79c8a5ba692aa94043f7)<br>\n[公式Document](https://albumentations.readthedocs.io/en/latest/api/augmentations.html?highlight=Transpose)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train用のAugumentationは、「転置」「左右反転」「上下反転」\n# pがどれくらいの確率でその操作を行うか、という引数\n\ntransforms_train = albumentations.Compose([\n    albumentations.Transpose(p=0.5),\n    albumentations.VerticalFlip(p=0.5),\n    albumentations.HorizontalFlip(p=0.5),\n])\n# validationのときはAugumentationしなくていいからそのまんま\ntransforms_val = albumentations.Compose([])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"いったん作った画像を可視化してみる","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_show = PANDADataset(df_train, image_size, n_tiles, 0, transform=transforms_train)\nfrom pylab import rcParams\nrcParams['figure.figsize'] = 20,10\nfor i in range(2):\n    f, axarr = plt.subplots(1,5)\n    for p in range(5):\n        idx = np.random.randint(0, len(dataset_show))\n        img, label = dataset_show[idx]\n        axarr[p].imshow(1. - img.transpose(0, 1).transpose(1,2).squeeze())\n        axarr[p].set_title(str(sum(label)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss\n\n今回はBCE Lossを使う","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = nn.BCEWithLogitsLoss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train & Val\n\ntrain, validation用の関数を定義\n\nここで注意する点は、Dataloaderはデータセットを使い切るまでバッチサイズ分データを固めて返してくれるだけ、<br>\nつまりはDataloade自体は1epoch分forループを回すだけなので学習時はエポック分さらにforループで回してあげる必要がある.\n\n---memo---<br>\nDataloaderあたりか[これ](https://qiita.com/mathlive/items/2a512831878b8018db02)が参考になる.<br>\nProgressBarを表示させるtqdmについては[これ](https://qiita.com/pontyo4/items/76145cb10e030ad8186a)がいい感じ.<br>\n検証用関数のno.grad()については[これ](https://qiita.com/a_yoshii/items/598365cf3b68955e11c5)<br>\n\n\n.cpu().numpy()に関してはmodel自体の計算はGPUで行っているが正解率やlossを計算するときはcpuで行いたいので一度CPUに落としてから出ないとダメ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# 訓練用関数の定義\n\ndef train_epoch(loader, optimizer):\n    \n    '''\n    params:\n        loader : データローダーを想定\n        optimizer: 最適化手法を想定\n    '''\n    \n    # modelをtrainモードに\n    model.train()\n    #tarin_lossが記録用配列\n    train_loss = []\n    bar = tqdm(loader)\n    for (data, target) in bar:\n        \n        # data(訓練データ[画像])とtarget(訓練ラベル[tensor配列])をGPUヘ\n        data, target = data.to(device), target.to(device)\n        # 損失関数を定義\n        loss_func = criterion\n        # optimizerの勾配をリセット\n        optimizer.zero_grad()\n        # logitsがモデルを通した出力\n        # 形式は[0.1, 0.5, 0.12, 0.18, 0.05, 0.05]などを想定\n        logits = model(data)\n        # lossを計算\n        loss = loss_func(logits, target)\n        # 勾配計算\n        loss.backward()\n        # パラメータ更新\n        optimizer.step()\n        # モデルから損失だけ取得\n        loss_np = loss.detach().cpu().numpy()\n        # lossを記録\n        train_loss.append(loss_np)\n        # smooth_lossは下の方法で平滑化したやつっぽい\n        smooth_loss = sum(train_loss[-100:]) / min(len(train_loss), 100)\n        # lossとsmooth_lossをprint\n        bar.set_description('loss: %.5f, smth: %.5f' % (loss_np, smooth_loss))\n    return train_loss\n\n\n\n# 検証用関数の定義\ndef val_epoch(loader, get_output=False):\n    \n    # modelをevalモードに\n    model.eval()\n    val_loss = []\n    LOGITS = \n    # 実際にLOGITからどのクラスを指定したかをPREDSに保存(下のTARGETSと同様)\n    PREDS = []\n    # 正解ラベル（ex:[1,0,0,0,0,0]）を合計して整数化したもの(0~5の範囲をとる)の記録用配列\n    TARGETS = []\n\n    with torch.no_grad():\n        for (data, target) in tqdm(loader):\n            data, target = data.to(device), target.to(device)\n            logits = model(data)\n\n            loss = criterion(logits, target)\n            \n            # modelからの出力であるlogitsの各要素に対してsigmoid()をかけ,\n            # その総和を取り、モデルから取得、小数点を切り上げ切り下げし整数化(0~5にしている)\n            # この段階ではdetach()のときGPU上で行っている\n            pred = logits.sigmoid().sum(1).detach().round()\n            LOGITS.append(logits)\n            PREDS.append(pred)\n            # 正解ラベル（ex:[1,0,0,0,0,0]）を合計して整数化したもの(0~5の範囲をとる)\n            TARGETS.append(target.sum(1))\n\n            val_loss.append(loss.detach().cpu().numpy())\n        val_loss = np.mean(val_loss)\n        \n    # 正解率を計算\n    # この段階でCPUに落とす\n    LOGITS = torch.cat(LOGITS).cpu().numpy()\n    PREDS = torch.cat(PREDS).cpu().numpy()\n    TARGETS = torch.cat(TARGETS).cpu().numpy()\n    acc = (PREDS == TARGETS).mean() * 100.\n    \n    # ここはよくわからん\n    qwk = cohen_kappa_score(PREDS, TARGETS, weights='quadratic')\n    qwk_k = cohen_kappa_score(PREDS[df_valid['data_provider'] == 'karolinska'], df_valid[df_valid['data_provider'] == 'karolinska'].isup_grade.values, weights='quadratic')\n    qwk_r = cohen_kappa_score(PREDS[df_valid['data_provider'] == 'radboud'], df_valid[df_valid['data_provider'] == 'radboud'].isup_grade.values, weights='quadratic')\n    print('qwk', qwk, 'qwk_k', qwk_k, 'qwk_r', qwk_r)\n\n    if get_output:\n        return LOGITS\n    else:\n        return val_loss, acc, qwk\n\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Dataloader & Model & Optimizer\n\n##GradualWarmupSchedulerの使い方\n\n以下を参照<br>\n[](http://)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# 今回はfold=0\ntrain_idx = np.where((df_train['fold'] != fold))[0]\nvalid_idx = np.where((df_train['fold'] == fold))[0]\n\ndf_this  = df_train.loc[train_idx]\ndf_valid = df_train.loc[valid_idx]\n\ndataset_train = PANDADataset(df_this , image_size, n_tiles, transform=transforms_train)\ndataset_valid = PANDADataset(df_valid, image_size, n_tiles, transform=transforms_val)\n\n# Samplerがtrainとvalidで違うので注意\ntrain_loader = torch.utils.data.DataLoader(dataset_train, batch_size=batch_size, sampler=RandomSampler(dataset_train), num_workers=num_workers)\nvalid_loader = torch.utils.data.DataLoader(dataset_valid, batch_size=batch_size, sampler=SequentialSampler(dataset_valid), num_workers=num_workers)\n\nmodel = enetv2(enet_type, out_dim=out_dim)\n# modelをGPUにおくる\nmodel = model.to(device)\n\n# optimizerはAdam\n# lr（学習率）はwarmup_factorを使って可変にしている(いまいちよく分かっていない)\noptimizer = optim.Adam(model.parameters(), lr=init_lr/warmup_factor)\nscheduler_cosine = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, n_epochs-warmup_epo)\nscheduler = GradualWarmupScheduler(optimizer, multiplier=warmup_factor, total_epoch=warmup_epo, after_scheduler=scheduler_cosine)\n\nprint(len(dataset_train), len(dataset_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Run Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"qwk_max = 0.\nbest_file = f'{kernel_type}_best_fold{fold}.pth'\nfor epoch in range(1, n_epochs+1):\n    print(time.ctime(), 'Epoch:', epoch)\n    scheduler.step(epoch-1)\n\n    train_loss = train_epoch(train_loader, optimizer)\n    val_loss, acc, qwk = val_epoch(valid_loader)\n\n    content = time.ctime() + ' ' + f'Epoch {epoch}, lr: {optimizer.param_groups[0][\"lr\"]:.7f}, train loss: {np.mean(train_loss):.5f}, val loss: {np.mean(val_loss):.5f}, acc: {(acc):.5f}, qwk: {(qwk):.5f}'\n    print(content)\n    with open(f'log_{kernel_type}.txt', 'a') as appender:\n        appender.write(content + '\\n')\n\n    if qwk > qwk_max:\n        print('score2 ({:.6f} --> {:.6f}).  Saving model ...'.format(qwk_max, qwk))\n        torch.save(model.state_dict(), best_file)\n        qwk_max = qwk\n\ntorch.save(model.state_dict(), os.path.join(f'{kernel_type}_final_fold{fold}.pth'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# My Local Train Log\n\n\n```\nTue June 2 15:39:21 2020 Epoch 1, lr: 0.0000300, train loss: 0.42295, val loss: 0.29257, acc: 47.50471, qwk: 0.77941\nTue June 2 15:51:56 2020 Epoch 2, lr: 0.0003000, train loss: 0.34800, val loss: 0.48723, acc: 29.09605, qwk: 0.58493\nTue June 2 16:04:28 2020 Epoch 3, lr: 0.0003000, train loss: 0.29207, val loss: 0.27091, acc: 52.49529, qwk: 0.81714\nTue June 2 16:17:01 2020 Epoch 4, lr: 0.0002965, train loss: 0.26521, val loss: 0.26736, acc: 57.15631, qwk: 0.80364\nTue June 2 16:29:33 2020 Epoch 5, lr: 0.0002921, train loss: 0.24412, val loss: 0.24422, acc: 56.07345, qwk: 0.84068\nTue June 2 16:42:05 2020 Epoch 6, lr: 0.0002861, train loss: 0.23085, val loss: 0.25306, acc: 58.05085, qwk: 0.84429\nTue June 2 16:54:38 2020 Epoch 7, lr: 0.0002785, train loss: 0.21998, val loss: 0.21920, acc: 62.14689, qwk: 0.86278\nTue June 2 17:07:10 2020 Epoch 8, lr: 0.0002694, train loss: 0.21062, val loss: 0.23400, acc: 61.91149, qwk: 0.86170\nTue June 2 17:19:47 2020 Epoch 9, lr: 0.0002589, train loss: 0.20040, val loss: 0.27417, acc: 57.10923, qwk: 0.81771\nTue June 2 17:32:25 2020 Epoch 10, lr: 0.0002471, train loss: 0.18900, val loss: 0.26732, acc: 64.92467, qwk: 0.84131\nTue June 2 17:45:05 2020 Epoch 11, lr: 0.0002342, train loss: 0.18640, val loss: 0.21936, acc: 63.27684, qwk: 0.86580\nTue June 2 17:57:42 2020 Epoch 12, lr: 0.0002203, train loss: 0.17387, val loss: 0.22863, acc: 61.25235, qwk: 0.86871\nTue June 2 18:10:23 2020 Epoch 13, lr: 0.0002055, train loss: 0.16491, val loss: 0.23071, acc: 66.85499, qwk: 0.87892\nTue June 2 18:23:00 2020 Epoch 14, lr: 0.0001901, train loss: 0.15448, val loss: 0.24338, acc: 68.45574, qwk: 0.87342\nTue June 2 18:35:39 2020 Epoch 15, lr: 0.0001743, train loss: 0.14536, val loss: 0.22043, acc: 65.11299, qwk: 0.87169\nTue June 2 18:48:18 2020 Epoch 16, lr: 0.0001581, train loss: 0.13918, val loss: 0.22007, acc: 67.65537, qwk: 0.88284\nTue June 2 19:00:55 2020 Epoch 17, lr: 0.0001419, train loss: 0.13121, val loss: 0.24287, acc: 66.71375, qwk: 0.86357\nTue June 2 19:13:35 2020 Epoch 18, lr: 0.0001257, train loss: 0.12249, val loss: 0.21583, acc: 66.80791, qwk: 0.88478\nTue June 2 19:26:14 2020 Epoch 19, lr: 0.0001099, train loss: 0.11325, val loss: 0.21401, acc: 71.13936, qwk: 0.89178\nTue June 2 19:38:55 2020 Epoch 20, lr: 0.0000945, train loss: 0.10602, val loss: 0.21250, acc: 70.00942, qwk: 0.89256\nTue June 2 19:51:32 2020 Epoch 21, lr: 0.0000797, train loss: 0.09965, val loss: 0.21149, acc: 70.33898, qwk: 0.89590\nTue June 2 20:03:59 2020 Epoch 22, lr: 0.0000658, train loss: 0.09425, val loss: 0.22203, acc: 70.76271, qwk: 0.89493\nTue June 2 20:16:28 2020 Epoch 23, lr: 0.0000529, train loss: 0.08843, val loss: 0.22948, acc: 71.70433, qwk: 0.89304\nTue June 2 20:28:56 2020 Epoch 24, lr: 0.0000411, train loss: 0.08448, val loss: 0.21200, acc: 71.18644, qwk: 0.89947\nTue June 2 20:41:25 2020 Epoch 25, lr: 0.0000306, train loss: 0.07898, val loss: 0.21873, acc: 72.55179, qwk: 0.90021\nTue June 2 20:53:53 2020 Epoch 26, lr: 0.0000215, train loss: 0.07369, val loss: 0.21842, acc: 72.64595, qwk: 0.90240\nTue June 2 21:06:20 2020 Epoch 27, lr: 0.0000139, train loss: 0.07264, val loss: 0.21501, acc: 73.21092, qwk: 0.90450\nTue June 2 21:18:49 2020 Epoch 28, lr: 0.0000079, train loss: 0.06950, val loss: 0.21616, acc: 73.35217, qwk: 0.90264\nTue June 2 21:31:16 2020 Epoch 29, lr: 0.0000035, train loss: 0.06787, val loss: 0.21195, acc: 73.11676, qwk: 0.90434\nTue June 2 21:43:43 2020 Epoch 30, lr: 0.0000009, train loss: 0.06801, val loss: 0.21014, acc: 73.11676, qwk: 0.90468\n```","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Thank you for reading!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}