{"cells":[{"metadata":{},"cell_type":"markdown","source":"# (1)사전학습, 점진적 학습데이터 확대를 통한 기존 LSTM 모델의 개선"},{"metadata":{"_cell_guid":"522ce330-d4f9-4fbe-9684-4b65fd684cca","_uuid":"174484daa5f084ce4970f5048d3e05d2c4429787"},"cell_type":"markdown","source":"## 19.11.05. 내용 수정\n1. 오타 수정(2.1.1. 에서) \"10epoch정도에서 0.33%\" >>> \"10epoch정도에서 33%\"\n\n## 목차\n- 요약\n- 가설 및 실험 내용\n- 결과 분석 및 고찰\n\n## 1. 요약\n본 \"Quick, Draw!\" 문제에서는 이미지 분류 문제로서, CIFAR의 이미지 분류문제가 단순히 이미지 정보만 주어진 것에 비해, \n획순정보라는 부가적인 정보가 들어가있어 획순정보의 활용 방식이 기대가 되는 흥미로운 주제입니다.\n본 competition의 Notebook에서 stroke 정보를 활용하고자 LSTM을 사용한 \n아래의 notebook을 기초로 하여 pretraining 방법과 학습데이터의 점진적 증가를 통해 해당 모델의 성능을 개선하고자 하였습니다.\n\nhttps://www.kaggle.com/kmader/quickdraw-baseline-lstm-reading-and-submission\n\nLSTM은 sequence에 대하여 적응할 수 있는 모델이기 때문에, 획순 정보를 다루는데 적합한 모델로 파악됩니다.\n\n## 2.1.1. 가설1. 단순한 그림으로 pretraining하면 획순 개념을 학습할 수 있을 것이다.\n\n도형(line.csv, triangle.csv, square.csv, otagon.csv, hexagon.csv, circle.csv, squiggle.csv) 을 구분하는 모델을 pretraining하고\n복잡한 그림을 점차 분석해나가는 방식으로 모델을 훈련하면, 모델의 layer를 줄이고 효과적으로 모델을 구현할 수 있지 않을까라는 착안을 하였습니다.\n\n간단하고 명확한 이미지를 갖는 class에 대해서 pretraining을 진행하여 충분히 정확도를 높이고, \n삼각형, 사각형, 원만을 학습데이터로 사용했을 때, early stopping 조건에 의해 10epoch정도에서 33% cat_acc(categorical accuracy)로 학습이 종료되었습니다.\nclass당 7500 이미지로 하였는데도, 33%를 넘을 수가 없었습니다.\n\n학습데이터에 도형을(직선, 삼각형, 사각형, 오각형, 육각형, 원, 구불구불한 선(squiggle)) 더 추가하고 훈련데이터를 class 당 7500개 이미지로 하여 훈련을 진행했습니다.\n이 경우에는 100epoch에서 84.7% cat_acc(85.3% val_cat_acc)로 학습이 종료되었습니다. 따라서, class 종류가 늘어나면 모델의 성능이 개선된 것으로 보여집니다.\n\n\n340 종의 class를 다시 11 개의 대분류(교통, 사물, 동물 등)로 분류하고, 각 대분류에서 단순한 이미지를 갖는 3가지의 class만 택하여 pretraining을 진행했습니다.\n73% cat_acc까지 pretraining을 한 후, 해당시점의 가중치를 저장하여 340 종 모든 class를 학습데이터로 하여 main training을 진행했습니다.\n\npretraining 후 main training을 시작할 때 cat_acc는 73%에서 0.003%로 감소하였습니다.\n\n## 2.1.2. 가설2. 학습데이터를 점진적으로 늘리면 학습이 잘 될 것이다.\n\npretraining(32 class, 7500 이미지/class) 이후 main training에서 학습데이터를 340 class를 각 class 당 750개의 이미지로 20 epoch을 학습 후 학습 이미지의 개수를 두 배로(1500개) 늘려서 학습을 진행했습니다.\n이 후 20 epoch 당 이미지의 개수를 두 배씩 늘려 학습하는 방식으로 학습을 진행합니다.\n\npretraining에서 main training으로 넘어갈 때 cat_acc가 73% 에서 3%로 급격히 떨어진 것에 비하여, main training에서는 20 epoch에서 다음 20 epoch 으로 넘어갈 때,\n41.7% 에서 35.1%로 정확도가 어느정도 유지되는 것을 볼 수 있습니다. 또 era가 증가할수록 그 감소폭이(-6.6p%, -0.2p%) 줄어들다가 era5로 넘어갈 때는 오히려 증가함을 볼 수 있습니다.\n\n||||||||||\n|--- |--- |--- |--- |--- |--- |--- |--- |--- |\n|era|era당<br>epoch 수|class 당<br> 이미지 수|첫-마지막<br> cat_acc|첫<br> cat_acc|마지막<br> cat_acc|마지막-마지막<br> cat_acc|epoch당<br> 소요시간|era<br> 총 소요시간|\n|era1_pretraining|100|7500|-|0%|73%|-|120s|40m|\n|era2_maintraining|20|750|-70p%|3%|41.7%|-31.3p%|120s|40m|\n|era3_maintraining|20|1500|-6.6p%|35.1%|49.3%|+7.6p%|180s|60m|\n|era4_maintraining|20|3000|-0.2p%|49.1%|56.9%|+7.6p%|6m|120m|      \n|era5_maintraining|20|6000|+1.5p%|58.4%|%|p%|12.5m|240m?|      \n\n## 3. 결과분석 및 고찰\nera5의 모델로 late submission을 한 결과 0.74238를 획득했습니다.\nera5 모델은 총 epoch이 많이 부족하기 때문에, score가 높지 못한 것으로 파악됩니다. \n충분히 많은 epoch을 돌리고, 대조군을 활용하여 비교를 한다면 본 가설에 대해서 더욱 정확한 고찰을 할 수 있습니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"cd /kaggle/input/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pwd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls quickdraw-doodle-recognition/train_simplified/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls quickdraw2/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# (2) 340개 class의 11 종 대분류\n사물이 161개로 가장 많고, 동물이 53개, 교통이 29개로 많습니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndf_11_category = pd.read_csv('quickdraw2/340classes_11_category_labeling_libre.csv')\ndisplay(df_11_category)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# (3) 340개 class의 이미지 plotting 및 pretraining 데이터로 단순한 그림 찾기"},{"metadata":{"trusted":true},"cell_type":"code","source":"pwd","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom keras.utils.np_utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences\nfrom sklearn.preprocessing import LabelEncoder\nimport pandas as pd\nfrom keras.metrics import top_k_categorical_accuracy\ndef top_3_accuracy(x,y): return top_k_categorical_accuracy(x,y, 3)\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom glob import glob\nimport gc\ngc.enable()\ndef get_available_gpus():\n    from tensorflow.python.client import device_lib\n    local_device_protos = device_lib.list_local_devices()\n    return [x.name for x in local_device_protos if x.device_type == 'GPU']\n\n# base 디렉토리와 test 용 csv 파일의 경로를 설정합니다.\nbase_dir = os.path.join('.', 'quickdraw-doodle-recognition')\ntest_path = os.path.join(base_dir, 'test_simplified.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7acacf8e960084782425ef1a1a3fd532a240ad48"},"cell_type":"code","source":"from ast import literal_eval\nALL_TRAIN_PATHS = glob(os.path.join(base_dir, 'train_simplified', '*.csv'))\nCOL_NAMES = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\n\ndef _stack_it(raw_strokes):\n    \"\"\"preprocess the string and make \n    a standard Nx3 stroke vector\"\"\"\n    stroke_vec = literal_eval(raw_strokes) # string->list\n    # unwrap the list\n    in_strokes = [(xi,yi,i)  \n     for i,(x,y) in enumerate(stroke_vec) \n     for xi,yi in zip(x,y)]\n    c_strokes = np.stack(in_strokes)\n    # replace stroke id with 1 for continue, 2 for new\n    c_strokes[:,2] = [1]+np.diff(c_strokes[:,2]).tolist()\n    c_strokes[:,2] += 1 # since 0 is no stroke\n    # pad the strokes with zeros\n    return pad_sequences(c_strokes.swapaxes(0, 1), \n                         maxlen=STROKE_COUNT, \n                         padding='post').swapaxes(0, 1)\n\ndef read_batch(var_l_category_list=[],\n               samples=5, \n               start_row=0,\n               max_rows = 1000):\n    \"\"\"\n    load and process the csv files\n    this function is horribly inefficient but simple\n    \"\"\"\n    out_df_list = []\n    if len(var_l_category_list) != 0:\n        for c_path in ALL_TRAIN_PATHS:\n            if c_path.split('/')[-1] in var_l_category_list:\n                c_df = pd.read_csv(c_path, nrows=max_rows, skiprows=start_row)\n                c_df.columns=COL_NAMES\n                out_df_list += [c_df.sample(samples)[['drawing', 'word']]]\n    elif len(var_l_category_list) == 0:\n        for c_path in ALL_TRAIN_PATHS:\n            c_df = pd.read_csv(c_path, nrows=max_rows, skiprows=start_row)\n            c_df.columns=COL_NAMES\n            out_df_list += [c_df.sample(samples)[['drawing', 'word']]]\n    full_df = pd.concat(out_df_list)\n    full_df['drawing'] = full_df['drawing'].\\\n        map(_stack_it)\n    \n    return full_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(ALL_TRAIN_PATHS))\nprint(base_dir)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"word_encoder = LabelEncoder()\ndef f_preparing_dataset(var_list=[]):\n    train_args = dict(var_l_category_list=var_list,\n                      samples=TRAIN_SAMPLES, \n                      start_row=0, \n                      max_rows=int(TRAIN_SAMPLES*1.5))\n    valid_args = dict(var_l_category_list=var_list,\n                      samples=VALID_SAMPLES, \n                      start_row=train_args['max_rows']+1, \n                      max_rows=VALID_SAMPLES+25)\n    test_args = dict(var_l_category_list=var_list,\n                     samples=TEST_SAMPLES, \n                     start_row=valid_args['max_rows']+train_args['max_rows']+1, \n                     max_rows=TEST_SAMPLES+25)\n    train_df, valid_df, test_df = [read_batch(**train_args), read_batch(**valid_args), read_batch(**test_args)]\n    \n    word_encoder.fit(train_df['word'])\n    print('words', len(word_encoder.classes_), '=>', ', '.join([x for x in word_encoder.classes_]))\n\n    time_f = time.time()\n    time_d = time_f - time_i\n    print(\"{:2.0f}mins {:2.2f}sec\".format(time_d//60, time_d%60))\n    return [train_df, valid_df, test_df]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import time\ntime_i = time.time()\n\nbatch_size = 4096\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 1    # 이미지 plot을 위해 각 dataset마다 1개 sample만 읽어옵니다.\nVALID_SAMPLES = 1    # 이미지 plot을 위해 각 dataset마다 1개 sample만 읽어옵니다.\nTEST_SAMPLES = 1     # 이미지 plot을 위해 각 dataset마다 1개 sample만 읽어옵니다.\n\ntrain_df, valid_df, test_df = f_preparing_dataset()\n\ntime_f = time.time()\ntime_d = time_f - time_i\nprint(\"{:2.0f}mins {:2.2f}sec\".format(time_d//60, time_d%60))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ff5ddced-d77e-473f-899d-82cf11ad2bd9","_uuid":"409468f1d5abd17b819482473a4f354a61f8d7ef","trusted":true},"cell_type":"code","source":"def get_Xy(in_df):\n    X = np.stack(in_df['drawing'], 0)\n    y = to_categorical(word_encoder.transform(in_df['word'].values))\n    return X, y\ntrain_X, train_y = get_Xy(train_df)\nvalid_X, valid_y = get_Xy(valid_df)\ntest_X, test_y = get_Xy(test_df)\nprint(train_X.shape)\nprint(valid_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\ndef f_plot_img_X(var_arr_X, nb_samples, n_img=1, n_class=1):\n    time_i = time.time()\n    # rand_idxs = np.random.choice(range(train_X.shape[0]), size = 9)\n    \n    n_tot = n_img * n_class\n    n_wide = 10\n    n_height = n_tot//10 + (n_tot%10 != 0)\n    print(n_tot)\n    print(n_height)\n    print(n_wide*n_height)\n    rand_idxs = np.array([nb_samples*i+j for i in range(n_class) for j in range(n_img) ])\n    fig, m_axs = plt.subplots(n_height, n_wide, figsize = (16, int(1.6*n_height)))\n\n    for c_id, c_ax in zip(rand_idxs, m_axs.flatten()):\n        test_arr = var_arr_X[c_id]\n        test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n        lab_idx = np.cumsum(test_arr[:,2]-1)\n\n\n        for i in np.unique(lab_idx):\n            c_ax.plot(test_arr[lab_idx==i,0], \n                    np.max(test_arr[:,1])-test_arr[lab_idx==i,1], '.-')\n        c_ax.axis('off')\n        c_ax.set_title(word_encoder.classes_[np.argmax(train_y[c_id])])\n\n    plt.show()\n    time_f = time.time()\n    time_d = time_f - time_i\n    print(\"{:2.0f}mins {:2.2f}sec\".format(time_d//60, time_d%60))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### (3).1. 340class 이미지 plotting\n340class에 대한 그림을 train_X, test_X, valid_X로부터 plot하였습니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"f_plot_img_X(train_X, nb_samples=TRAIN_SAMPLES, n_img=1, n_class=340)\nf_plot_img_X(valid_X, nb_samples=VALID_SAMPLES, n_img=1, n_class=340)\nf_plot_img_X(test_X, nb_samples=TEST_SAMPLES, n_img=1, n_class=340)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (4) 사전학습 era1-pretraining(100epoch/7500img/40mins)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 11가지의 카테고리로 나누고 각 카테고리에서 3개의 class를 가지고 사전훈련을 진행합니다.\n# 11가지 카테고리\n# 교통 /사물/ 동물/ 동작/ 음식/ 신체,인물/ 식물/ 과일/ 장소,건물/ 역할/ 개념\n# pretraining의 학습데이터는 총 32개 csv 파일\nstr_11_category_list = \\\n'''airplane.csv\nambulance.csv\nbicycle.csv\nalarm clock.csv\nanvil.csv\nbackpack.csv\nant.csv\nbat.csv\nbear.csv\nanimal migration.csv\ncamouflage.csv\nsnorkel.csv\nbanana.csv\nbirthday cake.csv\nbread.csv\nangel.csv\narm.csv\nbeard.csv\nbroccoli.csv\nbush.csv\ncactus.csv\napple.csv\nbanana.csv\nblueberry.csv\nbeach.csv\ncastle.csv\ngarden.csv\ncampfire.csv\nfire hydrant.csv\npool.csv\ncircle.csv\nhexagon.csv\nhurricane.csv'''\nl_category_list = str_11_category_list.split('\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 분당 900 TRAIN_SAMPLES 처리 함\nimport time\n\nbatch_size = 4096\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 7500\nVALID_SAMPLES = 75\nTEST_SAMPLES = 50\n\nword_encoder = LabelEncoder()\ntrain_df, valid_df, test_df = f_preparing_dataset(l_category_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing import sequence\n\n# main_training 의 모델 구조와 맞추기 위해 340 class 개수로 post padding을 준다.\ntrain_X, train_y = get_Xy(train_df)\ntrain_y = sequence.pad_sequences(train_y, maxlen=340, padding='post')\nvalid_X, valid_y = get_Xy(valid_df)\nvalid_y = sequence.pad_sequences(valid_y, maxlen=340, padding='post')\ntest_X, test_y = get_Xy(test_df)\ntest_y = np.array(sequence.pad_sequences(test_y, maxlen=340, padding='post'))\n\nprint(train_X.shape)\nprint(train_y.shape)\nprint(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f_plot_img_X(train_X, nb_samples=TRAIN_SAMPLES, n_img=1, n_class=32)\nf_plot_img_X(valid_X, nb_samples=VALID_SAMPLES, n_img=1, n_class=32)\nf_plot_img_X(test_X, nb_samples=TEST_SAMPLES, n_img=1, n_class=32)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d65490e7-302e-4232-afe7-4e9499010e31","_uuid":"ba9d55554ba9e4177df5f0645ca1e0f5e4393ca3","trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import BatchNormalization, Conv1D, LSTM, Dense, Dropout\nif len(get_available_gpus())>0:\n    # https://twitter.com/fchollet/status/918170264608817152?lang=en\n    from keras.layers import CuDNNLSTM as LSTM # this one is about 3x faster on GPU instances\nstroke_read_model = Sequential()\n# from keras.utils.training_utils import multi_gpu_model\n\n\nstroke_read_model.add(BatchNormalization(input_shape = (None,)+train_X.shape[2:]))\n# filter count and length are taken from the script https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py\nstroke_read_model.add(Conv1D(48, (5,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Conv1D(64, (5,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Conv1D(96, (3,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(LSTM(128, return_sequences = True))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(LSTM(128, return_sequences = False))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Dense(512))\nstroke_read_model.add(Dropout(0.3))\n\n# main trainig 에서 len(word_encoder.classes_)=340 이므로 Dense(340)으로 맞춰준다.\nstroke_read_model.add(Dense(340, activation = 'softmax'))\n\n# 멀티 GPU 사용하기\n# stroke_read_model = multi_gpu_model(stroke_read_model, gpus=1)\nstroke_read_model.compile(optimizer = 'adam', \n                          loss = 'categorical_crossentropy', \n                          metrics = ['categorical_accuracy', top_3_accuracy])\nstroke_read_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2a549512-a9d9-4afd-b748-3e1c3296e193","_uuid":"5fda10b30c47a8cf6ea822ed0a4a1d7cd2c81195","trusted":true},"cell_type":"code","source":"weight_path=\"pretrain_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n                                   verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n#                       patience=5) # probably needs to be more patient, but kaggle time is limited\n                      patience=5*2)\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = stroke_read_model.fit(train_X, train_y,\n                      validation_data = (valid_X, valid_y), \n                      batch_size = batch_size,\n                      epochs = 100,\n                      callbacks = callbacks_list,\n                      shuffle = True)\nstroke_read_model.save_weights('pretrain_weights.hd')\n\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\nEpoch 20/20\n255000/255000 [==============================] - 25s 96us/step - loss: 3.8444 - categorical_accuracy: 0.1638 - top_3_accuracy: 0.3205 - val_loss: 3.5913 - val_categorical_accuracy: 0.2022 - val_top_3_accuracy: 0.3750\n\nEpoch 00020: val_loss improved from 3.72432 to 3.59134, saving model to pretrain_weights.best.hdf5\n17000/17000 [==============================] - 1s 35us/step\nAccuracy: 20.4%, Top 3 Accuracy 37.4%\n\nEpoch 20/20\n255000/255000 [==============================] - 25s 96us/step - loss: 2.2690 - categorical_accuracy: 0.4540 - top_3_accuracy: 0.6680 - val_loss: 2.0096 - val_categorical_accuracy: 0.5108 - val_top_3_accuracy: 0.7212\n\nEpoch 00020: val_loss improved from 2.01422 to 2.00963, saving model to pretrain_weights.best.hdf5\n17000/17000 [==============================] - 1s 34us/step\nAccuracy: 50.5%, Top 3 Accuracy 71.7%\n\nEpoch 20/20\n255000/255000 [==============================] - 25s 96us/step - loss: 2.0509 - categorical_accuracy: 0.5009 - top_3_accuracy: 0.7121 - val_loss: 1.8247 - val_categorical_accuracy: 0.5543 - val_top_3_accuracy: 0.7570\n\nEpoch 00020: val_loss improved from 1.83847 to 1.82466, saving model to pretrain_weights.best.hdf5\n17000/17000 [==============================] - 1s 35us/step\nAccuracy: 54.7%, Top 3 Accuracy 75.2%\n\nEpoch 20/20\n255000/255000 [==============================] - 25s 96us/step - loss: 1.9077 - categorical_accuracy: 0.5315 - top_3_accuracy: 0.7380 - val_loss: 1.7446 - val_categorical_accuracy: 0.5709 - val_top_3_accuracy: 0.7711\n\nEpoch 00020: val_loss did not improve from 1.72911\n17000/17000 [==============================] - 1s 35us/step\nAccuracy: 56.6%, Top 3 Accuracy 76.5%\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_era1_pretraining = pd.DataFrame(history.history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (5).1. main training (era2_maintraining/20epoch/750img/ 41.7%cat_acc/40mins)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 분당 30 TRAIN_SAMPLES 처리 함\nimport time\n\nbatch_size = 4096\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 750\nVALID_SAMPLES = int(TRAIN_SAMPLES/10)\nTEST_SAMPLES = int(TRAIN_SAMPLES/75*50)\n\nword_encoder = LabelEncoder()\ntrain_df, valid_df, test_df = f_preparing_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing import sequence\n\n# main_training 의 모델 구조와 맞추기 위해 340 class 개수로 post padding을 준다.\ntrain_X, train_y = get_Xy(train_df)\ntrain_y = sequence.pad_sequences(train_y, maxlen=340, padding='post')\nvalid_X, valid_y = get_Xy(valid_df)\nvalid_y = sequence.pad_sequences(valid_y, maxlen=340, padding='post')\ntest_X, test_y = get_Xy(test_df)\ntest_y = np.array(sequence.pad_sequences(test_y, maxlen=340, padding='post'))\n\nprint(train_X.shape)\nprint(train_y.shape)\nprint(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weight_path=\"era2_maintraining_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n                                   verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n#                       patience=5) # probably needs to be more patient, but kaggle time is limited\n                      patience=5*2)\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"825b3af8-9451-487b-a1e1-538f2f1489e1","_uuid":"ed2fc26af74aed1a93bbc253d61b72db5a81f5cc","trusted":true},"cell_type":"code","source":"history2 = stroke_read_model.fit(train_X, train_y,\n                      validation_data = (valid_X, valid_y), \n                      batch_size = batch_size,\n                      epochs = 20,\n                      callbacks = callbacks_list,\n                      shuffle = True)\nstroke_read_model.save_weights('era2_maintraining_weights.hd')\n\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(pd.DataFrame(history2.history))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (5).2. main training (era3_maintraining/ 20epoch/ 1500img/ 49.3%cat_acc/ 60m)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 분당 60 TRAIN_SAMPLES 처리 함\nimport time\n\nbatch_size = 4096\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 1500\nVALID_SAMPLES = int(TRAIN_SAMPLES/10)\nTEST_SAMPLES = int(TRAIN_SAMPLES/75*50)\n\nword_encoder = LabelEncoder()\ntrain_df, valid_df, test_df = f_preparing_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing import sequence\n\n# main_training 의 모델 구조와 맞추기 위해 340 class 개수로 post padding을 준다.\ntrain_X, train_y = get_Xy(train_df)\ntrain_y = sequence.pad_sequences(train_y, maxlen=340, padding='post')\nvalid_X, valid_y = get_Xy(valid_df)\nvalid_y = sequence.pad_sequences(valid_y, maxlen=340, padding='post')\ntest_X, test_y = get_Xy(test_df)\ntest_y = np.array(sequence.pad_sequences(test_y, maxlen=340, padding='post'))\n\nprint(train_X.shape)\nprint(train_y.shape)\nprint(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weight_path=\"era3_maintraining_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n                                   verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n#                       patience=5) # probably needs to be more patient, but kaggle time is limited\n                      patience=5*2)\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history3 = stroke_read_model.fit(train_X, train_y,\n                      validation_data = (valid_X, valid_y), \n                      batch_size = batch_size,\n                      epochs = 20,\n                      callbacks = callbacks_list,\n                      shuffle = True)\nstroke_read_model.save_weights('era3_maintraining_weights.hd')\n\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(pd.DataFrame(history3.history))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (5).3. main training (era4_maintraining/20epoch/3000img/ 56.9%cat_acc/120mins)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# # 분당 60 TRAIN_SAMPLES 처리 함\n# import time\n\n# batch_size = 4096\n# STROKE_COUNT = 196\n# TRAIN_SAMPLES = 3000\n# VALID_SAMPLES = int(TRAIN_SAMPLES/10)\n# TEST_SAMPLES = int(TRAIN_SAMPLES/75*50)\n\n# word_encoder = LabelEncoder()\n# train_df, valid_df, test_df = f_preparing_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from keras.preprocessing import sequence\n\n# # main_training 의 모델 구조와 맞추기 위해 340 class 개수로 post padding을 준다.\n# train_X, train_y = get_Xy(train_df)\n# train_y = sequence.pad_sequences(train_y, maxlen=340, padding='post')\n# valid_X, valid_y = get_Xy(valid_df)\n# valid_y = sequence.pad_sequences(valid_y, maxlen=340, padding='post')\n# test_X, test_y = get_Xy(test_df)\n# test_y = np.array(sequence.pad_sequences(test_y, maxlen=340, padding='post'))\n\n# print(train_X.shape)\n# print(train_y.shape)\n# print(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# weight_path=\"era3_maintraining_weights.best.hdf5\".format('stroke_lstm_model')\n\n# checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n#                              save_best_only=True, mode='min', save_weights_only = True)\n\n\n# reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n#                                    verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\n# early = EarlyStopping(monitor=\"val_loss\", \n#                       mode=\"min\", \n# #                       patience=5) # probably needs to be more patient, but kaggle time is limited\n#                       patience=5*2)\n# callbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# history4 = stroke_read_model.fit(train_X, train_y,\n#                       validation_data = (valid_X, valid_y), \n#                       batch_size = batch_size,\n#                       epochs = 20,\n#                       callbacks = callbacks_list,\n#                       shuffle = True)\n# stroke_read_model.save_weights('era4_maintraining_weights.hd')\n\n# lstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\n# print('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display(pd.DataFrame(history4.history))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (5).4. main training (era5_maintraining/20epoch/6000img/ %cat_acc/240mins)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# # 분당 60 TRAIN_SAMPLES 처리 함\n# import time\n\n# batch_size = 4096\n# STROKE_COUNT = 196\n# TRAIN_SAMPLES = 6000\n# VALID_SAMPLES = int(TRAIN_SAMPLES/10)\n# TEST_SAMPLES = int(TRAIN_SAMPLES/75*50)\n\n# word_encoder = LabelEncoder()\n# train_df, valid_df, test_df = f_preparing_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from keras.preprocessing import sequence\n\n# # main_training 의 모델 구조와 맞추기 위해 340 class 개수로 post padding을 준다.\n# train_X, train_y = get_Xy(train_df)\n# train_y = sequence.pad_sequences(train_y, maxlen=340, padding='post')\n# valid_X, valid_y = get_Xy(valid_df)\n# valid_y = sequence.pad_sequences(valid_y, maxlen=340, padding='post')\n# test_X, test_y = get_Xy(test_df)\n# test_y = np.array(sequence.pad_sequences(test_y, maxlen=340, padding='post'))\n\n# print(train_X.shape)\n# print(train_y.shape)\n# print(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# weight_path=\"era3_maintraining_weights.best.hdf5\".format('stroke_lstm_model')\n\n# checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n#                              save_best_only=True, mode='min', save_weights_only = True)\n\n\n# reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n#                                    verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\n# early = EarlyStopping(monitor=\"val_loss\", \n#                       mode=\"min\", \n# #                       patience=5) # probably needs to be more patient, but kaggle time is limited\n#                       patience=5*2)\n# callbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# history5 = stroke_read_model.fit(train_X, train_y,\n#                       validation_data = (valid_X, valid_y), \n#                       batch_size = batch_size,\n#                       epochs = 20,\n#                       callbacks = callbacks_list,\n#                       shuffle = True)\n# stroke_read_model.save_weights('era5_maintraining_weights.hd')\n\n# lstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\n# print('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display(pd.DataFrame(history5.history))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## (6).1. 분석 (precision, recall, f1-score, support, confusion_matrix)"},{"metadata":{"_cell_guid":"ee75c585-b134-4ea2-b8f3-219e24efd1f1","_uuid":"6b9cdf52d233de60108d72f540db978801b578c1","trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\ntest_cat = np.argmax(test_y, 1)\npred_y = stroke_read_model.predict(test_X, batch_size = 4096)\npred_cat = np.argmax(pred_y, 1)\nplt.matshow(confusion_matrix(test_cat, pred_cat))\nprint(classification_report(test_cat, pred_cat, \n                            target_names = [x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"db1d371b-4b2c-478f-b6df-76db58a24fbe","_uuid":"bd9a16adcb46e07d7949644e69bf3483f7dce571"},"cell_type":"markdown","source":"## (6).2. 분석 (그리는 도중의 모델의 예측)"},{"metadata":{"_cell_guid":"bf7dff37-c634-4930-8dae-3dba8090c251","_uuid":"c43e87e7eccfb72dd35e64d872a7d658ffa535a3","scrolled":false,"trusted":true},"cell_type":"code","source":"points_to_use = [5, 15, 20, 30, 40, 50]\npoints_to_user = [108]\nsamples = 12\nword_dex = lambda x: word_encoder.classes_[x]\nrand_idxs = np.random.choice(range(test_X.shape[0]), size = samples)\nfig, m_axs = plt.subplots(len(rand_idxs), len(points_to_use), figsize = (24, samples/8*24))\nfor c_id, c_axs in zip(rand_idxs, m_axs):\n    res_idx = np.argmax(test_y[c_id])\n    goal_cat = word_encoder.classes_[res_idx]\n    \n    for pt_idx, (pts, c_ax) in enumerate(zip(points_to_use, c_axs)):\n        test_arr = test_X[c_id, :].copy()\n        test_arr[pts:] = 0 # short sequences make CudnnLSTM crash, ugh \n        stroke_pred = stroke_read_model.predict(np.expand_dims(test_arr,0))[0]\n        top_10_idx = np.argsort(-1*stroke_pred)[:10]\n        top_10_sum = np.sum(stroke_pred[top_10_idx])\n        \n        test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n        lab_idx = np.cumsum(test_arr[:,2]-1)\n        for i in np.unique(lab_idx):\n            c_ax.plot(test_arr[lab_idx==i,0], \n                    np.max(test_arr[:,1])-test_arr[lab_idx==i,1], # flip y\n                      '.-')\n        c_ax.axis('off')\n        if pt_idx == (len(points_to_use)-1):\n            c_ax.set_title('Answer: %s (%2.1f%%) \\nPredicted: %s (%2.1f%%)' % (goal_cat, 100*stroke_pred[res_idx]/top_10_sum, word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum))\n        else:\n            c_ax.set_title('%s (%2.1f%%), %s (%2.1f%%)\\nCorrect: (%2.1f%%)' % (word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum, \n                                                                 word_dex(top_10_idx[1]), 100*stroke_pred[top_10_idx[1]]/top_10_sum, \n                                                                 100*stroke_pred[res_idx]/top_10_sum))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e99b1ed154f26381d12918e2b4e12db807e6535f"},"cell_type":"markdown","source":"## (7).1. Late Submission"},{"metadata":{"_cell_guid":"436a4fce-3843-4c84-8eeb-0161fe3c4e04","_uuid":"4f3a40e23f2e917b68171822944491ab348e15b3","trusted":true},"cell_type":"code","source":"sub_df = pd.read_csv(test_path)\nsub_df['drawing'] = sub_df['drawing'].map(_stack_it)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72825ea87d35ad96b0254e3af5f5aaf64fb9c78f"},"cell_type":"code","source":"sub_vec = np.stack(sub_df['drawing'].values, 0)\nsub_pred = stroke_read_model.predict(sub_vec, verbose=True, batch_size=4096)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"639ca8a511e5e1a02b6cd0333cc04213f8497487"},"cell_type":"code","source":"top_3_pred = [word_encoder.classes_[np.argsort(-1*c_pred)[:3]] for c_pred in sub_pred]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68dd3629f5e5b30bede2d4b485a6f1dfabc8d5a4"},"cell_type":"code","source":"top_3_pred = [' '.join([col.replace(' ', '_') for col in row]) for row in top_3_pred]\ntop_3_pred[:3]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9708406fba8087c68fd1b525d29d63cb7f476976"},"cell_type":"markdown","source":"## (7).2. 모델의 submission 데이터셋을 통한 예측"},{"metadata":{"trusted":true,"_uuid":"6a60baa74045ff401dab7e14dd20710dc4535f67"},"cell_type":"code","source":"fig, m_axs = plt.subplots(3,3, figsize = (16, 16))\nrand_idxs = np.random.choice(range(sub_vec.shape[0]), size = 9)\nfor c_id, c_ax in zip(rand_idxs, m_axs.flatten()):\n    test_arr = sub_vec[c_id]\n    test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n    lab_idx = np.cumsum(test_arr[:,2]-1)\n    for i in np.unique(lab_idx):\n        c_ax.plot(test_arr[lab_idx==i,0], \n                np.max(test_arr[:,1])-test_arr[lab_idx==i,1], '.-')\n    c_ax.axis('off')\n    c_ax.set_title(top_3_pred[c_id])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pwd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b5ece83cb6095e95ef5741e73508d9129be1e3d"},"cell_type":"code","source":"sub_df['word'] = top_3_pred\nsub_df[['key_id', 'word']].to_csv('/kaggle/working/submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls /kaggle/working/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv('submission.csv')\ndf_submission.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}