{"cells":[{"metadata":{"_cell_guid":"522ce330-d4f9-4fbe-9684-4b65fd684cca","_uuid":"174484daa5f084ce4970f5048d3e05d2c4429787"},"cell_type":"markdown","source":"# Overview\nThe notebook is modified from one that was made for the [Quick, Draw Dataset](https://www.kaggle.com/google/tinyquickdraw), it would actually be interesting to see how beneficial a transfer learning approach using that data as a starting point could be.\n\n## This Notebook\nThe notebook takes and preprocesses the data from the QuickDraw Competition step (strokes) and trains an LSTM. The outcome variable (y) is always the same (category). The stroke-based LSTM. The model takes the stroke data and 'preprocesses' it a bit using 1D convolutions and then uses two stacked LSTMs followed by two dense layers to make the classification. The model can be thought to 'read' the drawing stroke by stroke.\n\n## Fun Models\n\nAfter the classification models, we try to build a few models to understand what the LSTM actually does. Here we experiment step by step to see how the prediction changes with each stop\n\n### Next Steps\nThe next steps could be\n- use more data to train\n- include the country code (different countries draw different things, different ways)\n- more complex models","execution_count":null},{"metadata":{"_uuid":"d8ccded02a5f2c4a9d9ee2f7688114bcd2e1f11a"},"cell_type":"markdown","source":"### Model Parameters\nHere we keep track of the relevant parameters for the data preprocessing, model construction and training","execution_count":null},{"metadata":{"trusted":true,"_uuid":"8b08fbab2000a563b388f126eac74362641e497c"},"cell_type":"code","source":"batch_size = 4096\nSTROKE_COUNT = 96\nTRAIN_SAMPLES = 1024\nVALID_SAMPLES = 128\nTEST_SAMPLES = 128","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nimport gc\nfrom glob import glob\nfrom keras.metrics import top_k_categorical_accuracy\nimport pandas as pd\nfrom sklearn.preprocessing import LabelEncoder\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.utils.np_utils import to_categorical\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\n%matplotlib inline\ndef top_3_accuracy(x, y): return top_k_categorical_accuracy(x, y, 3)\n\n\ngc.enable()\n\n\ndef get_available_gpus():\n    from tensorflow.python.client import device_lib\n    local_device_protos = device_lib.list_local_devices()\n    return [x.name for x in local_device_protos if x.device_type == 'GPU']\n\n\nbase_dir = os.path.join('..', 'input')\ntest_path = os.path.join(base_dir, 'test_simplified.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7acacf8e960084782425ef1a1a3fd532a240ad48"},"cell_type":"code","source":"occ = {}\n\nfrom ast import literal_eval\nALL_TRAIN_PATHS = glob(os.path.join(base_dir, 'train_simplified', '*.csv'))\nCOL_NAMES = ['countrycode', 'drawing',\n             'key_id', 'recognized', 'timestamp', 'word']\n\n\ndef _stack_it(raw_strokes):\n    \"\"\"preprocess the string and make \n    a standard Nx3 stroke vector\"\"\"\n    stroke_vec = literal_eval(raw_strokes)  # string->list\n    # unwrap the list\n    in_strokes = [(xi, yi, i)\n                  for i, (x, y) in enumerate(stroke_vec)\n                  for xi, yi in zip(x, y)]\n    c_strokes = np.stack(in_strokes)\n    # replace stroke id with 1 for continue, 2 for new\n    c_strokes[:, 2] = [1] + np.diff(c_strokes[:, 2]).tolist()\n    c_strokes[:, 2] += 1  # since 0 is no stroke\n    if len(c_strokes) in occ:\n        occ[len(c_strokes)] += 1\n    else:\n        occ[len(c_strokes)] = 1\n    # pad the strokes with zeros\n    return pad_sequences(c_strokes.swapaxes(0, 1),\n                         maxlen=STROKE_COUNT,\n                         padding='post').swapaxes(0, 1)\n\n\ndef read_batch(samples=5,\n               start_row=0,\n               max_rows=1000):\n    \"\"\"\n    load and process the csv files\n    this function is horribly inefficient but simple\n    \"\"\"\n    out_df_list = []\n    for c_path in ALL_TRAIN_PATHS:\n        c_df = pd.read_csv(c_path, nrows=max_rows, skiprows=start_row)\n        c_df.columns = COL_NAMES\n        out_df_list += [c_df.sample(samples)[['drawing', 'word']]]\n    full_df = pd.concat(out_df_list)\n    full_df['drawing'] = full_df['drawing'].\\\n        map(_stack_it)\n\n    return full_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ec1854b21b36cd2fd7f7d0717aaa8da32506a6a"},"cell_type":"markdown","source":"# Reading and Parsing\nSince it is too much data (23GB) to read in at once, we just take a portion of it for training, validation and hold-out testing. This should give us an idea about how well the model works, but leaves lots of room for improvement later","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_args = dict(samples=TRAIN_SAMPLES,\n                  start_row=0,\n                  max_rows=int(TRAIN_SAMPLES))\nvalid_args = dict(samples=VALID_SAMPLES,\n                  start_row=train_args['max_rows'] + 1,\n                  max_rows=VALID_SAMPLES)\ntest_args = dict(samples=TEST_SAMPLES,\n                 start_row=valid_args['max_rows'] + train_args['max_rows'] + 1,\n                 max_rows=TEST_SAMPLES)\ntrain_df = read_batch(**train_args)\nvalid_df = read_batch(**valid_args)\ntest_df = read_batch(**test_args)\nword_encoder = LabelEncoder()\nword_encoder.fit(train_df['word'])\nprint('words', len(word_encoder.classes_), '=>',\n      ', '.join([x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6d29237e-ece3-4dfd-9095-475296f4a608","_uuid":"8bae16a4973a215861fbb536a602c4f5abf3b4bf"},"cell_type":"markdown","source":"# Stroke-based Classification\nHere we use the stroke information to train a model and see if the strokes give us a better idea of what the shape could be. ","execution_count":null},{"metadata":{"_cell_guid":"ff5ddced-d77e-473f-899d-82cf11ad2bd9","_uuid":"409468f1d5abd17b819482473a4f354a61f8d7ef","trusted":true},"cell_type":"code","source":"def get_Xy(in_df):\n    X = np.stack(in_df['drawing'], 0)\n    y = to_categorical(word_encoder.transform(in_df['word'].values))\n    return X, y\n\n\ntrain_X, train_y = get_Xy(train_df)\nvalid_X, valid_y = get_Xy(valid_df)\ntest_X, test_y = get_Xy(test_df)\nprint(train_X.shape)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"56240ed9-42b0-4f62-b3d1-f92017f04e30","_uuid":"5cc79204a0a1da048d1d58ba8dfdafd0af3ebcb8","trusted":true},"cell_type":"code","source":"fig, m_axs = plt.subplots(3, 3, figsize=(16, 16))\nrand_idxs = np.random.choice(range(train_X.shape[0]), size=9)\nfor c_id, c_ax in zip(rand_idxs, m_axs.flatten()):\n    test_arr = train_X[c_id]\n    test_arr = test_arr[test_arr[:, 2] > 0, :]  # only keep valid points\n    lab_idx = np.cumsum(test_arr[:, 2] - 1)\n    for i in np.unique(lab_idx):\n        c_ax.plot(test_arr[lab_idx == i, 0],\n                  np.max(test_arr[:, 1]) - test_arr[lab_idx == i, 1], '.-')\n    c_ax.axis('off')\n    c_ax.set_title(word_encoder.classes_[np.argmax(train_y[c_id])])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d65490e7-302e-4232-afe7-4e9499010e31","_uuid":"ba9d55554ba9e4177df5f0645ca1e0f5e4393ca3","trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import BatchNormalization, Conv1D, LSTM, Dense, Dropout, Bidirectional\nstroke_read_model = Sequential()\nstroke_read_model.add(BatchNormalization(\n    input_shape=(None,)+train_X.shape[2:]))\n\nstroke_read_model.add(Conv1D(64, (5,)))\nstroke_read_model.add(Dropout(0.0))\nstroke_read_model.add(Conv1D(64, (5,)))\nstroke_read_model.add(Dropout(0.0))\nstroke_read_model.add(Conv1D(64, (3,)))\nstroke_read_model.add(Dropout(0.0))\nstroke_read_model.add(LSTM(160, dropout = 0.0, return_sequences=True))\nstroke_read_model.add(LSTM(160, dropout = 0.0, return_sequences=True))\nstroke_read_model.add(LSTM(160, dropout = 0.0, return_sequences=False))\nstroke_read_model.add(Dense(512))\nstroke_read_model.add(Dropout(0.0))\nstroke_read_model.add(Dense(len(word_encoder.classes_), activation='softmax'))\nstroke_read_model.compile(optimizer='adam',\n                          loss='categorical_crossentropy',\n                          metrics=['categorical_accuracy', top_3_accuracy])\nstroke_read_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import plot_model\nplot_model(stroke_read_model, show_shapes=True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2a549512-a9d9-4afd-b748-3e1c3296e193","_uuid":"5fda10b30c47a8cf6ea822ed0a4a1d7cd2c81195","trusted":true},"cell_type":"code","source":"weight_path = \"{}_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1,\n                             save_best_only=True, mode='min', save_weights_only=True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10,\n                                   verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\",\n                      mode=\"min\",\n                      patience=8)  # probably needs to be more patient, but kaggle time is limited\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"825b3af8-9451-487b-a1e1-538f2f1489e1","_uuid":"ed2fc26af74aed1a93bbc253d61b72db5a81f5cc","trusted":true},"cell_type":"code","source":"from IPython.display import clear_output\nhist = stroke_read_model.fit(train_X, train_y,\n                      validation_data=(valid_X, valid_y),\n                      batch_size=batch_size,\n                      epochs=50,\n                      callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_df = pd.DataFrame(hist.history)\nplt.figure(figsize=(16, 16))\nplt.rc('font', size=48)\nplt.plot(hist_df.val_categorical_accuracy, lw=5, label='Validation Accuracy')\nplt.plot(hist_df.categorical_accuracy, lw=5, label='Training Accuracy')\nplt.plot(hist_df.val_top_3_accuracy, lw=5, label='Validation Top 3 Accuracy')\nplt.plot(hist_df.top_3_accuracy, lw=5, label='Training Top 3 Accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.grid()\nplt.savefig('hist.png', dpi=300)\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a7eb5b62-cf57-4380-8786-9ddc05be658f","_uuid":"858059b6c16d81f86460bef8fcf595e0d68d12b2","trusted":true},"cell_type":"code","source":"stroke_read_model.load_weights(weight_path)\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size=batch_size)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' %\n      (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\ntest_cat = np.argmax(test_y, 1)\npred_y = stroke_read_model.predict(test_X, batch_size = 4096)\npred_cat = np.argmax(pred_y, 1)\nplt.matshow(confusion_matrix(test_cat, pred_cat))\nprint(classification_report(test_cat, pred_cat, \n                            target_names = [x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"db1d371b-4b2c-478f-b6df-76db58a24fbe","_uuid":"bd9a16adcb46e07d7949644e69bf3483f7dce571"},"cell_type":"markdown","source":"# Reading Point by Point","execution_count":null},{"metadata":{"_cell_guid":"ee75c585-b134-4ea2-b8f3-219e24efd1f1","_uuid":"6b9cdf52d233de60108d72f540db978801b578c1","trusted":true},"cell_type":"code","source":"points_to_use = [5, 15, 20, 30, 40, 50]\npoints_to_user = [108]\nsamples = 12\nword_dex = lambda x: word_encoder.classes_[x]\nrand_idxs = np.random.choice(range(test_X.shape[0]), size = samples)\nfig, m_axs = plt.subplots(len(rand_idxs), len(points_to_use), figsize = (24, samples/8*24))\nfor c_id, c_axs in zip(rand_idxs, m_axs):\n    res_idx = np.argmax(test_y[c_id])\n    goal_cat = word_encoder.classes_[res_idx]\n    \n    for pt_idx, (pts, c_ax) in enumerate(zip(points_to_use, c_axs)):\n        test_arr = test_X[c_id, :].copy()\n        test_arr[pts:] = 0 # short sequences make CudnnLSTM crash, ugh \n        stroke_pred = stroke_read_model.predict(np.expand_dims(test_arr,0))[0]\n        top_10_idx = np.argsort(-1*stroke_pred)[:10]\n        top_10_sum = np.sum(stroke_pred[top_10_idx])\n        \n        test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n        lab_idx = np.cumsum(test_arr[:,2]-1)\n        for i in np.unique(lab_idx):\n            c_ax.plot(test_arr[lab_idx==i,0], \n                    np.max(test_arr[:,1])-test_arr[lab_idx==i,1], # flip y\n                      '.-')\n        c_ax.axis('off')\n        if pt_idx == (len(points_to_use)-1):\n            c_ax.set_title('Answer: %s (%2.1f%%) \\nPredicted: %s (%2.1f%%)' % (goal_cat, 100*stroke_pred[res_idx]/top_10_sum, word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum))\n        else:\n            c_ax.set_title('%s (%2.1f%%), %s (%2.1f%%)\\nCorrect: (%2.1f%%)' % (word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum, \n                                                                 word_dex(top_10_idx[1]), 100*stroke_pred[top_10_idx[1]]/top_10_sum, \n                                                                 100*stroke_pred[res_idx]/top_10_sum))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}