{"cells":[{"metadata":{"_cell_guid":"522ce330-d4f9-4fbe-9684-4b65fd684cca","_uuid":"174484daa5f084ce4970f5048d3e05d2c4429787"},"cell_type":"markdown","source":"# Overview\nThe notebook is modified from one that was made for the [Quick, Draw Dataset](https://www.kaggle.com/google/tinyquickdraw), it would actually be interesting to see how beneficial a transfer learning approach using that data as a starting point could be.\n\n## This Notebook\nThe notebook takes and preprocesses the data from the QuickDraw Competition step (strokes) and trains an a WaveNet-style classifier (wavenet in its original implemention is https://deepmind.com/blog/high-fidelity-speech-synthesis-wavenet/ is intended for synthesis, but the dilated convolution approach can be applied). We use the implementation [here](https://github.com/mjpyeon/wavenet-classifier/blob/master/WaveNetClassifier.py) as a reference.\n\n## Fun Models\n\nAfter the classification models, we try to build a few models to understand what the WaveNet actually does. Here we experiment step by step to see how the prediction changes with each stop\n\n### Next Steps\nThe next steps could be\n- use more data to train\n- include the country code (different countries draw different things, different ways)\n- more complex models"},{"metadata":{"_uuid":"39077055d70da04344d314d12dba7f846a5e3721"},"cell_type":"markdown","source":"### Model Parameters\nHere we keep all of the parameters to make keeping track of versions and hyperparameter optimization  easier"},{"metadata":{"trusted":true,"_uuid":"4e8c68db0704a070ead7a5cb324ab59dd4b5aebd"},"cell_type":"code","source":"n_filters = 128+64+128\ndilation_depth = 8\nactivation = 'softmax'\nscale_ratio = 1\nkernel_size = 2\npool_size_1 = 4\npool_size_2 = 8\nbatch_size = 2048//4\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 3000\nVALID_SAMPLES = 750\n\nTEST_SAMPLES = 500","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom keras.utils.np_utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences\nfrom sklearn.preprocessing import LabelEncoder\nimport pandas as pd\nfrom keras.metrics import top_k_categorical_accuracy\ndef top_3_accuracy(x,y): return top_k_categorical_accuracy(x,y, 3)\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom glob import glob\nimport gc\ngc.enable()\ndef get_available_gpus():\n    from tensorflow.python.client import device_lib\n    local_device_protos = device_lib.list_local_devices()\n    return [x.name for x in local_device_protos if x.device_type == 'GPU']\nbase_dir = os.path.join('..', 'input')\ntest_path = os.path.join(base_dir, 'test_simplified.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7acacf8e960084782425ef1a1a3fd532a240ad48"},"cell_type":"code","source":"from ast import literal_eval\nALL_TRAIN_PATHS = glob(os.path.join(base_dir, 'train_simplified', '*.csv'))\nCOL_NAMES = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\n\ndef _stack_it(raw_strokes):\n    \"\"\"preprocess the string and make \n    a standard Nx3 stroke vector\"\"\"\n    stroke_vec = literal_eval(raw_strokes) # string->list\n    # unwrap the list\n    in_strokes = [(xi,yi,i)  \n     for i,(x,y) in enumerate(stroke_vec) \n     for xi,yi in zip(x,y)]\n    c_strokes = np.stack(in_strokes)\n    # replace stroke id with 1 for continue, 2 for new\n    c_strokes[:,2] = [1]+np.diff(c_strokes[:,2]).tolist()\n    c_strokes[:,2] += 1 # since 0 is no stroke\n    # pad the strokes with zeros\n    return pad_sequences(c_strokes.swapaxes(0, 1), \n                         maxlen=STROKE_COUNT, \n                         padding='post').swapaxes(0, 1)\ndef read_batch(samples=5, \n               start_row=0,\n               max_rows=1000):\n    \"\"\"\n    load and process the csv files\n    this function is horribly inefficient but simple\n    \"\"\"\n    out_df_list = []\n    for c_path in ALL_TRAIN_PATHS:\n        c_df = pd.read_csv(c_path, nrows=max_rows, skiprows=start_row)\n        c_df.columns=COL_NAMES\n        out_df_list += [c_df.sample(samples)[['drawing', 'word']]]\n    full_df = pd.concat(out_df_list)\n    full_df['drawing'] = full_df['drawing'].\\\n        map(_stack_it)\n    \n    return full_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43c53589bedc9a53ea398be98cacc3cca4b24a76"},"cell_type":"markdown","source":"# Reading and Parsing\nSince it is too much data (23GB) to read in at once, we just take a portion of it for training, validation and hold-out testing. This should give us an idea about how well the model works, but leaves lots of room for improvement later"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_args = dict(samples=TRAIN_SAMPLES, \n                  start_row=0, \n                  max_rows=int(TRAIN_SAMPLES*1.5))\nvalid_args = dict(samples=VALID_SAMPLES, \n                  start_row=train_args['max_rows']+1, \n                  max_rows=VALID_SAMPLES+25)\ntest_args = dict(samples=TEST_SAMPLES, \n                 start_row=valid_args['max_rows']+train_args['max_rows']+1, \n                 max_rows=TEST_SAMPLES+25)\ntrain_df = read_batch(**train_args)\nvalid_df = read_batch(**valid_args)\ntest_df = read_batch(**test_args)\nword_encoder = LabelEncoder()\nword_encoder.fit(train_df['word'])\nprint('words', len(word_encoder.classes_), '=>', ', '.join([x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6d29237e-ece3-4dfd-9095-475296f4a608","_uuid":"8bae16a4973a215861fbb536a602c4f5abf3b4bf"},"cell_type":"markdown","source":"# Stroke-based Classification\nHere we use the stroke information to train a model and see if the strokes give us a better idea of what the shape could be. "},{"metadata":{"_cell_guid":"ff5ddced-d77e-473f-899d-82cf11ad2bd9","_uuid":"409468f1d5abd17b819482473a4f354a61f8d7ef","trusted":true},"cell_type":"code","source":"def get_Xy(in_df):\n    X = np.stack(in_df['drawing'], 0)\n    y = to_categorical(word_encoder.transform(in_df['word'].values))\n    return X, y\ntrain_X, train_y = get_Xy(train_df)\nvalid_X, valid_y = get_Xy(valid_df)\ntest_X, test_y = get_Xy(test_df)\nprint(train_X.shape)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"56240ed9-42b0-4f62-b3d1-f92017f04e30","_uuid":"5cc79204a0a1da048d1d58ba8dfdafd0af3ebcb8","trusted":true},"cell_type":"code","source":"fig, m_axs = plt.subplots(3,3, figsize = (16, 16))\nrand_idxs = np.random.choice(range(train_X.shape[0]), size = 9)\nfor c_id, c_ax in zip(rand_idxs, m_axs.flatten()):\n    test_arr = train_X[c_id]\n    test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n    lab_idx = np.cumsum(test_arr[:,2]-1)\n    for i in np.unique(lab_idx):\n        c_ax.plot(test_arr[lab_idx==i,0], \n                np.max(test_arr[:,1])-test_arr[lab_idx==i,1], '.-')\n    c_ax.axis('off')\n    c_ax.set_title(word_encoder.classes_[np.argmax(train_y[c_id])])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5e1d5bba-0fb4-432c-bd0b-ad69be0ef9ac","_uuid":"b4a087a17798c2ec8eb520bc916bcad38d4ebff2","collapsed":true},"cell_type":"markdown","source":"# WaveNet to Parse Strokes\nThe model suggeted from the WaveNet article by DeepMind (taken from site mentioned above) is\n\n![Suggested Model](https://storage.googleapis.com/deepmind-live-cms/documents/wavenet_conv_gif.gif)"},{"metadata":{"_cell_guid":"d65490e7-302e-4232-afe7-4e9499010e31","_uuid":"ba9d55554ba9e4177df5f0645ca1e0f5e4393ca3","trusted":true},"cell_type":"code","source":"from keras.layers import Conv1D, Input, Activation, AveragePooling1D, Add, Multiply, GlobalAveragePooling1D\nfrom keras.models import Model\ninput_shape = train_X.shape[1:]\noutput_shape = train_y.shape[1:]\ndef residual_block(x, i):\n    tanh_out = Conv1D(n_filters, \n                      kernel_size, \n                      dilation_rate = kernel_size**i, \n                      padding='causal', \n                      name='dilated_conv_%d_tanh' % (kernel_size ** i), \n                      activation='tanh'\n                      )(x)\n    sigm_out = Conv1D(n_filters, \n                      kernel_size, \n                      dilation_rate = kernel_size**i, \n                      padding='causal', \n                      name='dilated_conv_%d_sigm' % (kernel_size ** i), \n                      activation='sigmoid'\n                      )(x)\n    z = Multiply(name='gated_activation_%d' % (i))([tanh_out, sigm_out])\n    skip = Conv1D(n_filters, 1, name='skip_%d'%(i))(z)\n    res = Add(name='residual_block_%d' % (i))([skip, x])\n    return res, skip\nx = Input(shape=input_shape, name='original_input')\nskip_connections = []\nout = Conv1D(n_filters, 2, dilation_rate=1, padding='causal', name='dilated_conv_1')(x)\nfor i in range(1, dilation_depth + 1):\n    out, skip = residual_block(out,i)\n    skip_connections.append(skip)\nout = Add(name='skip_connections')(skip_connections)\nout = Activation('relu')(out)\n\nout = Conv1D(n_filters, pool_size_1, strides = 1, padding='same', name='conv_5ms', activation = 'relu')(out)\nout = AveragePooling1D(pool_size_1, padding='same', name='downsample_to_200Hz')(out)\n\nout = Conv1D(n_filters, pool_size_2, padding='same', activation='relu', name='conv_500ms')(out)\nout = Conv1D(output_shape[0], pool_size_2, padding='same', activation='relu', name='conv_500ms_target_shape')(out)\nout = AveragePooling1D(pool_size_2, padding='same',name = 'downsample_to_2Hz')(out)\nout = Conv1D(output_shape[0], (int) (input_shape[0] / (pool_size_1*pool_size_2)), padding='same', name='final_conv')(out)\nout = GlobalAveragePooling1D(name='final_pooling')(out)\nout = Activation(activation, name='final_activation')(out)\n\nstroke_read_model = Model(x, out)  \nstroke_read_model.compile(optimizer='adam', \n                          loss='categorical_crossentropy', \n                          metrics=['accuracy', top_3_accuracy])\nstroke_read_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2a549512-a9d9-4afd-b748-3e1c3296e193","_uuid":"5fda10b30c47a8cf6ea822ed0a4a1d7cd2c81195","trusted":true},"cell_type":"code","source":"weight_path=\"{}_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = False)\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=1e-6)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n                      patience=10) # probably needs to be more patient, but kaggle time is limited\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"825b3af8-9451-487b-a1e1-538f2f1489e1","_uuid":"ed2fc26af74aed1a93bbc253d61b72db5a81f5cc","trusted":true},"cell_type":"code","source":"from IPython.display import clear_output\nstroke_read_model.fit(train_X, train_y,\n                      validation_data = (valid_X, valid_y), \n                      batch_size = batch_size,\n                      epochs = 50,\n                      callbacks = callbacks_list)\nclear_output()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a7eb5b62-cf57-4380-8786-9ddc05be658f","_uuid":"858059b6c16d81f86460bef8fcf595e0d68d12b2","trusted":true},"cell_type":"code","source":"stroke_read_model.load_weights(weight_path)\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ee75c585-b134-4ea2-b8f3-219e24efd1f1","_uuid":"6b9cdf52d233de60108d72f540db978801b578c1","trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\ntest_cat = np.argmax(test_y, 1)\npred_y = stroke_read_model.predict(test_X, batch_size = 4096)\npred_cat = np.argmax(pred_y, 1)\nplt.matshow(confusion_matrix(test_cat, pred_cat))\nprint(classification_report(test_cat, pred_cat, \n                            target_names = [x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"db1d371b-4b2c-478f-b6df-76db58a24fbe","_uuid":"bd9a16adcb46e07d7949644e69bf3483f7dce571"},"cell_type":"markdown","source":"# Reading Point by Point"},{"metadata":{"_cell_guid":"bf7dff37-c634-4930-8dae-3dba8090c251","_uuid":"c43e87e7eccfb72dd35e64d872a7d658ffa535a3","scrolled":false,"trusted":true},"cell_type":"code","source":"points_to_use = [5, 15, 20, 30, 40, 50]\npoints_to_user = [108]\nsamples = 12\nword_dex = lambda x: word_encoder.classes_[x]\nrand_idxs = np.random.choice(range(test_X.shape[0]), size = samples)\nfig, m_axs = plt.subplots(len(rand_idxs), len(points_to_use), figsize = (24, samples/8*24))\nfor c_id, c_axs in zip(rand_idxs, m_axs):\n    res_idx = np.argmax(test_y[c_id])\n    goal_cat = word_encoder.classes_[res_idx]\n    \n    for pt_idx, (pts, c_ax) in enumerate(zip(points_to_use, c_axs)):\n        test_arr = test_X[c_id, :].copy()\n        test_arr[pts:] = 0 # short sequences make CudnnLSTM crash, ugh \n        stroke_pred = stroke_read_model.predict(np.expand_dims(test_arr,0))[0]\n        top_10_idx = np.argsort(-1*stroke_pred)[:10]\n        top_10_sum = np.sum(stroke_pred[top_10_idx])\n        \n        test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n        lab_idx = np.cumsum(test_arr[:,2]-1)\n        for i in np.unique(lab_idx):\n            c_ax.plot(test_arr[lab_idx==i,0], \n                    np.max(test_arr[:,1])-test_arr[lab_idx==i,1], # flip y\n                      '.-')\n        c_ax.axis('off')\n        if pt_idx == (len(points_to_use)-1):\n            c_ax.set_title('Answer: %s (%2.1f%%) \\nPredicted: %s (%2.1f%%)' % (goal_cat, 100*stroke_pred[res_idx]/top_10_sum, word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum))\n        else:\n            c_ax.set_title('%s (%2.1f%%), %s (%2.1f%%)\\nCorrect: (%2.1f%%)' % (word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum, \n                                                                 word_dex(top_10_idx[1]), 100*stroke_pred[top_10_idx[1]]/top_10_sum, \n                                                                 100*stroke_pred[res_idx]/top_10_sum))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"193d66e021e6a0b29975ce42067bf65b10d283d7"},"cell_type":"markdown","source":"# Submission\nWe can create a submission using the model"},{"metadata":{"_cell_guid":"436a4fce-3843-4c84-8eeb-0161fe3c4e04","_uuid":"4f3a40e23f2e917b68171822944491ab348e15b3","trusted":true},"cell_type":"code","source":"sub_df = pd.read_csv(test_path)\nsub_df['drawing'] = sub_df['drawing'].map(_stack_it)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f2fb3a5463b7c71330771e4d84cbfe6f5c618c4"},"cell_type":"code","source":"sub_vec = np.stack(sub_df['drawing'].values, 0)\nsub_pred = stroke_read_model.predict(sub_vec, verbose=True, batch_size=4096)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d785199df3d8171c867a7b8016699bda8399c1df"},"cell_type":"code","source":"top_3_pred = [word_encoder.classes_[np.argsort(-1*c_pred)[:3]] for c_pred in sub_pred]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9e022e1391e1fb7dfe4481b940d78be96c59a19e"},"cell_type":"code","source":"top_3_pred = [' '.join([col.replace(' ', '_') for col in row]) for row in top_3_pred]\ntop_3_pred[:3]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd6ee8bab00ca3686472f8670c9a5bd823084e6b"},"cell_type":"markdown","source":"## Show some predictions on the submission dataset"},{"metadata":{"trusted":true,"_uuid":"eabd3abcd81094a334dda6540a155b6d51580940"},"cell_type":"code","source":"fig, m_axs = plt.subplots(3,3, figsize = (16, 16))\nrand_idxs = np.random.choice(range(sub_vec.shape[0]), size = 9)\nfor c_id, c_ax in zip(rand_idxs, m_axs.flatten()):\n    test_arr = sub_vec[c_id]\n    test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n    lab_idx = np.cumsum(test_arr[:,2]-1)\n    for i in np.unique(lab_idx):\n        c_ax.plot(test_arr[lab_idx==i,0], \n                np.max(test_arr[:,1])-test_arr[lab_idx==i,1], '.-')\n    c_ax.axis('off')\n    c_ax.set_title(top_3_pred[c_id])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c0ca90c42b2c8568675ff5064bfe1acae6fcbac"},"cell_type":"code","source":"sub_df['word'] = top_3_pred\nsub_df[['key_id', 'word']].to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"338ece2c2655ecc1cbb27ad8685c72348046a72e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}