{"cells":[{"metadata":{"_uuid":"ce4b601f35a563fbdf4798f6a015228a78bb0438"},"cell_type":"markdown","source":"# Quick, Draw! Doodle Recognition Challenge\n\nFirstly we import the nedded packages and libraries to run the python script.\nWe then show the directories in which the data we need are in."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport datetime as dt\n%matplotlib inline\nimport matplotlib.pyplot as plt \nimport seaborn as sns\nimport ast\nfrom datetime import date, timedelta\n\n\n\n\nimport pickle # Read/Write with Serialization\nimport requests # Makes HTTP requests\nfrom io import BytesIO # Use When expecting bytes-like objects\nstart = dt.datetime.now()\n\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9bbba77ec70dc4cb77421e105e2c4b2691b7fbad"},"cell_type":"markdown","source":"# Classes and images\nThe number of classes are easily found, looking at the amount of files in the *train_simplified* folder.\nAfterwards the amount of images in each class is found in a *for loop* by reading each csv file. The numbers are also used for counting the total amount of images."},{"metadata":{"trusted":true,"_uuid":"e0e7808b182727bf7d979358bc45dcd56e303966","scrolled":true},"cell_type":"code","source":"TRAIN_FILES_PATH = '../input/train_simplified/'\n\n\ntrainingFileNameArr = os.listdir(TRAIN_FILES_PATH)\ntotalClassesCount = len(trainingFileNameArr)\ntotalImagesCount = 0\n\nprint('classes:', totalClassesCount)\n\n#for trainingFileName in trainingFileNameArr:\n#  print(trainingFileName)\n#  dataset = pd.read_csv(TRAIN_FILES_PATH + trainingFileName, header=0).values\n#  print(dataset.shape[0])\n#  totalImagesCount += dataset.shape[0]\n\nprint(trainingFileNameArr)\nprint('images: 49707579')\nprint('Average in each class: 146198.76')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6481d339d17a67682c00c3448208964156cff34a"},"cell_type":"markdown","source":"There are 340 classes and 49,707,579 images in total. The amount of images in each class is printed with the class name in the ouput above averaging at 146198.76 per class.\n# Printing images from 8 classes.\nThe first 8 classes has been selected for printing."},{"metadata":{"trusted":true,"_uuid":"379ede15b610cf0040cbed408a1d17a3e7a31bc0","scrolled":false},"cell_type":"code","source":"%matplotlib inline\nfor i in range(8):\n    print(trainingFileNameArr[i]);\n    path = os.path.join(TRAIN_FILES_PATH,trainingFileNameArr[i]);\n    item = pd.read_csv(path);\n    item['timestamp'] = pd.to_datetime(item.timestamp);\n    item = item.sort_values(by='timestamp',ascending=False)[-20:]\n    item['drawing'] = item['drawing'].apply(ast.literal_eval);\n    \n    item.head()\n    \n    n = 10\n    fig, axs = plt.subplots(2, 10, figsize=(16, 3));\n    for i, drawing in enumerate(item.drawing):\n        ax = axs[i // n, i % n];\n        for x, y in drawing:\n            ax.plot(x, -np.array(y), lw=3);\n        ax.axis('off');\n        \n    fig.savefig('item.jpg', dpi=200)\n    fig = plt.show();\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d63b86d5f5d3d5ae2b6a02b29e45f581ac8e237"},"cell_type":"markdown","source":"# Preparing data for network\nWe first reset the kernel inputs to make sure we don't carry over any variables. This also measn imports must be done again."},{"metadata":{"trusted":true,"_uuid":"e960ba65af5807c834facc01fa792a8ebbb5fb47"},"cell_type":"code","source":"%reset -f ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78ca6d65178b56cf00da1b2cf166d2e358277e7c"},"cell_type":"markdown","source":"## Parameter setting\nThese have been changed a few times. Especially the amount of samples has been increased to use more data, and hopefully increase accuracy. This also requires more VRAM, meaning there is a line in how much data is usable. More data also means longer training time. <br>\nTo increase the amount of data usable, the batch size is decreased, taking up less VRAM."},{"metadata":{"trusted":true,"_uuid":"77f0759c4529126caf48ecdef896362addb4c06e"},"cell_type":"code","source":"batch_size = 2048\nSTROKE_COUNT = 196\nTRAIN_SAMPLES = 2000\nVALID_SAMPLES = 200\nTEST_SAMPLES = 150","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c05363e11aad13ce0c4749f33c6f5208c2fa4817"},"cell_type":"markdown","source":"## The different sizes of samples\n\nTRAIN_SAMPLES = 750,\nVALID_SAMPLES = 75,\nTEST_SAMPLES = 50\n<br>\n<br>\nTRAIN_SAMPLES = 1000,\nVALID_SAMPLES = 100,\nTEST_SAMPLES = 75\nwith 0.5 dropout in last 2\n<br>\n<br>\nTRAIN_SAMPLES = 1200,\nVALID_SAMPLES = 120,\nTEST_SAMPLES = 100\nBest precision\n<br>\n<br>\nTRAIN_SAMPLES = 2000,\nVALID_SAMPLES = 200,\nTEST_SAMPLES = 150,\nBatch size of 2048 necessary instead of 4096\n<br>\n"},{"metadata":{"trusted":true,"_uuid":"5d14b6561882d2c28126adf2a66b024e7f5e1b46","scrolled":false},"cell_type":"code","source":"%matplotlib inline\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom keras.utils.np_utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences\nfrom sklearn.preprocessing import LabelEncoder\nimport pandas as pd\nfrom keras.metrics import top_k_categorical_accuracy\ndef top_3_accuracy(x,y): return top_k_categorical_accuracy(x,y, 3)\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom glob import glob\nimport gc\ngc.enable()\ndef get_available_gpus():\n    from tensorflow.python.client import device_lib\n    local_device_protos = device_lib.list_local_devices()\n    return [x.name for x in local_device_protos if x.device_type == 'GPU']\nbase_dir = os.path.join('..', 'input')\ntest_path = os.path.join(base_dir, 'test_simplified.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c276bcb7273ff2aee3da57f146041e3fb56ffe7"},"cell_type":"markdown","source":"## Creating training data\nHere we're creating the training by drawing strokes, training the network to learn the drawings for each stroke making it try to predict from early on, just like the real program does."},{"metadata":{"trusted":true,"_uuid":"a18b26c47a70c0e09ea1b4bd303483cfa85a35c3"},"cell_type":"code","source":"from ast import literal_eval\nALL_TRAIN_PATHS = glob(os.path.join(base_dir, 'train_simplified', '*.csv'))\nCOL_NAMES = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\n\ndef _stack_it(raw_strokes):\n    \"\"\"preprocess the string and make \n    a standard Nx3 stroke vector\"\"\"\n    stroke_vec = literal_eval(raw_strokes) # string->list\n    # unwrap the list\n    in_strokes = [(xi,yi,i)  \n     for i,(x,y) in enumerate(stroke_vec) \n     for xi,yi in zip(x,y)]\n    c_strokes = np.stack(in_strokes)\n    # replace stroke id with 1 for continue, 2 for new\n    c_strokes[:,2] = [1]+np.diff(c_strokes[:,2]).tolist()\n    c_strokes[:,2] += 1 # since 0 is no stroke\n    # pad the strokes with zeros\n    return pad_sequences(c_strokes.swapaxes(0, 1), \n                         maxlen=STROKE_COUNT, \n                         padding='post').swapaxes(0, 1)\ndef read_batch(samples=5, \n               start_row=0,\n               max_rows = 1000):\n    \"\"\"\n    load and process the csv files\n    this function is horribly inefficient but simple\n    \"\"\"\n    out_df_list = []\n    for c_path in ALL_TRAIN_PATHS:\n        c_df = pd.read_csv(c_path, nrows=max_rows, skiprows=start_row)\n        c_df.columns=COL_NAMES\n        out_df_list += [c_df.sample(samples)[['drawing', 'word']]]\n    full_df = pd.concat(out_df_list)\n    full_df['drawing'] = full_df['drawing'].\\\n        map(_stack_it)\n    \n    return full_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c15c4b78b867e9e30fac642c5ac4781156cc8b6"},"cell_type":"markdown","source":"## Setup of the different dictionaries and making labels for the data."},{"metadata":{"trusted":true,"_uuid":"76b9f2ef1344600cad1eec101834d71dc8a56c66","scrolled":true},"cell_type":"code","source":"train_args = dict(samples=TRAIN_SAMPLES, \n                  start_row=0, \n                  max_rows=int(TRAIN_SAMPLES*1.5))\nvalid_args = dict(samples=VALID_SAMPLES, \n                  start_row=train_args['max_rows']+1, \n                  max_rows=VALID_SAMPLES+25)\ntest_args = dict(samples=TEST_SAMPLES, \n                 start_row=valid_args['max_rows']+train_args['max_rows']+1, \n                 max_rows=TEST_SAMPLES+25)\ntrain_df = read_batch(**train_args)\nvalid_df = read_batch(**valid_args)\ntest_df = read_batch(**test_args)\nword_encoder = LabelEncoder()\nword_encoder.fit(train_df['word'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d27a8926a8a4fd3ebff5e7ce51e8c1d87df58321"},"cell_type":"markdown","source":"## Splitting the data, before feeding it to the network, for training, validation, and testing."},{"metadata":{"trusted":true,"_uuid":"516cc96c5ebf5eea1f65f4b15b25a0b1e2f3e8d5","scrolled":true},"cell_type":"code","source":"def get_Xy(in_df):\n    X = np.stack(in_df['drawing'], 0)\n    y = to_categorical(word_encoder.transform(in_df['word'].values))\n    return X, y\ntrain_X, train_y = get_Xy(train_df)\nvalid_X, valid_y = get_Xy(valid_df)\ntest_X, test_y = get_Xy(test_df)\nprint(train_X.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0000a4aa3ffedbcba23f892d29dbeddd13ecd545"},"cell_type":"markdown","source":"# Network design\nThe network layout is chosen, as it is a rather small network  it will train faster.<br>\nThe network is a stroke based Long Short Term Memory network. By using the 1D convolutions the model takes the stroke data and 'preprocesses' it.<br>\nAfter the convolutions two LSTM layers are added followed by two dense layers to classify the data. With the LSTM we're reading the drawing 'stroke for stroke'."},{"metadata":{"trusted":true,"_uuid":"912891bf34fe8a9a4cc1eddf1878e0cabd51d95c","scrolled":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import BatchNormalization, Conv1D, LSTM, Dense, Dropout\nif len(get_available_gpus())>0:\n    # https://twitter.com/fchollet/status/918170264608817152?lang=en\n    from keras.layers import CuDNNLSTM as LSTM # this one is about 3x faster on GPU instances\nstroke_read_model = Sequential()\nstroke_read_model.add(BatchNormalization(input_shape = (None,)+train_X.shape[2:]))\n# filter count and length are taken from the script https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py\nstroke_read_model.add(Conv1D(48, (5,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Conv1D(64, (5,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Conv1D(96, (3,)))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(LSTM(128, return_sequences = True))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(LSTM(128, return_sequences = False))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Dense(512))\nstroke_read_model.add(Dropout(0.3))\nstroke_read_model.add(Dense(len(word_encoder.classes_), activation = 'softmax'))\nstroke_read_model.compile(optimizer = 'adam', \n                          loss = 'categorical_crossentropy', \n                          metrics = ['categorical_accuracy', top_3_accuracy])\nstroke_read_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"deac2b32c1219dade957cdd053841980cd97cda6"},"cell_type":"markdown","source":"# Checkpoints and Callbacks\nSetting checkpoints and making callbacks.<br>\nThis is have some control over the network while training without stopping it to tweak.<br>\nBy calling the ReduceLROnPlateau, we'll reduce the learning rate if no progression is made in 10 epochs.<br>\nCalling EarlyStopping does as the name say, stop early if no progression is made. This is to prevent the network 'running wild' without having any progress at all.\n"},{"metadata":{"trusted":true,"_uuid":"b871282fc152b5ba634224dd3aa6de5816acdaf7"},"cell_type":"code","source":"weight_path=\"{}_weights.best.hdf5\".format('stroke_lstm_model')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, \n                                   verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n                      patience=5) # probably needs to be more patient, but kaggle time is limited\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd779b5ea6814ea7652426cf9ae9eb81f1bd5a8b"},"cell_type":"code","source":"from IPython.display import clear_output\nstroke_read_model.fit(train_X, train_y,\n                      validation_data = (valid_X, valid_y), \n                      batch_size = batch_size,\n                      epochs = 50,\n                      callbacks = callbacks_list)\nclear_output()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"db3a521ff23184e2d8244f29cff6747f26d8d5d5"},"cell_type":"code","source":"stroke_read_model.load_weights(weight_path)\nlstm_results = stroke_read_model.evaluate(test_X, test_y, batch_size = 4096)\nprint('Accuracy: %2.1f%%, Top 3 Accuracy %2.1f%%' % (100*lstm_results[1], 100*lstm_results[2]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0926fcc053ad236110c3d720c16aea6687c7af0"},"cell_type":"markdown","source":"# Results\n### Variyng results depending on the settings in the network and data size\nwith 0.5 dropout after the two last layers we reach a precision of 49.8% and 70.6% <br>\nwith 0.3 dropout after the two last layers we reach a precision of 53.8% and 74.6%<br>\nwith 0.3 dropout after the two last layers: Accuracy: 59.2%, Top 3 Accuracy 78.6% larger amounts of data\n#### with 0.3 dropout after the two last layers: Accuracy: 65.3%, Top 3 Accuracy 83.2%, highest accuracy with largest amount of data\nThe precision can differ a little due to a regular run through and the commit made to build the notebook."},{"metadata":{"_uuid":"8a51025de4c7374c5ab2867e2da0aadc5ba56039"},"cell_type":"markdown","source":"# Overview\nWe the create an overview of the precision in every class"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"b9f7a63623c46ff00593fbb11d6cdd7ef28fc9e5"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\ntest_cat = np.argmax(test_y, 1)\npred_y = stroke_read_model.predict(test_X, batch_size = 4096)\npred_cat = np.argmax(pred_y, 1)\nplt.matshow(confusion_matrix(test_cat, pred_cat))\nprint(classification_report(test_cat, pred_cat, \n                            target_names = [x for x in word_encoder.classes_]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76e42de041dd13b7d65a0422b1d5dcb8c241d909"},"cell_type":"markdown","source":"Here we see, that the accuracy differs a lot from class to class. Where some classes has an accuracy above 80% others are below 40%\nIn the Confusion matrix we some scatters here an there, but in general classification is done correctly."},{"metadata":{"_uuid":"7d2b47b61fd6fe66b54388057219b25d61b3fdad"},"cell_type":"markdown","source":"# Submission and top 3 predictions\nWe prepare for submission of the results and show three top3 results"},{"metadata":{"trusted":true,"_uuid":"36f328c4cd6680b55528b35b9be4ae9cb857ed27"},"cell_type":"code","source":"sub_df = pd.read_csv(test_path)\nsub_df['drawing'] = sub_df['drawing'].map(_stack_it)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"0280f22d97473986102e49080ac901514fd6b2a6"},"cell_type":"code","source":"sub_vec = np.stack(sub_df['drawing'].values, 0)\nsub_pred = stroke_read_model.predict(sub_vec, verbose=True, batch_size=2048)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0965e81c04120f51a022974c34db8797e0ae1f9e"},"cell_type":"code","source":"top_3_pred = [word_encoder.classes_[np.argsort(-1*c_pred)[:3]] for c_pred in sub_pred]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"b97250fdc5ba498f708909e0cb7268451bd18e44"},"cell_type":"code","source":"top_3_pred = [' '.join([col.replace(' ', '_') for col in row]) for row in top_3_pred]\ntop_3_pred[:3]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"391a3d6e33d00cb6fda4b6d592354865900c0d68"},"cell_type":"markdown","source":"# Prediction\nWe can make some predictions on the test data using the weights from the trained network."},{"metadata":{"trusted":true,"_uuid":"20d0d73665f1fac753d9da4b32124e1fa5ec6d10"},"cell_type":"code","source":"points_to_use = [5, 15, 20, 30, 40, 50]\npoints_to_user = [108]\nsamples = 12\nword_dex = lambda x: word_encoder.classes_[x]\nrand_idxs = np.random.choice(range(test_X.shape[0]), size = samples)\nfig, m_axs = plt.subplots(len(rand_idxs), len(points_to_use), figsize = (24, samples/8*24))\nfor c_id, c_axs in zip(rand_idxs, m_axs):\n    res_idx = np.argmax(test_y[c_id])\n    goal_cat = word_encoder.classes_[res_idx]\n    \n    for pt_idx, (pts, c_ax) in enumerate(zip(points_to_use, c_axs)):\n        test_arr = test_X[c_id, :].copy()\n        test_arr[pts:] = 0 # short sequences make CudnnLSTM crash, ugh \n        stroke_pred = stroke_read_model.predict(np.expand_dims(test_arr,0))[0]\n        top_10_idx = np.argsort(-1*stroke_pred)[:10]\n        top_10_sum = np.sum(stroke_pred[top_10_idx])\n        \n        test_arr = test_arr[test_arr[:,2]>0, :] # only keep valid points\n        lab_idx = np.cumsum(test_arr[:,2]-1)\n        for i in np.unique(lab_idx):\n            c_ax.plot(test_arr[lab_idx==i,0], \n                    np.max(test_arr[:,1])-test_arr[lab_idx==i,1], # flip y\n                      '.-')\n        c_ax.axis('off')\n        if pt_idx == (len(points_to_use)-1):\n            c_ax.set_title('Answer: %s (%2.1f%%) \\nPredicted: %s (%2.1f%%)' % (goal_cat, 100*stroke_pred[res_idx]/top_10_sum, word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum))\n        else:\n            c_ax.set_title('%s (%2.1f%%), %s (%2.1f%%)\\nCorrect: (%2.1f%%)' % (word_dex(top_10_idx[0]), 100*stroke_pred[top_10_idx[0]]/top_10_sum, \n                                                                 word_dex(top_10_idx[1]), 100*stroke_pred[top_10_idx[1]]/top_10_sum, \n                                                                 100*stroke_pred[res_idx]/top_10_sum))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"577028e03abd55b784f49a951e6a2f07de8e0cab"},"cell_type":"markdown","source":"Here we see, that predictions made differ very much in precision, proving that the network made is not very precise.\nAs the sample size for the network is rather low, this can lead to bad learning, as deep learning needs a large database to learn properly."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}