{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Mini-Project RVGIS by Oliver G. H.\n**1. Explore the dataset, e.g. using the kernel provided at https://www.kaggle.com/gaborfodor/how-to-draw-an-owl-baseline-lb-0-002**\n**a.** The dataset consists of many thousands of images, specifically 112163 images in both full and simplified formats (simplified meaning not every keystroke was logged)\n**b.** There are 340 classes.\n**c.** Found under In[9]\n**d.** Classifiers always have issues when data is heavily skewed - with two classes and 850 images in one and 900 in the other the issue is not too big, but with 100 classes where 50% of the data comes from one class, this is a real problem as the network will learn the features of one class more often than the others and be much more likely to guess that class."},{"metadata":{"_uuid":"f7f2a9516140a84124bf7bbf538ee4c30860b778"},"cell_type":"markdown","source":"## Setup\nImport the necessary libraries and a few helper functions."},{"metadata":{"trusted":true,"_uuid":"ce6d2aa7de1fa341144def7d3a5b1ffdea26bc91","_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport os\nimport ast\nimport datetime as dt\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = [16, 10]\nplt.rcParams['font.size'] = 14\nimport seaborn as sns\nimport cv2\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Input, Reshape, ZeroPadding2D\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, BatchNormalization\nfrom tensorflow.keras.metrics import categorical_accuracy, top_k_categorical_accuracy, categorical_crossentropy\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.applications import MobileNet\nfrom tensorflow.keras.applications.mobilenet import preprocess_input\nstart = dt.datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"978b1e827e598c53df3ef09838a6d85591d83052"},"cell_type":"code","source":"DP_DIR = '../input/shuffle-csvs/'\nINPUT_DIR = '../input/quickdraw-doodle-recognition/'\n\nBASE_SIZE = 256\nNCSVS = 100\nNCATS = 340\nnp.random.seed(seed=1987)\ntf.set_random_seed(seed=1987)\n\ndef f2cat(filename: str) -> str:\n    return filename.split('.')[0]\n\ndef list_all_categories():\n    files = os.listdir(os.path.join(INPUT_DIR, 'train_simplified'))\n    return sorted([f2cat(f) for f in files], key=str.lower)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b2fcd1a08ae1ae0619be38a113a244eb6515b63b"},"cell_type":"code","source":"def apk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    if len(predicted) > k:\n        predicted = predicted[:k]\n    score = 0.0\n    num_hits = 0.0\n    for i, p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i + 1.0)\n    if not actual:\n        return 0.0\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])\n\ndef preds2catids(predictions):\n    return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], columns=['a', 'b', 'c'])\n\ndef top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"264156422a95e4b350886d558d516ae8bd2e25c0"},"cell_type":"markdown","source":"## MobileNet\n\nMobileNets are based on a streamlined architecture that uses depthwise separable convolutions to build light weight deep neural networks.\n\n[MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://arxiv.org/pdf/1704.04861.pdf)"},{"metadata":{"trusted":true,"_uuid":"54e5f0c637195b6624e2f3e6db5e7f8990e14eb7"},"cell_type":"code","source":"STEPS = 800\nEPOCHS = 16\nsize = 64\nbatchsize = 680","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4ad7e187fa2e9ab1de939874c101fb733d8d0ac"},"cell_type":"markdown","source":"# 2. Our Own Model\nHere is where I create my own model, discarding the MobileNet model and trying something new.\n1. I started with a simple model consisting of 2 convolutional layers of size 64 and kernel size 3x3, a max pooling layer and .25 dropout\nAfter a single epoch of 800 steps, the model reached top 3 accuracy of 0.6269 and categorical accuracy of 0.4450 which is quite impressive for such a simple network.\n2. However, two 3x3 kernels do not cover a particularly large area of the images we feed the network. Therefore, I decided to extend the network depth and allow it to pick up on features of a larger scale. I tested the model with twice the depth, only the size of these two new convolutional layers were 128 and 256, repsectively. This of course increased the training time of the model a fair bit (nearly doubled it), but it also improved the results significantly to categorical accuracy 0.5661 and top 3 0.7421.\n3. I figured that additional depth was very positive for the network, so I changed the two last layers to 128 size and added an additional block of two convolutional of size 256, max pooling, and dropout. Additionally, I added a final Dropout of 0.5 just before the dense layer to force the network to keep training poor nodes. This seemed to improve the network a bit, but not as much as hoped or expected. The new model achieved a categorical accuracy of 0.6205 and a top-3 accuracy of 0.7919.\n4. I tested another model with an additional layer of convolution, but the accuracy dropped to less than 0.01 which indicates that the filters grew too encompassing an could not distinguish anymore. Instead, I decided to test the impact of strides on training time and accuracy. After adding stride 2 to the second conv layer of every block, the accuracy dropped a bit to categorical accuracy 0.5415 and top-3 accuracy 0.7410. The training time improved by to just under half, which is very significant.\n5. I tested the model with double the sizes of every convolutional layer now that training time was acceptable. This improved the accuracy to 0.5935 categorical and 0.7846 top 3. However, this increased the training to 3000s running only a single epoch (three times). Seeing as the session only allows for 21600 seconds, this was far too much. \n6. Instead, I tried to reduce the size again, so the convolutional layers followed this sizing: 32->64->Max+Drop->128->128->Max+Drop->256->512->Max+Drop. This was not a significant enough reduction in accuracy to outweigh the time improvement. In total, running the script took 1158s and yielded a  categorical accuracy of 0.5623 and top 3 accuracy of 0.7569. Even at the end, the training accuracy steadily increased, and there was a rather large disparity in training and validation accuracy. The last training iteration had a top 3 accuracy of 0.6367 which is more than .1 under the validation accuracy, indicating huge rooms for improvement still.\n7. With this new information, I decided to revisit 16 epochs as the original notebook had. I do this to test the capabilities of this model versus the Greyscale MobileNet on equal footing. Of course, an issue that can arise here is that the training time is simply still too long. If that is the case, there are options to bring down the training time, however I do not expect it to be necessary. Instead, I believe the model may have room to expand as it currently has roughly 1/3 the parameters of the MobileNet. The resulting network reached a 0.7052 categorical accuracy and 0.8523 top 3 accuracy. \n8. Since the model trained so quickly (roughly 17000s), I decided to expand on it a bit. The first layer was changed to size 64, and the stride layers were switched to the former in their block. In the last block, I removed strides alltogether and instead implemented an additional Max Pooling layer. 'same' padding on stride layers were discarded in favor of 'valid' padding to reduce the size of the image accordingly. In order for this to work I added a zero padding layer before these. This brought the training time down to roughly 15000s and increased the categorical accuracy to 0.7205 and top 3 accuracy 0.8730.\n9. I noticed that following the final Max Pooling layer, the size of the resulting image was (2,2). This means that the final set of parameters were significantly increased, and by reducing that I could increase the size of each convolutional layer. To do this, I simply reduced the kernel size of the last final two convolutional layers to (2,2) and doubled each layer size, resulting in a final layer size of 1024. This resulted in a model with just over 5 million parameters. This model ended up scoring 0.8755 on top 3 accuracy, still unable to reach 0.9. \n10. The model trained very quickly, roughly 400ms per step, but as it did not improve by doubling the size, I figured the major bottleneck here is still the depth. However, the output size is currently (2,2), so I need to keep this size relatively uniform. I decided to remove strides from the convolutional layer in the second block, copy it and double the layer sizes. Then, I doubled the first layer's size in the last block to 1024. So now the three first block consist of a zero padding layer, two convolutional layer of sizes 128, 256, and 512, and (2,2) max pooling. The second convolutional layer has the 'same' keyword as padding, meaning each block outputs a POT number (16, 8, 4) after the max pooling. The final block uses only valid outputs, and so the output size is still (2,2). However, this ran into the issue of not improving after an entire epoch due to the learning rate. I figured, after plenty of tests, that the actual information left after the two final layers was near nothing. I removed the final layer, and after increasing the learning rate back to 0.005, it started improving again.\n\nAt this point Kaggle started not to work for me on either PC. Commits would never finish, instead brining up an interrupt error, and running the code would inevitably fail to train the network or the network would suddenly return to \"random chance\" accuracy. Therefore, I hand in this assignment. The best accuracy I achieved will have to be using the network described in step 9. \n\n<small>Here's to hoping the text changes will commit and I won't have to rewrite</small>"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"0860ec35bee03f0c5cd21202dc7471c2d201cf5f","_kg_hide-output":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(ZeroPadding2D(input_shape=(size,size,1)))\nmodel.add(Conv2D(128, (3,3), strides = 2, padding='valid', activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Conv2D(128, (3,3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(ZeroPadding2D())\nmodel.add(Conv2D(256, (3,3), padding='valid', activation='relu'))\nmodel.add(Conv2D(256, (3,3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(ZeroPadding2D())\nmodel.add(Conv2D(512, (3,3), activation='relu'))\nmodel.add(Conv2D(512, (3,3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(1024, (2,2), activation='relu'))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dropout(0.5))\nmodel.add(Dense(NCATS, activation='softmax'))\n\nmodel.compile(optimizer=Adam(lr=0.005), loss='categorical_crossentropy', metrics=[categorical_crossentropy, categorical_accuracy, top_3_accuracy])\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab1834ea2757a53d602a3508efffcc34bc190dc7"},"cell_type":"markdown","source":"## Training with Image Generator"},{"metadata":{"trusted":true,"_uuid":"f6455bf9555b8381b6a4292098a64a0eb7ff54dc"},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, time_color=True):\n    img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)\n    for t, stroke in enumerate(raw_strokes):\n        for i in range(len(stroke[0]) - 1):\n            color = 255 - min(t, 10) * 13 if time_color else 255\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]),\n                         (stroke[0][i + 1], stroke[1][i + 1]), color, lw)\n    if size != BASE_SIZE:\n        return cv2.resize(img, (size, size))\n    else:\n        return img\n\ndef image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n    while True:\n        for k in np.random.permutation(ks):\n            filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n            for df in pd.read_csv(filename, chunksize=batchsize):\n                df['drawing'] = df['drawing'].apply(ast.literal_eval)\n                x = np.zeros((len(df), size, size, 1))\n                for i, raw_strokes in enumerate(df.drawing.values):\n                    x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw,\n                                             time_color=time_color)\n                x = preprocess_input(x).astype(np.float32)\n                y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n                yield x, y\n\ndef df_to_image_array_xd(df, size, lw=6, time_color=True):\n    df['drawing'] = df['drawing'].apply(ast.literal_eval)\n    x = np.zeros((len(df), size, size, 1))\n    for i, raw_strokes in enumerate(df.drawing.values):\n        x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw, time_color=time_color)\n    x = preprocess_input(x).astype(np.float32)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98ff512e1a1b5e86e86d9eef4127525bedf3b9e1"},"cell_type":"code","source":"valid_df = pd.read_csv(os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(NCSVS - 1)), nrows=34000)\nx_valid = df_to_image_array_xd(valid_df, size)\ny_valid = keras.utils.to_categorical(valid_df.y, num_classes=NCATS)\nprint(x_valid.shape, y_valid.shape)\nprint('Validation array memory {:.2f} GB'.format(x_valid.nbytes / 1024.**3 ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d80ad7f4d378ea7f30479221d604eeeed559cae4"},"cell_type":"code","source":"train_datagen = image_generator_xd(size=size, batchsize=batchsize, ks=range(NCSVS - 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ce5fb89fbb77777316d6fca7689b6636c0e6021"},"cell_type":"code","source":"x, y = next(train_datagen)\nn = 8\nfig, axs = plt.subplots(nrows=n, ncols=n, sharex=True, sharey=True, figsize=(12, 12))\nfor i in range(n**2):\n    ax = axs[i // n, i % n]\n    (-x[i]+1)/2\n    ax.imshow((-x[i, :, :, 0] + 1)/2, cmap=plt.cm.gray)\n    ax.axis('off')\nplt.tight_layout()\nfig.savefig('gs.png', dpi=300)\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da72d70fc1781e80427d45a80c07b3571dda0b36"},"cell_type":"code","source":"callbacks = [\n    ReduceLROnPlateau(monitor='val_categorical_accuracy', factor=0.5, patience=5,\n                      min_delta=0.005, mode='max', cooldown=3, verbose=1)\n]\nhists = []\nhist = model.fit_generator(\n    train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n    validation_data=(x_valid, y_valid),\n    callbacks = callbacks\n)\nhists.append(hist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ada344bf3454765298e7b7ed7861c82bca2d2084","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"hist = model.fit_generator(\n    train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n    validation_data=(x_valid, y_valid),\n    callbacks = callbacks\n)\nhists.append(hist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a3a0cfef0d984d5b02872617e0eb2ad8a791964","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"hist = model.fit_generator(\n    train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n    validation_data=(x_valid, y_valid),\n    callbacks = callbacks\n)\nhists.append(hist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"05767778d356bc63b7cded355159fd4082eee1a5"},"cell_type":"code","source":"hist_df = pd.concat([pd.DataFrame(hist.history) for hist in hists], sort=True)\nhist_df.index = np.arange(1, len(hist_df)+1)\nfig, axs = plt.subplots(nrows=2, sharex=True, figsize=(16, 10))\naxs[0].plot(hist_df.val_categorical_accuracy, lw=5, label='Validation Accuracy')\naxs[0].plot(hist_df.categorical_accuracy, lw=5, label='Training Accuracy')\naxs[0].set_ylabel('Accuracy')\naxs[0].set_xlabel('Epoch')\naxs[0].grid()\naxs[0].legend(loc=0)\naxs[1].plot(hist_df.val_categorical_crossentropy, lw=5, label='Validation MLogLoss')\naxs[1].plot(hist_df.categorical_crossentropy, lw=5, label='Training MLogLoss')\naxs[1].set_ylabel('MLogLoss')\naxs[1].set_xlabel('Epoch')\naxs[1].grid()\naxs[1].legend(loc=0)\nfig.savefig('hist.png', dpi=300)\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c1927f22d3c45cba0bdee7d6f4b6c858d82d614"},"cell_type":"code","source":"valid_predictions = model.predict(x_valid, batch_size=128, verbose=1)\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)\nprint('Map3: {:.3f}'.format(map3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5687db83dda178c9f29614c68a0527529fb8f200"},"cell_type":"markdown","source":"# 3. Break down the results of your classifier\nAfter nearly 50 epochs, every network (before the major issues started) was still learning at a reasonable rate, meaning there was room to improve for a long time. Also, training accuracy was consistently lower than validation accuracy by nealy .1, which may indicate that the measures I took to avoid overfitting was a bit too agressive, but since Kaggle stopped working properly for me I can't validate this. \nSome of the major issues I faced was not wanting to discard too much \"arbitrarily\", so I tried to avoid stepping too much through the data, but that is impossible to do without increasing parameter amounts, leading to a very long training time. A long training time means less iterations and thus less room to experiment and improve. \nAdditionally, many of the networks that followed would either refuse to improve due to too low learning rate or suddenly improve massively in one aspect and no others when the learning rate was increased. \n\n\nThere were plenty of steps I had noted down that I wanted to test - the final dropout seemed a bit too agressive following so much dropout and batch normalization, and I wanted to test what removing it entirely would do for the training accuracy. Additionally, I would like to test the same model without the \"decay\" in color values to see if a more classical approach to CNN where the image is fed as is would work better or at least as good.\n\nI also wanted to do some more in-depth analysis of the results, such as testing which categories were easier and harder to distinguish, and whether larger or smaller data had any influence on the network's ability to guess correctly. However, since Kaggle no longer trains my network correctly (even when using older, tested and proven models), this has become impossible."},{"metadata":{"_uuid":"be4577a9ba00611697eea8f241a42c504981e86f"},"cell_type":"markdown","source":"## Create Submission"},{"metadata":{"trusted":true,"_uuid":"a7d14348150baf753e90cf2719b9f31dd564f6a2"},"cell_type":"code","source":"test = pd.read_csv(os.path.join(INPUT_DIR, 'test_simplified.csv'))\ntest.head()\nx_test = df_to_image_array_xd(test, size)\nprint(test.shape, x_test.shape)\nprint('Test array memory {:.2f} GB'.format(x_test.nbytes / 1024.**3 ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"608b02f5c7909ae62becbe5c931b7264171296e8"},"cell_type":"code","source":"test_predictions = model.predict(x_test, batch_size=128, verbose=1)\n\ntop3 = preds2catids(test_predictions)\ntop3.head()\ntop3.shape\n\ncats = list_all_categories()\nid2cat = {k: cat.replace(' ', '_') for k, cat in enumerate(cats)}\ntop3cats = top3.replace(id2cat)\ntop3cats.head()\ntop3cats.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52e0f9c44f2a9a38fd1550ffb9c07fb7ea22b17d"},"cell_type":"code","source":"test['word'] = top3cats['a'] + ' ' + top3cats['b'] + ' ' + top3cats['c']\nsubmission = test[['key_id', 'word']]\nsubmission.to_csv('gs_mn_submission_{}.csv'.format(int(map3 * 10**4)), index=False)\nsubmission.head()\nsubmission.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b418f4c06c4e4453aa1b5ab16dde344eb8b735c5"},"cell_type":"code","source":"end = dt.datetime.now()\nprint('Latest run {}.\\nTotal time {}s'.format(end, (end - start).seconds))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}