{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Quickdraw MLP using dyadic signature features\nIn this notebook, we investigate the effectiveness of a simple MLP neural network using dyadic signature features on the quickdraw dataset.\n\n## Approach\n1. We use all 340 categories of the `train_simplified` dataset and take 200 samples of each category, 90% for training and 10% for testing\n2. For each drawing, we compute the stroke embedding (pen-on pen-off) and then compute the dyadic signature features\n3. Use a simple 3 layer MLP network to classify"},{"metadata":{},"cell_type":"markdown","source":"## Installation"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -r ../input/quickdraw-requirements/requirements.txt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom scipy.interpolate import interp1d\nfrom iisignature import sig, logsig, prepare\n\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler, LabelBinarizer\nfrom sklearn.base import BaseEstimator, TransformerMixin, clone\nfrom sklearn.model_selection import train_test_split, cross_val_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definitions and implementation\n\nWe use `sklearn`'s pipeline functionality to do preprocessing. "},{"metadata":{"trusted":true},"cell_type":"code","source":"class SigFeatures(BaseEstimator, TransformerMixin):\n    def __init__(self, level=3):\n        self.level = level\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        return np.array([sig(x, self.level) for x in X])\n\n\nclass DyadicSigFeatures(BaseEstimator, TransformerMixin):\n    def __init__(self, sig_level=3, d_level=3):\n        self.sig_level = sig_level\n        self.d_level = d_level\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform_instance(self, X):\n        T = len(X)-1\n        current_times = np.arange(T+1)\n        X_fct = interp1d(current_times, X, axis=0)\n        features = []\n        for n in range(self.d_level+1):\n            N = 2**n\n            for i in range(N):\n                a = i*T/N\n                b = (i+1)*T/N\n                times = np.concatenate(([a], current_times[int(np.ceil(a)):int(np.ceil(b))], [b]))\n                path = X_fct(times)\n                features.append(sig(path, self.sig_level))\n        return np.concatenate(features)\n\n    def transform(self, X):\n        return [self.transform_instance(x) for x in X]\n    \n\nclass PenOnOff(BaseEstimator, TransformerMixin):\n    \"\"\"3D embedding as specified in http://discovery.ucl.ac.uk/10066168/1/arabic_handwriting_asar2018.pdf\"\"\"\n    def transform(self, X):\n        return [self.transform_instance(x) for x in X]\n    \n    def fit(self, X, y=None):\n        return self\n    \n    def transform_instance(self, data):\n        X = []\n        for index, stroke in enumerate(data):\n            embedded = np.transpose(stroke + [[2*index]*len(stroke[0])]).tolist()\n            if index >= 1:\n                X += [[stroke[0][0], stroke[1][0], 2*index-1]]\n            X += embedded\n            if index < len(data)-1:\n                X += [[stroke[0][-1], stroke[1][-1], 2*index+1]]\n        return X\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data loading\n\nWe load data from the CSV files."},{"metadata":{"trusted":true},"cell_type":"code","source":"from ast import literal_eval\n\ndef load_data(path, nrows=100):\n    data = pd.read_csv(path, index_col='key_id', nrows=nrows)\n    data['word'] = data['word'].replace(' ', '_', regex=True)\n    data['drawing'] = data['drawing'].apply(literal_eval)\n    return data\n\n\ndef load_multiple(filenames, size=400, folder='../input/quickdraw-doodle-recognition/train_simplified/'):\n    return pd.concat([load_data(folder+fname, nrows=size)\n                      for fname in filenames])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We load the first all 340 categories of the `train_simplified` dataset, and 200 samples of each category."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ncategories = !ls ../input/quickdraw-doodle-recognition/train_simplified/\ncategories = categories[0:340]\ndf = load_multiple(categories, size=200)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot a few of these drawings."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_drawing(X):\n    \"\"\"X is a collection of strokes\"\"\"\n    for x,y in X:\n        plt.plot(x, y, marker='.')\n    plt.gca().invert_yaxis()\n    plt.axis('equal')\n\nplt.figure(0)\nplot_drawing(df.drawing.values[12002])\nplt.figure(1)\nplot_drawing(df.drawing.values[15010])\nplt.figure(2)\nplot_drawing(df.drawing.values[42446])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features preprocessing\n\nWe set the dyadic level to 4 and signature truncation level to 3. See [this paper](https://ora.ox.ac.uk/objects/uuid:dd1ec888-c558-4385-8f48-4efcb867b682/download_file?file_format=pdf&safe_filename=Lyons%2Bet%2Bal%252C%2BRotation-free%2Bonline%2Bhandwritten%2Bcharacter%2Brecognition%2Busing%2Bdyadic%2Bpath%2Bsignature%2Bfeatures%252C%2Bhanging%2Bnormal.pdf&type_of_work=Conference+item) for details. These correspond to `n` and `m` in the paper respectively."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nd_level = 4 # dyadic level\nsig_level = 3 # signature truncation level\ndsigmodel = Pipeline([\n    ('penonoff',   PenOnOff()),\n    ('dsignature', DyadicSigFeatures(sig_level=sig_level, d_level=d_level)),\n    ('scale',      StandardScaler()),\n])\n\nX = dsigmodel.fit_transform(df.drawing.values)\ny = LabelBinarizer().fit_transform(df.word.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll use 90% of the data for training and 10% for testing."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Neural network setup"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import Dense, Dropout\nfrom keras.models import Sequential","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_features = X_train.shape[1]\nnum_classes = len(categories)\n\nmodel = Sequential()\nmodel.add(Dense(units=2048, activation='relu', input_shape=(num_features,)))\nmodel.add(Dense(units=2048, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(units=2048, activation='relu'))\nmodel.add(Dropout(0.5))\n\nmodel.add(Dense(units=num_classes, activation='softmax'))\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will use the same SGD optimizer as in the paper on [dyadic signatures](https://ora.ox.ac.uk/objects/uuid:dd1ec888-c558-4385-8f48-4efcb867b682/download_file?file_format=pdf&safe_filename=Lyons%2Bet%2Bal%252C%2BRotation-free%2Bonline%2Bhandwritten%2Bcharacter%2Brecognition%2Busing%2Bdyadic%2Bpath%2Bsignature%2Bfeatures%252C%2Bhanging%2Bnormal.pdf&type_of_work=Conference+item). In addition to the accuracy metric, we use the top 3 categorial accuracy (used by this Kaggle competition)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.optimizers import SGD\n\nopt = SGD(lr=0.02, decay=5e-4, momentum=0.9)\ndef top_3(y_true, y_pred): \n    return keras.metrics.top_k_categorical_accuracy(y_true, y_pred, k=3)\n\nmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy', top_3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training & Performance\n\nWe train for 20 epochs using batch sizes of 100."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(X_train, y_train, batch_size=100, epochs=20, validation_split=.1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['acc']) # blue\nplt.plot(history.history['val_acc']) # orange\nplt.plot(history.history['val_top_3']) # green","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our final score on the testing set is (loss, accuracy, top 3 accuracy):"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(X_test, y_test)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}