{"cells":[{"metadata":{"_uuid":"614ceba177164e8ae691ccfe0fc64031de5ba9a9"},"cell_type":"markdown","source":"# Raw Stroke 1D-CNN\n\nFor the [Doodle Recognition](https://www.kaggle.com/c/quickdraw-doodle-recognition) challenge I tried to use a **sequence of raw doodle strokes as input** to a 1D-CNN to predict the category.<br>\nBesides a **randomized sequence generator** you can find a deep network to tweak on your own.\n\n+ [1. Libraries](#1)<br>\n+ [2. Files And Label-Mapping](#2)<br>\n+ [3. Image Generator](#3)<br>\n+ [4. 1D CNN Network](#4)<br>\n+ [5. Training](#5)<br>\n+ [6. Submission](#6)<br>\n+ [7. Conclusion](#7)<br>\n\n***\n## <a id=1>1. Libraries</a>\n\nThe main libraries for this notebook are **keras for the model** and a mixture of **numpy and pandas for the data storage and preprocessing.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# To do linear algebra\nimport numpy as np\n\n# To store data\nimport pandas as pd\n\n# To walk directories\nimport os\n\n# To create models\nfrom keras.models import Model\nfrom keras.layers import Input, Conv1D, Dense, Dropout, BatchNormalization, Flatten, MaxPool1D\nfrom keras.utils import to_categorical\n\n# To read strings\nimport json\n\n# To create a submission\nimport csv\n\n# To create plots\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e54827d71c1083dd8fb0aef84b138d2abcb79ecd"},"cell_type":"markdown","source":"***\n## <a id=2>2. Files And Label-Mapping</a>\n\nThe doodles depict different categories of things and **each category/label is stored in a single file.**<br>\nThis cell creates a **mapping of labels to unique integers** for later one-hot encoding."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Path to the data\npath = '../input/train_simplified/'\n\n# Get all data files\nfiles = [os.path.join(path, file) for i, file in enumerate(os.listdir(path))]\n\n# Get number of categories\nn_categories = len(files)\n\n# Get dictionary to map labels to integers\nword_mapping = {file.split('/')[-1][:-4]:i for i, file in enumerate(files)}\n\nprint('Number of different files/categories:\\t{}'.format(n_categories))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c78acf7cba16df0dc759e12ef4ce2fe99e8d525d"},"cell_type":"markdown","source":"***\n## <a id=3>3. Image Generator</a>\n\nThe image generator uses the first **100.000 images of each file for learning** and the next **10.000 images as validation** dataset.<br>\nSince the generator samples from the files and shuffles the final batches the epochs will consist of mostly distinct doodles but the same images will occur eventually.<br>\nEach **doodle consists of a sequence of x- and y-coordinates.** The final preprocessed batch is a 2D-array of all concatenated x- and y-sequences. "},{"metadata":{"trusted":true,"_uuid":"6be8e3ff7c43b28f9692c395c49d0c998fd9809b","_kg_hide-input":true},"cell_type":"code","source":"def imageGenerator(batchsize, validation=False):\n    # Never ending iterator\n    while True:\n        \n        # Variable to store the data\n        df = []\n        \n        # Iterate over all files\n        for file in files:\n            # Get random samples of the data\n            if validation:\n                # Use rows 100000:110000 as validation data\n                df.append(pd.read_csv(file, nrows=110000, usecols=[1, 5]).tail(10000).sample(1000))\n            else:\n                # Use rows :100000 as training data\n                df.append(pd.read_csv(file, nrows=100000, usecols=[1, 5]).sample(1000))\n                \n        # Combine DataFrames\n        df = pd.concat(df)\n        \n        # Use mapping on labels\n        df['word'] = df['word'].map(word_mapping)\n        \n        # Shuffle DataFrame\n        df = df.sample(frac=1).reset_index(drop=True)\n        \n        # Convert labels to vectors\n        y = to_categorical(df['word'].values, n_categories)\n        \n        # Variable to store the sequence of strokes\n        X = []\n        # Iterate over all images\n        for values in df['drawing'].values:\n            # Convert string to list\n            image = json.loads(values)\n\n            strokes = []\n            # Concatenate all strokes\n            for x_axis, y_axis in image:\n                strokes.extend(list(zip(x_axis, y_axis)))\n            strokes = np.array(strokes)\n\n            # Create empty array for sequence padding\n            pad = np.zeros((sequence_length, 2))\n            \n            # Pad/slice data to correct format\n            if sequence_length>strokes.shape[0]:\n                pad[:strokes.shape[0],:] = strokes\n            else:\n                pad = strokes[:sequence_length, :]\n\n            X.append(pad)\n        X = np.array(X)\n        \n        \n        i = 0\n        # Iterate over all batches in the loaded data\n        while True:\n            # Slice a batch of data to yield\n            if i+batchsize<=y.shape[0]:\n                y_yield = y[i:i+batchsize]\n                X_yield = X[i:i+batchsize]\n                i += batchsize\n                yield (X_yield, y_yield)\n            else:\n                break","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e030c3c2bab01d399628d65fe57590a84bdd76a"},"cell_type":"markdown","source":"***\n## <a id=4>4. 1D CNN Network</a>\n\nThe model gets a **sequence of x- and y-coordinates as input.** To find patterns in the sequence the **network starts with some convolution layers.** To combine the found features the network **closes with some dense layers.**<br>\nThere has not been extensive architecture evaluation, therefore improvements should be easily possible."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f9d4a13d5dfa278f7a7af26beb1cf4284b08a2a8"},"cell_type":"code","source":"def createNetwork(seq_len):\n    \n    # Function to add a convolution layer with batch normalization\n    def addConv(network, features, kernel):\n        network = BatchNormalization()(network)\n        return Conv1D(features, kernel, padding='same', activation='relu')(network)\n    \n    # Function to add a dense layer with batch normalization and dropout\n    def addDense(network, size):\n        network = BatchNormalization()(network)\n        network = Dropout(0.2)(network)\n        return Dense(size, activation='relu')(network)\n    \n    \n    # Input layer\n    input = Input(shape=(seq_len, 2))\n    network = input\n    \n    # Add 1D Convolution\n    for features in [16, 24, 32]:\n        network = addConv(network, features, 5)\n    network = MaxPool1D(pool_size=5)(network)\n    \n    # Add 1D Convolution\n    for features in [64, 96, 128]:\n        network = addConv(network, features, 5)\n    network = MaxPool1D(pool_size=5)(network)\n\n    # Add 1D Convolution\n    for features in [256, 384, 512]:\n        network = addConv(network, features, 5)\n    #network = MaxPool1D(pool_size=5)(network)\n\n    # Flatten\n    network = Flatten()(network)\n    \n    # Dense layer for combination\n    for size in [128, 128]:\n        network = addDense(network, size)\n    \n    # Output layer\n    output = Dense(len(files), activation='softmax')(network)\n\n\n    # Create and compile model\n    model = Model(inputs = input, outputs = output)\n    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])\n\n    # Display model\n    model.summary()\n    return model\n\n\n# Length of the x-y-sequences\nsequence_length = 80\n\n# Create a model\nmodel = createNetwork(sequence_length)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30ff78b8ca33d2df962ff957fe397b61ea94c5c3"},"cell_type":"markdown","source":"***\n## <a id=5>5. Training</a>\n\nThe model has been **trained for the maximal time** a kernel is allowed to compute. "},{"metadata":{"trusted":true,"_uuid":"63c113350ff12d6fdd62423984a8fa7bbfddda61","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Instantiate generators for training and validation data\ntrain_generator = imageGenerator(batchsize=1000)\nvalid_generator = imageGenerator(batchsize=1000, validation=True)\n\n# Train the model\nmodel.fit_generator(train_generator, steps_per_epoch=340, epochs=130, validation_data=valid_generator, validation_steps=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c971b657167550712718490cea5d5dd26923c541","_kg_hide-input":true},"cell_type":"code","source":"# Get training history\nhistory = model.history.history\n\nprint('Final Validation Accuracy: {:.4f}'.format(history['val_acc'][-1]))\n\n# Create subplots\nfig, axarr = plt.subplots(2, 1, figsize=(12,9))\n\n# Plot accuracy\naxarr[0].plot(history['acc'], label='Train')\naxarr[0].plot(history['val_acc'], label='Valid')\naxarr[0].set_title('Accuracy')\naxarr[0].set_xlabel('Epochs')\naxarr[0].set_ylabel('Accuracy')\naxarr[0].legend()\n\n# Plot loss\naxarr[1].plot(history['loss'], label='Train')\naxarr[1].plot(history['val_loss'], label='Valid')\naxarr[1].set_title('Loss')\naxarr[1].set_xlabel('Epochs')\naxarr[1].set_ylabel('Loss')\naxarr[1].legend()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2762e1cc96ff7fd47d75153427cbcc62c7b7595a"},"cell_type":"markdown","source":"***\n## <a id=6>6. Submission</a>\n\nThe submission contains **roughly 100.000 images.**"},{"metadata":{"trusted":true,"_uuid":"4d9f9076e90fb06633e43bdd212582a54997d8be","_kg_hide-input":true},"cell_type":"code","source":"# Load submission data\nsubmission_data = pd.read_csv('../input/test_simplified.csv')\n\n\n# Preprocess the submission data\nX = []\n# Iterate over all images\nfor values in submission_data['drawing'].values:\n    # Convert string to list\n    image = json.loads(values)\n    \n    strokes = []\n    # Concatenate all strokes\n    for x_axis, y_axis in image:\n        strokes.extend(list(zip(x_axis, y_axis)))\n    strokes = np.array(strokes)\n\n    # Create empty array for padding\n    pad = np.zeros((sequence_length, 2))\n    # Pad/slice data to correct format\n    if sequence_length>strokes.shape[0]:\n        pad[:strokes.shape[0],:] = strokes\n    else:\n        pad = strokes[:sequence_length, :]\n\n    X.append(pad)\nX = np.array(X)\n\n\n# Predict categories\nprediction = model.predict(X)\n\n# Slice most probable three categories\nbest_prediction = np.flip(np.argsort(prediction, axis=1), axis=1)[:, :3]\n\n# Create reverse mapping dictionary\nreverse_word_mapping = {word:key for key, word in word_mapping.items()}\n\n\nsubmission = []\n# Iterate over each image\nfor key_id, label_ids in zip(submission_data['key_id'].values, best_prediction):\n    # reverse map and combine the most probable three categories for each image\n    labels = ' '.join([reverse_word_mapping[label_id].replace(' ', '_') for label_id in label_ids])\n    submission.append([key_id, labels])\n\n# Create and same submission\npd.DataFrame(submission, columns=['key_id', 'word']).to_csv('Submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8ae081f862cd23b6f596842d46d15e5c3aeb122"},"cell_type":"markdown","source":"***\n## <a id=7>7. Conclusion</a>\n\nUsing the raw sequences of x- and y-stroke coordinates as input yields **good but not state of the art results.**<br>\nUnfortunately the **image generator is slow** and therefore the training process is slowed down. Preprocessing and storing the doodles in a better format would accelerate the generator but can not be done in the kernel.<br>\nTo **increase the number of doodles** to learn from **varying the position of the stroke-points slightly** should improve the prediction. A more robust network should be the result of this artificial inflation of the dataset.<br>\n\n**I think a combination of this stroke approach and a classical 2D image CNN can be interesting.**<br>\n\nHave a good day!\n***\n***\n\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}