{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport numpy as np \nimport pandas as pd \nimport cv2\nimport ast\nfrom matplotlib import pyplot as plt\nimport keras\n\n\nimport keras\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras.layers import Dense, Dropout, Flatten, Activation\nfrom keras.metrics import categorical_accuracy, top_k_categorical_accuracy, categorical_crossentropy\nfrom keras.models import Sequential\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom keras.optimizers import Adam\nfrom keras.applications import MobileNet\nfrom keras.applications.mobilenet import preprocess_input\n\nimport os\nprint(os.listdir(\"../input/quickdraw-doodle-recognition\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_simplified_path = '../input/quickdraw-doodle-recognition/train_simplified/'\ntest_simplified_path = '../input/quickdraw-doodle-recognition/test_simplified.csv'\n\nBATCHSIZE = 128\nSIZE = 96\nNCLASSES = 340\nVALIDSAMPLES = 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The apk and mapk metrics are sourced from https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py and list of strings so they can't be used at the training phase without modification. Keras has a top k categorical accuracy metric that will be be used during the training and the provided metrics will be used for later to confirm the models perdormance."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def apk(actual, predicted, k=10):\n    \"\"\"\n    Computes the average precision at k.\n    This function computes the average prescision at k between two lists of\n    items.\n    Parameters\n    ----------\n    actual : list\n             A list of elements that are to be predicted (order doesn't matter)\n    predicted : list\n                A list of predicted elements (order does matter)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The average precision at k over the input lists\n    \"\"\"\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=10):\n    \"\"\"\n    Computes the mean average precision at k.\n    This function computes the mean average prescision at k between two lists\n    of lists of items.\n    Parameters\n    ----------\n    actual : list\n             A list of lists of elements that are to be predicted \n             (order doesn't matter in the lists)\n    predicted : list\n                A list of lists of predicted elements\n                (order matters in the lists)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The mean average precision at k over the input lists\n    \"\"\"\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load and display some of the data\nAs precised in the discription of the challenge, labels with multiple words need to be transformed into a single str by repalcing spaces with underscore. Also the drawing column has an str type and need to be transformed to a list."},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv(train_simplified_path + 'roller coaster.csv', index_col='key_id', nrows=10)\ndata['drawing'] = data['drawing'].apply(ast.literal_eval)\ndata['word'] = data['word'].apply(lambda x: x.replace(' ', '_'))\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To display the images, we use the function propsed by this [kernel](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892).\nThis method takes into acount the temperal information which is incoded in the color and defined by the parameter: ```time_color``` and also resizes the image."},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, time_color=True, base_size=256):\n    img = np.zeros((base_size, base_size), np.uint8)\n    for t, stroke in enumerate(raw_strokes):\n        for i in range(len(stroke[0]) - 1):\n            color = 255 - min(t, 10) * 13 if time_color else 255\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]),\n                         (stroke[0][i + 1], stroke[1][i + 1]), color, lw)\n    if size != base_size:\n        return cv2.resize(img, (size, size))\n    else:\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.title('Sample image of a roller coaster in grayscale')\nplt.imshow(255 - draw_cv2(data.iloc[9]['drawing']), cmap=plt.cm.gray)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Labels balance\nHere we display the count of each of the labels. They are sorted in order to be visualized in a single figure and still be able to provide a good representation of their count.  \nWe see that around 240 labels have between 113614 and 150000 samples while the around 15 labels have between 250000 and 340030 labels. These differences will affect the trained models and make them lean toward the over represented labels to some extent. Due to the large number of samples this effect might be limited and differences in the performance will depend on the balance of the test dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_files = os.listdir(train_simplified_path)\ntrain_count = []\nfor file in train_files:\n    with open(train_simplified_path + file) as f:\n        for (count, _) in enumerate(f, 1):\n            pass\n        train_count.append(count)\ntrain_count = np.sort(train_count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of classes: ', len(train_count))\nprint('Minimum count of a label: ', min(train_count))\nprint('Maximum count of a label: ', max(train_count))\nplt.title('The count of each of the labels sorted')\nplt.scatter(range(0, len(train_count)), train_count)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data generators\nSince the data size is large, we can't load it all in the memory and preprocess it into a list of images. So we will use a custom data generator that reads each csv and creates the images from the strockes while learning.  \nIn addition to having huge number of samples, each class is provided in a single csv seperatly which would add a computaional cost in order to create batchs with unifrom distribution of the classes. A possible way to solve this, is to load all the data, shuffle it and save it again to later sample from it. However, the method used here is to read randomly ```BATCHSIZE``` files and add a single sample from each one of them to the current batch."},{"metadata":{"trusted":true},"cell_type":"code","source":"# class index map\ntrain_files = os.listdir(train_simplified_path)\nclass_index = {}\nfor i, file in enumerate(train_files):\n    class_index[file[:-4].replace(' ', '_')] = i","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_generator(size=SIZE, files_list=train_files, class_index=class_index, batch_size=BATCHSIZE, n_classes=NCLASSES, lw=6, time_color=True):\n    while True:\n        ind = VALIDSAMPLES\n        while ind < 113610:\n            files = [files_list[i] for i in np.random.randint(n_classes, size=batch_size)]\n            x = np.zeros((batch_size, size, size, 1))\n            y = np.zeros((batch_size, n_classes))\n            for i, file in enumerate(files):\n                df = pd.read_csv(train_simplified_path + file, skiprows=range(1,ind), nrows=1)\n                df['drawing'] = df['drawing'].apply(ast.literal_eval)\n                df['word'] = df['word'].apply(lambda x: class_index[x.replace(' ', '_')])\n                y[i,:] = keras.utils.to_categorical(df['word'], num_classes=n_classes)\n                x[i,:,:,0] = draw_cv2(df.iloc[0]['drawing'], size=size, lw=lw, time_color=time_color)\n            x = preprocess_input(x).astype(np.float32)\n            ind += 1\n            yield x, y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the validation, the first ``VALIDSAMPLES`` samples from each class are loaded and preprocessed."},{"metadata":{"trusted":true},"cell_type":"code","source":"def validation_data(file_list=train_files, size=SIZE, class_index=class_index, n_samples_per_class=VALIDSAMPLES, n_classes=NCLASSES, lw=6, time_color=True):\n    x = np.zeros((len(file_list) * n_samples_per_class, size, size, 1))\n    y = np.zeros((len(file_list) * n_samples_per_class, n_classes))\n    for i, file in enumerate(file_list):\n        df = pd.read_csv(train_simplified_path + file, nrows=n_samples_per_class)\n        df['drawing'] = df['drawing'].apply(ast.literal_eval)\n        df['word'] = df['word'].apply(lambda x: class_index[x.replace(' ', '_')])\n        y[i*n_samples_per_class:(i+1)*n_samples_per_class,:] = keras.utils.to_categorical(df['word'], num_classes=n_classes)\n        for j, raw_strokes in enumerate(df.drawing.values):\n            x[i*n_samples_per_class+j,:,:,0] = draw_cv2(raw_strokes, size=size, lw=lw, time_color=time_color)\n    x = preprocess_input(x).astype(np.float32)\n    return x, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_validation, y_validation = validation_data()\nprint(x_validation.shape, y_validation.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_generator = train_generator()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model training\nWe have at our diposal a large number of sample that should be enough to reach high performance using the right model. So for the architecture choice, a predefined model is used and trained from scratch. As natural images and our sketchs differ widly, the weights are initialized randomly (note that even if we used a pretrained model after a certain number of epochs the pretrained and randomly initiallized model should converge).  \nDifferent popular architectures exist that make use of huge volumes of data, such as Resnet, InceptionV3 and Mobilnet. The use of these models allows to have deep architectures without suffering from vanishing gradient and optimization probelms, in addition to lower computaional cost. In particular MobileNet is fast and quite efficient as it is designed with objective to be applied in computer vision tasks that require quick responses such as autonomus vehicules.\nOur images contain far less features to uncover than natural images, so MobileNet is expected to perform well on our dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# metric\ndef top_3_categorical_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = MobileNet(input_shape=(SIZE, SIZE, 1), weights=None, classes=NCLASSES)\nmodel.compile(optimizer=Adam(lr=0.001), loss='categorical_crossentropy', metrics=[top_3_categorical_accuracy])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    ReduceLROnPlateau(monitor='val_top_3_categorical_accuracy', factor=0.5, patience=2, mode='max', min_lr=1e-5, verbose=1),\n    ModelCheckpoint('model.h5', monitor='val_top_3_categorical_accuracy', mode='max', save_best_only=True, save_weights_only=True)\n]\n\nhist = model.fit_generator(\n    train_data_generator, steps_per_epoch=500, epochs=15, verbose=1,\n    validation_data=(x_validation, y_validation),\n    callbacks = callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(hist.history.keys())\n#  \"top 3 Accuracy\"\nplt.plot(hist.history['top_3_categorical_accuracy'])\nplt.plot(hist.history['val_top_3_categorical_accuracy'])\nplt.title('model top 3 accuracy')\nplt.ylabel('top 3 accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()\n# \"Loss\"\nplt.plot(hist.history['loss'])\nplt.plot(hist.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# index to class mapping\nindex_class = {v: k for k, v in class_index.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction = model.predict(x_validation)\ny_true = []\nfor i in np.argmax(y_validation, axis=1):\n    y_true.append(index_class[i])\ny_pred = []\nfor i in range(prediction.shape[0]):\n    pred = ''\n    for j in prediction[i].argsort()[-3:][::-1]:\n        pred += index_class[j] + ' '\n    y_pred.append(pred[:-1])\n\ndel x_validation\ndel y_validation\nprint('Mean average precision (k=3) on the validation set: ', mapk(y_pred, y_true, k=3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(test_simplified_path, index_col='key_id')\nx_test = np.zeros((len(test), SIZE, SIZE, 1))\ntest['drawing'] = test['drawing'].apply(ast.literal_eval)\nfor i, raw_strokes in enumerate(test.drawing.values):\n    x_test[i,:,:,0] = draw_cv2(raw_strokes, size=SIZE, lw=6, time_color=True)\n    x_test[i,:,:,0] = preprocess_input(x_test[i,:,:,0]).astype(np.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = []\nprediction = model.predict(x_test)\nfor i in range(prediction.shape[0]):\n    pred = ''\n    for j in prediction[i].argsort()[-3:][::-1]:\n        pred += index_class[j] + ' '\n    y_pred.append(pred[:-1])\ntest['word']  = y_pred\ntest.drop(['drawing', 'countrycode'], axis=1, inplace=True)\ntest.to_csv('submission.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}