{"cells":[{"metadata":{"_uuid":"34e60b0a2f4ef25e8738e691aeab30cca24c0f3e"},"cell_type":"markdown","source":"# MobileNet 126x126x3 100k per class\n\nThis kernel is description of my final solution for Quick, Draw! Doodle Recognition Challenge.<br>\n\nI used MobileNet from keras.application package. Earlier I tried Resnet18 with Stochastic Depth, but it converged slower and had slightly lower performance. Changing model was first key decission to improve my score.<br>\nAfter that I experimented with image size and stroke encoding. It resulted with another rise of accuracy. Everything is described later in kernel.<br>\nI made some mistakes with reducing learning rate on plateou. LR was reduced to early and network hadn't possibility to converge because of too small learning rate. After increasing patient parameter to 10 was noticed significant advance.<br>\n\nI also experimented with some RNN-CNN solutions, but there were worse than normal CNN.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom skimage.draw import draw\nfrom glob import glob\nfrom sklearn.preprocessing import LabelEncoder\nfrom keras.utils.np_utils import to_categorical\nfrom multiprocessing.dummy import Pool\nfrom keras.models import load_model\nimport time\nimport ast\nimport keras\nimport random\nimport glob\nimport math","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"6e6055d177861e43830cb737f5cd9a182f812c38"},"cell_type":"code","source":"ALL_FILES = glob.glob('../input/shuffle-csvs*/*.csv.gz')\nVALIDATION_FILE = '../input/shuffle-csvs-75000-100000/train_k0.csv.gz'\nALL_FILES.remove(VALIDATION_FILE)\nINPUT_DIR = '../input/quickdraw-doodle-recognition/'\nBASE_SIZE = 256\nNCATS = 340\nnp.random.seed(seed=1987)\n\n\ndef apk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    if len(predicted) > k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i, p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i + 1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])\n\n\ndef preds2catids(predictions):\n    return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], columns=['a', 'b', 'c'])\n\ndef f2cat(filename: str) -> str:\n    return filename.split('.')[0]\n\ndef list_all_categories():\n    files = os.listdir(os.path.join(INPUT_DIR, 'train_simplified'))\n    return sorted([f2cat(f) for f in files], key=str.lower)\n\n\ndef plot_batch(x):    \n    cols = 4\n    rows = 6\n    fig, axs = plt.subplots(nrows=rows, ncols=cols, sharex=True, sharey=True, figsize=(18, 18))\n    for i in range(rows):\n        for k in range(0,3):\n            ax = axs[i, k]\n            ax.imshow(x[i, :, :, k], cmap=plt.cm.gray)\n            ax.axis('off')\n        ax = axs[i, 3]\n        ax.imshow(x[i, :, :], )\n        ax.axis('off')\n    fig.tight_layout()\n    plt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab5678c5ac0d93cc8df5ea626e40856359600bdb"},"cell_type":"markdown","source":"# Learning and data hyper parameters<br> \nIf **AUGMENTATION** set True, images were flipped horizontaly with probability equals to 0.5.<br>\n**BATCH_SIZE** were reduced from 512 to 448, because there were a problem with memory when used loaded keras model. There is a issue ticket on keras github where people have the same problems.<br>\nIn keras MobileNet documentation is stated that models trained on less downsized images have better accuracy. In this case, accuracy improvment was observed after changing **IMAGE_SIZE** from 64 to 128.<br>Strange image size which is not power of 2 is selected because of my typo in implementation, 126 would be much better choice. \n"},{"metadata":{"trusted":true,"_uuid":"4ac37f2a8eea854e12d220d25ff33bb5305aa8ad"},"cell_type":"code","source":"AUGMENTATION = True\nSTEPS = 500\nBATCH_SIZE = 448\nEPOCHS = 0\nLEARNING_RATE = 0.002\n\n\nIMG_SHAPE = (128,128,3)\nIMG_SIZE = IMG_SHAPE[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cf8bcf88ff0724fb09352a8e2088cf81d2de511"},"cell_type":"markdown","source":"# Image encoding\n\nFrom raw stokes I created 3 images with different encoding. Encoded images were concatenated to one image of size 128x128 and 3 channels.<br>\n* First channel represents presence of line. Single point had 255 value if there were stroke or 0 otherwise.\n* Second channel encoded strokes in time. Usually people firstly draw outline of object and details later. This assumption were used to set weights of each stroke. First stroke was encoded with 255 value which was deacreased with every next stroke by 13,  down to 125.\n* Third channel encoded stroke points in time. There are some patterns in stroke directions. For example when I draw a spider, I start draw legs from body, not the other way. First point of a stroke have 255 value which was deacreased gradually down to 20.<br>\n\nLater you can find some visualisation of presented encodings.\n"},{"metadata":{"trusted":true,"_uuid":"81b8377ad8198aa79d9b44a2ca039146c5771d6d"},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, augmentation = False):\n    img = np.zeros((BASE_SIZE, BASE_SIZE, 3), np.uint8)\n    for t, stroke in enumerate(raw_strokes):\n        points_count = len(stroke[0]) - 1\n        grad = 255//points_count\n        for i in range(len(stroke[0]) - 1):\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]), (stroke[0][i + 1], stroke[1][i + 1]), (255, 255 - min(t,10)*13, max(255 - grad*i, 20)), lw)\n    if size != BASE_SIZE:\n        img = cv2.resize(img, (size, size))\n    if augmentation:\n        if random.random() > 0.5:\n            img = np.fliplr(img)\n    return img\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d84a9bb0ade670167f062e0f8877162e4e91ebc"},"cell_type":"markdown","source":"# Data generators\n\nData generators are based on Beluga kernel. In original data set classes are splitted to separate files. To improve performance files were merged in separete kernels to single files cointaining all classes examples in random order. It gave much faster generator and time to generate single batch decreased 2 times."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"\n\ndef image_generator(size, batchsize, lw=6, augmentation = False):\n    while True:\n        for filename in ALL_FILES:\n            for df in pd.read_csv(filename, chunksize=batchsize):\n                df['drawing'] = df['drawing'].apply(eval)\n                x = np.zeros((len(df), size, size,3))\n                for i, raw_strokes in enumerate(df.drawing.values):\n                    x[i] = draw_cv2(raw_strokes, size=size, lw=lw, augmentation = augmentation)\n                x = x / 255.\n                x = x.reshape((len(df), size, size, 3)).astype(np.float32)\n                y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n                yield x, y\n\ndef valid_generator(valid_df, size, batchsize, lw=6):\n    while(True):\n        for i in range(0,len(valid_df),batchsize):\n            chunk = valid_df[i:i+batchsize]\n            x = np.zeros((len(chunk), size, size,3))\n            for i, raw_strokes in enumerate(chunk.drawing.values):\n                x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n            x = x / 255.\n            x = x.reshape((len(chunk), size, size,3)).astype(np.float32)\n            y = keras.utils.to_categorical(chunk.y, num_classes=NCATS)\n            yield x,y\n        \ndef test_generator(test_df, size, batchsize, lw=6):\n    for i in range(0,len(test_df),batchsize):\n        chunk = test_df[i:i+batchsize]\n        x = np.zeros((len(chunk), size, size,3))\n        for i, raw_strokes in enumerate(chunk.drawing.values):\n            x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n        x = x / 255.\n        x = x.reshape((len(chunk), size, size, 3)).astype(np.float32)\n        yield x\n        \n        \ntrain_datagen = image_generator(size=IMG_SIZE, batchsize=BATCH_SIZE, augmentation = AUGMENTATION)\n\nvalid_df = pd.read_csv(VALIDATION_FILE)\nvalid_df['drawing'] = valid_df['drawing'].apply(eval)\nvalidation_steps = len(valid_df)//BATCH_SIZE\nvalid_datagen = valid_generator(valid_df, size=IMG_SIZE, batchsize=BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"715ca7d165e3a4cc5ae9556daf55d336ea9dfb5c"},"cell_type":"markdown","source":"# Visualization of image encoding for \"ambulance\" class\n\nBelow is visualization of encoding for some ambulance images. First 3 columns represent described earlier channels and last 4th column is preview of whole image in RGB scale.<br>\nIn second column we can see that our assumption that people draw an outline first is right. Body of car have brighter lines than wheels or cross on the side. Also we can notice that there is pattern in drawing a wheel. Most people starts at top and sketch in anticlockwise direction."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6d8da2a2e6e7edbd72a7e019c50a88fc8fc763f3"},"cell_type":"code","source":"single_class_df = valid_df[valid_df['y'] == 2]\nsingle_class_gen = valid_generator(single_class_df, size=IMG_SIZE, batchsize=BATCH_SIZE)\nx, y = next(single_class_gen)\nplot_batch(x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bec4a9a56801c8d4a862843c1ffd4ec884afb291"},"cell_type":"markdown","source":"# Visualization of trainging batch"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8ea5cdee01222cf0fae6f4f683c00719e4eb867f"},"cell_type":"code","source":"x, y = next(train_datagen)\nplot_batch(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4a801340109f8fd7d4fc4879068f9bbcba9bed3"},"cell_type":"markdown","source":"# Visualization of validation batch"},{"metadata":{"trusted":true,"_uuid":"c6892231ebf536e1c68fe0efe92e6347900fc95f","_kg_hide-input":true},"cell_type":"code","source":"x, y = next(valid_datagen)\nplot_batch(x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"216ee535ab5a8b7c1bcca7ab705a3dde745760b6"},"cell_type":"markdown","source":"# Model definition\n\nAs I mentioned before, I used MobileNet from keras.application package. To train it for more epochs I piplined kernels to load previously trained model and continue whole process. It was annoying, but it helps me to save a lot of money on AWS or other cloud computing.<br>\n\nBecause network were trained from generator and there were a lot of data (3.4kk), each epoch means 500 batches for 488 examples.<br>\nFor the first time I used ReduceLROnPlateau callback and have learned that properly reduced LR can provide much better results. After 10 epochs without progress its reduced learning rate by half which usually gives noticable drop of loss value and helps network to converge."},{"metadata":{"trusted":true,"_uuid":"1d98bc5401d88dba539a17965cacb4b157acfcf9","scrolled":false,"_kg_hide-output":false},"cell_type":"code","source":"from keras.layers import Conv2D, MaxPooling2D\nfrom keras.layers import Dense, Dropout, Flatten, Activation\nfrom keras.metrics import categorical_accuracy, top_k_categorical_accuracy, categorical_crossentropy\nfrom keras.models import Sequential\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom keras.optimizers import Adam\nfrom keras.applications.mobilenet import MobileNet\nfrom keras.applications.mobilenet import preprocess_input\nfrom keras.models import load_model\n\ndef top_3_accuracy(y_true, y_pred):\n    return keras.metrics.top_k_categorical_accuracy(y_true, y_pred, k=3)\n\nreducer = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=10)\ncheckpointer = ModelCheckpoint(filepath='mobileNet-best.hdf5', verbose=0, save_best_only=True)\nmodel = load_model('../input/mobilenet-126x126x3-100k-per-class/mobileNet.hdf5', custom_objects = {'top_3_accuracy':top_3_accuracy})\nopt = Adam(lr = LEARNING_RATE)\nmodel.compile(optimizer = opt, loss = 'categorical_crossentropy', metrics = ['accuracy', top_3_accuracy])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1c74af76e612b50392a87cbdd5cda2720238116"},"cell_type":"markdown","source":"# Training"},{"metadata":{"trusted":true,"_uuid":"9f7114fd763a5f4ce83ccec76375988cb78107cb"},"cell_type":"code","source":"history = model.fit_generator(train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS , validation_data=valid_datagen, validation_steps = validation_steps, callbacks= [checkpointer, reducer])\nmodel.save('mobileNet.hdf5')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13cc85b6a42ff6d4750e16972d59d88762cb5a8b"},"cell_type":"code","source":"#merged log.csv files from each kernel version\nlog_df = pd.read_csv('../input/whole-training-log/log.csv')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"108fd8b96bb73bac34330b79dfc11250768d580a"},"cell_type":"code","source":"p = log_df[['loss','val_loss']].plot(figsize = (7,7))\np.set_xlabel('Epochs')\np","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"31e1f4fd14255eb0617c0fbb909985c2054584ad"},"cell_type":"code","source":"p = log_df[['acc','val_acc', 'top_3_accuracy','val_top_3_accuracy']].plot(figsize = (7,7))\np.set_xlabel('Epochs')\np","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9e878bdca0df954fc6f5f9a0ad8026d3357e2e26"},"cell_type":"code","source":"p = log_df[['lr']].plot(figsize = (5,5))\np.set_xlabel('Epochs')\np","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"11d7fe79172fefda973191ed30cba421d39d03a0"},"cell_type":"markdown","source":"Below we can observe mentioned loss drop after reducing LR."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ac337136f2189a0ddedbc2117c31f33594aa90c3"},"cell_type":"code","source":"p = log_df.iloc[150:200][['loss','val_loss']].plot(figsize = (7,7))\np.set_xlabel('Epochs')\np","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"71f21fdc794f028b6faaf15ca9f3442c2fe304ef","_kg_hide-input":true},"cell_type":"code","source":"log = pd.DataFrame.from_dict(history.history)\nlog.to_csv('train_log.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d2aef48269930b95eb0f444032e0913323aacab"},"cell_type":"markdown","source":"# Evaluation of model using MAP3"},{"metadata":{"trusted":true,"_uuid":"8613d4f32eaeba196cb0246b42d30065849b4d42","_kg_hide-input":true},"cell_type":"code","source":"gen = test_generator(valid_df, size=IMG_SIZE, batchsize=BATCH_SIZE)\nvalid_predictions = model.predict_generator(gen, steps = validation_steps, verbose=1)\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)\nprint('Map3: {:.3f}'.format(map3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76fe556e94afa18a18c0d5f805f9e67649bcf663","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"submission_df = pd.read_csv(os.path.join(INPUT_DIR, 'test_simplified.csv'))\nsubmission_df['drawing'] = submission_df['drawing'].apply(eval)\nsubmission_datagen = test_generator(submission_df, size=IMG_SIZE, batchsize=BATCH_SIZE)\nsubmission_predictions = model.predict_generator(submission_datagen, math.ceil(len(submission_df)/BATCH_SIZE))\ncats = list_all_categories()\nid2cat = {k: cat.replace(' ', '_') for k, cat in enumerate(cats)}\ntop3 = preds2catids(submission_predictions)\ntop3cats = top3.replace(id2cat)\nsubmission_df['word'] = top3cats['a'] + ' ' + top3cats['b'] + ' ' + top3cats['c']\nsubmission = submission_df[['key_id', 'word']]\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6dbb0b73d7e898eabf59fb02b2f12cf387dac49c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}