{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Keras DenseNet 121 for Quick, Draw! Doodle Recognition Challenge\n\nThis kernel is a fork of 🐘Greyscale MobileNet [LB=0.892] from @beluga. Huge thanks for his great kernel\n\nThis kernel has these main components:\n\n* DenseNet121\n* Fast and memory efficient Image Generator with temporal colored strokes\n\n"},{"metadata":{"_uuid":"f7f2a9516140a84124bf7bbf538ee4c30860b778"},"cell_type":"markdown","source":"## Setup\nImport the necessary libraries and a few helper functions."},{"metadata":{"trusted":true,"_uuid":"ce6d2aa7de1fa341144def7d3a5b1ffdea26bc91","_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport ast\nimport datetime as dt\nimport os\nimport time\nfrom math import trunc\n\nimport cv2\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow import keras\n\nimport pandas as pd\nimport seaborn as sns\nfrom keras.applications.densenet import DenseNet121, preprocess_input\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.layers import Dense, GlobalAveragePooling2D\nfrom keras.metrics import (categorical_accuracy, categorical_crossentropy,\n                           top_k_categorical_accuracy)\nfrom keras.models import Model, load_model\nfrom keras.optimizers import Adam\nfrom keras.callbacks import TensorBoard, ModelCheckpoint\n\nstart = dt.datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"978b1e827e598c53df3ef09838a6d85591d83052"},"cell_type":"code","source":"DP_DIR = '../input/shuffle-csvs/'\nINPUT_DIR = '../input/quickdraw-doodle-recognition/'\n\nBASE_SIZE = 256\nNCSVS = 100\nNCATS = 340\nnp.random.seed(seed=2018)\ntf.set_random_seed(seed=2018)\n\n\ndef f2cat(filename: str) -> str:\n    return filename.split('.')[0]\n\ndef list_all_categories():\n    files = os.listdir(os.path.join(INPUT_DIR, 'train_simplified'))\n    return sorted([f2cat(f) for f in files], key=str.lower)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b2fcd1a08ae1ae0619be38a113a244eb6515b63b"},"cell_type":"code","source":"def apk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    if len(predicted) > k:\n        predicted = predicted[:k]\n    score = 0.0\n    num_hits = 0.0\n    for i, p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i + 1.0)\n    if not actual:\n        return 0.0\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])\n\ndef preds2catids(predictions):\n    return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], columns=['a', 'b', 'c'])\n\ndef top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"264156422a95e4b350886d558d516ae8bd2e25c0"},"cell_type":"markdown","source":"## DenseNet 121\n\nDenseNet architecture is new, it is a logical extension of ResNet.\nResNet architecture has a fundamental building block (Identity) where you merge (additive) a previous layer into a future layer. Reasoning here is by adding additive merges we are forcing the network to learn residuals (errors i.e. diff between some previous layer and current one). In contrast, DenseNet paper proposes concatenating outputs from the previous layers instead of using the summation.\n\n[Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993)"},{"metadata":{"trusted":true,"_uuid":"54e5f0c637195b6624e2f3e6db5e7f8990e14eb7"},"cell_type":"code","source":"batchsize = 330\nsize = 64\nSTEPS = 10000\n\n# You can set as many epoch as you want until don't see any improvement \nEPOCHS = 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"0860ec35bee03f0c5cd21202dc7471c2d201cf5f","_kg_hide-output":true},"cell_type":"code","source":"base_model = DenseNet121(include_top=False, weights='imagenet',\n                         input_shape=(size, size, 3), classes=NCATS)\nx = base_model.output\nx = GlobalAveragePooling2D()(x)\nx = Dense(1024, activation='relu')(x)\npredictions = Dense(NCATS, activation='softmax')(x)\nmodel = Model(inputs=base_model.input, outputs=predictions)\nmodel.compile(optimizer=Adam(lr=1e-4, decay=1e-9), loss='categorical_crossentropy', metrics=[\n              categorical_crossentropy, categorical_accuracy, top_3_accuracy])\n\n# Load previous checkpoint (if you are tranning on a local machine)\n# model = load_model('path_to_checkpoint', custom_objects={'top_3_accuracy': top_3_accuracy})\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab1834ea2757a53d602a3508efffcc34bc190dc7"},"cell_type":"markdown","source":"## Training with Image Generator\nKeep in mind that Keras Densenet only accept RGB images, for this I only fill 3 channel with the same grayscale value so the image is not RGB, but it still works. You should try to enconde more information at this part, it will definitely help increase the LB accuracy"},{"metadata":{"trusted":true,"_uuid":"f6455bf9555b8381b6a4292098a64a0eb7ff54dc"},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, time_color=True):\n    img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)\n    for t, stroke in enumerate(raw_strokes):\n        for i in range(len(stroke[0]) - 1):\n            color = 255 - min(t, 10) * 13 if time_color else 255\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]),\n                         (stroke[0][i + 1], stroke[1][i + 1]), color, lw)\n    img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)\n    if size != BASE_SIZE:\n        return cv2.resize(img, (size, size))\n    else:\n        return img\n\n\ndef image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n    while True:\n        for k in np.random.permutation(ks):\n            filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n            for df in pd.read_csv(filename, chunksize=batchsize):\n                df['drawing'] = df['drawing'].apply(ast.literal_eval)\n                x = np.zeros((len(df), size, size, 3))\n                for i, raw_strokes in enumerate(df.drawing.values):\n                    x[i, :, :, :] = draw_cv2(raw_strokes, size=size, lw=lw,\n                                             time_color=time_color)\n                x = preprocess_input(x).astype(np.float32)\n                y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n                yield x, y\n\n\ndef df_to_image_array_xd(df, size, lw=6, time_color=True):\n    df['drawing'] = df['drawing'].apply(ast.literal_eval)\n    x = np.zeros((len(df), size, size, 3))\n    for i, raw_strokes in enumerate(df.drawing.values):\n        x[i, :, :, :] = draw_cv2(\n            raw_strokes, size=size, lw=lw, time_color=time_color)\n    x = preprocess_input(x).astype(np.float32)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98ff512e1a1b5e86e86d9eef4127525bedf3b9e1"},"cell_type":"code","source":"valid_df = pd.read_csv(os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(NCSVS - 1)), nrows=34000)\nx_valid = df_to_image_array_xd(valid_df, size)\ny_valid = keras.utils.to_categorical(valid_df.y, num_classes=NCATS)\nprint(x_valid.shape, y_valid.shape)\nprint('Validation array memory {:.2f} GB'.format(x_valid.nbytes / 1024.**3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d80ad7f4d378ea7f30479221d604eeeed559cae4"},"cell_type":"code","source":"train_datagen = image_generator_xd(size=size, batchsize=batchsize, ks=range(NCSVS - 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ce5fb89fbb77777316d6fca7689b6636c0e6021"},"cell_type":"code","source":"x, y = next(train_datagen)\nn = 8\nfig, axs = plt.subplots(nrows=n, ncols=n, sharex=True,\n                        sharey=True, figsize=(12, 12))\nfor i in range(n**2):\n    ax = axs[i // n, i % n]\n    (-x[i]+1)/2\n    ax.imshow((-x[i, :, :, 0] + 1)/2, cmap=plt.cm.gray)\n    ax.axis('off')\nplt.tight_layout()\nfig.savefig('gs.png', dpi=300)\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2afc64085dbd42e756eb4cea765ba322398ce608"},"cell_type":"markdown","source":"### I used Tensorboard for learning visualization"},{"metadata":{"trusted":true,"_uuid":"05767778d356bc63b7cded355159fd4082eee1a5"},"cell_type":"code","source":"tensorboard = TensorBoard(log_dir=\"logs/{}\".format(time.time()), batch_size=batchsize, write_images=True, update_freq=(STEPS/10)*batchsize)\nweightpath = \"./model/weights-{epoch:03d}-{top_3_accuracy:.3f}.hdf5\"\ncheckpoint = ModelCheckpoint(weightpath, monitor='val_loss', verbose=0,save_best_only=False, save_weights_only=False, mode='auto', period=1)\n\n\nmodel.fit_generator(\n    train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n    validation_data=(x_valid, y_valid),\n    callbacks=[tensorboard, checkpoint]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c1927f22d3c45cba0bdee7d6f4b6c858d82d614"},"cell_type":"code","source":"valid_predictions = model.predict(x_valid, batch_size=128, verbose=1)\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)\nprint('Map3: {:.3f}'.format(map3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be4577a9ba00611697eea8f241a42c504981e86f"},"cell_type":"markdown","source":"## Create Submission"},{"metadata":{"trusted":true,"_uuid":"a7d14348150baf753e90cf2719b9f31dd564f6a2"},"cell_type":"code","source":"test = pd.read_csv(os.path.join(INPUT_DIR, 'test_simplified.csv'))\ntest.head()\nx_test = df_to_image_array_xd(test, size)\nprint(test.shape, x_test.shape)\nprint('Test array memory {:.2f} GB'.format(x_test.nbytes / 1024.**3 ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"608b02f5c7909ae62becbe5c931b7264171296e8"},"cell_type":"code","source":"test_predictions = model.predict(x_test, batch_size=128, verbose=1)\n\ntop3 = preds2catids(test_predictions)\ntop3.head()\ntop3.shape\n\ncats = list_all_categories()\nid2cat = {k: cat.replace(' ', '_') for k, cat in enumerate(cats)}\ntop3cats = top3.replace(id2cat)\ntop3cats.head()\ntop3cats.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52e0f9c44f2a9a38fd1550ffb9c07fb7ea22b17d"},"cell_type":"code","source":"test['word'] = top3cats['a'] + ' ' + top3cats['b'] + ' ' + top3cats['c']\nsubmission = test[['key_id', 'word']]\nsubmission.to_csv('gs_mn_submission_{}.csv'.format(int(map3 * 10**4)), index=False)\nsubmission.head()\nsubmission.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b418f4c06c4e4453aa1b5ab16dde344eb8b735c5"},"cell_type":"code","source":"end = dt.datetime.now()\nprint('Latest run {}.\\nTotal time {}s'.format(end, (end - start).seconds))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e3a4ba36dfce2312437a9a598bfaaee20cc87fb"},"cell_type":"markdown","source":"# Suggession for improvement\n\nI'm able to achive LB 0.925 with a single DenseNet 121 model, no TTA, no ensemble. 100k/class for training, 340k for validation. After 45 epoch and 65 hours on a single GTX 1060 6GB. You can try these to improve the LB score:\n- Training with more data, you can use @beluga kernel to split and shuffle the data: https://www.kaggle.com/gaborfodor/shuffle-csvs\n- Bigger image size and bigger batch size (batch size should be as big as possible)\n- Try DenseNet 169\n- TTA (hflip) and ensemble\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}