{"cells":[{"metadata":{"_uuid":"3ba78cc685d3db80fbeae0c00d173d6ca794860b"},"cell_type":"markdown","source":"# Quick, Draw! Doodle Recognition Challenge\n\"Quick, Draw!\" was released as an experimental game to educate the public in a playful way about how AI works. The game prompts users to draw an image depicting a certain category, such as ”banana,” “table,” etc. The game generated more than 1B drawings, of which a subset was publicly released as the basis for this competition’s training set. That subset contains **50M drawings encompassing 340 label categories**.\n\nhttps://quickdraw.withgoogle.com/\n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport ast\nimport os\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport datetime as dt\nimport numpy as np\nimport cv2\nfrom tqdm import tqdm\nfrom sklearn.metrics import confusion_matrix\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Activation\nfrom tensorflow.keras.metrics import categorical_accuracy, top_k_categorical_accuracy, categorical_crossentropy\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.applications import MobileNet\nfrom tensorflow.keras.applications.mobilenet import preprocess_input\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.figure_factory as ff\nfrom IPython.core.display import HTML\nstart = dt.datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"DP_DIR = '../input/shuffle-animal-csvs/'\nINPUT_DIR = '../input/quickdraw-doodle-recognition/'\n# INPUT_DIR = '../input/'\nBW_DIR = '../input/black-white-cnn-animals/'\nGS_DIR = '../input/greyscale-mobilenet-animals/'\nBASE_SIZE = 256\nNCSVS = 100\nnp.random.seed(seed=1987)\ntf.set_random_seed(seed=1987)\n\ndef f2cat(filename: str) -> str:\n return filename.split('.')[0]\n\nclass Simplified():\n def __init__(self, input_path='./input'):\n self.input_path = input_path\n\n def list_all_categories(self):\n files = os.listdir(os.path.join(self.input_path, 'train_simplified'))\n return [f2cat(f) for f in files]\n\n def read_training_csv(self, category, nrows=None, usecols=None, drawing_transform=False):\n df = pd.read_csv(os.path.join(self.input_path, 'train_simplified', category + '.csv'),\n nrows=nrows, parse_dates=['timestamp'], usecols=usecols)\n if drawing_transform:\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"443d84891247f770f35486496546da9ee5f68fe0","trusted":true},"cell_type":"code","source":"s = Simplified(INPUT_DIR)\nanimals = ['ant', 'bat', 'bear', 'bee', 'bird', 'butterfly', 'camel', 'cat', 'cow',\n 'crab', 'crocodile', 'dog', 'dolphin', 'dragon', 'duck', 'elephant', 'fish',\n 'flamingo', 'frog', 'giraffe', 'hedgehog', 'horse', 'kangaroo', 'lion',\n 'lobster', 'monkey', 'mosquito', 'mouse', 'octopus', 'owl', 'panda',\n 'parrot', 'penguin', 'pig', 'rabbit', 'raccoon', 'rhinoceros', 'scorpion',\n 'sea turtle', 'shark', 'sheep', 'snail', 'snake', 'spider', 'squirrel',\n 'swan', 'teddy-bear', 'tiger', 'whale', 'zebra']\nNCATS = len(animals)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59d0902ce8bb3d92f83fc87b4a969a2fdf7ea9ae"},"cell_type":"markdown","source":"## Simplified drawings \n\nTwo versions of the data are given. The raw data is the exact input recorded from the user drawing, while the simplified version removes unnecessary points from the vector information. (For example, a straight line may have been recorded with 8 points, but since you only need 2 points to uniquely identify a line, 6 points can be dropped.) The simplified files are much smaller and provide effectively the same information. We will use the simplified files.\n\nDrawings are stored as strings in the drawing column for each category DataFrame."},{"metadata":{"trusted":true,"_uuid":"ccac887f621ef2a218b281e5506716b80fc5d7e2"},"cell_type":"code","source":"df = s.read_training_csv('owl', nrows=100, drawing_transform=True)\ndf.head()\n\ndrawing = df.drawing.values[0]\nprint(drawing)\nprint('--------------------------------------')\nprint('This drawing has {} strokes.'.format(len(drawing)))\nprint('i, [[xs], [ys]]')\nfor i, stroke in enumerate(drawing):\n print(i, stroke)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93e4c90c26b9e3f691ff9f2d54865cc43d541c73","trusted":true},"cell_type":"code","source":"n = 10\nfig, axs = plt.subplots(nrows=n, ncols=n, sharex=True, sharey=True, figsize=(16, 10))\nfor i, row in df[: n * n].iterrows():\n ax = axs[i // n, i % n]\n for x, y in row.drawing:\n color = 'green' if row.recognized else 'red'\n ax.plot(x, -np.array(y), lw=3, color=color)\n ax.axis('off')\nplt.suptitle('Recognized and unrecognized owls')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"b46ad026186b759c850a8c005b79c16dbc5d792a","trusted":true},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6):\n img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)\n for stroke in raw_strokes:\n for i in range(len(stroke[0]) - 1):\n _ = cv2.line(img, (stroke[0][i], stroke[1][i]), (stroke[0][i + 1], stroke[1][i + 1]), 255, lw)\n if size != BASE_SIZE:\n return cv2.resize(img, (size, size))\n else:\n return img\n\ndef image_generator(size, batchsize, ks, lw=6):\n while True:\n for k in np.random.permutation(ks):\n filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n for df in pd.read_csv(filename, chunksize=batchsize):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n x = np.zeros((len(df), size, size))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n x = x / 255.\n x = x.reshape((len(df), size, size, 1)).astype(np.float32)\n y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n yield x, y\n\ndef df_to_image_array(df, size, lw=6):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n x = np.zeros((len(df), size, size))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n x = x / 255.\n x = x.reshape((len(df), size, size, 1)).astype(np.float32)\n return x\n\ndef top_3_accuracy(y_true, y_pred):\n return top_k_categorical_accuracy(y_true, y_pred, k=3)\n\ndef apk(actual, predicted, k=3):\n \"\"\"\n Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n \"\"\"\n if len(predicted) > k:\n predicted = predicted[:k]\n\n score = 0.0\n num_hits = 0.0\n\n for i, p in enumerate(predicted):\n if p in actual and p not in predicted[:i]:\n num_hits += 1.0\n score += num_hits / (i + 1.0)\n\n if not actual:\n return 0.0\n\n return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=3):\n \"\"\"\n Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n \"\"\"\n return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])\n\n\ndef preds2catids(predictions):\n return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], columns=['a', 'b', 'c'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccc872d7dffaaa6dbbb3b5877b1283e441973278"},"cell_type":"markdown","source":"## Train simple CNN\n[Beginner’s guide to Understanding Convolutional Neural Networks](https://medium.com/@junehaoching/beginners-guide-to-understanding-convolutional-neural-networks-5209e5d9f717)\n![A](https://cdn-images-1.medium.com/max/800/1*RWn-qcAvt4tyu_sCDpXVUw.png)\n### Convolution\n![conv](https://cdn-images-1.medium.com/max/600/1*ZCjPUFrB6eHPRi4eyP6aaA.gif)\n### Pooling\n![pooling](https://cdn-images-1.medium.com/max/800/0*fZdIq8a_46j4rw4Q.png)\n\n### Fully connected\n![full](https://cdn-images-1.medium.com/max/800/0*yQAKlw1Zauh0ZsKv.jpg)\n\n### All together\n![together](https://cdn-images-1.medium.com/max/2000/0*ZuYqJj-W1JNsO8PE.png)\n\nhttps://www.kaggle.com/gaborfodor/black-white-cnn-animals\n\n\n## Load simple CNN"},{"metadata":{"_uuid":"9ab2df1837b584462a500e281ae85b6cef07cc91","trusted":true},"cell_type":"code","source":"size = 32\nmodel = Sequential()\nmodel.add(Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu',\n input_shape=(size, size, 1)))\nmodel.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\nmodel.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(NCATS, activation='softmax'))\nmodel.compile(optimizer=Adam(lr=0.002), loss='categorical_crossentropy',\n metrics=[categorical_crossentropy, categorical_accuracy, top_3_accuracy])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b08133c39c4e6bbf6aaf200d644862ab048ce5b","trusted":true},"cell_type":"code","source":"model.load_weights(filepath=os.path.join(BW_DIR, 'bw_animal_cnn.h5'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"610d02d4bd0a10fc7aca578eefd0b39bb5f0f8cf","trusted":true},"cell_type":"code","source":"valid_df = pd.read_csv(os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(NCSVS - 1)), nrows=10**5)\nx_valid = df_to_image_array(valid_df, size)\ny_valid = keras.utils.to_categorical(valid_df.y, num_classes=NCATS)\nprint(x_valid.shape, y_valid.shape)\nprint('Validation array memory {:.2f} GB'.format(x_valid.nbytes / 1024.**3 ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88d9d75198bcfd0a668166705df5eaeb5257225f","trusted":true},"cell_type":"code","source":"valid_predictions = model.predict(x_valid, batch_size=128, verbose=1)\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)\nprint('Top1 Accuracy: {:.3f}'.format(np.mean(valid_df.y.values == np.argmax(valid_predictions, 1))))\nprint('Map@3: {:.3f}'.format(map3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a32095605235f9c22d6e8bfa14571c3894de41cc","trusted":true},"cell_type":"code","source":"predicted_cat = np.argmax(valid_predictions, 1)\ncmx = confusion_matrix(valid_df.y.values, np.argmax(valid_predictions, 1))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"acf41415ebc73dad24de2b690a895cb5640eb839","trusted":true},"cell_type":"code","source":"k = 20\nlayout = dict(\n title = 'BW CNN Confusion Matrix',\n xaxis= dict(title='Predicted Class', ticklen=5, zeroline=False, gridwidth=2),\n yaxis=dict(title='Ground Truth', ticklen=5, gridwidth=2),\n width = 800,\n height = 800,\n margin=go.layout.Margin(l=200, r=50, b=50, t=250, pad=4),\n)\nfig = ff.create_annotated_heatmap(\n z=cmx[:k, :k],\n x=list(animals[:k]),\n y=list(animals[:k]),\n colorscale='Blues',\n reversescale=True,\n showscale=True,\n font_colors = ['#efecee', '#3c3636'])\nfig['layout'].update(layout)\npy.iplot(fig, filename='bw_confusion')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccc872d7dffaaa6dbbb3b5877b1283e441973278"},"cell_type":"markdown","source":"## Train MobileNet\nhttps://www.kaggle.com/gaborfodor/greyscale-mobilenet-animals/notebook"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0a175467ee7282badfa0c53a13edc79c5a05b8ba"},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, time_color=True):\n img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)\n for t, stroke in enumerate(raw_strokes):\n for i in range(len(stroke[0]) - 1):\n color = 255 - min(t, 10) * 13 if time_color else 255\n _ = cv2.line(img, (stroke[0][i], stroke[1][i]),\n (stroke[0][i + 1], stroke[1][i + 1]), color, lw)\n if size != BASE_SIZE:\n return cv2.resize(img, (size, size))\n else:\n return img\n\ndef image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n while True:\n for k in np.random.permutation(ks):\n filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n for df in pd.read_csv(filename, chunksize=batchsize):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n x = np.zeros((len(df), size, size, 1))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw,\n time_color=time_color)\n x = preprocess_input(x).astype(np.float32)\n y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n yield x, y\n\ndef df_to_image_array_xd(df, size, lw=6, time_color=True):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n x = np.zeros((len(df), size, size, 1))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw, time_color=time_color)\n x = preprocess_input(x).astype(np.float32)\n return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52a4a969af30b1ce88a3b29b69a6bb14106dec2c"},"cell_type":"markdown","source":"## Load MobileNet"},{"metadata":{"_uuid":"9ab2df1837b584462a500e281ae85b6cef07cc91","_kg_hide-output":true,"trusted":true,"scrolled":false},"cell_type":"code","source":"size = 64\nmodel = MobileNet(input_shape=(size, size, 1), alpha=1., weights=None, classes=NCATS)\nmodel.compile(optimizer=Adam(lr=0.002), loss='categorical_crossentropy',\n metrics=[categorical_crossentropy, categorical_accuracy, top_3_accuracy])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b08133c39c4e6bbf6aaf200d644862ab048ce5b","trusted":true},"cell_type":"code","source":"model.load_weights(filepath=os.path.join(GS_DIR, 'gs_animal_mobile.h5'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"610d02d4bd0a10fc7aca578eefd0b39bb5f0f8cf","trusted":true},"cell_type":"code","source":"valid_df = pd.read_csv(os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(NCSVS - 1)), nrows=10**5)\nx_valid = df_to_image_array_xd(valid_df, size)\ny_valid = keras.utils.to_categorical(valid_df.y, num_classes=NCATS)\nprint(x_valid.shape, y_valid.shape)\nprint('Validation array memory {:.2f} GB'.format(x_valid.nbytes / 1024.**3 ))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"334b02dfd3245c9d8924aecab5b995337d2a7993","trusted":true},"cell_type":"code","source":"bw_hist_df = pd.read_csv(os.path.join(BW_DIR, 'bw_cnn_history.csv'))\ngs_hist_df = pd.read_csv(os.path.join(GS_DIR, 'gs_mobile_history.csv'))\ndata = [\n go.Scatter(\n x=gs_hist_df.index.values,\n y=gs_hist_df.val_categorical_accuracy.values,\n mode='lines',\n name='64x64 MobileNet',\n line=dict(width=4, color='#5ac995')\n ),\n go.Scatter(\n x=bw_hist_df.index,\n y=bw_hist_df.val_categorical_accuracy.values,\n mode='lines',\n name='32x32 Simple CNN',\n line=dict(width=4, color='#007FB4')\n ),\n]\nlayout = go.Layout(\n title='Validation Performance',\n xaxis=dict(title='Epoch', ticklen=5, zeroline=False, gridwidth=2),\n yaxis=dict(title='Accuracy', ticklen=5, gridwidth=2),\n showlegend=True\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename='users')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88d9d75198bcfd0a668166705df5eaeb5257225f","trusted":true},"cell_type":"code","source":"valid_predictions = model.predict(x_valid, batch_size=128, verbose=1)\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)\nprint('Top1 Accuracy: {:.3f}'.format(np.mean(valid_df.y.values == np.argmax(valid_predictions, 1))))\nprint('Map@3: {:.3f}'.format(map3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a32095605235f9c22d6e8bfa14571c3894de41cc","trusted":true},"cell_type":"code","source":"predicted_cat = np.argmax(valid_predictions, 1)\ncmx = confusion_matrix(valid_df.y.values, np.argmax(valid_predictions, 1))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"acf41415ebc73dad24de2b690a895cb5640eb839","trusted":true},"cell_type":"code","source":"k = 20\nlayout = dict(\n title = 'GS Mobilenet Confusion Matrix',\n xaxis= dict(title='Predicted Class', ticklen=5, zeroline=False, gridwidth=2),\n yaxis=dict(title='Ground Truth', ticklen=5, gridwidth=2),\n width = 800,\n height = 800,\n margin=go.layout.Margin(l=200, r=50, b=50, t=250, pad=4),\n)\nfig = ff.create_annotated_heatmap(\n z=cmx[:k, :k],\n x=list(animals[:k]),\n y=list(animals[:k]),\n colorscale='Greens',\n reversescale=True,\n showscale=True,\n font_colors = ['#efecee', '#3c3636'])\nfig['layout'].update(layout)\npy.iplot(fig, filename='bw_confusion')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9c586fa6790e265e8f22562181cb2873554183c","trusted":true},"cell_type":"code","source":"end = dt.datetime.now()\nprint('Latest run {}.\\nTotal time {}s'.format(end, (end - start).seconds))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"117ecd6605b1a83e8e173f7b9f881a8e9f9fbd64"},"cell_type":"markdown","source":"# Try another network\n\n![ModelZoo](https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/113660/36e5c10e00918f72d404e381d83f1e0c/models.png)\n\n[Benchmark Analysis of Representative\nDeep Neural Network Architectures](https://arxiv.org/pdf/1810.00736.pdf)"},{"metadata":{"_kg_hide-input":true,"_uuid":"928488553122d9236a8e30bbfd35e9df5544b354","trusted":true},"cell_type":"code","source":"HTML('''\n

Available pretrained keras models

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelSizeTop-1 AccuracyTop-5 AccuracyParametersDepth
Xception88 MB0.7900.94522,910,480126
VGG16528 MB0.7130.901138,357,54423
VGG19549 MB0.7130.900143,667,24026
ResNet5099 MB0.7490.92125,636,712168
InceptionV392 MB0.7790.93723,851,784159
InceptionResNetV2215 MB0.8030.95355,873,736572
MobileNet16 MB0.7040.8954,253,86488
MobileNetV214 MB0.7130.9013,538,98488
DenseNet12133 MB0.7500.9238,062,504121
DenseNet16957 MB0.7620.93214,307,880169
DenseNet20180 MB0.7730.93620,242,984201
NASNetMobile23 MB0.7440.9195,326,716-
NASNetLarge343 MB0.8250.96088,949,818-
\n

The top-1 and top-5 accuracy refers to the model's performance on the ImageNet validation dataset.

\n''')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33c4bd90094be103f94b0f55afafd178dc881b8d"},"cell_type":"markdown","source":"Source: https://keras.io/applications/"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}