{"cells":[{"metadata":{"_uuid":"b787e4a0ce54ea38ae1c10fee2a75b135d368a11"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"205391a974e117899d0e6eeddd43349277419b3a"},"cell_type":"markdown","source":"## Data Exploartion\nBefore classification an understanding of the data is appropriate.  The data comes from the game Quick Draw where a user has 20 seconds to draw a doodle of an object. An algorithm than tries to guess what they drew. The data provided comes in two versions. The raw data which contains the x and y pixel positions of the strokes and the time in milliseconds from the first point in the stroke. The simplified data contains a normalised an pixel position between 0 and 255. The time information is removed. The dataset contains of 340 different classes. Here is an an example of the simplified data can be seen below. Code is taken from https://www.kaggle.com/jpmiller/image-based-cnn . The values in the 'drawing' column are a pair of two lists. One for the x values in a stroke and one for the corresponding y values for the stroke. The length of an entry in 'drawing' corresponds to the number of strokes used in that drawing. Other than that there is also a contry code, a timestamp of when the image was created and the word that the user was trying to draw.  \n\n"},{"metadata":{"trusted":true,"_uuid":"1444a84513cb0b25b535480cb01215bc3a7dc750","_kg_hide-input":true},"cell_type":"code","source":"import os\nimport re\nfrom glob import glob\nfrom tqdm import tqdm\nimport numpy as np\nimport pandas as pd\nimport ast\nimport gc\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nimport matplotlib.cbook\nwarnings.filterwarnings(\"ignore\",category=matplotlib.cbook.mplDeprecation)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2ffc31669cb879ad35a4c23884b3954ad6016a4a","_kg_hide-input":true},"cell_type":"code","source":"fnames = glob('../input/train_simplified/*.csv')\ncnames = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\ndrawlist = []\nfor f in fnames[0:8]:\n    first = pd.read_csv(f, nrows=40) # make sure we get a recognized drawing\n    first = first[first.recognized==True].head(20)\n    drawlist.append(first)\ndraw_df = pd.DataFrame(np.concatenate(drawlist), columns=cnames)\ndraw_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a73f809e62b0b74eddd1beaf4cd76a4fb076d084"},"cell_type":"markdown","source":"To see how images can look we will plot images from samples classes from 8 classes. "},{"metadata":{"trusted":true,"_uuid":"0aebb9f2126f6c03815519c85219fa951c4ac678","_kg_hide-input":true},"cell_type":"code","source":"\nlabels = unique_classes = draw_df['word'].unique()\nfor label in labels:\n    plt.figure(figsize=(10,10))\n    \n    \n    for ii in range(1,5):\n        examples = [ast.literal_eval(pts) for pts in draw_df[draw_df['word']==label].drawing.values]\n        #import pdb; pdb.set_trace()\n        for x,y in examples[ii]:\n            plt.subplot(2,2,ii)\n            plt.plot(x,y,marker='.')\n            plt.axis('off')\n            plt.title(label)\nplt.show()\n\ndraw_df = None\ndel draw_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f6b822621c176b5676d69b6db5201db69279b13"},"cell_type":"markdown","source":"Here we load the 50 first classes to at the distribution of the classes and how large each class is.."},{"metadata":{"trusted":true,"_uuid":"6d08c678a4aab4b3ab3ff13f8992068fec65a55d","_kg_hide-input":true},"cell_type":"code","source":"fnames = glob('../input/train_simplified/*.csv')\ncnames = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\ndrawlist = []\nfor f in fnames[0:50]:\n    df = pd.read_csv(f) # make sure we get a recognized drawing\n    df = df[df.recognized==True]\n    drawlist.append(len(df))\nplt.hist(drawlist,bins=15)\nplt.title('Histogram of number of images for first 50 classes.')\nplt.ylabel('Number of classes')\nplt.xlabel('Number of images')\n\ndf = None\ndrawlist = None\nfnames = None\ncnames = None\ndel df\ndel drawlist\ndel fnames\ndel cnames\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41cf7d6774130343e672707b8fd316ef1bef6b60"},"cell_type":"markdown","source":"When looking at the distribution of images with a quick glance two things become quite obvious. First of all just by judging the first 50 classes it looks like there are between 100000-200000 images in most of the classes. A dataset of that size will very quickly use up all the available memory. The second thing is that it is not an equal distribution of images per class. Some classes have 3 times the amount of images as others. This can pose a problem because a network can be overfitted to a few class since choosing the most represented classes will automaticly increase accuracy during training because there are more instances of them. Both problems can be solved simultaneously by limiting the number of images per class to be the same for all classes.  "},{"metadata":{"_uuid":"468506bcf85fc736eb9218e0f2024e42e62c125e"},"cell_type":"markdown","source":"# Choosing a network\n\nA simple CNN will be used as a baseline. This model is chosen because of it's simplicity and it's robust performance on the MNIST database which is kind of similar to the approach in this kernal. The model is from https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/ and is made up of:\n    1. Convolutional layer with 32 feature maps of size 5×5.\n    2. Pooling layer taking the max over 2*2 patches.\n    3. Dropout layer with a probability of 20%.\n    4. Flatten layer.\n    5. Fully connected layer with 128 neurons and rectifier activation.\n    6.. Output layer.\n    \nThe second model that will be chosen is a small version based on a VGG-net structure. The original model can be found here https://www.pyimagesearch.com/2018/04/16/keras-and-convolutional-neural-networks-cnns/. The charactaristic thing about the VGG-net is having more stacked 3x3 filters as we move further down the network. By increasing the number of filters further it should learn more and more abstract features. The end of the VGG net is usually a flattened layer followed by two fully connected layers. \n\n# Initial testing\nTo get a rough idea of what structure to use the following approach is used. The small VGG-net is split into three models with growing complexity.  The three models are tested together with the baseline using 5-fold cross validation. Since it takes quite some time to train on all the data a small subset of 20 classes with 500 images each are used to get a rough idea about the networks.\n\nThe following code to load the data is from https://www.kaggle.com/jpmiller/image-based-cnn which loads N images from N class and converts the stroke information into an actual image of size 32x32."},{"metadata":{"trusted":true,"_uuid":"f4a334ee51548da1c610420115faf203465335b7","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"#%% import\nimport os\nfrom glob import glob\nimport re\nimport ast\nimport numpy as np \nimport pandas as pd\nfrom PIL import Image, ImageDraw \nfrom tqdm import tqdm\nfrom dask import bag\nimport matplotlib.pyplot as plt\nimport keras\nimport gc\n%matplotlib inline\n## Import up sound alert dependencies\nfrom IPython.display import Audio, display\n\ndef allDone():\n  display(Audio(url='https://sound.peal.io/ps/audios/000/000/537/original/woo_vu_luvub_dub_dub.wav', autoplay=True))\n## Insert whatever audio file you want above\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b127a37aff539b3d2b21b950c1d29020a83ac8d5","_kg_hide-input":true},"cell_type":"code","source":"#%% set label dictionary and params\nclassfiles = os.listdir('../input/train_simplified/')\nnumstonames = {i: v[:-4].replace(\" \", \"_\") for i, v in enumerate(classfiles)} #adds underscores\n\nnum_classes = 20    #340 max \nimheight, imwidth = 32, 32  \nims_per_class = 500  # 1500 starts to crash the kernel often\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fd0d4e14e5dc98f4e6609ce4f398ed909af5d2a","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# faster conversion function\ndef draw_it(strokes):\n    image = Image.new(\"P\", (256,256), color=255)\n    image_draw = ImageDraw.Draw(image)\n    for stroke in ast.literal_eval(strokes):\n        for i in range(len(stroke[0])-1):\n            image_draw.line([stroke[0][i], \n                             stroke[1][i],\n                             stroke[0][i+1], \n                             stroke[1][i+1]],\n                            fill=0, width=5)\n    image = image.resize((imheight, imwidth))\n    return np.array(image)/255.\n\n#%% get train arrays\ntrain_grand = []\nclass_paths = glob('../input/train_simplified/*.csv')\nword_label = []\nfor i,c in enumerate(tqdm(class_paths[0: num_classes])):\n    train = pd.read_csv(c, usecols=['drawing', 'recognized','word'], nrows=ims_per_class*5//4)\n    train = train[train.recognized == True].head(ims_per_class)\n    word_label_current = train['word'].replace(' ', '_', regex=True)\n    imagebag = bag.from_sequence(train.drawing.values).map(draw_it) \n    trainarray = np.array(imagebag.compute())  # PARALLELIZE\n    trainarray = np.reshape(trainarray, (ims_per_class, -1))    \n    labelarray = np.full((train.shape[0], 1), i)\n    trainarray = np.concatenate((labelarray, trainarray), axis=1)\n    train_grand.append(trainarray)\n    word_label.append(word_label_current.values)\n    \nword_label = np.ravel(word_label) # flatten the labels. It's a janky workaround\ntrain_grand = np.array([train_grand.pop() for i in np.arange(num_classes)]) #less memory than np.concatenate\ntrain_grand = train_grand.reshape((-1, (imheight*imwidth+1)))\ntrain_grand= np.append(train_grand, word_label[:,None],axis=1)\n\ntrainarray = None\ntrain = None\ndel trainarray\ndel train\ngc.collect()\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4d3d0fe6ed905798f6018cbe2b4d90127f4d1ae","_kg_hide-input":true},"cell_type":"code","source":"# memory-friendly alternative to train_test_split?\nvalfrac = 0.1\ncutpt = int(valfrac * train_grand.shape[0])\n\nnp.random.shuffle(train_grand)\nword_array = train_grand[:,-1]\ntrain_grand = np.delete(train_grand,-1,axis=1)\ntrain_grand = train_grand.astype(np.float64)\ny_train, X_train = train_grand[cutpt: , 0], train_grand[cutpt: , 1:]\ny_val, X_val = train_grand[0:cutpt, 0], train_grand[0:cutpt, 1:] #validation set is recognized==True\nword_train, word_val = word_array[cutpt:], word_array[0:cutpt,]\n\ndel train_grand\ny_train_full_label = y_train\ny_train = keras.utils.to_categorical(y_train, num_classes)\nX_train = X_train.reshape(X_train.shape[0], imheight, imwidth, 1)\ny_val_full_label = y_val\ny_val = keras.utils.to_categorical(y_val, num_classes)\nX_val = X_val.reshape(X_val.shape[0], imheight, imwidth, 1)\n\nprint(y_train.shape, \"\\n\",\n      X_train.shape, \"\\n\",\n      y_val.shape, \"\\n\",\n      X_val.shape)\n\nplt.imshow(X_train[0,:,:,0])\nplt.title('Example from class number: {}'.format(numstonames.get([np.where(y_train[0]==1)][0][0][0])))\nplt.axis('off')\nplt.show()\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db84842a1caffdc09c43e855bb78a4a4837af4ce"},"cell_type":"markdown","source":"\n"},{"metadata":{"_uuid":"c8bedcd00ed3f8de8715eca787ee770a51da0920","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Load NN libraries\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers.core import Activation\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.layers.normalization import BatchNormalization\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom sklearn.model_selection import GridSearchCV\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d91cd03df643a8f85bdfef33a11beb2ee4d8d8e3"},"cell_type":"code","source":"def baseline_model(height = 32, width = 3, num_classes = 320):\n    # create model\n    model = Sequential()\n    model.add(Conv2D(32, (5, 5), input_shape=(height, width, 1), activation='relu'))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Dropout(0.2))\n    model.add(Flatten())\n    model.add(Dense(128, activation='relu'))\n    model.add(Dense(num_classes, activation='softmax'))\n    # Compile model\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"14e385a751266d8d62ac248cc3b2fbd4d54a7e64"},"cell_type":"code","source":"def smallVGGnet1(height = 32, width = 3, num_classes = 320):\n\t# initialize the model along with the input shape to be\n\t# \"channels last\" and the channels dimension itself\n\tmodel = Sequential()\n\tdepth = 1\n\tinputShape = (height, width, depth)\n\tchanDim = -1\n\n\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",\n\t\tinput_shape=inputShape))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(3, 3)))\n\tmodel.add(Dropout(0.25))\n\n\t# first (and only) set of FC => RELU layers\n\tmodel.add(Flatten())\n\tmodel.add(Dense(1024))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(0.5))\n\n\t# softmax classifier\n\tmodel.add(Dense(num_classes))\n\tmodel.add(Activation(\"softmax\"))\n\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\t# return the constructed network architecture\n\treturn model\t\t\n\ndef smallVGGnet2(height = 32, width = 3, num_classes = 320, optimizer = 'adam',dropout_rate = 0.5):\n\t# initialize the model along with the input shape to be\n\t# \"channels last\" and the channels dimension itself\n\tmodel = Sequential()\n\tdepth = 1\n\tinputShape = (height, width, depth)\n\tchanDim = -1\n\n\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",\n\t\tinput_shape=inputShape))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(3, 3)))\n\tmodel.add(Dropout(0.25))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(0.25))\n    \n\t# first (and only) set of FC => RELU layers\n\tmodel.add(Flatten())\n\tmodel.add(Dense(1024))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(dropout_rate))\n\n\t# softmax classifier\n\tmodel.add(Dense(num_classes))\n\tmodel.add(Activation(\"softmax\"))\n\tmodel.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n\t# return the constructed network architecture\n\treturn model\t\t\n\ndef smallVGGnet3(height = 32, width = 3, num_classes = 320):\n\t# initialize the model along with the input shape to be\n\t# \"channels last\" and the channels dimension itself\n\tmodel = Sequential()\n\tdepth = 1\n\tinputShape = (height, width, depth)\n\tchanDim = -1\n\n\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",\n\t\tinput_shape=inputShape))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(3, 3)))\n\tmodel.add(Dropout(0.25))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(0.25))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(0.25))\n\t# first (and only) set of FC => RELU layers\n\tmodel.add(Flatten())\n\tmodel.add(Dense(1024))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(0.5))\n    \n\t# softmax classifier\n\tmodel.add(Dense(num_classes))\n\tmodel.add(Activation(\"softmax\"))\n\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\t# return the constructed network architecture\n\treturn model\t\t","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22715cdea881b766445706be58d663425cb34a2d","_kg_hide-input":true},"cell_type":"code","source":"# Functions for plotting loss and accuracy\ndef plotAccuracy(History):\n    '''\n    Plots the accuracy of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt = the plt handle of the figure. Used for saving the plot\n    '''\n    fig1, ax_acc = plt.subplots()\n    plt.plot(History.history['acc'])\n    plt.plot(History.history['val_acc'])\n    plt.xlabel('Epoch')\n    plt.ylabel('Accuracy')\n    plt.title('Model - Accuracy')\n    plt.legend(['Training', 'Validation'], loc='lower right')\n    return plt\n\ndef plotLoss(History):\n    '''\n    Plots the loss of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt = the plt handle of the figure. Used for saving the plot  \n    '''\n    fig2, ax_loss = plt.subplots()\n    plt.xlabel('Epoch')\n    plt.ylabel('Loss')\n    plt.title('Model- Loss')\n    plt.legend(['Training', 'Validation'], loc='upper right')\n    plt.plot(History.history['loss'])\n    plt.plot(History.history['val_loss'])\n    plt.legend(['Training', 'Validation'], loc='lower right')\n    return plt\n\ndef plotHistory(History):\n    '''\n    Plots the accuracy and validation of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt_acc = the plt handle of the accuracy figure. Used for saving the plot\n        plt_val = the plt handle of the validation figure. Used for saving the plot\n    '''    \n    plt_acc = plotAccuracy(History)\n    plt_val = plotLoss(History)\n    return plt_acc,plt_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2100846415e4dad1300a55e9aeccd9fad94125f5","_kg_hide-output":true},"cell_type":"code","source":"from sklearn.cross_validation import cross_val_score\nfrom keras.wrappers.scikit_learn import KerasClassifier\n\n#model = baseline_model(imheight,imwidth,num_classes)\n_CV_EPOCH = 30\n_CV_BATCH_SIZE = 100\n'''\nmodel = KerasClassifier(build_fn=baseline_model,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=_CV_BATCH_SIZE, verbose=0)\nscores = cross_val_score(model, X_train, y_train, cv=5)\nprint('Mean of baseline CNN %.2f with +- %.2f std' % (np.mean(scores),np.std(scores)))\n\nmodel = KerasClassifier(build_fn=smallVGGnet1,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=_CV_BATCH_SIZE, verbose=0)\nscores = cross_val_score(model, X_train, y_train, cv=5)\nprint('Mean of 1 block small VGG %.2f with +- %.2f std' % (np.mean(scores),np.std(scores)))\n\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=_CV_BATCH_SIZE, verbose=0)\nscores = cross_val_score(model, X_train, y_train, cv=5)\nprint('Mean of 2 block small VGG %.2f with +- %.2f std' % (np.mean(scores),np.std(scores)))\n\nmodel = KerasClassifier(build_fn=smallVGGnet3,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=_CV_BATCH_SIZE, verbose=0)\nscores = cross_val_score(model, X_train, y_train, cv=5)\nprint('Mean of all 3 block small VGG %.2f with +- %.2f std' % (np.mean(scores),np.std(scores)))\n\nmodel = None\nscores = None\ndel model\ndel scores\ngc.collect()\n\nallDone()\n'''\n#Mean of baseline CNN 0.39 with +- 0.02 std\n#Mean of 1 block small VGG 0.50 with +- 0.06 std\n#Mean of 2 block small VGG 0.57 with +- 0.03 std\n#Mean of all 3 block small VGG 0.45 with +- 0.05 std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ed74a44e1ab9f2a86cd4ccc9170d3f181e72d2de"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n'''\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=_CV_BATCH_SIZE, verbose=0)\n\n# define the grid search parameters\nbatch_size = [10, 20, 50, 100, 200, 400]\nparam_grid = dict(batch_size=batch_size)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\nmodel = None\ngrid = None\ngrid_result = None\ndel model\ndel grid\ndel grid_result\ngc.collect()\n\nallDone()\n'''\n#Best: 0.576333 using {'batch_size': 10}\n#0.576333 (0.031041) with: {'batch_size': 10}\n#0.564222 (0.021351) with: {'batch_size': 20}\n#0.567556 (0.015478) with: {'batch_size': 50}\n#0.565556 (0.015110) with: {'batch_size': 100}\n#0.551333 (0.010965) with: {'batch_size': 200}\n#0.524556 (0.005724) with: {'batch_size': 400}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"81bada10ed84622036804fc5476fc4110c3ef0eb"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n'''\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_CV_EPOCH, batch_size=100, verbose=0)\n\n# define the grid search parameters\noptimizer = ['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam']\nparam_grid = dict(optimizer=optimizer)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\ndel model\ndel grid\ndel grid_result\nallDone()\n'''\n#Best: 0.579667 using {'optimizer': 'Adam'}\n#0.354444 (0.020531) with: {'optimizer': 'SGD'}\n#0.511444 (0.057758) with: {'optimizer': 'RMSprop'}\n#0.553778 (0.023892) with: {'optimizer': 'Adagrad'}\n#0.546222 (0.005814) with: {'optimizer': 'Adadelta'}\n#0.579667 (0.019530) with: {'optimizer': 'Adam'}\n#0.558111 (0.002183) with: {'optimizer': 'Adamax'}\n#0.545444 (0.021730) with: {'optimizer': 'Nadam'}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7d2ebfb4c345ea70eaefc88b66f01449d4477b80"},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n'''\n_GEN_EPOCH = 20\n_GEN_BATCH_SIZE = 100\nmodel_with_gen1 = smallVGGnet2(imheight,imwidth,num_classes)\nmodel_with_gen2 = smallVGGnet2(imheight,imwidth,num_classes)\n\ndatagen = ImageDataGenerator(\n    horizontal_flip=True)\n\n# This was the other generator that was tried. It reduced the accuracy of about 5%.\ndatagen2 = ImageDataGenerator(\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    horizontal_flip=True) \n\nhistory_gen1 = model_with_gen1.fit_generator(datagen.flow(X_train, y_train, batch_size=_GEN_BATCH_SIZE),steps_per_epoch=len(X_train) / _GEN_BATCH_SIZE, epochs=_GEN_EPOCH,verbose = 0)\nscores_gen1 = model_with_gen1.evaluate(X_val, y_val, verbose=0)\n\nhistory_gen2 = model_with_gen2.fit_generator(datagen2.flow(X_train, y_train, batch_size=_GEN_BATCH_SIZE),steps_per_epoch=len(X_train) / _GEN_BATCH_SIZE, epochs=_GEN_EPOCH,verbose = 0)\nscores_gen2 = model_with_gen2.evaluate(X_val, y_val, verbose=0)\n\nmodel = smallVGGnet2(imheight,imwidth,num_classes)\nhistory = model.fit(X_train, y_train, epochs=_GEN_EPOCH, batch_size=_GEN_BATCH_SIZE, verbose=0)\nscores = model.evaluate(X_val, y_val, verbose=0)\n\n\nprint(\"VGG with flip only Accuracy: %.2f%%\" % (scores_gen1[1]*100))\nprint(\"VGG with rotation, shift and flip Accuracy: %.2f%%\" % (scores_gen2[1]*100))\nprint(\"VGG without augmentation Accuracy: %.2f%%\" % (scores[1]*100))\nallDone()\ndel model_with_gen\ndel history_gen\ndel scores_gen\ndel model\ndel history\n'''\n#VGG with flip only Accuracy: 82.60% \n#VGG with rotation, shift and flip Accuracy: 23.00% \n#VGG without augmentation Accuracy: 82.00%","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dba98ac81a9dc4fd4298f2da691d1e50fe138098","_kg_hide-input":true},"cell_type":"code","source":"# Updating the input to include variable dropout rates\ndef smallVGGnet2(height = 32, width = 3, num_classes = 320, optimizer = 'adam',dropout_rate_last = 0.5, dropout_rate_middle=0.25,init_mode = 'he_normal'):\n\t# initialize the model along with the input shape to be\n\t# \"channels last\" and the channels dimension itself\n\tmodel = Sequential()\n\tdepth = 1\n\tinputShape = (height, width, depth)\n\tchanDim = -1\n\n\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",\n\t\tinput_shape=inputShape,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(3, 3)))\n\tmodel.add(Dropout(dropout_rate_middle))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\",kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(dropout_rate_middle))\n    \n\t# first (and only) set of FC => RELU layers\n\tmodel.add(Flatten())\n\tmodel.add(Dense(1024,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(dropout_rate_last))\n\n\t# softmax classifier\n\tmodel.add(Dense(num_classes,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"softmax\"))\n\tmodel.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n\t# return the constructed network architecture\n\treturn model\t\t","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46c87208c7a62358b463071572f510ed343a333c"},"cell_type":"code","source":"# Looking at the effect of dropout of the last layer\n'''\n_GRID_EPOCH = 20\n_GRID_BATCH_SIZE = 100\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth,\n                        num_classes = num_classes, nb_epoch=_GRID_EPOCH, batch_size=_GRID_BATCH_SIZE, verbose=0)\ndropout_rate_last = [0.1, 0.3, 0.5, 0.7, 0.9]\nparam_grid = dict(dropout_rate_last=dropout_rate_last)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\ndel model\ndel grid\ndel grid_result\nallDone()\n'''\n#Best: 0.601778 using {'dropout_rate_last': 0.3}\n#0.589444 (0.014838) with: {'dropout_rate_last': 0.1}\n#0.601778 (0.009445) with: {'dropout_rate_last': 0.3}\n#0.580000 (0.009361) with: {'dropout_rate_last': 0.5}\n#0.535000 (0.012223) with: {'dropout_rate_last': 0.7}\n#0.428111 (0.026609) with: {'dropout_rate_last': 0.9}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4f6b3dbc754fb16c588044c2b628ea09eb6c6ea"},"cell_type":"code","source":"# Looking at the effect of dropout of the layers after convolution\n'''\n_GRID_EPOCH = 20\n_GRID_BATCH_SIZE = 100\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth,\n                        num_classes = num_classes, nb_epoch=_GRID_EPOCH, batch_size=_GRID_BATCH_SIZE, verbose=0)\ndropout_rate_middle = [0.1, 0.3, 0.5, 0.7, 0.9]\nparam_grid = dict(dropout_rate_middle=dropout_rate_middle)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\ndel model\ndel grid\ndel grid_result\nallDone()\n'''\n#Best: 0.597444 using {'dropout_rate_middle': 0.1}\n#0.597444 (0.005977) with: {'dropout_rate_middle': 0.1}\n#0.552000 (0.009193) with: {'dropout_rate_middle': 0.3}\n#0.428667 (0.027475) with: {'dropout_rate_middle': 0.5}\n#0.199111 (0.021056) with: {'dropout_rate_middle': 0.7}\n#0.049667 (0.006771) with: {'dropout_rate_middle': 0.9}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5dff95b1c471a247860dfdb5a6441dee9557a83","collapsed":true},"cell_type":"code","source":"# Looking at the weight initialization\n'''\n_GRID_EPOCH = 20\n_GRID_BATCH_SIZE = 100\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes, nb_epoch=_GRID_EPOCH, batch_size=_GRID_BATCH_SIZE, verbose=0)\ninit_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']\nparam_grid = dict(init_mode=init_mode)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\nallDone()\n'''\n#Best: 0.608667 using {'init_mode': 'lecun_uniform'}\n#0.382444 (0.015945) with: {'init_mode': 'uniform'}\n#0.608667 (0.018409) with: {'init_mode': 'lecun_uniform'}\n#0.518778 (0.031582) with: {'init_mode': 'normal'}\n#0.044333 (0.001785) with: {'init_mode': 'zero'}\n#0.560000 (0.018799) with: {'init_mode': 'glorot_normal'}\n#0.558444 (0.024736) with: {'init_mode': 'glorot_uniform'}\n#0.578889 (0.014718) with: {'init_mode': 'he_normal'}\n#0.569556 (0.009979) with: {'init_mode': 'he_uniform'}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7921bd3e22389f3db33cee8d8abdee9d28b33f9a"},"cell_type":"code","source":"# Fine search the dropout rates\n'''\n_GRID_EPOCH = 20\n_GRID_BATCH_SIZE = 100\n\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes,\n                        nb_epoch=_GRID_EPOCH, batch_size=_GRID_BATCH_SIZE, verbose=0)\ndropout_rate_last = [0.3, 0.35, 0.40, 0.45, 0.5]\nparam_grid = dict(dropout_rate_last=dropout_rate_last)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\n\n\nmodel = KerasClassifier(build_fn=smallVGGnet2,height = imheight, width = imwidth, num_classes = num_classes,\n                        dropout_rate_last=0.45, nb_epoch=_GRID_EPOCH, batch_size=_GRID_BATCH_SIZE, verbose=0)\ndropout_rate_middle = [0.1, 0.15, 0.20, 0.25, 0.3]\nparam_grid = dict(dropout_rate_middle=dropout_rate_middle)\ngrid = GridSearchCV(estimator=model, param_grid=param_grid)\ngrid_result = grid.fit(X_train, y_train)\n# summarize results\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']\nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\nallDone()\ndel X_train\ndel y_train\ndel X_val\ndel y_val\n'''\n#Best: 0.585667 using {'dropout_rate_last': 0.45}\n#0.556222 (0.021832) with: {'dropout_rate_last': 0.3}\n#0.551667 (0.022283) with: {'dropout_rate_last': 0.35}\n#0.561111 (0.016153) with: {'dropout_rate_last': 0.4}\n#0.585667 (0.005361) with: {'dropout_rate_last': 0.45}\n#0.572667 (0.025692) with: {'dropout_rate_last': 0.5}\n#Best: 0.588778 using {'dropout_rate_middle': 0.1}\n#0.588778 (0.021666) with: {'dropout_rate_middle': 0.1}\n#0.586111 (0.011704) with: {'dropout_rate_middle': 0.15}\n#0.586556 (0.005370) with: {'dropout_rate_middle': 0.2}\n#0.570889 (0.016511) with: {'dropout_rate_middle': 0.25}\n#0.556222 (0.020004) with: {'dropout_rate_middle': 0.3}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"929c5925e3b1eecdf927da2d9c19c9ec8b15bb74"},"cell_type":"markdown","source":"Cross validation takes up too much memory to run all at once, together with the actual network. Therefore the code has been commented out but the results are left at the end of the the code.  Comment in to run it yourself.  The data used to perform cross validation and grid search was a small subset containing 20 classes and 500 images.\n\nOut of the four models that were tested VGG net with only 2 blocks of CNN performs the best at an accuracy of around 57% , which is a bit surprising. The deepest VGG net with 3 block achieved only 45% accuracy. And they all outperformed the baseline.  We will continue optimizing for the small VGGnet that contains 2 blocks. We perform a grid search for different parameters to get an idea of what to tweak on the network. We will test the effect of batch size, percentage of dropout layer, initial weights and optimizer.\n\nIt looks like a batch size between around between 10 to 200 seems to work the best. Since a too small or too large batch size can lead to over/under fitting it will probably be a good idea to go with a 100 as it is somewhere in the middle.\n\nThere doesn't seem to be that much difference between the optimizers, except SGD which performed about 10% worse than the rest.  Adam seems to perform the best so we will go further with this.\n\nWe also tried to see what result data augmentation has on the accuracy. Adding sheer, flip and rotation dropped the accuracy down to 23%. Horizontal flip increased it by 0.6% which is not a lot. But it could be argued that by adding horizontal flip we're adding more robustness to the network as people don't always draw the images from the same side so flipping would naturally occur. \n\nWe checked if weight initialization players a significant role in this network. There seems to be a big difference between the different weight initialization with zero and uniform being the bottom at 4.3% and 38.2% accuracy respectivly. The rest seem the all very close to each other as they range from 56% to 61.9%. lecun_uniform will be used as weight initialization since it scored the highest.\n\nIn regards to dropout rate we searched through the performence of the layers after convolution (middle layers) and the dense layers (the last layer). It looked like the best performence is found in the 0.3-0.5 dropout rate for the last dense layer and between 0.1 and 0.3 in the dropout after convolutions. A closer search of the values between those showed that there wasn't much difference between them. Therefore choosing the 0.5 for last layer and 0.3 for the last layer is the more conservative choice and should translate better when using all the classes.\n\nThe following hyperparameters that performed the best were the following:\n* Optimizer = Adam\n* Initial weights = lecun_uniform\n* Batch size = 100\n* Dropout_middle = 0.3\n* Dropout_last = 0.5\n\n\n\n# Final network\nTo see how it all comes together the network will train on the full data base. A small subset will be split into validation data to see how well it perform. The only time the validation data has been used was during optimizing the network was when seeing how data augmentation affected the accuracy. "},{"metadata":{"trusted":true,"_uuid":"e22ef73089c323fa20965521eea6972dbb1b910f"},"cell_type":"code","source":"#%% import\nimport os\nfrom glob import glob\nimport re\nimport ast\nimport numpy as np \nimport pandas as pd\nfrom PIL import Image, ImageDraw \nfrom tqdm import tqdm\nfrom dask import bag\nimport matplotlib.pyplot as plt\nimport keras\nimport gc\n\n%matplotlib inline\n## Import up sound alert dependencies\nfrom IPython.display import Audio, display\n\ndef allDone():\n  display(Audio(url='https://sound.peal.io/ps/audios/000/000/537/original/woo_vu_luvub_dub_dub.wav', autoplay=True))\n## Insert whatever audio file you want above\n\n# Load NN libraries\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers.core import Activation\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.layers.normalization import BatchNormalization\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.metrics import top_k_categorical_accuracy\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import confusion_matrix, classification_report\nfrom keras.preprocessing.image import ImageDataGenerator\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c30bee2ee56d884d03e122d181051c7833df3927"},"cell_type":"code","source":"#%% set label dictionary and params\nclassfiles = os.listdir('../input/train_simplified/')\nnumstonames = {i: v[:-4].replace(\" \", \"_\") for i, v in enumerate(classfiles)} #adds underscores\n\nnum_classes = 340    #340 max \nimheight, imwidth = 32, 32  \nims_per_class = 500  # 1000 kills the kernel. 500 images are able to fit in the memory together with the rest of the code\n\n# faster conversion function\ndef draw_it(strokes):\n    image = Image.new(\"P\", (256,256), color=255)\n    image_draw = ImageDraw.Draw(image)\n    for stroke in ast.literal_eval(strokes):\n        for i in range(len(stroke[0])-1):\n            image_draw.line([stroke[0][i], \n                             stroke[1][i],\n                             stroke[0][i+1], \n                             stroke[1][i+1]],\n                            fill=0, width=5)\n    image = image.resize((imheight, imwidth))\n    return np.array(image)/255.\n\n# Workaround to load test data without crashing kernal\ntest_df = pd.read_csv('../input/test_simplified.csv', index_col=['key_id'])\nimage_series=test_df['drawing']\ntest_array = []\nfor row in range(0,len(image_series)):\n    #import pdb; pdb.set_trace()\n    image = image_series.iloc[row]\n    image = np.array(draw_it(image))\n    image = np.reshape(image, image.shape + (1,))\n    test_array.append(image)\ntest_df['image'] = test_array\n\ndel image_series, image\ngc.collect()\n\n#%% get train arrays\ntrain_grand = []\nclass_paths = glob('../input/train_simplified/*.csv')\nword_label = []\nfor i,c in enumerate(tqdm(class_paths[0: num_classes])):\n    train = pd.read_csv(c, usecols=['drawing', 'recognized','word'], nrows=ims_per_class*5//4)\n    train = train[train.recognized == True].head(ims_per_class)\n    word_label_current = train['word'].replace(' ', '_', regex=True)\n    imagebag = bag.from_sequence(train.drawing.values).map(draw_it) \n    trainarray = np.array(imagebag.compute())  # PARALLELIZE\n    trainarray = np.reshape(trainarray, (ims_per_class, -1))    \n    labelarray = np.full((train.shape[0], 1), i)\n    trainarray = np.concatenate((labelarray, trainarray), axis=1)\n    train_grand.append(trainarray)\n    word_label.append(word_label_current.values)\n    \nword_label = np.ravel(word_label) # flatten the labels. It's a janky workaround\nword_encoder = LabelEncoder()\nword_encoder.fit(word_label)\ntrain_grand = np.array([train_grand.pop() for i in np.arange(num_classes)]) #less memory than np.concatenate\ntrain_grand = train_grand.reshape((-1, (imheight*imwidth+1)))\ntrain_grand= np.append(train_grand, word_label[:,None],axis=1)\n\ndel trainarray\ndel train\n\n# memory-friendly alternative to train_test_split?\nvalfrac = 0.1\ncutpt = int(valfrac * train_grand.shape[0])\n\nnp.random.shuffle(train_grand)\nword_array = train_grand[:,-1]\ntrain_grand = np.delete(train_grand,-1,axis=1)\ntrain_grand = train_grand.astype(np.float64)\ny_train, X_train = train_grand[cutpt: , 0], train_grand[cutpt: , 1:]\ny_val, X_val = train_grand[0:cutpt, 0], train_grand[0:cutpt, 1:] #validation set is recognized==True\nword_train, word_val = word_array[cutpt:], word_array[0:cutpt,]\n\ndel train_grand\ny_train_full_label = y_train\ny_train = keras.utils.to_categorical(y_train, num_classes)\nX_train = X_train.reshape(X_train.shape[0], imheight, imwidth, 1)\ny_val_full_label = y_val\ny_val = keras.utils.to_categorical(y_val, num_classes)\nX_val = X_val.reshape(X_val.shape[0], imheight, imwidth, 1)\n\nprint(y_train.shape, \"\\n\",\n      X_train.shape, \"\\n\",\n      y_val.shape, \"\\n\",\n      X_val.shape)\n\nplt.imshow(X_train[0,:,:,0])\nplt.title('Example from class number: {}'.format(numstonames.get(np.where(y_train[0]==1)[0][0])))\nplt.axis('off')\n\nplt.show()\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4eefd3eb9984c4d8b02b2f6c2623c2d402ff36a4"},"cell_type":"code","source":"def top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)\n\ndef smallVGGnet2(height = 32, width = 3, num_classes = 320, optimizer = 'adam',dropout_rate_last = 0.35, dropout_rate_middle=0.25, init_mode = 'lecun_uniform'):\n\t# initialize the model along with the input shape to be\n\t# \"channels last\" and the channels dimension itself\n\tmodel = Sequential()\n\tdepth = 1\n\tinputShape = (height, width, depth)\n\tchanDim = -1\n\n\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",\n\t\tinput_shape=inputShape,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(3, 3)))\n\tmodel.add(Dropout(dropout_rate_middle))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\",kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(dropout_rate_middle))\n\t# (CONV => RELU) * 2 => POOL\n\tmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization(axis=chanDim))\n\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\tmodel.add(Dropout(dropout_rate_middle))    \n\t# first (and only) set of FC => RELU layers\n\tmodel.add(Flatten())\n\tmodel.add(Dense(1024,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"relu\"))\n\tmodel.add(BatchNormalization())\n\tmodel.add(Dropout(dropout_rate_last))\n\n\t# softmax classifier\n\tmodel.add(Dense(num_classes,kernel_initializer=init_mode))\n\tmodel.add(Activation(\"softmax\"))\n\tmodel.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy', top_3_accuracy])\n\t# return the constructed network architecture\n\treturn model\t\t\n\n# Functions for plotting loss and accuracy\ndef plotAccuracy(History):\n    '''\n    Plots the accuracy of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt = the plt handle of the figure. Used for saving the plot\n    '''\n    fig1, ax_acc = plt.subplots()\n    plt.plot(History.history['acc'])\n    plt.plot(History.history['val_acc'])\n    plt.xlabel('Epoch')\n    plt.ylabel('Accuracy')\n    plt.title('Model - Accuracy')\n    plt.legend(['Training', 'Validation'], loc='lower right')\n    return plt\n\ndef plotLoss(History):\n    '''\n    Plots the loss of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt = the plt handle of the figure. Used for saving the plot  \n    '''\n    fig2, ax_loss = plt.subplots()\n    plt.xlabel('Epoch')\n    plt.ylabel('Loss')\n    plt.title('Model- Loss')\n    plt.legend(['Training', 'Validation'], loc='upper right')\n    plt.plot(History.history['loss'])\n    plt.plot(History.history['val_loss'])\n    plt.legend(['Training', 'Validation'], loc='lower right')\n    return plt\n\ndef plotHistory(History):\n    '''\n    Plots the accuracy and validation of the training and validation of the model\n    \n    input:\n        History = the history of the model.\n    \n    output:\n        plt_acc = the plt handle of the accuracy figure. Used for saving the plot\n        plt_val = the plt handle of the validation figure. Used for saving the plot\n    '''    \n    plt_acc = plotAccuracy(History)\n    plt_val = plotLoss(History)\n    return plt_acc,plt_val\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30f460ff0b81f944c622285fa96a4dbfb8460b9d","_kg_hide-input":false},"cell_type":"code","source":"_EPOCH = 50\n_BATCH_SIZE = 100\n_WEIGHT_INI = 'lecun_uniform' \n_DROPOUT_RATE_LAST = 0.5\n_DROPOUT_RATE_MIDDLE = 0.3\n\nmodel = smallVGGnet2(height = imheight, width = imwidth, num_classes = num_classes,\n                     dropout_rate_last=_DROPOUT_RATE_LAST,dropout_rate_middle=_DROPOUT_RATE_MIDDLE,\n                    init_mode = _WEIGHT_INI)\ndatagen = ImageDataGenerator(horizontal_flip=True)\n#datagen = ImageDataGenerator()\n\nearlystop = EarlyStopping(monitor='val_top_3_accuracy', mode='auto', patience=5) \ncallbacks = [earlystop]\n\nhistory = model.fit_generator(datagen.flow(X_train, y_train, batch_size=_BATCH_SIZE),\n                              validation_data=(X_val,y_val),\n                              callbacks=callbacks,steps_per_epoch=len(X_train) / _BATCH_SIZE, epochs=_EPOCH,verbose = 0)\n#history = model.fit(X_train, y_train, batch_size=_BATCH_SIZE,\n#                   validation_data=(X_val,y_val),\n#                   callbacks=callbacks,\n#                   epochs=_EPOCH,verbose = 0)\nscores = model.evaluate(X_val, y_val, verbose=0)\n\nprint(\"Optimized VGG Accuracy on validation set: %.2f%%, Top 3 Accuracy: %.2f%%\" % (scores[1]*100,scores[2]*100))\n\nplotHistory(history)\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"28b86203eea0bf37b3549a41483e8fd63c60dc16"},"cell_type":"code","source":"from io import StringIO\nimport re\ndef report_to_df(report):\n    report = re.sub(r\" +\", \" \", report).replace(\"avg / total\", \"avg/total\").replace(\"\\n \", \"\\n\")\n    report_df = pd.read_csv(StringIO(\"Classes\" + report), sep=' ', index_col=0)        \n    return(report_df)\n\n\ntest_cat = np.argmax(y_val, 1)\npred_y = model.predict(X_val)\npred_cat = np.argmax(pred_y, 1)\n#plt.matshow(confusion_matrix(test_cat, pred_cat))\ntemp = classification_report(test_cat, pred_cat, \n                           target_names = [x for x in numstonames.values()])\nclassification_df = report_to_df(temp)\n\nn_best_classes = classification_df.nlargest(10,'f1-score').reset_index()\nn_worst_classes = classification_df.nsmallest(10,'f1-score').reset_index()\nprint(n_best_classes)\nprint(n_worst_classes)\nworst_classes_labels = []\nfor class_number, class_word in numstonames.items():    \n    if class_word in n_worst_classes['Classes'].values:\n        worst_classes_labels.append(class_number)\n\nplt.figure(figsize=(10,10))\nfor x,_class in enumerate(worst_classes_labels[0:5]):\n    plt.subplot(3,2,x+1)\n    worst_class_array_loc = np.where(test_cat==_class)\n    wrong_prediction = np.where(test_cat[worst_class_array_loc]!=pred_cat[worst_class_array_loc])\n    wrong_img_loc = wrong_prediction[0][0] # first wrong prediction\n    plt.imshow(X_val[worst_class_array_loc][wrong_img_loc,:,:,0])\n    plt.axis('off')\n    plt.title('Real class: {}\\n Predicted class: {}'.format(numstonames.get(test_cat[worst_class_array_loc][wrong_img_loc]),\n                                                            numstonames.get(pred_cat[worst_class_array_loc][wrong_img_loc])))\nplt.show()\n\nbest_classes_labels = []\nfor class_number, class_word in numstonames.items():   \n    if class_word in n_best_classes['Classes'].values:\n        best_classes_labels.append(class_number)\nplt.figure(figsize=(10,10))\nfor x,_class in enumerate(best_classes_labels[0:5]):\n    plt.subplot(3,2,x+1)\n    best_class_array_loc = np.where(test_cat==_class)\n    right_prediction = np.where(test_cat[best_class_array_loc]==pred_cat[best_class_array_loc])\n    right_img_loc = right_prediction[0][0] # first wrong prediction\n    #print(right_img_loc)\n    plt.imshow(X_val[best_class_array_loc][right_img_loc,:,:,0])\n    plt.axis('off')\n    plt.title('Real class: {}\\n Predicted class: {}'.format(numstonames.get(test_cat[best_class_array_loc][right_img_loc]),\n                                                            numstonames.get(pred_cat[best_class_array_loc][right_img_loc])))    \n    \nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c05c910857d15b518e5b903bd5acb1da21c6d7e","_kg_hide-input":true},"cell_type":"code","source":"\nX_train = None\ny_train = None\nX_val = None\ny_val = None\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"03ddb769cd2d12447384f1843d3641343bbf7c33","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"ttvlist = []\ntest_array = test_df['image']\nfor test_image in test_array:\n    test_image = np.expand_dims(test_image, axis=0)\n    testpreds = model.predict(test_image, verbose=0)\n    ttvs = np.argsort(-testpreds)[:, 0:3]  # top 3\n    ttvlist.append(ttvs)\n    \nttvarray = np.concatenate(ttvlist)\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f350de503b366c7a5b941adfbde146b2b923cd08","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"preds_df = pd.DataFrame({'first': ttvarray[:,0], 'second': ttvarray[:,1], 'third': ttvarray[:,2]})\npreds_df = preds_df.replace(numstonames)\npreds_df['words'] = preds_df['first'] + \" \" + preds_df['second'] + \" \" + preds_df['third']\n\nsub = pd.read_csv('../input/sample_submission.csv', index_col=['key_id'])\nsub['word'] = preds_df.words.values\nsub.to_csv('small_VGG.csv')\nsub.head\nallDone()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a20aad373bdd833ec87e3d855c01c7ab214cb9d"},"cell_type":"markdown","source":"The optimized small VGG with 2 CNN blocks network has an accuracy of 57.69% when using all 340 classes with  500 images per class. 10% were used for validation. The top 3 accuracy is 77.98% and that is what this competition uses as the submission score. We can use F1-score to get a measurement of how well it classifies a certain class as F1 contains both precision and recall. When looking at the top 10 missclassified classes and examples of where the network fails to classify correctly we can see that the mistakes it makes are reasonable. Like identifying a bear as a mouse or a marker as a candle. Those can be hard to classify even for a human. The top 10 best classes on the other hand seem rather distinctable and the examples of correct classification look easy to classify with the human eye. \n\nTo see how well the network performs on the test set provided by kaggle we submit the top 3 prediction of the test set. We get a top 3 accuracy of around 59% which is substantially lower than the validation set.  There can be multiple reasons for this. The network is overfitting to the training data which can be seen on the accuracy and loss plots. There seems to be around 10% difference in train and validation accuracy. But this doesn't explain why the top 3 accuracy of the test set is around 18% less that the top 3 validation set. Since we don't know the distribution of classes in the test set one explanation could be that there are more images from classes that the network struggles to classify. This probably unlikely but cannot really be investigated as long as the test set distribution is unknown.  Out of curiosity the VGG with 3 CNN blocks was also run on the test data. The hyper parameters were the same as previous. This actually performed better and had an accuracy of 62% on the test data. It looks like even though it performed badly with only 20 classes, when extending to 340 classes the deeper network deeper network learned better features. "},{"metadata":{"_uuid":"5b0d99a90f6721110774f31dda0e69ff0d751ebd"},"cell_type":"markdown","source":"# Experiences with this Kaggle Competition\nIt took some time to figure out how a kaggle kernel actually works because a lot of the things are implied or assumed that the user already knows. But once your kernel is up it is easy to prototype in it. Some of the pros of kaggle are that there are a lot of kernels available so you can copy some of the trivial stuff like loading the data or submitting the data. The biggest pro is without a doubt that you have very easy access to a good GPU. The biggest con of kaggle is that for me the kernel died quite often making me lose my work and having to start over. I tried different ways to clear the RAM (deleting variables, setting them to None and using garbage collector) but I could not clear it. That is why I had to comment out my gridsearches to be able to commit my kernel. Jupyter also lacks a variable explorer, debug tool and other small things that an IDE like spyder has which makes the workflow much easier. Over all I found the Kaggle platform and competition as an interesting experience and could see my self doing the competitions on a sideproject level after my thesis.\n"},{"metadata":{"_uuid":"096db6c3a447543e405e1371529d767949b9aadf"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"5b12f60d3992203bdc6f54735f6b15d307cd1116","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}