{"cells":[{"metadata":{"_uuid":"b787e4a0ce54ea38ae1c10fee2a75b135d368a11"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"5b12f60d3992203bdc6f54735f6b15d307cd1116"},"cell_type":"markdown","source":"# Sketch-Buster 9000 \n### Even Though I have commited this code a couple of times, the results does not seem to update unfortunately, and I don't know why. It runs fine sometimes in the Interactive Session however\n\nBy me and with help from some nice public kernels around. "},{"metadata":{"trusted":true,"_uuid":"85fa10a42b5ae09d4259f41720e8d9ad2fd0ae41","_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\nimport pandas as pd\nimport numpy as np\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport datetime as dt\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport ast\nsns.set_palette(sns.color_palette('copper', 20))\nfrom datetime import date, timedelta\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfc74b0c36c8c8a472009fcb247e7f8fafb5bdf6"},"cell_type":"markdown","source":"# Exercise 1\nCounting classes and amount of data\nTakes a long time to run, but it prints: 49.707.579 total counts\n340 classes "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport os, os.path\nimport numpy as np\ntrainPath = '../input/train_simplified/'\ntrainClasses = os.listdir(trainPath)\namountClasses = len([name for name in trainClasses])\nprint('Amount of classes: ' + str(amountClasses))\n\"\"\"\ntotalCount = 0\nfor i in range(amountClasses):\n    df = pd.read_csv(trainPath + trainClasses[i])\n    count = len(df.index)\n    totalCount += count\n    print('Progress: ' + str(i/amountClasses))\n    \nprint('Total Count of classes:' + str(totalCount) + 'Count per class: ' + str(count)) \nTakes a long time to run, but it prints: 49.707.579 total counts and 116677 counts per class\nThis is not correct (as 116677*340 != 49707579) and must mean that the data is not evenly distributed between classes, so the question\n\"how many images per class\" will vary depending on what class we're talking about\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9820e92143f0248e4f5edf230c4f533aba27ed3f"},"cell_type":"code","source":"import gc\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6931d467a45c53ba38b6203f00fc5fee8d58f2f9"},"cell_type":"markdown","source":"Showing some example images down below: (takes a long time to process)"},{"metadata":{"trusted":true,"_uuid":"cc337c00d0509b1b97fc27eb93da003c4e5db2d5"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport ast\nimport json\n\n\nfor i in range(0, 8):\n    df = pd.read_csv(trainPath + trainClasses[i])\n    df['timestamp'] = pd.to_datetime(df.timestamp)\n    df.sort_values(by='timestamp', ascending=False)[-20:]\n    df['drawing'] = df['drawing'].apply(json.loads)\n    df.head()\n    \n    n = 10\n    j = 0\n    fig, axs = plt.subplots(nrows=5, ncols=4, sharex=True, sharey=True)\n    for j, drawing in enumerate(df.drawing):\n        if j//5 == 5: break\n        ax = axs[j // 5, j % 4]\n        for x, y in drawing:\n            ax.plot(x, -np.array(y), lw = 3);\n        ax.axis('off')\n        \n    plt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c492fb567fbfdbb886485386933cf4da5f73271"},"cell_type":"markdown","source":"# Considerations about data distributions:\n\n1. Drawings are placed differently on the canvas, meaning we should augment some data to train the network to handle translations, scales and rotations\n2. Data is not evenly distributed in the classes so we should avoid overfitting to the classes that are overrepresented by using regularzation\n3. Colours are used differently in each sample, one should consider converting to B/W to only get the shape. This has the drawback that colour sometimes emphasises what we draw and then we cant use that cue anymore.\n4. We have two types of data: simplified and raw. Raw will contain more information, but also noise. Simple will be much faster to train and still have the important and sensible parts\nof the drawing. Maybe using a combination of the two will be a good idea to increase accuracy? \n\n\n## MY NETWORK: \nI have chosen to convert the drawing to black and white images using the simplified data. This should help with training time and remove any noise there might have been introduced in the raw data. \nThe way I do it is starting with a blank canvas and then draw a line from point to point. This means that the line will also be straight and the edges will be sharp. \nTo speed up this process I use enumerates so that I can parallel the drawing process. \nI only use the drawings that has been recognized, as the other drawings would probably be invalid. \n\nThe training data is gathered in one large array that is then shuffled. \nI am splitting up my data to that I use 10 percent of it for validation and the rest for training the neural network \n\n\n"},{"metadata":{"_uuid":"a1ca9c950d3860e88f2529806fbfe228e171acf3"},"cell_type":"markdown","source":"# Exercise 2 "},{"metadata":{"trusted":true,"_uuid":"ac08165760c73d6253993c3f2b906455d6712304"},"cell_type":"code","source":"#%% import\nimport os\nfrom glob import glob\nimport re\nimport ast\nimport numpy as np \nimport pandas as pd\nimport json\nfrom PIL import Image, ImageDraw \nfrom tqdm import tqdm\nfrom dask import bag\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, LeakyReLU, BatchNormalization\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D\nfrom tensorflow.keras.metrics import top_k_categorical_accuracy\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\n\n#%% set label dictionary and params\nclassfiles = os.listdir('../input/train_simplified/')\nnumstonames = {i: v[:-4].replace(\" \", \"_\") for i, v in enumerate(classfiles)} #adds underscores\n\nnum_classes = 340    #340 max \nimheight, imwidth = 32, 32  \nims_per_class = 2000  #max - can always be increased later if we want to train with more images\n\n# faster conversion function since we are using the simplified data\ndef draw_it(strokes):\n    image = Image.new(\"P\", (256,256), color=255)\n    image_draw = ImageDraw.Draw(image)\n    for stroke in json.loads(strokes):\n        for i in range(len(stroke[0])-1):\n            image_draw.line([stroke[0][i], \n                             stroke[1][i],\n                             stroke[0][i+1], \n                             stroke[1][i+1]],\n                            fill=0, width=5)\n    image = image.resize((imheight, imwidth))\n    return np.array(image)/255.\n\n#%% get train arrays from the zip file\ntrain_grand = []\nclass_paths = glob('../input/train_simplified/*.csv')\nfor i,c in enumerate(tqdm(class_paths[0: num_classes])): #Basically saying that we want to run this 2000 times for 340 classess\n    train = pd.read_csv(c, usecols=['drawing', 'recognized'], nrows=ims_per_class*5//4)\n    train = train[train.recognized == True].head(ims_per_class)\n    imagebag = bag.from_sequence(train.drawing.values).map(draw_it) \n    trainarray = np.array(imagebag.compute())  # PARALLELIZING USING DASK.BAG.COMPUTE (essentially converting to np array)\n    trainarray = np.reshape(trainarray, (ims_per_class, -1))    \n    labelarray = np.full((train.shape[0], 1), i)\n    trainarray = np.concatenate((labelarray, trainarray), axis=1)\n    train_grand.append(trainarray)\n    \ntrain_grand = np.array([train_grand.pop() for i in np.arange(num_classes)]) #less memory than np.concatenate\ntrain_grand = train_grand.reshape((-1, (imheight*imwidth+1)))\n\ndel trainarray # Clearing some memory right here\ndel train\n\n# splitting up training data\nvalfrac = 0.1  #using 10 percent for validation\ncutpt = int(valfrac * train_grand.shape[0]) #Which means we must use 90% of the data \n\nnp.random.shuffle(train_grand) #shuffling the data around\ny_train, X_train = train_grand[cutpt: , 0], train_grand[cutpt: , 1:]  #Splitting it up into two arrays; 612000 instances of 340 classes, and 612000 instances of 32x32 images\ny_val, X_val = train_grand[0:cutpt, 0], train_grand[0:cutpt, 1:] #validation set is recognized==True\n\ndel train_grand #Removing the large grand array for memory management\n\ny_train = keras.utils.to_categorical(y_train, num_classes) #y contain the possible classes (output)\nX_train = X_train.reshape(X_train.shape[0], imheight, imwidth, 1) #x is the image itself (input)\ny_val = keras.utils.to_categorical(y_val, num_classes) #same\nX_val = X_val.reshape(X_val.shape[0], imheight, imwidth, 1)\n\nprint(y_train.shape, \"\\n\",\n      X_train.shape, \"\\n\",\n      y_val.shape, \"\\n\",\n      X_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b69552990a796adea47a423260a4ff0449f8b337"},"cell_type":"markdown","source":"I have created my model more or less from trial and error. This means setting the epoch low to a start, running it, changing a variable, and the running it again.\nI create my sequential model. I use a standard CNN and I try not to make it too deep, as the drawings should be relatively simple when they have been drawn from skratch in B/W. It also helps with training time when it is not too deep.\nI use the activation function RELU , as it turned out that it was the one that performed the best. I have tried sigmoid, linear and leakyRELU, however leakyRELU I found to give better initial results, but performing worse in the higher epochs. \n\nI use max pooling after each convolutional layer, as I follow the standard CNN architecture. I have experiemented with the size of the max pooling kernel, but 2x2  performs best. This might be because of data is sparse in the drawings, since they only consist of thin lines and sharp edges. \n\nAnother layer is added that is more than  twice as big as the previous, also with the relu activation. I increased the size of the second layer as I wanted to try to force more features to be detected. With the assumption that simplified data removes noise, that means that the neurons should only learn the actual features of the drawing. \n\nAnother pooling layer \n\nThen I dropout 10 percent of the hidden units to try and force learning upon the other 90 percent. This helps prevent overfitting. I have experimented with different values here, and it does not change the ending result too much. \n\nThen I flatten the model, so that the dimensionality is correct for the dense layer \n\nA densely connected layer that is double the amount of possible classes then connects the neurons followed by a 75 percent dropout that again randomly prevents overfitting. \n\nThe last dense layer is then connected, this one containing 340 neurons, and with the  softmax activation function since we are dealing with a classification problem "},{"metadata":{"trusted":true,"_uuid":"2b88ba95e3042aaf92cb864ba6dcf91d638b6fdd"},"cell_type":"code","source":"\n\n#Constructing the actual CNN \nmodel = Sequential()\n\nmodel.add(Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', input_shape=(imheight, imwidth, 1)))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\n\nmodel.add(Dense(num_classes*2, activation='relu'))\nmodel.add(Dropout(0.75))\n\nmodel.add(Dense(num_classes, activation='softmax'))\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e175ce341ee201041d6266b638e8c848d5dbbf64"},"cell_type":"markdown","source":"Here I define some callbacks and some rules, along with the learning rate and optimizer. \nComments in code describe what I am doing. \n"},{"metadata":{"trusted":true,"_uuid":"5957828ea55f668a878af7d83cd24ab2db4f8c51","scrolled":true},"cell_type":"code","source":"def top_3_accuracy(x,y): \n    t3 = top_k_categorical_accuracy(x,y, 3)\n    return t3\n\n#Callbacks: \n#ReduceLROnPlateau reduces the learning rate when a parameter has stopped improving: In this case, if it hasn't seen significant improvement for 3 batches\n# the learning rate is reduced by a factor 0.5 down to a minimum of 0.0001 learning rate\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3, \n                                   verbose=1, mode='autow', min_delta=0.005, cooldown=5, min_lr=0.0001)\n#Stops traning when top 3 accuracy has stopped increasing for a total of 5 epochs \nearlystop = EarlyStopping(monitor='val_top_3_accuracy', mode='max', patience=5) \n#Setting the actual callbacks\ncallbacks = [reduceLROnPlat, earlystop]\n\n#Compiling the mdel with cross entropy as performance measurer, adam optimizer, and with the top 3 accuracy metrics\nmodel.compile(loss='categorical_crossentropy',\n              optimizer='Adam',\n              metrics=['accuracy', top_3_accuracy])\n\nhists = []\n\n#Training the model with the parameters set above. \nhist = model.fit(x=X_train, y=y_train,\n          epochs = 40,\n          batch_size=450,\n          validation_data = (X_val, y_val),\n          callbacks = callbacks,\n          verbose = 1)\n\nhists.append(hist)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a0555462a64f8eda0e878dde96cc1f731798516","trusted":true},"cell_type":"markdown","source":"- I found that including more epochs gave a gradual increase until a certain point where the increase was so small that it was not worth it. \n- Kaggle seems to crash and not continiue working at ~30 epochs. (entire page just vanishes, when refreshing everything is reset)  \n- The max I have gotten is 78.68% using the adam optimizer, where other optimizers such as Adamax gave high initial accuracy, Adam ends with the highest. \n- Batch size  improves the initial results and the speed of the training significantly but results in a lower validation accuracy then if I choose a lower value. The model converges pretty quickly around 0.55\n- Increasing the dropout rate from 0.5 to 0.75 seems to increase the accuracy quite a lot so I did the same thing with the first dropout. This, again, seems to increase the accuracy. This makes sense, as we force neurons to learn by deactivating some of them randomly. \n- The best result I could get with this relatively small network is between 77-79 %, depending on some of the above discussed hyperparameters\n"},{"metadata":{"trusted":true,"_uuid":"2db495eba0b7331de5387425e361f47ac714ba87"},"cell_type":"code","source":"from tensorflow.keras.metrics import categorical_accuracy\n\n#creating a graph of the training: \nhist_df = pd.concat([pd.DataFrame(hist.history) for hist in hists], sort=True)\nhist_df.index = np.arange(1, len(hist_df)+1)\nfig, axs = plt.subplots(nrows=2, sharex=True, figsize=(16, 10))\naxs[0].plot(hist_df.val_top_3_accuracy, lw=5, label='Validation top 3 Accuracy', color=\"blue\")\naxs[0].plot(hist_df.top_3_accuracy, lw=5, label='Training top 3 Accuracy', color=\"red\")\naxs[0].set_ylabel('Accuracy')\naxs[0].set_xlabel('Epoch')\naxs[0].grid()\naxs[0].legend(loc=0)\n\n##Doing the same for loss\naxs[1].plot(hist_df.val_loss, lw=5, label='Validation MLogLoss', color=\"blue\")\naxs[1].plot(hist_df.loss, lw=5, label='Training MLogLoss', color = \"red\")\naxs[1].set_ylabel('MLogLoss')\naxs[1].set_xlabel('Epoch')\naxs[1].grid()\naxs[1].legend(loc=0)\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb4746eb3cbc23868d7c22bbdbab298456ea9a4a","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"markdown","source":"# Discussion \nAbove we see the accuracy over time, and we can see that it actually still increases after epch 30. However Kaggle will not allow me to run more then 30 epochs before I time out or something like that, so I cannot train my final network. As expected, the loss is almost the inverse of what we see in the accuracy. \n\nIf Kaggel ran the whoel network, I would have like to see what classes had the highest accruacy and what classes had the lowest. In this case, I speculate that more abstract and advanced classes would have a low accauracy, as people can interpret words differently. Also classes that might look alike, such as triangle and pyramid might get confused. \nEasy classes to identify would be classes with unique features such as the wall of china or the eiffel tower. \n\nAs the data is converted to B/W that means that if any class that looks alike was only seperated by their colour, this network might have confused the two and therefore this accuracy would also have been low. \n\n"},{"metadata":{"trusted":true,"_uuid":"3f50b7df4c9d80e3dc0393e3b7e3628df2591aa4"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb51bed5d52e3e26cbd998c38a4df48fd8b83092"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"798aae80897ced48ee8c0c99b2c237aca0efa11f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}