{"cells":[{"metadata":{"_uuid":"75748b0279371c3140e122232fcc527e6951f3fc"},"cell_type":"markdown","source":"Hi!\nSo, I'm not particularly experienced, and I lack my own GPU. However, I tried to put some effort into this competition, trying to work on my skills, and I really like this dataset.\nUnfortunately I haven't seen much improvement with a few iterations on a basic network.\n\nSo I thought, why not ask the community to help me improve.\n\nLet me describe what goes on here:\nI use a rather traditional CNN in Keras with a generator. I feed the network with binary images of size 64x64. 0 = background, 1 = line.\n\nI had a smaller network, and it got stuck at a roughly similar point. So I tried to make it bigger, but it didn't really help at all. Now because of the 6 hour limitation I can only go through so much data in one go, but I put the final model of this network into a second kernel and tried to train it on more data, but it just did not train at all.\n\nThe results I get are roughly like this:\naccuracy just below 60%\ntop 3 accuracy ~77% on train and validation set\n0.67 leaderboard score, I've been stuck here\n\nAnyway, if anyone can point me in a good direction to improve my skills and score, I'd be glad :)\nI take any advice I can get.\n\nMaybe some specific question from me:\n\nIs this \"traditional\" CNN architecture just outdated now? Is this dataset beyond its capabilities?\n\nWhat to do if I don't want to take the \"cheap\" route and use pre-trained models?\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true},"cell_type":"code","source":"# Definitions etc.\n# Lots of functions to convert drawing to a static binary image\n# Generator class that loads chunks from pandas dataframes\n\nimport os\nimport numpy as np\nimport pandas as pd\nimport random\n\nfrom PIL import Image, ImageDraw\nfrom keras.utils import np_utils\nfrom keras.utils import Sequence\n\n#%%\n\ndef ClassFromFileName(filename):\n    dot = filename.find('.')\n    return filename[:dot].replace(' ', '_')\n\n#%%\n\ndef MakeImage(drawing, line_width, orig_size=(256, 256), target_size=(128, 128)):\n    img = Image.new('1', orig_size)\n    draw = ImageDraw.Draw(img)\n    for stroke in drawing:\n        stroke_seq = list(zip(stroke[0], stroke[1]))\n        draw.line(stroke_seq, fill=1, width=line_width)\n    img_small = img.resize(target_size)\n    return img_small\n\n#def SaveImage(drawing, file_path, line_width, orig_size=(256, 256), target_size=(128, 128)):\n    #img = MakeImage(drawing, line_width, orig_size, target_size)\n    #img.save(file_path)\n    \ndef DrawingToImage(drawing, line_width, orig_size=(256, 256), target_size=(128, 128)):\n    img = MakeImage(drawing, line_width, orig_size, target_size)\n    return np.reshape(np.array(list(img.getdata()), dtype=np.uint8), target_size)\n\n#%%\n    \ndef MyRead(folder, file, count, header_index=0, start_index=0):\n    skip = list(range(1, start_index))\n    if len(skip) > 0:\n        return pd.read_csv(os.path.join(folder, file), nrows=count, header=header_index, skiprows=skip)\n    else:\n        return pd.read_csv(os.path.join(folder, file), nrows=count)\n            \n\ndef DrawingsToImages(drawings, line_width, target_size=(128, 128)):\n    return [DrawingToImage(drawing, line_width, target_size=target_size) for drawing in drawings]\n\ndef ExtractData(dataframes, column_name, line_width, target_size=(128, 128)):\n    column_data = [df[column_name].map(eval).tolist() for df in dataframes]\n    return [DrawingsToImages(drawings, line_width, target_size=target_size) for drawings in column_data]\n\ndef MakeAnswers(classes, count):\n    return [[c for i in range(count)] for c in classes]\n\n\n\n#%%\n    \ndef LoadData(folder, dim_2d, channels, samples_per_class, column='drawing', line_width=3, shuffle=True, header=0, start_index=0):\n    files = os.listdir(folder)\n    dataframes = [MyRead(folder, file, samples_per_class, header, start_index) for file in files]\n    image_data = ExtractData(dataframes, column, line_width, dim_2d)\n    \n    classes = [ClassFromFileName(file) for file in files]\n    classes_num = [i for i in range(len(files))]\n    answers = MakeAnswers(classes_num, samples_per_class)\n    \n    total_samples = len(classes) * samples_per_class\n    image_data = np.reshape(image_data, (total_samples, 1, dim_2d[0], dim_2d[1]))\n    answers = np.reshape(answers, (total_samples))\n    answers = np_utils.to_categorical(answers, len(classes), dtype='uint8')\n    \n    if shuffle:\n        images_answers = list(zip(image_data, answers))\n        random.shuffle(images_answers)\n        image_data, answers = zip(*images_answers)\n    \n    image_data = np.array(image_data, dtype=np.uint8)\n    answers = np.array(answers)\n    class_mapping = dict(zip(classes_num, classes))\n    \n    return (image_data, answers, class_mapping)\n\nclass DoodleGenerator(Sequence):\n    \n    def __init__(self, files, class_count, batchsize, chunksize, batches_per_epoch, line_width=3, target_size=(64, 64), skip=0):\n        self.answers = self.MakeAnswers(class_count)\n        self.batch = batchsize\n        self.chunksize = chunksize\n        #self.shuffle = shuffle\n        self.batches_per_epoch = batches_per_epoch\n        #self.max_chunks = max_chunks\n        self.line_width=line_width\n        self.target_size=target_size\n        \n        self.dfiterators = self.LoadDataframes(files)\n        self.chunks = self.LoadChunks(self.dfiterators, chunksize)\n        self.chunkpositions = [0 for d in files]\n        self.chunkcounters = [0 for d in files]\n    \n    def MakeAnswers(self, class_count):\n        indexes = [i for i in range(class_count)]\n        return np_utils.to_categorical(indexes, dtype='uint8')\n    \n    def LoadDataframes(self, files, skip=0):\n        return [self.LoadDataframe(f) for f in files]\n    \n    def LoadDataframe(self, file, skip=0):\n        return pd.read_csv(file, iterator=True, skiprows=lambda x: x > 0 and x < skip)\n    \n    def LoadChunks(self, dataframes, chunksize):\n        return [self.LoadChunk(d, chunksize) for d in dataframes]\n    \n    def LoadChunk(self, dataframe, chunksize):\n        return dataframe.get_chunk(chunksize)        \n    \n    def __len__(self):\n        return self.batches_per_epoch\n    \n    def DoGetOne(self, chunk, chunkposition):\n        dfrow = self.chunks[chunk].iloc[chunkposition]\n        drawing = eval(dfrow['drawing'])\n        return DrawingToImage(drawing, self.line_width, target_size=self.target_size)\n    \n    def GetOne(self):\n        chosen = random.randrange(0, len(self.answers))\n        x = self.DoGetOne(chosen, self.chunkpositions[chosen])\n        self.chunkpositions[chosen] = self.chunkpositions[chosen] + 1\n        if self.chunkpositions[chosen] == self.chunksize:\n            self.chunks[chosen] = self.LoadChunk(self.dfiterators[chosen], self.chunksize)\n            self.chunkcounters[chosen] = self.chunkcounters[chosen] + 1\n            self.chunkpositions[chosen] = 0\n        y = self.answers[chosen]\n        return (x, y)\n    \n    def UnzipReshape(self, tuples):\n        xs = np.reshape(np.array([t[0] for t in tuples]), (self.batch, 1, self.target_size[0], self.target_size[1]))\n        ys = np.array([t[1] for t in tuples])\n        return xs, ys\n    \n    def __getitem__(self, idx):\n        xytuples = [self.GetOne() for i in range(self.batch)]\n        return self.UnzipReshape(xytuples)\n    \n    def GetSamplesServed(self, i):\n        return self.chunkcounters[i] * self.chunksize + self.chunkpositions[i]\n    \n    def GetStatus(self):\n        return [self.GetSamplesServed(i) for i in range(len(self.chunkcounters))]\n#class end\nprint(\"defs done\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true},"cell_type":"code","source":"# Load validation data\n#\n\nfolder = '../input/train_simplified'\nfiles = os.listdir(folder)\npaths = [os.path.join(folder, file) for file in files]\n\ndim_1 = 64\ndim_2 = 64\nitems_per_class = 70\nval_start_index = 0\n\n# paint images with this line width, also used for generator\nline_width = 4\n\n\nval_data, val_answers, class_map = LoadData(folder, (dim_1, dim_2), 1, items_per_class, 'drawing', line_width, False, start_index=val_start_index)\nclass_count = len(class_map)\n\nprint('Data shape: {}'.format(np.shape(val_data)))\nprint('Answers shape: {}'.format(np.shape(val_answers)))\nprint('Classes: {}'.format(class_count))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"693ee471ce2d1ebe9fc775bf1b9dc699c71cc26e"},"cell_type":"code","source":"# Network architecture\n#\n\nfrom keras.models import Sequential\nfrom keras.layers import Convolution2D, MaxPooling2D\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Activation, LeakyReLU\nfrom keras.metrics import top_k_categorical_accuracy\n\ndef top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)\n\nfrom keras import backend as K\nK.set_image_dim_ordering('th')\n\nnet = Sequential()\nnet.add(Convolution2D(64, (3, 3), padding='same', input_shape=(1, dim_1, dim_2)))\nnet.add(LeakyReLU(alpha=0.1))\n\nnet.add(Convolution2D(128, (3, 3), padding='same'))\nnet.add(LeakyReLU(alpha=0.2))\n\nnet.add(Convolution2D(256, (3, 3), padding='same'))\nnet.add(LeakyReLU(alpha=0.2))\n\nnet.add(MaxPooling2D((2, 2)))\n\nnet.add(Convolution2D(256, (3, 3), padding='same'))\nnet.add(LeakyReLU(alpha=0.3))\n\nnet.add(MaxPooling2D((2, 2)))\n\nnet.add(Convolution2D(384, (3, 3)))\nnet.add(LeakyReLU(alpha=0.3))\n\nnet.add(Convolution2D(512, (3, 3)))\nnet.add(LeakyReLU(alpha=0.2))\n\nnet.add(MaxPooling2D((2, 2)))\n\nnet.add(Convolution2D(384, (1, 1)))\nnet.add(LeakyReLU(alpha=0.2))\n\nnet.add(MaxPooling2D((2, 2)))\n\nnet.add(Flatten())\n\nnet.add(Dense(600))\nnet.add(LeakyReLU(alpha=0.1))\n\nnet.add(Dropout(0.25))\n\nnet.add(Dense(class_count, activation='softmax'))\n\nnet.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy', top_3_accuracy])\n\ntry:\n    os.stat('models/')\nexcept:\n    os.mkdir('models/')\n\nnet_json = net.to_json()\nwith open(\"./models/model10.json\", \"w\") as json_file:\n    json_file.write(net_json)\n    \nprint(os.listdir('models'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3ab453db77165d4ea02a42c652d5d95cff4beb9"},"cell_type":"code","source":"# Print net summary\n#\nprint(net.metrics_names)\nprint(net.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3063c208bccc59a3d6cb2ae56eaee016d3360058","scrolled":false},"cell_type":"code","source":"# Prepare generator and fit model\n#\nfrom keras.callbacks import ModelCheckpoint, CSVLogger, EarlyStopping\n\ntrain_model_file = './models/model10.traing.{epoch:02d}.hdf5'\nval_model_file = './models/model10.test.{epoch:02d}.hdf5'\nfinal_model_file = './models/model10.final.hdf5'\nlog_file = './models/model10-log.csv'\n\ntrain_checkpointer = ModelCheckpoint(filepath=train_model_file, monitor='top_3_accuracy', save_best_only=True, save_weights_only=True, mode='max')\nval_checkpointer = ModelCheckpoint(filepath=val_model_file, monitor='val_top_3_accuracy', save_best_only=True, save_weights_only=True, mode='max')\nlogger = CSVLogger(log_file)\nval_loss_stop = EarlyStopping(monitor='val_loss', patience=8, mode='min', baseline=4.0)\n\nepoch_count = 30\nbatch_size = 128\nbatches_per_epoch = 900\nnum_classes = 340 # same as class_count somewhere above, but whatever\n\n# files, class_count, batchsize, chunksize, batches_per_epoch\ndoodlegen = DoodleGenerator(paths, num_classes, batch_size, 128, batches_per_epoch, line_width=line_width, skip=items_per_class+1)\nprint('generator line width: {}'.format(doodlegen.line_width))\n\nimgs_per_epoch = batch_size * batches_per_epoch\nimgs_total = imgs_per_epoch * epoch_count\nimgs_per_class = imgs_total / num_classes\nprint('Images per epoch: {}'.format(imgs_per_epoch))\nprint('Total images processed: {}'.format(imgs_total))\nprint('Expected images per class: {}'.format(imgs_per_class))\n\nmycallbacks = [logger, train_checkpointer, val_checkpointer, val_loss_stop]\nnet.fit_generator(doodlegen, epochs=epoch_count, validation_data=(val_data, val_answers), callbacks=mycallbacks)\nnet.save(final_model_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04bdbf75713992932e21c5f5ca16c6ebbefbd93a"},"cell_type":"code","source":"# Print generator status\n# I used this to verify that there is roughly the same number of samples per class when training\n\nstatus = doodlegen.GetStatus()\nprint('status:\\n', status)\nprint('max: ', np.amax(status))\nprint('avg: ', np.mean(status))\nprint('min: ', np.amin(status))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d5722ca2481f837ff46e2e2ab489d4fbc422a9a"},"cell_type":"code","source":"# load test data\n\ntest_file = '../input/test_simplified.csv'\ntest_df = pd.read_csv(test_file)\ntest_data = ExtractData([test_df], 'drawing', 3, (dim_1, dim_2))\n\ntest_keys = test_df['key_id'].tolist()\ntest_data = np.reshape(test_data[0], (len(test_data[0]), 1, dim_1, dim_2))\nprint(np.shape(test_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44b05d246c373928525852ffca2b760b16169c71"},"cell_type":"code","source":"# Get net output on test set and make a submission file\n\ndef Top3(vec):\n    top1, top2, top3 = -float('inf'), -float('inf'), -float('inf')\n    itop1, itop2, itop3 = 0, 0, 0\n    for idx, val in enumerate(vec):\n        if val > top1:\n            top3 = top2\n            itop3 = itop2\n            top2 = top1\n            itop2 = itop1\n            top1 = val\n            itop1 = idx\n        elif val > top2:\n            top3 = top2\n            itop3 = itop2\n            top2 = val\n            itop2 = idx\n        elif val > top3:\n            top3 = val\n            itop3 = idx\n    return ((itop1, top1), (itop2, top2), (itop3, top3))\n\ndef ToClasses(top, class_map):\n    return [class_map[c] for c, conf in top]\n\ndef GetOutput(net, test_data, test_keys, class_map):\n    net_output = net.predict(test_data, batch_size=128)\n    classes_output = [ToClasses(Top3(x), class_map) for x in net_output]\n    return list(zip(test_keys, classes_output))\n\nprint('Getting output')\noutput = GetOutput(net, test_data, test_keys, class_map)\nprint(output[0])\n\ndef SubmissionLine(item):\n    return str(item[0]) + ', ' + ' '.join(item[1]) + '\\n'\n\nprint('Writing submission file')\ndef MakeSubmission(output, file_name, header):\n    with open(file_name, 'w', newline='\\n') as file:\n        file.write(header + '\\n')\n        for item in output:\n            file.write(SubmissionLine(item))\n\nsubmission_file = './submission.csv'\nMakeSubmission(output, submission_file, 'key_id,word')\n\nprint('Done')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}