{"cells":[{"metadata":{"_uuid":"43a300127cae0aeadf4e02060c07eab10f5ae823"},"cell_type":"markdown","source":"# Area context with Keras\n\nAs some discussions and [my results](https://www.kaggle.com/qlasty/localization-context-country-continent) indicate, including countrycode does not seem to help in any way to improve the LB. In this approach I utilized a simple clustering of countries so now we can give information more general than the country name but also more precise than very rough continent context. The basic clustering is made upon localization [(kernel)](https://www.kaggle.com/qlasty/area-context-country-clustering), hence some clusters may reflect cultural common core.\nThis notebook covers:\n* creating train/valid/test generators\n* option of adding area context to MobileNet model\n* defining a proper model, depending on the selected context mode\n* making predictions\n\n### Significant part of the notebook relies on solutions of [@beluga](https://www.kaggle.com/gaborfodor) (e.g. drawing, MobileNet utilization, schuffling) - for whom: big kudos (and upvotes)!\n\n\nAt first, import necessary libs, define settings. Note that **additional package _(pycountry-convert)_** was installed."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport json\nimport time\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom enum import Enum\nimport matplotlib\n%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\n\n!pip install pycountry-convert\nimport pycountry_convert\n\nimport keras\nimport tensorflow as tf\nfrom keras.metrics import top_k_categorical_accuracy\nfrom keras.backend.tensorflow_backend import set_session\n\nconfig = tf.ConfigProto()\nconfig.gpu_options.allow_growth = True\nconfig.log_device_placement = True \nsess = tf.Session(config=config)\nset_session(sess) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"252200efbe7d07385ac5814ca59f5861d39dc20b"},"cell_type":"markdown","source":"## Settings, helpers\nInitial training settings. "},{"metadata":{"_uuid":"96f16821ff4705379599873a90b144620df8f99a","trusted":true},"cell_type":"code","source":"batch_size = 340*3\nvalid_percent = 1\ndata_files = 100\nsize = 64 # image size\nnum_classes = 340\n\nclass_paths = os.listdir(\"../input/quickdraw-doodle-recognition/train_simplified/\")\ncat_names = [item[:-4] for item in class_paths] # take file names, remove '.csv' extension\ncat_names.sort(key=lambda x: str.lower(x)) # sort names of classes regardless capital letters\nUNKNOWN_COUNTRY = 'YY' # code of unknown country\n\n# mode indicating whether to consider area context\nclass ContextMode(Enum):\n    no_context = 1\n    area_context = 2    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09dfcd16f6eb5f73af49502f8000423f5d9b2379"},"cell_type":"markdown","source":"Here I only consider countries from the ISO 3166-1 standard. More details are described in this [kernel](https://www.kaggle.com/qlasty/localization-context-country-continent)."},{"metadata":{"_uuid":"fae327ca8a10d99d5de270539911a4a1abcd8c7c","trusted":true},"cell_type":"code","source":"valid_countries_dict = pycountry_convert.map_countries(cn_name_format=\"default\")\nvalid_country_codes = list(set([v['alpha_2'] for k, v in valid_countries_dict.items()]))\nall_country_codes= set(valid_country_codes) | set([UNKNOWN_COUNTRY])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb39b813216b07676c97603f8064914c588599aa"},"cell_type":"markdown","source":"If received country code is not in our *all_country_codes* set let's assign it to UNKNOWN_COUNTRY."},{"metadata":{"_uuid":"6927f3854c3b153acf4461caa3653b44e1a12330","trusted":true},"cell_type":"code","source":"def ValidateCountry(country_code):    \n    return country_code if country_code in all_country_codes else UNKNOWN_COUNTRY","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36020f38e1d0784334621f622516ea4d8756b936"},"cell_type":"markdown","source":"Import area mapping (output from another kernel)"},{"metadata":{"trusted":true,"_uuid":"0e1d10813599ebf71fbb8b0de7315ea615bcc6a6"},"cell_type":"code","source":"our_mapping = pd.read_csv('../input/area-context-country-clustering/area_mapping.csv')\nprint(our_mapping.head())\n\nour_dict = pd.Series(our_mapping.groups.values, index = our_mapping.alpha3).to_dict()\n\ndef get_area(iso_code):\n    try:\n        return our_dict[iso_code]\n    except KeyError:\n        return 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5df3484b6342b7f7f15fca1ddddd1dfccd3b221"},"cell_type":"code","source":"num_groups = max(our_mapping.groups)+1\nprint(num_groups)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbad3e1e1d4d67c8aa449b1e37ff79b372f5ef3a"},"cell_type":"markdown","source":"Function **draw_cv2**, which I have first spotted in [@beluga](https://www.kaggle.com/gaborfodor) kernels as [here [1]](https://www.kaggle.com/gaborfodor/black-white-cnn-lb-0-77) or [here [2]](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-animals) (in the second example each stroke color gets lighter)."},{"metadata":{"_uuid":"1749d9f5cc04f31354089324b5883ba5730b06c3","trusted":true},"cell_type":"code","source":"#---drawing images: ref [1,2]--------------------------------------\ndef draw_cv2(raw_strokes, size=32, lw=6):    \n    BASE_SIZE = 256\n    img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)    \n    for stroke_no, stroke in enumerate(raw_strokes):\n        line_intensity = 255 - min(stroke_no, 10) * 10\n        for i in range(len(stroke[0]) - 1):\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]), \n                         (stroke[0][i + 1], stroke[1][i + 1]), line_intensity, lw)            \n    if size != BASE_SIZE:\n        return cv2.resize(img, (size, size))\n    return img","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d08958d034bcca3821405d4cf9ec037b3421a0c0"},"cell_type":"markdown","source":"Generators based on this [kernel [3].](https://www.kaggle.com/gaborfodor/shuffle-csvs)"},{"metadata":{"trusted":true,"_uuid":"528350e8a0358fb898295a6c4bccc3f986f31f7d"},"cell_type":"code","source":"def image_generator(batch_size, isTraining):\n    while True:\n        if isTraining: # if valid_percent==1 -> training will use 99 files of 100\n            index_table = np.random.permutation(data_files-valid_percent)\n        else: # if valid_percent==1 -> validation will use 1 file of 100\n            index_table = np.random.permutation(valid_percent)+(data_files-valid_percent)\n        \n        _path = '../input/shuffle-and-filter/'\n        for k in index_table:                         \n            filename = os.path.join(_path,'train_k{}.csv.gz'.format(k))\n            for chunk in pd.read_csv(filename, chunksize=batch_size, \n                                     usecols=['countrycode','y','drawing','word'], \n                                     keep_default_na=False):             \n                ch_size = len(chunk)                \n                if(ch_size<batch_size):\n                    continue # generator will reach to the beginning \n                    # of another file for full length batch\n                else:\n                    yield chunk","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f3d6b7aaf3e78c1beb644220c631c7d5dfc7933"},"cell_type":"markdown","source":"For the sake of submission, I define another generator, so we won't have to repeat parts of the code."},{"metadata":{"_uuid":"a45d2c29b46d3b7f755881208fcc7a7720abd1e2","trusted":true},"cell_type":"code","source":"def test_gen():\n    _path = '../input/quickdraw-doodle-recognition/test_simplified.csv'        \n    while True:        \n        for _chunk in pd.read_csv(_path, chunksize=batch_size, \n                                  usecols=['drawing','countrycode'], \n                                  keep_default_na=False):            \n            \n            # I made it as it helps overcomming some shape problems with\n            # predict_generator, but one has to take care of selecting\n            # appropriate range of lines for the output predictions\n            if (len(_chunk)<batch_size):\n                _chunk = pd.concat([_chunk, _chunk[0:batch_size-len(_chunk)]])                                               \n            yield _chunk            \n\ngen_TEST = test_gen()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc0c14f94ef8e12d827d019aea6ab99d35e2fc15"},"cell_type":"markdown","source":"One hot encoder for areas."},{"metadata":{"_kg_hide-output":true,"_uuid":"5a2ae872cc242ba991e449b99c4229219fa29c03","trusted":true},"cell_type":"code","source":"areas_oneh = OneHotEncoder().fit(np.arange(num_groups).reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63dc4ad1587f77f8efa596d14d54146d2e13a251"},"cell_type":"markdown","source":"Main generator"},{"metadata":{"_uuid":"d0379a608449d182036b8846c9cbf738be819335","trusted":true},"cell_type":"code","source":"class GeneratorMode(Enum):\n    training = 1\n    validation = 2\n    testing = 3\n    \ndef main_generator(whatGeneratorMode, whatContextMode=ContextMode.no_context):\n    while True:\n        \n        #---getting and simple filtering of data--------------------------\n        if whatGeneratorMode==GeneratorMode.training:\n            results = next(train_generator)            \n            \n        elif whatGeneratorMode==GeneratorMode.validation:\n            results = next(valid_generator)            \n            \n        elif whatGeneratorMode==GeneratorMode.testing:\n            results = next(gen_TEST)                                        \n                    \n                    \n        #---drawing images: ref [1,2]--------------------------------------\n        results['drawing'] = results['drawing'].apply(json.loads)        \n        x = np.zeros((batch_size, size, size))\n        for i, raw_strokes in enumerate(results.drawing.values):\n            x[i] = draw_cv2(raw_strokes, size=size)\n    \n        x = x / 255.\n        x = x.reshape((batch_size, size, size, 1)).astype(np.float32)        \n                \n        \n        if whatGeneratorMode!=GeneratorMode.testing:                            \n            #---converting labels to 1-hot---------------------------------\n            y = keras.utils.to_categorical(results.y, num_classes=num_classes)\n                \n        #---output: - yielded 'x' depends on the selected context mode----\n        if whatContextMode==ContextMode.no_context:\n            if whatGeneratorMode==GeneratorMode.testing:                            \n                yield x\n            else:\n                yield x, y\n        elif whatContextMode==ContextMode.area_context:\n            _countries = results['countrycode'].apply(ValidateCountry).values\n            _areas = np.array(list(map(get_area, _countries)))\n            xac = areas_oneh.transform(_areas.reshape(-1,1)).toarray()                        \n            \n            if whatGeneratorMode==GeneratorMode.testing:                         \n                yield [x, xac]\n            else:\n                yield [x, xac], y                ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b3ab34ac7e7d55da4a8d1d10b07b5f6854976f1"},"cell_type":"markdown","source":"### CNN MobileNet Model \nHelper function to evaluate top3 accuracy."},{"metadata":{"_uuid":"1dd455da3f6432b93281b57378769a3de3dd46d8","trusted":true},"cell_type":"code","source":"def in_top_3(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a1db76c3c2149f9caa4530bcdc17f4aac0a0696","trusted":true},"cell_type":"code","source":"def create_model(whatContextMode=ContextMode.no_context, alpha=1.0, show=False):\n    \n    from keras.models import Model\n    from keras.layers import Input, concatenate, Reshape, Activation, Conv2D\n    from keras.layers import Dropout, Flatten, GlobalAveragePooling2D\n    from keras.optimizers import Adam     \n    from keras.applications import MobileNet\n    from keras import backend as K\n    K.clear_session()\n        \n    if whatContextMode.value==ContextMode.no_context.value:        \n        my_model = MobileNet(input_shape=(size, size, 1), weights=None, alpha=alpha, classes=num_classes)\n\n        my_model.compile(optimizer=Adam(), loss='categorical_crossentropy',\n                  metrics=['accuracy',in_top_3])\n\n    elif whatContextMode.value==ContextMode.area_context.value:\n        \n        dropout=0.001        \n        shape0 = (1, 1, int(1024*alpha))\n        shape1 = (1, 1, int(1024*alpha+num_groups))        \n        Input_area = Input(shape=(num_groups,))     \n        \n        my_model = MobileNet(input_shape=(size, size, 1), weights=None, alpha=alpha, include_top=False)\n\n        x = my_model.get_layer('conv_pw_13_relu').output # call last layer of the model\n\n        # following implementation of architecture of top layers of MobileNet taken from:\n        # https://github.com/fchollet/deep-learning-models/blob/master/mobilenet.py\n        #--------- Start of implementation: ---------\n        x = GlobalAveragePooling2D()(x)\n        x = Reshape(shape0, name='reshape_0')(x)\n        \n        #---------------adding area context (not in original model)--\n        x = Flatten()(x)\n        x = concatenate([x, Input_area])        \n        x = Reshape(shape1, name='reshape_1')(x)\n        #------------------------------------------------------------\n        \n        x = Dropout(dropout, name='dropout')(x)\n        x = Conv2D(num_classes, (1, 1), padding='same', name='conv_preds')(x)\n        x = Activation('softmax', name='act_softmax')(x)\n        preds = Reshape((num_classes,), name='reshape_2')(x)\n        #--------- End of implementation ----------\n        \n        my_model = Model(inputs=[my_model.input, Input_area], outputs=preds)                \n        my_model.compile(optimizer=Adam(), loss='categorical_crossentropy',\n                  metrics=['accuracy',in_top_3])\n    else:\n        print('Context not recognized')\n        return\n    \n    if show:\n        my_model.summary()\n\n    return my_model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36aa0bea03d48b51df9c59f27c1047a254c54f1c"},"cell_type":"markdown","source":"Take a look how the context is included in the CNN model:"},{"metadata":{"_uuid":"8257ceebd87b4abc6416b81bfbf34eb1fe1cb210","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"ourContextMode=ContextMode.area_context\ncreate_model(whatContextMode=ourContextMode, alpha=0.5, show=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"99bfaffd78ba1dfc84206a439cb686365a7235a1"},"cell_type":"markdown","source":"### Model evaluation"},{"metadata":{"_uuid":"6cb7fd4847f944d49d07fb012f6099fdd8fbe22a","trusted":true},"cell_type":"code","source":"from numpy.random import seed as nseed\nnseed(2018)\nfrom tensorflow import set_random_seed\nset_random_seed(2018)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96a09792294b616fc4aa329bb94e94280c6dc017"},"cell_type":"code","source":"ourContextMode = ContextMode.area_context\n\ntrain_generator = image_generator(batch_size=batch_size, isTraining=True)\nvalid_generator = image_generator(batch_size=batch_size, isTraining=False)\n\ntrain_gen=main_generator(whatGeneratorMode=GeneratorMode.training,   \n                         whatContextMode=ourContextMode)\nvalid_gen=main_generator(whatGeneratorMode=GeneratorMode.validation, \n                         whatContextMode=ourContextMode)    \n\nmodel = create_model(whatContextMode=ourContextMode)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bb5da1cf21bf1abb24ea65d03fafa6e1ba0fd51","_kg_hide-output":true},"cell_type":"code","source":"train_epoch=44\ntrain_steps=1000\nvalidation_steps=34\n\nfrom keras.callbacks import ReduceLROnPlateau\ncb = [ReduceLROnPlateau(monitor='val_acc', factor=0.5, \n                        patience=5, mode='max', cooldown=3, verbose=0)]\n\nhistory = model.fit_generator(\n        generator=train_gen, steps_per_epoch=train_steps, \n        validation_data=valid_gen, validation_steps=validation_steps,\n        callbacks=cb, \n        epochs=train_epoch, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7bdfe3a674aa6f899dd61d23e7dd2f497e504c0e","trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(nrows=2, ncols=3, figsize=(12, 8))\noptions = ['val_loss','val_acc','val_in_top_3','loss','acc','in_top_3']\nprint(ourContextMode.name)\nfor i, opt in enumerate(options):    \n    ax = axs[i//3, i%3]        \n    ax.plot(np.arange(train_epoch)+1, history.history[opt], marker='o')\n    ax.set_xlabel('epochs')\n    ax.set_title(opt)    \n    ax.grid()\nplt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ef997634003f988cc1dc5ba9b10fa70baa62043"},"cell_type":"markdown","source":"\n______________________________\n\n### Predictions"},{"metadata":{"_uuid":"3c8cd3b8891052c2500f49cc903dcb7de5037915","trusted":true},"cell_type":"code","source":"_test_gen = main_generator(whatGeneratorMode=GeneratorMode.testing, \n                           whatContextMode=ourContextMode)\npredictions = model.predict_generator(_test_gen, steps=np.ceil(112199/batch_size), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83468a8cfd3f09016382b70344815aa76ba97e7b","trusted":true},"cell_type":"code","source":"print(np.shape(predictions))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0d1703383e3426df2791f4b05f376d0efa3a5d9","trusted":true},"cell_type":"code","source":"def top3cats(predictions):\n    return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], \n                        columns=['word','word2','word3'] )\n\ntest_simp = pd.read_csv('../input/quickdraw-doodle-recognition/test_simplified.csv', \n                        nrows=np.shape(predictions)[0])\ncategories_dict = {_id: cat_name.replace(' ', '_') for _id, cat_name in enumerate(cat_names)}\n\noutput = top3cats(predictions).replace(categories_dict)\noutput = (output.word + ' ').str.cat([output.word2 + ' ', output.word3])\noutput.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2dc97f3d93e3d029d3d9f7aa0d8c58b7ce046dde","trusted":true},"cell_type":"code","source":"output = pd.concat([test_simp['key_id'], output.loc[0:112198]], axis=1)\noutput.tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7efa6297461a6073be465c004a122efc2844e37","trusted":true},"cell_type":"code","source":"output.to_csv(ourContextMode.name + '_myResults.csv',index=False) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d426a827637e3fe60f23344974ca51b318a05d2e"},"cell_type":"markdown","source":"### References \n[1] [Beluga - Black&White CNN [LB=0.77], *kernel*](https://www.kaggle.com/gaborfodor/black-white-cnn-lb-0-77)  \n[2] [Beluga - Greyscale MobileNet Animals, *kernel*](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-animals)  \n[3] [Beluga - Shuffle csvs, *kernel*](https://www.kaggle.com/gaborfodor/shuffle-csvs)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}