{"cells":[{"metadata":{"_uuid":"43a300127cae0aeadf4e02060c07eab10f5ae823"},"cell_type":"markdown","source":"# Localization context with Keras\n\nIn the discussions about this competition I noticed some remarks that the country information does not increase the LB. I decided to go a little other direction and to wrap *countries* into a *continent* feature and tried to compare the approaches. This notebook covers:\n* creating train/valid/test generators based on a list of sub generators for each class\n* option of adding context data to custom CNN model: country or continent of competitor\n* little insight into localization contexted images\n* defining a proper model, depending on the selected context mode\n* models training and comparison\n* making predictions\n\nAt first, import necessary libs, define settings. Note that **additional package _(pycountry-convert)_** was installed."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport json\nimport time\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom enum import Enum\nimport matplotlib\n%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\n\n!pip install pycountry-convert\nimport pycountry_convert\n\nimport keras\nimport tensorflow as tf\nfrom keras.metrics import top_k_categorical_accuracy\nfrom keras.backend.tensorflow_backend import set_session\n\nconfig = tf.ConfigProto()\nconfig.gpu_options.allow_growth = True\nconfig.log_device_placement = True \nsess = tf.Session(config=config)\nset_session(sess) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"252200efbe7d07385ac5814ca59f5861d39dc20b"},"cell_type":"markdown","source":"## Settings, statistics, helpers\nInitial training settings. Take care of *num_classes x (items_in_class+1) > batch_size*."},{"metadata":{"_uuid":"96f16821ff4705379599873a90b144620df8f99a","trusted":true},"cell_type":"code","source":"num_classes = 340 #30 will be set later\nitems_in_class = 75\nbatch_size = 2048\ntraining_percent = 0.4 # how much data to take (40%)\nvalid_percent = 0.05 # size of the validation set (5%)\nsize = 32 # size of image: 32x32px\n\nmy_class_path = \"../input/train_simplified/\"\nclass_paths = os.listdir(\"../input/train_simplified/\")\ncat_names = [item[:-4] for item in class_paths] # take file names, remove '.csv' extension\ncat_names.sort(key=lambda x: str.lower(x)) # sort names of classes regardless capital letters\n\nUNKNOWN_COUNTRY = 'YYY'\nUNKNOWN_CONTINENT = 'XXX'\n\ncntntns = ['AF', 'AS', 'EU', 'NA', 'OC', 'SA']+[UNKNOWN_CONTINENT] # list of continents\n\n# mode indicating whether to consider localization context\nclass ContextMode(Enum):\n no_context = 1\n country_context = 2\n continent_context = 3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"632a73fed610db72e374f65796189c9c71add67d"},"cell_type":"markdown","source":"This section delievers some statistics of the classes files and extracts all of the countries present in the available training data set."},{"metadata":{"_uuid":"1f267fee1557d27c121e4a13e0f2f2c0d1602570","trusted":true},"cell_type":"code","source":"data_countries_codes = []\nfile_lengths = []\nrecog_stat_list = []\n\nfor _name in tqdm(class_paths[0:num_classes]):\n # use keep_default_na=False to prevent pandas parsing country code 'NA' as nan\n df = pd.read_csv(my_class_path+_name, keep_default_na=False)\n file_lengths.append(len(df)-1) # number of samples in file (header not counted)\n \n recog_stat = df['recognized'].value_counts()\n recog_stat_list.append(100.0*recog_stat[1]/(recog_stat[0]+recog_stat[1])) # % of recognized images in class\n \n ccode_stat = df['countrycode'].unique() \n set1 = set(data_countries_codes)\n set2 = set(ccode_stat)\n any_new = set2-set1\n data_countries_codes += list(any_new) # list of unique country two letter codes \n \n# set size of validation set for each class\nvalid_lengths=[int(leng*valid_percent) for leng in file_lengths]\n \nprint('Average file length: {:.0f}+-{:.0f} lines\\nRecognized images: {:.0f}+-{:.0f}%\\nUnique countries in data: {}'.format(np.mean(file_lengths), np.std(file_lengths), np.mean(recog_stat_list), np.std(recog_stat_list), len(data_countries_codes)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78daf4e6f3911305146ee3fb76a61993caf72153","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"myhist=plt.hist(file_lengths, bins=50)\nmyhist=plt.ylabel('number of classes[n]')\nmyhist=plt.xlabel('items in class [n]')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06c7cf1d878c9e2aa9d36a082b0fa2bef7b365f2"},"cell_type":"markdown","source":"Using **pycountry-convert** we can associate two-letter country names encoded in **iso-3166-1** standard with appropriate continent. Following lines should help us catching the exceptions (codes that do not match with the standard). Note that here I make a union of iso-3166-1 codes with codes from the training data but one can stick to the selected set only."},{"metadata":{"trusted":true,"_uuid":"fae327ca8a10d99d5de270539911a4a1abcd8c7c"},"cell_type":"code","source":"valid_countries_dict = pycountry_convert.map_countries(cn_name_format=\"default\")\nvalid_country_codes = list(set([value['alpha_2'] for key, value in valid_countries_dict.items()]))\nall_country_codes= set(valid_country_codes) | set(data_countries_codes) | set([UNKNOWN_COUNTRY])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb39b813216b07676c97603f8064914c588599aa"},"cell_type":"markdown","source":"If received country code is not in our *all_country_codes* set let's assign it to UNKNOWN_COUNTRY. Also set the continent as unknown for unexpected countries (which can come with the testing set)."},{"metadata":{"trusted":true,"_uuid":"6927f3854c3b153acf4461caa3653b44e1a12330"},"cell_type":"code","source":"def ValidateCountry(country_code): \n return country_code if country_code in all_country_codes else UNKNOWN_COUNTRY\n\ndef ValidateContinent(country_code):\n try:\n return pycountry_convert.country_alpha2_to_continent_code(country_code)\n except:\n return UNKNOWN_CONTINENT","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d36c0268376853f485ca018a26cc55d693cf82b8"},"cell_type":"markdown","source":"And let's test them:"},{"metadata":{"_uuid":"1e600086b8ee66ee221ad7a8c6764650806de9ec","trusted":true},"cell_type":"code","source":"data_countries_codes = [ValidateCountry(_cntry) for _cntry in data_countries_codes]\ncont_codes = [ValidateContinent(_cntry) for _cntry in data_countries_codes]\nprint(\"First 10 countries: {}\".format(data_countries_codes[0:10]))\nprint(\"Associated continents: {}\\n\".format(cont_codes[0:10]))\nprint(\"Unique continents: {}\".format(set(cont_codes))) # has to match defined cntntns list","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbad3e1e1d4d67c8aa449b1e37ff79b372f5ef3a"},"cell_type":"markdown","source":"Function **draw_cv2**, which I have first spotted in [@beluga](https://www.kaggle.com/gaborfodor) kernels as [here [1]](https://www.kaggle.com/gaborfodor/black-white-cnn-lb-0-77) or [here [2]](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-animals) (in the second example each stroke color gets lighter). It is very nice, however it is the main bottleneck in terms of execution time within my solution (at least in the way I call it from the generator). Parallelization tries with splitting the em.drawings.values into few tables, each given to other process in a pool did not help me so far."},{"metadata":{"_uuid":"1749d9f5cc04f31354089324b5883ba5730b06c3","trusted":true},"cell_type":"code","source":"#---drawing images: ref [1,2]--------------------------------------\ndef draw_cv2(raw_strokes):\n size = 32\n lw = 6\n BASE_SIZE = 256\n img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8) \n for stroke_no, stroke in enumerate(raw_strokes):\n line_intensity = 255 - min(stroke_no, 10) * 10\n for i in range(len(stroke[0]) - 1):\n _ = cv2.line(img, (stroke[0][i], stroke[1][i]), (stroke[0][i + 1], stroke[1][i + 1]), line_intensity, lw) \n if size != BASE_SIZE:\n return cv2.resize(img, (size, size))\n return img","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"494f575b6b92d6ffd872c8bb07d83cfe6f53ff6d"},"cell_type":"markdown","source":"### Generators\nThe following cell defines a sub generator, dedicated for just one file/class. When running the notebook for all 340 classes, 340 sub generators for training and 340 for validation will be produced."},{"metadata":{"_uuid":"b3f19ec137146c744223d47b35979f3c1dc7d5c4","trusted":true},"cell_type":"code","source":"num_classes = 30\n\ndef items_gen(_id, isTraining):\n _path = my_class_path + cat_names[_id] + '.csv'\n \n if isTraining: # skip the first valid_percent lines of the file\n start_index, end_index = 1, valid_lengths[_id] \n if training_percent<1: # or select only last training_ratio lines\n start_index, end_index = 1, int(file_lengths[_id]*(1-training_percent)) \n else: # validation: skip the last 1-valid_percent lines of the file\n start_index, end_index = valid_lengths[_id], file_lengths[_id]\n \n while True:\n # skiprows-> generator will chunk within the specified range, depending on its type [training/validation]\n for _chunk in pd.read_csv(_path, chunksize=items_in_class +1, usecols=['drawing', 'recognized', 'word', 'countrycode'],skiprows=range(start_index, end_index), keep_default_na=False):\n yield _chunk","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff9d167cc4ac9846673f8bb807b2b7812c0bd094"},"cell_type":"markdown","source":"Let's create a list of sub generators for training and validation. It is time consuming to call 340 generators - on my laptop (i6700HQ) it takes usually up to 4s to evaluate *[next(gen_list_TRAIN[id]) for id in range(num_classes)]* (or 0.8s using Pools) but on Kaggle server it is only around 0.6s. It is definately a part to replace/improve. However it is quite flexible in terms of removing/adding classes (separte files) to analysis and is at least good for prototyping."},{"metadata":{"_uuid":"d60c8207e0baafe4485597579bf2acd0b5326c90","trusted":true},"cell_type":"code","source":"gen_list_TRAIN = [items_gen(i, isTraining=True ) for i in range(num_classes)]\ngen_list_VALID = [items_gen(i, isTraining=False) for i in range(num_classes)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f3d6b7aaf3e78c1beb644220c631c7d5dfc7933"},"cell_type":"markdown","source":"For the sake of submission, why not to define another generator, so we won't have to repeat parts of the code."},{"metadata":{"trusted":true,"_uuid":"a45d2c29b46d3b7f755881208fcc7a7720abd1e2"},"cell_type":"code","source":"def test_gen():\n _path = '../input/test_simplified.csv' \n while True: \n for _chunk in pd.read_csv(_path, chunksize=batch_size, usecols=['drawing','countrycode'], keep_default_na=False):\n yield _chunk \n\ngen_TEST = test_gen()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc0c14f94ef8e12d827d019aea6ab99d35e2fc15"},"cell_type":"markdown","source":"Encoders for labels and context information. Since the country/continent feature is more qualitative rather than quantitative it should be represented as one-hot vector (so it will require *n* number of additional neurons, where *n* is the number of countries/continents)."},{"metadata":{"_uuid":"5a2ae872cc242ba991e449b99c4229219fa29c03","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"labels_encoder = LabelEncoder()\nccodes_encoder = LabelEncoder()\ncntnts_encoder = LabelEncoder()\nlabels_encoder.fit(cat_names[0:num_classes])\nccodes_encoder.fit(list(all_country_codes))\ncntnts_encoder.fit(cntntns)\n#------------------------------------------\nlabels_oneh = OneHotEncoder().fit(np.arange(num_classes).reshape(-1,1))\nccodes_oneh = OneHotEncoder().fit(np.arange(len(list(all_country_codes))).reshape(-1,1))\ncntnts_oneh = OneHotEncoder().fit(np.arange(len(cntntns)).reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"793f00d6f82bc5d6d96893633afc2882b83e4637"},"cell_type":"markdown","source":"Main data generator. Allows to be set as training/validation/testing data generator. Also yields input data with or without context, depending on the selected mode. "},{"metadata":{"_uuid":"d0379a608449d182036b8846c9cbf738be819335","trusted":true},"cell_type":"code","source":"class GeneratorMode(Enum):\n training = 1\n validation = 2\n testing = 3\n \ndef main_generator(whatGeneratorMode, whatContextMode=ContextMode.no_context):\n while True:\n \n #---getting and simple filtering of data--------------------------\n if whatGeneratorMode==GeneratorMode.training:\n results = [next(gen_list_TRAIN[id]) for id in range(num_classes)] \n results = pd.concat(results)\n results = results[results.recognized == True]\n \n elif whatGeneratorMode==GeneratorMode.validation:\n results = [next(gen_list_VALID[id]) for id in range(num_classes)] \n results = pd.concat(results)\n results = results[results.recognized == True]\n \n elif whatGeneratorMode==GeneratorMode.testing:\n results = next(gen_TEST) \n \n #---shuffling and batch size setting------------------------------\n #-(depending on the number of not recognized samples, we need to--\n #-add or subtract samples from the concatenated dataframe)--------\n _itms = len(results) \n results = results.sample(frac=1, random_state=2018).reset_index(drop=True)\n if(_itms>batch_size):\n results=results[0:batch_size]\n elif(_itms