{"cells":[{"metadata":{"_uuid":"1a923c2104562eadf00c5807ee9d5fcbd973f9ec"},"cell_type":"markdown","source":"# Keras Simple CNN Benchmark\nThis kernel consists of the following elements:\n\n* Convolutional Neural Network.\n* Training of the model.\n* Descriptions of earlier versions of the code.\n* Analysis of the kernels performance.\n"},{"metadata":{"_uuid":"c2529a98a02b39421f173f3267a13a113a8dc7a5"},"cell_type":"markdown","source":"# Problems with kaggle\nJust before comitting the final version of the code, something went wrong with Kaggle and I was not able to commit anything. Later in the text I will refer to previous versions of the code. To view them, please use the following link:\nhttps://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad/versions?scriptVersionId=7919141\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nimport os\nimport ast\nimport datetime as dt\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = [16, 10]\nplt.rcParams['font.size'] = 14\nimport seaborn as sns\nimport cv2\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.layers import BatchNormalization, Dense, Dropout, Flatten, Activation\nfrom tensorflow.keras.metrics import categorical_accuracy, top_k_categorical_accuracy, categorical_crossentropy\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import Adam\nfrom collections import deque\nimport random\nimport scipy.misc as misc\nstart = dt.datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"033879c8451c02295c28b3680ec9e51f023a059f","_kg_hide-input":true},"cell_type":"code","source":"DP_DIR = '../input/shuffle-csvs/'\nINPUT_DIR = '../input/quickdraw-doodle-recognition/'\nBASE_SIZE = 256\nNCSVS = 100\nNCATS = 340\nnp.random.seed(seed=1987)\ntf.set_random_seed(seed=1987)\ndef f2cat(filename: str) -> str:\n    return filename.split('.')[0]\n\ndef list_all_categories():\n    files = os.listdir(os.path.join(INPUT_DIR, 'train_simplified'))\n    return sorted([f2cat(f) for f in files], key=str.lower)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1b7f42049a9bda3b2a797abddfe55b8d8e83742","_kg_hide-input":true},"cell_type":"code","source":"def apk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    if len(predicted) > k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i, p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i + 1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=3):\n    \"\"\"\n    Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n    \"\"\"\n    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])\n\ndef preds2catids_val(valid_predictions, gt):\n    val_pred = np.argsort(-valid_predictions, axis=1)[:, :3]\n    val_pred_gt = np.zeros((val_pred.shape[0],val_pred.shape[1]+1), dtype=int)\n    val_pred_gt[:,1:] = val_pred\n    val_pred_gt[:,0] = gt[:, 0]\n    return pd.DataFrame(val_pred_gt, columns=['ground truth','a', 'b', 'c'])\n\ndef preds2catids(predictions):\n    return pd.DataFrame(np.argsort(-predictions, axis=1)[:, :3], columns=['a', 'b', 'c'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3b222e1752a10c07fda6daea78fec11bec950cb"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"e6b6da3ac9bbf135258b03b04a3e7fd49f617fed"},"cell_type":"markdown","source":"# Kernel versions description\n\nFor the purpose of the miniproject, several versions of an existing kernel [1] were created. \n\n**Version 3. Original version.**\nThis version was taken from [1] and has a few things added to it that helps to evaluate the code and answer the exercise questions. The model is evaluated on the validation set which consists of 10.000 images from the 340 categories. The mAP is equal to 0.763 for this version of the code. The next versions have some minor changes introduced to them, following suggestions mentioned in exercise 2.2. In order to evaluate the importance of the change the code has undergone, only one thing at a time was altered.\nVersion 3 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7833461\n\n**Version 19. Adding batch normalization.**\nWhen batch normalization was added after every convolutional layer, the mAP score increased to 0.768. This can be explained by the fact that by applying batch normalization, the output from the hidden units of the network is normalized. This helps in reducing the internal covariate shift and enables regularization of the model [2].\nVersion 19 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7906470\n"},{"metadata":{"trusted":true,"_uuid":"4eb6c530d54785696674d2b8240271c91b93f6f1"},"cell_type":"code","source":"#The model architecture is defined here\ndef custom_single_cnn(size, conv_layers=(8, 16, 32, 64), dense_layers=(512, 256), conv_dropout=0.2,\n                      dense_dropout=0.2):\n    model = Sequential()\n    model.add(\n        Conv2D(conv_layers[0], kernel_size=(3, 3), padding='same', activation='relu', input_shape=(size, size, 1)))\n    # In Version 19, this line was used to add batch normalization\n    #     model.add(BatchNormalization())\n    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n    if conv_dropout:\n        model.add(Dropout(conv_dropout))\n\n    for conv_layer_size in conv_layers[1:]:\n        model.add(Conv2D(conv_layer_size, kernel_size=(3, 3), activation='relu'))\n        # In Version 19, this line was used to add batch normalization\n#         model.add(BatchNormalization())\n        model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n        if conv_dropout:\n            model.add(Dropout(conv_dropout))\n\n    model.add(Flatten())\n\n    for dense_layer_size in dense_layers:\n        model.add(Dense(dense_layer_size, activation='relu'))\n        model.add(Activation('relu'))\n        if dense_dropout:\n            model.add(Dropout(dense_dropout))\n\n    model.add(Dense(NCATS, activation='softmax'))\n    return model\n\ndef top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1be08c5fa4807434a7bc30a7922e1a4ae3f0996f"},"cell_type":"markdown","source":"**Version 5. Changing batch size.**\nIn this version of the code, batch size was changed from 512 to 32. This makes the mAP score drop from 0.763 to 0.612. Such drop in performance can be explained by the fact that the batch size determines the number of images in each iteration. The more images there are in each iteration, the easier it is for the model to learn features that will apply to the entire dataset, not just to the small batch of images. \nVersion 5 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7848781"},{"metadata":{"trusted":true,"_uuid":"08c624ee6f13f928ca915829e99f31d5c783921a"},"cell_type":"code","source":"STEPS = 500\nsize = 32\n# This variable controls the batch size of the model and was modified in Version 5.\nbatchsize = 512","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"110accd7b812a57ea2fdec826f37c1a54efc7b28"},"cell_type":"markdown","source":"**Version 20. Increasing the number of network layers. **\nIn this version of the code, an extra convolutional layer was added to the network and this has decreased the mAP score to 0.707. This might be due to the fact that the model is too complex for the given data, thus overfitting it. Besides that, by adding the extra layer, the output from the convolutional layers has feature maps that are decreased to size 2 x 2, which might be too small for the given task. \nVersion 20 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7906557\n\n**Version 21. Adding dropout layers.**\nApplying dropout to 20% of the nodes decreased the mAP score to 0.752. This might suggest low model capacity. Therefore introducing regularization in the form of dropout will not help in increasing its mAP score.\nVersion 21 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7906611\n\n**Version 22. Changing the learning rate.**\nIn this version, the learning rate of Adam optimizer was increased from 0.0024 to 0.005. This reduced the mAP score to 0.72 which means that the learning rate was too big, making it hard for the model to reach the local minimum.\nVersion 22 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7906654"},{"metadata":{"trusted":true,"_uuid":"8cf540bbf09ab24bd34a207891eb9d1bd93e7f21"},"cell_type":"code","source":"model = custom_single_cnn(size=size,\n                          # In Version 20 the following line was cnahged to:\n                          # conv_layers=[128, 64, 32],\n                          # to increase the number of network layers\n                          conv_layers=[128, 64],\n                          \n                          dense_layers=[1024],\n                          \n                          # In Version 21 the following line was cnahged to:\n                          # conv_dropout=0.2,\n                          # to apply dropout to 20% of the nodes\n                          conv_dropout=False,\n                          \n                          dense_dropout=0.25)\n\n# # In Version 22 the following part of the next line was cnahged to:\n# optimizer=Adam(lr=0.005)\n# which changed the learning rate of Adam optimizer from 0.0024 to 0.005.\nmodel.compile(optimizer=Adam(lr=0.0024), loss='categorical_crossentropy',\n              metrics=[categorical_crossentropy, categorical_accuracy, top_3_accuracy])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d378ee2118a50627c8fd0930249b82d3e9ee42b5"},"cell_type":"markdown","source":"**Version 6. Using data augmentation.**\nIn this version of the code, data augmentation techniques were applied to the input data. Specifically, random cropping and horizontal flipping were utilized. Random erasing was implemented, but not used in the current code, due to it having too drastic changes on the dataset. The augmented data is combined with the original data in order to increase the amount of training images. With data augmentation the mAP score decreased to 0.750. This can occur if the augmentations introduce too drastic changes to the dataset. In this case, excluding random cropping, but keeping the horizontal flipping, might help in increasing the mAP score. Sadly, due to time constraints, this assumption cannot be tested.\nVersion 6 can be viewed at: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7863337"},{"metadata":{"trusted":true,"_uuid":"7c9a777674c6e20392037a3b5113a0e8c92e0a5e"},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6):\n    img = np.zeros((BASE_SIZE, BASE_SIZE), np.uint8)\n    for stroke in raw_strokes:\n        for i in range(len(stroke[0]) - 1):\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]), (stroke[0][i + 1], stroke[1][i + 1]), 255, lw)\n    if size != BASE_SIZE:\n        return cv2.resize(img, (size, size))\n    else:\n        return img\n\n# In Version 6 this function was modified to perform data augmentation on the input images.\n# The augmented images were used together with the original images during training.\ndef image_generator(size, batchsize, ks, lw=6):\n    while True:\n        for k in np.random.permutation(ks):\n            filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n            for df in pd.read_csv(filename, chunksize=batchsize):\n                df['drawing'] = df['drawing'].apply(ast.literal_eval)\n                # For the purpoce of data augmentation, this line was modified to:\n                # x = np.zeros((len(df)*2, size, size))\n                x = np.zeros((len(df), size, size))\n                \n                for i, raw_strokes in enumerate(df.drawing.values):\n                    x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n                    # This code performs random erasing\n                    # It was not used, since it made too drastic changes\n#                     new_raw_strokes = raw_strokes\n#                     first = True\n#                     for i in range(len(new_raw_strokes)):\n#                         if len(new_raw_strokes[i][0]) > 2 and len(new_raw_strokes) > 3 and first:\n#                             rand = random.randint(0,len(new_raw_strokes[i][0])-1)\n#                             del new_raw_strokes[i][0][rand]\n#                             del new_raw_strokes[i][1][rand]\n#                             first = False\n#                     x_t = draw_cv2(new_raw_strokes, size=size, lw=lw)\n \n                # This part performs random cropping and horizontal flipping\n                # It was used to augment the data\n#                 rand_start = random.randint(0, int(size * 0.05))\n#                     rand_end = random.randint(int(size * 0.95), size-1)\n#                     x_temp = x[i][rand_start:rand_end, rand_start:rand_end]\n#                     x_temp_2 = cv2.resize(x_temp, (size, size), interpolation = cv2.INTER_NEAREST)\n#                     x_temp_2 = cv2.flip(x_temp_2, 1)\n#                     x[i+len(df)] = x_temp_2\n                    \n                x = x / 255.\n                # For the purpose of data augmentation, the following lines were changed to:\n                # x = x.reshape((len(df)*2, size, size, 1)).astype(np.float32)\n                # y = keras.utils.to_categorical(df.y.append(df.y), num_classes=NCATS)\n                x = x.reshape((len(df), size, size, 1)).astype(np.float32)\n                y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n                yield x, y\n\ndef df_to_image_array(df, size, lw=6):\n    df['drawing'] = df['drawing'].apply(ast.literal_eval)\n    x = np.zeros((len(df), size, size))\n    for i, raw_strokes in enumerate(df.drawing.values):\n        x[i] = draw_cv2(raw_strokes, size=size, lw=lw)\n    x = x / 255.\n    x = x.reshape((len(df), size, size, 1)).astype(np.float32)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d20f0992f56f63efafb633b2e992f2a00746a22"},"cell_type":"code","source":"valid_df = pd.read_csv(os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(NCSVS - 1)), nrows=10**5)\nx_valid = df_to_image_array(valid_df, size)\ny_valid = keras.utils.to_categorical(valid_df.y, num_classes=NCATS)\n\ncats = list_all_categories()\nid2cat = {k: cat.replace(' ', '_') for k, cat in enumerate(cats)}\ny_valid_in_words = []\nfor i in valid_df[['y']].values:\n    y_valid_in_words.append(id2cat[i[0]])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9413b82070175b63f940ddb2c29c53ffa2d666b"},"cell_type":"code","source":"train_datagen = image_generator(size=size, batchsize=batchsize, ks=range(NCSVS - 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4e239911c2464297c9939d3487ccf747869d89c","scrolled":true},"cell_type":"code","source":"callbacks = [\n    EarlyStopping(monitor='val_categorical_accuracy', patience=7, min_delta=0.001, mode='max'),\n    ReduceLROnPlateau(monitor='val_categorical_accuracy', factor=0.5, patience=5, min_delta=0.005,\n                      mode='max', cooldown=3)\n]\nhist = model.fit_generator(\n    train_datagen, steps_per_epoch=STEPS, epochs=100, verbose=1,\n    validation_data=(x_valid, y_valid),\n    callbacks = callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc49e3018e0075c4f191faffbbdc2c84e34853b9"},"cell_type":"markdown","source":"## Exercise 3. Breaking down the results of the classifier.\nSince it is only possible to update the notebook by committing the code and since the network results may slightly vary even if the same model is being trained, the results from Version 24 are being analysed. In this version, the input image was changed to be twice the size in both dimentions and an additional layer was added to the network. The network results can also be viewed from the current version of the code. They are a bit more easy to understand, as all the unnecessary output is removed. However, the results of the current version might slightly differ from the results of Version 24. Link to Version 24: https://www.kaggle.com/bloonz/black-white-cnn-lb-0-77-6966ad?scriptVersionId=7919141\n\nTable 1 shows classification accuracy for each country. The plot below shows the accuracy plotted against the number of images for every country, except a single outlier that had more than 30.000 images and would otherwise have made the plot hard to read (its accuracy value was close to the mean). Each red cross represents a country. When plotting the data it seems to form a Gaussian distribution. Even if some countries achieved an accuracy that is above the mean, it could be by chance due to a low sample size. \n\nIn Table 2 accuracy is calculated for every category and the results are sorted from lowest accuracy to the highest one. As it can be observed, categories such as cooler and garden hose have the smallest accuracy. Furthermore, categories such as ladder and rainbow have the highest accuracy scores. This result might indicate that there is a lack of a clear mental picture of some objects in the participants imagination. For example, if one is asked to draw a ladder, then most people might draw two parallel lines with some perpendicular lines in between; but when it comes to drawing a cooler, it might not be such an easy task to accomplish, given that different people might not have the same mental picture of the object. Therefore, in the latter case, there might be more variety in the drawings. Besides that, it might appear difficult to some people to draw complex objects, like a garden hose, given a relatively simple drawing tool and having average drawing skills. \n\nFurthermore, five images with the correct classification are shown next to 5 images where classification failed. Below, assumptions are provided as to why the model failed to classify each of the 5 latter images:\n\n1. The image belongs to the peanut category and the model suggested that it might be a tornado, tennis racquet or a blackberry. The reason for such a mistake could be the shape of the drawing, which is not that clear and can be mistaken for a lot of other objects. The image also contains a lot of noise.\n\n2. A participant was asked to draw a garden and the model classified it as being rollerskates, computer or a bulldozer. The reason behind the classification mistake might be that “garden” is a very broad term and different people may have various interpretations of it. Therefore, if the is no clear definition of the word, the drawings for that word might vary a lot.\n\n3. The ground truth for the drawing is “fireplace” and it was misclassified as being a lantern, passport or dishwasher. This might have happened because the drawing seems to be rotated and occupies only part of the screen, thus resembling a small object.\n\n4. The image class is candle and it was classified as being a hedgehog, campfire or bush. The reason might be that the drawing is very noisy, making it difficult to classify correctly.\n\n5. The participant was asked to draw a television. However, the drawing was classified as being either a map, fireplace or sandwhich. This might have happened because of the pattern in the middle of the drawing. This might have caused confusion, as such a pattern is, maybe, more likely to be seen in other objects. \n\n**References:**\n\n[1] Original kernel. URL: https://www.kaggle.com/gaborfodor/black-white-cnn-lb-0-77\n\n[2] S. Ioffe and C. Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: ArXiv e-prints (Feb. 2015). arXiv: 1502.03167 [cs.LG]."},{"metadata":{"trusted":true,"_uuid":"0540513f0592b6fb258a425ecb000b6b9f074726"},"cell_type":"code","source":"valid_predictions = model.predict(x_valid, batch_size=128, verbose=0)\ntop3_val = preds2catids_val(valid_predictions, valid_df[['y']].values)\nnewcol = valid_df[['countrycode']].values[:, 0]\ntop3_cats_fin = top3_val.assign(country = newcol)\n\n# comparison of classification accuracy based on country\ntp_country = 0\nfirst = True\nfive_correct_ex = 0\nfive_wrong_ex = 0\nfig, axs = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True, figsize=(15, 15))\nfor i, row in top3_cats_fin.iterrows():\n    if row['ground truth'] == row['a'] or row['ground truth'] == row['b'] or row['ground truth'] == row['c']:\n        tp_country = 1\n        # show 5 examples with correct prediction\n        if five_correct_ex < 5:\n            ax = axs[0, five_correct_ex % 5]\n            _ = ax.imshow(x_valid[i, :, :, 0], cmap=plt.cm.gray)\n            name = id2cat[row['ground truth']]\n            _ = ax.set_xlabel(name)\n            _ = ax.set_ylabel('Correct predict.')\n            five_correct_ex += 1\n    else:\n        # show 5 examples with wrong prediction\n        if five_wrong_ex < 5:\n            ax = axs[1, five_wrong_ex % 5]\n            _ = ax.imshow(x_valid[i, :, :, 0], cmap=plt.cm.gray)\n            name_gt = id2cat[row['ground truth']]\n            name_a = id2cat[row['a']]\n            name_b = id2cat[row['b']]\n            name_c = id2cat[row['c']]\n            name = 'GT: ' + name_gt + '\\n' + name_a + ' ' + name_b + ' ' + name_c\n            _ = ax.set_xlabel(name)\n            _ = ax.set_ylabel('Wrong predict.')\n            five_wrong_ex += 1\n        \n    if first:\n        d = {'Accuracy': [tp_country], '# of imgs': [1]}\n        result_country = pd.DataFrame(data=d, index=[row['country']])\n        first = False\n    else:\n        if row['country'] in result_country.index:\n            result_country.at[row['country'], 'Accuracy'] += tp_country\n            result_country.at[row['country'], '# of imgs'] += 1\n        else:\n            d = {'Accuracy': [tp_country], '# of imgs': [1]}\n            result_con = pd.DataFrame(data=d, index=[row['country']])\n            result_country = result_country.append(result_con)\n    tp_country = 0\nplt.tight_layout()\n_ = plt.show();\nplt.clf()\nresult_country = result_country[result_country['# of imgs'] < 30000]\nTP_p_con = result_country['Accuracy']/result_country['# of imgs']\nresult_country['Accuracy'] = TP_p_con\nprint('Table 1')\nresult_country.nsmallest(result_country.shape[0], 'Accuracy')\n_ = plt.plot(result_country['Accuracy'], result_country['# of imgs'], 'r+')\n_ = plt.xlabel('Accuracy per country')\n_ = plt.ylabel('Number of images per country')\n_ = plt.title('How many images per country vs accuracy per country. Data gathered from validation set.')\n        \n        \n# comparison of classification accuracy based on class\nprint('Table 2')\nfirst = True\nfor i in range(len(id2cat)):\n    temp = top3_val.loc[top3_val['ground truth'] == i]\n    temp_2 = temp.loc[(temp['a'] == i) | (temp['b'] == i) | (temp['c'] == i)]\n    tp = temp_2.shape[0]/temp.shape[0]\n    d = {'Accuracy': [tp], '# of imgs': [temp.shape[0]]}\n    if first:\n        result_class = pd.DataFrame(data=d, index=[id2cat[i]])\n        first = False\n    else:\n        result_cl = pd.DataFrame(data=d, index=[id2cat[i]])\n        result_class = result_class.append(result_cl)\nresult_class.nsmallest(len(id2cat), 'Accuracy')\n\nmap3 = mapk(valid_df[['y']].values, preds2catids(valid_predictions).values)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}