{"cells":[{"metadata":{"_uuid":"4a33e396d43e595c4d98bfeb76321a43ae81a058"},"cell_type":"markdown","source":"https://www.kaggle.com/c/quickdraw-doodle-recognition //  'Quick! Draw! Doodle Recognition'의 원 커널은 여기 있습니다.<br>\n원본이 영어인 만큼, 해석하여 올립니다. ~~번역한다고 쓴 글이 영어랑 안 맞을 수 있어서 원본도 같이 올립니다. 정 이상하다 싶으면 번역기를 돌리겠습니다(혹은 돌리는 것을 추천합니다)~~\n\n스케치 데이터 갖고 논 경험을 나누기 위해 이 커널을 만들었습니다. 웹사이트랑 데이터들을 찾아보고, 스케치물을 분류하기 위해 간단한 CNN을 사용할 겁니다.\n\n\n## Quick, Draw\n이거 좀 재밌어요. [빨리 그리기](htps://quickdraw.withgoogle.com/#) 이 사이트로 가서 몇몇 물체에 대한 스케치를 그릴 수 있습니다 (저번에 어딘가에 걸려있었던 이 링크를 따라가서 해 봤는데 재밌더라구요.)<br>\n거기 가면, 20초 간격으로, '뭐 그리세요' 하고 던져놓고 그리라고 시킵니다. 만약에 인공지능(AI)이 그 스케치를 맞추면(같은 라벨(이름)이 붙은 훈렷 데이터셋과 맞춰볼 수 있으면) 맞았다고 표시를 하고 다른 스케치를 그리게끔 진행합니다. 맨 끝에 가면 이런 결과물을 볼 수 있어요.\n\n<img src=\"https://s3.amazonaws.com/nonwebstorage/Screenshot+from+2018-09-26+22-44-21.png\" alt=\"drawing1\" width=\"600\"/>\n  \n<p><br></p>\n... AI가 보기엔 제가 그린 새 그림이 용, 떨어지는 보드나 모기같아 보였나 보군요. 이렇게 AI가 추측한 그림들 중 제일 가깝다고 본 라벨(이름) 3개를 알려줘요. 그리고... 다른 사람들이 그린 새를 보여줄 겁니다. 저거 보고 더 잘 그리라고 말이죠.\n<img src=\"https://s3.amazonaws.com/nonwebstorage/Screenshot+from+2018-09-26+22-45-21.png\" alt=\"drawing1\" width=\"600\"/>\n\n```\n\nI wrote this kernel to mess around with the sketch data and share my experience so far.  After exploring the website and the data, I'll use a very basic CNN to classify sketches. This model gets 0.60 on the Public LB when run with 6000 recognized images per class.\n\n## Quick, Draw\n\nOk - I have to say this is kind of a fun thing. You go to the [Quick Draw](https://quickdraw.withgoogle.com/#)  website and agree to sketch several common objects. The host then gives you the object labels one by one, and you have 20 seconds to draw each one. If the AI guesses your sketch (that is, associates it with training set items of the same label) you get a check mark and move on. At the end you get something like this:\n\n<img src=\"https://s3.amazonaws.com/nonwebstorage/Screenshot+from+2018-09-26+22-44-21.png\" alt=\"drawing1\" width=\"600\"/>\n\n<p><br></p>\nNotice that it didn't like my bird. Apparently it looked more like a dragon or a diving board(??) or a mosquito.  They're nice enough to show you how other people draw birds so you maybe learn how to draw better.\n<img src=\"https://s3.amazonaws.com/nonwebstorage/Screenshot+from+2018-09-26+22-45-21.png\" alt=\"drawing1\" width=\"600\"/>\n```"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Training Images\n스케치 몇 개를 보고 훈련 데이터에 뭐가 있는지 보겠습니다. Inversion의 'Getting Started' 커널을 쓸 거에요.\n\n```\nWe can look at a few sketches and then see what the training data contains overall. I'll use an adaptation of Inversion's 'Getting Started' kernel.\n```"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport re\nfrom glob import glob\nfrom tqdm import tqdm\nimport numpy as np\nimport pandas as pd\nimport ast\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"376d2827a7f11e80f6591bfc439a9d5558ef10fc"},"cell_type":"markdown","source":"이 DataFrame draw_df는 실제론 훈련 셋에 있는 .csv 파일들 중 6개에서 각각 맨 앞 2줄의 데이터를 모아놓은 것입니다.\n\n```\nHere's what the training data looks like. This data frame is actually a concatenation of two rows from each of 6 csv files in the training set.\n```"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5090c701f9121207b67e13a0ae3f239514dff217","scrolled":true},"cell_type":"code","source":"fnames = glob('../input/train_simplified/*.csv')\ncnames = ['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word']\ndrawlist = []\nfor f in fnames[0:6]:\n    first = pd.read_csv(f, nrows=10) # make sure we get a recognized drawing\n    first = first[first.recognized==True].head(2)\n    drawlist.append(first)\ndraw_df = pd.DataFrame(np.concatenate(drawlist), columns=cnames)\ndraw_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce444dda1ba6bc08e9a7a7cc5fb09a02855ef2e9"},"cell_type":"markdown","source":"그림 그린 데이터는 이렇게 숫자들의 배열로 저장됩니다. 보아하니, 데이터 이미지에서 찍힌 점들의 (x, y) 좌표를 (y * 이미지 한변의 길이 + x)로 변환하고, 이 변환된 점들을 이어진 순서대로 저장한 듯 합니다. 분리된 - 연결되어 있지 않은 - 선의 경우엔 또다른 배열에 저장했구요. 가령...\n```\ndrawing.values = [[선1점1, 선1점2, 선1점3, ... 선1점n], [선2점1, 선2점2, 선2점3, ... 선2점n], ..., [선i점1, 선i점2, 선i점3, ... 선i점n]]\n```\n이렇게 저장되어 있는 듯 합니다. 밑에 있는 한 줄의 코드는 이러한 데이터를 보여줍니다."},{"metadata":{"trusted":true,"_uuid":"9fb589a3124831d088c853864ed749891c6d2c8d"},"cell_type":"code","source":"draw_df.drawing.values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d385aa310c192e16f290e1756c27e06c48a22ec"},"cell_type":"markdown","source":"숫자로 보면 뭐가 뭔지 모르니, 그림 그린 걸로 직접 표현해 봅시다.\n\n```\nAnd here are some nice sketches...\n```"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2be57621533d0e0aa75108febde01261d2444c37"},"cell_type":"code","source":"evens = range(0,11,2)\nodds = range(1,12,2)\n\n# We have drawing images, 2 per label, consecutively\ndf1 = draw_df[draw_df.index.isin(evens)] # drawing images A of each label(object)\ndf2 = draw_df[draw_df.index.isin(odds)] # drawing images B of each label(object)\n\n# ast.literal_eval(pts) is similar to __builtins__.eval(pts), since each element of draw_df.drawing.values has a type of string, not an array\nexample1s = [ast.literal_eval(pts) for pts in df1.drawing.values]\nexample2s = [ast.literal_eval(pts) for pts in df2.drawing.values]\n\n#print(\"evens = \", evens)\n#print(\"odds = \", odds)\n#print(\"df1 = \", df1)\n#print(\"df2 = \", df2)\n#print(\"example1s = \", example1s)\n#print(\"example2s = \", example2s)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"0a1aaca9129ad3748e597993c90ab65c5fdefd5b"},"cell_type":"code","source":"labels = df2.word.tolist()\n\nprint(labels)\n\nfor i, example in enumerate(example1s):\n    plt.figure(figsize=(6,3))\n    \n    for x,y in example:\n        plt.subplot(1,2,1)\n        plt.plot(x, y, marker='.')\n        plt.axis('off')\n\n    for x,y, in example2s[i]:\n        plt.subplot(1,2,2)\n        plt.plot(x, y, marker='.')\n        plt.axis('off')\n        label = labels[i]\n        plt.title(label, fontsize=10)\n\n    plt.show()  ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf8bc6e13cba236c6632cff33d9729d6eff728f4"},"cell_type":"markdown","source":"## The Quick, Draw Data Repository\n\n[GitHub Repository](https://github.com/googlecreativelab/quickdraw-dataset) // Numpy 비트맵 파일 등등 상당히 쓸만한 데이터가 있는 곳이 있습니다(Kaggle 사이트는 아니에요) 각각의 파일들은 특정 Label에 대한 Sketch 데이터들을 가지고 있으며 1차원 배열의 형태로 존재합니다. 여기 있는 건 그 중 일부에요.\n\n수정 : 이 프로젝트에 쓰이는 테스트 셋들은 NumPy 비트맵 형식으로 만들어진 파일들을 가지고 있지 않습니다. matplotlib에서 변환한 배열들을 테스트하게 된다면, 원래 변환 과정과의 차이/변화로 인해 손실이 있을 겁니다. 이미지 기반 기계학습 모델이 결과를 제일 좋게 내는 경우는 훈련용 데이터랑 시범용 데이터를 똑같이 처리하는 것이겠죠. 그러니... 이 방법도 좋긴 한데, 더 좋은 결과물을 내고 싶으면 다른 파일들도 같이 보는 게 어떨까 싶습니다.\n\n```\nThere is a direct data source outside of Kaggle that seems pretty useful. The main link is a [GitHub Repository](https://github.com/googlecreativelab/quickdraw-dataset) that leads to the data in several formats, including Numpy bitmap files. Each file in the dataset covers a specific type of sketch and is in the shape of a long 1d array. Here are a few.\n\nUpdate: The test set for this project doesn't have premade files in the numpy bitmap format. When you apply your model to test arrays converted from matplotlib, there is a loss due to the different conversion process. The best results for an image-based model are had by using the same processing for train and test (no surprise there I guess). So I'd say these are good for exploration but consider the other files for better results.\n```"},{"metadata":{"trusted":true,"_uuid":"73e317f188154d2b908f75ee8aeafc06da86db42","scrolled":true,"_kg_hide-input":false},"cell_type":"code","source":"# commented out to save memory\n\nimport urllib\n\nLABELS = np.array(['baseball', 'bowtie', 'clock', 'hand', 'hat'])\nfor b in LABELS:\n    url = \"https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/{}.npy\".format(b)\n    urllib.request.urlretrieve(url, \"{}.npy\".format(b)) # download baseball.npy, bowtie.npy, ..., hat.npy, from the url above\n    nb = np.load(\"{}.npy\".format(b))\n    print(\"\\n Class '{0}' has {1} examples of {2}x{2} images\".format(b, nb.shape[0], int(nb.shape[1]**0.5)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b9f5b85771add0d7672f7611a62e4668a28453c"},"cell_type":"markdown","source":"## Convolutional Neural Network (CNN)\n\n이제 이미지를 구별하는 모델을 만들어 볼 겁니다. Repository에, 전에 사람들이 데이터를 어떻게 썼는지 보여주었던 자료들이 있습니다. 그 중 하나가 여기에서 쓰일 CNN입니다.\n\n그린 그림을 이미지로 바꾸는 데에서 자원(리소스)를 많이 쓰는 것 같습니다. 여기서 하는 획(한줄) 하나하나에 따른 모델을 그대로 쓸 수도 있지만, 변환 과정을 거칠 수도 있습니다. 다만, 이 때 데이터가 공간을 차지하는 것에 신경을 써야 할 겁니다.\n\n```\nNext we'll build an image classifier. There are some resources in the repository I mentioned earlier showing how people have used the data. One of those resources is a CNN just like what you see here. \n\nThe biggest usage of resources seems to be converting the drawings to images. You can stick with a stroke-based model or go down the conversion path. Going that way requires watching data usage and managing space limits - deep learning on Kaggle can be like building a ship in a bottle:) \n\nUPDATE: I'll be experimenting with a separate kernel to use a fit_generator like in Beluga's kernel and others. This kernel has the most I could squeeze into memory with simple fit and predict.\n```"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"adc697e3e58ef810c0576614ae95fa889bfb3345"},"cell_type":"code","source":"%reset -f\n# reset this program, deleting all pre-made variables, modules, functions, etc, that were before this cell","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"779113b4b0ef5fbe50ddda59813b019b3b769b58"},"cell_type":"code","source":"#%% import\nimport os\nfrom glob import glob\nimport re\nimport ast\nimport numpy as np \nimport pandas as pd\nfrom PIL import Image, ImageDraw \nfrom tqdm import tqdm\nfrom dask import bag\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.metrics import top_k_categorical_accuracy\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44e5965471a53f3ca6d87a1df8c0e1b2538cd34c"},"cell_type":"code","source":"#%% set label dictionary and params\nclassfiles = os.listdir('../input/train_simplified/')\nnumstonames = {i: v[:-4].replace(\" \", \"_\") for i, v in enumerate(classfiles)} #adds underscores\n\nnum_classes = 340    #340 max \nimheight, imwidth = 32, 32 # size of an image\nims_per_class = 2000  #max? # in the code above and above, there existed more than 100 thousand images per class(/label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ce1101eea3b2a0787c1dbe6b675e2a481802f67","_kg_hide-output":false,"scrolled":true},"cell_type":"code","source":"# faster conversion function\ndef draw_it(strokes):\n    image = Image.new(\"P\", (256,256), color=255)\n    image_draw = ImageDraw.Draw(image)\n    for stroke in ast.literal_eval(strokes):\n        for i in range(len(stroke[0])-1):\n            image_draw.line([stroke[0][i], \n                             stroke[1][i],\n                             stroke[0][i+1], \n                             stroke[1][i+1]],\n                            fill=0, width=5)\n    image = image.resize((imheight, imwidth))\n    return np.array(image)/255.\n\n#%% get train arrays\ntrain_grand = []\nclass_paths = glob('../input/train_simplified/*.csv')\nfor i,c in enumerate(tqdm(class_paths[0: num_classes])):\n    train = pd.read_csv(c, usecols=['drawing', 'recognized'], nrows=ims_per_class*5//4)\n    train = train[train.recognized == True].head(ims_per_class)\n    imagebag = bag.from_sequence(train.drawing.values).map(draw_it) \n    trainarray = np.array(imagebag.compute())  # PARALLELIZE\n    trainarray = np.reshape(trainarray, (ims_per_class, -1))    \n    labelarray = np.full((train.shape[0], 1), i)\n    trainarray = np.concatenate((labelarray, trainarray), axis=1)\n    train_grand.append(trainarray)\n    \ntrain_grand = np.array([train_grand.pop() for i in np.arange(num_classes)]) #less memory than np.concatenate\ntrain_grand = train_grand.reshape((-1, (imheight*imwidth+1)))\n\ndel trainarray\ndel train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6788c33f4a9dd552d473e3667387491c40dbfe4b","_kg_hide-input":false,"scrolled":true},"cell_type":"code","source":"# memory-friendly alternative to train_test_split?\nvalfrac = 0.1\ncutpt = int(valfrac * train_grand.shape[0])\n\nnp.random.shuffle(train_grand)\ny_train, X_train = train_grand[cutpt: , 0], train_grand[cutpt: , 1:]\ny_val, X_val = train_grand[0:cutpt, 0], train_grand[0:cutpt, 1:] #validation set is recognized==True\n\ndel train_grand\n\ny_train = keras.utils.to_categorical(y_train, num_classes)\nX_train = X_train.reshape(X_train.shape[0], imheight, imwidth, 1)\ny_val = keras.utils.to_categorical(y_val, num_classes)\nX_val = X_val.reshape(X_val.shape[0], imheight, imwidth, 1)\n\nprint(y_train.shape, \"\\n\",\n      X_train.shape, \"\\n\",\n      y_val.shape, \"\\n\",\n      X_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24ef673ac1b339a6449a2e20ecf946b40dbc0ca3"},"cell_type":"markdown","source":"CNN 구조는 이렇게 되어 있습니다. 생각보다 간단하죠?\n```\nHere's the architecture for the CNN. It's fairly simple compared to what else you can do.\n```"},{"metadata":{"trusted":true,"_uuid":"002b62eec9f0b5c139c4c08f82476433ccc11026"},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', input_shape=(imheight, imwidth, 1)))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Conv2D(64, kernel_size=(3, 3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.2))\nmodel.add(Flatten())\nmodel.add(Dense(680, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(num_classes, activation='softmax'))\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6464e84f6f0c7303110db9e26c74a5c90a0a9ab9","_kg_hide-output":true,"scrolled":false},"cell_type":"code","source":"def top_3_accuracy(x,y): \n    t3 = top_k_categorical_accuracy(x,y, 3)\n    return t3\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3, \n                                   verbose=1, mode='auto', min_delta=0.005, cooldown=5, min_lr=0.0001)\nearlystop = EarlyStopping(monitor='val_top_3_accuracy', mode='max', patience=5) \ncallbacks = [reduceLROnPlat, earlystop]\n\nmodel.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy', top_3_accuracy])\n\nmodel.fit(x=X_train, y=y_train,\n          batch_size = 32,\n          epochs = 22,\n          validation_data = (X_val, y_val),\n          callbacks = callbacks,\n          verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4125359749cd237de207eca0ab62074431df8323"},"cell_type":"markdown","source":"## Predicting on the Test data\nCNN이 검증 자료에선 잘 돌아가는 것 같군요.  훈련용 데이터 일부에서도 괜찮게 돌아가는 것 같습니다. 그러면... 시범용 셋을 주고 예측하게 시켜 봅시다.\n\n```\nThe CNN does OK on the validation data, even with a basic model and limited training data. Let's generate predictions on the test set and submit.\n```"},{"metadata":{"trusted":true,"_uuid":"2828f46cdf053261a1435e6eca1a4aea55195273","_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"#%% get test set\nttvlist = []\nreader = pd.read_csv('../input/test_simplified.csv', index_col=['key_id'],\n    chunksize=2048)\nfor chunk in tqdm(reader, total=55):\n    imagebag = bag.from_sequence(chunk.drawing.values).map(draw_it)\n    testarray = np.array(imagebag.compute())\n    testarray = np.reshape(testarray, (testarray.shape[0], imheight, imwidth, 1))\n    testpreds = model.predict(testarray, verbose=0)\n    ttvs = np.argsort(-testpreds)[:, 0:3]  # top 3\n    ttvlist.append(ttvs)\n    \nttvarray = np.concatenate(ttvlist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c650c6f9a0ec6f99042e6132ffc4a25b2b2f0b3","_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"preds_df = pd.DataFrame({'first': ttvarray[:,0], 'second': ttvarray[:,1], 'third': ttvarray[:,2]})\npreds_df = preds_df.replace(numstonames)\npreds_df['words'] = preds_df['first'] + \" \" + preds_df['second'] + \" \" + preds_df['third']\n\nsub = pd.read_csv('../input/sample_submission.csv', index_col=['key_id'])\nsub['word'] = preds_df.words.values\nsub.to_csv('subcnn_small.csv')\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"494ef170cb02e2425e82a897a0ca6c794c676b5a"},"cell_type":"markdown","source":" A full version with 6000 images per class at 28x28 gets just under 0.60 on the public LB.  "},{"metadata":{"trusted":true,"_uuid":"f75a34af14218daf6865999b19d2766b48f339a2"},"cell_type":"code","source":"import sys\n\n# These are the usual ipython objects, including this one you are creating\nipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']\n\n# Get a sorted list of the objects and their sizes\nsorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not \n    x.startswith('_') and x not in sys.modules and x \n    not in ipython_vars], key=lambda x: x[1], reverse=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}