{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"### Easy Data Preprocessing\n--by Joo Kyung Song, <br>\n--Submit to \"Programmers, Winter Coding\" \n"},{"metadata":{},"cell_type":"markdown","source":"![test](https://user-images.githubusercontent.com/43398106/68266393-5769d400-0092-11ea-85b9-bc7b466d6799.gif)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### 데이터 전처리\n1. 데이터 규모 확인<br>\n2. 320개 주어진 csv 파일 shuffle하여 100개로 나눠 합침. 파일 규모가 너무 크기 때문에 csv.gz 파일로 압축해서 활용 <br>\n3. PIL 라이브러리 활용해 64*64 이미지로 convert 하여 그림으로 나타냄.<br>\n4. one_hot_coding 기법 이용: 324개의 y_label을 np.eye(324)를 사용해서 원-핫인코딩함. <br>\n"},{"metadata":{},"cell_type":"markdown","source":"### CNN 구현 -- keras 라이브러리 사용 \n\n네트워크 구성은 아래와 같음 <br>**** conv - relu - conv - relu - pool - <br> conv - relu - conv - relu - pool - <br> conv - relu - conv - relu - pool - <br>affine - relu - dropout - affine - dropout - softmax****<br>\n\noptimizer 기법은 adamOptimizer 사용, 오차계산법은 cross-entropy 사용함. \n"},{"metadata":{},"cell_type":"markdown","source":"### Required Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport ast\nimport numpy as np\nimport tensorflow as tf\nfrom tqdm import tqdm\nfrom glob import glob\nfrom dask import bag\nimport cv2\nfrom tensorflow import keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Activation, DepthwiseConv2D, BatchNormalization, ZeroPadding2D, Lambda\nfrom keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, AveragePooling2D , Layer\nfrom keras import optimizers\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\nfrom keras.metrics import top_k_categorical_accuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. ### Data Shuffle + Compression\n***\"beluga\" 님의 Shuffle Csv를 참고하여 만들었습니다***\n\n320개의 label을 가진 {}.csv 파일을 100개로 나누어서 서로 shuffle. \n처음에는 sklearn library의 shuffle함수를 쓰려고 했지만 14GB의 데이터를 전부 담을 배열이 필요했음.\n-> 아예 데이터 순서대로 각 chunk_size만큼 뽑아서 따로 csv파일을 만드는 것을 선택함\n\n***Data Shuffle과 Compression은 Kaggle 내 RAM 메모리 할당 문제 때문에 각주 처리함***\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n# cwd = os.getcwd()\nfile_path = r\"../input/quickdraw-doodle-recognition/train_simplified\"\nfiles = os.listdir(file_path)\nword_category = [f.split(\".\")[0] for f in files]\nchunk_size = 100\n\n# for index, word in enumerate(word_category):\n#     df = pd.read_csv(os.path.join(file_path, str(word+\".csv\")))\n#     for k in range(chunk_size):\n#         filename = 'train_{}.csv'.format(k)\n#         df['file_index'] = index\n#         df['cv'] = (df.key_id // 10 ** 7) % chunk_size\n#         chunk = df[df.cv == k]\n#         chunk = chunk.drop(['key_id'], axis=1)\n\n# print(\"===Data shuffle Finished===\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"===Data Compression Starting===\")    \n# for k in tqdm(range(chunk_size)):\n#     filename = 'train_{}.csv'.format(k)\n#     if os.path.exists(filename):\n#         df = pd.read_csv(filename)\n#         df['rnd'] = np.random.rand(len(df))\n#         df = df.sort_values(by='rnd').drop('rnd', axis=1)\n#         df.to_csv(filename + '.gz', compression='gzip', index=False)\n#         os.remove(filename)\n# print(\"===Data Compression Done==\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preprocessing"},{"metadata":{},"cell_type":"markdown","source":"1. 가장 먼저 PIL 라이브러리의 ImageDraw를 활용하여 주어진 데이터를 Height=64, Width=64, Channel=1 의 array로 바꿈\n2. One-hot encoding 실행. \n3. shuffle된 데이터 중 recognized가 True일때, df['drawing']을 ast.literal_eval 함수를 활용하여 string이 아닌 배열로 변환\n4. 3에서 변환된 데이터를 X라는 리스트 데이터에 저장함 \n5. 4에서 받은 데이터를 (64,64)로 reshape하여 새로운 X2 배열에 저장\n6. 원핫코딩한 y label을 Y2라는 배열에 저장함 \n\n***원래는 train_0.csv.gz 파일로 3~6 번째 단계를 실행하지만 Kaggle에서 데이터는 read-only data 이므로 horse.csv를 부득이하게 사용***"},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import ImageDraw, Image\ndef make_img(img_arr) :\n    image = Image.new(\"P\", (256,256), color=255)\n    image_draw = ImageDraw.Draw(image)\n    for stroke in img_arr:\n        for i in range(len(stroke[0])-1):\n            image_draw.line([stroke[0][i], \n                             stroke[1][i],\n                             stroke[0][i+1], \n                             stroke[1][i+1]],\n                            fill=0, width=5)\n    return image\n\n\ndef stroke_to_img(strokes): \n    img=np.zeros((256,256))\n    for each in ast.literal_eval(strokes):\n        for i in range(len(each[0])-1):\n            cv2.line(img,(each[0][i],each[1][i]),(each[0][i+1],each[1][i+1]),255,5)\n    img=cv2.resize(img,(32,32))\n    img=img/255\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"one_hot_encoding = np.eye(len(word_category))\ncategory_y_label = dict()\nindex = 0\nfor i in word_category:\n    category_y_label[i]=one_hot_encoding[index]\n    index+=1\n    \n# f = \"train_1.csv.gz\"\nf = os.path.join(file_path,\"horse.csv\")\ndf = pd.read_csv(f)\nX = []\nY = []\nnum=0\n# for i in df.values:\n#     if i[2]==True:   #recognized가 True일때 \n# #         x = make_img(ast.literal_eval(i[1]))\n# #         x = np.array(x.resize((64,64)))\n#         x = ast.literal_eval(i[1])\n#         X.append(x)\n#         Y.append(i[4])\n#         num+=1\n#         if n%1000 == 0:\n#             print(\"==={}번째 완료\".format(num))\nfor i in df.values:\n    if i[3]==True:   #recognized가 True일때 \n#         x = make_img(ast.literal_eval(i[1]))\n#         x = np.array(x.resize((64,64)))\n        x = ast.literal_eval(i[1])\n        X.append(x)\n        Y.append(i[5])\n        num+=1\n        if num%10000 == 0:\n            print(\"==={}번째 완료\".format(num))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X2 =[]\nn=0\nfor i in X:\n    x = make_img(i)\n    x = np.array(x.resize((64,64)))\n    X2.append(x)\n    if n%10000 == 0:\n        print(\"==={}번째 완료==\".format(n))\n    n+=1\nX2 = np.array(X2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y2 = []\nn = 0\nfor y in Y:\n    Y2.append(category_y_label[y])\n    if n%10000 == 0:\n        print(\"==={}번째 완료==\".format(n))\n    n+=1\nY2 = np.array(Y2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 데이터 그림으로 확인해보기"},{"metadata":{"trusted":true},"cell_type":"code","source":"index = 340\n\nfor key,value in category_y_label.items():\n    if str(Y2[index])==str(value):\n        print(key)\n\nfor x,y in X[index]:\n    plt.plot(x, -np.array(y), lw=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### IMAGE 기반 CNN 학습"},{"metadata":{},"cell_type":"markdown","source":"### GPU memory allocation을 줄여주는 코드"},{"metadata":{"trusted":true},"cell_type":"code","source":"## extra imports to set GPU options\nimport tensorflow as tf\nfrom keras import backend as k\n \n###################################\n# TensorFlow wizardry\nconfig = tf.compat.v1.ConfigProto()\n \n# Don't pre-allocate memory; allocate as-needed\nconfig.gpu_options.allow_growth = True\n \n# Only allow a total of half the GPU memory to be allocated\nconfig.gpu_options.per_process_gpu_memory_fraction = 0.5\n \n# Create a session with the above options specified.\ntf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))\n# k.backend.set_session(tf.compat.v1.Session(config=config))\n# k.tensorflow_backend.set_session(tf.compat.v1.Session(config=config))\n###################################","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_grand=[]\nnum_class = 340\nper_class=2000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_paths = glob('/kaggle/input/quickdraw-doodle-recognition/train_simplified/*.csv')\nfor i , c in enumerate(tqdm(class_paths[0:num_class])): \n    train=pd.read_csv(c,usecols=['drawing','recognized'],nrows=per_class*2)\n    train=train[train.recognized==True].head(per_class)\n    imagebag=bag.from_sequence(train.drawing.values).map(stroke_to_img)\n    train_array=np.array(imagebag.compute())\n    train_array=np.reshape(train_array,(per_class,-1))    \n    label_array=np.full((train.shape[0],1),i)\n    train_array=np.concatenate((label_array,train_array),axis=1)\n    train_grand.append(train_array)\ndel train_array\ndel label_array","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_grand=np.array([train_grand.pop() for i in np.arange(num_class)]) \nheight = 32\nwidth = 32\ntrain_grand=train_grand.reshape((-1,(height*width+1))) \nprint(train_grand)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"specific = 0.1 \nsequence_length = 50\ncut = int(specific * train_grand.shape[0])\nprint(cut)\n\nnp.random.shuffle(train_grand)\ny_train, X_train = train_grand[cut: , 0], train_grand[cut: , 1:]\ny_val, X_val = train_grand[0:cut, 0], train_grand[0:cut, 1:]\n\ndel train_grand\n\nx_train=X_train.reshape(X_train.shape[0],height,width,1)\nx_val=X_val.reshape(X_val.shape[0],height,width,1)\n\nprint(y_train.shape, \"\\n\",\n      x_train.shape, \"\\n\",\n      y_val.shape, \"\\n\",\n      x_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(32, (3, 3), strides=(1, 1), input_shape=(32, 32,1)))\nmodel.add(Activation('relu'))\n# model.add(ZeroPadding2D((1, 1)))\nmodel.add(Conv2D(32, (3, 3), strides=(1, 1)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2),strides=(2, 2),data_format=\"channels_last\"))\nmodel.add(Conv2D(64, (3, 3), strides=(1, 1)))\nmodel.add(Activation('relu'))\nmodel.add(ZeroPadding2D((1, 1)))\nmodel.add(Conv2D(64, (3, 3), strides=(1, 1)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2),strides=(2, 2),data_format=\"channels_last\"))\nmodel.add(Conv2D(128, (3, 3), strides=(1, 1)))\nmodel.add(Activation('relu'))\nmodel.add(ZeroPadding2D((1, 1)))\nmodel.add(Conv2D(128, (3, 3), strides=(1, 1)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2),strides=(2, 2)))\nmodel.add(Dropout(0.2))\nmodel.add(Flatten())\nmodel.add(Dense(256))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(num_class))\nmodel.add(Activation('softmax'))\n# model.compile(RMSprop(lr=self.learningRate), 'MSE')\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def top_3_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduceLROnPlat=ReduceLROnPlateau(monitor='val_loss',factor=0.5,patience=3,\n                                 verbose=1,mode='auto',min_delta=0.005,\n                                 cooldown=5,min_lr=0.0001)\n\ncallbacks=[reduceLROnPlat]\n\nmodel.compile(loss='sparse_categorical_crossentropy',optimizer='adam',\n              metrics=['accuracy',top_3_accuracy])\n\nhistory=model.fit(x=x_train,y=y_train,batch_size=32,epochs=200,\n                  validation_data=(x_val,y_val),callbacks=callbacks,verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"acc=history.history['accuracy']\nval_acc=history.history['val_accuracy']\nloss= history.history['loss']\nval_loss=history.history['val_loss']\n\nepochs=range(1,len(acc)+1)\n\nplt.plot(epochs,acc,label='Training acc')\nplt.plot(epochs,val_acc,label='Validation acc')\nplt.title('Training and validation accuracy')\nplt.legend()\n\nplt.figure()\n\nplt.plot(epochs,loss,label='Training loss')\nplt.plot(epochs,val_loss,label='Validation loss')\nplt.title('Training and validation loss')\nplt.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test Data Prediciton "},{"metadata":{"trusted":true},"cell_type":"code","source":"#%% get test set\nttvlist = []\nreader = pd.read_csv('../input/quickdraw-doodle-recognition/test_simplified.csv', index_col=['key_id'],\n    chunksize=2048)\nfor chunk in tqdm(reader, total=55):\n    imagebag = bag.from_sequence(chunk.drawing.values).map(stroke_to_img)\n    testarray = np.array(imagebag.compute())\n    testarray = np.reshape(testarray, (testarray.shape[0], 32, 32, 1))\n    testpreds = model.predict(testarray, verbose=0)\n    ttvs = np.argsort(-testpreds)[:, 0:3]  # top 3\n    ttvlist.append(ttvs)\n    \nttvarray = np.concatenate(ttvlist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"numstonames={i : v[:-4].replace(' ','_') for i , v in enumerate(os.listdir(file_path))}\npreds_df = pd.DataFrame({'first': ttvarray[:,0], 'second': ttvarray[:,1], 'third': ttvarray[:,2]})\npreds_df = preds_df.replace(numstonames)\npreds_df['words'] = preds_df['first'] + \" \" + preds_df['second'] + \" \" + preds_df['third']\n\nresult = pd.read_csv('../input/quickdraw-doodle-recognition/sample_submission.csv', index_col=['key_id'])\nresult['word'] = preds_df.words.values\nresult.to_csv('submission.csv')\nresult.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### test_simplified 그림으로 확인하기"},{"metadata":{"trusted":true},"cell_type":"code","source":"df2 = pd.read_csv('../input/quickdraw-doodle-recognition/test_simplified.csv', nrows=100)\ndf2['drawing'] = df2['drawing'].apply(ast.literal_eval)\nn = 10\nfig, axs = plt.subplots(nrows=n, ncols=n, sharex=True, sharey=True, figsize=(16, 10))\nfor i, drawing in enumerate(df2['drawing']):\n    ax = axs[i // n, i % n]\n    for x, y in drawing:\n        ax.plot(x, -np.array(y), lw=3)\nplt.savefig(\"test_img.png\")\nplt.show();","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}