{"cells":[{"metadata":{"_uuid":"331fbee8c5b4b49ee05eefc98bba3b60fa3a2073"},"cell_type":"markdown","source":"> **Problem overview**\n\n\"Quick, Draw!\" was released as an experimental game to educate the public in a playful way about how AI works. The game prompts users to draw an image depicting a certain category, such as ”banana,” “table,” etc. The game generated more than 1B drawings, of which a subset was publicly released as the basis for this competition’s training set. That subset contains 50M drawings encompassing 340 label categories.\n\nSounds fun, right? Here's the challenge: since the training data comes from the game itself, drawings can be incomplete or may not match the label. You’ll need to build a recognizer that can effectively learn from this noisy data and perform well on a manually-labeled test set from a different distribution.\n\nYour task is to build a better classifier for the existing Quick, Draw! dataset. By advancing models on this dataset, Kagglers can improve pattern recognition solutions more broadly. This will have an immediate impact on handwriting recognition and its robust applications in areas including OCR (Optical Character Recognition), ASR (Automatic Speech Recognition) & NLP (Natural Language Processing)."},{"metadata":{"_uuid":"9da7fb8f7ae88df5934a3aff7811972fcd139542","trusted":true},"cell_type":"code","source":"# import python standard library\nimport os\n\n# import data manipulation library\nimport numpy as np\nimport pandas as pd\n\n# import data visualization library\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b1e96984cf9718ba06e595da35f4d0df3d37955"},"cell_type":"markdown","source":"> **Acquiring training and testing data**\n\nWe start by acquiring the training and testing datasets into Pandas DataFrames."},{"metadata":{"trusted":true},"cell_type":"code","source":"def csvload(file: str, nrows: int = None) -> pd.DataFrame:\n    \"\"\" Return a loaded csv file. \"\"\"\n    \n    return pd.read_csv('../input/train_simplified/' + file, nrows=nrows)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# class files and dictionary\nfiles = sorted(os.listdir('../input/train_simplified/'), reverse=False)\nclass_dict = {file[:-4].replace(\" \", \"_\"): i for i, file in enumerate(files)}\n\n# data dimensions\nnum_shuffles = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# acquiring training and testing data\nfor i, file in tqdm(enumerate(files)):\n    df_data = csvload(file, nrows=30000)\n    df_data['shuffle'] = (df_data['key_id'] // 10 ** 7) % num_shuffles\n    for k in range(num_shuffles):\n        df_chunk = df_data[df_data['shuffle'] == k]\n        if i == 0: df_chunk.to_csv('train_k%d.csv' %k, index=False)\n        else: df_chunk.to_csv('train_k%d.csv' %k, header=False, index=False, mode='a')            ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3cbed4acd836ea7a8f0bebf2940b435d8e4d7f37","trusted":true},"cell_type":"code","source":"# shuffle and compress file\nfor k in tqdm(range(num_shuffles)):\n    df_data = pd.read_csv('train_k%d.csv' %k)\n    print(df_data.shape)\n    df_data['rand'] = np.random.rand(df_data.shape[0])\n    df_data = df_data.sort_values(['rand']).drop(['rand'], axis=1)\n    df_data.to_csv('train_k%d.csv.gz' %k, index=False, compression='gzip')\n    \n    # memory clean-up\n    os.remove('train_k%d.csv' %k)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}