{"cells":[{"metadata":{"_uuid":"766dfca62587e46e8161291f00815bb12136d600"},"cell_type":"markdown","source":"<h2><center>Quick, Draw! Doodle Recognition Challenge - Simple EDA</center></h2>\n\n#### Here i'll do some simple analysis on the competition data set, if you have any suggestion or feedback please let me know in the comments."},{"metadata":{"_uuid":"353278f8014fc70293c17565cb616e2c1fbe4f51"},"cell_type":"markdown","source":"### Dependencies"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport ast\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.utils import shuffle\n\npd.options.display.max_rows = 20\nsns.set(style=\"darkgrid\")\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d6985cfbe02aec51631e0bdda63e48b45c25a86f"},"cell_type":"markdown","source":"### Load data"},{"metadata":{"trusted":true,"_uuid":"b63546a902016c0294282290900f0763db18a691","_kg_hide-input":false},"cell_type":"code","source":"train_sample = pd.DataFrame()\nfiles_directory = os.listdir(\"../input/train_simplified\")\nfor file in files_directory:\n    train_sample = train_sample.append(pd.read_csv('../input/train_simplified/' + file, index_col='key_id', nrows=10))\n# Shuffle data\ntrain_sample = shuffle(train_sample, random_state=123)\n\ntrain = pd.DataFrame()\nfor file in files_directory[:185]:\n    train = train.append(pd.read_csv('../input/train_simplified/' + file, index_col='key_id', usecols=[1, 2, 3, 5]))\n# Shuffle data\ntrain = shuffle(train, random_state=123)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e0726ba57e63a7ca891e77c8da96cca072435e9"},"cell_type":"markdown","source":"### Let's take a look at the data"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ee4f26f55f38e5212537e081546c48c6598d6d47"},"cell_type":"code","source":"print('Train number of rows: ', train.shape[0])\nprint('Train number of columns: ', train_sample.shape[1])\nprint('Train set features: %s' % train_sample.columns.values)\nprint('Train number of label categories: %s' % len(files_directory))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ab37155bea23945fb28c5efce96952c20fbc967d","_kg_hide-input":true},"cell_type":"code","source":"train_sample.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e84f24a4d311e6c6bab6c76b5d7b2d9f6db088ed"},"cell_type":"markdown","source":"Our dataset seems to be pretty simple, ids, timestamps, countrycodes what seems to really matters is the drawings values (arrays that represents the drawings), \"recognized\" that means if this drawing was recognized or not by the algorithm and \"word\" that is our labels, maybe \"countrycode\" and the timestamp may be cool features to play with as well."},{"metadata":{"_uuid":"226d25deaf487f0597c4230a3e6c95b6908a25e3"},"cell_type":"markdown","source":"### In case you wanna check the label count\n * I wold like to make a count plot of the label categories, but as we have 340 it would be a mess."},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"trusted":true,"_uuid":"3ddc570d3afaf052d3e8c66ac65167fe4645fc9a"},"cell_type":"code","source":"count_gp = train.groupby(['word']).size().reset_index(name='count').sort_values('count', ascending=False)\ntop_10 = count_gp[:10]\nbottom_10 = count_gp[count_gp.shape[0]-10:count_gp.shape[0]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36e3663e29caeb4b2f29f92253da70b858d4f34c"},"cell_type":"markdown","source":"### The top 10 words"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2a957c22d03805dfadbc1f3d13ebe475e3709327"},"cell_type":"code","source":"ax_t10 = sns.barplot(x=\"word\", y=\"count\", data=top_10, palette=\"coolwarm\")\nax_t10.set_xticklabels(ax_t10.get_xticklabels(), rotation=40, ha=\"right\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f0c6c1f2d7182eff6816e8474c285508fbe3820"},"cell_type":"markdown","source":"### The bottom 10 words"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1c08aeeca2952d1f47a3e9c73ae8d0bda48e5a9d"},"cell_type":"code","source":"ax_b10 = sns.barplot(x=\"word\", y=\"count\", data=bottom_10, palette=\"BrBG\")\nax_b10.set_xticklabels(ax_b10.get_xticklabels(), rotation=40, ha=\"right\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36845164975a03841c490f221074348c6b50ad9f","_kg_hide-input":true},"cell_type":"code","source":"count_gp","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f8c21f785e1a3e1b7a3f37f925ba9ee5638154a"},"cell_type":"markdown","source":"### Rate of recognized true/false rows"},{"metadata":{"trusted":true,"_uuid":"977c60238120c78fd0248b2016be0c9962af44b1","_kg_hide-input":true},"cell_type":"code","source":"sns.countplot(x=\"recognized\", data=train)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4a52f37f379d6f093195624638cb387c65f683d4"},"cell_type":"code","source":"rec_gp = train.groupby(['word', 'recognized']).size().reset_index(name='count')\nrec_true = rec_gp[(rec_gp['recognized'] == True)].rename(index=str, columns={\"recognized\": \"recognized_true\", \"count\": \"count_true\"})\nrec_false = rec_gp[(rec_gp['recognized'] == False)].rename(index=str, columns={\"recognized\": \"recognized_false\", \"count\": \"count_false\"})\nrec_gp = rec_true.set_index('word').join(rec_false.set_index('word'), on='word')\nrec_gp","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09dc0d718a61001589d5f1cff8c5de4b18a4a6b7"},"cell_type":"markdown","source":"### View data as  drawings"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"927139f2bfcab00aaddc5058ef9fc230c14adfaa"},"cell_type":"code","source":"words = train['word'].tolist()\ndrawings = [ast.literal_eval(pts) for pts in train[:9]['drawing'].values]\n\nplt.figure(figsize=(10, 10))\nfor i, drawing in enumerate(drawings):\n    plt.subplot(330 + (i+1))\n    for x,y in drawing:\n        plt.plot(x, y, marker='.')\n        plt.tight_layout()\n        plt.title(words[i]);\n        plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82d12747ce957b51cb2dd44a633aba2c02bdc7cb"},"cell_type":"markdown","source":"As we can see some of these may be really hard to get good results with a model, other look a lot easier, what i think may be the challenge here is to get good results with this amount of label categories, some of them can get drawing that look a lot like others, but as we have lots of data, maybe something like deep learning may get good results."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}