{"cells":[{"metadata":{"_uuid":"a8fa809decd9b982621a76a6ea39724842c7fc42"},"cell_type":"markdown","source":"# Lets Play With Quick Draw!!"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"![Imgur](https://i.imgur.com/l7LgSGy.png)"},{"metadata":{"_uuid":"eaa29700a9ad14bf7c70e3508149808403c4d273"},"cell_type":"markdown","source":"# 📡 What is Quick Draw?\n![Imgur](https://i.imgur.com/HyPUqwF.png)\n“Quick, Draw!” was a game that was initially featured at Google I/O in 2016, as a game where one player would be prompted to draw a picture of an object, and the other player would need to guess what it was. Just like pictionary.\nIn 2017, the Magenta team at Google Research took that idea a step further by using this labeled dataset to train the [Sketch-RNN](https://magenta.tensorflow.org/assets/sketch_rnn_demo/index.html) model, to try to predict what the player was drawing, in real time, instead of requiring a second player to do the guessing. The game is [available online,](https://quickdraw.withgoogle.com) and has now collected over 1 billion hand-drawn doodles!"},{"metadata":{"_uuid":"63905e199f979fa5d3d4087bc7d0880ae9f3f5b1"},"cell_type":"markdown","source":"# 📜 Overview Of Quick Draw DataSet\nThe team has open sourced this data, and in a variety of formats. You can learn more at their [GitHub page.](https://github.com/googlecreativelab/quickdraw-dataset)Now we can get this dataset easily from [Our Data Science House Kaggle.](https://www.kaggle.com)\n\n**Here is Orginal Google DataSet demo picture.This picture collect from github. 👇** ![Imgur](https://i.imgur.com/MOziCCc.png)\n\n**Another picture of Kaggle DataSet picture. 👇 **![Imgur](https://i.imgur.com/36cyzNX.png)"},{"metadata":{"_uuid":"e6af65f9cd4c3a936a5caf8d9df070a190aeebe7"},"cell_type":"markdown","source":"There are 4 formats: First up are the raw files stored in (.ndjson) format. These files encode the full set of information for each doodle. It contains timing information for each stroke of every picture drawn.\n\nThere is also a simplified version, stored in the same format (.ndjson), which has some preprocessing applied to normalize the data. The simplified version is also available as a binary format for more efficient storage and transfer. There are examples of how to read the files using both Python and NodeJS.\n\n**This picture Google Cloud Platfrom of Quick Draw Datasets. 👇** ![Imgur](https://i.imgur.com/eYoCpkA.png)\n\n\nThe fourth format takes the simplified data and renders it into a 28x28 grayscale bitmap in numpy .npy format, which can be loaded using np.load().\n\nWhy is it 28x28? Well, it’s a perfect replacement for any existing code you might have for processing MNIST data. So if you’re looking for something fancier than 10 handwritten digits, you can try processing over 300 different classes of doodles."},{"metadata":{"_uuid":"f664867a714c0df8a6ad9a23f25a85b86a800f1c"},"cell_type":"markdown","source":"# 📊 Data exploration and visualization of Quick Draw Game\nIf you want to explore the dataset some more, you can visualize the quickdraw dataset using Facets. The Facets team has even taken the liberty of hosting it online and giving us some presets to play around with! You can [access the page here.](https://pair-code.github.io/facets/quickdraw.html) We can load up some random chairs and see how different players drew chairs from around the world.\n\n![Quick Draw gift](https://i.imgur.com/q1h49cE.gif)\n\n"},{"metadata":{"_kg_hide-input":true,"_uuid":"47ceaec0da7200f693c683ed991c43a4002db916","trusted":true},"cell_type":"code","source":"%%html\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd87bbd05d388558cd50a6543c1d84600e27fe36"},"cell_type":"markdown","source":"# 📑 Import Libraries"},{"metadata":{"_kg_hide-input":true,"_uuid":"559b0ac0126aff8fe1119c263f58cd493a31a1f0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\n\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.svm import SVC, LinearSVC\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.neural_network import MLPClassifier\n\nfrom sklearn.metrics import accuracy_score, confusion_matrix, roc_curve, roc_auc_score\nfrom sklearn.model_selection import train_test_split, GridSearchCV\n#deep lerning libraries\nimport keras\nfrom keras import backend as K\nfrom keras.utils import np_utils\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\n\nimport pickle # Read/Write with Serialization\nimport requests # Makes HTTP requests\nfrom io import BytesIO # Use When expecting bytes-like objects","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af2f869ed6a564543fcd42131662dcc5c603152d"},"cell_type":"markdown","source":"# 📥 Load and Read DataSets"},{"metadata":{"_uuid":"101c04df57140b6667fe2d3c771d737b17aab728","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Classes we will load\ncategories = ['cannon','eye', 'face', 'nail', 'pear','piano','radio','spider','star','sword']\n\n# Dictionary for URL and class labels\nURL_DATA = {}\nfor category in categories:\n URL_DATA[category] = 'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/' + category +'.npy'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"a830849886ba04ad738632b32068159296efb53a","trusted":true},"cell_type":"code","source":"classes_dict = {}\nfor key, value in URL_DATA.items():\n response = requests.get(value)\n classes_dict[key] = np.load(BytesIO(response.content))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"1161f996a355571d48d9722c576ec3515f965381","trusted":true},"cell_type":"code","source":"for i, (key, value) in enumerate(classes_dict.items()):\n value = value.astype('float32')/255.\n if i == 0:\n classes_dict[key] = np.c_[value, np.zeros(len(value))]\n else:\n classes_dict[key] = np.c_[value,i*np.ones(len(value))]\n\n# Create a dict with label codes\nlabel_dict = {0:'cannon',1:'eye', 2:'face', 3:'nail', 4:'pear', \n 5:'piana',6:'radio', 7:'spider', 8:'star', 9:'sword'}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"066fdbb37dba801f861566fd675f0725a4dafce0","trusted":true},"cell_type":"code","source":"lst = []\nfor key, value in classes_dict.items():\n lst.append(value[:3000])\ndoodles = np.concatenate(lst)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"1f3012b9c9d4b9d217e83a480e72f9c48c0e16c4","trusted":true},"cell_type":"code","source":"# Split the data into features and class labels (X & y respectively)\ny = doodles[:,-1].astype('float32')\nX = doodles[:,:784]\n\n# Split each dataset into train/test splits\nX_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3,random_state=1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"9147d9fcdb3dfd08fa4737e71d42940e17c9c25c","trusted":true},"cell_type":"code","source":"# Save X_train dataset as a pickle file\nwith open('xtrain_doodle.pickle', 'wb') as f:\n pickle.dump(X_train, f)\n \n# Save X_test dataset as a pickle file\nwith open('xtest_doodle.pickle', 'wb') as f:\n pickle.dump(X_test, f)\n \n# Save y_train dataset as a pickle file\nwith open('ytrain_doodle.pickle', 'wb') as f:\n pickle.dump(y_train, f)\n \n# Save y_test dataset as a pickle file\nwith open('ytest_doodle.pickle', 'wb') as f:\n pickle.dump(y_test, f)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8294823da84667f746ccf5b3bed0247a67b4bd5c"},"cell_type":"markdown","source":"# 💻 Predictive Modeling "},{"metadata":{"_kg_hide-input":false,"_uuid":"812183abf7e315c0a92b3c2a6de39df28e146f80"},"cell_type":"markdown","source":"# 📌 1. K-Nearest Neighbors\n![Imgur](https://i.imgur.com/HyPUqwF.png)\n![Imgur](https://i.imgur.com/IO58pDy.jpg)"},{"metadata":{"_uuid":"175006868a5ffa1ff04a39fc1299fecef62707eb"},"cell_type":"markdown","source":"### 📒 Note:\nBy simply randomly guessing, one should be able to reach ~10% accuracy (since there are only ten class labels). A machine learning algorithm will need to obtain > 10% accuracy in order to demonstrate that it has in fact “learned” something (or found an underlying pattern in the data).\n\nTo start, we’ll model the data with the k-Nearest Neighbor (k-NN) classifier, arguably the most simple, easy to understand machine learning algorithm. The k-NN algorithm classifies unknown data points by finding the most common class among the k-closest examples. Each data point in the k closest examples casts a vote and the category with the most votes is chosen."},{"metadata":{"_uuid":"2dc4e28b3e3dd132b85e57942a68374b06b0c232"},"cell_type":"markdown","source":"## 🔧 Base model Of KNN\n\nNext, I will try out a KNN classifier:"},{"metadata":{"_uuid":"e5a4b0b49d13cac7726c2a807e83462258b74770"},"cell_type":"markdown","source":"### Output:\n\n```Accuracy:80.5 ```"},{"metadata":{"_uuid":"252460f53958f66ea978da7089b8d7255df1c9dc"},"cell_type":"markdown","source":"## 🔢 Tuning number of neighbors\nThe KNN classifier looks promising, let's test different values of K:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d1c36ca36339faf91b4e5a55f0fb6a3df76bde5c"},"cell_type":"markdown","source":"![Imgur](https://i.imgur.com/vwadkbv.png)"},{"metadata":{"_uuid":"8695f39fab0ae99e804521505ed83df24fa8c2a1"},"cell_type":"markdown","source":"## 📉 Plot results of grid search\n"},{"metadata":{"_uuid":"43fb199fbcd81541aa3f9e4f0f01645090cb3cda"},"cell_type":"markdown","source":"### Output:\n![Imgur](https://i.imgur.com/DBn5Cyj.png)"},{"metadata":{"_uuid":"41e52d69cb30755907f7cb3ec7c0c08ffe661165"},"cell_type":"markdown","source":"### 📒 Note:\nFrom examining our plot and using the elbow-method using 3 neighbors seems like the best choice to avoid overfitting. The main advantage of the KNN algorithm is that it performs well with multi-modal classes because the basis of its decision is based on a small neighborhood of similar objects. This is why its results were fairly high with 80%. The main disadvantage is the computational cost are very high and the results take far too long.**"},{"metadata":{"_uuid":"41adcb2d0e06794e28695c417e63d52275981ff6"},"cell_type":"markdown","source":"# 🏞 2. Random Forest"},{"metadata":{"_uuid":"21be748d22c2341855d4a0f21150c840ae9cb64b"},"cell_type":"markdown","source":"![Imgur](https://i.imgur.com/HyPUqwF.png)\n![Imgur](https://i.imgur.com/lREy3CV.jpg)\n![Imgur](https://i.imgur.com/lEuwiKK.jpg)"},{"metadata":{"_uuid":"b3678c8135a159f4111ef5f29d2049421215217f"},"cell_type":"markdown","source":"### 📒 Note:\nRandom forests is an ensemble model which means that it uses the results from many different models to calculate a label"},{"metadata":{"_uuid":"b6497a974c7a98efc5f92e28455a08e6b5bf1994"},"cell_type":"markdown","source":"## 🔧 Base RFC model\n"},{"metadata":{"_uuid":"0998bcdeb10ebe33643ffab96866fbfb2062fe2d"},"cell_type":"markdown","source":"### Output:\n```\naccuracy:74.4\n```"},{"metadata":{"_uuid":"2dbc61ec51c9005ca39d90a07ead4412f2adf905"},"cell_type":"markdown","source":"We then tuned the max_features parameters, which are the maximum number of\nfeatures Random Forest can try in individual tree. By limiting the max features to\nthe square root of total features we improved the model and made it computationally\nless expensive.\nWe then plotted the pixel importances and saw that the edges of the doodles tend to\nbe the most important."},{"metadata":{"_uuid":"47d607bc748fcd5e102573dd643cc65a8007c82d"},"cell_type":"markdown","source":"## 🔧 Tuning number of estimators in the ensemble method\n"},{"metadata":{"_uuid":"d6a4b63ba27a0f0dc1325c3fc62d34514a292cc2"},"cell_type":"markdown","source":"### Output:\n![Imgur](https://i.imgur.com/Ds6OVNz.png)"},{"metadata":{"_uuid":"200b48818026f75f953ad9879c60aaa4f6872948"},"cell_type":"markdown","source":"## 📉 Plot results of grid search\n"},{"metadata":{"_uuid":"c85c97951ce1a0374b163f7e903e515e1474dc4d"},"cell_type":"markdown","source":"### Output:\n![Imgur](https://i.imgur.com/wKOndoI.png)"},{"metadata":{"_uuid":"5576aa641b9b673f25680d7e41994f87d60317f3"},"cell_type":"markdown","source":"## 🔧 Tuning max features"},{"metadata":{"_uuid":"28b699e5f93941539fdb2703a8af75a35949405a"},"cell_type":"markdown","source":"### Output:\n![Imgur](https://i.imgur.com/IbOqyhQ.png)"},{"metadata":{"_uuid":"6029b25cff90a6b2505ae23e3b35c2802b4271d8"},"cell_type":"markdown","source":"### Output:\n![Imgur](https://i.imgur.com/MhH5Wnz.png)"},{"metadata":{"_uuid":"c4ed3e1d75d25cc12e0bcff21d5a3826f8e18db6"},"cell_type":"markdown","source":"## Modeling RFC with best hyper-parameters"},{"metadata":{"_uuid":"31cb5cc30bea0584baf532f918ed4f04adf7a14f"},"cell_type":"markdown","source":"### Output:\n```\naccuracy:80.5\n```"},{"metadata":{"_uuid":"eb01a7bfe08a7e2b2db50d867de4910fbc094f27"},"cell_type":"markdown","source":"### Output:\n**Seeing what pixels are the most important in deciding the label\n**![Imgur](https://i.imgur.com/aJMgZJi.png)"},{"metadata":{"_uuid":"1cfe67f86dead933750b4ec5d6888dee6f344f62"},"cell_type":"markdown","source":"### 📒 Note:\nThe random forest had an accuracy score very close to the k-nn model. The features that are most important are on the edge and in the middle of each side."},{"metadata":{"_uuid":"b7ea9e50a43a6a01a90ec8691385f3b081f6e226"},"cell_type":"markdown","source":"# 📐 3.Support Vector Machine\n![Imgur](https://i.imgur.com/HyPUqwF.png)\n![Imgur](https://i.imgur.com/JHwkkBg.jpg)\n![Imgur](https://i.imgur.com/mILrxTO.png)\n"},{"metadata":{"_uuid":"722105671741bd94d897d157873b978e25274270"},"cell_type":"markdown","source":"### 📒 Note:\nSVM classification uses planes in space to divide data points. We can compared a linear divider a non-linear divider."},{"metadata":{"_uuid":"eaab7a76e76915b8a3c6bdf2e9e0775cf50c472e"},"cell_type":"markdown","source":"## 3.1.LinearSVC"},{"metadata":{"_uuid":"6511d49d4f0683632b5d8e4d82c0af478dd652a6"},"cell_type":"markdown","source":"### Output:\n```\nLinear SVC accuracy:71.7\n```"},{"metadata":{"_uuid":"f36bd79cfa06c83ee600848f3df5d2ee44cd382e"},"cell_type":"markdown","source":"## 3.2. Non-Linear SVM (Radial Basis Function)"},{"metadata":{"_uuid":"afa0f489ddd7293b01d69acf7045a36dd19c8e70"},"cell_type":"markdown","source":"### Output:\n```\nGaussian Radial Basis Function SVC Accuracy: 77.15\n```"},{"metadata":{"_uuid":"1eac31904dc50034bd41b257d540d165188f844c"},"cell_type":"markdown","source":"# 📚 4. Multi-Layer Perceptron\n\n![Imgur](https://i.imgur.com/HyPUqwF.png)\n\n![Imgur](https://i.imgur.com/1wtN0Ln.png)\n\n![Imgur](https://i.imgur.com/1S9bC75.jpg)"},{"metadata":{"_uuid":"4f7fc90c94ce62652e7472ea424cff1fb7ee8d2a"},"cell_type":"markdown","source":"### 📒 Note\n\nA perceptron is a neural network with a very basic architecture. This will be good to compare against the convolutional neural network. Neural Networks receive an input (a single vector), and transform it through a series of hidden layers. Each hidden layer is made up of a set of neurons, where each neuron is fully connected to all neurons in the previous layer, and where neurons in a single layer function completely independently and do not share any connections. The last fully-connected layer is called the “output layer” and in classification settings it represents the class scores.\n\nFor tuning the hyper-parameters for a Multi-Layer Perceptron, we can try different number of layers and number of neurons in each layer. The default activiation function (ReLu) is the most effective choice from sklearn and the the defualt optimization algorithm (adam) is the best choice due to the size of our dataset."},{"metadata":{"_uuid":"fdced6bf0bfeae483a4237127ba49ec1a2dab0e9"},"cell_type":"markdown","source":"### Output:\n```\nmlp accuracy: 81.9\n```"},{"metadata":{"_uuid":"aa69ff9592e9fc21a1b2c09beec94e7be96ae5f9"},"cell_type":"markdown","source":"### Output:\n**Mean Test Score of MLP:\n**![Imgur](https://i.imgur.com/lINgGl4.png)"},{"metadata":{"_uuid":"7922c709f9271a72cc61a2dc191747553a70ba38"},"cell_type":"markdown","source":"### Output:\n```\nmlp accuracy: 84.6\n```"},{"metadata":{"_uuid":"5b2e93def4cf4da788f3df11dd2c1834a6066e23"},"cell_type":"markdown","source":"# 📡 5.Convolutional Neural Networks (CNN / ConvNet)\nThe Convolutional Neural Network architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties into the architecture. These then make the forward function more efficient to implement and vastly reduce the amount of parameters in the network.\n\n![Imgur](https://i.imgur.com/HyPUqwF.png)\n![Imgur](https://i.imgur.com/jR5naeg.png)\n\n### How Work Convolutional Neural Networks (CNN)\n![Imgur](https://i.imgur.com/q5BBj8p.png)\n"},{"metadata":{"_uuid":"539f320a6bc922952d622242f15fd215ec623276"},"cell_type":"markdown","source":"### Output:\n**Train on 21000 samples, validate on 9000 samples\n**![Imgur](https://i.imgur.com/QoB4Bft.png)"},{"metadata":{"_uuid":"019769581176afe20f2493971ff28c265b176a82"},"cell_type":"markdown","source":"### Output:\n**Model accuracy & Model loss plot:\n**![Imgur](https://i.imgur.com/csePWWo.png)"},{"metadata":{"_uuid":"cf43e8909ed745b40d89ecf13846936835b40533"},"cell_type":"markdown","source":"### Output:\n```\nTest Loss: 41.8\nTest Accuracy: 91\n```"},{"metadata":{"_uuid":"2aef79487d6394a50f0c47b82190dc6814f11bdb"},"cell_type":"markdown","source":"# 8. Results"},{"metadata":{"_uuid":"77e22c756f1d3956ff439b6c7e10ee09d185108a"},"cell_type":"markdown","source":"### Output:\nFinal result of all\n```\nKNN accuracy: 0.819555555556 \n Random forest accuracy: 0.804555555556 \n Linear SVC accuracy: 0.717777777778 \n Gaussian Radial Basis Function SVC Accuracy: 0.781555555556 \n Multi-Layer Perceptron accuracy: 0.845666666667 \n Convolutional Neural Network Score: 0.909555555556 \n```\n "},{"metadata":{"_uuid":"7964eb0db61a24e8b1d98fe401d49f6d3771980d"},"cell_type":"markdown","source":"# Conclusion:\nThis notebook a demon for this compitions. I will only use around 5000 doodles for each label since the full dataset would be too much for my computer to handle. Then I will explore the drawings and graph random sketches of each\ncategory. Finally, I will test the dataset with Random Forest, Support-Vector Machine (SVM), KNearest\nNeighbors (KNN) and Multi-Layer Perceptron (MLP) classifiers in scikit-learn as well as\na Convolutional Neural Network (CNN) in Karas.\n\nHere is some reference :\n1. [Quick Draw's Article](https://towardsdatascience.com/quick-draw-the-worlds-largest-doodle-dataset-823c22ffce6b)\n2. [Quick Draw's Demo video](https://www.youtube.com/watch?list=PLIivdWyY5sqJxnwJhe3etaK7utrBiPBQ2&time_continue=2&v=8DEjphIfeYw)\n3. [ Quick Draw's Paper](https://github.com/nolanadams1230/Doodle_Classification/blob/master/Final_Report.pdf)\n\nIf you enjoyed reading the kernel , hit the upvote button !"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}