{"cells":[{"metadata":{"_uuid":"1e99c1e6d51186e4459acda86ab1316c8fef9019"},"cell_type":"markdown","source":"# Find outliers with unusual entropy\nInspired by the pre-processing step from the paper http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2763.pdf\n\n> <b>Noise Removal with Image Entropy</b>\n> Key ingredient to a successful sketch center loss is the guarantee of non-noisy data (outliers), as it will significantly affect the class feature centers. However, sketch data collected with crowdsourcing are inevitable to noise, where we propose a noisy data removal technique to alleviate such issue by resorting to image entropy. Given a category of sketch, we can get entropy for each sketch and the overall entropy distribution on a category basis. We empirically find that keeping the middle 90% of each category as normal samples gives us best results. In Figure 4, we visualize the entropy histogram of star samples in our training set. If we choose the middle 90% samples as normal samples for star category, we can calculate and get the 0.05 and 0.95 percentiles of star images entropy as 0.1051 and 0.1721, respectively. We then treat the remaining samples as outliers or noise points (entropy ∈ [0, 0.1051)   (0.1721, 1]). It can be observed that low entropy sketches tend to be overly-abstract, yet high entropy ones being messy, sometimes with meaningless scribbles. Nevertheless, sketch data falling into middle entropy range present more consistent and reasonable drawings.\n"},{"metadata":{"trusted":true,"_uuid":"f2cac137d7913d05b8e04a4c39a8c243f54eebb9"},"cell_type":"code","source":"import ast\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom dask import bag\nfrom tqdm import tqdm\nfrom PIL import Image, ImageDraw","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b1bd54ae26d12b76232f21bfea7da034cf89f94"},"cell_type":"code","source":"def entropy_it(x):\n    counts = np.bincount(x)\n    p = counts[counts > 0] / float(len(x))\n    # compute Shannon entropy in bits\n    return -np.sum(p * np.log2(p))\n\ndef draw_it(strokes):\n    image = Image.new(\"P\", (256,256), color=255)\n    draw = ImageDraw.Draw(image)\n    for stroke in ast.literal_eval(strokes):\n        for i in range(len(stroke[0])-1):\n            draw.line([stroke[0][i], stroke[1][i], stroke[0][i+1], stroke[1][i+1]], fill=0, width=5)\n    image = np.array(image)\n    return entropy_it(image.flatten()), image\n\ndef plot_it(entropy, images, indices, n=5):\n    fig, axs = plt.subplots(nrows=n, ncols=n, figsize=(12, 10))\n    for i, j in enumerate(indices[0][:n*n]):\n        ax = axs[i // n, i % n]\n        ax.set_title(\"%.4f\" % entropy[j])\n        ax.imshow(images[j], cmap=\"gray\")\n        ax.set_yticks([])\n        ax.set_xticks([])\n        plt.setp(ax.spines.values(), color=\"red\")\n    plt.subplots_adjust(bottom=-0.2)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"reader = pd.read_csv('../input/train_simplified/star.csv', index_col=['key_id'], chunksize=1024)\n\ndata = []\nfor chunk in tqdm(reader):\n    entropybag = bag.from_sequence(chunk.drawing.values).map(draw_it)\n    data.extend(entropybag.compute()) # PARALLELIZE\n\nentropy, images = zip(*data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e988d652a20ec614fc6cf2bf542da06091acdfce"},"cell_type":"markdown","source":"## Keep 98% of samples"},{"metadata":{"trusted":true,"_uuid":"184f0bf97c087adad3dbcf2bf63a22603b312eb4"},"cell_type":"code","source":"threshold = 1\nlower = np.percentile(entropy, threshold)\nupper = np.percentile(entropy, 100 - threshold)\nprint(np.min(entropy), np.max(entropy))\nprint(lower, upper)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f52b2c414cf41a7b0c36c818e86d0cacea3225cb"},"cell_type":"code","source":"plt.title(\"Histogram\")\nplt.xlabel('entropy')\nplt.ylabel('count')\nplt.hist(entropy, bins=100)\nplt.axvline(x=lower, color='r')\nplt.axvline(x=upper, color='r')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b772d50c2bd04ecfc7520783c7542083cdad0c0f"},"cell_type":"markdown","source":"## Low entropy samples"},{"metadata":{"trusted":true,"_uuid":"5984a58ffc27a19e15cda5405851fe08f396a698"},"cell_type":"code","source":"plot_it(entropy, images, np.where(entropy < lower))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58575b67be908d46564b21fdb6d4b27994974d71"},"cell_type":"markdown","source":"## High entropy samples"},{"metadata":{"trusted":true,"_uuid":"8455359246b67d05c2ad51310a5e06740e35756c"},"cell_type":"code","source":"plot_it(entropy, images, np.where(entropy > upper))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}