{
  "id": 72892,
  "title": "From LB 0.924 to 0.943",
  "url": "/competitions/quickdraw-doodle-recognition/discussion/72892",
  "author_name": "Appian",
  "post_date": "2018-11-28T05:28:34.894000",
  "votes": 50,
  "comment_count": 35,
  "views": 0,
  "content": "<p>I'd like to share some of my notes which improved the LB score from 0.924 to 0.943. I hope this is beneficial to some people to get notions to further improve the score.</p>\n\n<p>There are posts for getting LB 0.945 and above by Heng. It is very helpful.</p>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558\">https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558</a></li>\n<li><a href=\"https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912\">https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912</a></li>\n</ul>\n\n<h2>The baseline model (LB 0.924)</h2>\n\n<ul>\n<li>se_resnext26 (no pretrained)</li>\n<li>cross entropy loss</li>\n<li>adam optimizer</li>\n<li>image size 80x80</li>\n<li>batch size 512</li>\n<li>lr 3e-3 to 1e-4</li>\n<li>1.06 epochs</li>\n</ul>\n\n<p>I used raw version of training data.</p>\n\n<p>I encoded stroke information by calculating stroke velocity. Used 512x512 array to draw lines and shrink it to 80x80. </p>\n\n<p>I used Beluga's great kernel to divide train dataset into 100 files. Used 1 file for validation, 99 files for training. \n<a href=\"https://www.kaggle.com/gaborfodor/shuffle-csvs\">https://www.kaggle.com/gaborfodor/shuffle-csvs</a></p>\n\n<h2>Increased batch size (LB 0.935)</h2>\n\n<ul>\n<li>batch size 1024</li>\n<li>1.53 epochs total</li>\n</ul>\n\n<h2>Increased image size (LB 0.938)</h2>\n\n<ul>\n<li>image size 128x128 </li>\n<li>batch size 512 (because of GPU limit)</li>\n<li>3.07 epochs total</li>\n</ul>\n\n<h2>Excluded low quality images (LB 0.941)</h2>\n\n<ul>\n<li>Used the model above to exclude unrecognized low quality images.</li>\n<li>Resulted in excluding 20% of unrecognized images.</li>\n<li>3.88 epochs total</li>\n</ul>\n\n<h2>Ensemble, tta (LB 0.943)</h2>\n\n<ul>\n<li>tta by incomplete strokes (ie. use 80% of strokes)</li>\n<li>snapshot ensemble</li>\n</ul>\n\n<h3>To do list for the rest of the competition</h3>\n\n<ul>\n<li>larger image size</li>\n<li>country specific model</li>\n</ul>\n\n<p>I have not tested many things which might improve LB further. You probably get better accuracy than I get by using larger image size or a better way of encoding strokes. Thanks for reading. </p>",
  "messages": [
    {
      "id": 428940,
      "postDate": "2018-11-28T05:28:34.893Z",
      "content": "<p>I'd like to share some of my notes which improved the LB score from 0.924 to 0.943. I hope this is beneficial to some people to get notions to further improve the score.</p>\n\n<p>There are posts for getting LB 0.945 and above by Heng. It is very helpful.</p>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558\">https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558</a></li>\n<li><a href=\"https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912\">https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912</a></li>\n</ul>\n\n<h2>The baseline model (LB 0.924)</h2>\n\n<ul>\n<li>se_resnext26 (no pretrained)</li>\n<li>cross entropy loss</li>\n<li>adam optimizer</li>\n<li>image size 80x80</li>\n<li>batch size 512</li>\n<li>lr 3e-3 to 1e-4</li>\n<li>1.06 epochs</li>\n</ul>\n\n<p>I used raw version of training data.</p>\n\n<p>I encoded stroke information by calculating stroke velocity. Used 512x512 array to draw lines and shrink it to 80x80. </p>\n\n<p>I used Beluga's great kernel to divide train dataset into 100 files. Used 1 file for validation, 99 files for training. \n<a href=\"https://www.kaggle.com/gaborfodor/shuffle-csvs\">https://www.kaggle.com/gaborfodor/shuffle-csvs</a></p>\n\n<h2>Increased batch size (LB 0.935)</h2>\n\n<ul>\n<li>batch size 1024</li>\n<li>1.53 epochs total</li>\n</ul>\n\n<h2>Increased image size (LB 0.938)</h2>\n\n<ul>\n<li>image size 128x128 </li>\n<li>batch size 512 (because of GPU limit)</li>\n<li>3.07 epochs total</li>\n</ul>\n\n<h2>Excluded low quality images (LB 0.941)</h2>\n\n<ul>\n<li>Used the model above to exclude unrecognized low quality images.</li>\n<li>Resulted in excluding 20% of unrecognized images.</li>\n<li>3.88 epochs total</li>\n</ul>\n\n<h2>Ensemble, tta (LB 0.943)</h2>\n\n<ul>\n<li>tta by incomplete strokes (ie. use 80% of strokes)</li>\n<li>snapshot ensemble</li>\n</ul>\n\n<h3>To do list for the rest of the competition</h3>\n\n<ul>\n<li>larger image size</li>\n<li>country specific model</li>\n</ul>\n\n<p>I have not tested many things which might improve LB further. You probably get better accuracy than I get by using larger image size or a better way of encoding strokes. Thanks for reading. </p>",
      "rawMarkdown": "I'd like to share some of my notes which improved the LB score from 0.924 to 0.943. I hope this is beneficial to some people to get notions to further improve the score.\n\nThere are posts for getting LB 0.945 and above by Heng. It is very helpful.\n\n- https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558\n- https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912\n\n\n## The baseline model (LB 0.924)\n- se_resnext26 (no pretrained)\n- cross entropy loss\n- adam optimizer\n- image size 80x80\n- batch size 512\n- lr 3e-3 to 1e-4\n- 1.06 epochs\n\nI used raw version of training data.\n\nI encoded stroke information by calculating stroke velocity. Used 512x512 array to draw lines and shrink it to 80x80. \n\nI used Beluga's great kernel to divide train dataset into 100 files. Used 1 file for validation, 99 files for training. \nhttps://www.kaggle.com/gaborfodor/shuffle-csvs\n\n## Increased batch size (LB 0.935)\n- batch size 1024\n- 1.53 epochs total\n\n\n## Increased image size (LB 0.938)\n- image size 128x128 \n- batch size 512 (because of GPU limit)\n- 3.07 epochs total\n\n\n## Excluded low quality images (LB 0.941)\n- Used the model above to exclude unrecognized low quality images.\n- Resulted in excluding 20% of unrecognized images.\n- 3.88 epochs total\n\n\n## Ensemble, tta (LB 0.943)\n- tta by incomplete strokes (ie. use 80% of strokes)\n- snapshot ensemble\n\n\n### To do list for the rest of the competition\n- larger image size\n- country specific model\n\n\nI have not tested many things which might improve LB further. You probably get better accuracy than I get by using larger image size or a better way of encoding strokes. Thanks for reading. \n",
      "votes": 50
    },
    {
      "id": 430016,
      "postDate": "2018-11-29T16:55:25.873Z",
      "content": "<p>there is this trick to increase you batch size by freezing:</p>\n\n<ol>\n<li><p>train as normal, e.g. batch=256. Then reduce your learning rate ... etc ... until there is no improvement</p></li>\n<li><p>freeze the bottom layers and increase your batch size. The more you freeze, the larger is the batch size, but the improvement is limited by no. of trainable parameters</p></li>\n</ol>",
      "rawMarkdown": "there is this trick to increase you batch size by freezing:\n\n1. train as normal, e.g. batch=256. Then reduce your learning rate ... etc ... until there is no improvement\n\n2. freeze the bottom layers and increase your batch size. The more you freeze, the larger is the batch size, but the improvement is limited by no. of trainable parameters",
      "votes": 7,
      "replies": [
        {
          "id": 430168,
          "postDate": "2018-11-30T00:10:18.593Z",
          "content": "<p>Thanks for the advice. I probably try that at some point.</p>",
          "rawMarkdown": "Thanks for the advice. I probably try that at some point."
        }
      ]
    },
    {
      "id": 429626,
      "postDate": "2018-11-29T04:59:11.820Z",
      "content": "<p>Hey, how did you get 0.946 score now?</p>",
      "rawMarkdown": "Hey, how did you get 0.946 score now?",
      "votes": 1,
      "replies": [
        {
          "id": 429993,
          "postDate": "2018-11-29T16:25:04.230Z",
          "content": "<p>Actually I'm only inreasing the image size to 134x134.</p>",
          "rawMarkdown": "Actually I'm only inreasing the image size to 134x134.",
          "votes": 1
        },
        {
          "id": 430228,
          "postDate": "2018-11-30T03:31:46.460Z",
          "content": "<p>0.946 on 134x134, very impressive results</p>",
          "rawMarkdown": "0.946 on 134x134, very impressive results"
        }
      ]
    },
    {
      "id": 429222,
      "postDate": "2018-11-28T14:45:03.160Z",
      "content": "<p>@Appian  What do you mean by this \"I encoded stroke information by calculating stroke velocity\" ? Can you please share some reference how you did it ? currently I am using single channel no encoding, does encoding help a lot ?</p>",
      "rawMarkdown": "@Appian  What do you mean by this \"I encoded stroke information by calculating stroke velocity\" ? Can you please share some reference how you did it ? currently I am using single channel no encoding, does encoding help a lot ?",
      "votes": 1,
      "replies": [
        {
          "id": 429243,
          "postDate": "2018-11-28T15:22:35.687Z",
          "content": "<p>There are 3 channels. I use them seperately. For example, 1st channel</p>\n\n<p>I calculated the velocity using timestamps and coordinates and then draw a line which value is determined by the velocity. By doing this, the painted image can represents the speed of each strokes which I hope the model learn something useful. I think it helps, not much maybe.</p>",
          "rawMarkdown": "There are 3 channels. I use them seperately. For example, 1st channel\n\nI calculated the velocity using timestamps and coordinates and then draw a line which value is determined by the velocity. By doing this, the painted image can represents the speed of each strokes which I hope the model learn something useful. I think it helps, not much maybe."
        },
        {
          "id": 429274,
          "postDate": "2018-11-28T15:59:22.343Z",
          "content": "<p>@Appian</p>\n\n<p>are you using train_raw (65gb) strokes ?</p>",
          "rawMarkdown": "@Appian\n\nare you using train_raw (65gb) strokes ?"
        },
        {
          "id": 429994,
          "postDate": "2018-11-29T16:26:01.780Z",
          "content": "<p>Yes,</p>",
          "rawMarkdown": "Yes,"
        }
      ]
    },
    {
      "id": 429027,
      "postDate": "2018-11-28T08:46:06.837Z",
      "content": "<p>@Appian Hi, what is your corresponding local LB results of public LB 0.943 ?</p>",
      "rawMarkdown": " @Appian Hi, what is your corresponding local LB results of public LB 0.943 ?",
      "votes": -1,
      "replies": [
        {
          "id": 429123,
          "postDate": "2018-11-28T11:37:03.467Z",
          "content": "<p>0.93444 local LB for 0.943 public LB. (Ensemble, TTA)\n0.93320 local LB for 0.941 public LB. (Single model)</p>\n\n<p>I use 90,000 recognized images for validation.</p>",
          "rawMarkdown": "0.93444 local LB for 0.943 public LB. (Ensemble, TTA)\n0.93320 local LB for 0.941 public LB. (Single model)\n\nI use 90,000 recognized images for validation."
        },
        {
          "id": 429187,
          "postDate": "2018-11-28T13:41:25.560Z",
          "content": "<p>@Appian thanks for you reply.  My results are:\ntop1: 0.848   top3: 0.948  local lb: 0.894 for 0.942 public LB.</p>\n\n<p>It seems that our LB evaluation methods are different. Would you please share your top1 or top3 validation results? Thanks a lot.</p>",
          "rawMarkdown": "@Appian thanks for you reply.  My results are:\ntop1: 0.848   top3: 0.948  local lb: 0.894 for 0.942 public LB.\n\nIt seems that our LB evaluation methods are different. Would you please share your top1 or top3 validation results? Thanks a lot."
        },
        {
          "id": 429247,
          "postDate": "2018-11-28T15:25:29.357Z",
          "content": "<p>You are welcome. \nI don't have top1 score as I only use average precision at k=3. </p>\n\n<pre><code>score = ml_metrics.apk([target_class], top3, k=3)\n</code></pre>\n\n<p>ml_metrics is from \n<a href=\"https://github.com/benhamner/Metrics/tree/master/Python/ml_metrics\">https://github.com/benhamner/Metrics/tree/master/Python/ml_metrics</a></p>",
          "rawMarkdown": "You are welcome. \nI don't have top1 score as I only use average precision at k=3. \n\n    score = ml_metrics.apk([target_class], top3, k=3)\n\nml_metrics is from \nhttps://github.com/benhamner/Metrics/tree/master/Python/ml_metrics"
        },
        {
          "id": 429272,
          "postDate": "2018-11-28T15:57:08.783Z",
          "content": "<p>@凉宫ハルヒ</p>\n\n<p>what is your stroke encoding and lr schedule ?</p>",
          "rawMarkdown": "@凉宫ハルヒ\n\nwhat is your stroke encoding and lr schedule ?"
        },
        {
          "id": 429730,
          "postDate": "2018-11-29T09:02:34.673Z",
          "content": "<p>thx.</p>",
          "rawMarkdown": "thx."
        }
      ]
    },
    {
      "id": 432038,
      "postDate": "2018-12-03T09:56:26.850Z",
      "content": "<p>Thank you for sharing.\nI got 0.94 with Resnet50 and 0.943 with Se_Resnet50.</p>",
      "rawMarkdown": "Thank you for sharing.\nI got 0.94 with Resnet50 and 0.943 with Se_Resnet50."
    },
    {
      "id": 430018,
      "postDate": "2018-11-29T16:57:14.230Z",
      "content": "<p>Hi,\nWhat do you mean by 1.06 epochs?\nI tried similar epoch size in my model and the score was too low.\nIs this the same epoch which means number of times the training set is trained on the model?</p>",
      "rawMarkdown": "Hi,\nWhat do you mean by 1.06 epochs?\nI tried similar epoch size in my model and the score was too low.\nIs this the same epoch which means number of times the training set is trained on the model?",
      "replies": [
        {
          "id": 430169,
          "postDate": "2018-11-30T00:14:15.753Z",
          "content": "<p>Yes, considering the huge dataset we have, I split them to 100 files and call 1 file 0.01 epoch and run validation for every 0.01 epoch for convinience. 1.06 means I fed 106 files to the model for learning.</p>",
          "rawMarkdown": "Yes, considering the huge dataset we have, I split them to 100 files and call 1 file 0.01 epoch and run validation for every 0.01 epoch for convinience. 1.06 means I fed 106 files to the model for learning.",
          "votes": 1
        },
        {
          "id": 430515,
          "postDate": "2018-11-30T13:58:15.983Z",
          "content": "<p>But then how many epochs have you run those 106 files for? Like how many total times have you used them to train the model? Do you train each of those 106 files just once?\nWell I'm pretty new to image recognition challenges and so quite confused. Thanks for the help :)</p>",
          "rawMarkdown": "But then how many epochs have you run those 106 files for? Like how many total times have you used them to train the model? Do you train each of those 106 files just once?\nWell I'm pretty new to image recognition challenges and so quite confused. Thanks for the help :)"
        },
        {
          "id": 430915,
          "postDate": "2018-12-01T07:55:17.507Z",
          "content": "<p>Trained on all training images once plus 6% of all training images once. I called this 1.06 epochs.\nGood luck with the challenge!</p>",
          "rawMarkdown": "Trained on all training images once plus 6% of all training images once. I called this 1.06 epochs.\nGood luck with the challenge!"
        },
        {
          "id": 431032,
          "postDate": "2018-12-01T13:39:56Z",
          "content": "<p>Wow!! Your model and pre-processing might be working great then!! One of my kernels had score of 0.907 only after 30 epochs and when I implement the same with a 2 the accuracy falls big time.\nGood luck to you too!!</p>",
          "rawMarkdown": "Wow!! Your model and pre-processing might be working great then!! One of my kernels had score of 0.907 only after 30 epochs and when I implement the same with a 2 the accuracy falls big time.\nGood luck to you too!!",
          "votes": 1
        }
      ]
    },
    {
      "id": 429708,
      "postDate": "2018-11-29T08:17:38.987Z",
      "content": "<p>If you have already excluded unrecognized low quality images (I'm planing to do something similar but my model hasn't converged yet), increasing the batch size isn't that much of a boost anymore, maybe try larger images instead?</p>",
      "rawMarkdown": "If you have already excluded unrecognized low quality images (I'm planing to do something similar but my model hasn't converged yet), increasing the batch size isn't that much of a boost anymore, maybe try larger images instead?",
      "replies": [
        {
          "id": 429936,
          "postDate": "2018-11-29T15:11:20.903Z",
          "content": "<p>Yes, I'm trying larger images. The large batch size might not be very important now as you mentioned. </p>",
          "rawMarkdown": "Yes, I'm trying larger images. The large batch size might not be very important now as you mentioned. "
        }
      ]
    },
    {
      "id": 428984,
      "postDate": "2018-11-28T07:15:10.683Z",
      "content": "<p>Thanks @Appain for the great tips.</p>\n\n<p>I'm using DenseNet121 with 64x64 image size and 330 in batch size, no ensemble or TTA and I'm stuck at LB 0.925. Since I only have a GTX 1060 6GB, I can't increase the image size or batch size.</p>\n\n<p>Do you think se_resnext26 or ResNet will be better than DenseNet121 ? I planning to use a GPU with more VRAM to get to a bigger batch size like 1024. (since increased batch size helped you get to LB 0.935)</p>",
      "rawMarkdown": "Thanks @Appain for the great tips.\n\nI'm using DenseNet121 with 64x64 image size and 330 in batch size, no ensemble or TTA and I'm stuck at LB 0.925. Since I only have a GTX 1060 6GB, I can't increase the image size or batch size.\n\nDo you think se_resnext26 or ResNet will be better than DenseNet121 ? I planning to use a GPU with more VRAM to get to a bigger batch size like 1024. (since increased batch size helped you get to LB 0.935)",
      "replies": [
        {
          "id": 428996,
          "postDate": "2018-11-28T07:26:57.627Z",
          "content": "<p>It seems like many people including myself got stuck at 0.925 at some point. Try resetting your learning rate to something higher. </p>",
          "rawMarkdown": "It seems like many people including myself got stuck at 0.925 at some point. Try resetting your learning rate to something higher. ",
          "votes": 1
        },
        {
          "id": 429014,
          "postDate": "2018-11-28T08:15:16.370Z",
          "content": "<p>As I haven't tried DenseNet, I can not answer the question. I tried resnet50, se_resnext50, se_resnext26 and chose se_resnext26 as it is the smallest(fastest) and accuracy was as good as others in about 0.2 epochs.</p>\n\n<p>se_resnext26\n<a href=\"https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\">https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py</a></p>\n\n<p>Increasing batch size/image size probably helps. \nI really doubt the parameters I use is the best. Batch size 1024 turned out to be good in my case. Finding out a good batch size along with learning rate for your model is important I guess.</p>",
          "rawMarkdown": "As I haven't tried DenseNet, I can not answer the question. I tried resnet50, se_resnext50, se_resnext26 and chose se_resnext26 as it is the smallest(fastest) and accuracy was as good as others in about 0.2 epochs.\n\nse_resnext26\nhttps://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\n\nIncreasing batch size/image size probably helps. \nI really doubt the parameters I use is the best. Batch size 1024 turned out to be good in my case. Finding out a good batch size along with learning rate for your model is important I guess.",
          "votes": 1
        },
        {
          "id": 429028,
          "postDate": "2018-11-28T08:48:35.397Z",
          "content": "<p>@Appian Btw when you use seresnext50, what is the batchsize did you used? How did you implemented it ? And what hardware did you used to train it?</p>\n\n<p>Thanks again ;)</p>",
          "rawMarkdown": "\n@Appian Btw when you use seresnext50, what is the batchsize did you used? How did you implemented it ? And what hardware did you used to train it?\n\nThanks again ;)"
        },
        {
          "id": 429124,
          "postDate": "2018-11-28T11:40:16.147Z",
          "content": "<p>batch size is 512</p>\n\n<p>I did not implement the net. I use this implementation. <a href=\"https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\">https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py</a></p>\n\n<p>trained on 1080ti x2, core i5 8400</p>",
          "rawMarkdown": "batch size is 512\n\nI did not implement the net. I use this implementation. https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\n\ntrained on 1080ti x2, core i5 8400",
          "votes": 1
        },
        {
          "id": 429562,
          "postDate": "2018-11-29T02:44:06.040Z",
          "content": "<p>I only have one 1080ti. your 1080ti*2, how much memory is available!?</p>",
          "rawMarkdown": "I only have one 1080ti. your 1080ti*2, how much memory is available!?"
        },
        {
          "id": 429997,
          "postDate": "2018-11-29T16:28:13.967Z",
          "content": "<p>I have a 64GB machine with 2 1080ti</p>",
          "rawMarkdown": "I have a 64GB machine with 2 1080ti"
        },
        {
          "id": 430179,
          "postDate": "2018-11-30T00:58:49.520Z",
          "content": "<p>One of my model is Densenet 121, first I use 64*64*700, and I can get about lb 0.930, now I am doing fine-tuning with 128*128*170, increase the size of image will be helpful</p>",
          "rawMarkdown": "One of my model is Densenet 121, first I use 64*64*700, and I can get about lb 0.930, now I am doing fine-tuning with 128*128*170, increase the size of image will be helpful"
        }
      ]
    },
    {
      "id": 431029,
      "postDate": "2018-12-01T13:26:57.970Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 430202,
      "postDate": "2018-11-30T02:21:43.257Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true
    },
    {
      "id": 429852,
      "postDate": "2018-11-29T13:10:30.787Z",
      "content": "<p>Thanks for the the tips</p>",
      "rawMarkdown": "Thanks for the the tips"
    },
    {
      "id": 428980,
      "postDate": "2018-11-28T07:08:19.180Z",
      "content": "<p>thanks so much for your sharing!</p>",
      "rawMarkdown": "thanks so much for your sharing!"
    }
  ],
  "comments": [
    {
      "id": 430016,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2018-11-29T16:55:25.873000",
      "content": "<p>there is this trick to increase you batch size by freezing:</p>\n\n<ol>\n<li><p>train as normal, e.g. batch=256. Then reduce your learning rate ... etc ... until there is no improvement</p></li>\n<li><p>freeze the bottom layers and increase your batch size. The more you freeze, the larger is the batch size, but the improvement is limited by no. of trainable parameters</p></li>\n</ol>",
      "votes": 7,
      "replies": [
        {
          "id": 430168,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-30T00:10:18.593000",
          "content": "<p>Thanks for the advice. I probably try that at some point.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 429626,
      "author_name": "Gary",
      "author_url": "",
      "post_date": "2018-11-29T04:59:11.820000",
      "content": "<p>Hey, how did you get 0.946 score now?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 429993,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-29T16:25:04.230000",
          "content": "<p>Actually I'm only inreasing the image size to 134x134.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 430228,
          "author_name": "good good study",
          "author_url": "",
          "post_date": "2018-11-30T03:31:46.460000",
          "content": "<p>0.946 on 134x134, very impressive results</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 429222,
      "author_name": "remidi",
      "author_url": "",
      "post_date": "2018-11-28T14:45:03.160000",
      "content": "<p>@Appian  What do you mean by this \"I encoded stroke information by calculating stroke velocity\" ? Can you please share some reference how you did it ? currently I am using single channel no encoding, does encoding help a lot ?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 429243,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-28T15:22:35.687000",
          "content": "<p>There are 3 channels. I use them seperately. For example, 1st channel</p>\n\n<p>I calculated the velocity using timestamps and coordinates and then draw a line which value is determined by the velocity. By doing this, the painted image can represents the speed of each strokes which I hope the model learn something useful. I think it helps, not much maybe.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429274,
          "author_name": "remidi",
          "author_url": "",
          "post_date": "2018-11-28T15:59:22.343000",
          "content": "<p>@Appian</p>\n\n<p>are you using train_raw (65gb) strokes ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429994,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-29T16:26:01.780000",
          "content": "<p>Yes,</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 429027,
      "author_name": "good good study",
      "author_url": "",
      "post_date": "2018-11-28T08:46:06.837000",
      "content": "<p>@Appian Hi, what is your corresponding local LB results of public LB 0.943 ?</p>",
      "votes": -1,
      "replies": [
        {
          "id": 429123,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-28T11:37:03.467000",
          "content": "<p>0.93444 local LB for 0.943 public LB. (Ensemble, TTA)\n0.93320 local LB for 0.941 public LB. (Single model)</p>\n\n<p>I use 90,000 recognized images for validation.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429187,
          "author_name": "good good study",
          "author_url": "",
          "post_date": "2018-11-28T13:41:25.560000",
          "content": "<p>@Appian thanks for you reply.  My results are:\ntop1: 0.848   top3: 0.948  local lb: 0.894 for 0.942 public LB.</p>\n\n<p>It seems that our LB evaluation methods are different. Would you please share your top1 or top3 validation results? Thanks a lot.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429247,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-28T15:25:29.357000",
          "content": "<p>You are welcome. \nI don't have top1 score as I only use average precision at k=3. </p>\n\n<pre><code>score = ml_metrics.apk([target_class], top3, k=3)\n</code></pre>\n\n<p>ml_metrics is from \n<a href=\"https://github.com/benhamner/Metrics/tree/master/Python/ml_metrics\">https://github.com/benhamner/Metrics/tree/master/Python/ml_metrics</a></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429272,
          "author_name": "remidi",
          "author_url": "",
          "post_date": "2018-11-28T15:57:08.783000",
          "content": "<p>@凉宫ハルヒ</p>\n\n<p>what is your stroke encoding and lr schedule ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429730,
          "author_name": "good good study",
          "author_url": "",
          "post_date": "2018-11-29T09:02:34.673000",
          "content": "<p>thx.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 432038,
      "author_name": "Hai Nam Nguyen",
      "author_url": "",
      "post_date": "2018-12-03T09:56:26.850000",
      "content": "<p>Thank you for sharing.\nI got 0.94 with Resnet50 and 0.943 with Se_Resnet50.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 430018,
      "author_name": "Sreyan Ghosh",
      "author_url": "",
      "post_date": "2018-11-29T16:57:14.230000",
      "content": "<p>Hi,\nWhat do you mean by 1.06 epochs?\nI tried similar epoch size in my model and the score was too low.\nIs this the same epoch which means number of times the training set is trained on the model?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 430169,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-30T00:14:15.753000",
          "content": "<p>Yes, considering the huge dataset we have, I split them to 100 files and call 1 file 0.01 epoch and run validation for every 0.01 epoch for convinience. 1.06 means I fed 106 files to the model for learning.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 430515,
          "author_name": "Sreyan Ghosh",
          "author_url": "",
          "post_date": "2018-11-30T13:58:15.983000",
          "content": "<p>But then how many epochs have you run those 106 files for? Like how many total times have you used them to train the model? Do you train each of those 106 files just once?\nWell I'm pretty new to image recognition challenges and so quite confused. Thanks for the help :)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 430915,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-12-01T07:55:17.507000",
          "content": "<p>Trained on all training images once plus 6% of all training images once. I called this 1.06 epochs.\nGood luck with the challenge!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 431032,
          "author_name": "Sreyan Ghosh",
          "author_url": "",
          "post_date": "2018-12-01T13:39:56",
          "content": "<p>Wow!! Your model and pre-processing might be working great then!! One of my kernels had score of 0.907 only after 30 epochs and when I implement the same with a 2 the accuracy falls big time.\nGood luck to you too!!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 429708,
      "author_name": "vlad",
      "author_url": "",
      "post_date": "2018-11-29T08:17:38.987000",
      "content": "<p>If you have already excluded unrecognized low quality images (I'm planing to do something similar but my model hasn't converged yet), increasing the batch size isn't that much of a boost anymore, maybe try larger images instead?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 429936,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-29T15:11:20.903000",
          "content": "<p>Yes, I'm trying larger images. The large batch size might not be very important now as you mentioned. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 428984,
      "author_name": "Hoàng Tùng Lâm",
      "author_url": "",
      "post_date": "2018-11-28T07:15:10.683000",
      "content": "<p>Thanks @Appain for the great tips.</p>\n\n<p>I'm using DenseNet121 with 64x64 image size and 330 in batch size, no ensemble or TTA and I'm stuck at LB 0.925. Since I only have a GTX 1060 6GB, I can't increase the image size or batch size.</p>\n\n<p>Do you think se_resnext26 or ResNet will be better than DenseNet121 ? I planning to use a GPU with more VRAM to get to a bigger batch size like 1024. (since increased batch size helped you get to LB 0.935)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 428996,
          "author_name": "HuyenNguyen",
          "author_url": "",
          "post_date": "2018-11-28T07:26:57.627000",
          "content": "<p>It seems like many people including myself got stuck at 0.925 at some point. Try resetting your learning rate to something higher. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 429014,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-28T08:15:16.370000",
          "content": "<p>As I haven't tried DenseNet, I can not answer the question. I tried resnet50, se_resnext50, se_resnext26 and chose se_resnext26 as it is the smallest(fastest) and accuracy was as good as others in about 0.2 epochs.</p>\n\n<p>se_resnext26\n<a href=\"https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\">https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py</a></p>\n\n<p>Increasing batch size/image size probably helps. \nI really doubt the parameters I use is the best. Batch size 1024 turned out to be good in my case. Finding out a good batch size along with learning rate for your model is important I guess.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 429028,
          "author_name": "Hoàng Tùng Lâm",
          "author_url": "",
          "post_date": "2018-11-28T08:48:35.397000",
          "content": "<p>@Appian Btw when you use seresnext50, what is the batchsize did you used? How did you implemented it ? And what hardware did you used to train it?</p>\n\n<p>Thanks again ;)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429124,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-28T11:40:16.147000",
          "content": "<p>batch size is 512</p>\n\n<p>I did not implement the net. I use this implementation. <a href=\"https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py\">https://github.com/soeaver/pytorch-priv/blob/master/models/imagenet/se_resnext.py</a></p>\n\n<p>trained on 1080ti x2, core i5 8400</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 429562,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-11-29T02:44:06.040000",
          "content": "<p>I only have one 1080ti. your 1080ti*2, how much memory is available!?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 429997,
          "author_name": "Appian",
          "author_url": "",
          "post_date": "2018-11-29T16:28:13.967000",
          "content": "<p>I have a 64GB machine with 2 1080ti</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 430179,
          "author_name": "Dilapsky Lee",
          "author_url": "",
          "post_date": "2018-11-30T00:58:49.520000",
          "content": "<p>One of my model is Densenet 121, first I use 64*64*700, and I can get about lb 0.930, now I am doing fine-tuning with 128*128*170, increase the size of image will be helpful</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 431029,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-12-01T13:26:57.970000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 430202,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-11-30T02:21:43.257000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 429852,
      "author_name": "Vahid",
      "author_url": "",
      "post_date": "2018-11-29T13:10:30.787000",
      "content": "<p>Thanks for the the tips</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 428980,
      "author_name": "Salaryman",
      "author_url": "",
      "post_date": "2018-11-28T07:08:19.180000",
      "content": "<p>thanks so much for your sharing!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "428940": "I'd like to share some of my notes which improved the LB score from 0.924 to 0.943. I hope this is beneficial to some people to get notions to further improve the score.\n\nThere are posts for getting LB 0.945 and above by Heng. It is very helpful.\n\n- https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70558\n- https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70912\n\n\n## The baseline model (LB 0.924)\n- se_resnext26 (no pretrained)\n- cross entropy loss\n- adam optimizer\n- image size 80x80\n- batch size 512\n- lr 3e-3 to 1e-4\n- 1.06 epochs\n\nI used raw version of training data.\n\nI encoded stroke information by calculating stroke velocity. Used 512x512 array to draw lines and shrink it to 80x80. \n\nI used Beluga's great kernel to divide train dataset into 100 files. Used 1 file for validation, 99 files for training. \nhttps://www.kaggle.com/gaborfodor/shuffle-csvs\n\n## Increased batch size (LB 0.935)\n- batch size 1024\n- 1.53 epochs total\n\n\n## Increased image size (LB 0.938)\n- image size 128x128 \n- batch size 512 (because of GPU limit)\n- 3.07 epochs total\n\n\n## Excluded low quality images (LB 0.941)\n- Used the model above to exclude unrecognized low quality images.\n- Resulted in excluding 20% of unrecognized images.\n- 3.88 epochs total\n\n\n## Ensemble, tta (LB 0.943)\n- tta by incomplete strokes (ie. use 80% of strokes)\n- snapshot ensemble\n\n\n### To do list for the rest of the competition\n- larger image size\n- country specific model\n\n\nI have not tested many things which might improve LB further. You probably get better accuracy than I get by using larger image size or a better way of encoding strokes. Thanks for reading. \n",
    "430016": "there is this trick to increase you batch size by freezing:\n\n1. train as normal, e.g. batch=256. Then reduce your learning rate ... etc ... until there is no improvement\n\n2. freeze the bottom layers and increase your batch size. The more you freeze, the larger is the batch size, but the improvement is limited by no. of trainable parameters",
    "429626": "Hey, how did you get 0.946 score now?",
    "429222": "@Appian  What do you mean by this \"I encoded stroke information by calculating stroke velocity\" ? Can you please share some reference how you did it ? currently I am using single channel no encoding, does encoding help a lot ?",
    "429027": " @Appian Hi, what is your corresponding local LB results of public LB 0.943 ?",
    "432038": "Thank you for sharing.\nI got 0.94 with Resnet50 and 0.943 with Se_Resnet50.",
    "430018": "Hi,\nWhat do you mean by 1.06 epochs?\nI tried similar epoch size in my model and the score was too low.\nIs this the same epoch which means number of times the training set is trained on the model?",
    "429708": "If you have already excluded unrecognized low quality images (I'm planing to do something similar but my model hasn't converged yet), increasing the batch size isn't that much of a boost anymore, maybe try larger images instead?",
    "428984": "Thanks @Appain for the great tips.\n\nI'm using DenseNet121 with 64x64 image size and 330 in batch size, no ensemble or TTA and I'm stuck at LB 0.925. Since I only have a GTX 1060 6GB, I can't increase the image size or batch size.\n\nDo you think se_resnext26 or ResNet will be better than DenseNet121 ? I planning to use a GPU with more VRAM to get to a bigger batch size like 1024. (since increased batch size helped you get to LB 0.935)",
    "431029": "",
    "430202": "",
    "429852": "Thanks for the the tips",
    "428980": "thanks so much for your sharing!"
  }
}