{
  "id": 72791,
  "title": "LR schedule and Fine-tuning",
  "url": "/competitions/quickdraw-doodle-recognition/discussion/72791",
  "author_name": "remidi",
  "post_date": "2018-11-27T07:35:31.188000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>What is the lr schedule and fine-tuning that works ? I am using </p>\n\n<pre><code>Reduceonplateau with initial_lr = 0.001 and min_lr= 1e-6, \nbatchsize=128\nimgsize=128, \nmodel=resnet50\nall image samples - 34000 validation samples\n</code></pre>\n\n<p>but I can reach only <strong>0.935 Top-5 accuracy</strong> and <strong>0.922 LB</strong>.</p>\n\n<p>What is the fundamental mistake I might be doing ?</p>",
  "messages": [
    {
      "id": 428697,
      "postDate": "2018-11-27T18:18:04.637Z",
      "content": "<p>@FiyeroLeung\n(asked the question through mail)\nI was able to get the <code>0.929 LB</code> with the <strong>AdamAccumulator</strong> posted in the forum (I am unable to find) it increases the batch size of the network by accumulating the gradient,  it seems to give better results. I am yet to test the Sketch-a-Net implementation of channels versus single channel in resnet50.</p>\n\n<p>did anyone try the 6-channel and 3-channels (sketch-a-net) with resnet and observed better LB ?</p>\n\n<p>reference : <a href=\"https://arxiv.org/abs/1501.07873\">Sketch-A-Net Beats Humans</a></p>",
      "rawMarkdown": "@FiyeroLeung\n(asked the question through mail)\nI was able to get the `0.929 LB` with the **AdamAccumulator** posted in the forum (I am unable to find) it increases the batch size of the network by accumulating the gradient,  it seems to give better results. I am yet to test the Sketch-a-Net implementation of channels versus single channel in resnet50.\n\ndid anyone try the 6-channel and 3-channels (sketch-a-net) with resnet and observed better LB ?\n\nreference : [Sketch-A-Net Beats Humans](https://arxiv.org/abs/1501.07873)",
      "votes": 1,
      "replies": [
        {
          "id": 428839,
          "postDate": "2018-11-28T00:40:28.247Z",
          "content": "<p><a href=\"https://github.com/keras-team/keras/issues/3556\">https://github.com/keras-team/keras/issues/3556</a></p>",
          "rawMarkdown": "https://github.com/keras-team/keras/issues/3556"
        }
      ]
    },
    {
      "id": 428388,
      "postDate": "2018-11-27T07:35:31.187Z",
      "content": "<p>What is the lr schedule and fine-tuning that works ? I am using </p>\n\n<pre><code>Reduceonplateau with initial_lr = 0.001 and min_lr= 1e-6, \nbatchsize=128\nimgsize=128, \nmodel=resnet50\nall image samples - 34000 validation samples\n</code></pre>\n\n<p>but I can reach only <strong>0.935 Top-5 accuracy</strong> and <strong>0.922 LB</strong>.</p>\n\n<p>What is the fundamental mistake I might be doing ?</p>",
      "rawMarkdown": "What is the lr schedule and fine-tuning that works ? I am using \n\n    Reduceonplateau with initial_lr = 0.001 and min_lr= 1e-6, \n    batchsize=128\n    imgsize=128, \n    model=resnet50\n    all image samples - 34000 validation samples\n\nbut I can reach only **0.935 Top-5 accuracy** and **0.922 LB**.\n\nWhat is the fundamental mistake I might be doing ?",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 428697,
      "author_name": "remidi",
      "author_url": "",
      "post_date": "2018-11-27T18:18:04.637000",
      "content": "<p>@FiyeroLeung\n(asked the question through mail)\nI was able to get the <code>0.929 LB</code> with the <strong>AdamAccumulator</strong> posted in the forum (I am unable to find) it increases the batch size of the network by accumulating the gradient,  it seems to give better results. I am yet to test the Sketch-a-Net implementation of channels versus single channel in resnet50.</p>\n\n<p>did anyone try the 6-channel and 3-channels (sketch-a-net) with resnet and observed better LB ?</p>\n\n<p>reference : <a href=\"https://arxiv.org/abs/1501.07873\">Sketch-A-Net Beats Humans</a></p>",
      "votes": 1,
      "replies": [
        {
          "id": 428839,
          "author_name": "HuyenNguyen",
          "author_url": "",
          "post_date": "2018-11-28T00:40:28.247000",
          "content": "<p><a href=\"https://github.com/keras-team/keras/issues/3556\">https://github.com/keras-team/keras/issues/3556</a></p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "428697": "@FiyeroLeung\n(asked the question through mail)\nI was able to get the `0.929 LB` with the **AdamAccumulator** posted in the forum (I am unable to find) it increases the batch size of the network by accumulating the gradient,  it seems to give better results. I am yet to test the Sketch-a-Net implementation of channels versus single channel in resnet50.\n\ndid anyone try the 6-channel and 3-channels (sketch-a-net) with resnet and observed better LB ?\n\nreference : [Sketch-A-Net Beats Humans](https://arxiv.org/abs/1501.07873)",
    "428388": "What is the lr schedule and fine-tuning that works ? I am using \n\n    Reduceonplateau with initial_lr = 0.001 and min_lr= 1e-6, \n    batchsize=128\n    imgsize=128, \n    model=resnet50\n    all image samples - 34000 validation samples\n\nbut I can reach only **0.935 Top-5 accuracy** and **0.922 LB**.\n\nWhat is the fundamental mistake I might be doing ?"
  }
}