![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2411547%2Fc390fea597d401133107ece73e680a85%2Fdownload.png?generation=1590703728798221&alt=media\")
Hi there again!
\n\nI wanted to ask you guys what learning rates you guys are using to train transfer learning on your models? I originally used 1e-3, but after switching to a regression problem with MSELoss, I just had the feeling that my validation score would continue to fluctuate. \nThus, I believed that it's a too large learning rate, but I might be wrong.
\n\nI ran the fastai learning rate finder, and it suggests me a loss of around 1e-8, which in my opinion seems incredibly low. (See image below). Nonetheless, I am still running the experiments.
\n\n
What learning rates are you guys using, if I may ask? And should optimizers like Rectified Adam, or Over9000 converge to an optimum, inspite different starting learning rates?
\n\nCheers and take care!
", "messages": [ { "id": 865792, "postDate": "2020-05-28T22:13:36.867Z", "content": "Hi there again!
\n\nI wanted to ask you guys what learning rates you guys are using to train transfer learning on your models? I originally used 1e-3, but after switching to a regression problem with MSELoss, I just had the feeling that my validation score would continue to fluctuate. \nThus, I believed that it's a too large learning rate, but I might be wrong.
\n\nI ran the fastai learning rate finder, and it suggests me a loss of around 1e-8, which in my opinion seems incredibly low. (See image below). Nonetheless, I am still running the experiments.
\n\n
What learning rates are you guys using, if I may ask? And should optimizers like Rectified Adam, or Over9000 converge to an optimum, inspite different starting learning rates?
\n\nCheers and take care!
", "rawMarkdown": "Hi there again!\n\nI wanted to ask you guys what learning rates you guys are using to train transfer learning on your models? I originally used 1e-3, but after switching to a regression problem with MSELoss, I just had the feeling that my validation score would continue to fluctuate. \nThus, I believed that it's a too large learning rate, but I might be wrong.\n\nI ran the fastai learning rate finder, and it suggests me a loss of around 1e-8, which in my opinion seems incredibly low. (See image below). Nonetheless, I am still running the experiments.\n\n\n\nWhat learning rates are you guys using, if I may ask? And should optimizers like Rectified Adam, or Over9000 converge to an optimum, inspite different starting learning rates?\n\nCheers and take care!\n", "votes": 5 }, { "id": 866697, "postDate": "2020-05-29T15:57:57.470Z", "content": "I currently use 1e-4 for backbone and 1e-3 for head. 1e-3 overall gives me bad result and 1e-4 is just slower. Can also try 1e-5 1e-3 I think.
\n\nIn general the recommendation from lrfind be it fastai's or pytorch lightning's are awful because they are based on an automated analysis of the graph that is very naive. You should look at the graph and pick manually.
", "rawMarkdown": "I currently use 1e-4 for backbone and 1e-3 for head. 1e-3 overall gives me bad result and 1e-4 is just slower. Can also try 1e-5 1e-3 I think.\n\nIn general the recommendation from lrfind be it fastai's or pytorch lightning's are awful because they are based on an automated analysis of the graph that is very naive. You should look at the graph and pick manually.", "votes": 4, "replies": [ { "id": 866726, "postDate": "2020-05-29T16:24:33.323Z", "content": "Thank you very much for your recommendation. That sounds reasonable - i had no considered splitting the LR for backbone and head, but it makes completely sense!
", "rawMarkdown": "Thank you very much for your recommendation. That sounds reasonable - i had no considered splitting the LR for backbone and head, but it makes completely sense!" } ] }, { "id": 869391, "postDate": "2020-06-01T01:47:00.853Z", "content": "I believe 3e-4 is the best LR for Adam optim.
", "rawMarkdown": "I believe 3e-4 is the best LR for Adam optim.", "votes": 1 }, { "id": 865824, "postDate": "2020-05-28T23:12:29.880Z", "content": "Its highly dependent on network ... and other stuff... It is hard to give advice. Bet I use somewhere between 1e-3 to 1e-4
That is true! Currently I am mostly testing with an EffNetB0. I am still experimenting a bit around.\n1e-8 converged way too slow. Whereas 1e-2, had low train loss, but the valid loss almost exploded...
", "rawMarkdown": "That is true! Currently I am mostly testing with an EffNetB0. I am still experimenting a bit around.\n1e-8 converged way too slow. Whereas 1e-2, had low train loss, but the valid loss almost exploded...", "votes": 1 }, { "id": 865845, "postDate": "2020-05-28T23:45:19.600Z", "content": "B0 is too big for my poor gpu... I am still doing everything on resnet34 =) But good luck! =)
If I am not mistaken, EffnetB0 has significantly less parameters than Resnet34! The first one has around 5 mio. whereas the later has around 20 mio. parameters.\nWhich is kinda the whole point of EfficientNets. You should give it a shot! :)
", "rawMarkdown": "If I am not mistaken, EffnetB0 has significantly less parameters than Resnet34! The first one has around 5 mio. whereas the later has around 20 mio. parameters.\nWhich is kinda the whole point of EfficientNets. You should give it a shot! :) ", "votes": 1 }, { "id": 866213, "postDate": "2020-05-29T07:44:12.680Z", "content": "EffnetB0 has less parameters and achieved better performances than ResNet34 (at least on Imagenet).\n![](\"https://raw.githubusercontent.com/tensorflow/tpu/master/models/official/efficientnet/g3doc/params.png\")
()
()",
"votes": 1
},
{
"id": 866536,
"postDate": "2020-05-29T13:27:17.340Z",
"content": "Less parameters doesn't mean correlates with efficient of gpu usage. Efficient net uses more expensive activation function and many other stuff ... There is nice comparison and more in depth analysis here https://github.com/rwightman/pytorch-image-models/blob/master/notebooks/EffResNetComparison.ipynb . But yes in general they are more accurate than resnets =)
", "rawMarkdown": "Less parameters doesn't mean correlates with efficient of gpu usage. Efficient net uses more expensive activation function and many other stuff ... There is nice comparison and more in depth analysis here https://github.com/rwightman/pytorch-image-models/blob/master/notebooks/EffResNetComparison.ipynb . But yes in general they are more accurate than resnets =) ", "votes": 6 }, { "id": 866543, "postDate": "2020-05-29T13:33:16.110Z", "content": "That is true, the swish activation is more complex, as well as the mobile conv blocks! thanks for pointing this out! :)
", "rawMarkdown": "That is true, the swish activation is more complex, as well as the mobile conv blocks! thanks for pointing this out! :)", "votes": 1 }, { "id": 866839, "postDate": "2020-05-29T18:22:12.877Z", "content": "I was not aware of that, thank you! :)
", "rawMarkdown": "I was not aware of that, thank you! :)", "votes": 1 } ] } ], "comments": [ { "id": 866697, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-05-29T15:57:57.470000", "content": "I currently use 1e-4 for backbone and 1e-3 for head. 1e-3 overall gives me bad result and 1e-4 is just slower. Can also try 1e-5 1e-3 I think.
\n\nIn general the recommendation from lrfind be it fastai's or pytorch lightning's are awful because they are based on an automated analysis of the graph that is very naive. You should look at the graph and pick manually.
", "votes": 4, "replies": [ { "id": 866726, "author_name": "Claudio Fanconi", "author_url": "", "post_date": "2020-05-29T16:24:33.323000", "content": "Thank you very much for your recommendation. That sounds reasonable - i had no considered splitting the LR for backbone and head, but it makes completely sense!
", "votes": 0, "replies": [] } ] }, { "id": 869391, "author_name": "seefun", "author_url": "", "post_date": "2020-06-01T01:47:00.853000", "content": "I believe 3e-4 is the best LR for Adam optim.
", "votes": 1, "replies": [] }, { "id": 865824, "author_name": "DrHB", "author_url": "", "post_date": "2020-05-28T23:12:29.880000", "content": "Its highly dependent on network ... and other stuff... It is hard to give advice. Bet I use somewhere between 1e-3 to 1e-4
That is true! Currently I am mostly testing with an EffNetB0. I am still experimenting a bit around.\n1e-8 converged way too slow. Whereas 1e-2, had low train loss, but the valid loss almost exploded...
", "votes": 1, "replies": [] }, { "id": 865845, "author_name": "DrHB", "author_url": "", "post_date": "2020-05-28T23:45:19.600000", "content": "B0 is too big for my poor gpu... I am still doing everything on resnet34 =) But good luck! =)
If I am not mistaken, EffnetB0 has significantly less parameters than Resnet34! The first one has around 5 mio. whereas the later has around 20 mio. parameters.\nWhich is kinda the whole point of EfficientNets. You should give it a shot! :)
", "votes": 1, "replies": [] }, { "id": 866213, "author_name": "Pasquale", "author_url": "", "post_date": "2020-05-29T07:44:12.680000", "content": "EffnetB0 has less parameters and achieved better performances than ResNet34 (at least on Imagenet).\n()
Less parameters doesn't mean correlates with efficient of gpu usage. Efficient net uses more expensive activation function and many other stuff ... There is nice comparison and more in depth analysis here https://github.com/rwightman/pytorch-image-models/blob/master/notebooks/EffResNetComparison.ipynb . But yes in general they are more accurate than resnets =)
", "votes": 6, "replies": [] }, { "id": 866543, "author_name": "Claudio Fanconi", "author_url": "", "post_date": "2020-05-29T13:33:16.110000", "content": "That is true, the swish activation is more complex, as well as the mobile conv blocks! thanks for pointing this out! :)
", "votes": 1, "replies": [] }, { "id": 866839, "author_name": "Pasquale", "author_url": "", "post_date": "2020-05-29T18:22:12.877000", "content": "I was not aware of that, thank you! :)
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "865792": "Hi there again!\n\nI wanted to ask you guys what learning rates you guys are using to train transfer learning on your models? I originally used 1e-3, but after switching to a regression problem with MSELoss, I just had the feeling that my validation score would continue to fluctuate. \nThus, I believed that it's a too large learning rate, but I might be wrong.\n\nI ran the fastai learning rate finder, and it suggests me a loss of around 1e-8, which in my opinion seems incredibly low. (See image below). Nonetheless, I am still running the experiments.\n\n\n\nWhat learning rates are you guys using, if I may ask? And should optimizers like Rectified Adam, or Over9000 converge to an optimum, inspite different starting learning rates?\n\nCheers and take care!\n", "866697": "I currently use 1e-4 for backbone and 1e-3 for head. 1e-3 overall gives me bad result and 1e-4 is just slower. Can also try 1e-5 1e-3 I think.\n\nIn general the recommendation from lrfind be it fastai's or pytorch lightning's are awful because they are based on an automated analysis of the graph that is very naive. You should look at the graph and pick manually.", "869391": "I believe 3e-4 is the best LR for Adam optim.", "865824": "Its highly dependent on network ... and other stuff... It is hard to give advice. Bet I use somewhere between `1e-3` to `1e-4`" } }