Especially with lower resource settings, my experience is that imagenet-pretrained weights (always) score better to their unpretrained counterparts, although the link from imagenet and medical images are not that obvious. (yes the story about low levels learning lower level details: an edge is always an edge).
\nSo my question is: Especially with the focus on medical imaging and this competition, is there any opportunity for using unpretrained weights and get good results? My experience is that, they usually start low in performance, and I probably stop early or use hyperparameters that are optimized for pretrained models, so they never surpass their pretrained twins.
", "messages": [ { "id": 2153787, "postDate": "2023-02-21T16:36:46.193Z", "content": "I have read that training from scratch benefits from good initialization. The random initializations available in different frameworks might not be the best for all tasks.
\nIf you have some easier task you are able to train well, then you could use those weights to initialize the base of your model, given that you keep the base same in both tasks, right?
", "rawMarkdown": "I have read that training from scratch benefits from good initialization. The random initializations available in different frameworks might not be the best for all tasks.\n\nIf you have some easier task you are able to train well, then you could use those weights to initialize the base of your model, given that you keep the base same in both tasks, right?", "votes": 1, "replies": [ { "id": 2153830, "postDate": "2023-02-21T16:54:21.057Z", "content": "I'm not qualified enough to answer this, but I think such tàsks really aren't 'easy'. Even if there is, imagenet pretrained weights might score similar to our from scratch model.
\nTraining from scratch could be most useful in scenarios where the domain of application is drastically separate from the original data on which model was trained on (like in this task: medical vs generic objects). But resources to train are a huge stumbling block.
\nNo wonder why transfer learning is such a hit.
Not an expert, but my take is that one needs to train the network for far too long and that too with a bunch of GPUs which anyway we don't have access to in Kaggle, to produce any meaningful result. Hence Pretrained weights of imagenet would be better off than training the model from scratch.
\nOff course you could unfreeze some last layers of the model, with a very low LR, while training on this dataset, which again means you could easily run into an OOM issue if not careful.
I was thinking, having much less detail then an imagenette dataset in medical imaging with sufficient data and strong models..?
\nMaybe that would be an easy overfit for a capable model, causing the low performance on a test set,, unless pretrained.
Maybe, maybe not. You may test the performance of models with cross validation folds, having taken care of data leakage. You could also employ many regularisation techniques to tackle overfitting.
", "rawMarkdown": "Maybe, maybe not. You may test the performance of models with cross validation folds, having taken care of data leakage. You could also employ many regularisation techniques to tackle overfitting." } ] } ] }, { "id": 2153249, "postDate": "2023-02-21T09:58:11.803Z", "content": "Especially with lower resource settings, my experience is that imagenet-pretrained weights (always) score better to their unpretrained counterparts, although the link from imagenet and medical images are not that obvious. (yes the story about low levels learning lower level details: an edge is always an edge).
\nSo my question is: Especially with the focus on medical imaging and this competition, is there any opportunity for using unpretrained weights and get good results? My experience is that, they usually start low in performance, and I probably stop early or use hyperparameters that are optimized for pretrained models, so they never surpass their pretrained twins.
", "rawMarkdown": "Especially with lower resource settings, my experience is that imagenet-pretrained weights (always) score better to their unpretrained counterparts, although the link from imagenet and medical images are not that obvious. (yes the story about low levels learning lower level details: an edge is always an edge).\n\nSo my question is: Especially with the focus on medical imaging and this competition, is there any opportunity for using unpretrained weights and get good results? My experience is that, they usually start low in performance, and I probably stop early or use hyperparameters that are optimized for pretrained models, so they never surpass their pretrained twins. \n\n", "votes": 1 } ], "comments": [ { "id": 2153787, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-21T16:36:46.193000", "content": "I have read that training from scratch benefits from good initialization. The random initializations available in different frameworks might not be the best for all tasks.
\nIf you have some easier task you are able to train well, then you could use those weights to initialize the base of your model, given that you keep the base same in both tasks, right?
", "votes": 1, "replies": [ { "id": 2153830, "author_name": "Sandy", "author_url": "", "post_date": "2023-02-21T16:54:21.057000", "content": "I'm not qualified enough to answer this, but I think such tàsks really aren't 'easy'. Even if there is, imagenet pretrained weights might score similar to our from scratch model.
\nTraining from scratch could be most useful in scenarios where the domain of application is drastically separate from the original data on which model was trained on (like in this task: medical vs generic objects). But resources to train are a huge stumbling block.
\nNo wonder why transfer learning is such a hit.
Not an expert, but my take is that one needs to train the network for far too long and that too with a bunch of GPUs which anyway we don't have access to in Kaggle, to produce any meaningful result. Hence Pretrained weights of imagenet would be better off than training the model from scratch.
\nOff course you could unfreeze some last layers of the model, with a very low LR, while training on this dataset, which again means you could easily run into an OOM issue if not careful.
I was thinking, having much less detail then an imagenette dataset in medical imaging with sufficient data and strong models..?
\nMaybe that would be an easy overfit for a capable model, causing the low performance on a test set,, unless pretrained.
Maybe, maybe not. You may test the performance of models with cross validation folds, having taken care of data leakage. You could also employ many regularisation techniques to tackle overfitting.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2153787": "I have read that training from scratch benefits from good initialization. The random initializations available in different frameworks might not be the best for all tasks.\n\nIf you have some easier task you are able to train well, then you could use those weights to initialize the base of your model, given that you keep the base same in both tasks, right?", "2153420": "Not an expert, but my take is that one needs to train the network for far too long and that too with a bunch of GPUs which anyway we don't have access to in Kaggle, to produce any meaningful result. Hence Pretrained weights of imagenet would be better off than training the model from scratch. \nOff course you could unfreeze some last layers of the model, with a very low LR, while training on this dataset, which again means you could easily run into an OOM issue if not careful.", "2153249": "Especially with lower resource settings, my experience is that imagenet-pretrained weights (always) score better to their unpretrained counterparts, although the link from imagenet and medical images are not that obvious. (yes the story about low levels learning lower level details: an edge is always an edge).\n\nSo my question is: Especially with the focus on medical imaging and this competition, is there any opportunity for using unpretrained weights and get good results? My experience is that, they usually start low in performance, and I probably stop early or use hyperparameters that are optimized for pretrained models, so they never surpass their pretrained twins. \n\n" } }