For smaller abnormalities, it makes sense to use larger images. Previous discussions in the forum have also pointed out that original images could work better for training and to identify the bounding boxes with smaller sizes.
\nI am using Corochan's notebook for Detectron2 FasterRCNN on original image data. However, I kept getting OOM (Out of memeory) error for the GPU.
By making some changes to the parameters to reduce GPU memory usage, I managed to run it for 1000 iterations.
\nI am choosing ---
\n\n\n
\n- R50 backbone (
\nconfig_name = \"faster_rcnn_R_50_FPN_3x.yaml\")- \n
ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256- \n
HorizontalFlip\": {\"p\": 0.75}- \n
SOLVER.IMS_PER_BATCH = 1- \n
SOLVER.BASE_LR = 0.005However, this run takes 1 hr 20 minutes on Kaggle. That means for a 9 hour time limit on Kaggle GPUs, I can only run the models for (540/80)*1000 = 6750 iterations. I don't think this would be enough to finish the training.
\n
âť“ Can someone please suggest me how I can train the model for higher iterations?
\nâť“ Or, how I can use a different set of parameters to reduce the memory usage by the GPU?
\nI must be missing something straight forward with transfer learning. I am not sure how to set it up for Detectron2.
Thanks for your help.
\nPS- I can't train it on my laptop as it has just a 4GB GPU. :)
", "messages": [ { "id": 1231961, "postDate": "2021-03-09T11:58:51.893Z", "content": "@sourabhchauhan Colab pro gives you double the RAM size 25 gb (Think so ) . You can upgrade it for 10 usd a month
", "rawMarkdown": "@sourabhchauhan Colab pro gives you double the RAM size 25 gb (Think so ) . You can upgrade it for 10 usd a month", "votes": 1, "replies": [ { "id": 1232360, "postDate": "2021-03-09T17:34:13.993Z", "content": "@usharengaraju But only if you live in the US/Canada, isn't it?
", "rawMarkdown": "@usharengaraju But only if you live in the US/Canada, isn't it?" }, { "id": 1232606, "postDate": "2021-03-09T21:20:22.593Z", "content": "@hannes82 I'll let you in on a little secret - just put a US postal code and you can get it.
", "rawMarkdown": "@hannes82 I'll let you in on a little secret - just put a US postal code and you can get it. " }, { "id": 1232690, "postDate": "2021-03-09T23:46:28.140Z", "content": "Ah I see :-) For now I think I will stick to kaggle ressources but thanks for letting me know!
", "rawMarkdown": "Ah I see :-) For now I think I will stick to kaggle ressources but thanks for letting me know!", "votes": 1 } ] }, { "id": 1231440, "postDate": "2021-03-09T01:44:13.577Z", "content": "You can use checkpoint, save the model before kaggle 9 hour limit ends your session, then load the model and continue training.
", "rawMarkdown": "You can use checkpoint, save the model before kaggle 9 hour limit ends your session, then load the model and continue training. ", "votes": 2 }, { "id": 1231408, "postDate": "2021-03-09T00:39:34.597Z", "content": "For smaller abnormalities, it makes sense to use larger images. Previous discussions in the forum have also pointed out that original images could work better for training and to identify the bounding boxes with smaller sizes.
\nI am using Corochan's notebook for Detectron2 FasterRCNN on original image data. However, I kept getting OOM (Out of memeory) error for the GPU.
By making some changes to the parameters to reduce GPU memory usage, I managed to run it for 1000 iterations.
\nI am choosing ---
\n\n\n
\n- R50 backbone (
\nconfig_name = \"faster_rcnn_R_50_FPN_3x.yaml\")- \n
ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256- \n
HorizontalFlip\": {\"p\": 0.75}- \n
SOLVER.IMS_PER_BATCH = 1- \n
SOLVER.BASE_LR = 0.005However, this run takes 1 hr 20 minutes on Kaggle. That means for a 9 hour time limit on Kaggle GPUs, I can only run the models for (540/80)*1000 = 6750 iterations. I don't think this would be enough to finish the training.
\n
âť“ Can someone please suggest me how I can train the model for higher iterations?
\nâť“ Or, how I can use a different set of parameters to reduce the memory usage by the GPU?
\nI must be missing something straight forward with transfer learning. I am not sure how to set it up for Detectron2.
Thanks for your help.
\nPS- I can't train it on my laptop as it has just a 4GB GPU. :)
", "rawMarkdown": "For smaller abnormalities, it makes sense to use larger images. Previous discussions in the forum have also pointed out that original images could work better for training and to identify the bounding boxes with smaller sizes. \nI am using [Corochan's notebook] (https://www.kaggle.com/corochann/vinbigdata-detectron2-train) for Detectron2 FasterRCNN on original image data. However, I kept getting OOM (Out of memeory) error for the GPU. \n\nBy making some changes to the parameters to reduce GPU memory usage, I managed to run it for 1000 iterations.\nI am choosing ---\n> \n- R50 backbone (`config_name = \"faster_rcnn_R_50_FPN_3x.yaml\" `)\n- `ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256`\n- `HorizontalFlip\": {\"p\": 0.75}`\n- `SOLVER.IMS_PER_BATCH = 1 `\n- `SOLVER.BASE_LR = 0.005`\n>\nHowever, this run takes 1 hr 20 minutes on Kaggle. That means for a 9 hour time limit on Kaggle GPUs, I can only run the models for (540/80)*1000 = 6750 iterations. I don't think this would be enough to finish the training.\n\nâť“ Can someone please suggest me how I can train the model for higher iterations? \nâť“ Or, how I can use a different set of parameters to reduce the memory usage by the GPU?\nI must be missing something straight forward with transfer learning. I am not sure how to set it up for Detectron2. \n\nThanks for your help. \n\nPS- I can't train it on my laptop as it has just a 4GB GPU. :) \n \n\n\n", "votes": 2 } ], "comments": [ { "id": 1231961, "author_name": "Tensor Girl", "author_url": "", "post_date": "2021-03-09T11:58:51.893000", "content": "@sourabhchauhan Colab pro gives you double the RAM size 25 gb (Think so ) . You can upgrade it for 10 usd a month
", "votes": 1, "replies": [ { "id": 1232360, "author_name": "Hannes Ă–hler", "author_url": "", "post_date": "2021-03-09T17:34:13.993000", "content": "@usharengaraju But only if you live in the US/Canada, isn't it?
", "votes": 0, "replies": [] }, { "id": 1232606, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-03-09T21:20:22.593000", "content": "@hannes82 I'll let you in on a little secret - just put a US postal code and you can get it.
", "votes": 0, "replies": [] }, { "id": 1232690, "author_name": "Hannes Ă–hler", "author_url": "", "post_date": "2021-03-09T23:46:28.140000", "content": "Ah I see :-) For now I think I will stick to kaggle ressources but thanks for letting me know!
", "votes": 1, "replies": [] } ] }, { "id": 1231440, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-03-09T01:44:13.577000", "content": "You can use checkpoint, save the model before kaggle 9 hour limit ends your session, then load the model and continue training.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1231961": "@sourabhchauhan Colab pro gives you double the RAM size 25 gb (Think so ) . You can upgrade it for 10 usd a month", "1231440": "You can use checkpoint, save the model before kaggle 9 hour limit ends your session, then load the model and continue training. ", "1231408": "For smaller abnormalities, it makes sense to use larger images. Previous discussions in the forum have also pointed out that original images could work better for training and to identify the bounding boxes with smaller sizes. \nI am using [Corochan's notebook] (https://www.kaggle.com/corochann/vinbigdata-detectron2-train) for Detectron2 FasterRCNN on original image data. However, I kept getting OOM (Out of memeory) error for the GPU. \n\nBy making some changes to the parameters to reduce GPU memory usage, I managed to run it for 1000 iterations.\nI am choosing ---\n> \n- R50 backbone (`config_name = \"faster_rcnn_R_50_FPN_3x.yaml\" `)\n- `ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256`\n- `HorizontalFlip\": {\"p\": 0.75}`\n- `SOLVER.IMS_PER_BATCH = 1 `\n- `SOLVER.BASE_LR = 0.005`\n>\nHowever, this run takes 1 hr 20 minutes on Kaggle. That means for a 9 hour time limit on Kaggle GPUs, I can only run the models for (540/80)*1000 = 6750 iterations. I don't think this would be enough to finish the training.\n\nâť“ Can someone please suggest me how I can train the model for higher iterations? \nâť“ Or, how I can use a different set of parameters to reduce the memory usage by the GPU?\nI must be missing something straight forward with transfer learning. I am not sure how to set it up for Detectron2. \n\nThanks for your help. \n\nPS- I can't train it on my laptop as it has just a 4GB GPU. :) \n \n\n\n" } }