{ "id": 376879, "title": "[LB 0.47] MONAI training and inference pipeline", "url": "/competitions/rsna-breast-cancer-detection/discussion/376879", "author_name": "Yiheng Wang", "post_date": "2023-01-09T03:26:25.631000", "votes": 71, "comment_count": 61, "views": 0, "content": "

In this topic, I will share my experiences in this challenge, and open source all of my works on how to get a 0.47 (the number may change) model.

\n

Inference code link:
\nhttps://www.kaggle.com/code/yiheng/monai-baseline
\nTraining code link:
\nhttps://www.kaggle.com/code/yiheng/monai-pipeline-training/notebook

\n

you can also clone the training pipeline from github:

\n
git clone git@github.com:yiheng-wang-nv/rsna_breast_monai_solution.git\n
\n

The following is my local and lb results (will update if having any progresses):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
pf1 localbin pf1 localLB
fold 0, agg by max0.27530.33520.38
fold 0, agg by mean0.20420.33920.42
fold 0, agg by mean, 2 tta0.20450.38790.44
4 folds, agg by mean, 2 tta0.22340.31290.47
", "messages": [ { "id": 2092068, "postDate": "2023-01-09T03:26:25.630Z", "content": "

In this topic, I will share my experiences in this challenge, and open source all of my works on how to get a 0.47 (the number may change) model.

\n

Inference code link:
\nhttps://www.kaggle.com/code/yiheng/monai-baseline
\nTraining code link:
\nhttps://www.kaggle.com/code/yiheng/monai-pipeline-training/notebook

\n

you can also clone the training pipeline from github:

\n
git clone git@github.com:yiheng-wang-nv/rsna_breast_monai_solution.git\n
\n

The following is my local and lb results (will update if having any progresses):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
pf1 localbin pf1 localLB
fold 0, agg by max0.27530.33520.38
fold 0, agg by mean0.20420.33920.42
fold 0, agg by mean, 2 tta0.20450.38790.44
4 folds, agg by mean, 2 tta0.22340.31290.47
", "rawMarkdown": "In this topic, I will share my experiences in this challenge, and open source all of my works on how to get a 0.47 (the number may change) model.\n\nInference code link:\nhttps://www.kaggle.com/code/yiheng/monai-baseline\nTraining code link:\nhttps://www.kaggle.com/code/yiheng/monai-pipeline-training/notebook\n\nyou can also clone the training pipeline from github:\n```\ngit clone git@github.com:yiheng-wang-nv/rsna_breast_monai_solution.git\n```\n\nThe following is my local and lb results (will update if having any progresses):\n\n| | pf1 local | bin pf1 local | LB |\n| --- | --- | --- | --- |\n| fold 0, agg by max | 0.2753 | 0.3352 | 0.38 |\n| fold 0, agg by mean | 0.2042 | 0.3392 | 0.42 |\n| fold 0, agg by mean, 2 tta | 0.2045 | 0.3879 | 0.44 |\n| 4 folds, agg by mean, 2 tta | 0.2234 | 0.3129 | 0.47 |", "votes": 71 }, { "id": 2092074, "postDate": "2023-01-09T03:46:48.203Z", "content": "

[Update 1] My purpose is to make use of MONAI, as well as other libs (such as timm for network, dali for acceleration) to build a decent model.
\nFirst of all, I wish to thank @theoviel , I followed your topic and your code to design my training pipeline, which can be summarized as the following settings:

\n\n

Based on these settings, I started to do experiments, and my initial purpose is to reach pf1 ~0.083 and auc 0.755 (these two values are from theoviel's open sourced notebook I mentioned above). What I tuned include:

\n
    \n
  1. number of epochs
    2. \n
  2. pos sample weights on bce loss
    3. \n
  3. drop rate of efficientnet (this idea is achieved from @hengck23 in his topic
    \nafter ~ 8 experiments, I got pf1 0.10475 on one fold locally. Then I switched to use larger input size: 1024 * 1024 and do all 4 folds experiments. In this step, I did not tune any parameters, but just try to finish training, and then submit to LB.
    4. \n
", "rawMarkdown": "[Update 1] My purpose is to make use of MONAI, as well as other libs (such as timm for network, dali for acceleration) to build a decent model.\nFirst of all, I wish to thank @theoviel , I followed [your topic](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369769) and [your code](https://www.kaggle.com/code/theoviel/rsna-breast-baseline-inference) to design my training pipeline, which can be summarized as the following settings:\n\n- group stratified 4 fold data split\n- 512 * 512 input size\n- h flip augmentation\n- 4 epoch, lr decay from 3e-4\n- timm, efnv2s network\n\nBased on these settings, I started to do experiments, and my initial purpose is to reach pf1 ~0.083 and auc 0.755 (these two values are from theoviel's open sourced notebook I mentioned above). What I tuned include:\n\n1. number of epochs\n2. pos sample weights on bce loss\n3. drop rate of efficientnet (this idea is achieved from @hengck23 in [his topic](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333)\nafter ~ 8 experiments, I got pf1 0.10475 on one fold locally. Then I switched to use larger input size: 1024 * 1024 and do all 4 folds experiments. In this step, I did not tune any parameters, but just try to finish training, and then submit to LB.", "votes": 13, "replies": [ { "id": 2092075, "postDate": "2023-01-09T03:49:50.580Z", "content": "

My next plan:

\n
    \n
  1. try to train a 2048 * 2048 model, which I expect a ~0.51 LB result as @hengck23 shared : )
    2. \n
  2. adjust the input, since there are many black area in each figure, which may be useless.
    3. \n
", "rawMarkdown": "My next plan:\n1. try to train a 2048 * 2048 model, which I expect a ~0.51 LB result as @hengck23 shared : )\n2. adjust the input, since there are many black area in each figure, which may be useless.", "votes": 6, "replies": [ { "id": 2092470, "postDate": "2023-01-09T11:03:51.207Z", "content": "

\"adjust the input, since there are many black area in each figure\"
\nhttps://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region

\n

i think this will take about 10 min to process the all kaggle test images

", "rawMarkdown": "\"adjust the input, since there are many black area in each figure\"\nhttps://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region\n\ni think this will take about 10 min to process the all kaggle test images", "votes": 3 }, { "id": 2095384, "postDate": "2023-01-11T10:14:51.273Z", "content": "

Now I give up to train 2048 size model because of limited resources, and I will focus on roi based method as others suggested.
\nTo get a quick experiment, I will try to use the dataset provided by @remekkinas in his topic: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369754 (Thanks Remek!) locally, and if getting any progresses I will enhance my inference notebook and share my results.

", "rawMarkdown": "Now I give up to train 2048 size model because of limited resources, and I will focus on roi based method as others suggested.\nTo get a quick experiment, I will try to use the dataset provided by @remekkinas in his topic: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369754 (Thanks Remek!) locally, and if getting any progresses I will enhance my inference notebook and share my results.", "votes": 1 }, { "id": 2096719, "postDate": "2023-01-12T08:19:59.647Z", "content": "

if you are using crop strategy, do remember to take care of the case if no box can be detected. i spend a few days to debug this submission error.

\n
box = run_box_detection_model(image)\nif box is None:\n      use whole image\nelse:\n       crop image\n
", "rawMarkdown": "if you are using crop strategy, do remember to take care of the case if no box can be detected. i spend a few days to debug this submission error.\n\n```\nbox = run_box_detection_model(image)\nif box is None:\n use whole image\nelse:\n crop image\n\n\n```", "votes": 4 }, { "id": 2097348, "postDate": "2023-01-12T15:57:13.713Z", "content": "

and … use

\n
try:\n....\nexcept Exception as e \n... rollback operations in try\n
\n

I have one place in code which is unstable - probably in test DS is dicom file which break rules from train. I have spent a lot of time debugging - have not found any problem. Then decided to wrap code in try except and for this image rollback all operations - use oryginal image instead of cropped and postprocessed.

", "rawMarkdown": "and ... use \n\n```\ntry:\n....\nexcept Exception as e \n... rollback operations in try\n```\nI have one place in code which is unstable - probably in test DS is dicom file which break rules from train. I have spent a lot of time debugging - have not found any problem. Then decided to wrap code in try except and for this image rollback all operations - use oryginal image instead of cropped and postprocessed." }, { "id": 2101521, "postDate": "2023-01-16T01:11:18.770Z", "content": "

Which part of Heng's code did you wrap? I tried it but sub still fails.

", "rawMarkdown": "Which part of Heng's code did you wrap? I tried it but sub still fails. " } ] }, { "id": 2092868, "postDate": "2023-01-09T17:01:04.390Z", "content": "

Hi @yiheng

\n

Thanks for sharing your insights.. Can you mention the basis for your CV split..

\n

I am currently using StratifiedGroupKFold (5 folds) with patient_id for grouping

", "rawMarkdown": "Hi @yiheng \n\nThanks for sharing your insights.. Can you mention the basis for your CV split.. \n\nI am currently using StratifiedGroupKFold (5 folds) with patient_id for grouping", "replies": [ { "id": 2093695, "postDate": "2023-01-10T08:19:05.527Z", "content": "

Similarly, I also used StratifiedGroupKFold with k=4 (the same as Theo does in his notebook)

", "rawMarkdown": "Similarly, I also used StratifiedGroupKFold with k=4 (the same as Theo does in his notebook)", "votes": 2 } ] }, { "id": 2100708, "postDate": "2023-01-15T11:27:02.840Z", "content": "

Hi @yiheng I want to know when ensembling 4 folds, which validation set do you use to choose threshold to binarize ?

", "rawMarkdown": "Hi @yiheng I want to know when ensembling 4 folds, which validation set do you use to choose threshold to binarize ?", "replies": [ { "id": 2101783, "postDate": "2023-01-16T06:54:52.707Z", "content": "

Hi, the whole dataset. I used each fold's model to do predictions on each validation set, and then combine all predictions together to search the threshold.

", "rawMarkdown": "Hi, the whole dataset. I used each fold's model to do predictions on each validation set, and then combine all predictions together to search the threshold.", "votes": 1 } ] } ] }, { "id": 2100083, "postDate": "2023-01-14T23:32:17.640Z", "content": "

Great Job!

", "rawMarkdown": "Great Job!", "votes": 1 }, { "id": 2092163, "postDate": "2023-01-09T06:42:59.137Z", "content": "

Are you using Kaggle for training, some other cloud provider, or local GPU setup?

", "rawMarkdown": "Are you using Kaggle for training, some other cloud provider, or local GPU setup?", "votes": 1, "replies": [ { "id": 2092179, "postDate": "2023-01-09T07:17:24.047Z", "content": "

Hi @aadimator , I use local GPUs

", "rawMarkdown": "Hi @aadimator , I use local GPUs", "votes": 1 } ] }, { "id": 2104971, "postDate": "2023-01-18T07:17:55.217Z", "content": "

[Update 2] I used @hengck23 's trained model (in: https://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region) to infer the bounding boxes for all training images and then do the following preprocessing:
\ndicom ---> crop by bounding box ---> resize to 1024 * 1024
\nthe resized dataset is uploaded to: https://www.kaggle.com/datasets/yiheng/rsna-breast-1024

\n

Now I'm training with this dataset, if having any progresses, I will update the topic and the notebook.
\nThe next plans are:

\n
    \n
  1. train with different backbone
    2. \n
  2. train with different input size
    3. \n
  3. optimize the inference time
    4. \n
", "rawMarkdown": "[Update 2] I used @hengck23 's trained model (in: https://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region) to infer the bounding boxes for all training images and then do the following preprocessing:\ndicom ---> crop by bounding box ---> resize to 1024 * 1024\nthe resized dataset is uploaded to: https://www.kaggle.com/datasets/yiheng/rsna-breast-1024\n\nNow I'm training with this dataset, if having any progresses, I will update the topic and the notebook.\nThe next plans are:\n1. train with different backbone\n2. train with different input size\n3. optimize the inference time", "replies": [ { "id": 2110936, "postDate": "2023-01-22T14:39:57.640Z", "content": "

remember to use try except to capture dengerate case at submissiom
\nhttps://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example/comments#2104692

", "rawMarkdown": "remember to use try except to capture dengerate case at submissiom\nhttps://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example/comments#2104692", "votes": 1 } ] }, { "id": 2102690, "postDate": "2023-01-16T19:37:33.720Z", "content": "

Hi! Thanks for your work!
\nI tried to reproduce and set batch_size to 16 due to my single GPU… and I find that the eval loss is increasing instead of decreasing…
\nShould I decreas the lr for my smaller batch_size?

\n

Thanks!

", "rawMarkdown": "Hi! Thanks for your work!\nI tried to reproduce and set batch_size to 16 due to my single GPU... and I find that the eval loss is increasing instead of decreasing...\nShould I decreas the lr for my smaller batch_size?\n\nThanks!", "replies": [ { "id": 2102835, "postDate": "2023-01-16T21:29:17.630Z", "content": "

I tried this as well and failed, the model didn't converge, to make effnetv2 train I had to enable gradient checkpointing and keep batch size at 64, and the easier thing that I didn't test is to lower the learning rate, maybe it'll work comparatively to keeping batch size large

", "rawMarkdown": "I tried this as well and failed, the model didn't converge, to make effnetv2 train I had to enable gradient checkpointing and keep batch size at 64, and the easier thing that I didn't test is to lower the learning rate, maybe it'll work comparatively to keeping batch size large", "replies": [ { "id": 2102910, "postDate": "2023-01-16T22:06:47.737Z", "content": "

I tried many models and I found effnetv2 is really hard to train… The eval loss often didn't converge…
\nI will try to decrease the lr from 3e-4 to 1e-4, then maybe 5e-5 to have a look.
\nBut as you said, i don't know if this only works for large batch size

", "rawMarkdown": "I tried many models and I found effnetv2 is really hard to train... The eval loss often didn't converge...\nI will try to decrease the lr from 3e-4 to 1e-4, then maybe 5e-5 to have a look.\nBut as you said, i don't know if this only works for large batch size" } ] }, { "id": 2103174, "postDate": "2023-01-17T04:00:31.857Z", "content": "

could you try large batch size with smaller image size first? Or maybe you can upsample the positive samples, I guess a small batch size may make the task hard since there may have many batches that have no pos samples.

", "rawMarkdown": "could you try large batch size with smaller image size first? Or maybe you can upsample the positive samples, I guess a small batch size may make the task hard since there may have many batches that have no pos samples.", "replies": [ { "id": 2103803, "postDate": "2023-01-17T11:58:52.540Z", "content": "

Thanks! now I'm trying to use upsampling and also gradient accumulation to have a try!

", "rawMarkdown": "Thanks! now I'm trying to use upsampling and also gradient accumulation to have a try!" }, { "id": 2104315, "postDate": "2023-01-17T18:40:59.550Z", "content": "

I tried with upsampling the pos to 1/8 of each batch and gradient accumulation to get 64 a batch.
\nBut I still get the overfitting problems and cannot even close to your score…

", "rawMarkdown": "I tried with upsampling the pos to 1/8 of each batch and gradient accumulation to get 64 a batch.\nBut I still get the overfitting problems and cannot even close to your score..." }, { "id": 2104979, "postDate": "2023-01-18T07:22:45.210Z", "content": "

what is your val score? Have you tried to train on 512 * 512 size? It should get a ~0.1 val score.

", "rawMarkdown": "what is your val score? Have you tried to train on 512 * 512 size? It should get a ~0.1 val score." } ] } ] }, { "id": 2099092, "postDate": "2023-01-14T07:01:16.593Z", "content": "

Great Job!
\nSimple & Impressive for your sharing.

\n

Could you get my 3 simple questions?

\n
    \n

  1. As following your training config files, the pos_weight in loss function set to 1.
    \nDid your 0.47 LB score come from it? or other value?
    \nI'm just wondering because the pos_weight in most of other codes set to range of 10~50.

    2. \n

  2. In validation for each folds, how did you choose the best epoch weight?
    \n2 scores(bin pfbeta1(thres 0.1) & auc) can be printed.
    \nwhich scores did you follow?

    3. \n

  3. In your experiment, Was just 10 epoch training enough to get the result?

    4. \n
\n

Thank you!

", "rawMarkdown": "Great Job!\nSimple & Impressive for your sharing.\n\nCould you get my 3 simple questions?\n1. As following your training config files, the pos_weight in loss function set to 1.\nDid your 0.47 LB score come from it? or other value?\nI'm just wondering because the pos_weight in most of other codes set to range of 10~50.\n\n2. In validation for each folds, how did you choose the best epoch weight?\n2 scores(bin pfbeta1(thres 0.1) & auc) can be printed.\nwhich scores did you follow?\n\n3. In your experiment, Was just 10 epoch training enough to get the result?\n\nThank you!", "replies": [ { "id": 2101788, "postDate": "2023-01-16T06:56:48.293Z", "content": "
    \n
  1. Yes, I tried other pos weights but the final result is similar. 1 works on my solution.
    2. \n
  2. I select the best pf1 (non binariezed) weights.
    3. \n
  3. I think 10 epoch is enough, I also tried 20, but the total val score is similar.
    4. \n
", "rawMarkdown": "1. Yes, I tried other pos weights but the final result is similar. 1 works on my solution.\n2. I select the best pf1 (non binariezed) weights.\n3. I think 10 epoch is enough, I also tried 20, but the total val score is similar.", "votes": 1, "replies": [ { "id": 2105064, "postDate": "2023-01-18T08:42:03.500Z", "content": "

As for 3, I detected a new finding.
\nAccording to my latest experiments on cropped images, it's easier for training set to overfit, and reduce epoch number from 10 to 5 is enough to get similar val score and also lower train score (less overfit).

", "rawMarkdown": "As for 3, I detected a new finding.\nAccording to my latest experiments on cropped images, it's easier for training set to overfit, and reduce epoch number from 10 to 5 is enough to get similar val score and also lower train score (less overfit).", "votes": 1 } ] } ] }, { "id": 2097286, "postDate": "2023-01-12T14:57:56.947Z", "content": "

Thank you for sharing your work! As you already said, you use StratifiedGroupKFold with k=4 and binarize f1 to get the final prediction. How do you find the threshold to use in the submissions? As you compute the mean along all the folds, do you compute the mean of all the thresholds?

", "rawMarkdown": "Thank you for sharing your work! As you already said, you use StratifiedGroupKFold with k=4 and binarize f1 to get the final prediction. How do you find the threshold to use in the submissions? As you compute the mean along all the folds, do you compute the mean of all the thresholds?", "replies": [ { "id": 2104975, "postDate": "2023-01-18T07:20:29.123Z", "content": "

I search the threshold based on all predictions (from all folds models).

", "rawMarkdown": "I search the threshold based on all predictions (from all folds models)." } ] }, { "id": 2095373, "postDate": "2023-01-11T10:08:45.247Z", "content": "

Hi @yiheng . Are you also tracking AUC? If so, could you report it here? Thanks.

", "rawMarkdown": "Hi @yiheng . Are you also tracking AUC? If so, could you report it here? Thanks. ", "replies": [ { "id": 2095389, "postDate": "2023-01-11T10:18:57.003Z", "content": "

Hi @pheadrus , AUC is printed in my training pipeline however I did not record it in my sheet : (
\nI will add this part in my following experiments, and post my updates after getting a better result.

", "rawMarkdown": "Hi @pheadrus , AUC is printed in my training pipeline however I did not record it in my sheet : (\nI will add this part in my following experiments, and post my updates after getting a better result.", "votes": 3 } ] }, { "id": 2095341, "postDate": "2023-01-11T09:44:33.350Z", "content": "

@yiheng
\nDid you make any resampling (under or over)? And how much your fbeta fluctuated during training time on validation set? Did you get 0.10475 in a consistent progress?

", "rawMarkdown": "@yiheng \nDid you make any resampling (under or over)? And how much your fbeta fluctuated during training time on validation set? Did you get 0.10475 in a consistent progress? ", "replies": [ { "id": 2095390, "postDate": "2023-01-11T10:20:21.073Z", "content": "

Hi @simonalerdic , in 512 * 512 experiments, val score increases consistently.
\nSo far I did not try resampling but just change the pos weight in bce loss.

", "rawMarkdown": "Hi @simonalerdic , in 512 * 512 experiments, val score increases consistently.\nSo far I did not try resampling but just change the pos weight in bce loss.", "votes": 1, "replies": [ { "id": 2096561, "postDate": "2023-01-12T06:17:58.130Z", "content": "

Is it possible to reproduce your training notebook score on kaggle env? Have you tried? In here, model size and image size, batch size gets limited due to resource and adjusting them from yours gave poor result. Could you please confirm?

", "rawMarkdown": "Is it possible to reproduce your training notebook score on kaggle env? Have you tried? In here, model size and image size, batch size gets limited due to resource and adjusting them from yours gave poor result. Could you please confirm?" }, { "id": 2105047, "postDate": "2023-01-18T08:27:34.160Z", "content": "

Hi @simonalerdic , I'm afraid I cannot do this kind of works recently since we have limited time for this challenge. I prefer finishing my current experiments first:

\n
    \n
  1. try training with cropped images
    2. \n
  2. try different networks
    3. \n
  3. try different input size
    4. \n
\n

If having 512*512 results, I will post my log here if it is helpful.

", "rawMarkdown": "Hi @simonalerdic , I'm afraid I cannot do this kind of works recently since we have limited time for this challenge. I prefer finishing my current experiments first:\n1. try training with cropped images\n2. try different networks\n3. try different input size\n\nIf having 512*512 results, I will post my log here if it is helpful." } ] } ] }, { "id": 2095096, "postDate": "2023-01-11T07:43:59.407Z", "content": "

Is the batch size an important parameter? I reduced batch size to avoid out of memory, but the validation score will be very low.

", "rawMarkdown": "Is the batch size an important parameter? I reduced batch size to avoid out of memory, but the validation score will be very low.", "replies": [ { "id": 2095376, "postDate": "2023-01-11T10:10:02.600Z", "content": "

I think it has some impacts, since the percentage of pos samples is low. Maybe you can try to accumulate gradients if not having enough gpu memory.

", "rawMarkdown": "I think it has some impacts, since the percentage of pos samples is low. Maybe you can try to accumulate gradients if not having enough gpu memory.", "votes": 3, "replies": [ { "id": 2103574, "postDate": "2023-01-17T08:43:27.883Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!" } ] }, { "id": 2095496, "postDate": "2023-01-11T12:02:35.327Z", "content": "

Hi @juntaojiang , batch size is really trick, while the relationship between BS and LR is also very important.

\n
    \n
  1. When you reduce BS, try to scale LR at the same time. For example, the original BS is 32 and the LR is 1e-4, and you want to reduce the BS to 16, try to reduce LR to 5e-5. If you have more time, you can search the optimal LR around 5e-5.
    2. \n
  2. If you use multiple GPU to train, never put only one image per gpus. For example, if you use 8 cards, your minimum batch size should be 16, while there are 2 images per gpus. If you set BS=8, then there is 1 image per gpu, which will cause the model to perform much worse. It's caused by the BatchNorm in most CNN models.
    3. \n
", "rawMarkdown": "Hi @juntaojiang , batch size is really trick, while the relationship between BS and LR is also very important.\n\n1. When you reduce BS, try to scale LR at the same time. For example, the original BS is 32 and the LR is 1e-4, and you want to reduce the BS to 16, try to reduce LR to 5e-5. If you have more time, you can search the optimal LR around 5e-5.\n2. If you use multiple GPU to train, never put only one image per gpus. For example, if you use 8 cards, your minimum batch size should be 16, while there are 2 images per gpus. If you set BS=8, then there is 1 image per gpu, which will cause the model to perform much worse. It's caused by the BatchNorm in most CNN models.", "votes": 10, "replies": [ { "id": 2103572, "postDate": "2023-01-17T08:43:15.090Z", "content": "

Hi @forcewithme Thank you for your kind suggestion! I will have a try.

", "rawMarkdown": "Hi @forcewithme Thank you for your kind suggestion! I will have a try." } ] } ] }, { "id": 2095041, "postDate": "2023-01-11T07:04:35.680Z", "content": "

good information…

", "rawMarkdown": "good information..." }, { "id": 2095019, "postDate": "2023-01-11T06:44:23.623Z", "content": "

Nice work! btw, how do u determine clf_threshold = 0.1?

", "rawMarkdown": "Nice work! btw, how do u determine `clf_threshold = 0.1`?", "replies": [ { "id": 2095347, "postDate": "2023-01-11T09:50:54.737Z", "content": "

In the training step, I just manually set it to 0.1 to take a look (avoid doing search on every epoch).
\nThe final threshold I used is searched on all 4 folds predictions.

", "rawMarkdown": "In the training step, I just manually set it to 0.1 to take a look (avoid doing search on every epoch). \nThe final threshold I used is searched on all 4 folds predictions.", "votes": 1, "replies": [ { "id": 2095377, "postDate": "2023-01-11T10:10:28.667Z", "content": "

Got it, thx!

", "rawMarkdown": "Got it, thx!" } ] } ] }, { "id": 2094563, "postDate": "2023-01-10T20:44:22.340Z", "content": "

Excellent post, very helpful and concise. Thanks for sharing!

", "rawMarkdown": "Excellent post, very helpful and concise. Thanks for sharing!" }, { "id": 2092574, "postDate": "2023-01-09T12:36:00.047Z", "content": "

Great findings, thanks for sharing!
\nHow much VRAM do you have for 1024x1024 input images with EffNetV2S? 👀

", "rawMarkdown": "Great findings, thanks for sharing!\nHow much VRAM do you have for 1024x1024 input images with EffNetV2S? 👀", "replies": [ { "id": 2092592, "postDate": "2023-01-09T12:44:38.877Z", "content": "

Hi @davidlandup , for training, I used batch size = 64 on 4 * 32G gpus.

", "rawMarkdown": "Hi @davidlandup , for training, I used batch size = 64 on 4 * 32G gpus.", "votes": 1, "replies": [ { "id": 2092606, "postDate": "2023-01-09T12:50:00.063Z", "content": "

Thanks for the quick response! Packing some serious compute 😨

", "rawMarkdown": "Thanks for the quick response! Packing some serious compute 😨", "votes": 1 }, { "id": 2092648, "postDate": "2023-01-09T13:24:27.417Z", "content": "

today I tried to train a 2048 * 2048 model but failed. My GPUs are also not enough, but I think it's interesting to find better ways to optimize our solution with limited resources. I think crop foreground is a good way to avoid using large size, maybe you can try it, and I will also have a try and share my findings later.

", "rawMarkdown": "today I tried to train a 2048 * 2048 model but failed. My GPUs are also not enough, but I think it's interesting to find better ways to optimize our solution with limited resources. I think crop foreground is a good way to avoid using large size, maybe you can try it, and I will also have a try and share my findings later." }, { "id": 2093767, "postDate": "2023-01-10T09:58:14.953Z", "content": "

I wanted to create a baseline before cropping. Locally, I had an unreasonably high validation AUC (97%) which is a clear indication of a messed up data pipeline.
\nI'm currently getting ~0.06 pF1 (75% val AUC) locally on small images (224x224).

\n

Not sure if the same pipeline on larger images will translate to a much higher pF1, but I'd like to get it up to ~0.1 before kicking off a 1024x1024 or 2048x2048 run, at which point ROI extraction will likely play a much larger role. I'll probably experiment with cropping in the next couple of days to see if that helps get a higher base pF1…

\n

Posting updates if there are any :)

", "rawMarkdown": "I wanted to create a baseline before cropping. Locally, I had an unreasonably high validation AUC (97%) which is a clear indication of a messed up data pipeline.\nI'm currently getting ~0.06 pF1 (75% val AUC) locally on small images (224x224).\n\nNot sure if the same pipeline on larger images will translate to a much higher pF1, but I'd like to get it up to ~0.1 before kicking off a 1024x1024 or 2048x2048 run, at which point ROI extraction will likely play a much larger role. I'll probably experiment with cropping in the next couple of days to see if that helps get a higher base pF1...\n\nPosting updates if there are any :)", "votes": 2 }, { "id": 2094047, "postDate": "2023-01-10T15:07:42.807Z", "content": "

In my experiments, I found that increasing the image size improved by quite a lot the results. But as you said, I would suggest having a strong pipeline and playing with it. ROI extraction also played for me a huge improvement.

", "rawMarkdown": "In my experiments, I found that increasing the image size improved by quite a lot the results. But as you said, I would suggest having a strong pipeline and playing with it. ROI extraction also played for me a huge improvement.", "votes": 1 } ] } ] }, { "id": 2092484, "postDate": "2023-01-09T11:17:29.120Z", "content": "

What's the inference time on the 4 fold 1024 * 1024 setup?

", "rawMarkdown": "What's the inference time on the 4 fold 1024 * 1024 setup?", "replies": [ { "id": 2092591, "postDate": "2023-01-09T12:43:53.213Z", "content": "

Hi @kiranik , sorry I did not record the time as I submitted before going to bed : ) Is there any easy way to check the inference time?

", "rawMarkdown": "Hi @kiranik , sorry I did not record the time as I submitted before going to bed : ) Is there any easy way to check the inference time?", "votes": 1, "replies": [ { "id": 2092624, "postDate": "2023-01-09T13:03:21.177Z", "content": "

Alright, yes it's not easy, only way is to manually check if the personally schedule allow at the time and get some ~time intervall for a new setup, some also use the train set for time and memory check.

", "rawMarkdown": "Alright, yes it's not easy, only way is to manually check if the personally schedule allow at the time and get some ~time intervall for a new setup, some also use the train set for time and memory check. ", "votes": 1 } ] } ] }, { "id": 2116345, "postDate": "2023-01-26T12:38:28.017Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2115421, "postDate": "2023-01-25T18:50:37.333Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2110815, "postDate": "2023-01-22T12:52:09.127Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2095271, "postDate": "2023-01-11T09:04:37.840Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2093668, "postDate": "2023-01-10T07:53:42.943Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2093522, "postDate": "2023-01-10T04:10:12.667Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2097241, "postDate": "2023-01-12T14:32:09.333Z", "content": "

Thanks for sharing your insights

", "rawMarkdown": "Thanks for sharing your insights\n" }, { "id": 2095106, "postDate": "2023-01-11T07:51:37.493Z", "content": "

thank you!

", "rawMarkdown": "thank you!" } ], "comments": [ { "id": 2092074, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T03:46:48.203000", "content": "

[Update 1] My purpose is to make use of MONAI, as well as other libs (such as timm for network, dali for acceleration) to build a decent model.
\nFirst of all, I wish to thank @theoviel , I followed your topic and your code to design my training pipeline, which can be summarized as the following settings:

\n\n

Based on these settings, I started to do experiments, and my initial purpose is to reach pf1 ~0.083 and auc 0.755 (these two values are from theoviel's open sourced notebook I mentioned above). What I tuned include:

\n
    \n
  1. number of epochs
    2. \n
  2. pos sample weights on bce loss
    3. \n
  3. drop rate of efficientnet (this idea is achieved from @hengck23 in his topic
    \nafter ~ 8 experiments, I got pf1 0.10475 on one fold locally. Then I switched to use larger input size: 1024 * 1024 and do all 4 folds experiments. In this step, I did not tune any parameters, but just try to finish training, and then submit to LB.
    4. \n
", "votes": 13, "replies": [ { "id": 2092075, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T03:49:50.580000", "content": "

My next plan:

\n
    \n
  1. try to train a 2048 * 2048 model, which I expect a ~0.51 LB result as @hengck23 shared : )
    2. \n
  2. adjust the input, since there are many black area in each figure, which may be useless.
    3. \n
", "votes": 6, "replies": [ { "id": 2092470, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-09T11:03:51.207000", "content": "

\"adjust the input, since there are many black area in each figure\"
\nhttps://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region

\n

i think this will take about 10 min to process the all kaggle test images

", "votes": 3, "replies": [] }, { "id": 2095384, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-11T10:14:51.273000", "content": "

Now I give up to train 2048 size model because of limited resources, and I will focus on roi based method as others suggested.
\nTo get a quick experiment, I will try to use the dataset provided by @remekkinas in his topic: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369754 (Thanks Remek!) locally, and if getting any progresses I will enhance my inference notebook and share my results.

", "votes": 1, "replies": [] }, { "id": 2096719, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T08:19:59.647000", "content": "

if you are using crop strategy, do remember to take care of the case if no box can be detected. i spend a few days to debug this submission error.

\n
box = run_box_detection_model(image)\nif box is None:\n      use whole image\nelse:\n       crop image\n
", "votes": 4, "replies": [] }, { "id": 2097348, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-12T15:57:13.713000", "content": "

and … use

\n
try:\n....\nexcept Exception as e \n... rollback operations in try\n
\n

I have one place in code which is unstable - probably in test DS is dicom file which break rules from train. I have spent a lot of time debugging - have not found any problem. Then decided to wrap code in try except and for this image rollback all operations - use oryginal image instead of cropped and postprocessed.

", "votes": 0, "replies": [] }, { "id": 2101521, "author_name": "Phaedrus", "author_url": "", "post_date": "2023-01-16T01:11:18.770000", "content": "

Which part of Heng's code did you wrap? I tried it but sub still fails.

", "votes": 0, "replies": [] } ] }, { "id": 2092868, "author_name": "Balaji Selvaraj", "author_url": "", "post_date": "2023-01-09T17:01:04.390000", "content": "

Hi @yiheng

\n

Thanks for sharing your insights.. Can you mention the basis for your CV split..

\n

I am currently using StratifiedGroupKFold (5 folds) with patient_id for grouping

", "votes": 0, "replies": [ { "id": 2093695, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-10T08:19:05.527000", "content": "

Similarly, I also used StratifiedGroupKFold with k=4 (the same as Theo does in his notebook)

", "votes": 2, "replies": [] } ] }, { "id": 2100708, "author_name": "i_love_huyen_tran", "author_url": "", "post_date": "2023-01-15T11:27:02.840000", "content": "

Hi @yiheng I want to know when ensembling 4 folds, which validation set do you use to choose threshold to binarize ?

", "votes": 0, "replies": [ { "id": 2101783, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-16T06:54:52.707000", "content": "

Hi, the whole dataset. I used each fold's model to do predictions on each validation set, and then combine all predictions together to search the threshold.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2100083, "author_name": "AbanoubSamir004", "author_url": "", "post_date": "2023-01-14T23:32:17.640000", "content": "

Great Job!

", "votes": 1, "replies": [] }, { "id": 2092163, "author_name": "Aadam", "author_url": "", "post_date": "2023-01-09T06:42:59.137000", "content": "

Are you using Kaggle for training, some other cloud provider, or local GPU setup?

", "votes": 1, "replies": [ { "id": 2092179, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T07:17:24.047000", "content": "

Hi @aadimator , I use local GPUs

", "votes": 1, "replies": [] } ] }, { "id": 2104971, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-18T07:17:55.217000", "content": "

[Update 2] I used @hengck23 's trained model (in: https://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region) to infer the bounding boxes for all training images and then do the following preprocessing:
\ndicom ---> crop by bounding box ---> resize to 1024 * 1024
\nthe resized dataset is uploaded to: https://www.kaggle.com/datasets/yiheng/rsna-breast-1024

\n

Now I'm training with this dataset, if having any progresses, I will update the topic and the notebook.
\nThe next plans are:

\n
    \n
  1. train with different backbone
    2. \n
  2. train with different input size
    3. \n
  3. optimize the inference time
    4. \n
", "votes": 0, "replies": [ { "id": 2110936, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-22T14:39:57.640000", "content": "

remember to use try except to capture dengerate case at submissiom
\nhttps://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example/comments#2104692

", "votes": 1, "replies": [] } ] }, { "id": 2102690, "author_name": "nicehzj", "author_url": "", "post_date": "2023-01-16T19:37:33.720000", "content": "

Hi! Thanks for your work!
\nI tried to reproduce and set batch_size to 16 due to my single GPU… and I find that the eval loss is increasing instead of decreasing…
\nShould I decreas the lr for my smaller batch_size?

\n

Thanks!

", "votes": 0, "replies": [ { "id": 2102835, "author_name": "slime", "author_url": "", "post_date": "2023-01-16T21:29:17.630000", "content": "

I tried this as well and failed, the model didn't converge, to make effnetv2 train I had to enable gradient checkpointing and keep batch size at 64, and the easier thing that I didn't test is to lower the learning rate, maybe it'll work comparatively to keeping batch size large

", "votes": 0, "replies": [ { "id": 2102910, "author_name": "nicehzj", "author_url": "", "post_date": "2023-01-16T22:06:47.737000", "content": "

I tried many models and I found effnetv2 is really hard to train… The eval loss often didn't converge…
\nI will try to decrease the lr from 3e-4 to 1e-4, then maybe 5e-5 to have a look.
\nBut as you said, i don't know if this only works for large batch size

", "votes": 0, "replies": [] } ] }, { "id": 2103174, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-17T04:00:31.857000", "content": "

could you try large batch size with smaller image size first? Or maybe you can upsample the positive samples, I guess a small batch size may make the task hard since there may have many batches that have no pos samples.

", "votes": 0, "replies": [ { "id": 2103803, "author_name": "nicehzj", "author_url": "", "post_date": "2023-01-17T11:58:52.540000", "content": "

Thanks! now I'm trying to use upsampling and also gradient accumulation to have a try!

", "votes": 0, "replies": [] }, { "id": 2104315, "author_name": "nicehzj", "author_url": "", "post_date": "2023-01-17T18:40:59.550000", "content": "

I tried with upsampling the pos to 1/8 of each batch and gradient accumulation to get 64 a batch.
\nBut I still get the overfitting problems and cannot even close to your score…

", "votes": 0, "replies": [] }, { "id": 2104979, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-18T07:22:45.210000", "content": "

what is your val score? Have you tried to train on 512 * 512 size? It should get a ~0.1 val score.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2099092, "author_name": "3rd Company 3rd Platoon Leader", "author_url": "", "post_date": "2023-01-14T07:01:16.593000", "content": "

Great Job!
\nSimple & Impressive for your sharing.

\n

Could you get my 3 simple questions?

\n
    \n

  1. As following your training config files, the pos_weight in loss function set to 1.
    \nDid your 0.47 LB score come from it? or other value?
    \nI'm just wondering because the pos_weight in most of other codes set to range of 10~50.

    2. \n

  2. In validation for each folds, how did you choose the best epoch weight?
    \n2 scores(bin pfbeta1(thres 0.1) & auc) can be printed.
    \nwhich scores did you follow?

    3. \n

  3. In your experiment, Was just 10 epoch training enough to get the result?

    4. \n
\n

Thank you!

", "votes": 0, "replies": [ { "id": 2101788, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-16T06:56:48.293000", "content": "
    \n
  1. Yes, I tried other pos weights but the final result is similar. 1 works on my solution.
    2. \n
  2. I select the best pf1 (non binariezed) weights.
    3. \n
  3. I think 10 epoch is enough, I also tried 20, but the total val score is similar.
    4. \n
", "votes": 1, "replies": [ { "id": 2105064, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-18T08:42:03.500000", "content": "

As for 3, I detected a new finding.
\nAccording to my latest experiments on cropped images, it's easier for training set to overfit, and reduce epoch number from 10 to 5 is enough to get similar val score and also lower train score (less overfit).

", "votes": 1, "replies": [] } ] } ] }, { "id": 2097286, "author_name": "David Serrano", "author_url": "", "post_date": "2023-01-12T14:57:56.947000", "content": "

Thank you for sharing your work! As you already said, you use StratifiedGroupKFold with k=4 and binarize f1 to get the final prediction. How do you find the threshold to use in the submissions? As you compute the mean along all the folds, do you compute the mean of all the thresholds?

", "votes": 0, "replies": [ { "id": 2104975, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-18T07:20:29.123000", "content": "

I search the threshold based on all predictions (from all folds models).

", "votes": 0, "replies": [] } ] }, { "id": 2095373, "author_name": "Phaedrus", "author_url": "", "post_date": "2023-01-11T10:08:45.247000", "content": "

Hi @yiheng . Are you also tracking AUC? If so, could you report it here? Thanks.

", "votes": 0, "replies": [ { "id": 2095389, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-11T10:18:57.003000", "content": "

Hi @pheadrus , AUC is printed in my training pipeline however I did not record it in my sheet : (
\nI will add this part in my following experiments, and post my updates after getting a better result.

", "votes": 3, "replies": [] } ] }, { "id": 2095341, "author_name": "Simon Alerdic", "author_url": "", "post_date": "2023-01-11T09:44:33.350000", "content": "

@yiheng
\nDid you make any resampling (under or over)? And how much your fbeta fluctuated during training time on validation set? Did you get 0.10475 in a consistent progress?

", "votes": 0, "replies": [ { "id": 2095390, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-11T10:20:21.073000", "content": "

Hi @simonalerdic , in 512 * 512 experiments, val score increases consistently.
\nSo far I did not try resampling but just change the pos weight in bce loss.

", "votes": 1, "replies": [ { "id": 2096561, "author_name": "Simon Alerdic", "author_url": "", "post_date": "2023-01-12T06:17:58.130000", "content": "

Is it possible to reproduce your training notebook score on kaggle env? Have you tried? In here, model size and image size, batch size gets limited due to resource and adjusting them from yours gave poor result. Could you please confirm?

", "votes": 0, "replies": [] }, { "id": 2105047, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-18T08:27:34.160000", "content": "

Hi @simonalerdic , I'm afraid I cannot do this kind of works recently since we have limited time for this challenge. I prefer finishing my current experiments first:

\n
    \n
  1. try training with cropped images
    2. \n
  2. try different networks
    3. \n
  3. try different input size
    4. \n
\n

If having 512*512 results, I will post my log here if it is helpful.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2095096, "author_name": "Juntao Jiang", "author_url": "", "post_date": "2023-01-11T07:43:59.407000", "content": "

Is the batch size an important parameter? I reduced batch size to avoid out of memory, but the validation score will be very low.

", "votes": 0, "replies": [ { "id": 2095376, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-11T10:10:02.600000", "content": "

I think it has some impacts, since the percentage of pos samples is low. Maybe you can try to accumulate gradients if not having enough gpu memory.

", "votes": 3, "replies": [ { "id": 2103574, "author_name": "Juntao Jiang", "author_url": "", "post_date": "2023-01-17T08:43:27.883000", "content": "

Thank you!

", "votes": 0, "replies": [] } ] }, { "id": 2095496, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-01-11T12:02:35.327000", "content": "

Hi @juntaojiang , batch size is really trick, while the relationship between BS and LR is also very important.

\n
    \n
  1. When you reduce BS, try to scale LR at the same time. For example, the original BS is 32 and the LR is 1e-4, and you want to reduce the BS to 16, try to reduce LR to 5e-5. If you have more time, you can search the optimal LR around 5e-5.
    2. \n
  2. If you use multiple GPU to train, never put only one image per gpus. For example, if you use 8 cards, your minimum batch size should be 16, while there are 2 images per gpus. If you set BS=8, then there is 1 image per gpu, which will cause the model to perform much worse. It's caused by the BatchNorm in most CNN models.
    3. \n
", "votes": 10, "replies": [ { "id": 2103572, "author_name": "Juntao Jiang", "author_url": "", "post_date": "2023-01-17T08:43:15.090000", "content": "

Hi @forcewithme Thank you for your kind suggestion! I will have a try.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2095041, "author_name": "Vaidehi Savaliya", "author_url": "", "post_date": "2023-01-11T07:04:35.680000", "content": "

good information…

", "votes": 0, "replies": [] }, { "id": 2095019, "author_name": "阳光开朗大男孩", "author_url": "", "post_date": "2023-01-11T06:44:23.623000", "content": "

Nice work! btw, how do u determine clf_threshold = 0.1?

", "votes": 0, "replies": [ { "id": 2095347, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-11T09:50:54.737000", "content": "

In the training step, I just manually set it to 0.1 to take a look (avoid doing search on every epoch).
\nThe final threshold I used is searched on all 4 folds predictions.

", "votes": 1, "replies": [ { "id": 2095377, "author_name": "阳光开朗大男孩", "author_url": "", "post_date": "2023-01-11T10:10:28.667000", "content": "

Got it, thx!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2094563, "author_name": "Gustavo Ruiz", "author_url": "", "post_date": "2023-01-10T20:44:22.340000", "content": "

Excellent post, very helpful and concise. Thanks for sharing!

", "votes": 0, "replies": [] }, { "id": 2092574, "author_name": "David Landup", "author_url": "", "post_date": "2023-01-09T12:36:00.047000", "content": "

Great findings, thanks for sharing!
\nHow much VRAM do you have for 1024x1024 input images with EffNetV2S? 👀

", "votes": 0, "replies": [ { "id": 2092592, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T12:44:38.877000", "content": "

Hi @davidlandup , for training, I used batch size = 64 on 4 * 32G gpus.

", "votes": 1, "replies": [ { "id": 2092606, "author_name": "David Landup", "author_url": "", "post_date": "2023-01-09T12:50:00.063000", "content": "

Thanks for the quick response! Packing some serious compute 😨

", "votes": 1, "replies": [] }, { "id": 2092648, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T13:24:27.417000", "content": "

today I tried to train a 2048 * 2048 model but failed. My GPUs are also not enough, but I think it's interesting to find better ways to optimize our solution with limited resources. I think crop foreground is a good way to avoid using large size, maybe you can try it, and I will also have a try and share my findings later.

", "votes": 0, "replies": [] }, { "id": 2093767, "author_name": "David Landup", "author_url": "", "post_date": "2023-01-10T09:58:14.953000", "content": "

I wanted to create a baseline before cropping. Locally, I had an unreasonably high validation AUC (97%) which is a clear indication of a messed up data pipeline.
\nI'm currently getting ~0.06 pF1 (75% val AUC) locally on small images (224x224).

\n

Not sure if the same pipeline on larger images will translate to a much higher pF1, but I'd like to get it up to ~0.1 before kicking off a 1024x1024 or 2048x2048 run, at which point ROI extraction will likely play a much larger role. I'll probably experiment with cropping in the next couple of days to see if that helps get a higher base pF1…

\n

Posting updates if there are any :)

", "votes": 2, "replies": [] }, { "id": 2094047, "author_name": "David Serrano", "author_url": "", "post_date": "2023-01-10T15:07:42.807000", "content": "

In my experiments, I found that increasing the image size improved by quite a lot the results. But as you said, I would suggest having a strong pipeline and playing with it. ROI extraction also played for me a huge improvement.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2092484, "author_name": "Kirderf", "author_url": "", "post_date": "2023-01-09T11:17:29.120000", "content": "

What's the inference time on the 4 fold 1024 * 1024 setup?

", "votes": 0, "replies": [ { "id": 2092591, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2023-01-09T12:43:53.213000", "content": "

Hi @kiranik , sorry I did not record the time as I submitted before going to bed : ) Is there any easy way to check the inference time?

", "votes": 1, "replies": [ { "id": 2092624, "author_name": "Kirderf", "author_url": "", "post_date": "2023-01-09T13:03:21.177000", "content": "

Alright, yes it's not easy, only way is to manually check if the personally schedule allow at the time and get some ~time intervall for a new setup, some also use the train set for time and memory check.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2116345, "author_name": "", "author_url": "", "post_date": "2023-01-26T12:38:28.017000", "content": "", "votes": 0, "replies": [] }, { "id": 2115421, "author_name": "", "author_url": "", "post_date": "2023-01-25T18:50:37.333000", "content": "", "votes": 0, "replies": [] }, { "id": 2110815, "author_name": "", "author_url": "", "post_date": "2023-01-22T12:52:09.127000", "content": "", "votes": 0, "replies": [] }, { "id": 2095271, "author_name": "", "author_url": "", "post_date": "2023-01-11T09:04:37.840000", "content": "", "votes": 0, "replies": [] }, { "id": 2093668, "author_name": "", "author_url": "", "post_date": "2023-01-10T07:53:42.943000", "content": "", "votes": 0, "replies": [] }, { "id": 2093522, "author_name": "", "author_url": "", "post_date": "2023-01-10T04:10:12.667000", "content": "", "votes": 0, "replies": [] }, { "id": 2097241, "author_name": "LIUZIWEI0827", "author_url": "", "post_date": "2023-01-12T14:32:09.333000", "content": "

Thanks for sharing your insights

", "votes": 0, "replies": [] }, { "id": 2095106, "author_name": "yang zhao", "author_url": "", "post_date": "2023-01-11T07:51:37.493000", "content": "

thank you!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2092068": "In this topic, I will share my experiences in this challenge, and open source all of my works on how to get a 0.47 (the number may change) model.\n\nInference code link:\nhttps://www.kaggle.com/code/yiheng/monai-baseline\nTraining code link:\nhttps://www.kaggle.com/code/yiheng/monai-pipeline-training/notebook\n\nyou can also clone the training pipeline from github:\n```\ngit clone git@github.com:yiheng-wang-nv/rsna_breast_monai_solution.git\n```\n\nThe following is my local and lb results (will update if having any progresses):\n\n| | pf1 local | bin pf1 local | LB |\n| --- | --- | --- | --- |\n| fold 0, agg by max | 0.2753 | 0.3352 | 0.38 |\n| fold 0, agg by mean | 0.2042 | 0.3392 | 0.42 |\n| fold 0, agg by mean, 2 tta | 0.2045 | 0.3879 | 0.44 |\n| 4 folds, agg by mean, 2 tta | 0.2234 | 0.3129 | 0.47 |", "2092074": "[Update 1] My purpose is to make use of MONAI, as well as other libs (such as timm for network, dali for acceleration) to build a decent model.\nFirst of all, I wish to thank @theoviel , I followed [your topic](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369769) and [your code](https://www.kaggle.com/code/theoviel/rsna-breast-baseline-inference) to design my training pipeline, which can be summarized as the following settings:\n\n- group stratified 4 fold data split\n- 512 * 512 input size\n- h flip augmentation\n- 4 epoch, lr decay from 3e-4\n- timm, efnv2s network\n\nBased on these settings, I started to do experiments, and my initial purpose is to reach pf1 ~0.083 and auc 0.755 (these two values are from theoviel's open sourced notebook I mentioned above). What I tuned include:\n\n1. number of epochs\n2. pos sample weights on bce loss\n3. drop rate of efficientnet (this idea is achieved from @hengck23 in [his topic](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333)\nafter ~ 8 experiments, I got pf1 0.10475 on one fold locally. Then I switched to use larger input size: 1024 * 1024 and do all 4 folds experiments. In this step, I did not tune any parameters, but just try to finish training, and then submit to LB.", "2100083": "Great Job!", "2092163": "Are you using Kaggle for training, some other cloud provider, or local GPU setup?", "2104971": "[Update 2] I used @hengck23 's trained model (in: https://www.kaggle.com/code/hengck23/proprocess-function-e-g-crop-breast-region) to infer the bounding boxes for all training images and then do the following preprocessing:\ndicom ---> crop by bounding box ---> resize to 1024 * 1024\nthe resized dataset is uploaded to: https://www.kaggle.com/datasets/yiheng/rsna-breast-1024\n\nNow I'm training with this dataset, if having any progresses, I will update the topic and the notebook.\nThe next plans are:\n1. train with different backbone\n2. train with different input size\n3. optimize the inference time", "2102690": "Hi! Thanks for your work!\nI tried to reproduce and set batch_size to 16 due to my single GPU... and I find that the eval loss is increasing instead of decreasing...\nShould I decreas the lr for my smaller batch_size?\n\nThanks!", "2099092": "Great Job!\nSimple & Impressive for your sharing.\n\nCould you get my 3 simple questions?\n1. As following your training config files, the pos_weight in loss function set to 1.\nDid your 0.47 LB score come from it? or other value?\nI'm just wondering because the pos_weight in most of other codes set to range of 10~50.\n\n2. In validation for each folds, how did you choose the best epoch weight?\n2 scores(bin pfbeta1(thres 0.1) & auc) can be printed.\nwhich scores did you follow?\n\n3. In your experiment, Was just 10 epoch training enough to get the result?\n\nThank you!", "2097286": "Thank you for sharing your work! As you already said, you use StratifiedGroupKFold with k=4 and binarize f1 to get the final prediction. How do you find the threshold to use in the submissions? As you compute the mean along all the folds, do you compute the mean of all the thresholds?", "2095373": "Hi @yiheng . Are you also tracking AUC? If so, could you report it here? Thanks. ", "2095341": "@yiheng \nDid you make any resampling (under or over)? And how much your fbeta fluctuated during training time on validation set? Did you get 0.10475 in a consistent progress? ", "2095096": "Is the batch size an important parameter? I reduced batch size to avoid out of memory, but the validation score will be very low.", "2095041": "good information...", "2095019": "Nice work! btw, how do u determine `clf_threshold = 0.1`?", "2094563": "Excellent post, very helpful and concise. Thanks for sharing!", "2092574": "Great findings, thanks for sharing!\nHow much VRAM do you have for 1024x1024 input images with EffNetV2S? 👀", "2092484": "What's the inference time on the 4 fold 1024 * 1024 setup?", "2116345": "", "2115421": "", "2110815": "", "2095271": "", "2093668": "", "2093522": "", "2097241": "Thanks for sharing your insights\n", "2095106": "thank you!" } }