If you are using WeightedRandomSampler in this game, you may have some interest in this.
A simple idea, WeightedRandomSampler is randomly sampling negative samples repeatedly, it means that some samples may have been sampled many times but some others may not be sampled. So I write this simple sampler to address this problem: https://github.com/louis-she/exhaustive-weighted-random-sampler, the goal here is not to waste any samples even if there is a lot of them!
A demo notebook can be found here: https://www.kaggle.com/code/snaker/exhaustiveweightedrandomsampler/notebook
", "messages": [ { "id": 2088617, "postDate": "2023-01-06T13:45:14.190Z", "content": "If you are using WeightedRandomSampler in this game, you may have some interest in this.
A simple idea, WeightedRandomSampler is randomly sampling negative samples repeatedly, it means that some samples may have been sampled many times but some others may not be sampled. So I write this simple sampler to address this problem: https://github.com/louis-she/exhaustive-weighted-random-sampler, the goal here is not to waste any samples even if there is a lot of them!
A demo notebook can be found here: https://www.kaggle.com/code/snaker/exhaustiveweightedrandomsampler/notebook
", "rawMarkdown": "If you are using `WeightedRandomSampler` in this game, you may have some interest in this.\n\nA simple idea, `WeightedRandomSampler` is randomly sampling negative samples repeatedly, it means that some samples may have been sampled many times but some others may not be sampled. So I write this simple sampler to address this problem: https://github.com/louis-she/exhaustive-weighted-random-sampler, the goal here is not to waste any samples even if there is a lot of them!\n\nA demo notebook can be found here: https://www.kaggle.com/code/snaker/exhaustiveweightedrandomsampler/notebook", "votes": 24 }, { "id": 2114022, "postDate": "2023-01-24T16:54:34.687Z", "content": "I've always taken the approach of:
\nThis means every epoch, every negative case gets used, and which positive cases get oversampled changes each epoch.
\nI can't see why using this exhaustive weighted random sampler is any better than this, but would be keen to hear what you think @snaker?
", "rawMarkdown": "I've always taken the approach of:\n- ...at the start of every epoch\n- ...oversample all the positive cases using the epoch number as the random seed and make that your dataset for the epoch\n- ...iterate over all that dataset in the standard pytorch way\n\nThis means every epoch, every negative case gets used, and which positive cases get oversampled changes each epoch.\n\nI can't see why using this exhaustive weighted random sampler is any better than this, but would be keen to hear what you think @snaker? ", "replies": [ { "id": 2114493, "postDate": "2023-01-25T03:08:23.687Z", "content": "It depends on how to define one \"epoch\". If the dataset is large and is very imbalanced ( like this one ), I prefer to use WeightedRandomSampler with num_samples much smaller than the whole dataset, usually it's number of positive samples * N.
Ok, that makes sense, but it's effectively the same strategy but just a different way of setting training time?
\nAlso, I may be wrong, but I think your method doesn't ensure a uniform oversampling of the minority class, because replace=True?
I think my approach from above might have some slight advantages, but not 100% sure.
", "rawMarkdown": "Ok, that makes sense, but it's effectively the same strategy but just a different way of setting training time?\nAlso, I may be wrong, but I think your method doesn't ensure a uniform oversampling of the minority class, because `replace=True`?\n\nI think my approach from above might have some slight advantages, but not 100% sure." }, { "id": 2115836, "postDate": "2023-01-26T01:40:55.700Z", "content": "\"uniform oversampling of the minority class\" - that's true, WeightedRandomSampler can not ensure equal sample every minority class, maybe some class sample 5 time while others sample 2 or 3 times.
Thanks for sharing. But when I use ExhaustiveWeightedRandomSampler, my CV improved and my LB didn't improve. It confused me a lot
", "rawMarkdown": "Thanks for sharing. But when I use ExhaustiveWeightedRandomSampler, my CV improved and my LB didn't improve. It confused me a lot", "replies": [ { "id": 2108950, "postDate": "2023-01-21T01:08:17.900Z", "content": "LB is not stable… My CV boosted from 0.43 to 0.51, but LB is still the same…
", "rawMarkdown": "LB is not stable... My CV boosted from 0.43 to 0.51, but LB is still the same...", "votes": 1, "replies": [ { "id": 2108984, "postDate": "2023-01-21T02:48:22.570Z", "content": "So shall we trust cv or lb? haha
", "rawMarkdown": "So shall we trust cv or lb? haha" }, { "id": 2109004, "postDate": "2023-01-21T03:37:39.307Z", "content": "In my opinion, both, but give them a weight
", "rawMarkdown": "In my opinion, both, but give them a weight" }, { "id": 2109076, "postDate": "2023-01-21T05:24:17.463Z", "content": "use this playground code to see what really happens.
\nsee how oversampling changes the decision space
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/372628
", "rawMarkdown": "use this playground code to see what really happens.\nsee how oversampling changes the decision space\n\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/372628\n", "votes": 2 }, { "id": 2109085, "postDate": "2023-01-21T05:33:41.850Z", "content": "why choose the best?
\nyou can choose the best k-models from different weights in loss, etc
[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
\nuse larger learning rate to learn a coarse model.
\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)
ensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.
", "rawMarkdown": "why choose the best?\nyou can choose the best k-models from different weights in loss, etc\n\n[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time\n\nuse larger learning rate to learn a coarse model.\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)\n\nensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.\n" }, { "id": 2109268, "postDate": "2023-01-21T08:37:27.767Z", "content": "Really cool visualizations, thanks heng
", "rawMarkdown": "Really cool visualizations, thanks heng" }, { "id": 2114855, "postDate": "2023-01-25T09:48:55.643Z", "content": "Yes, but surely that is only a one fold boost, not the entire CV.. 😲
", "rawMarkdown": "Yes, but surely that is only a one fold boost, not the entire CV.. 😲" } ] } ] }, { "id": 2090365, "postDate": "2023-01-07T08:52:44.363Z", "content": "thanks for sharing. down sample or up sample, should improve the LB.
", "rawMarkdown": "thanks for sharing. down sample or up sample, should improve the LB.", "replies": [ { "id": 2090449, "postDate": "2023-01-07T10:46:02.627Z", "content": "When you find way to recognize (or show to model) cancer it will definitely help. For me challange is to find such way. Some images are classified very very well … some of them … no.
\nWhen I see cancer … model see it as well but when I don't (even label is cancer) model has the same problem. We are both poor in cancer recognition. 😂
that is true. It is impossible for no specialist to deal with abnormal data or wrong labelled data.
", "rawMarkdown": "that is true. It is impossible for no specialist to deal with abnormal data or wrong labelled data." } ] } ] } ], "comments": [ { "id": 2114022, "author_name": "James Howard", "author_url": "", "post_date": "2023-01-24T16:54:34.687000", "content": "I've always taken the approach of:
\nThis means every epoch, every negative case gets used, and which positive cases get oversampled changes each epoch.
\nI can't see why using this exhaustive weighted random sampler is any better than this, but would be keen to hear what you think @snaker?
", "votes": 0, "replies": [ { "id": 2114493, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-25T03:08:23.687000", "content": "It depends on how to define one \"epoch\". If the dataset is large and is very imbalanced ( like this one ), I prefer to use WeightedRandomSampler with num_samples much smaller than the whole dataset, usually it's number of positive samples * N.
Ok, that makes sense, but it's effectively the same strategy but just a different way of setting training time?
\nAlso, I may be wrong, but I think your method doesn't ensure a uniform oversampling of the minority class, because replace=True?
I think my approach from above might have some slight advantages, but not 100% sure.
", "votes": 0, "replies": [] }, { "id": 2115836, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-26T01:40:55.700000", "content": "\"uniform oversampling of the minority class\" - that's true, WeightedRandomSampler can not ensure equal sample every minority class, maybe some class sample 5 time while others sample 2 or 3 times.
Thanks for sharing. But when I use ExhaustiveWeightedRandomSampler, my CV improved and my LB didn't improve. It confused me a lot
", "votes": 0, "replies": [ { "id": 2108950, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-21T01:08:17.900000", "content": "LB is not stable… My CV boosted from 0.43 to 0.51, but LB is still the same…
", "votes": 1, "replies": [ { "id": 2108984, "author_name": "Leimeng46", "author_url": "", "post_date": "2023-01-21T02:48:22.570000", "content": "So shall we trust cv or lb? haha
", "votes": 0, "replies": [] }, { "id": 2109004, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-21T03:37:39.307000", "content": "In my opinion, both, but give them a weight
", "votes": 0, "replies": [] }, { "id": 2109076, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-21T05:24:17.463000", "content": "use this playground code to see what really happens.
\nsee how oversampling changes the decision space
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/372628
", "votes": 2, "replies": [] }, { "id": 2109085, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-21T05:33:41.850000", "content": "why choose the best?
\nyou can choose the best k-models from different weights in loss, etc
[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
\nuse larger learning rate to learn a coarse model.
\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)
ensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.
", "votes": 0, "replies": [] }, { "id": 2109268, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-21T08:37:27.767000", "content": "Really cool visualizations, thanks heng
", "votes": 0, "replies": [] }, { "id": 2114855, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-01-25T09:48:55.643000", "content": "Yes, but surely that is only a one fold boost, not the entire CV.. 😲
", "votes": 0, "replies": [] } ] } ] }, { "id": 2090365, "author_name": "dragon zhang", "author_url": "", "post_date": "2023-01-07T08:52:44.363000", "content": "thanks for sharing. down sample or up sample, should improve the LB.
", "votes": 0, "replies": [ { "id": 2090449, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-07T10:46:02.627000", "content": "When you find way to recognize (or show to model) cancer it will definitely help. For me challange is to find such way. Some images are classified very very well … some of them … no.
\nWhen I see cancer … model see it as well but when I don't (even label is cancer) model has the same problem. We are both poor in cancer recognition. 😂
that is true. It is impossible for no specialist to deal with abnormal data or wrong labelled data.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2088617": "If you are using `WeightedRandomSampler` in this game, you may have some interest in this.\n\nA simple idea, `WeightedRandomSampler` is randomly sampling negative samples repeatedly, it means that some samples may have been sampled many times but some others may not be sampled. So I write this simple sampler to address this problem: https://github.com/louis-she/exhaustive-weighted-random-sampler, the goal here is not to waste any samples even if there is a lot of them!\n\nA demo notebook can be found here: https://www.kaggle.com/code/snaker/exhaustiveweightedrandomsampler/notebook", "2114022": "I've always taken the approach of:\n- ...at the start of every epoch\n- ...oversample all the positive cases using the epoch number as the random seed and make that your dataset for the epoch\n- ...iterate over all that dataset in the standard pytorch way\n\nThis means every epoch, every negative case gets used, and which positive cases get oversampled changes each epoch.\n\nI can't see why using this exhaustive weighted random sampler is any better than this, but would be keen to hear what you think @snaker? ", "2108919": "Thanks for sharing. But when I use ExhaustiveWeightedRandomSampler, my CV improved and my LB didn't improve. It confused me a lot", "2090365": "thanks for sharing. down sample or up sample, should improve the LB." } }