{ "id": 357695, "title": "🔥 Best CV-LB results 🔥", "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/357695", "author_name": "Mark Baushenko", "post_date": "2022-10-05T09:39:03.657000", "votes": 17, "comment_count": 20, "views": 0, "content": "

Let's share CV-LB pairs in this thread.

", "messages": [ { "id": 1972704, "postDate": "2022-10-05T09:39:03.657Z", "content": "

Let's share CV-LB pairs in this thread.

", "rawMarkdown": "Let's share CV-LB pairs in this thread.", "votes": 17 }, { "id": 2007848, "postDate": "2022-10-28T14:41:54.553Z", "content": "
CV - 0.751 (5 FOLD) (only comp data)\nLB - 0.706\n
", "rawMarkdown": "\n```\nCV - 0.751 (5 FOLD) (only comp data)\nLB - 0.706\n```", "votes": 5, "replies": [ { "id": 2009150, "postDate": "2022-10-29T17:32:17.143Z", "content": "

Wow, @drhabib . Your CV-LB relationship looks much better than mine. My cv is always 0.8+, but lb(5-fold) still around 0.7.

", "rawMarkdown": "Wow, @drhabib . Your CV-LB relationship looks much better than mine. My cv is always 0.8+, but lb(5-fold) still around 0.7. ", "votes": 1 }, { "id": 2009301, "postDate": "2022-10-29T21:19:16.033Z", "content": "

Are you doing Augs ? Without augs the gap is bigger for me …

", "rawMarkdown": "Are you doing Augs ? Without `augs` the gap is bigger for me ...", "votes": 3 }, { "id": 2010561, "postDate": "2022-10-30T21:14:16.727Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2013684, "postDate": "2022-11-02T04:37:48.923Z", "content": "

After running some experiments .. I think the biggest challenge of this competition will be to build a proper validation set …

", "rawMarkdown": "After running some experiments .. I think the biggest challenge of this competition will be to build a proper validation set ...", "votes": 2 }, { "id": 2017677, "postDate": "2022-11-05T04:57:08.037Z", "content": "

@drhabib I totally agree with you.🤝

", "rawMarkdown": "@drhabib I totally agree with you.🤝", "votes": 3 } ] }, { "id": 2009588, "postDate": "2022-10-30T06:36:29.147Z", "content": "

CV - 0.789 (5 fold)
\nLB - 0.690

\n

--- updated 11/1

\n

CV - 0.829 (5 fold)
\nLB - 0.714

\n

for now, there's usually 0.1 gap between CV & LB on my experiments.

", "rawMarkdown": "CV - 0.789 (5 fold)\nLB - 0.690\n\n--- updated 11/1\n\nCV - 0.829 (5 fold)\nLB - 0.714\n\nfor now, there's usually 0.1 gap between CV & LB on my experiments.", "votes": 4, "replies": [ { "id": 2019912, "postDate": "2022-11-07T04:27:28.923Z", "content": "

My results are similar to yours, but cv and lb don't always correlate…..

", "rawMarkdown": "My results are similar to yours, but cv and lb don't always correlate.....", "votes": 1 }, { "id": 2020318, "postDate": "2022-11-07T10:56:38.963Z", "content": "

mine also starts to negatively correlate too :(

\n\n

cv 0.829 lb 0.714
\ncv 0.845 lb 0.706
\ncv 0.852 lb 0.701

\n\n

cv 0.862 lb 0.721
\ncv 0.868 lb 0.715

\n

I guess It's not a meaningful difference (0.00x) in the aspect of CV & LB, but need to build a robust validation set and take more time to probe the test set.

", "rawMarkdown": "mine also starts to negatively correlate too :(\n\n* single model\n\ncv 0.829 lb 0.714\ncv 0.845 lb 0.706\ncv 0.852 lb 0.701\n\n* ensemble\n\ncv 0.862 lb 0.721\ncv 0.868 lb 0.715\n\nI guess It's not a meaningful difference (0.00x) in the aspect of CV & LB, but need to build a robust validation set and take more time to probe the test set.", "votes": 2 }, { "id": 2020366, "postDate": "2022-11-07T12:01:40.897Z", "content": "

@hanzhou0315 , @kozistr may I ask if your reported CV scores are from evaluating the kaggle train set?

\n

Asking because otherwise it's not possible to do comparisons. Score obtained on a generated dataset will depend not only on model performance but also on the distribution of depths chosen.

\n

I don't have the exact numbers right now but I can only get AUC ~ 0.77-0.78 for kaggle train set (without using it for train at all), which translates into ~ 0.68 LB in the best case.
\nFor my CV value itself, it depends completely on my own generated data, so it's not very meaningful for other people. But it's high (>0.90) if I choose to include samples with lower depths.

", "rawMarkdown": "@hanzhou0315 , @kozistr may I ask if your reported CV scores are from evaluating the kaggle train set?\n\nAsking because otherwise it's not possible to do comparisons. Score obtained on a generated dataset will depend not only on model performance but also on the distribution of depths chosen.\n\nI don't have the exact numbers right now but I can only get AUC ~ 0.77-0.78 for kaggle train set (without using it for train at all), which translates into ~ 0.68 LB in the best case.\nFor my CV value itself, it depends completely on my own generated data, so it's not very meaningful for other people. But it's high (>0.90) if I choose to include samples with lower depths.", "votes": 1 }, { "id": 2020372, "postDate": "2022-11-07T12:14:31.473Z", "content": "

@kozistr I'm in the same situation as you.
\ncv: 0.838 lb:0.712
\ncv:0. 842 lb: 0.689
\nI agree with you，a robust validation set is very important.

", "rawMarkdown": "@kozistr I'm in the same situation as you.\ncv: 0.838 lb:0.712\ncv:0. 842 lb: 0.689\nI agree with you，a robust validation set is very important.", "votes": 2 }, { "id": 2020373, "postDate": "2022-11-07T12:15:38.487Z", "content": "

@darkbitur yes， it's on the kaggle train set.

", "rawMarkdown": "@darkbitur yes， it's on the kaggle train set." } ] }, { "id": 2032928, "postDate": "2022-11-16T22:17:57.983Z", "content": "

CV 0.74
\nLB: 0.72

", "rawMarkdown": "CV 0.74\nLB: 0.72", "votes": 1 }, { "id": 2020414, "postDate": "2022-11-07T13:06:33.187Z", "content": "

CV: 0.7806 (on given train set)
\nLB: 0.660

\n

I'm using generated data. Also, I didn't start using CV until pretty late and so far I've been unable to reach my max LB of 0.677 which was without CV.

", "rawMarkdown": "CV: 0.7806 (on given train set)\nLB: 0.660\n\nI'm using generated data. Also, I didn't start using CV until pretty late and so far I've been unable to reach my max LB of 0.677 which was without CV.", "votes": 1 }, { "id": 2014872, "postDate": "2022-11-02T23:37:47.337Z", "content": "

From same experiment (tf_efficientnet_b5_ns, 15 epochs, stratifiedkfold 5 folds):

\n\n

Both the loss value and the auc of the validation vary greatly from epoch to epoch, but it doesn't seem to be a meaningful variation.

", "rawMarkdown": "From same experiment (tf_efficientnet_b5_ns, 15 epochs, stratifiedkfold 5 folds):\n\n- best loss model\nCV: 0.7633 -> LB: 0.642\n\n- best score model\nCV: 0.7803 -> LB: 0.644\n\n- last epoch model\nCV: 0.7481 -> LB: 0.647\n\nBoth the loss value and the auc of the validation vary greatly from epoch to epoch, but it doesn't seem to be a meaningful variation.", "votes": 1 }, { "id": 2005384, "postDate": "2022-10-27T00:49:16.120Z", "content": "

CV 0.67825 (5 stratified folds) - LB 0.558
\nI was also able to get CV 0.71 with the same folds (by changing the random seed) and I got a slightly worse LB score

", "rawMarkdown": "CV 0.67825 (5 stratified folds) - LB 0.558\nI was also able to get CV 0.71 with the same folds (by changing the random seed) and I got a slightly worse LB score", "votes": 1 }, { "id": 2003856, "postDate": "2022-10-25T20:37:11.603Z", "content": "

train_acc=0.75 CV=0.408 LB=0.531. I think there is something wrong with my code.😂😂

", "rawMarkdown": "train_acc=0.75 CV=0.408 LB=0.531. I think there is something wrong with my code.😂😂", "votes": 1 }, { "id": 2019877, "postDate": "2022-11-07T03:53:46.650Z", "content": "

single model
\nCV: 0.82x
\nLB: 0.712

\n
\n

It looks like the gap between cv and lb is large.
\nand cv and lb are not always positively correlated……

", "rawMarkdown": "single model\nCV: 0.82x \nLB: 0.712\n\n--------------------------------------------------\nIt looks like the gap between cv and lb is large.\nand cv and lb are not always positively correlated......", "votes": 2 }, { "id": 2072234, "postDate": "2022-12-21T22:02:02.050Z", "content": "

CV0.8 +vs LB:0.700
\nBut is not relevant since we have our own datasets for training :) i have one dataset with high SNR :CV is 0.9 and LB 0.5-0.6

", "rawMarkdown": "CV0.8 +vs LB:0.700\nBut is not relevant since we have our own datasets for training :) i have one dataset with high SNR :CV is 0.9 and LB 0.5-0.6" }, { "id": 2007857, "postDate": "2022-10-28T14:48:47.293Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2007848, "author_name": "DrHB", "author_url": "", "post_date": "2022-10-28T14:41:54.553000", "content": "
CV - 0.751 (5 FOLD) (only comp data)\nLB - 0.706\n
", "votes": 5, "replies": [ { "id": 2009150, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-10-29T17:32:17.143000", "content": "

Wow, @drhabib . Your CV-LB relationship looks much better than mine. My cv is always 0.8+, but lb(5-fold) still around 0.7.

", "votes": 1, "replies": [] }, { "id": 2009301, "author_name": "DrHB", "author_url": "", "post_date": "2022-10-29T21:19:16.033000", "content": "

Are you doing Augs ? Without augs the gap is bigger for me …

", "votes": 3, "replies": [] }, { "id": 2010561, "author_name": "", "author_url": "", "post_date": "2022-10-30T21:14:16.727000", "content": "", "votes": 0, "replies": [] }, { "id": 2013684, "author_name": "DrHB", "author_url": "", "post_date": "2022-11-02T04:37:48.923000", "content": "

After running some experiments .. I think the biggest challenge of this competition will be to build a proper validation set …

", "votes": 2, "replies": [] }, { "id": 2017677, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-11-05T04:57:08.037000", "content": "

@drhabib I totally agree with you.🤝

", "votes": 3, "replies": [] } ] }, { "id": 2009588, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2022-10-30T06:36:29.147000", "content": "

CV - 0.789 (5 fold)
\nLB - 0.690

\n

--- updated 11/1

\n

CV - 0.829 (5 fold)
\nLB - 0.714

\n

for now, there's usually 0.1 gap between CV & LB on my experiments.

", "votes": 4, "replies": [ { "id": 2019912, "author_name": "zh", "author_url": "", "post_date": "2022-11-07T04:27:28.923000", "content": "

My results are similar to yours, but cv and lb don't always correlate…..

", "votes": 1, "replies": [] }, { "id": 2020318, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2022-11-07T10:56:38.963000", "content": "

mine also starts to negatively correlate too :(

\n\n

cv 0.829 lb 0.714
\ncv 0.845 lb 0.706
\ncv 0.852 lb 0.701

\n\n

cv 0.862 lb 0.721
\ncv 0.868 lb 0.715

\n

I guess It's not a meaningful difference (0.00x) in the aspect of CV & LB, but need to build a robust validation set and take more time to probe the test set.

", "votes": 2, "replies": [] }, { "id": 2020366, "author_name": "Victor Gonzalez", "author_url": "", "post_date": "2022-11-07T12:01:40.897000", "content": "

@hanzhou0315 , @kozistr may I ask if your reported CV scores are from evaluating the kaggle train set?

\n

Asking because otherwise it's not possible to do comparisons. Score obtained on a generated dataset will depend not only on model performance but also on the distribution of depths chosen.

\n

I don't have the exact numbers right now but I can only get AUC ~ 0.77-0.78 for kaggle train set (without using it for train at all), which translates into ~ 0.68 LB in the best case.
\nFor my CV value itself, it depends completely on my own generated data, so it's not very meaningful for other people. But it's high (>0.90) if I choose to include samples with lower depths.

", "votes": 1, "replies": [] }, { "id": 2020372, "author_name": "zh", "author_url": "", "post_date": "2022-11-07T12:14:31.473000", "content": "

@kozistr I'm in the same situation as you.
\ncv: 0.838 lb:0.712
\ncv:0. 842 lb: 0.689
\nI agree with you，a robust validation set is very important.

", "votes": 2, "replies": [] }, { "id": 2020373, "author_name": "zh", "author_url": "", "post_date": "2022-11-07T12:15:38.487000", "content": "

@darkbitur yes， it's on the kaggle train set.

", "votes": 0, "replies": [] } ] }, { "id": 2032928, "author_name": "arutema47", "author_url": "", "post_date": "2022-11-16T22:17:57.983000", "content": "

CV 0.74
\nLB: 0.72

", "votes": 1, "replies": [] }, { "id": 2020414, "author_name": "Will Rice", "author_url": "", "post_date": "2022-11-07T13:06:33.187000", "content": "

CV: 0.7806 (on given train set)
\nLB: 0.660

\n

I'm using generated data. Also, I didn't start using CV until pretty late and so far I've been unable to reach my max LB of 0.677 which was without CV.

", "votes": 1, "replies": [] }, { "id": 2014872, "author_name": "YYama", "author_url": "", "post_date": "2022-11-02T23:37:47.337000", "content": "

From same experiment (tf_efficientnet_b5_ns, 15 epochs, stratifiedkfold 5 folds):

\n\n

Both the loss value and the auc of the validation vary greatly from epoch to epoch, but it doesn't seem to be a meaningful variation.

", "votes": 1, "replies": [] }, { "id": 2005384, "author_name": "Firas Baba", "author_url": "", "post_date": "2022-10-27T00:49:16.120000", "content": "

CV 0.67825 (5 stratified folds) - LB 0.558
\nI was also able to get CV 0.71 with the same folds (by changing the random seed) and I got a slightly worse LB score

", "votes": 1, "replies": [] }, { "id": 2003856, "author_name": "yoyobar", "author_url": "", "post_date": "2022-10-25T20:37:11.603000", "content": "

train_acc=0.75 CV=0.408 LB=0.531. I think there is something wrong with my code.😂😂

", "votes": 1, "replies": [] }, { "id": 2019877, "author_name": "zh", "author_url": "", "post_date": "2022-11-07T03:53:46.650000", "content": "

single model
\nCV: 0.82x
\nLB: 0.712

\n
\n

It looks like the gap between cv and lb is large.
\nand cv and lb are not always positively correlated……

", "votes": 2, "replies": [] }, { "id": 2072234, "author_name": "George Chirita", "author_url": "", "post_date": "2022-12-21T22:02:02.050000", "content": "

CV0.8 +vs LB:0.700
\nBut is not relevant since we have our own datasets for training :) i have one dataset with high SNR :CV is 0.9 and LB 0.5-0.6

", "votes": 0, "replies": [] }, { "id": 2007857, "author_name": "", "author_url": "", "post_date": "2022-10-28T14:48:47.293000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1972704": "Let's share CV-LB pairs in this thread.", "2007848": "\n```\nCV - 0.751 (5 FOLD) (only comp data)\nLB - 0.706\n```", "2009588": "CV - 0.789 (5 fold)\nLB - 0.690\n\n--- updated 11/1\n\nCV - 0.829 (5 fold)\nLB - 0.714\n\nfor now, there's usually 0.1 gap between CV & LB on my experiments.", "2032928": "CV 0.74\nLB: 0.72", "2020414": "CV: 0.7806 (on given train set)\nLB: 0.660\n\nI'm using generated data. Also, I didn't start using CV until pretty late and so far I've been unable to reach my max LB of 0.677 which was without CV.", "2014872": "From same experiment (tf_efficientnet_b5_ns, 15 epochs, stratifiedkfold 5 folds):\n\n- best loss model\nCV: 0.7633 -> LB: 0.642\n\n- best score model\nCV: 0.7803 -> LB: 0.644\n\n- last epoch model\nCV: 0.7481 -> LB: 0.647\n\nBoth the loss value and the auc of the validation vary greatly from epoch to epoch, but it doesn't seem to be a meaningful variation.", "2005384": "CV 0.67825 (5 stratified folds) - LB 0.558\nI was also able to get CV 0.71 with the same folds (by changing the random seed) and I got a slightly worse LB score", "2003856": "train_acc=0.75 CV=0.408 LB=0.531. I think there is something wrong with my code.😂😂", "2019877": "single model\nCV: 0.82x \nLB: 0.712\n\n--------------------------------------------------\nIt looks like the gap between cv and lb is large.\nand cv and lb are not always positively correlated......", "2072234": "CV0.8 +vs LB:0.700\nBut is not relevant since we have our own datasets for training :) i have one dataset with high SNR :CV is 0.9 and LB 0.5-0.6", "2007857": "" } }