![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F991320%2F0b532244523e9a24e7fc37dbb307ca14%2FScreen%20Shot%202020-05-05%20at%2010.33.55%20PM.png?generation=1588732500014948&alt=media\")
model: se_resnext50 classification (no use of masks yet)\nimage size: 256x256\nCV: 0.6279 (Fold0: 0.6714 / Fold1: 0.6171 / Fold2: 0.6407 / Fold3: 0.6420 / Fold4: 0.6573)\nLB: 0.62x\nNot so much difference between CV & LB in my case
\n\n[update]\nCV: 0.7771 (fold0: 0.8132 / fold1: 0.7877 / fold2: 0.8143 / fold3: 0.8027 / fold4: 0.8057)\nLB: 0.76x
\n\nWhy some have much difference, others have not so much difference?
", "messages": [ { "id": 816821, "postDate": "2020-04-22T16:10:47.310Z", "content": "model: se_resnext50 classification (no use of masks yet)\nimage size: 256x256\nCV: 0.6279 (Fold0: 0.6714 / Fold1: 0.6171 / Fold2: 0.6407 / Fold3: 0.6420 / Fold4: 0.6573)\nLB: 0.62x\nNot so much difference between CV & LB in my case
\n\n[update]\nCV: 0.7771 (fold0: 0.8132 / fold1: 0.7877 / fold2: 0.8143 / fold3: 0.8027 / fold4: 0.8057)\nLB: 0.76x
\n\nWhy some have much difference, others have not so much difference?
", "rawMarkdown": "model: se_resnext50 classification (no use of masks yet)\nimage size: 256x256\nCV: 0.6279 (Fold0: 0.6714 / Fold1: 0.6171 / Fold2: 0.6407 / Fold3: 0.6420 / Fold4: 0.6573)\nLB: 0.62x\nNot so much difference between CV & LB in my case\n\n[update]\nCV: 0.7771 (fold0: 0.8132 / fold1: 0.7877 / fold2: 0.8143 / fold3: 0.8027 / fold4: 0.8057)\nLB: 0.76x\n\nWhy some have much difference, others have not so much difference?", "votes": 50 }, { "id": 841449, "postDate": "2020-05-10T19:48:46.693Z", "content": "Finally organized information on my more recent submissions, here is some info:
\n\n\n CV LB delta comment
\n----- ---- ------ -------
\n0.710 0.72 +0.010 level2 10 epochs resnet34
\n0.726 0.71 -0.016 rerun of the same? level2
\n0.788 0.74 -0.048 level1 scale 0.5 (so 2x smaller)
\n0.801 0.81 +0.009 level1 scale 1 resnet18
\n0.820 0.83 +0.010 level1 scale 0.875 resnet34
\n0.832 0.81 -0.022 level1 scale 0.5: switched to regression
\n0.856 0.82 -0.036 same with more aug + better loss
\n0.871 0.85 -0.021 level1 scale 1, 20 epochs
\n0.842 0.73 -0.112 level2 80 epochs
\n0.806 0.75 -0.056 level2 20 epochs\n
All submissions from one fold, without TTA. Using a stratified 5-fold split (except for the first few submissions).
\n\nSome observations:
\n\nThanks for share! Is level2 == lowest res?
", "rawMarkdown": "Thanks for share! Is level2 == lowest res?", "votes": 1 }, { "id": 842271, "postDate": "2020-05-11T09:44:13.697Z", "content": "\n\n\nIs level2 == lowest res?
\n
Yes
", "rawMarkdown": "> Is level2 == lowest res?\n\nYes" }, { "id": 842462, "postDate": "2020-05-11T12:10:08.643Z", "content": "thanks. appreciate your sharing.
\n\n> it seems that switching to regression made LB worse but CV better, although I'm not sure (could be just different scale)
\n\njudging from the aptos experiments it's the scale. All revealed solutions from top 45 used regression.
\n\n> 0.832 0.81 -0.022 level1 scale 0.5: switched to regression
\n> 0.856 0.82 -0.036 same with more aug + better loss
So it is still regression but different loss function. (common loss functions for regression)\nOn Aptos (as you could note i find aptos very relevant here) MSE worked like a charm
", "rawMarkdown": "thanks. appreciate your sharing. \n\n> it seems that switching to regression made LB worse but CV better, although I'm not sure (could be just different scale)\n\njudging from the aptos experiments it's the scale. All revealed solutions from top 45 used regression.\n\n> 0.832 0.81 -0.022 level1 scale 0.5: switched to regression \n> 0.856 0.82 -0.036 same with more aug + better loss \n\nSo it is still regression but different loss function. [(common loss functions for regression)](https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0)\nOn Aptos (as you could note i find aptos very relevant here) MSE worked like a charm ", "votes": 1 }, { "id": 843417, "postDate": "2020-05-12T03:32:28.993Z", "content": "training for longer might improve CV a lot but make LB worse. This happens to me too @lopuhin!
", "rawMarkdown": "training for longer might improve CV a lot but make LB worse. This happens to me too @lopuhin!", "votes": 1 }, { "id": 845817, "postDate": "2020-05-13T12:49:35.330Z", "content": "May I ask whether you use @Iafoss 's tile method ?
", "rawMarkdown": "May I ask whether you use @Iafoss 's tile method ?" }, { "id": 846336, "postDate": "2020-05-13T17:33:31.007Z", "content": "\n\n\nMay I ask whether you use @Iafoss 's tile method ?
\n
Yes, with only minor modifications.
", "rawMarkdown": "> May I ask whether you use @Iafoss 's tile method ?\n\nYes, with only minor modifications." }, { "id": 846448, "postDate": "2020-05-13T19:08:58.273Z", "content": "I'm also having these weird results where higher cv leads to lower lb, im not sure if i should trust cv or lb now haha
", "rawMarkdown": "I'm also having these weird results where higher cv leads to lower lb, im not sure if i should trust cv or lb now haha" }, { "id": 852395, "postDate": "2020-05-18T12:10:31.233Z", "content": "Hi. Nice results. I wonder how to make use th the obtained 4/5 fold weights to make predictions in case of regression. I am confused because for each fold we get qwk optimized coefs. Which one to use for inference? Or do we make predictions using individual folds using their own coefs and then take mode of the predictions as the final prediction?
", "rawMarkdown": "Hi. Nice results. I wonder how to make use th the obtained 4/5 fold weights to make predictions in case of **regression**. I am confused because for each fold we get qwk optimized coefs. Which one to use for inference? Or do we make predictions using individual folds using their own coefs and then take mode of the predictions as the final prediction? " }, { "id": 853295, "postDate": "2020-05-19T04:37:44.853Z", "content": "Hi @lopuhin , can u pls explain what is scale 0.5, 0.875, 1?
", "rawMarkdown": "Hi @lopuhin , can u pls explain what is scale 0.5, 0.875, 1? " }, { "id": 853501, "postDate": "2020-05-19T08:31:16.397Z", "content": "\n\n\ncan u pls explain what is scale 0.5, 0.875, 1?
\n
0.5 means 2x smaller image
\n\n\n\n\nI am confused because for each fold we get qwk optimized coefs. Which one to use for inference?
\n
good question - I didn't explore per-fold blending much yet, my initial idea in case of regression was to average raw predictions and average bins, but not sure it will work well.
", "rawMarkdown": "> can u pls explain what is scale 0.5, 0.875, 1? \n\n0.5 means 2x smaller image\n\n> I am confused because for each fold we get qwk optimized coefs. Which one to use for inference? \n\ngood question - I didn't explore per-fold blending much yet, my initial idea in case of regression was to average raw predictions and average bins, but not sure it will work well." }, { "id": 853515, "postDate": "2020-05-19T08:41:19.997Z", "content": "0.5 means 2x smaller image That means you resize the level1 image to half its size and then train on it after some other preprocessing?
\n\n\nThat means you resize the level1 image to half its size and then train on it after some other preprocessing?
\n
Yes - I build the slides from it like everyone else seems to do.
", "rawMarkdown": "> That means you resize the level1 image to half its size and then train on it after some other preprocessing?\n\nYes - I build the slides from it like everyone else seems to do.", "votes": 1 }, { "id": 853575, "postDate": "2020-05-19T09:51:33.637Z", "content": "Are you using kaggle kernels for training with slides from level1 images or u r training locally? Looking at GPU usage for level2 images in my kernels, I don't think I will be able to use slides from level1. I just want to know how do u do it?
", "rawMarkdown": "Are you using kaggle kernels for training with slides from level1 images or u r training locally? Looking at GPU usage for level2 images in my kernels, I don't think I will be able to use slides from level1. I just want to know how do u do it? " }, { "id": 853666, "postDate": "2020-05-19T11:28:48.447Z", "content": "I'm mostly training locally on 1 GPU which has less memory than on Kaggle (11 GB vs 16 GB on Kaggle). With higher resolution you have to reduce batch size and/or number of tiles but it's still totally doable with even higher resolution than level 1.
", "rawMarkdown": "I'm mostly training locally on 1 GPU which has less memory than on Kaggle (11 GB vs 16 GB on Kaggle). With higher resolution you have to reduce batch size and/or number of tiles but it's still totally doable with even higher resolution than level 1.", "votes": 2 }, { "id": 853791, "postDate": "2020-05-19T13:34:44.590Z", "content": "Thanks for galvanizing me! I'll surely try.
", "rawMarkdown": "Thanks for galvanizing me! I'll surely try. " } ] }, { "id": 834999, "postDate": "2020-05-06T01:10:47.177Z", "content": "Just a starter baseline: \n```
\n\nmodel: resnset50 (pertained: Imagenet) \nFold: 1\nsplit: random (80/20)\nepoch: 5\ntype: classifcation\nloss_valid: 0.815392\nqwk_valid: 0.837932\nlb: 0.81\naugment: rotate(20), brightens and contrast
\n\n```
\n\nedit : \nmy submission history, had to fight with kernels =);\n
\n\n\nJust a starter baseline\n lb: 0.81
\n
Nice baseline!
", "rawMarkdown": "> Just a starter baseline\n> lb: 0.81\n\nNice baseline!", "votes": 1 }, { "id": 842307, "postDate": "2020-05-11T10:22:13.190Z", "content": "epoch: 5 and lb: 0.81 !!\nThat is pretty amazing
", "rawMarkdown": "epoch: 5 and lb: 0.81 !!\nThat is pretty amazing", "votes": 2 } ] }, { "id": 831776, "postDate": "2020-05-03T15:47:25.093Z", "content": "Single fold resnet18: 0.80 CV, 0.81 LB\nSingle fold resnet34: 0.82 CV, 0.83 LB
", "rawMarkdown": "Single fold resnet18: 0.80 CV, 0.81 LB\nSingle fold resnet34: 0.82 CV, 0.83 LB", "votes": 10 }, { "id": 819896, "postDate": "2020-04-25T01:00:26.780Z", "content": "Regression model as APTOS competition , seresnext_50 with 5 folds, image size: 512 from here, cv with optimized threshold 0.768 and lb with optimized threshold 0.72; cv with fixed threshold 0.757 and lb with fixed threshold 0.71.
", "rawMarkdown": "Regression model as [APTOS competition](https://www.kaggle.com/c/aptos2019-blindness-detection) , seresnext_50 with 5 folds, image size: 512 from [here](https://www.kaggle.com/xhlulu/panda-resized-train-data-512x512), cv with optimized threshold 0.768 and lb with optimized threshold 0.72; cv with fixed threshold 0.757 and lb with fixed threshold 0.71.", "votes": 7, "replies": [ { "id": 821175, "postDate": "2020-04-26T01:25:26.540Z", "content": "Hey! Ive been trying to make a cnn regression model but its my first time and im encountering some problems. It seems like my model isnt converging, my loss is staying a arround 2.5... I followed this kernel from APTOS:https://www.kaggle.com/chanhu/eye-efficientnet-pytorch-lb-0-777
", "rawMarkdown": "Hey! Ive been trying to make a cnn regression model but its my first time and im encountering some problems. It seems like my model isnt converging, my loss is staying a arround 2.5... I followed this kernel from APTOS:https://www.kaggle.com/chanhu/eye-efficientnet-pytorch-lb-0-777" }, { "id": 821188, "postDate": "2020-04-26T01:49:28.730Z", "content": "Maybe you should check the processing part and model weights, this kernel uses special processing inside Dataset class for APTOS task, and it loads weight from the authors' trained weight for APTOS instead of imagenet weight.
", "rawMarkdown": "Maybe you should check the processing part and model weights, this kernel uses special processing inside Dataset class for APTOS task, and it loads weight from the authors' trained weight for APTOS instead of imagenet weight. ", "votes": 1 }, { "id": 821204, "postDate": "2020-04-26T02:21:59.407Z", "content": "Yes of course, im not reruning the kernel, im training locally and just looked at the main differences of a regression model! \nI used the isup value as label and one output as the prediction with mseloss but cant seem to converge..
", "rawMarkdown": "Yes of course, im not reruning the kernel, im training locally and just looked at the main differences of a regression model! \nI used the isup value as label and one output as the prediction with mseloss but cant seem to converge.." }, { "id": 821210, "postDate": "2020-04-26T02:34:44.707Z", "content": "I used SmoothL1Loss, maybe you can try it. :)
", "rawMarkdown": "I used SmoothL1Loss, maybe you can try it. :)" }, { "id": 821219, "postDate": "2020-04-26T02:49:01.417Z", "content": "I had tried it but no success :( thanks anyways! It's probably a bug in my code and ill find it one day haha
", "rawMarkdown": "I had tried it but no success :( thanks anyways! It's probably a bug in my code and ill find it one day haha", "votes": 1 }, { "id": 822865, "postDate": "2020-04-27T07:50:46.470Z", "content": "What does cv with optimized threshold and cv with fixed threshold means?
", "rawMarkdown": "What does cv with optimized threshold and cv with fixed threshold means?", "votes": 1 } ] }, { "id": 829989, "postDate": "2020-05-02T08:24:47.583Z", "content": "single fold seresnext50\n\ncv: 0.92\nlb: 0.80\nsplit: 80:20 \npretrained weights: imagenet\n
![\"\"](\"https://raw.githubusercontent.com/appian42/kaggle/master/PANDA/confusion.png\")
single fold model. Much of the code follows @lafoss work\nsee\n - https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb\n - https://www.kaggle.com/hengck23/kernel16867b0575
\n\n```\nresnet34, 20 384x384 patches per image at 0.5 scale (i.e. use highest resolution of tiff image and apply scale of 0.5)
\n\npublic LB 0.85
\n\nvalidation:\naugment = ['null', 'all_flip']\nkapp = 0.844811\naccuracy_top2 = 0.710156, 0.894637\nlog_loss = 0.775169
\n\n```
", "rawMarkdown": "single fold model. Much of the code follows @lafoss work\nsee\n - https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb\n - https://www.kaggle.com/hengck23/kernel16867b0575\n \n\n```\nresnet34, 20 384x384 patches per image at 0.5 scale (i.e. use highest resolution of tiff image and apply scale of 0.5)\n\npublic LB 0.85\n\nvalidation:\naugment = ['null', 'all_flip']\nkapp = 0.844811\naccuracy_top2 = 0.710156, 0.894637\nlog_loss = 0.775169\n\n\n```\n", "votes": 5, "replies": [ { "id": 852676, "postDate": "2020-05-18T15:50:43.770Z", "content": "so you mean for each image you take 20 tiles at sz 384*384 each? isn't it super slow and requires lot of hardwares to train like that?
", "rawMarkdown": "so you mean for each image you take 20 tiles at sz 384*384 each? isn't it super slow and requires lot of hardwares to train like that?" } ] }, { "id": 836136, "postDate": "2020-05-06T18:36:12.177Z", "content": "My results from 5-fold CV:
\n\n\nCV/LB:\n0.88/0.87\n0.91/0.87\n0.89/0.88\n
Depending on the approach, there can be a lot of bias. I think the more non-tissue elements of the image you include, the more bias. This may explain larger CV-LB differences for those using full-size image approaches on the lowest magnification.
", "rawMarkdown": "My results from 5-fold CV:\n\n```\nCV/LB:\n0.88/0.87\n0.91/0.87\n0.89/0.88\n```\n\nDepending on the approach, there can be a lot of bias. I think the more non-tissue elements of the image you include, the more bias. This may explain larger CV-LB differences for those using full-size image approaches on the lowest magnification. ", "votes": 5, "replies": [ { "id": 836459, "postDate": "2020-05-07T02:22:00.073Z", "content": "Great result. Thanks for the information!
", "rawMarkdown": "Great result. Thanks for the information!", "votes": 2 } ] }, { "id": 834204, "postDate": "2020-05-05T11:42:24.513Z", "content": "Single fold resnet50: 0.860 CV, 0.84 LB\nSingle fold effb4: 0.862 CV, 0.85 LB
", "rawMarkdown": "Single fold resnet50: 0.860 CV, 0.84 LB\nSingle fold effb4: 0.862 CV, 0.85 LB", "votes": 5 }, { "id": 818752, "postDate": "2020-04-24T05:21:04.537Z", "content": "model: resnet50 regression\nimg_size: 256*256\nsingle fold cv: 0.718\nsingle fold lb: 0.65
\n\nupdate:\nmodel: seresnext50 classification\nimg_size: 512*512\nsingle fold cv: 0.75\nlb: 0.73 (w/ TTA, w/ label smoothing)
", "rawMarkdown": "model: resnet50 regression\nimg_size: 256*256\nsingle fold cv: 0.718\nsingle fold lb: 0.65\n\nupdate:\nmodel: seresnext50 classification\nimg_size: 512*512\nsingle fold cv: 0.75\nlb: 0.73 (w/ TTA, w/ label smoothing)", "votes": 6 }, { "id": 845708, "postDate": "2020-05-13T11:34:17.290Z", "content": "5 Fold CV/LB\n\nresnet34 classification 0.80/0.78\nresnet34 regression 0.81/0.76\n\nI let each model learn 50 epochs.\nI'm using @Iafoss 's tile method by level 2 resolution.\nAlthough there are still few experiments, it seems that the gap between CV and LB is larger in regression.
Hi. In the case of regression, if you are using qwk optimised coefficients, which fold coefs do you use when making inference with 5 fold. Or you use coefs found in individual fold to make test set predictions with individual fold and then take the mode of the predictions for each test sample?
", "rawMarkdown": "Hi. In the case of regression, if you are using qwk optimised coefficients, which fold coefs do you use when making inference with 5 fold. Or you use coefs found in individual fold to make test set predictions with individual fold and then take the mode of the predictions for each test sample?" }, { "id": 853159, "postDate": "2020-05-19T02:17:23Z", "content": "I tried both the method of optimizing the coefficients after averaging the regression values, and the method of voting on the results of applying the coefficients in each fold.\nAs a result, the latter was slightly better in both CV and LB, so I used the latter.
", "rawMarkdown": "I tried both the method of optimizing the coefficients after averaging the regression values, and the method of voting on the results of applying the coefficients in each fold.\nAs a result, the latter was slightly better in both CV and LB, so I used the latter." } ] }, { "id": 841876, "postDate": "2020-05-11T04:38:39.190Z", "content": "Single Fold: 0.85 | LB: 0.81 | LB with TTA: 0.81 | LB with 5 fold blending: 0.81 | LB with 5 fold blending and TTA: 0.81
", "rawMarkdown": "Single Fold: 0.85 | LB: 0.81 | LB with TTA: 0.81 | LB with 5 fold blending: 0.81 | LB with 5 fold blending and TTA: 0.81", "votes": 3 }, { "id": 835567, "postDate": "2020-05-06T10:57:13.503Z", "content": "seresnext50, regression, single fold: CV 0.815 / LB 0.81
\n\nUPDATED:\nseresnext50, classification, single fold: CV 0.865 / LB 0.87
", "rawMarkdown": "seresnext50, regression, single fold: CV 0.815 / LB 0.81\n\nUPDATED:\nseresnext50, classification, single fold: CV 0.865 / LB 0.87", "votes": 3, "replies": [ { "id": 856635, "postDate": "2020-05-21T23:53:40.367Z", "content": "Great results @analokamus. I have the same configuration but can't get over 0.79 after 40 epochs for some reason. What Loss/Metric did you use? Any preprocessing trick that helped your CV/LB? π
", "rawMarkdown": "Great results @analokamus. I have the same configuration but can't get over 0.79 after 40 epochs for some reason. What Loss/Metric did you use? Any preprocessing trick that helped your CV/LB? π ", "votes": 1 } ] }, { "id": 831041, "postDate": "2020-05-03T04:03:53.093Z", "content": "EfficientNetB0 segmentation model - single fold(0.8:0.2)\n\ncv : 0.44\nLB : 0.53\n
EfficientNet-B0\n\n- 4 folds (75:25 splits)\n- CV: 0.83\n- LB: 0.83\n
seresnext50, regression\ncv: 0.815, lb: 0.60
", "rawMarkdown": "seresnext50, regression\ncv: 0.815, lb: 0.60", "votes": 3 }, { "id": 828631, "postDate": "2020-05-01T07:40:19.850Z", "content": "seresnet50, regression, 5fold avg\ncv: 0.83+ lb:0.77\n\ndf = pd.read_csv(os.path.join(data_dir,'train.csv'))\nskf = StratifiedKFold(n_splits=5, shuffle = True, random_state = 2020)\nfor fold, (train_index, val_index) in enumerate(skf.split(df.values, df['isup_grade'])):\n df.loc[val_index, 'fold'] = int(fold)\n
I met a weird cv/lb gap...\n- dataset: PANDA: Level 1 and 2 images\n- model: efficientnet b0\n- split: random split 0.8:0.2\n- fold: single fold\n- augment: flip and rotate\n- normalize: imagenet mean and std\n- image-size: 300 x 300 each tile ( use 5 tiles from level-1(intermediate) image, process like iafoss's )\n- epochs(the same model, as I train it 5+5 epochs, so I have a result of 5, and another of 10)\n - 5 cv: 0.7672814726829529 lb: 0.79\n - 10 cv: 0.7833058834075928 lb 0.75
\n\nPS: the tile size 300 is from my observation of visualizing the tiles after cutting by my eyes. I only compare 224 and 300 because the input-size of b0 is 224... And 300 has more raw-data pixels for just 5 tiles, so I choose 300. I visualize 16 tiles, as I ready to seam 16 tile into a image(4x4) like PANDA / se_resnext50 regression baseline did.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3149580%2F9a1d2084a054f0f5a8d6544391706889%2Ftiles_224vs300.png?generation=1590581574462695&alt=media\")
\nafter seaming, cv2.vconcat(cv2.hconcat...)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3149580%2Fd97b2b5e240a5cfc0f1a3ecd3245041c%2Fseam_224vs300.png?generation=1590581817847055&alt=media\")
As we can see the 300 has more blank, so I perfer to feed 224 to model.
", "rawMarkdown": "I met a weird cv/lb gap...\n- dataset: [PANDA: Level 1 and 2 images](https://www.kaggle.com/lopuhin/panda-2020-level-1-2)\n- model: efficientnet b0\n- split: random split 0.8:0.2\n- fold: single fold\n- augment: flip and rotate\n- normalize: imagenet mean and std\n- image-size: 300 x 300 each tile ( use 5 tiles from level-1(intermediate) image, process like iafoss's )\n- epochs(the same model, as I train it 5+5 epochs, so I have a result of 5, and another of 10)\n - 5 cv: 0.7672814726829529 lb: 0.79\n - 10 cv: 0.7833058834075928 lb 0.75\n\nPS: the tile size 300 is from my observation of visualizing the tiles after cutting by my eyes. I only compare 224 and 300 because the input-size of b0 is 224... And 300 has more raw-data pixels for just 5 tiles, so I choose 300. I visualize 16 tiles, as I ready to seam 16 tile into a image(4x4) like [PANDA / se_resnext50 regression baseline](https://www.kaggle.com/yasufuminakama/panda-se-resnext50-regression-baseline) did.\n\nafter seaming, cv2.vconcat(cv2.hconcat...)\n\n\nAs we can see the 300 has more blank, so I perfer to feed 224 to model.", "votes": 1, "replies": [ { "id": 863783, "postDate": "2020-05-27T15:02:06.943Z", "content": "Same thing happens for me when i train for more epochs my CV can increase but LB decrease, but many people see a correlation between Karolinska CV with LB score (if im not wrong)
", "rawMarkdown": "Same thing happens for me when i train for more epochs my CV can increase but LB decrease, but many people see a correlation between Karolinska CV with LB score (if im not wrong)" }, { "id": 864336, "postDate": "2020-05-28T00:34:15.877Z", "content": "Yes, you are right and it also happens to me.\nAs I split the data by provider, karolinska's score is similar to lb.\nAnd if I train more epochs(just from 6 to 10), the loss of train decrease but valid loss is stagnant(and cv increase, lb decrease).. loss(8,9,10 epoch) like below, my loss is crossentropy\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3149580%2F044828c764a69c7d5a63b63b4c538f02%2Fstagnant_loss.png?generation=1590625917481566&alt=media\")
So I plan to add more augmentations and use deeper backbone. What do you think about it?
", "rawMarkdown": "Yes, you are right and it also happens to me.\nAs I split the data by provider, karolinska's score is similar to lb.\nAnd if I train more epochs(just from 6 to 10), the loss of train decrease but valid loss is stagnant(and cv increase, lb decrease).. loss(8,9,10 epoch) like below, my loss is crossentropy\n\n\nSo I plan to add more augmentations and use deeper backbone. What do you think about it?" }, { "id": 864365, "postDate": "2020-05-28T00:55:38.377Z", "content": "I think you should try iafoss method of concat pooling and yes add a little more augmentations or regularization methods! Also i think having more tiles of smaller size is a little better
", "rawMarkdown": "I think you should try iafoss method of concat pooling and yes add a little more augmentations or regularization methods! Also i think having more tiles of smaller size is a little better", "votes": 1 }, { "id": 864381, "postDate": "2020-05-28T01:08:49.967Z", "content": "Thanks for your fast reply! I use GeM pool as mentioned using lafoss minimum model and how to improve by DrHB. And actually the GeM pool give me the most consistent trend of train and valid loos(and best cv and lb) compared with avg/max/concat.\nI will try more tiles with smaller size! Thank you for your adviceπ
", "rawMarkdown": "Thanks for your fast reply! I use GeM pool as mentioned [using lafoss minimum model and how to improve](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/152265) by DrHB. And actually the GeM pool give me the most consistent trend of train and valid loos(and best cv and lb) compared with avg/max/concat.\nI will try more tiles with smaller size! Thank you for your adviceπ ", "votes": 2 }, { "id": 864560, "postDate": "2020-05-28T04:23:15.217Z", "content": "Yes GeM pooling is what im using too, ive always had great results with this pooling method!
", "rawMarkdown": "Yes GeM pooling is what im using too, ive always had great results with this pooling method!" } ] }, { "id": 818549, "postDate": "2020-04-24T01:02:47.557Z", "content": "EDIT: Starting over with new method
\n\nSingle Fold Validation: .767\nLB: .75
\n\nSingle Fold Validation: .765\nLB: .78
\n\nVal: .776\nLB: .77
", "rawMarkdown": "EDIT: Starting over with new method\n\nSingle Fold Validation: .767\nLB: .75\n\nSingle Fold Validation: .765\nLB: .78\n\nVal: .776\nLB: .77", "votes": 1 }, { "id": 817978, "postDate": "2020-04-23T15:14:58.587Z", "content": "updated
\n\n|CV|LB|\n| --- | --- |\n|0.687|0.62|\n|0.692|0.62|\n|0.698|0.62|\n|0.654|0.64|\n|0.654|0.57|
\n\n4fold, se_resnext50 classification
", "rawMarkdown": "updated\n\n|CV|LB|\n| --- | --- |\n|0.687|0.62|\n|0.692|0.62|\n|0.698|0.62|\n|0.654|0.64|\n|0.654|0.57|\n\n4fold, se_resnext50 classification", "votes": 1, "replies": [ { "id": 818069, "postDate": "2020-04-23T16:02:55.120Z", "content": "Single fold score or Cross Validation score?
", "rawMarkdown": "Single fold score or Cross Validation score?", "votes": 2 }, { "id": 818519, "postDate": "2020-04-24T00:22:15.683Z", "content": "4fold score.\nMy code is based on your baseline notebook. Thank you so much!
", "rawMarkdown": "4fold score.\nMy code is based on your baseline [notebook](https://www.kaggle.com/yasufuminakama/panda-se-resnext50-classification-baseline). Thank you so much!", "votes": 1 }, { "id": 819998, "postDate": "2020-04-25T04:44:45.740Z", "content": "I've done 5 submits so far and CV and LB don't seem to correlate. I want to create a reliable validation set.
", "rawMarkdown": "I've done 5 submits so far and CV and LB don't seem to correlate. I want to create a reliable validation set." } ] }, { "id": 817479, "postDate": "2020-04-23T07:33:43.470Z", "content": "single efficientnet-b0 model \nOnly trained on 1000 training data\nCV: 0.38\nLB: 0.36
", "rawMarkdown": "single efficientnet-b0 model \nOnly trained on 1000 training data\nCV: 0.38\nLB: 0.36", "votes": 1 }, { "id": 816834, "postDate": "2020-04-22T16:33:59.483Z", "content": "That's rather a good news
", "rawMarkdown": "That's rather a good news", "votes": 1 }, { "id": 822861, "postDate": "2020-04-27T07:44:01.507Z", "content": "model: se_resnext50 classification\nCV-0.77 | LB-0.60 | Single Fold | 30 epoch\nCV-0.69 | LB-0.69 | Single Fold | 5 epcoh\nupdata:\nCV-0.869 | LB-0.86 | Single Fold
", "rawMarkdown": "model: se_resnext50 classification\nCV-0.77 | LB-0.60 | Single Fold | 30 epoch\nCV-0.69 | LB-0.69 | Single Fold | 5 epcoh\nupdata:\nCV-0.869 | LB-0.86 | Single Fold ", "votes": 2, "replies": [ { "id": 822940, "postDate": "2020-04-27T09:10:58.647Z", "content": "Hi, He, what's the img size do you use?
", "rawMarkdown": "Hi, He, what's the img size do you use?" }, { "id": 822960, "postDate": "2020-04-27T09:29:45.393Z", "content": "512*512
", "rawMarkdown": "512*512", "votes": 1 }, { "id": 822962, "postDate": "2020-04-27T09:30:53.277Z", "content": "thank you
", "rawMarkdown": "thank you" } ] }, { "id": 821142, "postDate": "2020-04-26T00:09:13.373Z", "content": "Se-ResNext50 single fold\nCV: 0.629154\nLB: 0.59, 0.63 w/ TTA, 0.67 w/ label smoothing
\n\nIn the spirit of public sharing here's the kernel
\n\nAlso, check out the precursor kernel over here
\n\nEDIT: new score with Label Smoothing
", "rawMarkdown": "Se-ResNext50 single fold\nCV: 0.629154\nLB: 0.59, 0.63 w/ TTA, 0.67 w/ label smoothing\n\nIn the spirit of public sharing here's the [kernel](https://www.kaggle.com/tanlikesmath/prostate-cancer-grading-intro-fastai2-starter)\n\nAlso, check out the precursor kernel over [here](https://www.kaggle.com/tanlikesmath/fastai2-training-baseline)\n\nEDIT: new score with Label Smoothing", "votes": 2 }, { "id": 819090, "postDate": "2020-04-24T10:52:10.283Z", "content": "resnet + fastfcn\ncv: 0.83, lb: 0.50
", "rawMarkdown": "resnet + fastfcn\ncv: 0.83, lb: 0.50", "votes": 2, "replies": [ { "id": 819852, "postDate": "2020-04-24T23:50:40.553Z", "content": "Hm, any idea why there's such a large difference between CV & LB for your model?
", "rawMarkdown": "Hm, any idea why there's such a large difference between CV & LB for your model?" }, { "id": 821578, "postDate": "2020-04-26T09:08:20.190Z", "content": "Hi @phalanx if you dont mind, do you also use classification model? Or segmentation model?
", "rawMarkdown": "Hi @phalanx if you dont mind, do you also use classification model? Or segmentation model?" } ] }, { "id": 861520, "postDate": "2020-05-26T04:59:14.313Z", "content": "I've been experimenting with low resolution since the tile method was published. However, the LB score cannot exceed 0.79 (and the CV score improve, but the LB is not correlated). What's the best CV/LB with low resolution?
", "rawMarkdown": "I've been experimenting with low resolution since the tile method was published. However, the LB score cannot exceed 0.79 (and the CV score improve, but the LB is not correlated). What's the best CV/LB with low resolution?" }, { "id": 857584, "postDate": "2020-05-22T18:11:58.290Z", "content": "Any idea for improvement is welcome π πͺ
\n\nFramework:
\n\nSingle fold:
\n\nX_train, X_val = train_test_split(train, test_size=.2, stratify=train['isup_grade'], random_state=SEED)\nPre processing:
\n\nConfig:
\n\nLR: 1e-3 \nBS: 16\nEpoch: 40\nModel: Seresnext50 backbone
\nClassification:
loss='categorical_crossentropy'\noptimizer=optimizers.Adam(lr=LR)\nmetrics=[qw_kappa_score]\nCallback= ReduceLROnPlateau
CV: 0.78\nLB: 0.79\nSingle Fold, cv/lb:\nse_resnext regression: 0.8233/0.78\nresnet34 regression 0.772/0.75
", "rawMarkdown": "Single Fold, cv/lb:\nse_resnext regression: 0.8233/0.78\nresnet34 regression 0.772/0.75" }, { "id": 835310, "postDate": "2020-05-06T07:21:53.100Z", "content": "mine is 0.81 CV and 0.82 LB, single fold resnet18
", "rawMarkdown": "mine is 0.81 CV and 0.82 LB, single fold resnet18" }, { "id": 825803, "postDate": "2020-04-29T09:00:45.270Z", "content": "Model - seresnext50 imagenet pretrained\nimg size - 384x384\nCV - 0.63\nLB - 0.66
", "rawMarkdown": "Model - seresnext50 imagenet pretrained\nimg size - 384x384\nCV - 0.63\nLB - 0.66" }, { "id": 822836, "postDate": "2020-04-27T07:16:08.940Z", "content": "Using classifiction for baseline-making.\nModel: Mixnet-L single-fold\nUsing 1:5 aspect ratio\nCV 0.80, LB .66. Not sure if I did my inference script totally right...
", "rawMarkdown": "Using classifiction for baseline-making.\nModel: Mixnet-L single-fold\nUsing 1:5 aspect ratio\nCV 0.80, LB .66. Not sure if I did my inference script totally right..." }, { "id": 820280, "postDate": "2020-04-25T09:45:14.720Z", "content": "efficientnet-b3, 256x256\n| CV | LB |\n| --- | --- |\n| 0.658 | 0.62 |\n| 0.687 | 0.61 |
\n\nMy CV-LB seems uncorrelated ...
", "rawMarkdown": "efficientnet-b3, 256x256\n| CV | LB |\n| --- | --- |\n| 0.658 | 0.62 |\n| 0.687 | 0.61 |\n\nMy CV-LB seems uncorrelated ..." }, { "id": 818508, "postDate": "2020-04-24T00:07:03.217Z", "content": "Single fold\nVal: 0.6426\nLB: 0.57
\n\nMmh i got a larger gap weird :\\
", "rawMarkdown": "Single fold\nVal: 0.6426\nLB: 0.57\n\nMmh i got a larger gap weird :\\" } ], "comments": [ { "id": 841449, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-10T19:48:46.693000", "content": "Finally organized information on my more recent submissions, here is some info:
\n\n\n CV LB delta comment
\n----- ---- ------ -------
\n0.710 0.72 +0.010 level2 10 epochs resnet34
\n0.726 0.71 -0.016 rerun of the same? level2
\n0.788 0.74 -0.048 level1 scale 0.5 (so 2x smaller)
\n0.801 0.81 +0.009 level1 scale 1 resnet18
\n0.820 0.83 +0.010 level1 scale 0.875 resnet34
\n0.832 0.81 -0.022 level1 scale 0.5: switched to regression
\n0.856 0.82 -0.036 same with more aug + better loss
\n0.871 0.85 -0.021 level1 scale 1, 20 epochs
\n0.842 0.73 -0.112 level2 80 epochs
\n0.806 0.75 -0.056 level2 20 epochs\n
All submissions from one fold, without TTA. Using a stratified 5-fold split (except for the first few submissions).
\n\nSome observations:
\n\nThanks for share! Is level2 == lowest res?
", "votes": 1, "replies": [] }, { "id": 842271, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-11T09:44:13.697000", "content": "\n\n\nIs level2 == lowest res?
\n
Yes
", "votes": 0, "replies": [] }, { "id": 842462, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-05-11T12:10:08.643000", "content": "thanks. appreciate your sharing.
\n\n> it seems that switching to regression made LB worse but CV better, although I'm not sure (could be just different scale)
\n\njudging from the aptos experiments it's the scale. All revealed solutions from top 45 used regression.
\n\n> 0.832 0.81 -0.022 level1 scale 0.5: switched to regression
\n> 0.856 0.82 -0.036 same with more aug + better loss
So it is still regression but different loss function. (common loss functions for regression)\nOn Aptos (as you could note i find aptos very relevant here) MSE worked like a charm
", "votes": 1, "replies": [] }, { "id": 843417, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-05-12T03:32:28.993000", "content": "training for longer might improve CV a lot but make LB worse. This happens to me too @lopuhin!
", "votes": 1, "replies": [] }, { "id": 845817, "author_name": "seefun", "author_url": "", "post_date": "2020-05-13T12:49:35.330000", "content": "May I ask whether you use @Iafoss 's tile method ?
", "votes": 0, "replies": [] }, { "id": 846336, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-13T17:33:31.007000", "content": "\n\n\nMay I ask whether you use @Iafoss 's tile method ?
\n
Yes, with only minor modifications.
", "votes": 0, "replies": [] }, { "id": 846448, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-05-13T19:08:58.273000", "content": "I'm also having these weird results where higher cv leads to lower lb, im not sure if i should trust cv or lb now haha
", "votes": 0, "replies": [] }, { "id": 852395, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-05-18T12:10:31.233000", "content": "Hi. Nice results. I wonder how to make use th the obtained 4/5 fold weights to make predictions in case of regression. I am confused because for each fold we get qwk optimized coefs. Which one to use for inference? Or do we make predictions using individual folds using their own coefs and then take mode of the predictions as the final prediction?
", "votes": 0, "replies": [] }, { "id": 853295, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-05-19T04:37:44.853000", "content": "Hi @lopuhin , can u pls explain what is scale 0.5, 0.875, 1?
", "votes": 0, "replies": [] }, { "id": 853501, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-19T08:31:16.397000", "content": "\n\n\ncan u pls explain what is scale 0.5, 0.875, 1?
\n
0.5 means 2x smaller image
\n\n\n\n\nI am confused because for each fold we get qwk optimized coefs. Which one to use for inference?
\n
good question - I didn't explore per-fold blending much yet, my initial idea in case of regression was to average raw predictions and average bins, but not sure it will work well.
", "votes": 0, "replies": [] }, { "id": 853515, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-05-19T08:41:19.997000", "content": "0.5 means 2x smaller image That means you resize the level1 image to half its size and then train on it after some other preprocessing?
\n\n\nThat means you resize the level1 image to half its size and then train on it after some other preprocessing?
\n
Yes - I build the slides from it like everyone else seems to do.
", "votes": 1, "replies": [] }, { "id": 853575, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-05-19T09:51:33.637000", "content": "Are you using kaggle kernels for training with slides from level1 images or u r training locally? Looking at GPU usage for level2 images in my kernels, I don't think I will be able to use slides from level1. I just want to know how do u do it?
", "votes": 0, "replies": [] }, { "id": 853666, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-19T11:28:48.447000", "content": "I'm mostly training locally on 1 GPU which has less memory than on Kaggle (11 GB vs 16 GB on Kaggle). With higher resolution you have to reduce batch size and/or number of tiles but it's still totally doable with even higher resolution than level 1.
", "votes": 2, "replies": [] }, { "id": 853791, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-05-19T13:34:44.590000", "content": "Thanks for galvanizing me! I'll surely try.
", "votes": 0, "replies": [] } ] }, { "id": 834999, "author_name": "DrHB", "author_url": "", "post_date": "2020-05-06T01:10:47.177000", "content": "Just a starter baseline: \n```
\n\nmodel: resnset50 (pertained: Imagenet) \nFold: 1\nsplit: random (80/20)\nepoch: 5\ntype: classifcation\nloss_valid: 0.815392\nqwk_valid: 0.837932\nlb: 0.81\naugment: rotate(20), brightens and contrast
\n\n```
\n\nedit : \nmy submission history, had to fight with kernels =);\n
\n\n\nJust a starter baseline\n lb: 0.81
\n
Nice baseline!
", "votes": 1, "replies": [] }, { "id": 842307, "author_name": "YS", "author_url": "", "post_date": "2020-05-11T10:22:13.190000", "content": "epoch: 5 and lb: 0.81 !!\nThat is pretty amazing
", "votes": 2, "replies": [] } ] }, { "id": 831776, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-03T15:47:25.093000", "content": "Single fold resnet18: 0.80 CV, 0.81 LB\nSingle fold resnet34: 0.82 CV, 0.83 LB
", "votes": 10, "replies": [] }, { "id": 819896, "author_name": "jionie", "author_url": "", "post_date": "2020-04-25T01:00:26.780000", "content": "Regression model as APTOS competition , seresnext_50 with 5 folds, image size: 512 from here, cv with optimized threshold 0.768 and lb with optimized threshold 0.72; cv with fixed threshold 0.757 and lb with fixed threshold 0.71.
", "votes": 7, "replies": [ { "id": 821175, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-04-26T01:25:26.540000", "content": "Hey! Ive been trying to make a cnn regression model but its my first time and im encountering some problems. It seems like my model isnt converging, my loss is staying a arround 2.5... I followed this kernel from APTOS:https://www.kaggle.com/chanhu/eye-efficientnet-pytorch-lb-0-777
", "votes": 0, "replies": [] }, { "id": 821188, "author_name": "jionie", "author_url": "", "post_date": "2020-04-26T01:49:28.730000", "content": "Maybe you should check the processing part and model weights, this kernel uses special processing inside Dataset class for APTOS task, and it loads weight from the authors' trained weight for APTOS instead of imagenet weight.
", "votes": 1, "replies": [] }, { "id": 821204, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-04-26T02:21:59.407000", "content": "Yes of course, im not reruning the kernel, im training locally and just looked at the main differences of a regression model! \nI used the isup value as label and one output as the prediction with mseloss but cant seem to converge..
", "votes": 0, "replies": [] }, { "id": 821210, "author_name": "jionie", "author_url": "", "post_date": "2020-04-26T02:34:44.707000", "content": "I used SmoothL1Loss, maybe you can try it. :)
", "votes": 0, "replies": [] }, { "id": 821219, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-04-26T02:49:01.417000", "content": "I had tried it but no success :( thanks anyways! It's probably a bug in my code and ill find it one day haha
", "votes": 1, "replies": [] }, { "id": 822865, "author_name": "torch", "author_url": "", "post_date": "2020-04-27T07:50:46.470000", "content": "What does cv with optimized threshold and cv with fixed threshold means?
", "votes": 1, "replies": [] } ] }, { "id": 829989, "author_name": "Appian", "author_url": "", "post_date": "2020-05-02T08:24:47.583000", "content": "single fold seresnext50\n\ncv: 0.92\nlb: 0.80\nsplit: 80:20 \npretrained weights: imagenet\n
single fold model. Much of the code follows @lafoss work\nsee\n - https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb\n - https://www.kaggle.com/hengck23/kernel16867b0575
\n\n```\nresnet34, 20 384x384 patches per image at 0.5 scale (i.e. use highest resolution of tiff image and apply scale of 0.5)
\n\npublic LB 0.85
\n\nvalidation:\naugment = ['null', 'all_flip']\nkapp = 0.844811\naccuracy_top2 = 0.710156, 0.894637\nlog_loss = 0.775169
\n\n```
", "votes": 5, "replies": [ { "id": 852676, "author_name": "DatNT", "author_url": "", "post_date": "2020-05-18T15:50:43.770000", "content": "so you mean for each image you take 20 tiles at sz 384*384 each? isn't it super slow and requires lot of hardwares to train like that?
", "votes": 0, "replies": [] } ] }, { "id": 836136, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-05-06T18:36:12.177000", "content": "My results from 5-fold CV:
\n\n\nCV/LB:\n0.88/0.87\n0.91/0.87\n0.89/0.88\n
Depending on the approach, there can be a lot of bias. I think the more non-tissue elements of the image you include, the more bias. This may explain larger CV-LB differences for those using full-size image approaches on the lowest magnification.
", "votes": 5, "replies": [ { "id": 836459, "author_name": "Y.Nakama", "author_url": "", "post_date": "2020-05-07T02:22:00.073000", "content": "Great result. Thanks for the information!
", "votes": 2, "replies": [] } ] }, { "id": 834204, "author_name": "Artur Fattakhov (MIPT DIHT)", "author_url": "", "post_date": "2020-05-05T11:42:24.513000", "content": "Single fold resnet50: 0.860 CV, 0.84 LB\nSingle fold effb4: 0.862 CV, 0.85 LB
", "votes": 5, "replies": [] }, { "id": 818752, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2020-04-24T05:21:04.537000", "content": "model: resnet50 regression\nimg_size: 256*256\nsingle fold cv: 0.718\nsingle fold lb: 0.65
\n\nupdate:\nmodel: seresnext50 classification\nimg_size: 512*512\nsingle fold cv: 0.75\nlb: 0.73 (w/ TTA, w/ label smoothing)
", "votes": 6, "replies": [] }, { "id": 845708, "author_name": "tattaka", "author_url": "", "post_date": "2020-05-13T11:34:17.290000", "content": "5 Fold CV/LB\n\nresnet34 classification 0.80/0.78\nresnet34 regression 0.81/0.76\n\nI let each model learn 50 epochs.\nI'm using @Iafoss 's tile method by level 2 resolution.\nAlthough there are still few experiments, it seems that the gap between CV and LB is larger in regression.
Hi. In the case of regression, if you are using qwk optimised coefficients, which fold coefs do you use when making inference with 5 fold. Or you use coefs found in individual fold to make test set predictions with individual fold and then take the mode of the predictions for each test sample?
", "votes": 0, "replies": [] }, { "id": 853159, "author_name": "tattaka", "author_url": "", "post_date": "2020-05-19T02:17:23", "content": "I tried both the method of optimizing the coefficients after averaging the regression values, and the method of voting on the results of applying the coefficients in each fold.\nAs a result, the latter was slightly better in both CV and LB, so I used the latter.
", "votes": 0, "replies": [] } ] }, { "id": 841876, "author_name": "seefun", "author_url": "", "post_date": "2020-05-11T04:38:39.190000", "content": "Single Fold: 0.85 | LB: 0.81 | LB with TTA: 0.81 | LB with 5 fold blending: 0.81 | LB with 5 fold blending and TTA: 0.81
", "votes": 3, "replies": [] }, { "id": 835567, "author_name": "RabotniKuma", "author_url": "", "post_date": "2020-05-06T10:57:13.503000", "content": "seresnext50, regression, single fold: CV 0.815 / LB 0.81
\n\nUPDATED:\nseresnext50, classification, single fold: CV 0.865 / LB 0.87
", "votes": 3, "replies": [ { "id": 856635, "author_name": "Jeremy Berros", "author_url": "", "post_date": "2020-05-21T23:53:40.367000", "content": "Great results @analokamus. I have the same configuration but can't get over 0.79 after 40 epochs for some reason. What Loss/Metric did you use? Any preprocessing trick that helped your CV/LB? π
", "votes": 1, "replies": [] } ] }, { "id": 831041, "author_name": "Tsai29", "author_url": "", "post_date": "2020-05-03T04:03:53.093000", "content": "EfficientNetB0 segmentation model - single fold(0.8:0.2)\n\ncv : 0.44\nLB : 0.53\n
EfficientNet-B0\n\n- 4 folds (75:25 splits)\n- CV: 0.83\n- LB: 0.83\n
seresnext50, regression\ncv: 0.815, lb: 0.60
", "votes": 3, "replies": [] }, { "id": 828631, "author_name": "hirune924", "author_url": "", "post_date": "2020-05-01T07:40:19.850000", "content": "seresnet50, regression, 5fold avg\ncv: 0.83+ lb:0.77\n\ndf = pd.read_csv(os.path.join(data_dir,'train.csv'))\nskf = StratifiedKFold(n_splits=5, shuffle = True, random_state = 2020)\nfor fold, (train_index, val_index) in enumerate(skf.split(df.values, df['isup_grade'])):\n df.loc[val_index, 'fold'] = int(fold)\n
I met a weird cv/lb gap...\n- dataset: PANDA: Level 1 and 2 images\n- model: efficientnet b0\n- split: random split 0.8:0.2\n- fold: single fold\n- augment: flip and rotate\n- normalize: imagenet mean and std\n- image-size: 300 x 300 each tile ( use 5 tiles from level-1(intermediate) image, process like iafoss's )\n- epochs(the same model, as I train it 5+5 epochs, so I have a result of 5, and another of 10)\n - 5 cv: 0.7672814726829529 lb: 0.79\n - 10 cv: 0.7833058834075928 lb 0.75
\n\nPS: the tile size 300 is from my observation of visualizing the tiles after cutting by my eyes. I only compare 224 and 300 because the input-size of b0 is 224... And 300 has more raw-data pixels for just 5 tiles, so I choose 300. I visualize 16 tiles, as I ready to seam 16 tile into a image(4x4) like PANDA / se_resnext50 regression baseline did.\n\nafter seaming, cv2.vconcat(cv2.hconcat...)\n
As we can see the 300 has more blank, so I perfer to feed 224 to model.
", "votes": 1, "replies": [ { "id": 863783, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-05-27T15:02:06.943000", "content": "Same thing happens for me when i train for more epochs my CV can increase but LB decrease, but many people see a correlation between Karolinska CV with LB score (if im not wrong)
", "votes": 0, "replies": [] }, { "id": 864336, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-05-28T00:34:15.877000", "content": "Yes, you are right and it also happens to me.\nAs I split the data by provider, karolinska's score is similar to lb.\nAnd if I train more epochs(just from 6 to 10), the loss of train decrease but valid loss is stagnant(and cv increase, lb decrease).. loss(8,9,10 epoch) like below, my loss is crossentropy\n
So I plan to add more augmentations and use deeper backbone. What do you think about it?
", "votes": 0, "replies": [] }, { "id": 864365, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-05-28T00:55:38.377000", "content": "I think you should try iafoss method of concat pooling and yes add a little more augmentations or regularization methods! Also i think having more tiles of smaller size is a little better
", "votes": 1, "replies": [] }, { "id": 864381, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-05-28T01:08:49.967000", "content": "Thanks for your fast reply! I use GeM pool as mentioned using lafoss minimum model and how to improve by DrHB. And actually the GeM pool give me the most consistent trend of train and valid loos(and best cv and lb) compared with avg/max/concat.\nI will try more tiles with smaller size! Thank you for your adviceπ
", "votes": 2, "replies": [] }, { "id": 864560, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-05-28T04:23:15.217000", "content": "Yes GeM pooling is what im using too, ive always had great results with this pooling method!
", "votes": 0, "replies": [] } ] }, { "id": 818549, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-04-24T01:02:47.557000", "content": "EDIT: Starting over with new method
\n\nSingle Fold Validation: .767\nLB: .75
\n\nSingle Fold Validation: .765\nLB: .78
\n\nVal: .776\nLB: .77
", "votes": 1, "replies": [] }, { "id": 817978, "author_name": "currypurin", "author_url": "", "post_date": "2020-04-23T15:14:58.587000", "content": "updated
\n\n|CV|LB|\n| --- | --- |\n|0.687|0.62|\n|0.692|0.62|\n|0.698|0.62|\n|0.654|0.64|\n|0.654|0.57|
\n\n4fold, se_resnext50 classification
", "votes": 1, "replies": [ { "id": 818069, "author_name": "Y.Nakama", "author_url": "", "post_date": "2020-04-23T16:02:55.120000", "content": "Single fold score or Cross Validation score?
", "votes": 2, "replies": [] }, { "id": 818519, "author_name": "currypurin", "author_url": "", "post_date": "2020-04-24T00:22:15.683000", "content": "4fold score.\nMy code is based on your baseline notebook. Thank you so much!
", "votes": 1, "replies": [] }, { "id": 819998, "author_name": "currypurin", "author_url": "", "post_date": "2020-04-25T04:44:45.740000", "content": "I've done 5 submits so far and CV and LB don't seem to correlate. I want to create a reliable validation set.
", "votes": 0, "replies": [] } ] }, { "id": 817479, "author_name": "0xFunky", "author_url": "", "post_date": "2020-04-23T07:33:43.470000", "content": "single efficientnet-b0 model \nOnly trained on 1000 training data\nCV: 0.38\nLB: 0.36
", "votes": 1, "replies": [] }, { "id": 816834, "author_name": "Loulou", "author_url": "", "post_date": "2020-04-22T16:33:59.483000", "content": "That's rather a good news
", "votes": 1, "replies": [] }, { "id": 822861, "author_name": "He", "author_url": "", "post_date": "2020-04-27T07:44:01.507000", "content": "model: se_resnext50 classification\nCV-0.77 | LB-0.60 | Single Fold | 30 epoch\nCV-0.69 | LB-0.69 | Single Fold | 5 epcoh\nupdata:\nCV-0.869 | LB-0.86 | Single Fold
", "votes": 2, "replies": [ { "id": 822940, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2020-04-27T09:10:58.647000", "content": "Hi, He, what's the img size do you use?
", "votes": 0, "replies": [] }, { "id": 822960, "author_name": "He", "author_url": "", "post_date": "2020-04-27T09:29:45.393000", "content": "512*512
", "votes": 1, "replies": [] }, { "id": 822962, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2020-04-27T09:30:53.277000", "content": "thank you
", "votes": 0, "replies": [] } ] }, { "id": 821142, "author_name": "ilovescience", "author_url": "", "post_date": "2020-04-26T00:09:13.373000", "content": "Se-ResNext50 single fold\nCV: 0.629154\nLB: 0.59, 0.63 w/ TTA, 0.67 w/ label smoothing
\n\nIn the spirit of public sharing here's the kernel
\n\nAlso, check out the precursor kernel over here
\n\nEDIT: new score with Label Smoothing
", "votes": 2, "replies": [] }, { "id": 819090, "author_name": "phalanx", "author_url": "", "post_date": "2020-04-24T10:52:10.283000", "content": "resnet + fastfcn\ncv: 0.83, lb: 0.50
", "votes": 2, "replies": [ { "id": 819852, "author_name": "Matt", "author_url": "", "post_date": "2020-04-24T23:50:40.553000", "content": "Hm, any idea why there's such a large difference between CV & LB for your model?
", "votes": 0, "replies": [] }, { "id": 821578, "author_name": "Tsai29", "author_url": "", "post_date": "2020-04-26T09:08:20.190000", "content": "Hi @phalanx if you dont mind, do you also use classification model? Or segmentation model?
", "votes": 0, "replies": [] } ] }, { "id": 861520, "author_name": "tattaka", "author_url": "", "post_date": "2020-05-26T04:59:14.313000", "content": "I've been experimenting with low resolution since the tile method was published. However, the LB score cannot exceed 0.79 (and the CV score improve, but the LB is not correlated). What's the best CV/LB with low resolution?
", "votes": 0, "replies": [] }, { "id": 857584, "author_name": "Jeremy Berros", "author_url": "", "post_date": "2020-05-22T18:11:58.290000", "content": "Any idea for improvement is welcome π πͺ
\n\nFramework:
\n\nSingle fold:
\n\nX_train, X_val = train_test_split(train, test_size=.2, stratify=train['isup_grade'], random_state=SEED)\nPre processing:
\n\nConfig:
\n\nLR: 1e-3 \nBS: 16\nEpoch: 40\nModel: Seresnext50 backbone
\nClassification:
loss='categorical_crossentropy'\noptimizer=optimizers.Adam(lr=LR)\nmetrics=[qw_kappa_score]\nCallback= ReduceLROnPlateau
CV: 0.78\nLB: 0.79\nSingle Fold, cv/lb:\nse_resnext regression: 0.8233/0.78\nresnet34 regression 0.772/0.75
", "votes": 0, "replies": [] }, { "id": 835310, "author_name": "abzaliev", "author_url": "", "post_date": "2020-05-06T07:21:53.100000", "content": "mine is 0.81 CV and 0.82 LB, single fold resnet18
", "votes": 0, "replies": [] }, { "id": 825803, "author_name": "Satwik", "author_url": "", "post_date": "2020-04-29T09:00:45.270000", "content": "Model - seresnext50 imagenet pretrained\nimg size - 384x384\nCV - 0.63\nLB - 0.66
", "votes": 0, "replies": [] }, { "id": 822836, "author_name": "", "author_url": "", "post_date": "2020-04-27T07:16:08.940000", "content": "", "votes": 0, "replies": [] }, { "id": 820280, "author_name": "", "author_url": "", "post_date": "2020-04-25T09:45:14.720000", "content": "", "votes": 0, "replies": [] }, { "id": 818508, "author_name": "", "author_url": "", "post_date": "2020-04-24T00:07:03.217000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "816821": "model: se_resnext50 classification (no use of masks yet)\nimage size: 256x256\nCV: 0.6279 (Fold0: 0.6714 / Fold1: 0.6171 / Fold2: 0.6407 / Fold3: 0.6420 / Fold4: 0.6573)\nLB: 0.62x\nNot so much difference between CV & LB in my case\n\n[update]\nCV: 0.7771 (fold0: 0.8132 / fold1: 0.7877 / fold2: 0.8143 / fold3: 0.8027 / fold4: 0.8057)\nLB: 0.76x\n\nWhy some have much difference, others have not so much difference?", "841449": "Finally organized information on my more recent submissions, here is some info:\n\n```\n CV LB delta comment \n----- ---- ------ ------- \n0.710 0.72 +0.010 level2 10 epochs resnet34 \n0.726 0.71 -0.016 rerun of the same? level2 \n0.788 0.74 -0.048 level1 scale 0.5 (so 2x smaller) \n0.801 0.81 +0.009 level1 scale 1 resnet18 \n0.820 0.83 +0.010 level1 scale 0.875 resnet34 \n0.832 0.81 -0.022 level1 scale 0.5: switched to regression \n0.856 0.82 -0.036 same with more aug + better loss \n0.871 0.85 -0.021 level1 scale 1, 20 epochs \n0.842 0.73 -0.112 level2 80 epochs \n0.806 0.75 -0.056 level2 20 epochs\n```\n\nAll submissions from one fold, without TTA. Using a stratified 5-fold split (except for the first few submissions).\n\nSome observations:\n\n- as others noticed, different levels might behave in a different way on CV/LB\n- it seems that switching to regression made LB worse but CV better, although I'm not sure (could be just different scale)\n- training for longer might improve CV a lot but make LB worse", "834999": "Just a starter baseline: \n```\n\nmodel: resnset50 (pertained: Imagenet) \nFold: 1\nsplit: random (80/20)\nepoch: 5\ntype: classifcation\nloss_valid: 0.815392\nqwk_valid: 0.837932\nlb: 0.81\naugment: rotate(20), brightens and contrast\n\n```\n\nedit : \nmy submission history, had to fight with kernels =);\n\n", "831776": "Single fold resnet18: 0.80 CV, 0.81 LB\nSingle fold resnet34: 0.82 CV, 0.83 LB", "819896": "Regression model as [APTOS competition](https://www.kaggle.com/c/aptos2019-blindness-detection) , seresnext_50 with 5 folds, image size: 512 from [here](https://www.kaggle.com/xhlulu/panda-resized-train-data-512x512), cv with optimized threshold 0.768 and lb with optimized threshold 0.72; cv with fixed threshold 0.757 and lb with fixed threshold 0.71.", "829989": "single fold seresnext50\n```\ncv: 0.92\nlb: 0.80\nsplit: 80:20 \npretrained weights: imagenet\n```\n\n", "852655": "single fold model. Much of the code follows @lafoss work\nsee\n - https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb\n - https://www.kaggle.com/hengck23/kernel16867b0575\n \n\n```\nresnet34, 20 384x384 patches per image at 0.5 scale (i.e. use highest resolution of tiff image and apply scale of 0.5)\n\npublic LB 0.85\n\nvalidation:\naugment = ['null', 'all_flip']\nkapp = 0.844811\naccuracy_top2 = 0.710156, 0.894637\nlog_loss = 0.775169\n\n\n```\n", "836136": "My results from 5-fold CV:\n\n```\nCV/LB:\n0.88/0.87\n0.91/0.87\n0.89/0.88\n```\n\nDepending on the approach, there can be a lot of bias. I think the more non-tissue elements of the image you include, the more bias. This may explain larger CV-LB differences for those using full-size image approaches on the lowest magnification. ", "834204": "Single fold resnet50: 0.860 CV, 0.84 LB\nSingle fold effb4: 0.862 CV, 0.85 LB", "818752": "model: resnet50 regression\nimg_size: 256*256\nsingle fold cv: 0.718\nsingle fold lb: 0.65\n\nupdate:\nmodel: seresnext50 classification\nimg_size: 512*512\nsingle fold cv: 0.75\nlb: 0.73 (w/ TTA, w/ label smoothing)", "845708": "5 Fold CV/LB\n```\nresnet34 classification 0.80/0.78\nresnet34 regression 0.81/0.76\n```\nI let each model learn 50 epochs.\nI'm using @Iafoss 's tile method by level 2 resolution.\nAlthough there are still few experiments, it seems that the gap between CV and LB is larger in regression.", "841876": "Single Fold: 0.85 | LB: 0.81 | LB with TTA: 0.81 | LB with 5 fold blending: 0.81 | LB with 5 fold blending and TTA: 0.81", "835567": "seresnext50, regression, single fold: CV 0.815 / LB 0.81\n\nUPDATED:\nseresnext50, classification, single fold: CV 0.865 / LB 0.87", "831041": "EfficientNetB0 segmentation model - single fold(0.8:0.2)\n```\ncv : 0.44\nLB : 0.53\n```", "830103": "EfficientNet-B0\n```\n- 4 folds (75:25 splits)\n- CV: 0.83\n- LB: 0.83\n```\n", "827920": "seresnext50, regression\ncv: 0.815, lb: 0.60", "828631": "seresnet50, regression, 5fold avg\ncv: 0.83+ lb:0.77\n```\ndf = pd.read_csv(os.path.join(data_dir,'train.csv'))\nskf = StratifiedKFold(n_splits=5, shuffle = True, random_state = 2020)\nfor fold, (train_index, val_index) in enumerate(skf.split(df.values, df['isup_grade'])):\n df.loc[val_index, 'fold'] = int(fold)\n```", "863537": "I met a weird cv/lb gap...\n- dataset: [PANDA: Level 1 and 2 images](https://www.kaggle.com/lopuhin/panda-2020-level-1-2)\n- model: efficientnet b0\n- split: random split 0.8:0.2\n- fold: single fold\n- augment: flip and rotate\n- normalize: imagenet mean and std\n- image-size: 300 x 300 each tile ( use 5 tiles from level-1(intermediate) image, process like iafoss's )\n- epochs(the same model, as I train it 5+5 epochs, so I have a result of 5, and another of 10)\n - 5 cv: 0.7672814726829529 lb: 0.79\n - 10 cv: 0.7833058834075928 lb 0.75\n\nPS: the tile size 300 is from my observation of visualizing the tiles after cutting by my eyes. I only compare 224 and 300 because the input-size of b0 is 224... And 300 has more raw-data pixels for just 5 tiles, so I choose 300. I visualize 16 tiles, as I ready to seam 16 tile into a image(4x4) like [PANDA / se_resnext50 regression baseline](https://www.kaggle.com/yasufuminakama/panda-se-resnext50-regression-baseline) did.\n\nafter seaming, cv2.vconcat(cv2.hconcat...)\n\n\nAs we can see the 300 has more blank, so I perfer to feed 224 to model.", "818549": "EDIT: Starting over with new method\n\nSingle Fold Validation: .767\nLB: .75\n\nSingle Fold Validation: .765\nLB: .78\n\nVal: .776\nLB: .77", "817978": "updated\n\n|CV|LB|\n| --- | --- |\n|0.687|0.62|\n|0.692|0.62|\n|0.698|0.62|\n|0.654|0.64|\n|0.654|0.57|\n\n4fold, se_resnext50 classification", "817479": "single efficientnet-b0 model \nOnly trained on 1000 training data\nCV: 0.38\nLB: 0.36", "816834": "That's rather a good news", "822861": "model: se_resnext50 classification\nCV-0.77 | LB-0.60 | Single Fold | 30 epoch\nCV-0.69 | LB-0.69 | Single Fold | 5 epcoh\nupdata:\nCV-0.869 | LB-0.86 | Single Fold ", "821142": "Se-ResNext50 single fold\nCV: 0.629154\nLB: 0.59, 0.63 w/ TTA, 0.67 w/ label smoothing\n\nIn the spirit of public sharing here's the [kernel](https://www.kaggle.com/tanlikesmath/prostate-cancer-grading-intro-fastai2-starter)\n\nAlso, check out the precursor kernel over [here](https://www.kaggle.com/tanlikesmath/fastai2-training-baseline)\n\nEDIT: new score with Label Smoothing", "819090": "resnet + fastfcn\ncv: 0.83, lb: 0.50", "861520": "I've been experimenting with low resolution since the tile method was published. However, the LB score cannot exceed 0.79 (and the CV score improve, but the LB is not correlated). What's the best CV/LB with low resolution?", "857584": "Any idea for improvement is welcome π πͺ \n\nFramework:\n\n- Tensorflow, Keras \n\nSingle fold:\n\n X_train, X_val = train_test_split(train, test_size=.2, stratify=train['isup_grade'], random_state=SEED)\n\nPre processing: \n\n- 4X4 tiled images (384, 384, 3)\n- Augmentations (Hor/Ver Flip, ShiftScaleRotate)\n\nConfig:\n\n LR: 1e-3 \n BS: 16\n Epoch: 40\n\nModel: Seresnext50 backbone