Just a running thread of your best single model LB score. I will update weekly. Please share! \n
Sept 25: EfficientNet B1 224x224 - 0.098\nSept 26: EfficientNet B0 224x224 - 0.093\nOct 24: EfficientNet B0 512x512- 0.076
", "messages": [ { "id": 634193, "postDate": "2019-09-26T02:40:36.080Z", "content": "Just a running thread of your best single model LB score. I will update weekly. Please share! \n
Sept 25: EfficientNet B1 224x224 - 0.098\nSept 26: EfficientNet B0 224x224 - 0.093\nOct 24: EfficientNet B0 512x512- 0.076
", "rawMarkdown": "Just a running thread of your best single model LB score. I will update weekly. Please share! \nSingle-fold SE ResNext50, 512x512 raw HU image: lb 0.066 w/o tta
", "rawMarkdown": "Single-fold SE ResNext50, 512x512 raw HU image: lb 0.066 w/o tta", "votes": 16, "replies": [ { "id": 652087, "postDate": "2019-10-18T10:26:30.510Z", "content": "@andy2709 Nice result! You're not using any windows? How many epochs? I tried using raw dicom, converted to float and divided by max int16 value, but wasn't able to get improvements in score. I used much simpler architecture though.
", "rawMarkdown": "@andy2709 Nice result! You're not using any windows? How many epochs? I tried using raw dicom, converted to float and divided by max int16 value, but wasn't able to get improvements in score. I used much simpler architecture though." }, { "id": 652513, "postDate": "2019-10-19T00:45:18.473Z", "content": "I plugged in a trainable module (reference here: https://arxiv.org/pdf/1812.00572.pdf, the sigmoid one) to transform the raw hu into relevant windows.\n1 epoch took 45 mins on a v100 and i trained for 30 epochs. Data sampling is the key here :)
", "rawMarkdown": "I plugged in a trainable module (reference here: https://arxiv.org/pdf/1812.00572.pdf, the sigmoid one) to transform the raw hu into relevant windows.\n1 epoch took 45 mins on a v100 and i trained for 30 epochs. Data sampling is the key here :)", "votes": 9 }, { "id": 652813, "postDate": "2019-10-19T12:59:08.417Z", "content": "Really interesting idea thanks for sharing the link :)
", "rawMarkdown": "Really interesting idea thanks for sharing the link :) " }, { "id": 655738, "postDate": "2019-10-23T12:53:21.243Z", "content": "hello, I want to know how you used the trainable module, the Nan would occur when I added the module?
", "rawMarkdown": "hello, I want to know how you used the trainable module, the Nan would occur when I added the module? " }, { "id": 656392, "postDate": "2019-10-24T07:46:39.317Z", "content": "Interesting Paper. Thanks for sharing!\nOut of curiosity: what kind of windows did the Window Optimization layer learn to use in your case?
", "rawMarkdown": "Interesting Paper. Thanks for sharing!\nOut of curiosity: what kind of windows did the Window Optimization layer learn to use in your case?" }, { "id": 656763, "postDate": "2019-10-24T15:38:31.433Z", "rawMarkdown": "", "isDeleted": true }, { "id": 656776, "postDate": "2019-10-24T15:55:52.120Z", "content": "@lw827380444 I transformed the DICOM arrays into HU images using the rescale intercept and slope before passing it into the module. I never encountered nan when training with different architectures.
\n\n@ambpro Here's my implementation, with different weight init. from the original paper's repo. I inserted this module right before any pretrained ImageNet model and it worked pretty well for me.
\n\n```\nfrom collections import OrderedDict\nimport math \nimport torch \nimport torch.nn as nn
\n\ndef get_init_conv_params_sigmoid(ww, wl, smooth=1., upbound_value=255.):\n w = 2./ww * math.log(upbound_value/smooth - 1.)\n b = -2.*wl/ww * math.log(upbound_value/smooth - 1.)\n return (w, b)\n\n\nclass WSO(nn.Module):\n def init(self, windows=OrderedDict({\n 'brain': {'W': 80, 'L': 40},\n 'subdural': {'W': 215, 'L': 75},\n 'bony': {'W': 2800, 'L': 600},\n 'tissue': {'W': 375, 'L': 40},\n }), U=255., eps=1.):\n super(WSO, self).init()\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)\n self.bn = nn.BatchNorm2d(len(windows)\n nn.init.ones_(self.conv1x1.weight.data)\n nn.init.zeros_(self.conv1x1.bias.data)\n nn.init.ones_(self.bn.weight.data)\n nn.init.zeros_(self.bn.bias.data)\n weight, bias = self._get_window_params()\n self.register_buffer(\"weight\", weight)\n self.register_buffer(\"bias\", bias)
def _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\ndef forward(self, x):\n x = self.conv1x1(x)\n if x.dtype == torch.float16:\n self.weight = self.weight.half()\n self.bias = self.bias.half()\n x = x.mul(self.weight[None, :, None, None]) + self.bias[None, :, None, None]\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n x = self.bn(x)\n return x\n```
", "rawMarkdown": "@lw827380444 I transformed the DICOM arrays into HU images using the rescale intercept and slope before passing it into the module. I never encountered nan when training with different architectures.\n\n@ambpro Here's my implementation, with different weight init. from the original paper's repo. I inserted this module right before any pretrained ImageNet model and it worked pretty well for me.\n\n```\nfrom collections import OrderedDict\nimport math \nimport torch \nimport torch.nn as nn \n\ndef get_init_conv_params_sigmoid(ww, wl, smooth=1., upbound_value=255.):\n w = 2./ww * math.log(upbound_value/smooth - 1.)\n b = -2.*wl/ww * math.log(upbound_value/smooth - 1.)\n return (w, b)\n```\n```\nclass WSO(nn.Module):\n def __init__(self, `windows=OrderedDict({\n 'brain': {'W': 80, 'L': 40},\n 'subdural': {'W': 215, 'L': 75},\n 'bony': {'W': 2800, 'L': 600},\n 'tissue': {'W': 375, 'L': 40},\n })`, U=255., eps=1.):\n super(WSO, self).__init__()\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)\n self.bn = nn.BatchNorm2d(len(windows)\n nn.init.ones_(self.conv1x1.weight.data)\n nn.init.zeros_(self.conv1x1.bias.data)\n nn.init.ones_(self.bn.weight.data)\n nn.init.zeros_(self.bn.bias.data)\n weight, bias = self._get_window_params()\n self.register_buffer(\"weight\", weight)\n self.register_buffer(\"bias\", bias)\n\n def _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\n def forward(self, x):\n x = self.conv1x1(x)\n if x.dtype == torch.float16:\n self.weight = self.weight.half()\n self.bias = self.bias.half()\n x = x.mul(self.weight[None, :, None, None]) + self.bias[None, :, None, None]\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n x = self.bn(x)\n return x\n```", "votes": 13 }, { "id": 657763, "postDate": "2019-10-25T12:13:42.737Z", "content": "Sorry, I had a question, that is there are some differences between your implementation and original codes in keras? Maybe, the weight and bias in WSO should be from conv1x1, but you have set the new weight and bias? please give me some instructions if possible? Thanks for your sharing!
", "rawMarkdown": "Sorry, I had a question, that is there are some differences between your implementation and original codes in keras? Maybe, the weight and bias in WSO should be from conv1x1, but you have set the new weight and bias? please give me some instructions if possible? Thanks for your sharing!" }, { "id": 659765, "postDate": "2019-10-28T08:35:45.457Z", "content": "I think it's equivalent, slightly less efficient as both 1x1 conv and additional linear transformation used. It's also possible to directly initialize 1x1 conv with similar result.
", "rawMarkdown": "I think it's equivalent, slightly less efficient as both 1x1 conv and additional linear transformation used. It's also possible to directly initialize 1x1 conv with similar result.", "votes": 1 }, { "id": 660142, "postDate": "2019-10-28T18:45:51.387Z", "content": "To echo what @dmytropoplavskiy said , I have mistakenly applied linear transformation twice. Thank you guys for helping me correct it 😄. \nIn the new code, I just initialized the weight and bias of the conv1x1 with the values obtained from equation (2) in the paper:
\n\n```\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)
\n\n weight, bias = self._get_window_params()\n self.conv1x1.weight.data = weight[:, None, None, None]\n self.conv1x1.bias.data = bias\n\ndef _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\ndef forward(self, x):\n x = self.conv1x1(x)\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n return x\n```
\n\nI re-ran two experiments with the old and new WSO module (same training configs, ResNet18 etc) and observed that the results were similar (difference of 1e-4).
", "rawMarkdown": "To echo what @dmytropoplavskiy said , I have mistakenly applied linear transformation twice. Thank you guys for helping me correct it 😄. \nIn the new code, I just initialized the weight and bias of the conv1x1 with the values obtained from equation (2) in the paper:\n\n```\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)\n \n weight, bias = self._get_window_params()\n self.conv1x1.weight.data = weight[:, None, None, None]\n self.conv1x1.bias.data = bias\n\n def _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\n def forward(self, x):\n x = self.conv1x1(x)\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n return x\n```\n\nI re-ran two experiments with the old and new WSO module (same training configs, ResNet18 etc) and observed that the results were similar (difference of 1e-4).", "votes": 3 }, { "id": 662726, "postDate": "2019-10-31T23:11:50.810Z", "content": "Hi, I see the WSO model generate in output a [4,512,512] but the net you use (SE ResNext50) require in input [3,512,512] how did you connect the two ?
", "rawMarkdown": "Hi, I see the WSO model generate in output a [4,512,512] but the net you use (SE ResNext50) require in input [3,512,512] how did you connect the two ?", "votes": 1 } ] }, { "id": 659411, "postDate": "2019-10-27T16:02:06.410Z", "content": "Resnet34 with modifications, 256 image size, 0.062 LB
", "rawMarkdown": "Resnet34 with modifications, 256 image size, 0.062 LB", "votes": 9, "replies": [ { "id": 659431, "postDate": "2019-10-27T16:57:20.930Z", "content": "Single fold?
", "rawMarkdown": "Single fold?" }, { "id": 659467, "postDate": "2019-10-27T18:04:00.593Z", "content": "That's a great score! What windowing, if any, you used, if you don't mind me asking?
", "rawMarkdown": "That's a great score! What windowing, if any, you used, if you don't mind me asking?" }, { "id": 659571, "postDate": "2019-10-27T23:11:16.423Z", "content": "@yaroshevskiy Great! How many epochs do you train? Do you use fine tuning or transfer learning directly? Single fold?
", "rawMarkdown": "@yaroshevskiy Great! How many epochs do you train? Do you use fine tuning or transfer learning directly? Single fold?" }, { "id": 659972, "postDate": "2019-10-28T14:08:21.060Z", "content": "not a single fold, imagenet pretrained, 20-30 epochs
", "rawMarkdown": "not a single fold, imagenet pretrained, 20-30 epochs" } ] }, { "id": 636257, "postDate": "2019-09-29T05:58:10.133Z", "content": "size 224x224, model: inceptionV3, single fold --> LB 0.079
", "rawMarkdown": "size 224x224, model: inceptionV3, single fold --> LB 0.079", "votes": 9, "replies": [ { "id": 636564, "postDate": "2019-09-29T19:30:15.190Z", "content": "Thanks for your sharing! How many epoch do you use for training!
", "rawMarkdown": "Thanks for your sharing! How many epoch do you use for training!" }, { "id": 636658, "postDate": "2019-09-30T01:58:41.140Z", "content": "I trained the network for 20 epochs with Adam optimizer and one cycle learning rate schedule.
", "rawMarkdown": "I trained the network for 20 epochs with Adam optimizer and one cycle learning rate schedule.", "votes": 3 }, { "id": 636664, "postDate": "2019-09-30T02:29:15.783Z", "content": "How long did it take for your model to go through 20 epochs?
", "rawMarkdown": "How long did it take for your model to go through 20 epochs?" }, { "id": 636666, "postDate": "2019-09-30T02:35:57.663Z", "content": "It took about 35 mins/epoch to run my code (written in Keras) on my local server. With the same idea + same hardware, you can reduce the time to 20 mins/epoch using PyTorch + mixed-precision training (Apex)
", "rawMarkdown": "It took about 35 mins/epoch to run my code (written in Keras) on my local server. With the same idea + same hardware, you can reduce the time to 20 mins/epoch using PyTorch + mixed-precision training (Apex)", "votes": 3 }, { "id": 636715, "postDate": "2019-09-30T05:07:39.787Z", "content": "@mathormad Thank you for your sharing! I have a question that did you face with overfitting problem! My model trained in all train dataset and when I trained it over 10 or more, my score just down.
", "rawMarkdown": "@mathormad Thank you for your sharing! I have a question that did you face with overfitting problem! My model trained in all train dataset and when I trained it over 10 or more, my score just down." }, { "id": 637601, "postDate": "2019-10-01T06:41:18.937Z", "content": "hi @linhlpv , to avoid overfitting you should split the data to training set and validation set, then monitor the loss function of the validation set, pick the best weight and predict on the test set.
", "rawMarkdown": "hi @linhlpv , to avoid overfitting you should split the data to training set and validation set, then monitor the loss function of the validation set, pick the best weight and predict on the test set.", "votes": 1 }, { "id": 638379, "postDate": "2019-10-01T20:58:35.947Z", "content": "May I ask what are u using as loss function and how do you make ur epoch run so fast? My best model at the moment (Xception) can run maximum 3 epochs, even though I only tried 2 epochs till now...
", "rawMarkdown": "May I ask what are u using as loss function and how do you make ur epoch run so fast? My best model at the moment (Xception) can run maximum 3 epochs, even though I only tried 2 epochs till now...", "votes": 2 }, { "id": 638857, "postDate": "2019-10-02T13:52:44.490Z", "content": "hi @mathormad Thank you for your suggestion. I see that you said your model just take 35 mins per epoch. How do you make your model run so fast? I have 4 GPUs 1080TI and I am using Resnet 50 and mixed-precision training for training in whole trainset and it takes me about 2 hours per epoch. Please help me! Thank you so much!
", "rawMarkdown": "hi @mathormad Thank you for your suggestion. I see that you said your model just take 35 mins per epoch. How do you make your model run so fast? I have 4 GPUs 1080TI and I am using Resnet 50 and mixed-precision training for training in whole trainset and it takes me about 2 hours per epoch. Please help me! Thank you so much!" }, { "id": 638922, "postDate": "2019-10-02T15:07:25.587Z", "content": "@linhlpv What image size are you using? Are you loading Dicom files?
", "rawMarkdown": "@linhlpv What image size are you using? Are you loading Dicom files?" }, { "id": 638997, "postDate": "2019-10-02T16:51:19.217Z", "content": "@drhabib I use 224x224 image size and image from png file. My code apply multi GPUs and Apex bellow:\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n model = nn.DataParallel(model)
How many epochs?
", "rawMarkdown": "How many epochs?" }, { "id": 639020, "postDate": "2019-10-02T17:16:02.963Z", "content": "@synysterjeet I just train 3 to 5 epoch. Because training one epoch took me 2 hours :(. I dont know how it so slow. I train in whole dataset
", "rawMarkdown": "@synysterjeet I just train 3 to 5 epoch. Because training one epoch took me 2 hours :(. I dont know how it so slow. I train in whole dataset" }, { "id": 639184, "postDate": "2019-10-02T21:14:07.240Z", "content": "@linhlpv , try these:\n- Check your CPU/GPU load. Can your CPUs load enough images to your GPU. \n- All (most) of your cores are working?!\n- Some of the transformations are slow, add them one by one.\n- Try a bigger batch size (increased accumulation steps).\n- Some preprocessing techniques usually help. (save as Numpy arrays instead of PNGs; int16 instead of float32)
", "rawMarkdown": "@linhlpv , try these:\n- Check your CPU/GPU load. Can your CPUs load enough images to your GPU. \n- All (most) of your cores are working?!\n- Some of the transformations are slow, add them one by one.\n- Try a bigger batch size (increased accumulation steps).\n- Some preprocessing techniques usually help. (save as Numpy arrays instead of PNGs; int16 instead of float32)\n", "votes": 5 }, { "id": 639298, "postDate": "2019-10-03T03:32:32.783Z", "content": "Hi @pestipeti, thank you for mention me! I will try it and update results.
", "rawMarkdown": "Hi @pestipeti, thank you for mention me! I will try it and update results." }, { "id": 639636, "postDate": "2019-10-03T12:25:22.800Z", "content": "Please enlighten us when you find which of these sugestions speed up the process.
", "rawMarkdown": "Please enlighten us when you find which of these sugestions speed up the process.", "votes": 1 }, { "id": 640803, "postDate": "2019-10-04T08:42:14.323Z", "content": "When I tried using more core CPUs in dataloader, I got my model run 2 faster than old version. But when I tried using bigger batch size, I got a problem that estimate time is not stable, it something was 1h30, sometime was 30 minutes. I dont know why and I am searching and try to fix that. Anyone has some ideas. Please tell my how to fix it.\nMy num_workers I used:\nnum_workers=os.cpu_count()*2
@linhlpv \nHard to say, but my bet is augmentation. It is random, and there are a few computationally heavy transformation. And if you use all of your cores for data loading, then every transformation has to wait for CPU. Try num_workers=os.cpu_count()*2 - 1 (or -2)\nProbably, you have to optimize your code a bit too. Especially when and how you move your data between the GPU and the CPU. Try some different settings, you will find the optimal, I am sure.
@pestipeti \nThank you for your suggestion! I will try more setting and optimize my code. And I will post result when I find something good for speed up code!
", "rawMarkdown": "@pestipeti \nThank you for your suggestion! I will try more setting and optimize my code. And I will post result when I find something good for speed up code!" } ] }, { "id": 634491, "postDate": "2019-09-26T11:13:11.197Z", "content": "Custom model -- 0.069
", "rawMarkdown": "Custom model -- 0.069 ", "votes": 8, "replies": [ { "id": 634521, "postDate": "2019-09-26T11:57:36.373Z", "content": "Just to be sure , this is a single model and not an ensemble?
", "rawMarkdown": "Just to be sure , this is a single model and not an ensemble?", "votes": 2 }, { "id": 634528, "postDate": "2019-09-26T12:10:34.063Z", "content": "single fold without any ensembling :)
", "rawMarkdown": "single fold without any ensembling :) ", "votes": 4 }, { "id": 634660, "postDate": "2019-09-26T15:13:14.367Z", "content": "is your custom pretrained on any external data or from scratch. Or you changed part of the pretrained model and called it custom
", "rawMarkdown": "is your custom pretrained on any external data or from scratch. Or you changed part of the pretrained model and called it custom", "votes": 1 }, { "id": 634668, "postDate": "2019-09-26T15:34:35.683Z", "content": "Why don't you just ask for the source code?
", "rawMarkdown": "Why don't you just ask for the source code?", "votes": -9 }, { "id": 634693, "postDate": "2019-09-26T16:15:35.677Z", "content": "@julianmukaj I find your comment rude. I asked a question and it's on @igorkrashenyi to decide whether to answer it.
\n\nAnswer to the question will not reveal his solution. Not answering it is also fine.
", "rawMarkdown": "@julianmukaj I find your comment rude. I asked a question and it's on @igorkrashenyi to decide whether to answer it. \n\nAnswer to the question will not reveal his solution. Not answering it is also fine. ", "votes": 4 }, { "id": 634799, "postDate": "2019-09-26T19:13:06.757Z", "content": "ImagNet pre-trained with modifications
", "rawMarkdown": "ImagNet pre-trained with modifications ", "votes": 4 }, { "id": 636011, "postDate": "2019-09-28T15:36:38.237Z", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "rawMarkdown": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ? ", "votes": 1 }, { "id": 636271, "postDate": "2019-09-29T06:51:13.253Z", "content": "512x512 resolution and the full dataset
", "rawMarkdown": "512x512 resolution and the full dataset", "votes": 4 }, { "id": 641886, "postDate": "2019-10-05T09:12:36.920Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 642728, "postDate": "2019-10-06T14:45:09.547Z", "content": "About 2.5 days using 4x2080ti and Apex
", "rawMarkdown": "About 2.5 days using 4x2080ti and Apex ", "votes": 3 }, { "id": 646043, "postDate": "2019-10-10T19:20:03.307Z", "content": "@igorkrashenyi Are you running Apex on windows? Running into installation issues for Apex. I have pytorch installed on windows and it can train no prob.
", "rawMarkdown": "@igorkrashenyi Are you running Apex on windows? Running into installation issues for Apex. I have pytorch installed on windows and it can train no prob." }, { "id": 647703, "postDate": "2019-10-13T05:39:20.960Z", "content": "I tried running Apex on Windows before and it wasn't fully supported by NVIDIA. Eventually I just resorted to Ubuntu
", "rawMarkdown": "I tried running Apex on Windows before and it wasn't fully supported by NVIDIA. Eventually I just resorted to Ubuntu", "votes": 1 } ] }, { "id": 646930, "postDate": "2019-10-11T22:30:14.243Z", "content": ".073 is achievable with b0 224x224. I grouped on patient and used some tricks, though
", "rawMarkdown": ".073 is achievable with b0 224x224. I grouped on patient and used some tricks, though", "votes": 7, "replies": [ { "id": 646946, "postDate": "2019-10-11T23:03:12.300Z", "content": "Have you solved local/LB correlation problem, by the way?
", "rawMarkdown": "Have you solved local/LB correlation problem, by the way?" }, { "id": 647044, "postDate": "2019-10-12T02:10:43.110Z", "content": "@cateek Solved it via grouping via patients. I think this is very important for establishing reliable validation score, though I have not done any control experiments. Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)
", "rawMarkdown": "@cateek Solved it via grouping via patients. I think this is very important for establishing reliable validation score, though I have not done any control experiments. Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)", "votes": 4 }, { "id": 648205, "postDate": "2019-10-13T21:33:21.767Z", "content": "Very interesting! I think I got it, I will test
", "rawMarkdown": "Very interesting! I think I got it, I will test\n" }, { "id": 648590, "postDate": "2019-10-14T11:51:22.537Z", "content": "thanks for sharing! What's your val loss function?
", "rawMarkdown": "thanks for sharing! What's your val loss function?" }, { "id": 648637, "postDate": "2019-10-14T12:39:41.087Z", "content": "Good work! Wouldn't it make sense for your LB score to be higher than validation if there are samples from the same patients in the test data set. Seems like you should just trust your validation
", "rawMarkdown": "Good work! Wouldn't it make sense for your LB score to be higher than validation if there are samples from the same patients in the test data set. Seems like you should just trust your validation" }, { "id": 648801, "postDate": "2019-10-14T16:01:54.920Z", "content": "@roguekk007 \"Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)\"\nIs this not comparing apples to oranges because you compare your validation loss to the LB metric?
", "rawMarkdown": "@roguekk007 \"Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)\"\nIs this not comparing apples to oranges because you compare your validation _loss_ to the LB _metric_?" }, { "id": 649073, "postDate": "2019-10-14T22:56:24.587Z", "content": "@micpie I think it is not comparing apples to oranges personally, because according to the LB probing, the weights of the metric is revealed, so my validation loss should be kind of equal to LB
", "rawMarkdown": "@micpie I think it is not comparing apples to oranges personally, because according to the LB probing, the weights of the metric is revealed, so my validation loss *should* be kind of equal to LB", "votes": 2 }, { "id": 649206, "postDate": "2019-10-15T04:16:14.073Z", "content": "@roguekk007 Thanks, you are right, with the correctly weighted BCE loss they are comparable. 👍
", "rawMarkdown": "@roguekk007 Thanks, you are right, with the correctly weighted BCE loss they are comparable. 👍 " }, { "id": 651546, "postDate": "2019-10-17T15:35:31.050Z", "content": "Hi! what do you mean by b0 ? I am unfamiliar with this term ! Edit: never mind I figured this out
", "rawMarkdown": "Hi! what do you mean by b0 ? I am unfamiliar with this term ! Edit: never mind I figured this out" }, { "id": 653864, "postDate": "2019-10-21T04:40:56.927Z", "content": "B0 is one type of the efficientnet architecture s ( there is b0, b1, ... B7)
", "rawMarkdown": "B0 is one type of the efficientnet architecture s ( there is b0, b1, ... B7)" }, { "id": 657621, "postDate": "2019-10-25T09:52:31.183Z", "content": "after the competition I look forward to how you did the \"probing\" that helped you figure out the weights !
", "rawMarkdown": "after the competition I look forward to how you did the \"probing\" that helped you figure out the weights !" } ] }, { "id": 636104, "postDate": "2019-09-28T20:18:40.787Z", "content": "EfficientNet-B5 @ 512 x 512 - 0.070
", "rawMarkdown": "EfficientNet-B5 @ 512 x 512 - 0.070", "votes": 7, "replies": [ { "id": 638059, "postDate": "2019-10-01T13:48:50.513Z", "content": "How many epochs are you running , if you don't mind me asking? And are the images in png or jpg?
", "rawMarkdown": "How many epochs are you running , if you don't mind me asking? And are the images in png or jpg?", "votes": 1 }, { "id": 638794, "postDate": "2019-10-02T12:43:09.850Z", "content": "100 epochs, 16,000 images per epoch, loaded from DICOM.
", "rawMarkdown": "100 epochs, 16,000 images per epoch, loaded from DICOM.", "votes": 4 }, { "id": 638918, "postDate": "2019-10-02T15:03:25.150Z", "content": "Wow, so how much time does it take for your model to train completely?
", "rawMarkdown": "Wow, so how much time does it take for your model to train completely?", "votes": 1 }, { "id": 639759, "postDate": "2019-10-03T14:11:23.907Z", "content": "About 60-65 hours.
", "rawMarkdown": "About 60-65 hours.", "votes": 3 }, { "id": 639908, "postDate": "2019-10-03T17:26:27.003Z", "content": "Hmm, I wonder how long EfficientNet B7 would take with those same parameters, 80-100 hours? How to use GCP credit wisely.
", "rawMarkdown": "Hmm, I wonder how long EfficientNet B7 would take with those same parameters, 80-100 hours? How to use GCP credit wisely.", "votes": 1 } ] }, { "id": 636131, "postDate": "2019-09-28T21:03:38.160Z", "content": "LB: 0.080
\n\n> MODEL: Efficient Net B0 \nSIZE: 224 (PNG)\nCV: 1 Fold\nAUG: [zoom, rotate]\nTTA: No\nPRETRAINED: True\nEPOCH: 20\nLR: 1e-3
", "rawMarkdown": "LB: 0.080\n\n> MODEL: Efficient Net B0 \nSIZE: 224 (PNG)\nCV: 1 Fold\nAUG: [zoom, rotate]\nTTA: No\nPRETRAINED: True\nEPOCH: 20\nLR: 1e-3\n\n", "votes": 8, "replies": [ { "id": 636150, "postDate": "2019-09-28T21:45:31.640Z", "content": "Thanks for sharing. How is your local validation compared to LB? Which loss func are you using? I'm getting 0.088 with 4 epochs at 1e-3 at B0 and no hold outset (yet)
", "rawMarkdown": "Thanks for sharing. How is your local validation compared to LB? Which loss func are you using? I'm getting 0.088 with 4 epochs at 1e-3 at B0 and no hold outset (yet)", "votes": 1 }, { "id": 636153, "postDate": "2019-09-28T22:06:41.523Z", "content": "I use for now torch.nn.BCEWithLogitsLoss() I did not implement yet the offical metric (I will update lter). But my val loss is almost identical to trn loss =)
I used the same settting as yours but in 4 epochs and LR=1.20E-3.\nTreating 'any' as a class to be predicted.\nLB: 0.113
", "rawMarkdown": "I used the same settting as yours but in 4 epochs and LR=1.20E-3.\nTreating 'any' as a class to be predicted.\nLB: 0.113", "votes": 1 }, { "id": 637722, "postDate": "2019-10-01T08:11:41.100Z", "content": "@drhabib May I ask if you are undersampling the training data? Compared to my training, 20 epochs are a lot.
", "rawMarkdown": "@drhabib May I ask if you are undersampling the training data? Compared to my training, 20 epochs are a lot. ", "votes": 1 }, { "id": 638056, "postDate": "2019-10-01T13:44:21.557Z", "content": "No under sampling... Just using random split =)
", "rawMarkdown": "No under sampling... Just using random split =) ", "votes": 1 }, { "id": 638380, "postDate": "2019-10-01T20:59:44.163Z", "content": "Could you elaborate further? I am having time issues as well and would like to understand better what I can do to overcome it.
", "rawMarkdown": "Could you elaborate further? I am having time issues as well and would like to understand better what I can do to overcome it.", "votes": 1 }, { "id": 638506, "postDate": "2019-10-02T01:21:28.240Z", "content": "I don't do any under sampling. I take whole data and split randomly 80% to train and 20% to validation. Let me know if you have other questions=)
", "rawMarkdown": "I don't do any under sampling. I take whole data and split randomly 80% to train and 20% to validation. Let me know if you have other questions=) ", "votes": 2 }, { "id": 638532, "postDate": "2019-10-02T02:36:02.357Z", "content": "Around how long does your model take to train?
", "rawMarkdown": "Around how long does your model take to train?", "votes": 1 }, { "id": 638866, "postDate": "2019-10-02T14:04:47.800Z", "content": "around 12
", "rawMarkdown": "around 12 " }, { "id": 638987, "postDate": "2019-10-02T16:44:35.370Z", "content": "Thanks for the sharing!!! Did you use any normalization(like ImageNet/competition dataset mean and std) or regularization?
", "rawMarkdown": "Thanks for the sharing!!! Did you use any normalization(like ImageNet/competition dataset mean and std) or regularization?", "votes": 1 }, { "id": 640949, "postDate": "2019-10-04T10:53:01.533Z", "content": "I used Imagenet stars :) Good luck
", "rawMarkdown": "I used Imagenet stars :) Good luck ", "votes": 1 }, { "id": 641000, "postDate": "2019-10-04T11:24:28.953Z", "content": "@drhabib What LB score do you mange to get with EfficentNetB0, 224x224, single fold, (no TTA perhaps)? I'm also finetuning on EfficientNetB0, and so far got 0.079, which I think is much worse than yours? ;-)
", "rawMarkdown": "@drhabib What LB score do you mange to get with EfficentNetB0, 224x224, single fold, (no TTA perhaps)? I'm also finetuning on EfficientNetB0, and so far got 0.079, which I think is much worse than yours? ;-)", "votes": 1 }, { "id": 643365, "postDate": "2019-10-07T13:15:52.823Z", "content": "@drhabib Is it convenient to disclose the train/val split that you are using? I observe very different difference between LB/CV scores using different seeds in sklearn's train_val_split. Anyone willing to disclose their seed and validation portion that they are using?
", "rawMarkdown": "@drhabib Is it convenient to disclose the train/val split that you are using? I observe very different difference between LB/CV scores using different seeds in sklearn's train_val_split. Anyone willing to disclose their seed and validation portion that they are using?", "votes": 1 }, { "id": 644422, "postDate": "2019-10-08T20:01:30.357Z", "content": "@akensert vanilla B0 0.70-0.72 depending on my GPU mood =)
", "rawMarkdown": "@akensert vanilla B0 0.70-0.72 depending on my GPU mood =)", "votes": 1 }, { "id": 644424, "postDate": "2019-10-08T20:04:11.937Z", "content": "@roguekk007 Are you using weighted loss ? If not that willl explain why you have a big difference. My LB and CV are almost identical or have very small difference. \nRegarding splitting. \nI did not observe huge difference if I split by patient or do StratifiedKFold.
are you using any special training procedure? lr scheduling? thanks
", "rawMarkdown": "are you using any special training procedure? lr scheduling? thanks", "votes": 1 }, { "id": 644668, "postDate": "2019-10-09T06:55:50.137Z", "content": "@drhabib Do you use Pytorch or keras?
", "rawMarkdown": "@drhabib Do you use Pytorch or keras? ", "votes": 1 }, { "id": 646224, "postDate": "2019-10-11T02:17:29.773Z", "content": "Verne, Nothing special, just cosine decay=)
", "rawMarkdown": "Verne, Nothing special, just cosine decay=)", "votes": 1 }, { "id": 646225, "postDate": "2019-10-11T02:18:47.673Z", "content": "@custodiogabriel I use Pytroch, but at the end of the day it doesn't matter which software you use to achieve top results =)
", "rawMarkdown": "@custodiogabriel I use Pytroch, but at the end of the day it doesn't matter which software you use to achieve top results =) ", "votes": 1 }, { "id": 647755, "postDate": "2019-10-13T08:19:38.617Z", "content": "@drhabib I’m currently getting .066 with b0 with minimal tuning. You must also be using some cool tricks to get b0 to .070? Will you consider teaming up?
", "rawMarkdown": "@drhabib I’m currently getting .066 with b0 with minimal tuning. You must also be using some cool tricks to get b0 to .070? Will you consider teaming up?" }, { "id": 648300, "postDate": "2019-10-14T02:27:06.087Z", "content": "Thanks for your sharing! How much batch size have you set?
", "rawMarkdown": "Thanks for your sharing! How much batch size have you set?" }, { "id": 648329, "postDate": "2019-10-14T03:22:48.360Z", "content": "48, max that can fit on my local RTX 2080 (not ti)
", "rawMarkdown": "48, max that can fit on my local RTX 2080 (not ti)" }, { "id": 649544, "postDate": "2019-10-15T14:11:13.827Z", "content": "Is your current score (0.061) from a single model?\nBTW, @roguekk007 @drhabib Perhaps, the best ensemble (team)👍 . Wish you all the best!
", "rawMarkdown": "Is your current score (0.061) from a single model?\nBTW, @roguekk007 @drhabib Perhaps, the best ensemble (team)👍 . Wish you all the best!" }, { "id": 650015, "postDate": "2019-10-16T02:26:16.577Z", "content": "Could you tell me what's the meaning of tta? Much Thanks
", "rawMarkdown": "Could you tell me what's the meaning of tta? Much Thanks" }, { "id": 650057, "postDate": "2019-10-16T03:47:55.933Z", "content": "@pupil3 Test Time Augmentaion (TTA)
", "rawMarkdown": "@pupil3 Test Time Augmentaion (TTA)" }, { "id": 650089, "postDate": "2019-10-16T04:58:03.967Z", "content": "Much Thanks
", "rawMarkdown": "Much Thanks" }, { "id": 650711, "postDate": "2019-10-16T16:03:45.270Z", "content": "Hi @drhabib
\n\nJust curious did you do window pre-processing? Or you use Jermey’s special preprocessing method?
\n\nThanks in advance :)
", "rawMarkdown": "Hi @drhabib \n\nJust curious did you do window pre-processing? Or you use Jermey’s special preprocessing method?\n\nThanks in advance :)", "votes": 1 } ] }, { "id": 652609, "postDate": "2019-10-19T05:43:05.577Z", "content": "0.069 - single fold, no tta, 300x300, 80:20 train/val split using the full dataset, custom model.
", "rawMarkdown": " 0.069 - single fold, no tta, 300x300, 80:20 train/val split using the full dataset, custom model.", "votes": 5, "replies": [ { "id": 653156, "postDate": "2019-10-20T02:32:23.860Z", "content": "If you allow me, did you use any data normalization like Multi channel windowing?
", "rawMarkdown": "If you allow me, did you use any data normalization like Multi channel windowing?\n" }, { "id": 653804, "postDate": "2019-10-21T01:18:24.550Z", "content": "Yea, I'm using Sigmoid (Brain + Subdural + Bone) Windowing from Gradient & Sigmoid Windowing.
", "rawMarkdown": "Yea, I'm using *Sigmoid (Brain + Subdural + Bone) Windowing* from [Gradient & Sigmoid Windowing](https://www.kaggle.com/reppic/gradient-sigmoid-windowing).", "votes": 2 }, { "id": 655230, "postDate": "2019-10-22T20:49:59.113Z", "content": "Did you use GroupKfold using PatientID? Many people are using... Do you think this is essential to increase the score?
", "rawMarkdown": "Did you use GroupKfold using PatientID? Many people are using... Do you think this is essential to increase the score?\n" }, { "id": 655377, "postDate": "2019-10-23T01:47:54.963Z", "content": "Yea, I haven't been doing multiple folds yet and I'm not using that function specifically, but I am splitting my train/validation sets using PatientId.
\n\nIt's not because splitting that way necessarily improves the score of a single model, but if you don't, you risk leaking info to your validation set and you can overfit without realizing it. Check out Similarity between train and test images
\n\nI imagine it also helps improve model diversity when ensembling models trained on different folds, but idk for sure.
", "rawMarkdown": "Yea, I haven't been doing multiple folds yet and I'm not using that function specifically, but I am splitting my train/validation sets using PatientId. \n\nIt's not because splitting that way necessarily improves the score of a single model, but if you don't, you risk leaking info to your validation set and you can overfit without realizing it. Check out [Similarity between train and test images](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/111729#latest-647909) \n\nI imagine it also helps improve model [diversity](https://ieeexplore.ieee.org/document/4371035) when ensembling models trained on different folds, but idk for sure.", "votes": 1 }, { "id": 655409, "postDate": "2019-10-23T02:47:31.580Z", "content": "I understand! Thanks for the feedback! My models usually have very low val_loss and very different from LB (larger), I think that if I do GroupKFold (or other techniques) I think I will have a much closer val_loss to LB then! I will see this kernel! Thanks!
", "rawMarkdown": "I understand! Thanks for the feedback! My models usually have very low val_loss and very different from LB (larger), I think that if I do GroupKFold (or other techniques) I think I will have a much closer val_loss to LB then! I will see this kernel! Thanks!", "votes": 1 }, { "id": 664619, "postDate": "2019-11-04T01:28:49.490Z", "content": "Update 11/3: \n0.066- single fold, no tta, 300x300, 80:20 train/val split using the full dataset (no CQ500 data), custom model and a custom data generator.
", "rawMarkdown": "Update 11/3: \n0.066- single fold, no tta, 300x300, 80:20 train/val split using the full dataset (no CQ500 data), custom model and a custom data generator.", "votes": 1 }, { "id": 664816, "postDate": "2019-11-04T09:13:32.470Z", "content": "By custom you meant you're not using imagenet weights, or it's just a custom head?
", "rawMarkdown": "By custom you meant you're not using imagenet weights, or it's just a custom head?" } ] }, { "id": 651109, "postDate": "2019-10-17T03:45:17.483Z", "content": "efficientnet-b2 \n256x256\nw/o 3 windowing preprocess\npublicLB: 0.069
", "rawMarkdown": "efficientnet-b2 \n256x256\nw/o 3 windowing preprocess\npublicLB: 0.069", "votes": 5, "replies": [ { "id": 651114, "postDate": "2019-10-17T04:03:02.377Z", "content": "Cool!
\nWhat is the meaning of w/o?
I think he means without
", "rawMarkdown": "I think he means without" }, { "id": 651143, "postDate": "2019-10-17T05:04:27.393Z", "content": "So he didn't use windowing?
", "rawMarkdown": "So he didn't use windowing?" }, { "id": 651355, "postDate": "2019-10-17T11:48:22.430Z", "content": "I used this kernel's windowing.\nI'm trying Appian's windowing now.
", "rawMarkdown": "I used [this kernel's](https://www.kaggle.com/omission/eda-view-dicom-images-with-correct-windowing) windowing.\nI'm trying Appian's windowing now." }, { "id": 651419, "postDate": "2019-10-17T13:28:11.613Z", "content": "@uiiurz1 That's a great result! Did you normalize input images?
", "rawMarkdown": "@uiiurz1 That's a great result! Did you normalize input images? " }, { "id": 651430, "postDate": "2019-10-17T13:34:40.040Z", "content": "I normalized input images with ImageNet's means and stds.
", "rawMarkdown": "I normalized input images with ImageNet's means and stds.", "votes": 1 }, { "id": 653160, "postDate": "2019-10-20T02:36:08.850Z", "content": "@uiiurz1 how many epochs?
", "rawMarkdown": "@uiiurz1 how many epochs?" } ] }, { "id": 651127, "postDate": "2019-10-17T04:26:00.700Z", "content": "EfficientNet-B0\n224x224\nPublic LB: 0.069 without TTA
", "rawMarkdown": "EfficientNet-B0\n224x224\nPublic LB: 0.069 without TTA", "votes": 6, "replies": [ { "id": 651192, "postDate": "2019-10-17T06:56:53.323Z", "content": "That's a great result with such a simple setup! May I wonder, did you use some special windowing? Have you used all images, or found a way to remove uninformative ones? I've done quite a lot of experiments and already decided that it's simply impossible for such to go below 0.75 with small resolution w/o some tricks. Thanks in advance
", "rawMarkdown": "That's a great result with such a simple setup! May I wonder, did you use some special windowing? Have you used all images, or found a way to remove uninformative ones? I've done quite a lot of experiments and already decided that it's simply impossible for such to go below 0.75 with small resolution w/o some tricks. Thanks in advance" }, { "id": 651199, "postDate": "2019-10-17T07:05:14.850Z", "content": "try undersampling
", "rawMarkdown": "try undersampling" }, { "id": 651206, "postDate": "2019-10-17T07:13:23.463Z", "content": "I'm using all of the images and averaging 5 folds. Nothing fancy with the network - my parameters are identical to DrHBs in earlier this thread. The image preprocessing is the key, but I think it's best if I share later 😉
", "rawMarkdown": "I'm using all of the images and averaging 5 folds. Nothing fancy with the network - my parameters are identical to DrHBs in earlier this thread. The image preprocessing is the key, but I think it's best if I share later 😉 ", "votes": 3 }, { "id": 651211, "postDate": "2019-10-17T07:16:01.363Z", "content": "Thanks! By preprocessing you mean window/no window, window parameters, and normalization, right?
", "rawMarkdown": "Thanks! By preprocessing you mean window/no window, window parameters, and normalization, right? " }, { "id": 651215, "postDate": "2019-10-17T07:21:44.177Z", "content": "@anjum48 coulld you please describe about 5 folds , how you are doing K-Fold . Thanks
", "rawMarkdown": "@anjum48 coulld you please describe about 5 folds , how you are doing K-Fold . Thanks\n" }, { "id": 651217, "postDate": "2019-10-17T07:25:00.843Z", "content": "@cateek Yes, but there are more ways than just windowing to make sure the images you pass to the model are as informative as possible\n@rajnishe I'm using GroupKfold using PatientID as the grouping variable
Interesting!
\nI am looking forward to see your preprocessing after competition end;)
One more thing. Do you treat any class in some specific way or just 6 classes with weights? I've tried some variation of 2-stage model but it didn't work well
", "rawMarkdown": "One more thing. Do you treat any class in some specific way or just 6 classes with weights? I've tried some variation of 2-stage model but it didn't work well" }, { "id": 651306, "postDate": "2019-10-17T09:56:00.483Z", "content": "No, I'm using the standard 6 class loss already shared here. I've not had a chance to test the 2-step pipeline yet
", "rawMarkdown": "No, I'm using the standard 6 class loss already shared here. I've not had a chance to test the 2-step pipeline yet", "votes": 1 }, { "id": 651335, "postDate": "2019-10-17T10:51:03.737Z", "content": "@anjum48 \nI'm using GroupKfold using PatientID as the grouping variable
Are CV score and LB score correlated?\nin fact, my CV score with GroupKfold using PatientID doesnt help to early stopping...
", "rawMarkdown": "@anjum48 \n```I'm using GroupKfold using PatientID as the grouping variable``` \n\nAre CV score and LB score correlated?\nin fact, my CV score with GroupKfold using PatientID doesnt help to early stopping..." } ] }, { "id": 634541, "postDate": "2019-09-26T12:29:04.213Z", "content": "inceptionv4 -> 0.78
", "rawMarkdown": "inceptionv4 -> 0.78", "votes": 5, "replies": [ { "id": 636010, "postDate": "2019-09-28T15:35:39.037Z", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "rawMarkdown": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ? ", "votes": 1 } ] }, { "id": 642485, "postDate": "2019-10-06T06:21:46.473Z", "content": "LB 0.079 (update 0.074)\n- se_resnext50_32x4d\n- 224x224\n- 2 epochs\n- hflip, crop
", "rawMarkdown": "LB 0.079 (update 0.074)\n- se\\_resnext50\\_32x4d\n- 224x224\n- 2 epochs\n- hflip, crop", "votes": 6, "replies": [ { "id": 643370, "postDate": "2019-10-07T13:25:55.800Z", "content": "@appian Hi! Is it convenient to disclose your train/val split? How is your correlation between train/test LB score?
", "rawMarkdown": "@appian Hi! Is it convenient to disclose your train/val split? How is your correlation between train/test LB score?" }, { "id": 643615, "postDate": "2019-10-07T17:29:05.250Z", "content": "The train valid split is 80:20. I haven't made enough submissions to see the correlation but the difference between public lb and local cv is relatively small so far.
\n\nnice result. i trained the same network on 200k images 256 resolution brain window and have a 0.14 result. what kind of images and windowing are you using? thanks
", "rawMarkdown": "nice result. i trained the same network on 200k images 256 resolution brain window and have a 0.14 result. what kind of images and windowing are you using? thanks" }, { "id": 646222, "postDate": "2019-10-11T02:12:57.783Z", "content": "@appian What LR and scheduling are you using? I am unable to replicate your results. I am already grouping on patient
", "rawMarkdown": "@appian What LR and scheduling are you using? I am unable to replicate your results. I am already grouping on patient" }, { "id": 646609, "postDate": "2019-10-11T13:31:57.627Z", "content": "@appian Great, why are you public LB and Local LB is so closer? my pubic LB always larger than local LB, about 0.02.
", "rawMarkdown": "@appian Great, why are you public LB and Local LB is so closer? my pubic LB always larger than local LB, about 0.02." }, { "id": 646680, "postDate": "2019-10-11T15:03:42.267Z", "content": "My guess is that you randomly split the data for train/val. Because one patient is at least composed of more than 20 slices, it introduces a leak especially when adjacent slices are split between train and val. Somethinkg like GroupKFold can deal with it.
", "rawMarkdown": "My guess is that you randomly split the data for train/val. Because one patient is at least composed of more than 20 slices, it introduces a leak especially when adjacent slices are split between train and val. Somethinkg like GroupKFold can deal with it.", "votes": 2 }, { "id": 646688, "postDate": "2019-10-11T15:17:58.320Z", "content": "from my own experience, I could not get stable cv without using weighted log-loss. I think a lot of the issues with cv not looking like LB is due to not using the proper loss function and not due to how you split your training/validation sets. And I think this makes sense - if you're optimizing your model to a different loss, CV is not going to reflect on LB closely. I used random split and weighted log-loss and my CV and LB look exactly the same.
", "rawMarkdown": "from my own experience, I could not get stable cv without using weighted log-loss. I think a lot of the issues with cv not looking like LB is due to not using the proper loss function and not due to how you split your training/validation sets. And I think this makes sense - if you're optimizing your model to a different loss, CV is not going to reflect on LB closely. I used random split and weighted log-loss and my CV and LB look exactly the same. " }, { "id": 647002, "postDate": "2019-10-12T00:59:41.927Z", "content": "Thanks@Appian@Tim Yee, I split the data for train/val using PatientID, so the slices for one patient ara all in train or val. Maybe my weights of logloss is [1,1,1,1,1,1], I'll try more, thanks.
", "rawMarkdown": "Thanks@Appian@Tim Yee, I split the data for train/val using PatientID, so the slices for one patient ara all in train or val. Maybe my weights of logloss is [1,1,1,1,1,1], I'll try more, thanks." } ] }, { "id": 646623, "postDate": "2019-10-11T13:42:08.373Z", "content": "Seresnext50->LB0.082\nimg size->224*224
", "rawMarkdown": "Seresnext50->LB0.082\nimg size->224*224", "votes": 3, "replies": [ { "id": 647795, "postDate": "2019-10-13T09:39:28.350Z", "content": "update\nSeresnext50->LB0.078\nimg size->224*224\nsubdural_window preprocessing
", "rawMarkdown": "update\nSeresnext50->LB0.078\nimg size->224*224\n[subdural_window preprocessing](https://www.kaggle.com/dcstang/see-like-a-radiologist-with-systematic-windowing)" }, { "id": 659957, "postDate": "2019-10-28T13:45:44.227Z", "content": "for how many epochs do you train? when do you notice overfitting? thanks!
", "rawMarkdown": "for how many epochs do you train? when do you notice overfitting? thanks!" } ] }, { "id": 638792, "postDate": "2019-10-02T12:37:03.580Z", "content": "EfficientNet B0 - 512x512 --> 0.078
", "rawMarkdown": "EfficientNet B0 - 512x512 --> 0.078", "votes": 3, "replies": [ { "id": 638801, "postDate": "2019-10-02T12:52:03.430Z", "content": "If you don't mind me asking, what's your validation split, and how many epochs?
", "rawMarkdown": "If you don't mind me asking, what's your validation split, and how many epochs?" }, { "id": 644817, "postDate": "2019-10-09T11:44:16.667Z", "content": "Random split 95% train & 5% validation... trained for 10 epochs
", "rawMarkdown": "Random split 95% train & 5% validation... trained for 10 epochs" }, { "id": 644975, "postDate": "2019-10-09T15:35:13.177Z", "content": "And how do you fit the 512 images in the B0 model? It takes 224 by default right?
", "rawMarkdown": "And how do you fit the 512 images in the B0 model? It takes 224 by default right?" }, { "id": 644978, "postDate": "2019-10-09T15:37:54.427Z", "content": "You can fit any image size. You can put in 128x128 if you want.
", "rawMarkdown": "You can fit any image size. You can put in 128x128 if you want." }, { "id": 644987, "postDate": "2019-10-09T16:03:28.053Z", "content": "But when I try to put 512x512 images in my B0 model it gives me \"invalid argument 0: sizes of tensor must match except in dimension 0\" error
", "rawMarkdown": "But when I try to put 512x512 images in my B0 model it gives me \"invalid argument 0: sizes of tensor must match except in dimension 0\" error" }, { "id": 645368, "postDate": "2019-10-10T03:46:16.700Z", "content": "Do you train using all images?
", "rawMarkdown": "Do you train using all images?" }, { "id": 648422, "postDate": "2019-10-14T06:43:14.103Z", "content": "yes, i do use all images
", "rawMarkdown": "yes, i do use all images", "votes": 1 } ] }, { "id": 634225, "postDate": "2019-09-26T03:48:01.747Z", "content": "Using only 30,000 samples(15k positive and 15k negative)
\n\nEfficient Net B4 10 epochs 256x256 - 0.113\nEfficient Net B4 3 epochs 256x256 - 0.107
\n\nLB improving with less number of epochs? (Edit - Severely Overfitting?)
", "rawMarkdown": "Using only 30,000 samples(15k positive and 15k negative)\n\nEfficient Net B4 10 epochs 256x256 - 0.113\nEfficient Net B4 3 epochs 256x256 - 0.107 \n\nLB improving with less number of epochs? (Edit - Severely Overfitting?)", "votes": 3, "replies": [ { "id": 634256, "postDate": "2019-09-26T05:23:20.430Z", "content": "0.098 4 epochs. I haven't tried greater than 4 epochs yet. I already used this weeks GPU quota.
", "rawMarkdown": "0.098 4 epochs. I haven't tried greater than 4 epochs yet. I already used this weeks GPU quota.", "votes": 2 }, { "id": 634305, "postDate": "2019-09-26T06:20:06.027Z", "content": "Are you using the full dataset? How long it takes for 1 epoch?
", "rawMarkdown": "Are you using the full dataset? How long it takes for 1 epoch?", "votes": 2 }, { "id": 634796, "postDate": "2019-09-26T19:08:20.097Z", "content": "In my opinon, its not about the number of epochs, but about not overfitting. 🙂
", "rawMarkdown": "In my opinon, its not about the number of epochs, but about not overfitting. 🙂 ", "votes": 2 }, { "id": 636567, "postDate": "2019-09-29T19:49:20.130Z", "content": "@bopengiowa Do you have a suggestion about removing overfitting? Thank you!
", "rawMarkdown": "@bopengiowa Do you have a suggestion about removing overfitting? Thank you!", "votes": 1 }, { "id": 636626, "postDate": "2019-09-29T22:59:08.330Z", "content": "Common for keras is it saves the weights from the best val score correct? So that should solve it right there. I think there are other learning rate things that people do such as scheduling and decay. Could also check if local val score matches with leaderboard scores.
", "rawMarkdown": "Common for keras is it saves the weights from the best val score correct? So that should solve it right there. I think there are other learning rate things that people do such as scheduling and decay. Could also check if local val score matches with leaderboard scores.", "votes": 2 }, { "id": 636718, "postDate": "2019-09-30T05:09:07.800Z", "content": "@bopengiowa thank you! I will try it 👍
", "rawMarkdown": "@bopengiowa thank you! I will try it 👍 ", "votes": 1 } ] }, { "id": 653855, "postDate": "2019-10-21T04:02:27.207Z", "content": "efficientnet-b2\n410x410\nsingle fold\nhflip tta\npublicLB: 0.064
", "rawMarkdown": "efficientnet-b2\n410x410\nsingle fold\nhflip tta\npublicLB: 0.064", "votes": 4, "replies": [ { "id": 653896, "postDate": "2019-10-21T06:11:57.740Z", "content": "Thanks for sharing! Are you using 5 epochs, png images as you mentioned before also?
", "rawMarkdown": "Thanks for sharing! Are you using 5 epochs, png images as you mentioned before also?" }, { "id": 653901, "postDate": "2019-10-21T06:40:36.767Z", "content": "Yes.
", "rawMarkdown": "Yes.", "votes": 2 }, { "id": 654556, "postDate": "2019-10-22T02:52:26.817Z", "content": "Cool. Could you please share the windowing method?
", "rawMarkdown": "Cool. Could you please share the windowing method?" }, { "id": 654594, "postDate": "2019-10-22T04:22:20.083Z", "content": "Are you using the new CQ500 dataset?
", "rawMarkdown": "Are you using the new CQ500 dataset?" }, { "id": 654637, "postDate": "2019-10-22T05:56:50.023Z", "content": "I used Appian's windowing and didn't use CQ500.
", "rawMarkdown": "I used Appian's windowing and didn't use CQ500." }, { "id": 654643, "postDate": "2019-10-22T06:12:18.353Z", "content": "How long one epoch takes
", "rawMarkdown": "How long one epoch takes " }, { "id": 654732, "postDate": "2019-10-22T08:30:44.570Z", "content": "4-5 hours
", "rawMarkdown": "4-5 hours" }, { "id": 654897, "postDate": "2019-10-22T12:56:27.053Z", "content": "Thanks for sharing, I see the efficientnet-b2 was trained with input imm 224x224. How did our shape your net to receive in input imm 410x410 ?
", "rawMarkdown": "Thanks for sharing, I see the efficientnet-b2 was trained with input imm 224x224. How did our shape your net to receive in input imm 410x410 ?" }, { "id": 655399, "postDate": "2019-10-23T02:30:16.160Z", "content": "@uiiurz1 Nice results indeed. If I understand correctly, you only used hflip during training, I was just wondering how you apply tta? Thanks once again.
", "rawMarkdown": "@uiiurz1 Nice results indeed. If I understand correctly, you only used hflip during training, I was just wondering how you apply tta? Thanks once again." } ] }, { "id": 651563, "postDate": "2019-10-17T15:58:05.833Z", "content": "Update: \nresnet50, 256x256, train on 150k images per epochs, split by patient, lb 0.071 no tta single fold
", "rawMarkdown": "Update: \nresnet50, 256x256, train on 150k images per epochs, split by patient, lb 0.071 no tta single fold", "votes": 4, "replies": [ { "id": 651594, "postDate": "2019-10-17T16:38:34.343Z", "content": "Are you sampling new 150K images every epoch?
", "rawMarkdown": "Are you sampling new 150K images every epoch? " }, { "id": 651599, "postDate": "2019-10-17T16:44:32.797Z", "content": "Yes, 150k images is randomly selected
", "rawMarkdown": "Yes, 150k images is randomly selected" } ] }, { "id": 636711, "postDate": "2019-09-30T04:56:56.933Z", "content": "resnet34, image 256x256, 10k pos, 10k neg, single fold, 0.088 lb\nUpdate: resnet34, image 256x256, single fold -> 0.078 lb
", "rawMarkdown": "resnet34, image 256x256, 10k pos, 10k neg, single fold, 0.088 lb\nUpdate: resnet34, image 256x256, single fold -> 0.078 lb", "votes": 4, "replies": [ { "id": 636882, "postDate": "2019-09-30T09:57:56.167Z", "content": "I think you mean 0.088?
", "rawMarkdown": "I think you mean 0.088?", "votes": 1 }, { "id": 637804, "postDate": "2019-10-01T09:16:03.360Z", "content": "I then use the the best checkpoint and finetune on the whole dataset, get lb 0.081 in the first epochs. However, but when I continue to finetune, the lb get worse, even though local cv still improve. Previous cv score is consistent w/ lb. I currently have no idea why. Still working on that :))
", "rawMarkdown": "I then use the the best checkpoint and finetune on the whole dataset, get lb 0.081 in the first epochs. However, but when I continue to finetune, the lb get worse, even though local cv still improve. Previous cv score is consistent w/ lb. I currently have no idea why. Still working on that :))", "votes": 1 }, { "id": 637833, "postDate": "2019-10-01T09:50:05.150Z", "content": "@dattran2346 your model seem overffiting :))
", "rawMarkdown": "@dattran2346 your model seem overffiting :)) " }, { "id": 637896, "postDate": "2019-10-01T10:55:51.310Z", "content": "My train and val loss still improving, hardly say it is overfit. Maybe the problem is how I up-train from under-sampling to all dataset 🤔
", "rawMarkdown": "My train and val loss still improving, hardly say it is overfit. Maybe the problem is how I up-train from under-sampling to all dataset 🤔 " }, { "id": 638427, "postDate": "2019-10-01T22:17:59.863Z", "content": "Forgive my stupidity, but what do u mean by 10k pos, 10k neg and r34?
", "rawMarkdown": "Forgive my stupidity, but what do u mean by 10k pos, 10k neg and r34?", "votes": 1 }, { "id": 638502, "postDate": "2019-10-02T01:20:22.140Z", "content": "he means: \n10 k postive examples\n10 k negative example \nresent 34 (r34?)\n256 image size
", "rawMarkdown": "he means: \n10 k postive examples\n10 k negative example \nresent 34 (r34?)\n256 image size ", "votes": 3 }, { "id": 638726, "postDate": "2019-10-02T11:02:11.877Z", "content": "U mean he only uses 20k IDs of the train data, 10k which any = 0 and 10k which any = 1?
", "rawMarkdown": "U mean he only uses 20k IDs of the train data, 10k which any = 0 and 10k which any = 1?", "votes": 1 }, { "id": 638734, "postDate": "2019-10-02T11:13:50.710Z", "content": "yep
", "rawMarkdown": "yep ", "votes": 1 }, { "id": 638745, "postDate": "2019-10-02T11:34:38.060Z", "content": "Yes, I use resnet 34 with image size of 256 and train under-sampling now to speed up experiment. The samples are selected such that each diseases have the same number of sample, and the total number of positive and negative are the same.
", "rawMarkdown": "Yes, I use resnet 34 with image size of 256 and train under-sampling now to speed up experiment. The samples are selected such that each diseases have the same number of sample, and the total number of positive and negative are the same.", "votes": 1 }, { "id": 643971, "postDate": "2019-10-08T07:22:03.347Z", "content": "Thanks @dattran2346 for this information.
\n\nI'm curious, I've been trying the 10k +/10k - approach too since I lack access to a big GPU and want to iterate not too slowly. I imagine that your CV numbers are different from the public LB scores right? Since our validation set contains way more examples from minority classes, I'd expect the CV score to be significantly worse than the public LB scores. Is it the case for you too?
", "rawMarkdown": "Thanks @dattran2346 for this information.\n\nI'm curious, I've been trying the 10k +/10k - approach too since I lack access to a big GPU and want to iterate not too slowly. I imagine that your CV numbers are different from the public LB scores right? Since our validation set contains way more examples from minority classes, I'd expect the CV score to be significantly worse than the public LB scores. Is it the case for you too?" }, { "id": 644092, "postDate": "2019-10-08T10:00:43.113Z", "content": "Hi @juliencs \nFor me, the local CV is about 0.08 at epoch 75, and the LB for that checkpoint is 0.086. How do you split your CV, do you stratify base on labels, or base on metadata like Patient or Study ID. Do you pick hard sample and train on them.\nMy CV is stratified by labels, and I just pick random samples.
", "rawMarkdown": "Hi @juliencs \nFor me, the local CV is about 0.08 at epoch 75, and the LB for that checkpoint is 0.086. How do you split your CV, do you stratify base on labels, or base on metadata like Patient or Study ID. Do you pick hard sample and train on them.\nMy CV is stratified by labels, and I just pick random samples.", "votes": 1 }, { "id": 644147, "postDate": "2019-10-08T12:15:31.803Z", "content": "Ooooh, 75 epochs. I never tried that long. My CV is same as yours it seems, but I never went past 15 epochs, and the loss was still pretty high by then, hence my question.
\n\n75 epochs must last a long time. The idea for me to take a really small sample was to iterate quicker, but if we have to wait for 75 epochs on the smaller sample to reach a minimum, maybe it's not worth it, and taking a bigger (and thus more diverse) sample, for less epochs, might yield at least similar results (just thinking out loud here really).
\n\nThanks for your answer. I'll keep experimenting.
", "rawMarkdown": "Ooooh, 75 epochs. I never tried that long. My CV is same as yours it seems, but I never went past 15 epochs, and the loss was still pretty high by then, hence my question.\n\n75 epochs must last a long time. The idea for me to take a really small sample was to iterate quicker, but if we have to wait for 75 epochs on the smaller sample to reach a minimum, maybe it's not worth it, and taking a bigger (and thus more diverse) sample, for less epochs, might yield at least similar results (just thinking out loud here really).\n\nThanks for your answer. I'll keep experimenting." }, { "id": 644155, "postDate": "2019-10-08T12:27:18.043Z", "content": "Yes, totally agree with you, 75 epochs is way too long for a under-sample model. And train for that long may actually hurt the model performance. Another approach is to filter out samples in the training, i.e, remove obvious images that have no brain in it for example.
", "rawMarkdown": "Yes, totally agree with you, 75 epochs is way too long for a under-sample model. And train for that long may actually hurt the model performance. Another approach is to filter out samples in the training, i.e, remove obvious images that have no brain in it for example. " }, { "id": 644308, "postDate": "2019-10-08T15:49:05.910Z", "content": "So you remove obvious images with no brain in it by hand? or you have some filter function? I could not look at more than 100 even If I tried.
", "rawMarkdown": "So you remove obvious images with no brain in it by hand? or you have some filter function? I could not look at more than 100 even If I tried." }, { "id": 644313, "postDate": "2019-10-08T15:55:26.367Z", "content": "You can use a simple brain window for this task, just filter by the HU of the brain.
", "rawMarkdown": "You can use a simple brain window for this task, just filter by the HU of the brain." }, { "id": 644348, "postDate": "2019-10-08T16:55:48.047Z", "content": "But just by filtering by brain window you cant judge whether there is brain or not right? And even the brain window values can vary, there isn't really an optimum range right?
", "rawMarkdown": "But just by filtering by brain window you cant judge whether there is brain or not right? And even the brain window values can vary, there isn't really an optimum range right?\n" }, { "id": 644354, "postDate": "2019-10-08T17:07:23.587Z", "content": "According to wiki the HU value of the brain is just a narrow range, you can just filter images that don't have (or very small number of pixel) of brain HU. There are about 2-3 images per study where there is no brain (just empty, or part of the CT machine, or just top of the skull).
", "rawMarkdown": "According to [wiki](https://en.wikipedia.org/wiki/Hounsfield_scale) the HU value of the brain is just a narrow range, you can just filter images that don't have (or very small number of pixel) of brain HU. There are about 2-3 images per study where there is no brain (just empty, or part of the CT machine, or just top of the skull)." } ] }, { "id": 636110, "postDate": "2019-09-28T20:36:56.447Z", "content": "ResNeXt-101 32x8d: 0.082
\n\nKernel with benchmark (0.089): https://www.kaggle.com/taindow/pytorch-resnext-101-32x8d-benchmark
", "rawMarkdown": "ResNeXt-101 32x8d: 0.082\n\nKernel with benchmark (0.089): https://www.kaggle.com/taindow/pytorch-resnext-101-32x8d-benchmark", "votes": 4, "replies": [ { "id": 636300, "postDate": "2019-09-29T08:15:02.430Z", "content": "I'm sure it sounds stupid, but what's the difference between your kernel with benchmark and your actual model that gives better results? Some extra preprocessing? Because I am also using the ResNeXt-101 model and getting 0.089
", "rawMarkdown": "I'm sure it sounds stupid, but what's the difference between your kernel with benchmark and your actual model that gives better results? Some extra preprocessing? Because I am also using the ResNeXt-101 model and getting 0.089" }, { "id": 636431, "postDate": "2019-09-29T14:14:35.937Z", "content": "Same dataset, but more epochs and splitting the data to monitor validation performance for the competition metric :)
", "rawMarkdown": "Same dataset, but more epochs and splitting the data to monitor validation performance for the competition metric :)", "votes": 2 }, { "id": 636471, "postDate": "2019-09-29T15:30:23.563Z", "content": "Thanks :) I got 0.086 by using 32x16d benchmark
", "rawMarkdown": "Thanks :) I got 0.086 by using 32x16d benchmark", "votes": 1 }, { "id": 638111, "postDate": "2019-10-01T14:43:36.420Z", "content": "@taindow Thank you for sharing! It sounds stupid but what metric do you use for your kernel.
", "rawMarkdown": "@taindow Thank you for sharing! It sounds stupid but what metric do you use for your kernel." }, { "id": 638381, "postDate": "2019-10-01T21:02:57.100Z", "content": "U mean you are using the metric as a loss function in your model? My model ran for 2 epochs and, between epochs 1 and 2, my validation score (BCE) increased while the competition metric continued going down (same as training loss). Are u having this problem as well?
", "rawMarkdown": "U mean you are using the metric as a loss function in your model? My model ran for 2 epochs and, between epochs 1 and 2, my validation score (BCE) increased while the competition metric continued going down (same as training loss). Are u having this problem as well?" } ] }, { "id": 634704, "postDate": "2019-09-26T16:37:40.677Z", "content": "resnet34 -- 0.084
", "rawMarkdown": "resnet34 -- 0.084", "votes": 4, "replies": [ { "id": 636009, "postDate": "2019-09-28T15:35:10.650Z", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "rawMarkdown": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ? ", "votes": 1 }, { "id": 636278, "postDate": "2019-09-29T07:08:15.800Z", "content": "Image size - 512 x 512...all the images used
", "rawMarkdown": "Image size - 512 x 512...all the images used", "votes": 3 } ] }, { "id": 660412, "postDate": "2019-10-29T04:56:29.687Z", "content": "a simple resnet34 without modification on 288x288 get decent results of LB 0.080 (local cv 0.057)\nsee: https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/109272#latest-660570
\n\n```\nwindowing input:
\n\nbrain = window_image(hu, 40, 80)\nsubdural = window_image(hu, 80, 200)\nsoft_tissue = window_image(hu, 40, 380)\nsingle model, TTA = ['null', 'flip_left_right']
\n\n```
", "rawMarkdown": "a simple resnet34 without modification on 288x288 get decent results of LB 0.080 (local cv 0.057)\nsee: https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/109272#latest-660570\n\n```\nwindowing input:\n\n brain = window_image(hu, 40, 80)\n subdural = window_image(hu, 80, 200)\n soft_tissue = window_image(hu, 40, 380)\n\nsingle model, TTA = ['null', 'flip_left_right']\n\n```\n", "votes": 1, "replies": [ { "id": 660511, "postDate": "2019-10-29T08:38:39.957Z", "content": "update: the same setup achieve LB 0.70 after increasing size to 512x512 input
", "rawMarkdown": "update: the same setup achieve LB 0.70 after increasing size to 512x512 input", "votes": 1 }, { "id": 660530, "postDate": "2019-10-29T09:17:11.380Z", "content": "other results:\nEfficientNet B4 224x224 - 0.081\nSeResNext50 512x512 - 0.070
", "rawMarkdown": "other results:\nEfficientNet B4 224x224 - 0.081\nSeResNext50 512x512 - 0.070\n", "votes": 1 }, { "id": 660533, "postDate": "2019-10-29T09:28:34.683Z", "content": "@hengck23 interested in teaming up?
", "rawMarkdown": "@hengck23 interested in teaming up?" }, { "id": 660671, "postDate": "2019-10-29T13:02:12.743Z", "content": "@hengck23 Do you have any ideas on how to go below 0.065? It seems that a well trained, reasonably complex model on full resolution and one of publicly known windowing variants will give ~0.07-0.074 on single fold and ~0.065-0.067 ensembling several folds
", "rawMarkdown": "@hengck23 Do you have any ideas on how to go below 0.065? It seems that a well trained, reasonably complex model on full resolution and one of publicly known windowing variants will give ~0.07-0.074 on single fold and ~0.065-0.067 ensembling several folds " }, { "id": 660691, "postDate": "2019-10-29T13:33:36.873Z", "content": "the training loss can go as low as 0.030~0.040. Hence with good generalization, it think you can get 0.060 without tricks.
\n\ntry ensemble network of different architecture, difference size input, different windowing, etc
\n\ntry tta of crop, scale, flip
\n\ntry smooth your output by
\n\n```\nlogit = net(input)\nlogit += some small noise
\n\nloss =(logit,truth)\nloss.backward()
\n\n```\n etc ...
", "rawMarkdown": "the training loss can go as low as 0.030~0.040. Hence with good generalization, it think you can get 0.060 without tricks. \n\ntry ensemble network of different architecture, difference size input, different windowing, etc\n\ntry tta of crop, scale, flip\n\ntry smooth your output by \n\n\n```\nlogit = net(input)\nlogit += some small noise\n\nloss =(logit,truth)\nloss.backward()\n\n```\n etc ...", "votes": 2 }, { "id": 660809, "postDate": "2019-10-29T16:31:02.057Z", "content": "https://www.kaggle.com/braquino/correct-images-sequece
\n\nyou can try to to use nearby slice of the same patient to post correct prediction results
", "rawMarkdown": "https://www.kaggle.com/braquino/correct-images-sequece\n\nyou can try to to use nearby slice of the same patient to post correct prediction results" } ] }, { "id": 644742, "postDate": "2019-10-09T09:10:38.833Z", "content": "update\nEfficientNet B0 - 256x256 --> 0.072
", "rawMarkdown": "update\nEfficientNet B0 - 256x256 --> 0.072", "votes": 1, "replies": [ { "id": 644759, "postDate": "2019-10-09T09:45:02.697Z", "content": "Are you loading images from dicom? And how many epochs?
", "rawMarkdown": "Are you loading images from dicom? And how many epochs?" }, { "id": 644875, "postDate": "2019-10-09T13:05:31.317Z", "content": "@uiiurz1 Hi, do u use Keras? I would like to know if it is allowed to import EfficientNet from some repository, since it doesn't come built in Keras and I didn't see anybody anouncing this model in the external data thread.
", "rawMarkdown": "@uiiurz1 Hi, do u use Keras? I would like to know if it is allowed to import EfficientNet from some repository, since it doesn't come built in Keras and I didn't see anybody anouncing this model in the external data thread." }, { "id": 645235, "postDate": "2019-10-09T23:25:53.100Z", "content": "Do you train on all images or just a sub-set?
", "rawMarkdown": "Do you train on all images or just a sub-set?" }, { "id": 645522, "postDate": "2019-10-10T07:14:39.773Z", "content": "Do you have a validation split?
", "rawMarkdown": "Do you have a validation split?" }, { "id": 645611, "postDate": "2019-10-10T09:38:43.407Z", "content": "Thanks, I guess your preprocessing must be very good to get you that LB score
", "rawMarkdown": "Thanks, I guess your preprocessing must be very good to get you that LB score" }, { "id": 651186, "postDate": "2019-10-17T06:45:46.777Z", "content": "pip install efficientnet\npip install keras_efficientnet
", "rawMarkdown": "pip install efficientnet\npip install keras_efficientnet" } ] }, { "id": 641962, "postDate": "2019-10-05T11:32:11.260Z", "content": "VGG19 - 224x224 - 0.073
", "rawMarkdown": "VGG19 - 224x224 - 0.073", "votes": 1, "replies": [ { "id": 641990, "postDate": "2019-10-05T12:10:40.497Z", "content": "Thank you for sharing! Do you mind if I ask How many epoch and data do you use for training?
", "rawMarkdown": "Thank you for sharing! Do you mind if I ask How many epoch and data do you use for training? " }, { "id": 642340, "postDate": "2019-10-05T23:10:22.213Z", "content": "I'm using model checkpoint based on val_loss, but I think it was around 20 epochs. And I'm using one-fold with 80% for train e 20% for validation.
", "rawMarkdown": "I'm using model checkpoint based on val_loss, but I think it was around 20 epochs. And I'm using one-fold with 80% for train e 20% for validation.", "votes": 1 }, { "id": 642374, "postDate": "2019-10-06T00:56:10.057Z", "content": "@fernandocamargo how long your model take to training one epoch? Thank you for replying me!
", "rawMarkdown": "@fernandocamargo how long your model take to training one epoch? Thank you for replying me! " }, { "id": 643596, "postDate": "2019-10-07T17:01:38.393Z", "content": "It takes around 40min.
", "rawMarkdown": "It takes around 40min.", "votes": 1 }, { "id": 643667, "postDate": "2019-10-07T18:55:40.143Z", "content": "Nice work! I am trying to speed my code for fast training. Could you give me some suggestion for this? Thank you so much!
", "rawMarkdown": "Nice work! I am trying to speed my code for fast training. Could you give me some suggestion for this? Thank you so much!" }, { "id": 643901, "postDate": "2019-10-08T05:27:28.427Z", "content": "The easiest way to speed up training would be using FP16 computations. Also I posted a paper here which talks about only updating the top losses when backpropagating the error. It can also provide some speed up.
", "rawMarkdown": "The easiest way to speed up training would be using FP16 computations. Also I posted a paper [here](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/111383#latest-641873) which talks about only updating the top losses when backpropagating the error. It can also provide some speed up.", "votes": 2 }, { "id": 643957, "postDate": "2019-10-08T07:00:29.720Z", "content": "hi @salilm23 . Thanks for your suggestion. May I ask you How can I find implementation of this paper?
", "rawMarkdown": "hi @salilm23 . Thanks for your suggestion. May I ask you How can I find implementation of this paper?\n" }, { "id": 643970, "postDate": "2019-10-08T07:18:58.707Z", "content": "Hey @linhlpv , implementation is also linked in the post :)
\n\nEdit - Sorry, the implementation linked to the paper has now been removed
", "rawMarkdown": "Hey @linhlpv , implementation is also linked in the post :)\n\nEdit - Sorry, the implementation linked to the paper has now been removed ", "votes": 2 }, { "id": 643998, "postDate": "2019-10-08T07:59:37.797Z", "content": "@salilm23 - hi, do you have a local copy of the code? maybe you could upload it somewehere
", "rawMarkdown": "@salilm23 - hi, do you have a local copy of the code? maybe you could upload it somewehere" }, { "id": 644117, "postDate": "2019-10-08T10:52:52.180Z", "content": "Well, the implementation seems to be back now. https://anonymous.4open.science/r/c6d4060d-bdac-4d31-839e-8579650255b3/
", "rawMarkdown": "Well, the implementation seems to be back now. https://anonymous.4open.science/r/c6d4060d-bdac-4d31-839e-8579650255b3/", "votes": 1 } ] }, { "id": 640833, "postDate": "2019-10-04T09:02:46.363Z", "content": "ResNeXt 32x8d - 0.087 with 50/50 % sampler
", "rawMarkdown": "ResNeXt 32x8d - 0.087 with 50/50 % sampler", "votes": 1, "replies": [ { "id": 640848, "postDate": "2019-10-04T09:13:19.170Z", "content": "What do you mean by a 50/50% sampler exactly?
", "rawMarkdown": "What do you mean by a 50/50% sampler exactly?\n" }, { "id": 640860, "postDate": "2019-10-04T09:20:50.817Z", "content": "I took all data with any == 1 and add randomly with any !=0 so that the ratio would be 50 to 50 %
", "rawMarkdown": "I took all data with any == 1 and add randomly with any !=0 so that the ratio would be 50 to 50 %" }, { "id": 640877, "postDate": "2019-10-04T09:34:59.760Z", "content": "I'm not sure I'm getting you here, wouldn't any==1 and any !=0 give the same images?
", "rawMarkdown": "I'm not sure I'm getting you here, wouldn't any==1 and any !=0 give the same images?" }, { "id": 641073, "postDate": "2019-10-04T12:24:26.867Z", "content": "Sorry. It is my mistake. I mean any==1 and any !=1
", "rawMarkdown": "Sorry. It is my mistake. I mean any==1 and any !=1", "votes": 1 }, { "id": 641579, "postDate": "2019-10-04T20:01:27.870Z", "content": "And how many epochs?
", "rawMarkdown": "And how many epochs?\n" }, { "id": 641757, "postDate": "2019-10-05T03:48:24.087Z", "content": "1 epoch)
", "rawMarkdown": "1 epoch)" } ] }, { "id": 650323, "postDate": "2019-10-16T09:21:56.890Z", "content": "Seresnext50->LB0.072\nimg size->512*512\nthree windowing preprocessor
", "rawMarkdown": "Seresnext50->LB0.072\nimg size->512*512\nthree windowing preprocessor", "votes": 2, "replies": [ { "id": 659275, "postDate": "2019-10-27T11:46:14.093Z", "content": "@takuok I am just wondering if you can share what augmentations you are using? Also, is the result of a single fold?
", "rawMarkdown": "@takuok I am just wondering if you can share what augmentations you are using? Also, is the result of a single fold?" } ] }, { "id": 647449, "postDate": "2019-10-12T16:35:22.083Z", "content": "With raw input from dcmread, random ShuffleSplit, no tta,\nB0 0.079 -> 0.074 224x224
", "rawMarkdown": "With raw input from dcmread, random ShuffleSplit, no tta,\nB0 0.079 -> 0.074 224x224", "votes": 2, "replies": [ { "id": 647547, "postDate": "2019-10-12T20:39:20.787Z", "content": "Epochs?
", "rawMarkdown": "Epochs?" }, { "id": 647550, "postDate": "2019-10-12T20:45:03.347Z", "content": "I'm pretty sure that's subjective to many other factors. For .95/.05 split, default class weight it seems 4-6 epochs works well and then starts to overfit real bad real fast afterwards.
", "rawMarkdown": "I'm pretty sure that's subjective to many other factors. For .95/.05 split, default class weight it seems 4-6 epochs works well and then starts to overfit real bad real fast afterwards." }, { "id": 647551, "postDate": "2019-10-12T20:46:41.027Z", "content": "Noticed that for resnext, more than 4 epochs is generally overfittnig
", "rawMarkdown": "Noticed that for resnext, more than 4 epochs is generally overfittnig" }, { "id": 650016, "postDate": "2019-10-16T02:26:43.553Z", "content": "could you tell me what's the meaning of tta?\nMuch thanks
", "rawMarkdown": "could you tell me what's the meaning of tta?\nMuch thanks" }, { "id": 650091, "postDate": "2019-10-16T04:59:59.483Z", "content": "Test-time augmentation: https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/
", "rawMarkdown": "Test-time augmentation: https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/" }, { "id": 659954, "postDate": "2019-10-28T13:41:23.760Z", "content": "what sort of training loss and validation loss do you have just before it starts to overfit? thanks
", "rawMarkdown": "what sort of training loss and validation loss do you have just before it starts to overfit? thanks" } ] }, { "id": 642349, "postDate": "2019-10-05T23:31:55.780Z", "content": "InceptionResnetv2 -224*224 --> 0.086
", "rawMarkdown": "InceptionResnetv2 -224*224 --> 0.086", "votes": 2 }, { "id": 636490, "postDate": "2019-09-29T16:18:37.737Z", "content": "ResNeXt-101 32x16d : 0.086
", "rawMarkdown": "ResNeXt-101 32x16d : 0.086", "votes": 2, "replies": [ { "id": 636568, "postDate": "2019-09-29T19:50:08.533Z", "content": "Thanks for your sharing! How many epoch that you train you model?
", "rawMarkdown": "Thanks for your sharing! How many epoch that you train you model?" }, { "id": 636657, "postDate": "2019-09-30T01:56:38.690Z", "content": "1 epoch for now, due to size of dataset even that takes a long time
", "rawMarkdown": "1 epoch for now, due to size of dataset even that takes a long time", "votes": 1 }, { "id": 636714, "postDate": "2019-09-30T05:04:37.243Z", "content": "Nice work! I am also using ResNeXt 32x16d and archive 0.088 :D. i using Adam optimizer with lr=1e-5. Thanks for aswering me!
", "rawMarkdown": "Nice work! I am also using ResNeXt 32x16d and archive 0.088 :D. i using Adam optimizer with lr=1e-5. Thanks for aswering me!\n" }, { "id": 638923, "postDate": "2019-10-02T15:07:56.093Z", "content": "Did you change your model to better your accuracy? Or did you change some hyperparameters like learning rate?
", "rawMarkdown": "Did you change your model to better your accuracy? Or did you change some hyperparameters like learning rate?\n" }, { "id": 641589, "postDate": "2019-10-04T20:06:24.517Z", "content": "Now I use RAdam for optimizer. You should try it.
", "rawMarkdown": "Now I use RAdam for optimizer. You should try it." }, { "id": 643083, "postDate": "2019-10-07T04:37:05.440Z", "content": "So you're using keras then?
", "rawMarkdown": "So you're using keras then?\n" }, { "id": 643170, "postDate": "2019-10-07T07:52:27.010Z", "content": "I have both Keras and Pytorch Base code.
", "rawMarkdown": "I have both Keras and Pytorch Base code." } ] }, { "id": 649196, "postDate": "2019-10-15T03:54:12.313Z", "content": "VGG19 \nepoch20\nLB：0.082
", "rawMarkdown": "VGG19 \nepoch20\nLB：0.082", "replies": [ { "id": 649393, "postDate": "2019-10-15T09:42:36.517Z", "content": "Did you train it on the whole dataset?
", "rawMarkdown": "Did you train it on the whole dataset?" }, { "id": 659780, "postDate": "2019-10-28T08:54:39.633Z", "content": "undata: VGG19_BN\nepoches:15\nLB:0.075
", "rawMarkdown": "undata: VGG19_BN\nepoches:15\nLB:0.075" }, { "id": 659781, "postDate": "2019-10-28T08:55:37.440Z", "content": "yes， I use the whole data, I will try three window process, but i have no a hug disk
", "rawMarkdown": "yes， I use the whole data, I will try three window process, but i have no a hug disk" } ] }, { "id": 643575, "postDate": "2019-10-07T16:28:33.317Z", "content": "So far Resnet50 .0094 w/o any augmentation or tta.
", "rawMarkdown": "So far Resnet50 .0094 w/o any augmentation or tta. ", "replies": [ { "id": 650014, "postDate": "2019-10-16T02:25:52.067Z", "content": "could you tell me what's the meaning of tta?\nMuch Thanks
", "rawMarkdown": "could you tell me what's the meaning of tta?\nMuch Thanks" }, { "id": 653123, "postDate": "2019-10-20T01:13:20.010Z", "content": "TTA = Test Time Augmentation
", "rawMarkdown": "TTA = Test Time Augmentation", "votes": 1 } ] }, { "id": 660794, "postDate": "2019-10-29T16:09:22.333Z", "content": "hm.
", "rawMarkdown": "hm.\n", "votes": -1 }, { "id": 648597, "postDate": "2019-10-14T12:01:47.100Z", "content": "@appian thanks for reply but it takes me to 4 hours for an epoch on kernel what should i do
", "rawMarkdown": "@appian thanks for reply but it takes me to 4 hours for an epoch on kernel what should i do" }, { "id": 648113, "postDate": "2019-10-13T19:03:22.423Z", "content": "@appian how much time does it take you train up to 2 epochs
", "rawMarkdown": "@appian how much time does it take you train up to 2 epochs", "replies": [ { "id": 648470, "postDate": "2019-10-14T08:35:28.220Z", "content": "Took less than 3 hours with a single 1080ti (one epoch was about 5,000 seconds)
", "rawMarkdown": "Took less than 3 hours with a single 1080ti (one epoch was about 5,000 seconds)" } ] }, { "id": 647975, "postDate": "2019-10-13T15:02:09.523Z", "content": "EfficientNet B0 - 256x256 --> 0.077\nImage augmentation included horizontal flip and rotation of up to 10 degrees\nTrained for 10 epochs using cyclical learning rate, max 0.009
", "rawMarkdown": "EfficientNet B0 - 256x256 --> 0.077\nImage augmentation included horizontal flip and rotation of up to 10 degrees\nTrained for 10 epochs using cyclical learning rate, max 0.009", "replies": [ { "id": 648012, "postDate": "2019-10-13T16:01:55.150Z", "content": "@joerengland did you add anything to your B0? mine seems to be overfitting and stuck above 0.086 no matter the regularization method I'm using, and I'm basically using everything I know (globalpooling, dropout, weight decay)
", "rawMarkdown": "@joerengland did you add anything to your B0? mine seems to be overfitting and stuck above 0.086 no matter the regularization method I'm using, and I'm basically using everything I know (globalpooling, dropout, weight decay)" }, { "id": 648183, "postDate": "2019-10-13T20:48:46.207Z", "content": "No I didn't add anything. I used the pretrained version from this Pytorch implementation (https://github.com/lukemelas/EfficientNet-PyTorch) pretty much right out of the box using the fastai library. Every time I try to regularize strongly or add more data augmentation, performance drops. I have also tried unfreezing and fine tuning the pretrained layers a few times, but that has thus far been less successful than just doing transfer learning.
", "rawMarkdown": "No I didn't add anything. I used the pretrained version from this Pytorch implementation (https://github.com/lukemelas/EfficientNet-PyTorch) pretty much right out of the box using the fastai library. Every time I try to regularize strongly or add more data augmentation, performance drops. I have also tried unfreezing and fine tuning the pretrained layers a few times, but that has thus far been less successful than just doing transfer learning.", "votes": 1 } ] }, { "id": 647830, "postDate": "2019-10-13T11:06:54.310Z", "content": "@XingJian Lyu are you training on full dataset
", "rawMarkdown": "@XingJian Lyu are you training on full dataset ", "replies": [ { "id": 656649, "postDate": "2019-10-24T13:50:13.223Z", "content": "@pranshu29 Yes, I train on full dataset by default:)
", "rawMarkdown": "@pranshu29 Yes, I train on full dataset by default:)" } ] }, { "id": 647633, "postDate": "2019-10-13T02:27:32.177Z", "content": "As Oct. 12th, size 256x256, model: Resnet50 0.089
", "rawMarkdown": "As Oct. 12th, size 256x256, model: Resnet50 0.089" }, { "id": 642803, "postDate": "2019-10-06T17:13:29.340Z", "content": "ResNet50 - 224*224, 5epochs, full dataset (23905 seconds) --> 0.108...
", "rawMarkdown": "ResNet50 - 224*224, 5epochs, full dataset (23905 seconds) --> 0.108...", "replies": [ { "id": 643184, "postDate": "2019-10-07T08:06:30.730Z", "content": "@ihelon have you tried a different architecture? I don't know why, but I find ResNet50 weak this competition.
", "rawMarkdown": "@ihelon have you tried a different architecture? I don't know why, but I find ResNet50 weak this competition. " }, { "id": 644125, "postDate": "2019-10-08T11:21:06.623Z", "content": "Same with me...resnet50 is not performing well. resnet34 is better.
Score - 0.107\n- EfficientNetb2 , 224x \n- With just 2 epoch .\n- steps = len(generator)\n- Little touch on image pre processing for Gaussian Blur\n- loss - log_loss
", "rawMarkdown": "Score - 0.107\n- EfficientNetb2 , 224x \n- With just 2 epoch .\n- steps = len(generator)\n- Little touch on image pre processing for Gaussian Blur\n- loss - log_loss\n" }, { "id": 639303, "postDate": "2019-10-03T03:40:48.720Z", "content": "Inception , 224x224 , ---> 0.091
", "rawMarkdown": "Inception , 224x224 , ---> 0.091" }, { "id": 638909, "postDate": "2019-10-02T14:44:03.017Z", "content": "You guys are awesome!\nI just submit a public kernel to apply the GCP credits😜 \nStill writing the dataloader🙈
", "rawMarkdown": "You guys are awesome!\nI just submit a public kernel to apply the GCP credits😜 \nStill writing the dataloader🙈 " }, { "id": 650505, "postDate": "2019-10-16T12:59:12.447Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 636904, "postDate": "2019-09-30T10:21:05.513Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" } ], "comments": [ { "id": 651862, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-10-18T02:58:39.550000", "content": "Single-fold SE ResNext50, 512x512 raw HU image: lb 0.066 w/o tta
", "votes": 16, "replies": [ { "id": 652087, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-18T10:26:30.510000", "content": "@andy2709 Nice result! You're not using any windows? How many epochs? I tried using raw dicom, converted to float and divided by max int16 value, but wasn't able to get improvements in score. I used much simpler architecture though.
", "votes": 0, "replies": [] }, { "id": 652513, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-10-19T00:45:18.473000", "content": "I plugged in a trainable module (reference here: https://arxiv.org/pdf/1812.00572.pdf, the sigmoid one) to transform the raw hu into relevant windows.\n1 epoch took 45 mins on a v100 and i trained for 30 epochs. Data sampling is the key here :)
", "votes": 9, "replies": [] }, { "id": 652813, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-10-19T12:59:08.417000", "content": "Really interesting idea thanks for sharing the link :)
", "votes": 0, "replies": [] }, { "id": 655738, "author_name": "baijuguoxi", "author_url": "", "post_date": "2019-10-23T12:53:21.243000", "content": "hello, I want to know how you used the trainable module, the Nan would occur when I added the module?
", "votes": 0, "replies": [] }, { "id": 656392, "author_name": "Alexander Abstreiter", "author_url": "", "post_date": "2019-10-24T07:46:39.317000", "content": "Interesting Paper. Thanks for sharing!\nOut of curiosity: what kind of windows did the Window Optimization layer learn to use in your case?
", "votes": 0, "replies": [] }, { "id": 656763, "author_name": "", "author_url": "", "post_date": "2019-10-24T15:38:31.433000", "content": "", "votes": 0, "replies": [] }, { "id": 656776, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-10-24T15:55:52.120000", "content": "@lw827380444 I transformed the DICOM arrays into HU images using the rescale intercept and slope before passing it into the module. I never encountered nan when training with different architectures.
\n\n@ambpro Here's my implementation, with different weight init. from the original paper's repo. I inserted this module right before any pretrained ImageNet model and it worked pretty well for me.
\n\n```\nfrom collections import OrderedDict\nimport math \nimport torch \nimport torch.nn as nn
\n\ndef get_init_conv_params_sigmoid(ww, wl, smooth=1., upbound_value=255.):\n w = 2./ww * math.log(upbound_value/smooth - 1.)\n b = -2.*wl/ww * math.log(upbound_value/smooth - 1.)\n return (w, b)\n\n\nclass WSO(nn.Module):\n def init(self, windows=OrderedDict({\n 'brain': {'W': 80, 'L': 40},\n 'subdural': {'W': 215, 'L': 75},\n 'bony': {'W': 2800, 'L': 600},\n 'tissue': {'W': 375, 'L': 40},\n }), U=255., eps=1.):\n super(WSO, self).init()\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)\n self.bn = nn.BatchNorm2d(len(windows)\n nn.init.ones_(self.conv1x1.weight.data)\n nn.init.zeros_(self.conv1x1.bias.data)\n nn.init.ones_(self.bn.weight.data)\n nn.init.zeros_(self.bn.bias.data)\n weight, bias = self._get_window_params()\n self.register_buffer(\"weight\", weight)\n self.register_buffer(\"bias\", bias)
def _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\ndef forward(self, x):\n x = self.conv1x1(x)\n if x.dtype == torch.float16:\n self.weight = self.weight.half()\n self.bias = self.bias.half()\n x = x.mul(self.weight[None, :, None, None]) + self.bias[None, :, None, None]\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n x = self.bn(x)\n return x\n```
", "votes": 13, "replies": [] }, { "id": 657763, "author_name": "baijuguoxi", "author_url": "", "post_date": "2019-10-25T12:13:42.737000", "content": "Sorry, I had a question, that is there are some differences between your implementation and original codes in keras? Maybe, the weight and bias in WSO should be from conv1x1, but you have set the new weight and bias? please give me some instructions if possible? Thanks for your sharing!
", "votes": 0, "replies": [] }, { "id": 659765, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2019-10-28T08:35:45.457000", "content": "I think it's equivalent, slightly less efficient as both 1x1 conv and additional linear transformation used. It's also possible to directly initialize 1x1 conv with similar result.
", "votes": 1, "replies": [] }, { "id": 660142, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-10-28T18:45:51.387000", "content": "To echo what @dmytropoplavskiy said , I have mistakenly applied linear transformation twice. Thank you guys for helping me correct it 😄. \nIn the new code, I just initialized the weight and bias of the conv1x1 with the values obtained from equation (2) in the paper:
\n\n```\n self.windows = windows\n self.U = U\n self.eps = eps\n self.conv1x1 = nn.Conv2d(1, len(windows), kernel_size=1, stride=1, padding=0)
\n\n weight, bias = self._get_window_params()\n self.conv1x1.weight.data = weight[:, None, None, None]\n self.conv1x1.bias.data = bias\n\ndef _get_window_params(self):\n weight = []\n bias = []\n for _, window in self.windows.items():\n ww, wl = window[\"W\"], window[\"L\"]\n w, b = get_init_conv_params_sigmoid(ww, wl, self.eps, self.U)\n weight.append(w)\n bias.append(b)\n weight = torch.as_tensor(weight)\n bias = torch.as_tensor(bias)\n # print(weight, bias)\n return weight, bias\n\ndef forward(self, x):\n x = self.conv1x1(x)\n x = torch.sigmoid(x)\n x = x.mul(self.U)\n return x\n```
\n\nI re-ran two experiments with the old and new WSO module (same training configs, ResNet18 etc) and observed that the results were similar (difference of 1e-4).
", "votes": 3, "replies": [] }, { "id": 662726, "author_name": "AlGiLa", "author_url": "", "post_date": "2019-10-31T23:11:50.810000", "content": "Hi, I see the WSO model generate in output a [4,512,512] but the net you use (SE ResNext50) require in input [3,512,512] how did you connect the two ?
", "votes": 1, "replies": [] } ] }, { "id": 659411, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-10-27T16:02:06.410000", "content": "Resnet34 with modifications, 256 image size, 0.062 LB
", "votes": 9, "replies": [ { "id": 659431, "author_name": "Arthur Stsepanenka", "author_url": "", "post_date": "2019-10-27T16:57:20.930000", "content": "Single fold?
", "votes": 0, "replies": [] }, { "id": 659467, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-27T18:04:00.593000", "content": "That's a great score! What windowing, if any, you used, if you don't mind me asking?
", "votes": 0, "replies": [] }, { "id": 659571, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-27T23:11:16.423000", "content": "@yaroshevskiy Great! How many epochs do you train? Do you use fine tuning or transfer learning directly? Single fold?
", "votes": 0, "replies": [] }, { "id": 659972, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-10-28T14:08:21.060000", "content": "not a single fold, imagenet pretrained, 20-30 epochs
", "votes": 0, "replies": [] } ] }, { "id": 636257, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2019-09-29T05:58:10.133000", "content": "size 224x224, model: inceptionV3, single fold --> LB 0.079
", "votes": 9, "replies": [ { "id": 636564, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-29T19:30:15.190000", "content": "Thanks for your sharing! How many epoch do you use for training!
", "votes": 0, "replies": [] }, { "id": 636658, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2019-09-30T01:58:41.140000", "content": "I trained the network for 20 epochs with Adam optimizer and one cycle learning rate schedule.
", "votes": 3, "replies": [] }, { "id": 636664, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-09-30T02:29:15.783000", "content": "How long did it take for your model to go through 20 epochs?
", "votes": 0, "replies": [] }, { "id": 636666, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2019-09-30T02:35:57.663000", "content": "It took about 35 mins/epoch to run my code (written in Keras) on my local server. With the same idea + same hardware, you can reduce the time to 20 mins/epoch using PyTorch + mixed-precision training (Apex)
", "votes": 3, "replies": [] }, { "id": 636715, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-30T05:07:39.787000", "content": "@mathormad Thank you for your sharing! I have a question that did you face with overfitting problem! My model trained in all train dataset and when I trained it over 10 or more, my score just down.
", "votes": 0, "replies": [] }, { "id": 637601, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2019-10-01T06:41:18.937000", "content": "hi @linhlpv , to avoid overfitting you should split the data to training set and validation set, then monitor the loss function of the validation set, pick the best weight and predict on the test set.
", "votes": 1, "replies": [] }, { "id": 638379, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-01T20:58:35.947000", "content": "May I ask what are u using as loss function and how do you make ur epoch run so fast? My best model at the moment (Xception) can run maximum 3 epochs, even though I only tried 2 epochs till now...
", "votes": 2, "replies": [] }, { "id": 638857, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-02T13:52:44.490000", "content": "hi @mathormad Thank you for your suggestion. I see that you said your model just take 35 mins per epoch. How do you make your model run so fast? I have 4 GPUs 1080TI and I am using Resnet 50 and mixed-precision training for training in whole trainset and it takes me about 2 hours per epoch. Please help me! Thank you so much!
", "votes": 0, "replies": [] }, { "id": 638922, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-02T15:07:25.587000", "content": "@linhlpv What image size are you using? Are you loading Dicom files?
", "votes": 0, "replies": [] }, { "id": 638997, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-02T16:51:19.217000", "content": "@drhabib I use 224x224 image size and image from png file. My code apply multi GPUs and Apex bellow:\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n model = nn.DataParallel(model)
How many epochs?
", "votes": 0, "replies": [] }, { "id": 639020, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-02T17:16:02.963000", "content": "@synysterjeet I just train 3 to 5 epoch. Because training one epoch took me 2 hours :(. I dont know how it so slow. I train in whole dataset
", "votes": 0, "replies": [] }, { "id": 639184, "author_name": "Peter", "author_url": "", "post_date": "2019-10-02T21:14:07.240000", "content": "@linhlpv , try these:\n- Check your CPU/GPU load. Can your CPUs load enough images to your GPU. \n- All (most) of your cores are working?!\n- Some of the transformations are slow, add them one by one.\n- Try a bigger batch size (increased accumulation steps).\n- Some preprocessing techniques usually help. (save as Numpy arrays instead of PNGs; int16 instead of float32)
", "votes": 5, "replies": [] }, { "id": 639298, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-03T03:32:32.783000", "content": "Hi @pestipeti, thank you for mention me! I will try it and update results.
", "votes": 0, "replies": [] }, { "id": 639636, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-03T12:25:22.800000", "content": "Please enlighten us when you find which of these sugestions speed up the process.
", "votes": 1, "replies": [] }, { "id": 640803, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-04T08:42:14.323000", "content": "When I tried using more core CPUs in dataloader, I got my model run 2 faster than old version. But when I tried using bigger batch size, I got a problem that estimate time is not stable, it something was 1h30, sometime was 30 minutes. I dont know why and I am searching and try to fix that. Anyone has some ideas. Please tell my how to fix it.\nMy num_workers I used:\nnum_workers=os.cpu_count()*2
@linhlpv \nHard to say, but my bet is augmentation. It is random, and there are a few computationally heavy transformation. And if you use all of your cores for data loading, then every transformation has to wait for CPU. Try num_workers=os.cpu_count()*2 - 1 (or -2)\nProbably, you have to optimize your code a bit too. Especially when and how you move your data between the GPU and the CPU. Try some different settings, you will find the optimal, I am sure.
@pestipeti \nThank you for your suggestion! I will try more setting and optimize my code. And I will post result when I find something good for speed up code!
", "votes": 0, "replies": [] } ] }, { "id": 634491, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-26T11:13:11.197000", "content": "Custom model -- 0.069
", "votes": 8, "replies": [ { "id": 634521, "author_name": "DecentMakeover", "author_url": "", "post_date": "2019-09-26T11:57:36.373000", "content": "Just to be sure , this is a single model and not an ensemble?
", "votes": 2, "replies": [] }, { "id": 634528, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-26T12:10:34.063000", "content": "single fold without any ensembling :)
", "votes": 4, "replies": [] }, { "id": 634660, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-09-26T15:13:14.367000", "content": "is your custom pretrained on any external data or from scratch. Or you changed part of the pretrained model and called it custom
", "votes": 1, "replies": [] }, { "id": 634668, "author_name": "JM", "author_url": "", "post_date": "2019-09-26T15:34:35.683000", "content": "Why don't you just ask for the source code?
", "votes": -9, "replies": [] }, { "id": 634693, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-09-26T16:15:35.677000", "content": "@julianmukaj I find your comment rude. I asked a question and it's on @igorkrashenyi to decide whether to answer it.
\n\nAnswer to the question will not reveal his solution. Not answering it is also fine.
", "votes": 4, "replies": [] }, { "id": 634799, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-26T19:13:06.757000", "content": "ImagNet pre-trained with modifications
", "votes": 4, "replies": [] }, { "id": 636011, "author_name": "AlGiLa", "author_url": "", "post_date": "2019-09-28T15:36:38.237000", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "votes": 1, "replies": [] }, { "id": 636271, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-29T06:51:13.253000", "content": "512x512 resolution and the full dataset
", "votes": 4, "replies": [] }, { "id": 641886, "author_name": "", "author_url": "", "post_date": "2019-10-05T09:12:36.920000", "content": "", "votes": 1, "replies": [] }, { "id": 642728, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-10-06T14:45:09.547000", "content": "About 2.5 days using 4x2080ti and Apex
", "votes": 3, "replies": [] }, { "id": 646043, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-10T19:20:03.307000", "content": "@igorkrashenyi Are you running Apex on windows? Running into installation issues for Apex. I have pytorch installed on windows and it can train no prob.
", "votes": 0, "replies": [] }, { "id": 647703, "author_name": "ChinHuiC", "author_url": "", "post_date": "2019-10-13T05:39:20.960000", "content": "I tried running Apex on Windows before and it wasn't fully supported by NVIDIA. Eventually I just resorted to Ubuntu
", "votes": 1, "replies": [] } ] }, { "id": 646930, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-11T22:30:14.243000", "content": ".073 is achievable with b0 224x224. I grouped on patient and used some tricks, though
", "votes": 7, "replies": [ { "id": 646946, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-11T23:03:12.300000", "content": "Have you solved local/LB correlation problem, by the way?
", "votes": 0, "replies": [] }, { "id": 647044, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-12T02:10:43.110000", "content": "@cateek Solved it via grouping via patients. I think this is very important for establishing reliable validation score, though I have not done any control experiments. Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)
", "votes": 4, "replies": [] }, { "id": 648205, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-13T21:33:21.767000", "content": "Very interesting! I think I got it, I will test
", "votes": 0, "replies": [] }, { "id": 648590, "author_name": "Gussssss", "author_url": "", "post_date": "2019-10-14T11:51:22.537000", "content": "thanks for sharing! What's your val loss function?
", "votes": 0, "replies": [] }, { "id": 648637, "author_name": "bright", "author_url": "", "post_date": "2019-10-14T12:39:41.087000", "content": "Good work! Wouldn't it make sense for your LB score to be higher than validation if there are samples from the same patients in the test data set. Seems like you should just trust your validation
", "votes": 0, "replies": [] }, { "id": 648801, "author_name": "Michael Pieler", "author_url": "", "post_date": "2019-10-14T16:01:54.920000", "content": "@roguekk007 \"Right now I have another problem which is that my val loss is much greater than my LB (.0814 v. .073)\"\nIs this not comparing apples to oranges because you compare your validation loss to the LB metric?
", "votes": 0, "replies": [] }, { "id": 649073, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-14T22:56:24.587000", "content": "@micpie I think it is not comparing apples to oranges personally, because according to the LB probing, the weights of the metric is revealed, so my validation loss should be kind of equal to LB
", "votes": 2, "replies": [] }, { "id": 649206, "author_name": "Michael Pieler", "author_url": "", "post_date": "2019-10-15T04:16:14.073000", "content": "@roguekk007 Thanks, you are right, with the correctly weighted BCE loss they are comparable. 👍
", "votes": 0, "replies": [] }, { "id": 651546, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2019-10-17T15:35:31.050000", "content": "Hi! what do you mean by b0 ? I am unfamiliar with this term ! Edit: never mind I figured this out
", "votes": 0, "replies": [] }, { "id": 653864, "author_name": "Hilal Shaath", "author_url": "", "post_date": "2019-10-21T04:40:56.927000", "content": "B0 is one type of the efficientnet architecture s ( there is b0, b1, ... B7)
", "votes": 0, "replies": [] }, { "id": 657621, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2019-10-25T09:52:31.183000", "content": "after the competition I look forward to how you did the \"probing\" that helped you figure out the weights !
", "votes": 0, "replies": [] } ] }, { "id": 636104, "author_name": "Ian Pan", "author_url": "", "post_date": "2019-09-28T20:18:40.787000", "content": "EfficientNet-B5 @ 512 x 512 - 0.070
", "votes": 7, "replies": [ { "id": 638059, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-01T13:48:50.513000", "content": "How many epochs are you running , if you don't mind me asking? And are the images in png or jpg?
", "votes": 1, "replies": [] }, { "id": 638794, "author_name": "Ian Pan", "author_url": "", "post_date": "2019-10-02T12:43:09.850000", "content": "100 epochs, 16,000 images per epoch, loaded from DICOM.
", "votes": 4, "replies": [] }, { "id": 638918, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-02T15:03:25.150000", "content": "Wow, so how much time does it take for your model to train completely?
", "votes": 1, "replies": [] }, { "id": 639759, "author_name": "Ian Pan", "author_url": "", "post_date": "2019-10-03T14:11:23.907000", "content": "About 60-65 hours.
", "votes": 3, "replies": [] }, { "id": 639908, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-03T17:26:27.003000", "content": "Hmm, I wonder how long EfficientNet B7 would take with those same parameters, 80-100 hours? How to use GCP credit wisely.
", "votes": 1, "replies": [] } ] }, { "id": 636131, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-28T21:03:38.160000", "content": "LB: 0.080
\n\n> MODEL: Efficient Net B0 \nSIZE: 224 (PNG)\nCV: 1 Fold\nAUG: [zoom, rotate]\nTTA: No\nPRETRAINED: True\nEPOCH: 20\nLR: 1e-3
", "votes": 8, "replies": [ { "id": 636150, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-09-28T21:45:31.640000", "content": "Thanks for sharing. How is your local validation compared to LB? Which loss func are you using? I'm getting 0.088 with 4 epochs at 1e-3 at B0 and no hold outset (yet)
", "votes": 1, "replies": [] }, { "id": 636153, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-28T22:06:41.523000", "content": "I use for now torch.nn.BCEWithLogitsLoss() I did not implement yet the offical metric (I will update lter). But my val loss is almost identical to trn loss =)
I used the same settting as yours but in 4 epochs and LR=1.20E-3.\nTreating 'any' as a class to be predicted.\nLB: 0.113
", "votes": 1, "replies": [] }, { "id": 637722, "author_name": "Alex", "author_url": "", "post_date": "2019-10-01T08:11:41.100000", "content": "@drhabib May I ask if you are undersampling the training data? Compared to my training, 20 epochs are a lot.
", "votes": 1, "replies": [] }, { "id": 638056, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-01T13:44:21.557000", "content": "No under sampling... Just using random split =)
", "votes": 1, "replies": [] }, { "id": 638380, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-01T20:59:44.163000", "content": "Could you elaborate further? I am having time issues as well and would like to understand better what I can do to overcome it.
", "votes": 1, "replies": [] }, { "id": 638506, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-02T01:21:28.240000", "content": "I don't do any under sampling. I take whole data and split randomly 80% to train and 20% to validation. Let me know if you have other questions=)
", "votes": 2, "replies": [] }, { "id": 638532, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-02T02:36:02.357000", "content": "Around how long does your model take to train?
", "votes": 1, "replies": [] }, { "id": 638866, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-02T14:04:47.800000", "content": "around 12
", "votes": 0, "replies": [] }, { "id": 638987, "author_name": "jayjhlin", "author_url": "", "post_date": "2019-10-02T16:44:35.370000", "content": "Thanks for the sharing!!! Did you use any normalization(like ImageNet/competition dataset mean and std) or regularization?
", "votes": 1, "replies": [] }, { "id": 640949, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-04T10:53:01.533000", "content": "I used Imagenet stars :) Good luck
", "votes": 1, "replies": [] }, { "id": 641000, "author_name": "Alex", "author_url": "", "post_date": "2019-10-04T11:24:28.953000", "content": "@drhabib What LB score do you mange to get with EfficentNetB0, 224x224, single fold, (no TTA perhaps)? I'm also finetuning on EfficientNetB0, and so far got 0.079, which I think is much worse than yours? ;-)
", "votes": 1, "replies": [] }, { "id": 643365, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-07T13:15:52.823000", "content": "@drhabib Is it convenient to disclose the train/val split that you are using? I observe very different difference between LB/CV scores using different seeds in sklearn's train_val_split. Anyone willing to disclose their seed and validation portion that they are using?
", "votes": 1, "replies": [] }, { "id": 644422, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-08T20:01:30.357000", "content": "@akensert vanilla B0 0.70-0.72 depending on my GPU mood =)
", "votes": 1, "replies": [] }, { "id": 644424, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-08T20:04:11.937000", "content": "@roguekk007 Are you using weighted loss ? If not that willl explain why you have a big difference. My LB and CV are almost identical or have very small difference. \nRegarding splitting. \nI did not observe huge difference if I split by patient or do StratifiedKFold.
are you using any special training procedure? lr scheduling? thanks
", "votes": 1, "replies": [] }, { "id": 644668, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-09T06:55:50.137000", "content": "@drhabib Do you use Pytorch or keras?
", "votes": 1, "replies": [] }, { "id": 646224, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-11T02:17:29.773000", "content": "Verne, Nothing special, just cosine decay=)
", "votes": 1, "replies": [] }, { "id": 646225, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-11T02:18:47.673000", "content": "@custodiogabriel I use Pytroch, but at the end of the day it doesn't matter which software you use to achieve top results =)
", "votes": 1, "replies": [] }, { "id": 647755, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-13T08:19:38.617000", "content": "@drhabib I’m currently getting .066 with b0 with minimal tuning. You must also be using some cool tricks to get b0 to .070? Will you consider teaming up?
", "votes": 0, "replies": [] }, { "id": 648300, "author_name": "Gussssss", "author_url": "", "post_date": "2019-10-14T02:27:06.087000", "content": "Thanks for your sharing! How much batch size have you set?
", "votes": 0, "replies": [] }, { "id": 648329, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-14T03:22:48.360000", "content": "48, max that can fit on my local RTX 2080 (not ti)
", "votes": 0, "replies": [] }, { "id": 649544, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-10-15T14:11:13.827000", "content": "Is your current score (0.061) from a single model?\nBTW, @roguekk007 @drhabib Perhaps, the best ensemble (team)👍 . Wish you all the best!
", "votes": 0, "replies": [] }, { "id": 650015, "author_name": "pupil3", "author_url": "", "post_date": "2019-10-16T02:26:16.577000", "content": "Could you tell me what's the meaning of tta? Much Thanks
", "votes": 0, "replies": [] }, { "id": 650057, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-10-16T03:47:55.933000", "content": "@pupil3 Test Time Augmentaion (TTA)
", "votes": 0, "replies": [] }, { "id": 650089, "author_name": "pupil3", "author_url": "", "post_date": "2019-10-16T04:58:03.967000", "content": "Much Thanks
", "votes": 0, "replies": [] }, { "id": 650711, "author_name": "Hao He", "author_url": "", "post_date": "2019-10-16T16:03:45.270000", "content": "Hi @drhabib
\n\nJust curious did you do window pre-processing? Or you use Jermey’s special preprocessing method?
\n\nThanks in advance :)
", "votes": 1, "replies": [] } ] }, { "id": 652609, "author_name": "Ryan Epp", "author_url": "", "post_date": "2019-10-19T05:43:05.577000", "content": "0.069 - single fold, no tta, 300x300, 80:20 train/val split using the full dataset, custom model.
", "votes": 5, "replies": [ { "id": 653156, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-20T02:32:23.860000", "content": "If you allow me, did you use any data normalization like Multi channel windowing?
", "votes": 0, "replies": [] }, { "id": 653804, "author_name": "Ryan Epp", "author_url": "", "post_date": "2019-10-21T01:18:24.550000", "content": "Yea, I'm using Sigmoid (Brain + Subdural + Bone) Windowing from Gradient & Sigmoid Windowing.
", "votes": 2, "replies": [] }, { "id": 655230, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-22T20:49:59.113000", "content": "Did you use GroupKfold using PatientID? Many people are using... Do you think this is essential to increase the score?
", "votes": 0, "replies": [] }, { "id": 655377, "author_name": "Ryan Epp", "author_url": "", "post_date": "2019-10-23T01:47:54.963000", "content": "Yea, I haven't been doing multiple folds yet and I'm not using that function specifically, but I am splitting my train/validation sets using PatientId.
\n\nIt's not because splitting that way necessarily improves the score of a single model, but if you don't, you risk leaking info to your validation set and you can overfit without realizing it. Check out Similarity between train and test images
\n\nI imagine it also helps improve model diversity when ensembling models trained on different folds, but idk for sure.
", "votes": 1, "replies": [] }, { "id": 655409, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-23T02:47:31.580000", "content": "I understand! Thanks for the feedback! My models usually have very low val_loss and very different from LB (larger), I think that if I do GroupKFold (or other techniques) I think I will have a much closer val_loss to LB then! I will see this kernel! Thanks!
", "votes": 1, "replies": [] }, { "id": 664619, "author_name": "Ryan Epp", "author_url": "", "post_date": "2019-11-04T01:28:49.490000", "content": "Update 11/3: \n0.066- single fold, no tta, 300x300, 80:20 train/val split using the full dataset (no CQ500 data), custom model and a custom data generator.
", "votes": 1, "replies": [] }, { "id": 664816, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-11-04T09:13:32.470000", "content": "By custom you meant you're not using imagenet weights, or it's just a custom head?
", "votes": 0, "replies": [] } ] }, { "id": 651109, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-17T03:45:17.483000", "content": "efficientnet-b2 \n256x256\nw/o 3 windowing preprocess\npublicLB: 0.069
", "votes": 5, "replies": [ { "id": 651114, "author_name": "takuoko", "author_url": "", "post_date": "2019-10-17T04:03:02.377000", "content": "Cool!
\nWhat is the meaning of w/o?
I think he means without
", "votes": 0, "replies": [] }, { "id": 651143, "author_name": "takuoko", "author_url": "", "post_date": "2019-10-17T05:04:27.393000", "content": "So he didn't use windowing?
", "votes": 0, "replies": [] }, { "id": 651355, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-17T11:48:22.430000", "content": "I used this kernel's windowing.\nI'm trying Appian's windowing now.
", "votes": 0, "replies": [] }, { "id": 651419, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-17T13:28:11.613000", "content": "@uiiurz1 That's a great result! Did you normalize input images?
", "votes": 0, "replies": [] }, { "id": 651430, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-17T13:34:40.040000", "content": "I normalized input images with ImageNet's means and stds.
", "votes": 1, "replies": [] }, { "id": 653160, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-20T02:36:08.850000", "content": "@uiiurz1 how many epochs?
", "votes": 0, "replies": [] } ] }, { "id": 651127, "author_name": "datasaurus", "author_url": "", "post_date": "2019-10-17T04:26:00.700000", "content": "EfficientNet-B0\n224x224\nPublic LB: 0.069 without TTA
", "votes": 6, "replies": [ { "id": 651192, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-17T06:56:53.323000", "content": "That's a great result with such a simple setup! May I wonder, did you use some special windowing? Have you used all images, or found a way to remove uninformative ones? I've done quite a lot of experiments and already decided that it's simply impossible for such to go below 0.75 with small resolution w/o some tricks. Thanks in advance
", "votes": 0, "replies": [] }, { "id": 651199, "author_name": "Rajnish Chauhan", "author_url": "", "post_date": "2019-10-17T07:05:14.850000", "content": "try undersampling
", "votes": 0, "replies": [] }, { "id": 651206, "author_name": "datasaurus", "author_url": "", "post_date": "2019-10-17T07:13:23.463000", "content": "I'm using all of the images and averaging 5 folds. Nothing fancy with the network - my parameters are identical to DrHBs in earlier this thread. The image preprocessing is the key, but I think it's best if I share later 😉
", "votes": 3, "replies": [] }, { "id": 651211, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-17T07:16:01.363000", "content": "Thanks! By preprocessing you mean window/no window, window parameters, and normalization, right?
", "votes": 0, "replies": [] }, { "id": 651215, "author_name": "Rajnish Chauhan", "author_url": "", "post_date": "2019-10-17T07:21:44.177000", "content": "@anjum48 coulld you please describe about 5 folds , how you are doing K-Fold . Thanks
", "votes": 0, "replies": [] }, { "id": 651217, "author_name": "datasaurus", "author_url": "", "post_date": "2019-10-17T07:25:00.843000", "content": "@cateek Yes, but there are more ways than just windowing to make sure the images you pass to the model are as informative as possible\n@rajnishe I'm using GroupKfold using PatientID as the grouping variable
Interesting!
\nI am looking forward to see your preprocessing after competition end;)
One more thing. Do you treat any class in some specific way or just 6 classes with weights? I've tried some variation of 2-stage model but it didn't work well
", "votes": 0, "replies": [] }, { "id": 651306, "author_name": "datasaurus", "author_url": "", "post_date": "2019-10-17T09:56:00.483000", "content": "No, I'm using the standard 6 class loss already shared here. I've not had a chance to test the 2-step pipeline yet
", "votes": 1, "replies": [] }, { "id": 651335, "author_name": "Sugawarya", "author_url": "", "post_date": "2019-10-17T10:51:03.737000", "content": "@anjum48 \nI'm using GroupKfold using PatientID as the grouping variable
Are CV score and LB score correlated?\nin fact, my CV score with GroupKfold using PatientID doesnt help to early stopping...
", "votes": 0, "replies": [] } ] }, { "id": 634541, "author_name": "OrKatz", "author_url": "", "post_date": "2019-09-26T12:29:04.213000", "content": "inceptionv4 -> 0.78
", "votes": 5, "replies": [ { "id": 636010, "author_name": "AlGiLa", "author_url": "", "post_date": "2019-09-28T15:35:39.037000", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "votes": 1, "replies": [] } ] }, { "id": 642485, "author_name": "Appian", "author_url": "", "post_date": "2019-10-06T06:21:46.473000", "content": "LB 0.079 (update 0.074)\n- se_resnext50_32x4d\n- 224x224\n- 2 epochs\n- hflip, crop
", "votes": 6, "replies": [ { "id": 643370, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-07T13:25:55.800000", "content": "@appian Hi! Is it convenient to disclose your train/val split? How is your correlation between train/test LB score?
", "votes": 0, "replies": [] }, { "id": 643615, "author_name": "Appian", "author_url": "", "post_date": "2019-10-07T17:29:05.250000", "content": "The train valid split is 80:20. I haven't made enough submissions to see the correlation but the difference between public lb and local cv is relatively small so far.
\n\nnice result. i trained the same network on 200k images 256 resolution brain window and have a 0.14 result. what kind of images and windowing are you using? thanks
", "votes": 0, "replies": [] }, { "id": 646222, "author_name": "Nicholas Lyu", "author_url": "", "post_date": "2019-10-11T02:12:57.783000", "content": "@appian What LR and scheduling are you using? I am unable to replicate your results. I am already grouping on patient
", "votes": 0, "replies": [] }, { "id": 646609, "author_name": "Mu Song", "author_url": "", "post_date": "2019-10-11T13:31:57.627000", "content": "@appian Great, why are you public LB and Local LB is so closer? my pubic LB always larger than local LB, about 0.02.
", "votes": 0, "replies": [] }, { "id": 646680, "author_name": "Appian", "author_url": "", "post_date": "2019-10-11T15:03:42.267000", "content": "My guess is that you randomly split the data for train/val. Because one patient is at least composed of more than 20 slices, it introduces a leak especially when adjacent slices are split between train and val. Somethinkg like GroupKFold can deal with it.
", "votes": 2, "replies": [] }, { "id": 646688, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-11T15:17:58.320000", "content": "from my own experience, I could not get stable cv without using weighted log-loss. I think a lot of the issues with cv not looking like LB is due to not using the proper loss function and not due to how you split your training/validation sets. And I think this makes sense - if you're optimizing your model to a different loss, CV is not going to reflect on LB closely. I used random split and weighted log-loss and my CV and LB look exactly the same.
", "votes": 0, "replies": [] }, { "id": 647002, "author_name": "Mu Song", "author_url": "", "post_date": "2019-10-12T00:59:41.927000", "content": "Thanks@Appian@Tim Yee, I split the data for train/val using PatientID, so the slices for one patient ara all in train or val. Maybe my weights of logloss is [1,1,1,1,1,1], I'll try more, thanks.
", "votes": 0, "replies": [] } ] }, { "id": 646623, "author_name": "takuoko", "author_url": "", "post_date": "2019-10-11T13:42:08.373000", "content": "Seresnext50->LB0.082\nimg size->224*224
", "votes": 3, "replies": [ { "id": 647795, "author_name": "takuoko", "author_url": "", "post_date": "2019-10-13T09:39:28.350000", "content": "update\nSeresnext50->LB0.078\nimg size->224*224\nsubdural_window preprocessing
", "votes": 0, "replies": [] }, { "id": 659957, "author_name": "Verne", "author_url": "", "post_date": "2019-10-28T13:45:44.227000", "content": "for how many epochs do you train? when do you notice overfitting? thanks!
", "votes": 0, "replies": [] } ] }, { "id": 638792, "author_name": "Jayaram", "author_url": "", "post_date": "2019-10-02T12:37:03.580000", "content": "EfficientNet B0 - 512x512 --> 0.078
", "votes": 3, "replies": [ { "id": 638801, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-02T12:52:03.430000", "content": "If you don't mind me asking, what's your validation split, and how many epochs?
", "votes": 0, "replies": [] }, { "id": 644817, "author_name": "Jayaram", "author_url": "", "post_date": "2019-10-09T11:44:16.667000", "content": "Random split 95% train & 5% validation... trained for 10 epochs
", "votes": 0, "replies": [] }, { "id": 644975, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-09T15:35:13.177000", "content": "And how do you fit the 512 images in the B0 model? It takes 224 by default right?
", "votes": 0, "replies": [] }, { "id": 644978, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-09T15:37:54.427000", "content": "You can fit any image size. You can put in 128x128 if you want.
", "votes": 0, "replies": [] }, { "id": 644987, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-09T16:03:28.053000", "content": "But when I try to put 512x512 images in my B0 model it gives me \"invalid argument 0: sizes of tensor must match except in dimension 0\" error
", "votes": 0, "replies": [] }, { "id": 645368, "author_name": "Gabriel", "author_url": "", "post_date": "2019-10-10T03:46:16.700000", "content": "Do you train using all images?
", "votes": 0, "replies": [] }, { "id": 648422, "author_name": "Jayaram", "author_url": "", "post_date": "2019-10-14T06:43:14.103000", "content": "yes, i do use all images
", "votes": 1, "replies": [] } ] }, { "id": 634225, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-09-26T03:48:01.747000", "content": "Using only 30,000 samples(15k positive and 15k negative)
\n\nEfficient Net B4 10 epochs 256x256 - 0.113\nEfficient Net B4 3 epochs 256x256 - 0.107
\n\nLB improving with less number of epochs? (Edit - Severely Overfitting?)
", "votes": 3, "replies": [ { "id": 634256, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-09-26T05:23:20.430000", "content": "0.098 4 epochs. I haven't tried greater than 4 epochs yet. I already used this weeks GPU quota.
", "votes": 2, "replies": [] }, { "id": 634305, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-09-26T06:20:06.027000", "content": "Are you using the full dataset? How long it takes for 1 epoch?
", "votes": 2, "replies": [] }, { "id": 634796, "author_name": "Bo Peng", "author_url": "", "post_date": "2019-09-26T19:08:20.097000", "content": "In my opinon, its not about the number of epochs, but about not overfitting. 🙂
", "votes": 2, "replies": [] }, { "id": 636567, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-29T19:49:20.130000", "content": "@bopengiowa Do you have a suggestion about removing overfitting? Thank you!
", "votes": 1, "replies": [] }, { "id": 636626, "author_name": "Bo Peng", "author_url": "", "post_date": "2019-09-29T22:59:08.330000", "content": "Common for keras is it saves the weights from the best val score correct? So that should solve it right there. I think there are other learning rate things that people do such as scheduling and decay. Could also check if local val score matches with leaderboard scores.
", "votes": 2, "replies": [] }, { "id": 636718, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-30T05:09:07.800000", "content": "@bopengiowa thank you! I will try it 👍
", "votes": 1, "replies": [] } ] }, { "id": 653855, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-21T04:02:27.207000", "content": "efficientnet-b2\n410x410\nsingle fold\nhflip tta\npublicLB: 0.064
", "votes": 4, "replies": [ { "id": 653896, "author_name": "ChinHuiC", "author_url": "", "post_date": "2019-10-21T06:11:57.740000", "content": "Thanks for sharing! Are you using 5 epochs, png images as you mentioned before also?
", "votes": 0, "replies": [] }, { "id": 653901, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-21T06:40:36.767000", "content": "Yes.
", "votes": 2, "replies": [] }, { "id": 654556, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-10-22T02:52:26.817000", "content": "Cool. Could you please share the windowing method?
", "votes": 0, "replies": [] }, { "id": 654594, "author_name": "Ryan Epp", "author_url": "", "post_date": "2019-10-22T04:22:20.083000", "content": "Are you using the new CQ500 dataset?
", "votes": 0, "replies": [] }, { "id": 654637, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-22T05:56:50.023000", "content": "I used Appian's windowing and didn't use CQ500.
", "votes": 0, "replies": [] }, { "id": 654643, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-10-22T06:12:18.353000", "content": "How long one epoch takes
", "votes": 0, "replies": [] }, { "id": 654732, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-22T08:30:44.570000", "content": "4-5 hours
", "votes": 0, "replies": [] }, { "id": 654897, "author_name": "AlGiLa", "author_url": "", "post_date": "2019-10-22T12:56:27.053000", "content": "Thanks for sharing, I see the efficientnet-b2 was trained with input imm 224x224. How did our shape your net to receive in input imm 410x410 ?
", "votes": 0, "replies": [] }, { "id": 655399, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-10-23T02:30:16.160000", "content": "@uiiurz1 Nice results indeed. If I understand correctly, you only used hflip during training, I was just wondering how you apply tta? Thanks once again.
", "votes": 0, "replies": [] } ] }, { "id": 651563, "author_name": "nan", "author_url": "", "post_date": "2019-10-17T15:58:05.833000", "content": "Update: \nresnet50, 256x256, train on 150k images per epochs, split by patient, lb 0.071 no tta single fold
", "votes": 4, "replies": [ { "id": 651594, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-17T16:38:34.343000", "content": "Are you sampling new 150K images every epoch?
", "votes": 0, "replies": [] }, { "id": 651599, "author_name": "nan", "author_url": "", "post_date": "2019-10-17T16:44:32.797000", "content": "Yes, 150k images is randomly selected
", "votes": 0, "replies": [] } ] }, { "id": 636711, "author_name": "nan", "author_url": "", "post_date": "2019-09-30T04:56:56.933000", "content": "resnet34, image 256x256, 10k pos, 10k neg, single fold, 0.088 lb\nUpdate: resnet34, image 256x256, single fold -> 0.078 lb
", "votes": 4, "replies": [ { "id": 636882, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-09-30T09:57:56.167000", "content": "I think you mean 0.088?
", "votes": 1, "replies": [] }, { "id": 637804, "author_name": "nan", "author_url": "", "post_date": "2019-10-01T09:16:03.360000", "content": "I then use the the best checkpoint and finetune on the whole dataset, get lb 0.081 in the first epochs. However, but when I continue to finetune, the lb get worse, even though local cv still improve. Previous cv score is consistent w/ lb. I currently have no idea why. Still working on that :))
", "votes": 1, "replies": [] }, { "id": 637833, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-01T09:50:05.150000", "content": "@dattran2346 your model seem overffiting :))
", "votes": 0, "replies": [] }, { "id": 637896, "author_name": "nan", "author_url": "", "post_date": "2019-10-01T10:55:51.310000", "content": "My train and val loss still improving, hardly say it is overfit. Maybe the problem is how I up-train from under-sampling to all dataset 🤔
", "votes": 0, "replies": [] }, { "id": 638427, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-01T22:17:59.863000", "content": "Forgive my stupidity, but what do u mean by 10k pos, 10k neg and r34?
", "votes": 1, "replies": [] }, { "id": 638502, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-02T01:20:22.140000", "content": "he means: \n10 k postive examples\n10 k negative example \nresent 34 (r34?)\n256 image size
", "votes": 3, "replies": [] }, { "id": 638726, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-02T11:02:11.877000", "content": "U mean he only uses 20k IDs of the train data, 10k which any = 0 and 10k which any = 1?
", "votes": 1, "replies": [] }, { "id": 638734, "author_name": "DrHB", "author_url": "", "post_date": "2019-10-02T11:13:50.710000", "content": "yep
", "votes": 1, "replies": [] }, { "id": 638745, "author_name": "nan", "author_url": "", "post_date": "2019-10-02T11:34:38.060000", "content": "Yes, I use resnet 34 with image size of 256 and train under-sampling now to speed up experiment. The samples are selected such that each diseases have the same number of sample, and the total number of positive and negative are the same.
", "votes": 1, "replies": [] }, { "id": 643971, "author_name": "juliencs", "author_url": "", "post_date": "2019-10-08T07:22:03.347000", "content": "Thanks @dattran2346 for this information.
\n\nI'm curious, I've been trying the 10k +/10k - approach too since I lack access to a big GPU and want to iterate not too slowly. I imagine that your CV numbers are different from the public LB scores right? Since our validation set contains way more examples from minority classes, I'd expect the CV score to be significantly worse than the public LB scores. Is it the case for you too?
", "votes": 0, "replies": [] }, { "id": 644092, "author_name": "nan", "author_url": "", "post_date": "2019-10-08T10:00:43.113000", "content": "Hi @juliencs \nFor me, the local CV is about 0.08 at epoch 75, and the LB for that checkpoint is 0.086. How do you split your CV, do you stratify base on labels, or base on metadata like Patient or Study ID. Do you pick hard sample and train on them.\nMy CV is stratified by labels, and I just pick random samples.
", "votes": 1, "replies": [] }, { "id": 644147, "author_name": "juliencs", "author_url": "", "post_date": "2019-10-08T12:15:31.803000", "content": "Ooooh, 75 epochs. I never tried that long. My CV is same as yours it seems, but I never went past 15 epochs, and the loss was still pretty high by then, hence my question.
\n\n75 epochs must last a long time. The idea for me to take a really small sample was to iterate quicker, but if we have to wait for 75 epochs on the smaller sample to reach a minimum, maybe it's not worth it, and taking a bigger (and thus more diverse) sample, for less epochs, might yield at least similar results (just thinking out loud here really).
\n\nThanks for your answer. I'll keep experimenting.
", "votes": 0, "replies": [] }, { "id": 644155, "author_name": "nan", "author_url": "", "post_date": "2019-10-08T12:27:18.043000", "content": "Yes, totally agree with you, 75 epochs is way too long for a under-sample model. And train for that long may actually hurt the model performance. Another approach is to filter out samples in the training, i.e, remove obvious images that have no brain in it for example.
", "votes": 0, "replies": [] }, { "id": 644308, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-08T15:49:05.910000", "content": "So you remove obvious images with no brain in it by hand? or you have some filter function? I could not look at more than 100 even If I tried.
", "votes": 0, "replies": [] }, { "id": 644313, "author_name": "nan", "author_url": "", "post_date": "2019-10-08T15:55:26.367000", "content": "You can use a simple brain window for this task, just filter by the HU of the brain.
", "votes": 0, "replies": [] }, { "id": 644348, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-08T16:55:48.047000", "content": "But just by filtering by brain window you cant judge whether there is brain or not right? And even the brain window values can vary, there isn't really an optimum range right?
", "votes": 0, "replies": [] }, { "id": 644354, "author_name": "nan", "author_url": "", "post_date": "2019-10-08T17:07:23.587000", "content": "According to wiki the HU value of the brain is just a narrow range, you can just filter images that don't have (or very small number of pixel) of brain HU. There are about 2-3 images per study where there is no brain (just empty, or part of the CT machine, or just top of the skull).
", "votes": 0, "replies": [] } ] }, { "id": 636110, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-09-28T20:36:56.447000", "content": "ResNeXt-101 32x8d: 0.082
\n\nKernel with benchmark (0.089): https://www.kaggle.com/taindow/pytorch-resnext-101-32x8d-benchmark
", "votes": 4, "replies": [ { "id": 636300, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-09-29T08:15:02.430000", "content": "I'm sure it sounds stupid, but what's the difference between your kernel with benchmark and your actual model that gives better results? Some extra preprocessing? Because I am also using the ResNeXt-101 model and getting 0.089
", "votes": 0, "replies": [] }, { "id": 636431, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-09-29T14:14:35.937000", "content": "Same dataset, but more epochs and splitting the data to monitor validation performance for the competition metric :)
", "votes": 2, "replies": [] }, { "id": 636471, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-09-29T15:30:23.563000", "content": "Thanks :) I got 0.086 by using 32x16d benchmark
", "votes": 1, "replies": [] }, { "id": 638111, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-01T14:43:36.420000", "content": "@taindow Thank you for sharing! It sounds stupid but what metric do you use for your kernel.
", "votes": 0, "replies": [] }, { "id": 638381, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-01T21:02:57.100000", "content": "U mean you are using the metric as a loss function in your model? My model ran for 2 epochs and, between epochs 1 and 2, my validation score (BCE) increased while the competition metric continued going down (same as training loss). Are u having this problem as well?
", "votes": 0, "replies": [] } ] }, { "id": 634704, "author_name": "Abhilash Awasthi", "author_url": "", "post_date": "2019-09-26T16:37:40.677000", "content": "resnet34 -- 0.084
", "votes": 4, "replies": [ { "id": 636009, "author_name": "AlGiLa", "author_url": "", "post_date": "2019-09-28T15:35:10.650000", "content": "Hi,\nwhat input image size you used ? did you used the full number of images of just a sub portion ?
", "votes": 1, "replies": [] }, { "id": 636278, "author_name": "Abhilash Awasthi", "author_url": "", "post_date": "2019-09-29T07:08:15.800000", "content": "Image size - 512 x 512...all the images used
", "votes": 3, "replies": [] } ] }, { "id": 660412, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-29T04:56:29.687000", "content": "a simple resnet34 without modification on 288x288 get decent results of LB 0.080 (local cv 0.057)\nsee: https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/109272#latest-660570
\n\n```\nwindowing input:
\n\nbrain = window_image(hu, 40, 80)\nsubdural = window_image(hu, 80, 200)\nsoft_tissue = window_image(hu, 40, 380)\nsingle model, TTA = ['null', 'flip_left_right']
\n\n```
", "votes": 1, "replies": [ { "id": 660511, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-29T08:38:39.957000", "content": "update: the same setup achieve LB 0.70 after increasing size to 512x512 input
", "votes": 1, "replies": [] }, { "id": 660530, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-29T09:17:11.380000", "content": "other results:\nEfficientNet B4 224x224 - 0.081\nSeResNext50 512x512 - 0.070
", "votes": 1, "replies": [] }, { "id": 660533, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-29T09:28:34.683000", "content": "@hengck23 interested in teaming up?
", "votes": 0, "replies": [] }, { "id": 660671, "author_name": "Eek The Cat", "author_url": "", "post_date": "2019-10-29T13:02:12.743000", "content": "@hengck23 Do you have any ideas on how to go below 0.065? It seems that a well trained, reasonably complex model on full resolution and one of publicly known windowing variants will give ~0.07-0.074 on single fold and ~0.065-0.067 ensembling several folds
", "votes": 0, "replies": [] }, { "id": 660691, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-29T13:33:36.873000", "content": "the training loss can go as low as 0.030~0.040. Hence with good generalization, it think you can get 0.060 without tricks.
\n\ntry ensemble network of different architecture, difference size input, different windowing, etc
\n\ntry tta of crop, scale, flip
\n\ntry smooth your output by
\n\n```\nlogit = net(input)\nlogit += some small noise
\n\nloss =(logit,truth)\nloss.backward()
\n\n```\n etc ...
", "votes": 2, "replies": [] }, { "id": 660809, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-29T16:31:02.057000", "content": "https://www.kaggle.com/braquino/correct-images-sequece
\n\nyou can try to to use nearby slice of the same patient to post correct prediction results
", "votes": 0, "replies": [] } ] }, { "id": 644742, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-09T09:10:38.833000", "content": "update\nEfficientNet B0 - 256x256 --> 0.072
", "votes": 1, "replies": [ { "id": 644759, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-09T09:45:02.697000", "content": "Are you loading images from dicom? And how many epochs?
", "votes": 0, "replies": [] }, { "id": 644875, "author_name": "Gurgel", "author_url": "", "post_date": "2019-10-09T13:05:31.317000", "content": "@uiiurz1 Hi, do u use Keras? I would like to know if it is allowed to import EfficientNet from some repository, since it doesn't come built in Keras and I didn't see anybody anouncing this model in the external data thread.
", "votes": 0, "replies": [] }, { "id": 645235, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-09T23:25:53.100000", "content": "Do you train on all images or just a sub-set?
", "votes": 0, "replies": [] }, { "id": 645522, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-10T07:14:39.773000", "content": "Do you have a validation split?
", "votes": 0, "replies": [] }, { "id": 645611, "author_name": "4ui_iurz1", "author_url": "", "post_date": "2019-10-10T09:38:43.407000", "content": "Thanks, I guess your preprocessing must be very good to get you that LB score
", "votes": 0, "replies": [] }, { "id": 651186, "author_name": "Rajnish Chauhan", "author_url": "", "post_date": "2019-10-17T06:45:46.777000", "content": "pip install efficientnet\npip install keras_efficientnet
", "votes": 0, "replies": [] } ] }, { "id": 641962, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2019-10-05T11:32:11.260000", "content": "VGG19 - 224x224 - 0.073
", "votes": 1, "replies": [ { "id": 641990, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-05T12:10:40.497000", "content": "Thank you for sharing! Do you mind if I ask How many epoch and data do you use for training?
", "votes": 0, "replies": [] }, { "id": 642340, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2019-10-05T23:10:22.213000", "content": "I'm using model checkpoint based on val_loss, but I think it was around 20 epochs. And I'm using one-fold with 80% for train e 20% for validation.
", "votes": 1, "replies": [] }, { "id": 642374, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-06T00:56:10.057000", "content": "@fernandocamargo how long your model take to training one epoch? Thank you for replying me!
", "votes": 0, "replies": [] }, { "id": 643596, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2019-10-07T17:01:38.393000", "content": "It takes around 40min.
", "votes": 1, "replies": [] }, { "id": 643667, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-07T18:55:40.143000", "content": "Nice work! I am trying to speed my code for fast training. Could you give me some suggestion for this? Thank you so much!
", "votes": 0, "replies": [] }, { "id": 643901, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-10-08T05:27:28.427000", "content": "The easiest way to speed up training would be using FP16 computations. Also I posted a paper here which talks about only updating the top losses when backpropagating the error. It can also provide some speed up.
", "votes": 2, "replies": [] }, { "id": 643957, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-08T07:00:29.720000", "content": "hi @salilm23 . Thanks for your suggestion. May I ask you How can I find implementation of this paper?
", "votes": 0, "replies": [] }, { "id": 643970, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-10-08T07:18:58.707000", "content": "Hey @linhlpv , implementation is also linked in the post :)
\n\nEdit - Sorry, the implementation linked to the paper has now been removed
", "votes": 2, "replies": [] }, { "id": 643998, "author_name": "steelrose", "author_url": "", "post_date": "2019-10-08T07:59:37.797000", "content": "@salilm23 - hi, do you have a local copy of the code? maybe you could upload it somewehere
", "votes": 0, "replies": [] }, { "id": 644117, "author_name": "Salil Mishra", "author_url": "", "post_date": "2019-10-08T10:52:52.180000", "content": "Well, the implementation seems to be back now. https://anonymous.4open.science/r/c6d4060d-bdac-4d31-839e-8579650255b3/
", "votes": 1, "replies": [] } ] }, { "id": 640833, "author_name": "Alimbekov Renat [dsmlkz]", "author_url": "", "post_date": "2019-10-04T09:02:46.363000", "content": "ResNeXt 32x8d - 0.087 with 50/50 % sampler
", "votes": 1, "replies": [ { "id": 640848, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-04T09:13:19.170000", "content": "What do you mean by a 50/50% sampler exactly?
", "votes": 0, "replies": [] }, { "id": 640860, "author_name": "Alimbekov Renat [dsmlkz]", "author_url": "", "post_date": "2019-10-04T09:20:50.817000", "content": "I took all data with any == 1 and add randomly with any !=0 so that the ratio would be 50 to 50 %
", "votes": 0, "replies": [] }, { "id": 640877, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-04T09:34:59.760000", "content": "I'm not sure I'm getting you here, wouldn't any==1 and any !=0 give the same images?
", "votes": 0, "replies": [] }, { "id": 641073, "author_name": "Alimbekov Renat [dsmlkz]", "author_url": "", "post_date": "2019-10-04T12:24:26.867000", "content": "Sorry. It is my mistake. I mean any==1 and any !=1
", "votes": 1, "replies": [] }, { "id": 641579, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-04T20:01:27.870000", "content": "And how many epochs?
", "votes": 0, "replies": [] }, { "id": 641757, "author_name": "Alimbekov Renat [dsmlkz]", "author_url": "", "post_date": "2019-10-05T03:48:24.087000", "content": "1 epoch)
", "votes": 0, "replies": [] } ] }, { "id": 650323, "author_name": "takuoko", "author_url": "", "post_date": "2019-10-16T09:21:56.890000", "content": "Seresnext50->LB0.072\nimg size->512*512\nthree windowing preprocessor
", "votes": 2, "replies": [ { "id": 659275, "author_name": "Mohammad Azam Khan", "author_url": "", "post_date": "2019-10-27T11:46:14.093000", "content": "@takuok I am just wondering if you can share what augmentations you are using? Also, is the result of a single fold?
", "votes": 0, "replies": [] } ] }, { "id": 647449, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-10-12T16:35:22.083000", "content": "With raw input from dcmread, random ShuffleSplit, no tta,\nB0 0.079 -> 0.074 224x224
", "votes": 2, "replies": [ { "id": 647547, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-12T20:39:20.787000", "content": "Epochs?
", "votes": 0, "replies": [] }, { "id": 647550, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-10-12T20:45:03.347000", "content": "I'm pretty sure that's subjective to many other factors. For .95/.05 split, default class weight it seems 4-6 epochs works well and then starts to overfit real bad real fast afterwards.
", "votes": 0, "replies": [] }, { "id": 647551, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-12T20:46:41.027000", "content": "Noticed that for resnext, more than 4 epochs is generally overfittnig
", "votes": 0, "replies": [] }, { "id": 650016, "author_name": "pupil3", "author_url": "", "post_date": "2019-10-16T02:26:43.553000", "content": "could you tell me what's the meaning of tta?\nMuch thanks
", "votes": 0, "replies": [] }, { "id": 650091, "author_name": "Josh Myers", "author_url": "", "post_date": "2019-10-16T04:59:59.483000", "content": "Test-time augmentation: https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/
", "votes": 0, "replies": [] }, { "id": 659954, "author_name": "Verne", "author_url": "", "post_date": "2019-10-28T13:41:23.760000", "content": "what sort of training loss and validation loss do you have just before it starts to overfit? thanks
", "votes": 0, "replies": [] } ] }, { "id": 642349, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2019-10-05T23:31:55.780000", "content": "InceptionResnetv2 -224*224 --> 0.086
", "votes": 2, "replies": [] }, { "id": 636490, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-09-29T16:18:37.737000", "content": "ResNeXt-101 32x16d : 0.086
", "votes": 2, "replies": [ { "id": 636568, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-29T19:50:08.533000", "content": "Thanks for your sharing! How many epoch that you train you model?
", "votes": 0, "replies": [] }, { "id": 636657, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-09-30T01:56:38.690000", "content": "1 epoch for now, due to size of dataset even that takes a long time
", "votes": 1, "replies": [] }, { "id": 636714, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-09-30T05:04:37.243000", "content": "Nice work! I am also using ResNeXt 32x16d and archive 0.088 :D. i using Adam optimizer with lr=1e-5. Thanks for aswering me!
", "votes": 0, "replies": [] }, { "id": 638923, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-02T15:07:56.093000", "content": "Did you change your model to better your accuracy? Or did you change some hyperparameters like learning rate?
", "votes": 0, "replies": [] }, { "id": 641589, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-04T20:06:24.517000", "content": "Now I use RAdam for optimizer. You should try it.
", "votes": 0, "replies": [] }, { "id": 643083, "author_name": "Arijit Gupta", "author_url": "", "post_date": "2019-10-07T04:37:05.440000", "content": "So you're using keras then?
", "votes": 0, "replies": [] }, { "id": 643170, "author_name": "L3viEvil", "author_url": "", "post_date": "2019-10-07T07:52:27.010000", "content": "I have both Keras and Pytorch Base code.
", "votes": 0, "replies": [] } ] }, { "id": 649196, "author_name": "Kun Jiang ", "author_url": "", "post_date": "2019-10-15T03:54:12.313000", "content": "VGG19 \nepoch20\nLB：0.082
", "votes": 0, "replies": [ { "id": 649393, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2019-10-15T09:42:36.517000", "content": "Did you train it on the whole dataset?
", "votes": 0, "replies": [] }, { "id": 659780, "author_name": "Kun Jiang ", "author_url": "", "post_date": "2019-10-28T08:54:39.633000", "content": "undata: VGG19_BN\nepoches:15\nLB:0.075
", "votes": 0, "replies": [] }, { "id": 659781, "author_name": "Kun Jiang ", "author_url": "", "post_date": "2019-10-28T08:55:37.440000", "content": "yes， I use the whole data, I will try three window process, but i have no a hug disk
", "votes": 0, "replies": [] } ] }, { "id": 643575, "author_name": "", "author_url": "", "post_date": "2019-10-07T16:28:33.317000", "content": "", "votes": 0, "replies": [ { "id": 650014, "author_name": "", "author_url": "", "post_date": "2019-10-16T02:25:52.067000", "content": "", "votes": 0, "replies": [] }, { "id": 653123, "author_name": "", "author_url": "", "post_date": "2019-10-20T01:13:20.010000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 660794, "author_name": "", "author_url": "", "post_date": "2019-10-29T16:09:22.333000", "content": "", "votes": -1, "replies": [] }, { "id": 648597, "author_name": "", "author_url": "", "post_date": "2019-10-14T12:01:47.100000", "content": "", "votes": 0, "replies": [] }, { "id": 648113, "author_name": "", "author_url": "", "post_date": "2019-10-13T19:03:22.423000", "content": "", "votes": 0, "replies": [ { "id": 648470, "author_name": "", "author_url": "", "post_date": "2019-10-14T08:35:28.220000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 647975, "author_name": "", "author_url": "", "post_date": "2019-10-13T15:02:09.523000", "content": "", "votes": 0, "replies": [ { "id": 648012, "author_name": "", "author_url": "", "post_date": "2019-10-13T16:01:55.150000", "content": "", "votes": 0, "replies": [] }, { "id": 648183, "author_name": "", "author_url": "", "post_date": "2019-10-13T20:48:46.207000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 647830, "author_name": "", "author_url": "", "post_date": "2019-10-13T11:06:54.310000", "content": "", "votes": 0, "replies": [ { "id": 656649, "author_name": "", "author_url": "", "post_date": "2019-10-24T13:50:13.223000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 647633, "author_name": "", "author_url": "", "post_date": "2019-10-13T02:27:32.177000", "content": "", "votes": 0, "replies": [] }, { "id": 642803, "author_name": "", "author_url": "", "post_date": "2019-10-06T17:13:29.340000", "content": "", "votes": 0, "replies": [ { "id": 643184, "author_name": "", "author_url": "", "post_date": "2019-10-07T08:06:30.730000", "content": "", "votes": 0, "replies": [] }, { "id": 644125, "author_name": "", "author_url": "", "post_date": "2019-10-08T11:21:06.623000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 642646, "author_name": "", "author_url": "", "post_date": "2019-10-06T12:36:43.493000", "content": "", "votes": 0, "replies": [] }, { "id": 639303, "author_name": "", "author_url": "", "post_date": "2019-10-03T03:40:48.720000", "content": "", "votes": 0, "replies": [] }, { "id": 638909, "author_name": "", "author_url": "", "post_date": "2019-10-02T14:44:03.017000", "content": "", "votes": 0, "replies": [] }, { "id": 650505, "author_name": "", "author_url": "", "post_date": "2019-10-16T12:59:12.447000", "content": "", "votes": 0, "replies": [] }, { "id": 636904, "author_name": "", "author_url": "", "post_date": "2019-09-30T10:21:05.513000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "634193": "Just a running thread of your best single model LB score. I will update weekly. Please share! \n