It's very likely that if you are using U-Net for this competition, you implemented it yourself. After reading the paper (or any documentation that refer to U-Net), you realised that there was this Conv -> Relu -> Conv -> Relu that was being repeated at every downsampling and upsampling stage, so naturally, you made it into a function (or most likely a Class):
\n ():\n nn.Sequential(\n nn.Conv2d(in_channels, out_channels, , padding=, bias=),\n nn.ReLU(inplace=),\n nn.Conv2d(out_channels, out_channels, , padding=, bias=),\n nn.ReLU(inplace=)\n ) \nEven if this implementation is reasonable and produces good results in itself, kaggle is all about improving the score at all cost, and sometimes, at a very low cost. Indeed, by simply adding BatchNorm, I was able to increase my CV and LB at the same time while having a lower number of epochs:
\n ():\n nn.Sequential(\n nn.Conv2d(in_channels, out_channels, , padding=, bias=), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=),\n nn.Conv2d(out_channels, out_channels, , padding=, bias=), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=)\n ) \nIf you are wondering why they didn't use BatchNorm, it's because the two papers came out at the same time:
\nBatchNorm paper - U-Net paper
It's very likely that if you are using U-Net for this competition, you implemented it yourself. After reading the paper (or any documentation that refer to U-Net), you realised that there was this Conv -> Relu -> Conv -> Relu that was being repeated at every downsampling and upsampling stage, so naturally, you made it into a function (or most likely a Class):
\n ():\n nn.Sequential(\n nn.Conv2d(in_channels, out_channels, , padding=, bias=),\n nn.ReLU(inplace=),\n nn.Conv2d(out_channels, out_channels, , padding=, bias=),\n nn.ReLU(inplace=)\n ) \nEven if this implementation is reasonable and produces good results in itself, kaggle is all about improving the score at all cost, and sometimes, at a very low cost. Indeed, by simply adding BatchNorm, I was able to increase my CV and LB at the same time while having a lower number of epochs:
\n ():\n nn.Sequential(\n nn.Conv2d(in_channels, out_channels, , padding=, bias=), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=),\n nn.Conv2d(out_channels, out_channels, , padding=, bias=), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=)\n ) \nIf you are wondering why they didn't use BatchNorm, it's because the two papers came out at the same time:
\nBatchNorm paper - U-Net paper
When asked ChatGPT why U-Net doesn't use BatchNorm:
\nThe original U-Net architecture, proposed by Ronneberger et al. in 2015, does not use Batch Normalization (BatchNorm). There could be several reasons for this:\n\nHistorical Context: BatchNorm was proposed in a paper by Ioffe and Szegedy in 2015, around the same time that the U-Net paper was published. It's possible that the U-Net authors were not aware of BatchNorm, or they didn't have enough time to incorporate it into their model and test it thoroughly.\n\nModel Simplicity: The U-Net was designed to be a relatively simple and straightforward architecture for biomedical image segmentation. The authors may have preferred to keep the architecture as simple as possible to make it easier for other researchers to understand and implement.\n\nDifferent Objectives: BatchNorm is primarily used to address the problem of internal covariate shift, where the distribution of each layer's inputs changes during training. This can make training deep networks difficult. However, the U-Net was designed for a different purpose: to perform precise localization in biomedical image segmentation tasks. The authors may have felt that addressing internal covariate shift was not a major concern for this task.\n\nComputational Resources: BatchNorm requires additional computations during both the forward and backward passes of training, which increases the computational resources required. If the authors were working with limited computational resources, they may have chosen to omit BatchNorm to reduce the computational requirements of the model.\n\nThat being said, variations of U-Net that do include BatchNorm have been proposed since the original paper, and these variations have shown improved performance in some cases. The decision to use BatchNorm or not often depends on the specific task and dataset, as well as the computational resources available.\nThank you for sharing your great comparison and explanation!
", "rawMarkdown": "Thank you for sharing your great comparison and explanation! ", "votes": 1 }, { "id": 2324201, "postDate": "2023-06-30T13:17:51.993Z", "content": "Thanks for sharing. What's the difference in CV and LB scores? How much boost are you getting?
", "rawMarkdown": "Thanks for sharing. What's the difference in CV and LB scores? How much boost are you getting?", "replies": [ { "id": 2324276, "postDate": "2023-06-30T14:02:17.893Z", "content": "I don't remember since i use smp implementation now which has a parameter for enabling BatchNorm. I recall it did improve the CV but also sped up training by lowering the number of epochs needed.
", "rawMarkdown": "I don't remember since i use smp implementation now which has a parameter for enabling BatchNorm. I recall it did improve the CV but also sped up training by lowering the number of epochs needed.", "votes": 1, "replies": [ { "id": 2339677, "postDate": "2023-07-11T02:35:21.043Z", "content": "@janmpia You mean smp.Unet(decoder_use_batchnorm=True)?
hello @riow1983,
\nyes thats what I mean.
Very interesting finding! Thank you!
", "rawMarkdown": "Very interesting finding! Thank you!" } ], "comments": [ { "id": 2280303, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-05-30T03:47:37.727000", "content": "When asked ChatGPT why U-Net doesn't use BatchNorm:
\nThe original U-Net architecture, proposed by Ronneberger et al. in 2015, does not use Batch Normalization (BatchNorm). There could be several reasons for this:\n\nHistorical Context: BatchNorm was proposed in a paper by Ioffe and Szegedy in 2015, around the same time that the U-Net paper was published. It's possible that the U-Net authors were not aware of BatchNorm, or they didn't have enough time to incorporate it into their model and test it thoroughly.\n\nModel Simplicity: The U-Net was designed to be a relatively simple and straightforward architecture for biomedical image segmentation. The authors may have preferred to keep the architecture as simple as possible to make it easier for other researchers to understand and implement.\n\nDifferent Objectives: BatchNorm is primarily used to address the problem of internal covariate shift, where the distribution of each layer's inputs changes during training. This can make training deep networks difficult. However, the U-Net was designed for a different purpose: to perform precise localization in biomedical image segmentation tasks. The authors may have felt that addressing internal covariate shift was not a major concern for this task.\n\nComputational Resources: BatchNorm requires additional computations during both the forward and backward passes of training, which increases the computational resources required. If the authors were working with limited computational resources, they may have chosen to omit BatchNorm to reduce the computational requirements of the model.\n\nThat being said, variations of U-Net that do include BatchNorm have been proposed since the original paper, and these variations have shown improved performance in some cases. The decision to use BatchNorm or not often depends on the specific task and dataset, as well as the computational resources available.\nThank you for sharing your great comparison and explanation!
", "votes": 1, "replies": [] }, { "id": 2324201, "author_name": "CroDoc", "author_url": "", "post_date": "2023-06-30T13:17:51.993000", "content": "Thanks for sharing. What's the difference in CV and LB scores? How much boost are you getting?
", "votes": 0, "replies": [ { "id": 2324276, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-30T14:02:17.893000", "content": "I don't remember since i use smp implementation now which has a parameter for enabling BatchNorm. I recall it did improve the CV but also sped up training by lowering the number of epochs needed.
", "votes": 1, "replies": [ { "id": 2339677, "author_name": "Ryosuke Horiuchi", "author_url": "", "post_date": "2023-07-11T02:35:21.043000", "content": "@janmpia You mean smp.Unet(decoder_use_batchnorm=True)?
hello @riow1983,
\nyes thats what I mean.
Very interesting finding! Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2280300": "## Spoiler: It doesn't use BatchNorm\nIt's very likely that if you are using U-Net for this competition, you implemented it yourself. After reading the paper (or any documentation that refer to U-Net), you realised that there was this **Conv -> Relu -> Conv -> Relu** that was being repeated at every downsampling and upsampling stage, so naturally, you made it into a function (or most likely a Class):\n```python\ndef double_conv(in_channels, out_channels):\n return nn.Sequential(\n nn.Conv2d(in_channels, out_channels, 3, padding=1, bias=True),\n nn.ReLU(inplace=True),\n nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=True),\n nn.ReLU(inplace=True)\n ) \n```\n\nEven if this implementation is reasonable and produces good results in itself, kaggle is all about improving the score at all cost, and sometimes, at **a very low cost**. Indeed, by simply adding BatchNorm, I was able to increase my CV and LB at the same time while having a lower number of epochs:\n\n```python\ndef double_conv(in_channels, out_channels):\n return nn.Sequential(\n nn.Conv2d(in_channels, out_channels, 3, padding=1, bias=False), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=True),\n nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False), \n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=True)\n ) \n```\n\n*If you are wondering why they didn't use BatchNorm, it's because the two papers came out at the same time:*\n[BatchNorm paper](https://arxiv.org/abs/1502.03167) - [U-Net paper](https://arxiv.org/abs/1505.04597)", "2280303": "When asked ChatGPT why U-Net doesn't use BatchNorm:\n\n```\nThe original U-Net architecture, proposed by Ronneberger et al. in 2015, does not use Batch Normalization (BatchNorm). There could be several reasons for this:\n\nHistorical Context: BatchNorm was proposed in a paper by Ioffe and Szegedy in 2015, around the same time that the U-Net paper was published. It's possible that the U-Net authors were not aware of BatchNorm, or they didn't have enough time to incorporate it into their model and test it thoroughly.\n\nModel Simplicity: The U-Net was designed to be a relatively simple and straightforward architecture for biomedical image segmentation. The authors may have preferred to keep the architecture as simple as possible to make it easier for other researchers to understand and implement.\n\nDifferent Objectives: BatchNorm is primarily used to address the problem of internal covariate shift, where the distribution of each layer's inputs changes during training. This can make training deep networks difficult. However, the U-Net was designed for a different purpose: to perform precise localization in biomedical image segmentation tasks. The authors may have felt that addressing internal covariate shift was not a major concern for this task.\n\nComputational Resources: BatchNorm requires additional computations during both the forward and backward passes of training, which increases the computational resources required. If the authors were working with limited computational resources, they may have chosen to omit BatchNorm to reduce the computational requirements of the model.\n\nThat being said, variations of U-Net that do include BatchNorm have been proposed since the original paper, and these variations have shown improved performance in some cases. The decision to use BatchNorm or not often depends on the specific task and dataset, as well as the computational resources available.\n```", "2280525": "Thank you for sharing your great comparison and explanation! ", "2324201": "Thanks for sharing. What's the difference in CV and LB scores? How much boost are you getting?", "2282848": "Very interesting finding! Thank you!" } }