{
  "id": 409925,
  "title": "Means and Standard Deviation for Normalization",
  "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/409925",
  "author_name": "Soumyadeep Khandual",
  "post_date": "2023-05-13T07:15:57.680000",
  "votes": 12,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi,<br>\nSince we are working on 9 infrared channel at different wavelengths which are converted to brightness temperatures, we cannot simply divide each pixel intensity by 255. <br>\nI have calculated the mean and std of train dataset for the 9 channels which can be used to normalize images,</p>\n<pre><code>Mean = [, , , , , , , , ]\n Std = [ ,  , , , , , , , ]\n</code></pre>\n<p>The 9 channels correspond to band_08 to band_16. The code i used is given below:</p>\n<pre><code>total_sum = torch.zeros()\ntotal_sum_of_sq = torch.zeros()\n\n\n\n\n data, _  tqdm(train_loader):\n    total_sum += torch.(data, axis=[,,,]) \n    total_sum_of_sq += torch.(data**, axis=[,,,])\n\ntotal_count = (train_loader)*train_loader.batch_size***\ntotal_mean = total_sum / total_count\ntotal_mean_of_sq = total_sum_of_sq/total_count\ntotal_std = torch.sqrt(total_mean_of_sq - total_mean**)\n</code></pre>\n<p>Let me know if it was helpful to you or if you find any mistakes. It would be great if someone can verify the mean and std.</p>",
  "messages": [
    {
      "id": 2257251,
      "postDate": "2023-05-13T07:15:57.680Z",
      "content": "<p>Hi,<br>\nSince we are working on 9 infrared channel at different wavelengths which are converted to brightness temperatures, we cannot simply divide each pixel intensity by 255. <br>\nI have calculated the mean and std of train dataset for the 9 channels which can be used to normalize images,</p>\n<pre><code>Mean = [, , , , , , , , ]\n Std = [ ,  , , , , , , , ]\n</code></pre>\n<p>The 9 channels correspond to band_08 to band_16. The code i used is given below:</p>\n<pre><code>total_sum = torch.zeros()\ntotal_sum_of_sq = torch.zeros()\n\n\n\n\n data, _  tqdm(train_loader):\n    total_sum += torch.(data, axis=[,,,]) \n    total_sum_of_sq += torch.(data**, axis=[,,,])\n\ntotal_count = (train_loader)*train_loader.batch_size***\ntotal_mean = total_sum / total_count\ntotal_mean_of_sq = total_sum_of_sq/total_count\ntotal_std = torch.sqrt(total_mean_of_sq - total_mean**)\n</code></pre>\n<p>Let me know if it was helpful to you or if you find any mistakes. It would be great if someone can verify the mean and std.</p>",
      "rawMarkdown": "Hi,\nSince we are working on 9 infrared channel at different wavelengths which are converted to brightness temperatures, we cannot simply divide each pixel intensity by 255. \nI have calculated the mean and std of train dataset for the 9 channels which can be used to normalize images,\n```python\nMean = [233.6771, 242.2548, 250.7509, 274.4108, 255.5268, 276.6016, 275.3604, 272.5643, 260.4260]\n Std = [ 7.0181,  9.1566, 11.3484, 19.6334, 13.1177, 20.7182, 21.0882, 20.5616, 15.8269]\n```\nThe 9 channels correspond to band_08 to band_16. The code i used is given below:\n```python\ntotal_sum = torch.zeros(9)\ntotal_sum_of_sq = torch.zeros(9)\n\n# train_loder output shape = (Batch_size, Time_frame, Channel, H, W)\n# Time_frame = 8, Channel = 9, H=256 , W =256 \n# in my case batch size = 1\nfor data, _ in tqdm(train_loader):\n    total_sum += torch.sum(data, axis=[0,1,3,4]) \n    total_sum_of_sq += torch.sum(data**2, axis=[0,1,3,4])\n\ntotal_count = len(train_loader)*train_loader.batch_size*8*256*256\ntotal_mean = total_sum / total_count\ntotal_mean_of_sq = total_sum_of_sq/total_count\ntotal_std = torch.sqrt(total_mean_of_sq - total_mean**2)\n```\n\nLet me know if it was helpful to you or if you find any mistakes. It would be great if someone can verify the mean and std.",
      "votes": 10
    },
    {
      "id": 2257398,
      "postDate": "2023-05-13T10:18:20.733Z",
      "content": "<p>Hi! It looks like you have correctly calculated the mean and standard deviation for your 9 infrared channels. Using the mean and standard deviation for normalization is a common practice in image processing, and it is good that you have calculated them from your training dataset rather than using standard values like 255.</p>\n<p>The code you used to calculate the mean and standard deviation also seems correct. You summed up the pixel values across all batches, time frames, height, and width dimensions and then divided by the total number of pixels to get the mean. To calculate the standard deviation, you used the formula for variance, which involves subtracting the squared mean from the mean of squared values, and then taking the square root.</p>\n<p>Overall, it seems like you have done a good job in calculating the mean and standard deviation for your dataset.</p>",
      "rawMarkdown": "Hi! It looks like you have correctly calculated the mean and standard deviation for your 9 infrared channels. Using the mean and standard deviation for normalization is a common practice in image processing, and it is good that you have calculated them from your training dataset rather than using standard values like 255.\n\nThe code you used to calculate the mean and standard deviation also seems correct. You summed up the pixel values across all batches, time frames, height, and width dimensions and then divided by the total number of pixels to get the mean. To calculate the standard deviation, you used the formula for variance, which involves subtracting the squared mean from the mean of squared values, and then taking the square root.\n\nOverall, it seems like you have done a good job in calculating the mean and standard deviation for your dataset.",
      "votes": -7
    }
  ],
  "comments": [
    {
      "id": 2257398,
      "author_name": "Ericka42",
      "author_url": "",
      "post_date": "2023-05-13T10:18:20.733000",
      "content": "<p>Hi! It looks like you have correctly calculated the mean and standard deviation for your 9 infrared channels. Using the mean and standard deviation for normalization is a common practice in image processing, and it is good that you have calculated them from your training dataset rather than using standard values like 255.</p>\n<p>The code you used to calculate the mean and standard deviation also seems correct. You summed up the pixel values across all batches, time frames, height, and width dimensions and then divided by the total number of pixels to get the mean. To calculate the standard deviation, you used the formula for variance, which involves subtracting the squared mean from the mean of squared values, and then taking the square root.</p>\n<p>Overall, it seems like you have done a good job in calculating the mean and standard deviation for your dataset.</p>",
      "votes": -7,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2257251": "Hi,\nSince we are working on 9 infrared channel at different wavelengths which are converted to brightness temperatures, we cannot simply divide each pixel intensity by 255. \nI have calculated the mean and std of train dataset for the 9 channels which can be used to normalize images,\n```python\nMean = [233.6771, 242.2548, 250.7509, 274.4108, 255.5268, 276.6016, 275.3604, 272.5643, 260.4260]\n Std = [ 7.0181,  9.1566, 11.3484, 19.6334, 13.1177, 20.7182, 21.0882, 20.5616, 15.8269]\n```\nThe 9 channels correspond to band_08 to band_16. The code i used is given below:\n```python\ntotal_sum = torch.zeros(9)\ntotal_sum_of_sq = torch.zeros(9)\n\n# train_loder output shape = (Batch_size, Time_frame, Channel, H, W)\n# Time_frame = 8, Channel = 9, H=256 , W =256 \n# in my case batch size = 1\nfor data, _ in tqdm(train_loader):\n    total_sum += torch.sum(data, axis=[0,1,3,4]) \n    total_sum_of_sq += torch.sum(data**2, axis=[0,1,3,4])\n\ntotal_count = len(train_loader)*train_loader.batch_size*8*256*256\ntotal_mean = total_sum / total_count\ntotal_mean_of_sq = total_sum_of_sq/total_count\ntotal_std = torch.sqrt(total_mean_of_sq - total_mean**2)\n```\n\nLet me know if it was helpful to you or if you find any mistakes. It would be great if someone can verify the mean and std.",
    "2257398": "Hi! It looks like you have correctly calculated the mean and standard deviation for your 9 infrared channels. Using the mean and standard deviation for normalization is a common practice in image processing, and it is good that you have calculated them from your training dataset rather than using standard values like 255.\n\nThe code you used to calculate the mean and standard deviation also seems correct. You summed up the pixel values across all batches, time frames, height, and width dimensions and then divided by the total number of pixels to get the mean. To calculate the standard deviation, you used the formula for variance, which involves subtracting the squared mean from the mean of squared values, and then taking the square root.\n\nOverall, it seems like you have done a good job in calculating the mean and standard deviation for your dataset."
  }
}