Hi @gordonyip!
\nThanks for the notebook (https://www.kaggle.com/code/gordonyip/calibrating-and-time-binning-astronomical-data/notebook). I have a few additional questions:
\nAre we ignoring read noise (file read)? Typically, when creating master dark files, you subtract the read noise from the darks. However, you also need to subtract the read noise from the light frames separately.
\nIt looks like the same dark is subtracted from both 0.1 sec and 4.5 sec images (for 'AIRS-CH0'). This seems incorrect since dark current is proportional to the exposure time.
\nYou’re subtracting the dark from the flat. Were the flats also taken with a 4.5 sec exposure? The darks and flats (for 'AIRS-CH0') seem quite different:
\nDark: Min -0.3225, Max 8.9267, Mean 0.0100, Std 0.1472
\nFlat: Min -0.0248, Max 1.0933, Mean 1.0057, Std 0.0239
\nThe dark has a much higher standard deviation. Also, the dark maximum is 8, while the flat (which supposedly includes the dark) has a maximum of around 1.
What does negative dark current mean? Does it indicate that the sensor signal slowly decays over time?
\nAre all the darks, flats, reads, and correction coefficients the same for all the images? I sampled a few hundred, and they all seem identical.
", "messages": [ { "id": 2957556, "postDate": "2024-08-13T08:03:29.127Z", "content": "Hi @gordonyip!
\nThanks for the notebook (https://www.kaggle.com/code/gordonyip/calibrating-and-time-binning-astronomical-data/notebook). I have a few additional questions:
\nAre we ignoring read noise (file read)? Typically, when creating master dark files, you subtract the read noise from the darks. However, you also need to subtract the read noise from the light frames separately.
\nIt looks like the same dark is subtracted from both 0.1 sec and 4.5 sec images (for 'AIRS-CH0'). This seems incorrect since dark current is proportional to the exposure time.
\nYou’re subtracting the dark from the flat. Were the flats also taken with a 4.5 sec exposure? The darks and flats (for 'AIRS-CH0') seem quite different:
\nDark: Min -0.3225, Max 8.9267, Mean 0.0100, Std 0.1472
\nFlat: Min -0.0248, Max 1.0933, Mean 1.0057, Std 0.0239
\nThe dark has a much higher standard deviation. Also, the dark maximum is 8, while the flat (which supposedly includes the dark) has a maximum of around 1.
What does negative dark current mean? Does it indicate that the sensor signal slowly decays over time?
\nAre all the darks, flats, reads, and correction coefficients the same for all the images? I sampled a few hundred, and they all seem identical.
", "rawMarkdown": "Hi @gordonyip!\n\nThanks for the notebook (https://www.kaggle.com/code/gordonyip/calibrating-and-time-binning-astronomical-data/notebook). I have a few additional questions:\n\nAre we ignoring read noise (file read)? Typically, when creating master dark files, you subtract the read noise from the darks. However, you also need to subtract the read noise from the light frames separately.\n\nIt looks like the same dark is subtracted from both 0.1 sec and 4.5 sec images (for 'AIRS-CH0'). This seems incorrect since dark current is proportional to the exposure time.\n\nYou’re subtracting the dark from the flat. Were the flats also taken with a 4.5 sec exposure? The darks and flats (for 'AIRS-CH0') seem quite different:\nDark: Min -0.3225, Max 8.9267, Mean 0.0100, Std 0.1472\nFlat: Min -0.0248, Max 1.0933, Mean 1.0057, Std 0.0239\nThe dark has a much higher standard deviation. Also, the dark maximum is 8, while the flat (which supposedly includes the dark) has a maximum of around 1.\n\nWhat does negative dark current mean? Does it indicate that the sensor signal slowly decays over time?\n\nAre all the darks, flats, reads, and correction coefficients the same for all the images? I sampled a few hundred, and they all seem identical.\n", "votes": 12 }, { "id": 2959467, "postDate": "2024-08-14T22:25:37.443Z", "content": "Hi,
\nI tried to answer these questions in this post:
\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/528066
Please let me know if it helps.
", "rawMarkdown": "Hi, \nI tried to answer these questions in this post:\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/528066\n\nPlease let me know if it helps.", "votes": 1 }, { "id": 2964521, "postDate": "2024-08-19T21:51:54.837Z", "content": "From what I understood the read noise or bias frames are used for correction of flat frames. Ideally, someone would have an exposure-matched dark for flats. If they don't, considering that the flats can be obtained with short exposure time the next best thing is to subtract bias frames from the flats.
\nHere it seems the flats of the competition are already went through some pre-processing (they do not have integer values unlike the science frames).
I disagree with subtracting the bias from the dark frames. Even this website hints at subtracting biases from dark frames, but I think this is unnecessary.
\nThis is the overall goal:
\nit's a very good summary of all the things that confuse me too. thanks!
", "rawMarkdown": "it's a very good summary of all the things that confuse me too. thanks!" } ], "comments": [ { "id": 2959467, "author_name": "Lorenzo Mugnai", "author_url": "", "post_date": "2024-08-14T22:25:37.443000", "content": "Hi,
\nI tried to answer these questions in this post:
\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/528066
Please let me know if it helps.
", "votes": 1, "replies": [] }, { "id": 2964521, "author_name": "Reza R. Choubeh", "author_url": "", "post_date": "2024-08-19T21:51:54.837000", "content": "From what I understood the read noise or bias frames are used for correction of flat frames. Ideally, someone would have an exposure-matched dark for flats. If they don't, considering that the flats can be obtained with short exposure time the next best thing is to subtract bias frames from the flats.
\nHere it seems the flats of the competition are already went through some pre-processing (they do not have integer values unlike the science frames).
I disagree with subtracting the bias from the dark frames. Even this website hints at subtracting biases from dark frames, but I think this is unnecessary.
\nThis is the overall goal:
\nit's a very good summary of all the things that confuse me too. thanks!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2957556": "Hi @gordonyip!\n\nThanks for the notebook (https://www.kaggle.com/code/gordonyip/calibrating-and-time-binning-astronomical-data/notebook). I have a few additional questions:\n\nAre we ignoring read noise (file read)? Typically, when creating master dark files, you subtract the read noise from the darks. However, you also need to subtract the read noise from the light frames separately.\n\nIt looks like the same dark is subtracted from both 0.1 sec and 4.5 sec images (for 'AIRS-CH0'). This seems incorrect since dark current is proportional to the exposure time.\n\nYou’re subtracting the dark from the flat. Were the flats also taken with a 4.5 sec exposure? The darks and flats (for 'AIRS-CH0') seem quite different:\nDark: Min -0.3225, Max 8.9267, Mean 0.0100, Std 0.1472\nFlat: Min -0.0248, Max 1.0933, Mean 1.0057, Std 0.0239\nThe dark has a much higher standard deviation. Also, the dark maximum is 8, while the flat (which supposedly includes the dark) has a maximum of around 1.\n\nWhat does negative dark current mean? Does it indicate that the sensor signal slowly decays over time?\n\nAre all the darks, flats, reads, and correction coefficients the same for all the images? I sampled a few hundred, and they all seem identical.\n", "2959467": "Hi, \nI tried to answer these questions in this post:\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/528066\n\nPlease let me know if it helps.", "2964521": "From what I understood the read noise or bias frames are used for correction of flat frames. Ideally, someone would have an exposure-matched dark for flats. If they don't, considering that the flats can be obtained with short exposure time the next best thing is to subtract bias frames from the flats.\nHere it seems the flats of the competition are already went through some pre-processing (they do not have integer values unlike the science frames).\n\nI disagree with subtracting the bias from the dark frames. Even [this website](http://deepskystacker.free.fr/english/) hints at subtracting biases from dark frames, but I think this is unnecessary. \n\nThis is the overall goal:\n1. We have science frames and we need an exposure-matched dark frame to subtract the noise. Both the science and dark frames already included the bias or read-out noise so subtracting them does the job.\n2. We need to use flat frames to correct for uneven response of camera to different wavelengths. We need exposure-matched dark frames for flat frames. If we do not have, the next best thing is the bias frames.", "2958091": "it's a very good summary of all the things that confuse me too. thanks!" } }