{ "id": 524003, "title": "Unsure on definition of \"ground truth spectrum\" and wavelength.csv file", "url": "/competitions/ariel-data-challenge-2024/discussion/524003", "author_name": "Daniel Chen", "post_date": "2024-08-04T01:36:14.866000", "votes": 2, "comment_count": 9, "views": 0, "content": "

It says

\n
\n

wavelength.csv: The wavelength grid for each ground truth spectrum in the dataset.

\n
\n

I'm not sure what this means, I've tried looking around for \"ground truth spectrum\" online and can't find any definition of this. It's a little unclear to me what spectra we are referring to throughout the writings. As such, I'm unable to understand what the wavelength.csv file is supposed to represent.

", "messages": [ { "id": 2947665, "postDate": "2024-08-05T11:15:34.867Z", "content": "

Hi @d223chen

\n

In scientific literature, what we refer to as the \"ground truth spectrum\" is more commonly known as the transmission spectrum or exoplanetary spectrum. These spectra contain valuable information about an exoplanet's atmosphere.

\n

Ariel, the space mission, will employ four different instruments to simultaneously observe an exoplanet or a transiting event. These instruments will capture data across various wavelengths of light. The combined information from these different wavelengths forms the spectrum we analyze. The wavelength.csv tells you the spectral range and resolution of the instruments.

\n

Hope it helps!

", "rawMarkdown": "Hi @d223chen \n\nIn scientific literature, what we refer to as the \"ground truth spectrum\" is more commonly known as the transmission spectrum or exoplanetary spectrum. These spectra contain valuable information about an exoplanet's atmosphere.\n\nAriel, the space mission, will employ four different instruments to simultaneously observe an exoplanet or a transiting event. These instruments will capture data across various wavelengths of light. The combined information from these different wavelengths forms the spectrum we analyze. The wavelength.csv tells you the spectral range and resolution of the instruments.\n\nHope it helps! \n", "votes": 3, "replies": [ { "id": 2950140, "postDate": "2024-08-07T09:16:38.393Z", "content": "

Hi @gordonyip

\n

May I ask why the dimension of ground truth spectrum is 283?

", "rawMarkdown": "Hi @gordonyip \n\nMay I ask why the dimension of ground truth spectrum is 283?", "replies": [ { "id": 2950442, "postDate": "2024-08-07T15:15:11.200Z", "content": "

yes of course. The combined spectrum from FGS1 observation and AIRS-CH0 observation will have a length of 283 values, or 283 values of transit depths against wavelength. Since we are using the same instrument, the wavelength grid will always be the same (as described in wavelength.csv). They are what we called instrument spectrum resolution and is defined by the specification of the instrument (see Ariel Red Book). Hope it helps!

", "rawMarkdown": "yes of course. The combined spectrum from FGS1 observation and AIRS-CH0 observation will have a length of 283 values, or 283 values of transit depths against wavelength. Since we are using the same instrument, the wavelength grid will always be the same (as described in wavelength.csv). They are what we called instrument spectrum resolution and is defined by the specification of the instrument (see Ariel Red Book). Hope it helps!", "votes": 4 }, { "id": 2950826, "postDate": "2024-08-08T00:14:09.153Z", "content": "

Thank you very much. That's very helpful.

", "rawMarkdown": "Thank you very much. That's very helpful." } ] }, { "id": 2987861, "postDate": "2024-09-13T05:06:15.740Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2946116, "postDate": "2024-08-04T01:36:14.867Z", "content": "

It says

\n
\n

wavelength.csv: The wavelength grid for each ground truth spectrum in the dataset.

\n
\n

I'm not sure what this means, I've tried looking around for \"ground truth spectrum\" online and can't find any definition of this. It's a little unclear to me what spectra we are referring to throughout the writings. As such, I'm unable to understand what the wavelength.csv file is supposed to represent.

", "rawMarkdown": "It says \n> wavelength.csv: The wavelength grid for each ground truth spectrum in the dataset.\n\nI'm not sure what this means, I've tried looking around for \"ground truth spectrum\" online and can't find any definition of this. It's a little unclear to me what spectra we are referring to throughout the writings. As such, I'm unable to understand what the wavelength.csv file is supposed to represent.", "votes": 2 }, { "id": 2950219, "postDate": "2024-08-07T11:29:21.263Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2950765, "postDate": "2024-08-07T21:49:01.097Z", "content": "

Hi @edotirapuka,

\n

Indeed, if the participants are able to pinpoint exactly the transit depths (intensities) in the test set, the best strategy is to submit their network with 10ppm as uncertainties. This conclusion builds on the assumption that the model can predict the transit depths with a very high degree of accuracy. Given our understanding of the noise within the dataset, there will be information loss (due to the noise), which may prevent the model from predicting the transit depths perfectly (at least in the test set).

\n

Also, note that 10ppm is a very narrow value for uncertainty, and their model will be heavily penalized if the predictions are outside the uncertainty range (a characteristic of the GLL function), resulting in a worse score.

", "rawMarkdown": "Hi @edotirapuka, \n\nIndeed, if the participants are able to pinpoint exactly the transit depths (intensities) in the test set, the best strategy is to submit their network with 10ppm as uncertainties. This conclusion builds on the assumption that the model can predict the transit depths with a very high degree of accuracy. Given our understanding of the noise within the dataset, there will be information loss (due to the noise), which may prevent the model from predicting the transit depths perfectly (at least in the test set).\n\nAlso, note that 10ppm is a very narrow value for uncertainty, and their model will be heavily penalized if the predictions are outside the uncertainty range (a characteristic of the GLL function), resulting in a worse score.", "votes": 4, "replies": [ { "id": 2966973, "postDate": "2024-08-22T12:07:25.870Z", "content": "

Is L_ideal (GLL_true) a fixed value? Like we fix the value of sigma_true at 1e-5 and calculate GLL_true?
\nGLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))
\nGLL_mean is definitely fixed.

", "rawMarkdown": "Is L_ideal (GLL_true) a fixed value? Like we fix the value of sigma_true at 1e-5 and calculate GLL_true?\nGLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))\nGLL_mean is definitely fixed." }, { "id": 2967007, "postDate": "2024-08-22T12:53:06.477Z", "content": "

yes, it is a fixed value.

", "rawMarkdown": "yes, it is a fixed value. ", "votes": 1 } ] } ] } ], "comments": [ { "id": 2947665, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-05T11:15:34.867000", "content": "

Hi @d223chen

\n

In scientific literature, what we refer to as the \"ground truth spectrum\" is more commonly known as the transmission spectrum or exoplanetary spectrum. These spectra contain valuable information about an exoplanet's atmosphere.

\n

Ariel, the space mission, will employ four different instruments to simultaneously observe an exoplanet or a transiting event. These instruments will capture data across various wavelengths of light. The combined information from these different wavelengths forms the spectrum we analyze. The wavelength.csv tells you the spectral range and resolution of the instruments.

\n

Hope it helps!

", "votes": 3, "replies": [ { "id": 2950140, "author_name": "Xiaolei Lian", "author_url": "", "post_date": "2024-08-07T09:16:38.393000", "content": "

Hi @gordonyip

\n

May I ask why the dimension of ground truth spectrum is 283?

", "votes": 0, "replies": [ { "id": 2950442, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-07T15:15:11.200000", "content": "

yes of course. The combined spectrum from FGS1 observation and AIRS-CH0 observation will have a length of 283 values, or 283 values of transit depths against wavelength. Since we are using the same instrument, the wavelength grid will always be the same (as described in wavelength.csv). They are what we called instrument spectrum resolution and is defined by the specification of the instrument (see Ariel Red Book). Hope it helps!

", "votes": 4, "replies": [] }, { "id": 2950826, "author_name": "Xiaolei Lian", "author_url": "", "post_date": "2024-08-08T00:14:09.153000", "content": "

Thank you very much. That's very helpful.

", "votes": 0, "replies": [] } ] }, { "id": 2987861, "author_name": "", "author_url": "", "post_date": "2024-09-13T05:06:15.740000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2950219, "author_name": "", "author_url": "", "post_date": "2024-08-07T11:29:21.263000", "content": "", "votes": 1, "replies": [ { "id": 2950765, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-07T21:49:01.097000", "content": "

Hi @edotirapuka,

\n

Indeed, if the participants are able to pinpoint exactly the transit depths (intensities) in the test set, the best strategy is to submit their network with 10ppm as uncertainties. This conclusion builds on the assumption that the model can predict the transit depths with a very high degree of accuracy. Given our understanding of the noise within the dataset, there will be information loss (due to the noise), which may prevent the model from predicting the transit depths perfectly (at least in the test set).

\n

Also, note that 10ppm is a very narrow value for uncertainty, and their model will be heavily penalized if the predictions are outside the uncertainty range (a characteristic of the GLL function), resulting in a worse score.

", "votes": 4, "replies": [ { "id": 2966973, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-08-22T12:07:25.870000", "content": "

Is L_ideal (GLL_true) a fixed value? Like we fix the value of sigma_true at 1e-5 and calculate GLL_true?
\nGLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))
\nGLL_mean is definitely fixed.

", "votes": 0, "replies": [] }, { "id": 2967007, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-22T12:53:06.477000", "content": "

yes, it is a fixed value.

", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2947665": "Hi @d223chen \n\nIn scientific literature, what we refer to as the \"ground truth spectrum\" is more commonly known as the transmission spectrum or exoplanetary spectrum. These spectra contain valuable information about an exoplanet's atmosphere.\n\nAriel, the space mission, will employ four different instruments to simultaneously observe an exoplanet or a transiting event. These instruments will capture data across various wavelengths of light. The combined information from these different wavelengths forms the spectrum we analyze. The wavelength.csv tells you the spectral range and resolution of the instruments.\n\nHope it helps! \n", "2946116": "It says \n> wavelength.csv: The wavelength grid for each ground truth spectrum in the dataset.\n\nI'm not sure what this means, I've tried looking around for \"ground truth spectrum\" online and can't find any definition of this. It's a little unclear to me what spectra we are referring to throughout the writings. As such, I'm unable to understand what the wavelength.csv file is supposed to represent.", "2950219": "" } }