The host notebook carries the normalization by signal += offset. If you check the values for offset, they are all negative, and after signal += offset, we get for soma values signal<0, which does not make sense to me from a physical POV- I expect the absolute minimum for the sensor to be zero. It makes much more sense that the normalization should be signal -= offset. Also, they already made the same mistake with signal *= gain, which they repaired to signal /= gain. But maybe I missed something so I would like to hear your thoughts, and especially I would be delighted If the hosts give input on this matter. @lorenzomugnai, @gordonyip
", "messages": [ { "id": 2969895, "postDate": "2024-08-25T14:15:41.500Z", "content": "The host notebook carries the normalization by signal += offset. If you check the values for offset, they are all negative, and after signal += offset, we get for soma values signal<0, which does not make sense to me from a physical POV- I expect the absolute minimum for the sensor to be zero. It makes much more sense that the normalization should be signal -= offset. Also, they already made the same mistake with signal *= gain, which they repaired to signal /= gain. But maybe I missed something so I would like to hear your thoughts, and especially I would be delighted If the hosts give input on this matter. @lorenzomugnai, @gordonyip
", "rawMarkdown": "The [host notebook](https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data) carries the normalization by signal += offset. If you check the values for offset, they are all negative, and after signal += offset, we get for soma values signal<0, which does not make sense to me from a physical POV- I expect the absolute minimum for the sensor to be zero. It makes much more sense that the normalization should be signal -= offset. Also, they already made the same mistake with signal \\*= gain, which they repaired to signal /= gain. But maybe I missed something so I would like to hear your thoughts, and especially I would be delighted If the hosts give input on this matter. @lorenzomugnai, @gordonyip ", "votes": 13 }, { "id": 2971197, "postDate": "2024-08-27T00:10:41.350Z", "content": "Just an FYI, changing my data processing to clip the values below 0 before applying the polynomial correction led to an increase in my validation score. Thank you all for pointing me towards that!
", "rawMarkdown": "Just an FYI, changing my data processing to clip the values below 0 before applying the polynomial correction led to an increase in my validation score. Thank you all for pointing me towards that!", "votes": 5, "replies": [ { "id": 2971214, "postDate": "2024-08-27T00:54:40.403Z", "content": "Do you care to share how much it bumped you? (If you prefer not to, it's understandable and OK)
", "rawMarkdown": "Do you care to share how much it bumped you? (If you prefer not to, it's understandable and OK)", "replies": [ { "id": 2972090, "postDate": "2024-08-28T01:00:50.817Z", "content": "It was a small improvement. I don't recall the exact value. Mostly I'm glad to have updated this in case of test set surprises.
", "rawMarkdown": "It was a small improvement. I don't recall the exact value. Mostly I'm glad to have updated this in case of test set surprises.", "votes": 2 } ] } ] }, { "id": 2970703, "postDate": "2024-08-26T12:11:45.590Z", "content": "I found this https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data/comments#2951175
\n\"There will still be some negative pixel values, they are caused by the random noise we injected into the dataset during the simulation. \"
", "rawMarkdown": "I found this https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data/comments#2951175\n\n\"There will still be some negative pixel values, they are caused by the random noise we injected into the dataset during the simulation. \"", "votes": 4, "replies": [ { "id": 2970709, "postDate": "2024-08-26T12:25:24.213Z", "content": "Good find. So normalization is good, but I think we need to be careful with applying polynomials to negative values (I wonder if just clipping them at zero is best…)
", "rawMarkdown": "Good find. So normalization is good, but I think we need to be careful with applying polynomials to negative values (I wonder if just clipping them at zero is best...)", "votes": 2 } ] }, { "id": 2970161, "postDate": "2024-08-25T19:18:04.063Z", "content": "when we proceed to the differences (cds signal), the offset doesn't matter. right? This only matters for polynom calibration, it seems and I don't really think it is correct in the host notebook for some reasons. I think you are totally right.
", "rawMarkdown": "when we proceed to the differences (cds signal), the offset doesn't matter. right? This only matters for polynom calibration, it seems and I don't really think it is correct in the host notebook for some reasons. I think you are totally right.", "votes": 2, "replies": [ { "id": 2970177, "postDate": "2024-08-25T19:35:06.800Z", "content": "Indeed. Without polynomial calibration, it wouldn't matter. However, with the calibration, things become complicated, especially since I suspect that polynomials are not supposed to be used for negative values. (Although, when I checked both normalizations, it had a negligible effect on the validation score. But who knows what would happen in test…)
", "rawMarkdown": "Indeed. Without polynomial calibration, it wouldn't matter. However, with the calibration, things become complicated, especially since I suspect that polynomials are not supposed to be used for negative values. (Although, when I checked both normalizations, it had a negligible effect on the validation score. But who knows what would happen in test...)", "votes": 4, "replies": [ { "id": 2970190, "postDate": "2024-08-25T19:42:09Z", "content": "yes, until the hosts posted their version of processing, I used many different ones and chose the one that works for 0->1 star validation and on the public leaderboard. Surprisingly, my best option is not the same at all. I don't understand anything about this data, unfortunately.
", "rawMarkdown": "yes, until the hosts posted their version of processing, I used many different ones and chose the one that works for 0->1 star validation and on the public leaderboard. Surprisingly, my best option is not the same at all. I don't understand anything about this data, unfortunately.\n", "votes": 4 }, { "id": 2970204, "postDate": "2024-08-25T20:22:55.333Z", "content": "If we check only for star 1/star 2, calibration is almost redundant. I get a bump of ~0.003 from all the calibration steps combined.
\nBut it's important to calibrate correctly for the test set since it can have 'surprises.'
\nEven if something works for public LB, private may still be from a different distribution. (Host neither confirmed nor denied it yet)
Just an FYI, changing my data processing to clip the values below 0 before applying the polynomial correction led to an increase in my validation score. Thank you all for pointing me towards that!
", "votes": 5, "replies": [ { "id": 2971214, "author_name": "greySnow", "author_url": "", "post_date": "2024-08-27T00:54:40.403000", "content": "Do you care to share how much it bumped you? (If you prefer not to, it's understandable and OK)
", "votes": 0, "replies": [ { "id": 2972090, "author_name": "Andrew Matteson", "author_url": "", "post_date": "2024-08-28T01:00:50.817000", "content": "It was a small improvement. I don't recall the exact value. Mostly I'm glad to have updated this in case of test set surprises.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2970703, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-08-26T12:11:45.590000", "content": "I found this https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data/comments#2951175
\n\"There will still be some negative pixel values, they are caused by the random noise we injected into the dataset during the simulation. \"
", "votes": 4, "replies": [ { "id": 2970709, "author_name": "greySnow", "author_url": "", "post_date": "2024-08-26T12:25:24.213000", "content": "Good find. So normalization is good, but I think we need to be careful with applying polynomials to negative values (I wonder if just clipping them at zero is best…)
", "votes": 2, "replies": [] } ] }, { "id": 2970161, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-08-25T19:18:04.063000", "content": "when we proceed to the differences (cds signal), the offset doesn't matter. right? This only matters for polynom calibration, it seems and I don't really think it is correct in the host notebook for some reasons. I think you are totally right.
", "votes": 2, "replies": [ { "id": 2970177, "author_name": "greySnow", "author_url": "", "post_date": "2024-08-25T19:35:06.800000", "content": "Indeed. Without polynomial calibration, it wouldn't matter. However, with the calibration, things become complicated, especially since I suspect that polynomials are not supposed to be used for negative values. (Although, when I checked both normalizations, it had a negligible effect on the validation score. But who knows what would happen in test…)
", "votes": 4, "replies": [ { "id": 2970190, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-08-25T19:42:09", "content": "yes, until the hosts posted their version of processing, I used many different ones and chose the one that works for 0->1 star validation and on the public leaderboard. Surprisingly, my best option is not the same at all. I don't understand anything about this data, unfortunately.
", "votes": 4, "replies": [] }, { "id": 2970204, "author_name": "greySnow", "author_url": "", "post_date": "2024-08-25T20:22:55.333000", "content": "If we check only for star 1/star 2, calibration is almost redundant. I get a bump of ~0.003 from all the calibration steps combined.
\nBut it's important to calibrate correctly for the test set since it can have 'surprises.'
\nEven if something works for public LB, private may still be from a different distribution. (Host neither confirmed nor denied it yet)