@wouterbulten and hosts,
\n\nI was wondering if you would consider adding an extra decimal point to the public LB scores. It would help us better characterize CV/LB discrepancies. However, I understand if you want to limit LB overfitting/probing by keeping it at 2 decimal points.
\n\nThanks,\nIan
", "messages": [ { "id": 841468, "postDate": "2020-05-10T20:01:20.380Z", "content": "@wouterbulten and hosts,
\n\nI was wondering if you would consider adding an extra decimal point to the public LB scores. It would help us better characterize CV/LB discrepancies. However, I understand if you want to limit LB overfitting/probing by keeping it at 2 decimal points.
\n\nThanks,\nIan
", "rawMarkdown": "@wouterbulten and hosts,\n\nI was wondering if you would consider adding an extra decimal point to the public LB scores. It would help us better characterize CV/LB discrepancies. However, I understand if you want to limit LB overfitting/probing by keeping it at 2 decimal points. \n\nThanks,\nIan", "votes": 14 }, { "id": 841671, "postDate": "2020-05-11T00:11:00.900Z", "content": "My intuition is that adding an extra decimal point will make it LB-probing-able. That is, it will be possible to achieve the perfect public LB score. I developed my intuition here and here. So I was actually glad to see that in this competition Kaggle addressed the issue, and from multiple directions. Still, given the small test size, - just 1000 for public+private test, only 6 possible values per sample, and good priors existing (0.8+ on QWK), I think 3 digits is already hack-able.
\n\nOn the other hand, reaching the perfect public LB score will help very little with the private.
", "rawMarkdown": "My intuition is that adding an extra decimal point will make it LB-probing-able. That is, it will be possible to achieve the perfect public LB score. I developed my intuition [here](https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110335) and [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/94043). So I was actually glad to see that in this competition Kaggle addressed the issue, and from multiple directions. Still, given the small test size, - just 1000 for public+private test, only 6 possible values per sample, and good priors existing (0.8+ on QWK), I think 3 digits is already hack-able.\n\nOn the other hand, reaching the perfect public LB score will help very little with the private.", "votes": 3 }, { "id": 844640, "postDate": "2020-05-12T18:23:39.440Z", "content": "I know it is not the perfect solution as adding a decimal point, but you can still sort your submissions by public score (in case you didn't know).
", "rawMarkdown": "I know it is not the perfect solution as adding a decimal point, but you can still sort your submissions by public score (in case you didn't know).", "votes": 4, "replies": [ { "id": 845439, "postDate": "2020-05-13T07:57:28.287Z", "content": "I'm not sure that works? I have 3 scores that are 0.86. When I sort by this, my most recent one is at the top, despite the leaderboard saying \"Your submission scored 0.86, which is not an improvement of your best score. Keep trying!\"
\n\nI suppose it's possible my last 2 models got the exact same score, but I think it's unlikely?
", "rawMarkdown": "I'm not sure that works? I have 3 scores that are 0.86. When I sort by this, my most recent one is at the top, despite the leaderboard saying \"Your submission scored 0.86, which is not an improvement of your best score. Keep trying!\"\n\nI suppose it's possible my last 2 models got the exact same score, but I think it's unlikely?", "votes": 1 }, { "id": 845522, "postDate": "2020-05-13T08:56:47.077Z", "content": "I think this message \"which is not an improvement of your best score\" is just a comparison after the truncation. But it says nothing, of course, about the sorting comparison logic, it still can be both ways. I agree @jamesphoward , the proof is missing.
", "rawMarkdown": "I think this message \"which is not an improvement of your best score\" is just a comparison **after** the truncation. But it says nothing, of course, about the sorting comparison logic, it still can be both ways. I agree @jamesphoward , the proof is missing.", "votes": 1 }, { "id": 845602, "postDate": "2020-05-13T10:08:46.103Z", "content": "@jamesphoward @zaharch sure, there is no proof, but I think the way your submissions and how LB is sorted is the same. In my previous competitions I used this feature, which helped me to choose better submissions for private. Of course, it is up to you ;-)
", "rawMarkdown": "@jamesphoward @zaharch sure, there is no proof, but I think the way your submissions and how LB is sorted is the same. In my previous competitions I used this feature, which helped me to choose better submissions for private. Of course, it is up to you ;-) ", "votes": 1 }, { "id": 845749, "postDate": "2020-05-13T12:07:21.693Z", "content": "Another trick to get more precision is to check your position on the LB among people with the same truncated score.
", "rawMarkdown": "Another trick to get more precision is to check your position on the LB among people with the same truncated score.", "votes": 1 } ] }, { "id": 892372, "postDate": "2020-06-18T20:45:52.493Z", "content": "@wouterbulten It would be great if you consider adding an extra decimal point to LB.
", "rawMarkdown": "@wouterbulten It would be great if you consider adding an extra decimal point to LB." }, { "id": 841584, "postDate": "2020-05-10T21:41:49.947Z", "content": "Yes, It would be useful
", "rawMarkdown": "Yes, It would be useful" }, { "id": 841471, "postDate": "2020-05-10T20:06:01.490Z", "content": "I would like this as well!
", "rawMarkdown": "I would like this as well!" } ], "comments": [ { "id": 841671, "author_name": "nosound", "author_url": "", "post_date": "2020-05-11T00:11:00.900000", "content": "My intuition is that adding an extra decimal point will make it LB-probing-able. That is, it will be possible to achieve the perfect public LB score. I developed my intuition here and here. So I was actually glad to see that in this competition Kaggle addressed the issue, and from multiple directions. Still, given the small test size, - just 1000 for public+private test, only 6 possible values per sample, and good priors existing (0.8+ on QWK), I think 3 digits is already hack-able.
\n\nOn the other hand, reaching the perfect public LB score will help very little with the private.
", "votes": 3, "replies": [] }, { "id": 844640, "author_name": "Zhanseri Ikram", "author_url": "", "post_date": "2020-05-12T18:23:39.440000", "content": "I know it is not the perfect solution as adding a decimal point, but you can still sort your submissions by public score (in case you didn't know).
", "votes": 4, "replies": [ { "id": 845439, "author_name": "James Howard", "author_url": "", "post_date": "2020-05-13T07:57:28.287000", "content": "I'm not sure that works? I have 3 scores that are 0.86. When I sort by this, my most recent one is at the top, despite the leaderboard saying \"Your submission scored 0.86, which is not an improvement of your best score. Keep trying!\"
\n\nI suppose it's possible my last 2 models got the exact same score, but I think it's unlikely?
", "votes": 1, "replies": [] }, { "id": 845522, "author_name": "nosound", "author_url": "", "post_date": "2020-05-13T08:56:47.077000", "content": "I think this message \"which is not an improvement of your best score\" is just a comparison after the truncation. But it says nothing, of course, about the sorting comparison logic, it still can be both ways. I agree @jamesphoward , the proof is missing.
", "votes": 1, "replies": [] }, { "id": 845602, "author_name": "Zhanseri Ikram", "author_url": "", "post_date": "2020-05-13T10:08:46.103000", "content": "@jamesphoward @zaharch sure, there is no proof, but I think the way your submissions and how LB is sorted is the same. In my previous competitions I used this feature, which helped me to choose better submissions for private. Of course, it is up to you ;-)
", "votes": 1, "replies": [] }, { "id": 845749, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-13T12:07:21.693000", "content": "Another trick to get more precision is to check your position on the LB among people with the same truncated score.
", "votes": 1, "replies": [] } ] }, { "id": 892372, "author_name": "SumanSudhir", "author_url": "", "post_date": "2020-06-18T20:45:52.493000", "content": "@wouterbulten It would be great if you consider adding an extra decimal point to LB.
", "votes": 0, "replies": [] }, { "id": 841584, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-05-10T21:41:49.947000", "content": "Yes, It would be useful
", "votes": 0, "replies": [] }, { "id": 841471, "author_name": "Shujun", "author_url": "", "post_date": "2020-05-10T20:06:01.490000", "content": "I would like this as well!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "841468": "@wouterbulten and hosts,\n\nI was wondering if you would consider adding an extra decimal point to the public LB scores. It would help us better characterize CV/LB discrepancies. However, I understand if you want to limit LB overfitting/probing by keeping it at 2 decimal points. \n\nThanks,\nIan", "841671": "My intuition is that adding an extra decimal point will make it LB-probing-able. That is, it will be possible to achieve the perfect public LB score. I developed my intuition [here](https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/110335) and [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/94043). So I was actually glad to see that in this competition Kaggle addressed the issue, and from multiple directions. Still, given the small test size, - just 1000 for public+private test, only 6 possible values per sample, and good priors existing (0.8+ on QWK), I think 3 digits is already hack-able.\n\nOn the other hand, reaching the perfect public LB score will help very little with the private.", "844640": "I know it is not the perfect solution as adding a decimal point, but you can still sort your submissions by public score (in case you didn't know).", "892372": "@wouterbulten It would be great if you consider adding an extra decimal point to LB.", "841584": "Yes, It would be useful", "841471": "I would like this as well!" } }