I've been seeing a lot of copies of the same notebook (TBM + Protenix) in the 'Code' tab for this competition that all seem to score fairly high (around 0.43-0.44). However, when I copy and submit the notebook, I get varying results each submission (around 0.41-0.43). I understand that there is likely some variance coming from Protenix that is causing the variation in the score, but is it reasonable that such variation could account for a 0.03 swing in public LB score?
\nAnother thing I think I need clarification on is the public score shown for public notebooks. Does that score shown at the top of a public notebook mean that the exact version that you are viewing scored that high, or that this notebook has scored that high after modifications were made that are not visible in the version that is public?
", "messages": [ { "id": 3425441, "postDate": "2026-03-20T20:14:22.250Z", "content": "I've been seeing a lot of copies of the same notebook (TBM + Protenix) in the 'Code' tab for this competition that all seem to score fairly high (around 0.43-0.44). However, when I copy and submit the notebook, I get varying results each submission (around 0.41-0.43). I understand that there is likely some variance coming from Protenix that is causing the variation in the score, but is it reasonable that such variation could account for a 0.03 swing in public LB score?
\nAnother thing I think I need clarification on is the public score shown for public notebooks. Does that score shown at the top of a public notebook mean that the exact version that you are viewing scored that high, or that this notebook has scored that high after modifications were made that are not visible in the version that is public?
", "rawMarkdown": "I've been seeing a lot of copies of the same notebook (TBM + Protenix) in the 'Code' tab for this competition that all seem to score fairly high (around 0.43-0.44). However, when I copy and submit the notebook, I get varying results each submission (around 0.41-0.43). I understand that there is likely some variance coming from Protenix that is causing the variation in the score, but is it reasonable that such variation could account for a 0.03 swing in public LB score?\n\nAnother thing I think I need clarification on is the public score shown for public notebooks. Does that score shown at the top of a public notebook mean that the exact version that you are viewing scored that high, or that this notebook has scored that high after modifications were made that are not visible in the version that is public?", "votes": 1 }, { "id": 3425740, "postDate": "2026-03-21T07:39:44.527Z", "content": "The variance you are observing comes from an unfixed seed. By comparing those public notebooks, you'll notice that some of them are just a copy of another notebook but shaken to a higher score.\nThe scores shown at the top of a public notebook should include both the public score of the current version you are browsing and the best score of this notebook (for example, Public Score 0.xxx, Best Score 0.xxx V1).
", "rawMarkdown": "The variance you are observing comes from an unfixed seed. By comparing those public notebooks, you'll notice that some of them are just a copy of another notebook but shaken to a higher score.\nThe scores shown at the top of a public notebook should include both the public score of the current version you are browsing and the best score of this notebook (for example, Public Score 0.xxx, Best Score 0.xxx V1).", "replies": [ { "id": 3425788, "postDate": "2026-03-21T09:19:33.593Z", "content": "Will the organizers remove copied notebooks during final evaluation? because this is leading to unfair lb rankings. And the bigger concern is that, anyone with randomness in their notebook, can score higher than a fixed and good notebook. for example say i have a lb score of 0.43 (with seed contamination fixed). But someone elses notebook having seed contamination issue, can randomly score higher than me. How to deal with this?
", "rawMarkdown": "Will the organizers remove copied notebooks during final evaluation? because this is leading to unfair lb rankings. And the bigger concern is that, anyone with randomness in their notebook, can score higher than a fixed and good notebook. for example say i have a lb score of 0.43 (with seed contamination fixed). But someone elses notebook having seed contamination issue, can randomly score higher than me. How to deal with this?", "votes": 1, "replies": [ { "id": 3428076, "postDate": "2026-03-24T23:24:12.243Z", "content": "That’s a very good question!
", "rawMarkdown": "That’s a very good question!" }, { "id": 3428700, "postDate": "2026-03-25T20:16:10.933Z", "content": "My personal opinion is that those public notebooks or private ones like them with wild variations in scores for the same submissions are very unstable. From my previous experience of such things, their scores on the private LB will be just as wild. What that means is a serious shuffling of the LB when the private scores is revealed at the end of the competition. A drop or a jump of hundreds of positions on the LB is a possibility. We will see.
", "rawMarkdown": "My personal opinion is that those public notebooks or private ones like them with wild variations in scores for the same submissions are very unstable. From my previous experience of such things, their scores on the private LB will be just as wild. What that means is a serious shuffling of the LB when the private scores is revealed at the end of the competition. A drop or a jump of hundreds of positions on the LB is a possibility. We will see." } ] } ] } ], "comments": [ { "id": 3425740, "author_name": "Bianco Chiu", "author_url": "", "post_date": "2026-03-21T07:39:44.527000", "content": "The variance you are observing comes from an unfixed seed. By comparing those public notebooks, you'll notice that some of them are just a copy of another notebook but shaken to a higher score.\nThe scores shown at the top of a public notebook should include both the public score of the current version you are browsing and the best score of this notebook (for example, Public Score 0.xxx, Best Score 0.xxx V1).
", "votes": 0, "replies": [ { "id": 3425788, "author_name": "Adi_3375", "author_url": "", "post_date": "2026-03-21T09:19:33.593000", "content": "Will the organizers remove copied notebooks during final evaluation? because this is leading to unfair lb rankings. And the bigger concern is that, anyone with randomness in their notebook, can score higher than a fixed and good notebook. for example say i have a lb score of 0.43 (with seed contamination fixed). But someone elses notebook having seed contamination issue, can randomly score higher than me. How to deal with this?
", "votes": 1, "replies": [ { "id": 3428076, "author_name": "Dido Grigorov", "author_url": "", "post_date": "2026-03-24T23:24:12.243000", "content": "That’s a very good question!
", "votes": 0, "replies": [] }, { "id": 3428700, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2026-03-25T20:16:10.933000", "content": "My personal opinion is that those public notebooks or private ones like them with wild variations in scores for the same submissions are very unstable. From my previous experience of such things, their scores on the private LB will be just as wild. What that means is a serious shuffling of the LB when the private scores is revealed at the end of the competition. A drop or a jump of hundreds of positions on the LB is a possibility. We will see.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3425441": "I've been seeing a lot of copies of the same notebook (TBM + Protenix) in the 'Code' tab for this competition that all seem to score fairly high (around 0.43-0.44). However, when I copy and submit the notebook, I get varying results each submission (around 0.41-0.43). I understand that there is likely some variance coming from Protenix that is causing the variation in the score, but is it reasonable that such variation could account for a 0.03 swing in public LB score?\n\nAnother thing I think I need clarification on is the public score shown for public notebooks. Does that score shown at the top of a public notebook mean that the exact version that you are viewing scored that high, or that this notebook has scored that high after modifications were made that are not visible in the version that is public?", "3425740": "The variance you are observing comes from an unfixed seed. By comparing those public notebooks, you'll notice that some of them are just a copy of another notebook but shaken to a higher score.\nThe scores shown at the top of a public notebook should include both the public score of the current version you are browsing and the best score of this notebook (for example, Public Score 0.xxx, Best Score 0.xxx V1)." } }