Edit: The issue I raised has been fixed, see @sohier comment below. TL;DR The competition metric should be used with a value of 57.846 for the fgs_weight.
\n---------- original issue description below ----------
\nOverview page states this:
\nThe channels are weighted proportionally to the spectral channel width divided by the number of spectral points per instrument. Additionally, FGS1 weight has been doubled to emphasize its importance:
\n: × (./) = .\n\n-Ch0: ./ ≈ . per spectral point\nThe score will return a float in the interval [0, 1], with higher scores corresponding to better performing models. Any score below 0 will be treated as 0.
\nHowever, the metric code ends with this:
\n weights = np.append(np.array([fgs_weight]), np.ones((solution.columns) - ))\n weights = weights * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\nwith a default value for fgs_weight equal to 1.
\nAs a result the weights are all ones when they should be [0,4, 0.0069, 0.0069, …, 0.0069]
\nA correct code would be this:
\n weights_0 = np.append(np.array([]), np.ones((solution.columns) - ) * )\n weights = weights_0 * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n (np.clip(submit_score, , )), ind_scores, weights_0\n@sohier what do you think?
", "messages": [ { "id": 3248289, "postDate": "2025-07-14T09:47:58.207Z", "content": "Edit: The issue I raised has been fixed, see @sohier comment below. TL;DR The competition metric should be used with a value of 57.846 for the fgs_weight.
\n---------- original issue description below ----------
\nOverview page states this:
\nThe channels are weighted proportionally to the spectral channel width divided by the number of spectral points per instrument. Additionally, FGS1 weight has been doubled to emphasize its importance:
\n: × (./) = .\n\n-Ch0: ./ ≈ . per spectral point\nThe score will return a float in the interval [0, 1], with higher scores corresponding to better performing models. Any score below 0 will be treated as 0.
\nHowever, the metric code ends with this:
\n weights = np.append(np.array([fgs_weight]), np.ones((solution.columns) - ))\n weights = weights * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\nwith a default value for fgs_weight equal to 1.
\nAs a result the weights are all ones when they should be [0,4, 0.0069, 0.0069, …, 0.0069]
\nA correct code would be this:
\n weights_0 = np.append(np.array([]), np.ones((solution.columns) - ) * )\n weights = weights_0 * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n (np.clip(submit_score, , )), ind_scores, weights_0\n@sohier what do you think?
", "rawMarkdown": "**Edit:** The issue I raised has been fixed, see @sohier comment below. TL;DR The competition metric should be used with a value of 57.846 for the fgs_weight.\n\n---------- original issue description below ----------\n\nOverview page states this:\n\n*The channels are weighted proportionally to the spectral channel width divided by the number of spectral points per instrument. Additionally, FGS1 weight has been doubled to emphasize its importance:*\n\n FGS1: 2 × (0.2/1) = 0.4\n\n AIRS-Ch0: 1.95/282 ≈ 0.0069 per spectral point\n\n*The score will return a float in the interval [0, 1], with higher scores corresponding to better performing models. Any score below 0 will be treated as 0.*\n\nHowever, the metric code ends with this:\n\n```python\n weights = np.append(np.array([fgs_weight]), np.ones(len(solution.columns) - 1))\n weights = weights * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n```\nwith a default value for fgs_weight equal to 1.\n\nAs a result the weights are all ones when they should be [0,4, 0.0069, 0.0069, ..., 0.0069]\n\nA correct code would be this:\n\n```python\n weights_0 = np.append(np.array([0.4]), np.ones(len(solution.columns) - 1) * 0.0069)\n weights = weights_0 * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n return float(np.clip(submit_score, 0.0, 1.0)), ind_scores, weights_0\n\n```\n\n@sohier what do you think?", "votes": 11 }, { "id": 3248378, "postDate": "2025-07-14T13:00:43.303Z", "content": "This notebook describes this in detail competition metric in detail, at least with regards to the weights.
\nIf you use the competition metric code and change the default fgs_weight=57.846…, you will get the same answer you have above; (.4/.0069=57.846). Because np.average is a weighted average, multiplying all the weights by a constant does not change the result.
@solverworld Thanks, i was lazy to document the fgs_weight default that should be used with the original code.
\nMy point is that the provided competition code is wrong, and you confirmed it.
\n@sohier Is the LB score computed correctly? This is the question that matters.
", "rawMarkdown": "@solverworld Thanks, i was lazy to document the fgs_weight default that should be used with the original code. \n\nMy point is that the provided competition code is wrong, and you confirmed it.\n\n@sohier Is the LB score computed correctly? This is the question that matters.", "votes": 1, "replies": [ { "id": 3248421, "postDate": "2025-07-14T14:49:59.507Z", "content": "Agreed. I made a mistake originally by setting fgs_weight=2, which is also wrong.
", "rawMarkdown": "Agreed. I made a mistake originally by setting fgs_weight=2, which is also wrong.", "votes": 1 }, { "id": 3248469, "postDate": "2025-07-14T16:40:27.860Z", "content": "I also doubt whether LB is correct
", "rawMarkdown": "I also doubt whether LB is correct" }, { "id": 3249876, "postDate": "2025-07-17T09:22:36.610Z", "content": "@horikitasaku You can change the sigma for FGS1 only while those for AIRS fixed and see how the Public LB change. My understanding is that the weights and other constants are consistent with their description.
", "rawMarkdown": "@horikitasaku You can change the sigma for FGS1 only while those for AIRS fixed and see how the Public LB change. My understanding is that the weights and other constants are consistent with their description." }, { "id": 3249923, "postDate": "2025-07-17T11:15:40.947Z", "content": "You're right. I tried and found no problem.
", "rawMarkdown": "You're right. I tried and found no problem.", "votes": 2 } ] }, { "id": 3248455, "postDate": "2025-07-14T16:12:34.680Z", "content": "@solverworld is correct, the deployed metric uses the normalized fgs_weight of exactly 57.846. I'll coordinate with the host on updating the overview tab for clarity.
\nEdit: We've updated the evaluation tab to clarify that the metric requires the normalized weight.
", "rawMarkdown": "@solverworld is correct, the deployed metric uses the normalized fgs_weight of exactly 57.846. I'll coordinate with the host on updating the overview tab for clarity.\n\nEdit: We've updated the evaluation tab to clarify that the metric requires the normalized weight.", "votes": 3, "replies": [ { "id": 3248456, "postDate": "2025-07-14T16:13:28.103Z", "content": "Good to know, thanks for confirming.
", "rawMarkdown": "Good to know, thanks for confirming.", "votes": 1 } ] } ] } ], "comments": [ { "id": 3248378, "author_name": "SolverWorld", "author_url": "", "post_date": "2025-07-14T13:00:43.303000", "content": "This notebook describes this in detail competition metric in detail, at least with regards to the weights.
\nIf you use the competition metric code and change the default fgs_weight=57.846…, you will get the same answer you have above; (.4/.0069=57.846). Because np.average is a weighted average, multiplying all the weights by a constant does not change the result.
@solverworld Thanks, i was lazy to document the fgs_weight default that should be used with the original code.
\nMy point is that the provided competition code is wrong, and you confirmed it.
\n@sohier Is the LB score computed correctly? This is the question that matters.
", "votes": 1, "replies": [ { "id": 3248421, "author_name": "SolverWorld", "author_url": "", "post_date": "2025-07-14T14:49:59.507000", "content": "Agreed. I made a mistake originally by setting fgs_weight=2, which is also wrong.
", "votes": 1, "replies": [] }, { "id": 3248469, "author_name": "Horikita Saku", "author_url": "", "post_date": "2025-07-14T16:40:27.860000", "content": "I also doubt whether LB is correct
", "votes": 0, "replies": [] }, { "id": 3249876, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2025-07-17T09:22:36.610000", "content": "@horikitasaku You can change the sigma for FGS1 only while those for AIRS fixed and see how the Public LB change. My understanding is that the weights and other constants are consistent with their description.
", "votes": 0, "replies": [] }, { "id": 3249923, "author_name": "Horikita Saku", "author_url": "", "post_date": "2025-07-17T11:15:40.947000", "content": "You're right. I tried and found no problem.
", "votes": 2, "replies": [] } ] }, { "id": 3248455, "author_name": "Sohier Dane", "author_url": "", "post_date": "2025-07-14T16:12:34.680000", "content": "@solverworld is correct, the deployed metric uses the normalized fgs_weight of exactly 57.846. I'll coordinate with the host on updating the overview tab for clarity.
\nEdit: We've updated the evaluation tab to clarify that the metric requires the normalized weight.
", "votes": 3, "replies": [ { "id": 3248456, "author_name": "CPMP", "author_url": "", "post_date": "2025-07-14T16:13:28.103000", "content": "Good to know, thanks for confirming.
", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3248289": "**Edit:** The issue I raised has been fixed, see @sohier comment below. TL;DR The competition metric should be used with a value of 57.846 for the fgs_weight.\n\n---------- original issue description below ----------\n\nOverview page states this:\n\n*The channels are weighted proportionally to the spectral channel width divided by the number of spectral points per instrument. Additionally, FGS1 weight has been doubled to emphasize its importance:*\n\n FGS1: 2 × (0.2/1) = 0.4\n\n AIRS-Ch0: 1.95/282 ≈ 0.0069 per spectral point\n\n*The score will return a float in the interval [0, 1], with higher scores corresponding to better performing models. Any score below 0 will be treated as 0.*\n\nHowever, the metric code ends with this:\n\n```python\n weights = np.append(np.array([fgs_weight]), np.ones(len(solution.columns) - 1))\n weights = weights * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n```\nwith a default value for fgs_weight equal to 1.\n\nAs a result the weights are all ones when they should be [0,4, 0.0069, 0.0069, ..., 0.0069]\n\nA correct code would be this:\n\n```python\n weights_0 = np.append(np.array([0.4]), np.ones(len(solution.columns) - 1) * 0.0069)\n weights = weights_0 * np.ones_like(ind_scores)\n submit_score = np.average(ind_scores, weights=weights)\n return float(np.clip(submit_score, 0.0, 1.0)), ind_scores, weights_0\n\n```\n\n@sohier what do you think?", "3248378": "This notebook describes this in detail [competition metric in detail](https://www.kaggle.com/code/junkoda/competition-metric-in-detail), at least with regards to the weights.\nIf you use the competition metric code and change the default fgs_weight=57.846..., you will get the same answer you have above; (.4/.0069=57.846). Because np.average is a weighted average, multiplying all the weights by a constant does not change the result.\n" } }