![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Fc56fd08247bc1cfeb3b3883cb1e7e91a%2Fmetric.png?generation=1688392382886578&alt=media\")
Hello everyone,
\nI want to express my gratitude for those who reported the bug and provided the fix. After investigating the issue, I can confirm that it was affecting the evaluation of non-propagated predictions and those where parents were predicted with lower scores than their children. This bug had a significant impact on the dashboard, and I apologize for any inconvenience it may have caused.
\nI have pushed the fix to the main branch of the CAFA-evaluator repository. The Kaggle team will now take over and recalculate the dashboard with the updated code. This process should happen as soon as possible, likely at the beginning of next week. They will communicate all the details and timeline on a pinned post.
" }, { "id": 2324438, "authorName": "Ogurtsov", "votes": 1, "postDate": "2023-06-30T15:46:54.297000", "content": "\n\nThe submission metric value depends on rows order
\n
No, LB metric is not order-sensitive.
" }, { "id": 2324570, "authorName": "Btbpanda", "votes": 2, "postDate": "2023-06-30T17:36:26.020000", "content": "If you mean, that shuffling the predictions doesn't affect your score, you are right. But the order of ground truth matters here. This is the part of pred_parser function:
(pred_file) f:\n line f:\n line = line.strip().split()\n line (line) > :\n p_id, term_id, prob = line[:]\n ns = ns_dict.get(term_id)\n ns gts p_id gts[ns].ids:\n i = gts[ns].ids[p_id]\n max_terms np.count_nonzero(matrix[ns][i]) <= max_terms:\n j = onts[ns].terms_dict.get(term_id)[]\n ids[ns][p_id] = i\n matrix[ns][i, j] = (matrix[ns][i, j], (prob))\nHere i is the row number in the prediction matrix. They are trying to obtain it from gts dict if the protein id p_id is there. So the order of prediction is defined by the ground truth file, which is parsed first. And I believe, if ones the organizers shuffle their targets, scores will be changed at least a little. Anyway, even if you freeze all the orders, I just follow the propagate function logic, and I suppose it is not a desired behavior.
To add more examples:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F6d66f303d33711db202fe24e0b2f6561%2Fmetric2.png?generation=1688392666181201&alt=media\")
The difference in scores can be quite big.
\nAs OP proposed it might depends on the order of the columns of \"Y\",
\nhere the order of Y columns is not by the number of \"1\", but different
\n(taken from Zoltan's script: \"Combine embeddings\")
This might only happen to those who do not perform parent propagation prior to the submission.
\nI just checked my propagated version results, and this correction has no effect on the evaluation scoring.
Thanks for pointing this out.
\nI believe you are correct, this would also fill protein with max child predictions of the first row.
\nUsing the following demo as an example,
\nassuming that col 1 is the term we are doing propagation, and col 2,3,4 are col 1's children.
\nThe correct value for row1 (protein1) should be 7 and for row2 (protein2) should be 12.
\nHowever, using the provided code, both of them are 7.
\nYour solution would fix this problem.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13769621%2F2a10c93cfa56e6e36ffcc96e659b755a%2FScreenshot%202023-06-30%20140952.png?generation=1688148606000433&alt=media\")
We'll take a look at this but since we're heading into July 4th weekend I don't expect that we'll be able to get back to you until the middle of next week.
" } ], "index": { "id": "420241", "title": "(Probably) Error in the metric computation", "authorName": "", "commentCount": "7", "votes": "28", "postDate": "2023-06-29 21:02:17.072000" }, "competition": "cafa-5-protein-function-prediction" }