The fix to remove duplicate adjacent nodes has been integrated into the github repo. Thanks for catching this, @btbpanda!
" }, { "id": 3360961, "authorName": "An Phan", "votes": 2, "postDate": "2025-12-02T19:37:12.620000", "content": "The problem occurred for 4 parent-child pairs because these child terms have both \"is_a\" and \"part_of\" relationship with their parent, leading to duplicated children in the children list for the parent term. During topological sort, the parent term reached in_degree of 0 earlier than expected, resulting in it being added to order before exhausting all of its children (and thus smaller index than its children).
Example: GO:1990700 has both \"is_a\" and \"part_of\" with parent GO:0006325
\n[Term]\n: GO:\nname: nucleolar chromatin organization\nnamespace: biological_process\n: [PMID:]\nsynonym: EXACT []\nsynonym: EXACT []\nis_a: GO:0006325 ! chromatin organization\nrelationship: part_of GO:0006325 ! chromatin organization\nrelationship: part_of GO:0007000 ! nucleolus organization\nI have tested a temporary fix for the \"Graph\" class, as detailed in my notebook. The children and adj list of each term needs to be de-duplicated after constructing adjacency matrix, before topological sorting. With the topological sort fixed, the remaining propagation process should behave correctly without modifying. We will keep testing this and eventually update the cafaevaluator code to handle this bug.
Interesting edge case, nice find! I encourage PRs to the repo for any issues found (and fixed).
" }, { "id": 3360141, "authorName": "An Phan", "votes": 2, "postDate": "2025-12-01T22:15:12.887000", "content": "Hello, thanks for pointing it out. I did a quick check at the code, and I did observe that some pairs of parent-child terms are not indexed like we would expect (child terms indices should be smaller than parent terms). This is probably due to the structure of the GO DAG where a child term sometimes has a parent that are a few levels further up the DAG (but still considered direct parent due to a direct relationship), which I think led to the mix-up in the resulting topological sort (not sure, I'm checking to see if this was the cause).
\nI created a toy prediction file which only has one prediction for term GO:0006338 (at index 2116 in G.terms_list, and order 25602 in G.order), and the expected result (after propagating) should have its parent term GO:0006325 (at index 2112 in G.terms_list, and order 20878 in G.order) with the same score. During propagation, the parent term GO:0006325 at order 20878 was completely removed from order_ in the \"remove leaves\" step, leading to incorrect propagation. More details in my notebook.
The fastest way to fix this is just to comment out the \"remove leaves\" part in the function propagate and only loop through indexes in G.order, so no parent terms get removed by accident; however, this fix will slow down the evaluation code. Right now, I am figuring out a way to fix the topological sort so that all child terms get smaller indexes than their parents in G.order. It only affects a very small number of terms, and these terms might not show up in the ground truth leaderboard, so no rescoring for now until I can nail down the cause and a good solution.
Hi An Phan.
\nThanks for your reply. I checked your quick fix solution, and, unfortunately, can say it is not enough. Even if we don't remove any terms from order_, you can imagine the case:
There is a graph G with terms_list ['a', 'b', 'c']. There is a chain of terms (from parent to child) a -> b -> c,
There is a ground truth: {a: 0, b: 0, c: 1}
There is G.order, which is wrong: [0, 2, 1]. The correct order supposed to be [2, 1, 0]
Imagine, we applied your fix, and iterate through that order while propagating the ground truth:
\nWe take a as a parent and assign it with the max children value, which is 0 comes from b. The result will be {a: 0, b: 0, c: 1}.
We take c as a parent, but there is no children to propagate, so we keep the same result: {a: 0, b: 0, c: 1}
We take 'b' as a parent and assign it with the max children value, which is 1 comes from c. The result will be {a: 0, b: 1, c: 1}
After the final iteration, you can see that result is different from expected which is {a: 1, b: 1, c: 1}
The most simple, however, a bit inefficient solution that works here (only for the given Graph, not general case) is to apply your fix + add propagation twice, so just replace line 152 CAFA-evaluator-PK/src/cafaeval/parser.py together with comment remove leaves part
with
\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\npropagate(matrix, ontologies[ns], ontologies[ns].order, mode=)\nlogging.debug(.(ns, matrix))\nTo compensate the inefficiency, I suggest you to replace your propagate function with something more efficient, like I did in my CAFA5 code. To be honest, many things could be done more efficient in that code, but let's focus on propagation now😀
\n numba njit, prange\n\n\n ():\n i prange(mat.shape[]):\n mat[i, k] == :\n \n\n j adj:\n mat[i, j] == :\n mat[i, k] = \n \n\n \n\n\n ():\n i prange(mat.shape[]):\n j adj:\n mat[i, j] > mat[i, k]:\n mat[i, k] = mat[i, j] \n\n \n\n ():\n\n fn = prop_max_cpu_binary binary prop_max_cpu\n\n f G.order:\n adj = G.terms_list[f][]\n\n (adj) == :\n \n\n fn(mat, f, np.asarray(adj))\n\n \nand then replace the target propagation at line 152 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix, ontologies[ns], binary=) \npropagate_max(matrix, ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nand replace prediction propagation at line 223 CAFA-evaluator-PK/src/cafaeval/parser.py with
propagate_max(matrix[ns], ontologies[ns], binary=) \npropagate_max(matrix[ns], ontologies[ns], binary=) \nlogging.debug(.(ns, matrix))\nBut of course, the most proper way for the research purposes is to fix .top_sort method of Graph