There have been a number of questions on inference time in this forum. However, they were either unanswered or not verified by the official organizers, leaving much confusion on the available inference time still:
\nI think they can be summarized by the following 3 underlying questions:
\nQ1: on average, how much time should we reserve for one test case for inference? Is it the estimated 17 seconds per case?
\nQ2: how do we know a successful submission to the public LB will not result in 12-hour timeout error in the final submission?
\nQ3: will the private and public testsets be evaluated in parallel and each will have 12-hour time limit?
\nCan the organizers please help clarify on the available inference time? Right now I think many people are worried about the 12 hours error without clear information.
", "messages": [ { "id": 3289981, "postDate": "2025-09-16T22:43:41.257Z", "content": "There have been a number of questions on inference time in this forum. However, they were either unanswered or not verified by the official organizers, leaving much confusion on the available inference time still:
\nI think they can be summarized by the following 3 underlying questions:
\nQ1: on average, how much time should we reserve for one test case for inference? Is it the estimated 17 seconds per case?
\nQ2: how do we know a successful submission to the public LB will not result in 12-hour timeout error in the final submission?
\nQ3: will the private and public testsets be evaluated in parallel and each will have 12-hour time limit?
\nCan the organizers please help clarify on the available inference time? Right now I think many people are worried about the 12 hours error without clear information.
", "rawMarkdown": "There have been a number of questions on inference time in this forum. However, they were either unanswered or not verified by the official organizers, leaving much confusion on the available inference time still:\n\n- 17 seconds inference per case (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/602640)\n- private vs public LB evaluation (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/606980)\n- inference time per case (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/603707)\n- dicom loading time and 12 hours (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/599253)\n- notebook timeout error local vs 12 hours (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/603776)\n\nI think they can be summarized by the following 3 underlying questions:\n\nQ1: on average, how much time should we reserve for one test case for inference? Is it the estimated 17 seconds per case?\n\nQ2: how do we know a successful submission to the public LB will not result in 12-hour timeout error in the final submission?\n\nQ3: will the private and public testsets be evaluated in parallel and each will have 12-hour time limit?\n\nCan the organizers please help clarify on the available inference time? Right now I think many people are worried about the 12 hours error without clear information.", "votes": 5 }, { "id": 3290846, "postDate": "2025-09-18T13:56:20.787Z", "content": "Hi @makelove,
\nQ1: From the Data page: \"There are nearly 2500 series in the test set.\" You should divide up the 12 hour allowed runtime across these ~2500 instances. You could consider a \"failsafe\" mechanism that just submits a default prediction if your submission approaches the 12 hour limit.
\nQ2: Every submission to this competition runs on the full test set, both public and private. This is in contrast to competitions like Jane Street which reruns submissions on a new private set after the competition ends. In other words, in this competition, if your submission succeeds on the public LB, it has also already succeeded on the private LB. The private LB scores will be revealed (but not computed) once the competition ends.
\nQ3: The combined runtime on both test sets is 12 hours. Your submission makes predictions on all test instances in sequence, randomly interleaved between public and private. Your submission doesn't know which are public and which are private.
", "rawMarkdown": "Hi @makelove,\n\nQ1: From the Data page: \"There are nearly 2500 series in the test set.\" You should divide up the 12 hour allowed runtime across these ~2500 instances. You could consider a \"failsafe\" mechanism that just submits a default prediction if your submission approaches the 12 hour limit.\n\nQ2: Every submission to this competition runs on the full test set, both public and private. This is in contrast to competitions like [Jane Street](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/data) which reruns submissions on a new private set after the competition ends. In other words, in this competition, if your submission succeeds on the public LB, it has also already succeeded on the private LB. The private LB scores will be revealed (but not computed) once the competition ends.\n\nQ3: The combined runtime on both test sets is 12 hours. Your submission makes predictions on all test instances in sequence, randomly interleaved between public and private. Your submission doesn't know which are public and which are private.", "votes": 6, "replies": [ { "id": 3291385, "postDate": "2025-09-19T15:02:22.263Z", "content": "Thank you very much for your detailed clarification! It is very helpful!
", "rawMarkdown": "Thank you very much for your detailed clarification! It is very helpful!", "votes": 1 }, { "id": 3296995, "postDate": "2025-10-02T05:18:00.613Z", "content": "@ryanholbrook approximately, how many image files can there be in each of these 2500 series
", "rawMarkdown": "@ryanholbrook approximately, how many image files can there be in each of these 2500 series" } ] }, { "id": 3290248, "postDate": "2025-09-17T13:06:17.410Z", "content": "Unfortunately I don't know the answers to these questions. We really need input from @ryanholbrook. I will try to point his attention towards this thread.
", "rawMarkdown": "Unfortunately I don't know the answers to these questions. We really need input from @ryanholbrook. I will try to point his attention towards this thread.", "votes": 1 } ], "comments": [ { "id": 3290846, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2025-09-18T13:56:20.787000", "content": "Hi @makelove,
\nQ1: From the Data page: \"There are nearly 2500 series in the test set.\" You should divide up the 12 hour allowed runtime across these ~2500 instances. You could consider a \"failsafe\" mechanism that just submits a default prediction if your submission approaches the 12 hour limit.
\nQ2: Every submission to this competition runs on the full test set, both public and private. This is in contrast to competitions like Jane Street which reruns submissions on a new private set after the competition ends. In other words, in this competition, if your submission succeeds on the public LB, it has also already succeeded on the private LB. The private LB scores will be revealed (but not computed) once the competition ends.
\nQ3: The combined runtime on both test sets is 12 hours. Your submission makes predictions on all test instances in sequence, randomly interleaved between public and private. Your submission doesn't know which are public and which are private.
", "votes": 6, "replies": [ { "id": 3291385, "author_name": "MakeLoveAndPeace", "author_url": "", "post_date": "2025-09-19T15:02:22.263000", "content": "Thank you very much for your detailed clarification! It is very helpful!
", "votes": 1, "replies": [] }, { "id": 3296995, "author_name": "Suman Kumar Gangopadhyay", "author_url": "", "post_date": "2025-10-02T05:18:00.613000", "content": "@ryanholbrook approximately, how many image files can there be in each of these 2500 series
", "votes": 0, "replies": [] } ] }, { "id": 3290248, "author_name": "Evan Calabrese", "author_url": "", "post_date": "2025-09-17T13:06:17.410000", "content": "Unfortunately I don't know the answers to these questions. We really need input from @ryanholbrook. I will try to point his attention towards this thread.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3289981": "There have been a number of questions on inference time in this forum. However, they were either unanswered or not verified by the official organizers, leaving much confusion on the available inference time still:\n\n- 17 seconds inference per case (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/602640)\n- private vs public LB evaluation (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/606980)\n- inference time per case (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/603707)\n- dicom loading time and 12 hours (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/599253)\n- notebook timeout error local vs 12 hours (https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/603776)\n\nI think they can be summarized by the following 3 underlying questions:\n\nQ1: on average, how much time should we reserve for one test case for inference? Is it the estimated 17 seconds per case?\n\nQ2: how do we know a successful submission to the public LB will not result in 12-hour timeout error in the final submission?\n\nQ3: will the private and public testsets be evaluated in parallel and each will have 12-hour time limit?\n\nCan the organizers please help clarify on the available inference time? Right now I think many people are worried about the 12 hours error without clear information.", "3290846": "Hi @makelove,\n\nQ1: From the Data page: \"There are nearly 2500 series in the test set.\" You should divide up the 12 hour allowed runtime across these ~2500 instances. You could consider a \"failsafe\" mechanism that just submits a default prediction if your submission approaches the 12 hour limit.\n\nQ2: Every submission to this competition runs on the full test set, both public and private. This is in contrast to competitions like [Jane Street](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/data) which reruns submissions on a new private set after the competition ends. In other words, in this competition, if your submission succeeds on the public LB, it has also already succeeded on the private LB. The private LB scores will be revealed (but not computed) once the competition ends.\n\nQ3: The combined runtime on both test sets is 12 hours. Your submission makes predictions on all test instances in sequence, randomly interleaved between public and private. Your submission doesn't know which are public and which are private.", "3290248": "Unfortunately I don't know the answers to these questions. We really need input from @ryanholbrook. I will try to point his attention towards this thread." } }