{ "id": 682880, "title": "Clarification on GPU Configuration in the Submission Environment", "url": "/competitions/stanford-rna-3d-folding-2/discussion/682880", "author_name": "Yusaku Muroya", "post_date": "2026-03-19T08:10:05.402000", "votes": 5, "comment_count": 9, "views": 0, "content": "

Dear organizers and fellow participants,
\nThank you for hosting this wonderful competition. I hope this is the right place to ask this question.

\n

I have been experiencing an issue with submission timeouts that I cannot fully explain, and I would greatly appreciate any clarification.

\n

What I observed:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
NotebookGPU SettingRuntimeSubmission Result
Notebook AT4 x2 (parallel computation)~4h 20minTimeout
Notebook BT4 x2~1hSuccess
Notebook CP100~1hSuccess
\n

Notebook A leverages both T4 GPUs in parallel to achieve its runtime of approximately 4 hours 20 minutes, which should be well within the 8-hour limit. Despite this, it consistently times out upon submission.

\n

My hypothesis:

\n

Given that shorter-running notebooks (both T4 x2 and P100) succeed while Notebook A (T4 x2 parallel, ~4h 20min) fails, I am wondering whether the submission environment might always use a P100 (single GPU) regardless of the GPU option selected in the notebook settings. If so, this would explain the timeout, as my parallel processing code would lose access to the second GPU and would also need to run on different hardware than it was optimized for.

\n

Of course, this is just my speculation, and I may be overlooking something else entirely.

\n

My questions:

\n
    \n
  1. Could you kindly confirm what GPU configuration is used when notebooks are evaluated during submission?
    2. \n
  2. If only P100 is available for submission, is this documented somewhere that I may have missed?
    3. \n
\n

I ask because, with about one week remaining in the competition, understanding the exact hardware constraints would help me (and likely other participants as well) plan our final approach more effectively.\nThank you very much for your time, and apologies if this has already been addressed elsewhere.

\n

Best regards

", "messages": [ { "id": 3423979, "postDate": "2026-03-19T08:10:05.403Z", "content": "

Dear organizers and fellow participants,
\nThank you for hosting this wonderful competition. I hope this is the right place to ask this question.

\n

I have been experiencing an issue with submission timeouts that I cannot fully explain, and I would greatly appreciate any clarification.

\n

What I observed:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
NotebookGPU SettingRuntimeSubmission Result
Notebook AT4 x2 (parallel computation)~4h 20minTimeout
Notebook BT4 x2~1hSuccess
Notebook CP100~1hSuccess
\n

Notebook A leverages both T4 GPUs in parallel to achieve its runtime of approximately 4 hours 20 minutes, which should be well within the 8-hour limit. Despite this, it consistently times out upon submission.

\n

My hypothesis:

\n

Given that shorter-running notebooks (both T4 x2 and P100) succeed while Notebook A (T4 x2 parallel, ~4h 20min) fails, I am wondering whether the submission environment might always use a P100 (single GPU) regardless of the GPU option selected in the notebook settings. If so, this would explain the timeout, as my parallel processing code would lose access to the second GPU and would also need to run on different hardware than it was optimized for.

\n

Of course, this is just my speculation, and I may be overlooking something else entirely.

\n

My questions:

\n
    \n
  1. Could you kindly confirm what GPU configuration is used when notebooks are evaluated during submission?
    2. \n
  2. If only P100 is available for submission, is this documented somewhere that I may have missed?
    3. \n
\n

I ask because, with about one week remaining in the competition, understanding the exact hardware constraints would help me (and likely other participants as well) plan our final approach more effectively.\nThank you very much for your time, and apologies if this has already been addressed elsewhere.

\n

Best regards

", "rawMarkdown": "Dear organizers and fellow participants, \nThank you for hosting this wonderful competition. I hope this is the right place to ask this question. \n\nI have been experiencing an issue with submission timeouts that I cannot fully explain, and I would greatly appreciate any clarification.\n\n### What I observed: \n| Notebook | GPU Setting | Runtime | Submission Result | \n|---|---|---|---|\n| Notebook A | T4 x2 (parallel computation) | ~4h 20min | **Timeout** |\n| Notebook B | T4 x2 | ~1h | Success |\n| Notebook C | P100 | ~1h | Success | \n\nNotebook A leverages both T4 GPUs in parallel to achieve its runtime of approximately 4 hours 20 minutes, which should be well within the 8-hour limit. Despite this, it consistently times out upon submission.\n \n### My hypothesis:\nGiven that shorter-running notebooks (both T4 x2 and P100) succeed while Notebook A (T4 x2 parallel, ~4h 20min) fails, I am wondering whether the submission environment might always use a **P100 (single GPU)** regardless of the GPU option selected in the notebook settings. If so, this would explain the timeout, as my parallel processing code would lose access to the second GPU and would also need to run on different hardware than it was optimized for.\n\nOf course, this is just my speculation, and I may be overlooking something else entirely.\n\n### My questions: \n1. Could you kindly confirm what GPU configuration is used when notebooks are evaluated during submission?\n2. If only P100 is available for submission, is this documented somewhere that I may have missed?\n\nI ask because, with about one week remaining in the competition, understanding the exact hardware constraints would help me (and likely other participants as well) plan our final approach more effectively.\nThank you very much for your time, and apologies if this has already been addressed elsewhere.\n\nBest regards", "votes": 5 }, { "id": 3424460, "postDate": "2026-03-19T17:39:20.780Z", "content": "

We always run the submission on the same GPU type as the originating notebook in order to guarantee performance characteristics are the same.

\n

Without looking at your code, I'd just suspect that your \"parallel\" workflow introduces some other bottleneck that actually slows things down more.

", "rawMarkdown": "We always run the submission on the same GPU type as the originating notebook in order to guarantee performance characteristics are the same.\n\nWithout looking at your code, I'd just suspect that your \"parallel\" workflow introduces some other bottleneck that actually slows things down more.", "votes": 1, "isPinned": true, "replies": [ { "id": 3424958, "postDate": "2026-03-20T06:16:06.863Z", "content": "

Thank you very much for the clarification, Dustin! That's reassuring to know. I've been running my notebooks with T4\n x2 through the UI, so the GPU type should be consistent. However, I noticed that my submissions take significantly\n longer than Save & Run (~3h Save & Run vs 5h+ still scoring on the same notebook). This might suggest the hidden test\n dataset has different characteristics, as Andrew mentioned. Thanks again for the quick response!

", "rawMarkdown": "Thank you very much for the clarification, Dustin! That's reassuring to know. I've been running my notebooks with T4\n x2 through the UI, so the GPU type should be consistent. However, I noticed that my submissions take significantly\n longer than Save & Run (~3h Save & Run vs 5h+ still scoring on the same notebook). This might suggest the hidden test\n dataset has different characteristics, as Andrew mentioned. Thanks again for the quick response!" } ] }, { "id": 3424391, "postDate": "2026-03-19T16:36:17.533Z", "content": "

I’m assuming your submission is a mix of TBM and Protenix or another deep learning model that utilizes the GPU. My best guess is that in the private leaderboard dataset there is less structural homologous in the train dataset, so more predictions are routed to Protenix/other deep learning models compared to running on the public test dataset, as many targets in that dataset already have near matches in the available train dataset.

", "rawMarkdown": "I’m assuming your submission is a mix of TBM and Protenix or another deep learning model that utilizes the GPU. My best guess is that in the private leaderboard dataset there is less structural homologous in the train dataset, so more predictions are routed to Protenix/other deep learning models compared to running on the public test dataset, as many targets in that dataset already have near matches in the available train dataset.", "replies": [ { "id": 3424400, "postDate": "2026-03-19T16:45:13.143Z", "content": "

The 8 hour runtime limit is based on the submission scoring using the private leaderboard dataset, not the public test dataset.

", "rawMarkdown": "The 8 hour runtime limit is based on the submission scoring using the private leaderboard dataset, not the public test dataset.", "replies": [ { "id": 3424955, "postDate": "2026-03-20T06:13:35.087Z", "content": "

Great insight, Andrew — I think you might be right. My submission is taking much longer than Save & Run on the same\n GPU setting, which is consistent with your hypothesis about the private dataset routing more predictions to deep learning models. Thanks for pointing this out!

", "rawMarkdown": "Great insight, Andrew — I think you might be right. My submission is taking much longer than Save & Run on the same\n GPU setting, which is consistent with your hypothesis about the private dataset routing more predictions to deep learning models. Thanks for pointing this out!" } ] } ] }, { "id": 3424356, "postDate": "2026-03-19T16:03:31.923Z", "content": "

my parallel config using 2xT4 runs about the same time as sequential P100 in submission rerun and i can't really explain why. But i tested it locally everything works out fine, so I share your concern…

", "rawMarkdown": "my parallel config using 2xT4 runs about the same time as sequential P100 in submission rerun and i can't really explain why. But i tested it locally everything works out fine, so I share your concern...", "replies": [ { "id": 3424956, "postDate": "2026-03-20T06:14:14.437Z", "content": "

Thanks for sharing your experience — it's helpful to know I'm not alone in this. I'm also seeing submissions take\n significantly longer than Save & Run, even with the same GPU configuration. Based on the other comments here, it may\n be related to the hidden test dataset having different characteristics rather than a GPU issue.

", "rawMarkdown": "Thanks for sharing your experience — it's helpful to know I'm not alone in this. I'm also seeing submissions take\n significantly longer than Save & Run, even with the same GPU configuration. Based on the other comments here, it may\n be related to the hidden test dataset having different characteristics rather than a GPU issue.", "replies": [ { "id": 3424969, "postDate": "2026-03-20T06:22:57.503Z", "content": "

yes, i'm aware of the hidden test set, however, still my parallel setup runs about the same time as sequential setup in the hidden test set, maybe i have a bug in the code or something tho.

", "rawMarkdown": "yes, i'm aware of the hidden test set, however, still my parallel setup runs about the same time as sequential setup in the hidden test set, maybe i have a bug in the code or something tho." } ] } ] }, { "id": 3424086, "postDate": "2026-03-19T10:47:29.227Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3424460, "author_name": "Dustin", "author_url": "", "post_date": "2026-03-19T17:39:20.780000", "content": "

We always run the submission on the same GPU type as the originating notebook in order to guarantee performance characteristics are the same.

\n

Without looking at your code, I'd just suspect that your \"parallel\" workflow introduces some other bottleneck that actually slows things down more.

", "votes": 1, "replies": [ { "id": 3424958, "author_name": "Yusaku Muroya", "author_url": "", "post_date": "2026-03-20T06:16:06.863000", "content": "

Thank you very much for the clarification, Dustin! That's reassuring to know. I've been running my notebooks with T4\n x2 through the UI, so the GPU type should be consistent. However, I noticed that my submissions take significantly\n longer than Save & Run (~3h Save & Run vs 5h+ still scoring on the same notebook). This might suggest the hidden test\n dataset has different characteristics, as Andrew mentioned. Thanks again for the quick response!

", "votes": 0, "replies": [] } ] }, { "id": 3424391, "author_name": "Andrew Lee", "author_url": "", "post_date": "2026-03-19T16:36:17.533000", "content": "

I’m assuming your submission is a mix of TBM and Protenix or another deep learning model that utilizes the GPU. My best guess is that in the private leaderboard dataset there is less structural homologous in the train dataset, so more predictions are routed to Protenix/other deep learning models compared to running on the public test dataset, as many targets in that dataset already have near matches in the available train dataset.

", "votes": 0, "replies": [ { "id": 3424400, "author_name": "Andrew Lee", "author_url": "", "post_date": "2026-03-19T16:45:13.143000", "content": "

The 8 hour runtime limit is based on the submission scoring using the private leaderboard dataset, not the public test dataset.

", "votes": 0, "replies": [ { "id": 3424955, "author_name": "Yusaku Muroya", "author_url": "", "post_date": "2026-03-20T06:13:35.087000", "content": "

Great insight, Andrew — I think you might be right. My submission is taking much longer than Save & Run on the same\n GPU setting, which is consistent with your hypothesis about the private dataset routing more predictions to deep learning models. Thanks for pointing this out!

", "votes": 0, "replies": [] } ] } ] }, { "id": 3424356, "author_name": "hongan", "author_url": "", "post_date": "2026-03-19T16:03:31.923000", "content": "

my parallel config using 2xT4 runs about the same time as sequential P100 in submission rerun and i can't really explain why. But i tested it locally everything works out fine, so I share your concern…

", "votes": 0, "replies": [ { "id": 3424956, "author_name": "Yusaku Muroya", "author_url": "", "post_date": "2026-03-20T06:14:14.437000", "content": "

Thanks for sharing your experience — it's helpful to know I'm not alone in this. I'm also seeing submissions take\n significantly longer than Save & Run, even with the same GPU configuration. Based on the other comments here, it may\n be related to the hidden test dataset having different characteristics rather than a GPU issue.

", "votes": 0, "replies": [ { "id": 3424969, "author_name": "hongan", "author_url": "", "post_date": "2026-03-20T06:22:57.503000", "content": "

yes, i'm aware of the hidden test set, however, still my parallel setup runs about the same time as sequential setup in the hidden test set, maybe i have a bug in the code or something tho.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3424086, "author_name": "", "author_url": "", "post_date": "2026-03-19T10:47:29.227000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3423979": "Dear organizers and fellow participants, \nThank you for hosting this wonderful competition. I hope this is the right place to ask this question. \n\nI have been experiencing an issue with submission timeouts that I cannot fully explain, and I would greatly appreciate any clarification.\n\n### What I observed: \n| Notebook | GPU Setting | Runtime | Submission Result | \n|---|---|---|---|\n| Notebook A | T4 x2 (parallel computation) | ~4h 20min | **Timeout** |\n| Notebook B | T4 x2 | ~1h | Success |\n| Notebook C | P100 | ~1h | Success | \n\nNotebook A leverages both T4 GPUs in parallel to achieve its runtime of approximately 4 hours 20 minutes, which should be well within the 8-hour limit. Despite this, it consistently times out upon submission.\n \n### My hypothesis:\nGiven that shorter-running notebooks (both T4 x2 and P100) succeed while Notebook A (T4 x2 parallel, ~4h 20min) fails, I am wondering whether the submission environment might always use a **P100 (single GPU)** regardless of the GPU option selected in the notebook settings. If so, this would explain the timeout, as my parallel processing code would lose access to the second GPU and would also need to run on different hardware than it was optimized for.\n\nOf course, this is just my speculation, and I may be overlooking something else entirely.\n\n### My questions: \n1. Could you kindly confirm what GPU configuration is used when notebooks are evaluated during submission?\n2. If only P100 is available for submission, is this documented somewhere that I may have missed?\n\nI ask because, with about one week remaining in the competition, understanding the exact hardware constraints would help me (and likely other participants as well) plan our final approach more effectively.\nThank you very much for your time, and apologies if this has already been addressed elsewhere.\n\nBest regards", "3424460": "We always run the submission on the same GPU type as the originating notebook in order to guarantee performance characteristics are the same.\n\nWithout looking at your code, I'd just suspect that your \"parallel\" workflow introduces some other bottleneck that actually slows things down more.", "3424391": "I’m assuming your submission is a mix of TBM and Protenix or another deep learning model that utilizes the GPU. My best guess is that in the private leaderboard dataset there is less structural homologous in the train dataset, so more predictions are routed to Protenix/other deep learning models compared to running on the public test dataset, as many targets in that dataset already have near matches in the available train dataset.", "3424356": "my parallel config using 2xT4 runs about the same time as sequential P100 in submission rerun and i can't really explain why. But i tested it locally everything works out fine, so I share your concern...", "3424086": "" } }