{ "id": 357818, "title": "Test data exceeds memory limitation?", "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/357818", "author_name": "Marco Ciavarella", "post_date": "2022-10-05T16:13:17.768000", "votes": 3, "comment_count": 4, "views": 0, "content": "

Hello guys,
\nI'm having a hard time actually loading in the test data to do inference.

\n

I'm only able to load in about ~10% of the test data, then I get a memory allocation error.
\n\"\"

\n

Is anyone else having this issue?

", "messages": [ { "id": 1973471, "postDate": "2022-10-05T16:13:17.767Z", "content": "

Hello guys,
\nI'm having a hard time actually loading in the test data to do inference.

\n

I'm only able to load in about ~10% of the test data, then I get a memory allocation error.
\n\"\"

\n

Is anyone else having this issue?

", "rawMarkdown": "Hello guys,\nI'm having a hard time actually loading in the test data to do inference.\n\nI'm only able to load in about ~10% of the test data, then I get a memory allocation error.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5247984%2F7a1598b203781bfbceba18280b59a7cb%2Foom_error_g2net.png?generation=1664986366840355&alt=media)\n\nIs anyone else having this issue?", "votes": 3 }, { "id": 1973656, "postDate": "2022-10-05T18:44:21.733Z", "content": "

The test data is something like 200GB. The RAM on the notebooks is ~16GB. You won't be able to load all of it at once.

", "rawMarkdown": "The test data is something like 200GB. The RAM on the notebooks is ~16GB. You won't be able to load all of it at once.", "votes": 1, "replies": [ { "id": 1973664, "postDate": "2022-10-05T18:47:28.500Z", "content": "

So what do you suggest? Multiple notebooks handling batches of test data?

", "rawMarkdown": "So what do you suggest? Multiple notebooks handling batches of test data?" }, { "id": 1973804, "postDate": "2022-10-05T21:07:45.283Z", "content": "

If you're just doing inference, then loop over them, scoring them in batches or one-by-one.

", "rawMarkdown": "If you're just doing inference, then loop over them, scoring them in batches or one-by-one.", "votes": 4 }, { "id": 1974562, "postDate": "2022-10-06T10:10:54.390Z", "content": "

Thanks for the tip, trying out batch inference right now!

", "rawMarkdown": "Thanks for the tip, trying out batch inference right now!" } ] } ], "comments": [ { "id": 1973656, "author_name": "Branden Murray", "author_url": "", "post_date": "2022-10-05T18:44:21.733000", "content": "

The test data is something like 200GB. The RAM on the notebooks is ~16GB. You won't be able to load all of it at once.

", "votes": 1, "replies": [ { "id": 1973664, "author_name": "Marco Ciavarella", "author_url": "", "post_date": "2022-10-05T18:47:28.500000", "content": "

So what do you suggest? Multiple notebooks handling batches of test data?

", "votes": 0, "replies": [] }, { "id": 1973804, "author_name": "Branden Murray", "author_url": "", "post_date": "2022-10-05T21:07:45.283000", "content": "

If you're just doing inference, then loop over them, scoring them in batches or one-by-one.

", "votes": 4, "replies": [] }, { "id": 1974562, "author_name": "Marco Ciavarella", "author_url": "", "post_date": "2022-10-06T10:10:54.390000", "content": "

Thanks for the tip, trying out batch inference right now!

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1973471": "Hello guys,\nI'm having a hard time actually loading in the test data to do inference.\n\nI'm only able to load in about ~10% of the test data, then I get a memory allocation error.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5247984%2F7a1598b203781bfbceba18280b59a7cb%2Foom_error_g2net.png?generation=1664986366840355&alt=media)\n\nIs anyone else having this issue?", "1973656": "The test data is something like 200GB. The RAM on the notebooks is ~16GB. You won't be able to load all of it at once." } }