![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2675447%2F1af82c9eb4c1fbc52593484743f77b68%2Fcsv_not_found.png?generation=1723819856728481&alt=media\")
If you were following my data preparation notebook (https://www.kaggle.com/code/ilu000/ariel24-data-prep) or used a similar approach that naively, writes a lot of files to the /kaggle/working/ folder, you probably also had this error:
Unfortunately, the error is rather cryptic and not even true. I am only speculating that kaggle does something like a glob.glob(\"/kaggle/working/\")[:500] and only searches the first 500 or so files for a file named submission.csv. The output count for notebooks is a good indication for this behavior (is this a bug @mylesoneill ?).
Shows 500 files, but there are more than 1000 files created.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2675447%2F74463a0d4f99ef14996547b33cdaa055%2Fout.png?generation=1723827627960163&alt=media\")
To solve this, just delete all other temporary files that you needed to prepare the solution at the end of your notebook. Only keep the required submission.csv.
For example:
\n! -rf *.npy\n! -rf *.npz\nIf you were following my data preparation notebook (https://www.kaggle.com/code/ilu000/ariel24-data-prep) or used a similar approach that naively, writes a lot of files to the /kaggle/working/ folder, you probably also had this error:
Unfortunately, the error is rather cryptic and not even true. I am only speculating that kaggle does something like a glob.glob(\"/kaggle/working/\")[:500] and only searches the first 500 or so files for a file named submission.csv. The output count for notebooks is a good indication for this behavior (is this a bug @mylesoneill ?).
Shows 500 files, but there are more than 1000 files created.
\n
To solve this, just delete all other temporary files that you needed to prepare the solution at the end of your notebook. Only keep the required submission.csv.
For example:
\n! -rf *.npy\n! -rf *.npz\n@ilu000 => with this process, we do not worry about /kaggle/working 20GB limit and temp files.
\n! mkdir /home/jupyter/neuips_2024\n%cd /home/jupyter/neuips_2024\n\n.. code \n\n%cd /kaggle/working\n.. save final submissions.csv\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Fe95b95c0ea3d32361530aa310055eef9%2FScreenshot%202024-08-16%20at%2023.09.11.png?generation=1723829963555758&alt=media\")
That is sneaky, I didn't know that trick to go over the 20 GB limit. Thank you for sharing!
\nSo, 20 GB is still the limit for any data that can be shared by a notebook, but there is way more to use in intermediate steps.
\n\nSo, 20 GB is still the limit for any data that can be shared by a notebook, but there is way more to use in intermediate steps.
\n
Oh that is such a cool insight.
\nKnowing that we can use a lot more data in the middle and submit only about 20 GB in the end.
Thanks!
", "rawMarkdown": "Oh that is such a cool insight.\nKnowing that we can use a lot more data in the middle and submit only about 20 GB in the end.\n\nThanks!" } ], "comments": [ { "id": 2961550, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-16T17:36:51.723000", "content": "@ilu000 => with this process, we do not worry about /kaggle/working 20GB limit and temp files.
\n! mkdir /home/jupyter/neuips_2024\n%cd /home/jupyter/neuips_2024\n\n.. code \n\n%cd /kaggle/working\n.. save final submissions.csv\n
That is sneaky, I didn't know that trick to go over the 20 GB limit. Thank you for sharing!
\nSo, 20 GB is still the limit for any data that can be shared by a notebook, but there is way more to use in intermediate steps.
\n\nSo, 20 GB is still the limit for any data that can be shared by a notebook, but there is way more to use in intermediate steps.
\n
Oh that is such a cool insight.
\nKnowing that we can use a lot more data in the middle and submit only about 20 GB in the end.
Thanks!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2961503": "If you were following my data preparation notebook (https://www.kaggle.com/code/ilu000/ariel24-data-prep) or used a similar approach that naively, writes a lot of files to the `/kaggle/working/` folder, you probably also had this error:\n\n\n\nUnfortunately, the error is rather cryptic and not even true. I am only speculating that kaggle does something like a `glob.glob(\"/kaggle/working/\")[:500]` and only searches the first 500 or so files for a file named `submission.csv`. The output count for notebooks is a good indication for this behavior (is this a bug @mylesoneill ?). \n\nShows 500 files, but there are more than 1000 files created.\n\n\nTo solve this, just delete all other temporary files that you needed to prepare the solution at the end of your notebook. Only keep the required `submission.csv`.\n\nFor example:\n```\n!rm -rf *.npy\n!rm -rf *.npz\n```\n", "2961550": "@ilu000 => with this process, we do not worry about /kaggle/working 20GB limit and temp files.\n\n```python\n! mkdir /home/jupyter/neuips_2024\n%cd /home/jupyter/neuips_2024\n \n.. all code \n\n%cd /kaggle/working\n.. save final submissions.csv\n```\n\n", "2961811": "Oh that is such a cool insight.\nKnowing that we can use a lot more data in the middle and submit only about 20 GB in the end.\n\nThanks!" } }