Hi everyone！I got the Notebook threw exception problem while submitting, it occurs within 2 minutes after I submitted.
\nIt's really confusing, I've disabled the internet connection. And here is my notebook I would be so grateful if someone could tell me what causes the problem and how to solve it, Thanks!🙏🙏
Hi everyone！I got the Notebook threw exception problem while submitting, it occurs within 2 minutes after I submitted.
\nIt's really confusing, I've disabled the internet connection. And here is my notebook I would be so grateful if someone could tell me what causes the problem and how to solve it, Thanks!🙏🙏
Are you making an array of all the test images? I suspect you will run out of memory. Try running your notebook interactively with the train data substituted for the test data.
", "rawMarkdown": "Are you making an array of all the test images? I suspect you will run out of memory. Try running your notebook interactively with the train data substituted for the test data.", "votes": 3, "replies": [ { "id": 2056220, "postDate": "2022-12-05T21:40:50.940Z", "content": "That is a great suggestion!
", "rawMarkdown": "That is a great suggestion!" }, { "id": 2056286, "postDate": "2022-12-06T00:42:00.530Z", "content": "Thank you👍! Actually, I used to make a test dataset in tf.Dataset format, which assumes much less RAM, still the problem occurred, so it may not be the reason. However, it's quite likely that an array will run out of memory. It might cause some other error like time out…
", "rawMarkdown": "Thank you👍! Actually, I used to make a test dataset in tf.Dataset format, which assumes much less RAM, still the problem occurred, so it may not be the reason. However, it's quite likely that an array will run out of memory. It might cause some other error like time out...", "votes": 1 } ] }, { "id": 2056221, "postDate": "2022-12-05T21:41:55.407Z", "content": "Also, to read all images in test, you need this (make sure you run this before importing pydicom:
try:\n import pylibjpeg\nexcept:\n !rm -rf /root/.cache/torch/hub/checkpoints/\n !mkdir -p /root/.cache/torch/hub/checkpoints/\n !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}\nalong with the rsna-2022-whl attached to your notebook. I have a public notebook with prediction that I am hoping will pass 😅 (running now). Maybe taking a look at how I go about this there might be helpful 🙂
", "rawMarkdown": "Also, to read all images in test, you need this (make sure you run this before importing `pydicom`:\n\n```\ntry:\n import pylibjpeg\nexcept:\n !rm -rf /root/.cache/torch/hub/checkpoints/\n !mkdir -p /root/.cache/torch/hub/checkpoints/\n !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}\n``` \n\nalong with the rsna-2022-whl attached to your notebook. I have a public notebook with prediction that I am hoping will pass 😅 (running now). Maybe taking a look at how I go about this there might be helpful 🙂", "votes": 4, "replies": [ { "id": 2056287, "postDate": "2022-12-06T00:42:04.737Z", "content": "Thank you😊! I'm trying this method now. And I want to ask something about details. All I need to do is just to import the pydicom? Or I also need to create the whole process part in my submission notebook?
", "rawMarkdown": "Thank you😊! I'm trying this method now. And I want to ask something about details. All I need to do is just to import the pydicom? Or I also need to create the whole process part in my submission notebook?" }, { "id": 2056298, "postDate": "2022-12-06T01:14:32.483Z", "content": "You first need to run the code above and then import pydicom and you should be able to read all the images in test without error 🙂
I ‘ve added the code above to my notebook and import pydicom, but the exception remains😭
", "rawMarkdown": "I ‘ve added the code above to my notebook and import pydicom, but the exception remains😭" }, { "id": 2059404, "postDate": "2022-12-08T20:03:28.110Z", "content": "Were you able to resolve the issue? I'm in the similar situation.
", "rawMarkdown": "Were you able to resolve the issue? I'm in the similar situation. ", "replies": [ { "id": 2059538, "postDate": "2022-12-09T01:58:59.440Z", "content": "Yes, and you can look at the answer above to find out how I did:)
", "rawMarkdown": "Yes, and you can look at the answer above to find out how I did:)" } ] }, { "id": 2143759, "postDate": "2023-02-14T13:38:46.040Z", "content": "thank you very much for your work, it helped me! but I didn't import pylibjpeg, just installed the libraries
", "rawMarkdown": "thank you very much for your work, it helped me! but I didn't import pylibjpeg, just installed the libraries" } ] }, { "id": 2065263, "postDate": "2022-12-14T13:02:53.447Z", "content": "I am also facing the Submission Error : My Notebook
\nI am not getting what I am doing wrong here in the submission. I have trained models but now I'm not able to submit any !! Can anybody help me with this
@jay2333 I have the same problem; but you can try modifying the model
", "rawMarkdown": "@jay2333 I have the same problem; but you can try modifying the model", "votes": 1, "replies": [ { "id": 2058048, "postDate": "2022-12-07T15:27:26.770Z", "content": "Thanks for replying! I've successfully solved this problem:) Actually it's not the model that went wrong, it's the dataset.
\nI didn't notice at first how the notebook will be run after I submitted, so I didn't add the image process part (i.e. transform the image from dcm into png.) into my notebook. Instead, I loaded the test image from a dataset created by myself, in which all the images are in png format. As a result, when submitting, the test dataset will be replaced by another larger test dataset. You will need a process part to transform the test images into png format, then put them into the model. Without the image processing part the notebook will throw exception…
\nIf you have made such trivial mistake as I did, All you need to do is to add the processing part into your submission notebook:)
\nGood luck!
okay! thanks!!🙏
", "rawMarkdown": "okay! thanks!!🙏" } ] }, { "id": 2162880, "postDate": "2023-02-28T13:13:02.167Z", "content": "Hi,
\nI also have the same error. But the sample_submission gives me a score. Not sure what is the issue. Where should I check. The test image loader converts the .dcm to .jpg. So when scoring the private dataset there should not be a problem right?
\nThanks & Best Regards
\nAMJS
Hi, you may need to show more information about your problem, such as how many time does your notebook take to threw an exception, so others can analyze your problem. If you don't mind you can also make your notebook public.
\nBut in my experience, the most efficient way to make a successful submission is to copy&modify others submission notebook, you just need to replace the model with your own.
\nIf you insist making your own submission notebook, you can also compare yours with others'.
Hi, I think the issue is with the testing right? When Kaggle plugs in the private data instead of the test images. Otherwise, there is no need for Kaggle to run the whole notebook right? As you, for the train data I have the converted DICOM images as jpg in a separate dataset to train. The testing of images uses the saved .H5 file to make predictions. The issue with the other's kernels is most of the time it's quite complicated with huge code blocks. I will try to find a simpler kernal and edit it as you said. That seems to be the easiest thing to do.
\nAs for your questions. In one instance where, the code took are around half an hour to give an error. But the latest modification it takes less than 3 minutes to give the error. Maybe because, not much people are submitting now. Anyway, please find the below code for a notebook which I use to try to get the score because training every time takes more time.
\nhttps://www.kaggle.com/michaelschroter/h5-catsndogs
\nThanks & Best Regards
\nAMJS
Hi, If it takes a few minutes to give an error, I guess it was because the image processing part (i.e.Converted DICOM to JPG) that went wrong. My suggestion is that you need to check your notebook again, compare it with others' notebook. Good luck to you!
", "rawMarkdown": "Hi, If it takes a few minutes to give an error, I guess it was because the image processing part (i.e.Converted DICOM to JPG) that went wrong. My suggestion is that you need to check your notebook again, compare it with others' notebook. Good luck to you!", "votes": 1 } ] } ] }, { "id": 2099180, "postDate": "2023-01-14T08:40:53.550Z", "content": "Actually, I am also having trouble resolving a similar error(Notebook threw exception). Have you already solved the error? If possible, please let me know how you solved it.
", "rawMarkdown": "Actually, I am also having trouble resolving a similar error(Notebook threw exception). Have you already solved the error? If possible, please let me know how you solved it." }, { "id": 2056297, "postDate": "2022-12-06T01:12:19.527Z", "content": "It seems that the problem is in here.
\ntest_image_paths = []\nfor i in range(4):\n img_path = '/kaggle/input/rsna-test-image-in-png/output' + '/' + os.listdir('/kaggle/input/rsna-test-image-in-png/output')[i]\n test_image_paths.append(img_path)\nWe must process 8,000 patients in the hidden test set. Other inference submission notebooks will help.
", "rawMarkdown": "It seems that the problem is in here.\n```\ntest_image_paths = []\nfor i in range(4):\n img_path = '/kaggle/input/rsna-test-image-in-png/output' + '/' + os.listdir('/kaggle/input/rsna-test-image-in-png/output')[i]\n test_image_paths.append(img_path)\n```\n\nWe must process 8,000 patients in the hidden test set. Other inference submission notebooks will help.", "replies": [ { "id": 2056303, "postDate": "2022-12-06T01:32:52.440Z", "content": "I changed the range from 4 to the length of os.listdir('/kaggle/input/rsna-test-image-in-png/output'), and it seems the problem remains
This problem doesn't seem to happen with your commit, but it will most likely happen with your model (because as you said it throws an error when it runs for 2 minutes, so the data should go into the model at exactly this time)
", "rawMarkdown": "This problem doesn't seem to happen with your commit, but it will most likely happen with your model (because as you said it throws an error when it runs for 2 minutes, so the data should go into the model at exactly this time)" } ] }, { "id": 2058819, "postDate": "2022-12-08T08:30:34.620Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2059058, "postDate": "2022-12-08T12:48:35.947Z", "content": "When you mentioned the \"messages\", do you mean the \"Cleanup Called\" that keeps showing up? I have the same problem and I also solved it by installing the tensorflow 2.4.1, but I only do this in my training notebook. In the submission notebook I used the default version of tensorflow, the message still keeps showing up, but it's unlikely to make the submission throws an exception.
\nIt seems some other reasons cause the problem.
Have you added the image processing part into your notebook?
", "rawMarkdown": "Have you added the image processing part into your notebook? " }, { "id": 2062590, "postDate": "2022-12-12T08:36:05.507Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 2056035, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2022-12-05T17:04:49.083000", "content": "Are you making an array of all the test images? I suspect you will run out of memory. Try running your notebook interactively with the train data substituted for the test data.
", "votes": 3, "replies": [ { "id": 2056220, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-05T21:40:50.940000", "content": "That is a great suggestion!
", "votes": 0, "replies": [] }, { "id": 2056286, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-06T00:42:00.530000", "content": "Thank you👍! Actually, I used to make a test dataset in tf.Dataset format, which assumes much less RAM, still the problem occurred, so it may not be the reason. However, it's quite likely that an array will run out of memory. It might cause some other error like time out…
", "votes": 1, "replies": [] } ] }, { "id": 2056221, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-05T21:41:55.407000", "content": "Also, to read all images in test, you need this (make sure you run this before importing pydicom:
try:\n import pylibjpeg\nexcept:\n !rm -rf /root/.cache/torch/hub/checkpoints/\n !mkdir -p /root/.cache/torch/hub/checkpoints/\n !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}\nalong with the rsna-2022-whl attached to your notebook. I have a public notebook with prediction that I am hoping will pass 😅 (running now). Maybe taking a look at how I go about this there might be helpful 🙂
", "votes": 4, "replies": [ { "id": 2056287, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-06T00:42:04.737000", "content": "Thank you😊! I'm trying this method now. And I want to ask something about details. All I need to do is just to import the pydicom? Or I also need to create the whole process part in my submission notebook?
", "votes": 0, "replies": [] }, { "id": 2056298, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-06T01:14:32.483000", "content": "You first need to run the code above and then import pydicom and you should be able to read all the images in test without error 🙂
I ‘ve added the code above to my notebook and import pydicom, but the exception remains😭
", "votes": 0, "replies": [] }, { "id": 2059404, "author_name": "Dance_with_Data", "author_url": "", "post_date": "2022-12-08T20:03:28.110000", "content": "Were you able to resolve the issue? I'm in the similar situation.
", "votes": 0, "replies": [ { "id": 2059538, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-09T01:58:59.440000", "content": "Yes, and you can look at the answer above to find out how I did:)
", "votes": 0, "replies": [] } ] }, { "id": 2143759, "author_name": "ivan", "author_url": "", "post_date": "2023-02-14T13:38:46.040000", "content": "thank you very much for your work, it helped me! but I didn't import pylibjpeg, just installed the libraries
", "votes": 0, "replies": [] } ] }, { "id": 2065263, "author_name": "Bibhabasu Mohapatra", "author_url": "", "post_date": "2022-12-14T13:02:53.447000", "content": "I am also facing the Submission Error : My Notebook
\nI am not getting what I am doing wrong here in the submission. I have trained models but now I'm not able to submit any !! Can anybody help me with this
@jay2333 I have the same problem; but you can try modifying the model
", "votes": 1, "replies": [ { "id": 2058048, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-07T15:27:26.770000", "content": "Thanks for replying! I've successfully solved this problem:) Actually it's not the model that went wrong, it's the dataset.
\nI didn't notice at first how the notebook will be run after I submitted, so I didn't add the image process part (i.e. transform the image from dcm into png.) into my notebook. Instead, I loaded the test image from a dataset created by myself, in which all the images are in png format. As a result, when submitting, the test dataset will be replaced by another larger test dataset. You will need a process part to transform the test images into png format, then put them into the model. Without the image processing part the notebook will throw exception…
\nIf you have made such trivial mistake as I did, All you need to do is to add the processing part into your submission notebook:)
\nGood luck!
okay! thanks!!🙏
", "votes": 0, "replies": [] } ] }, { "id": 2162880, "author_name": "Schroter", "author_url": "", "post_date": "2023-02-28T13:13:02.167000", "content": "Hi,
\nI also have the same error. But the sample_submission gives me a score. Not sure what is the issue. Where should I check. The test image loader converts the .dcm to .jpg. So when scoring the private dataset there should not be a problem right?
\nThanks & Best Regards
\nAMJS
Hi, you may need to show more information about your problem, such as how many time does your notebook take to threw an exception, so others can analyze your problem. If you don't mind you can also make your notebook public.
\nBut in my experience, the most efficient way to make a successful submission is to copy&modify others submission notebook, you just need to replace the model with your own.
\nIf you insist making your own submission notebook, you can also compare yours with others'.
Hi, I think the issue is with the testing right? When Kaggle plugs in the private data instead of the test images. Otherwise, there is no need for Kaggle to run the whole notebook right? As you, for the train data I have the converted DICOM images as jpg in a separate dataset to train. The testing of images uses the saved .H5 file to make predictions. The issue with the other's kernels is most of the time it's quite complicated with huge code blocks. I will try to find a simpler kernal and edit it as you said. That seems to be the easiest thing to do.
\nAs for your questions. In one instance where, the code took are around half an hour to give an error. But the latest modification it takes less than 3 minutes to give the error. Maybe because, not much people are submitting now. Anyway, please find the below code for a notebook which I use to try to get the score because training every time takes more time.
\nhttps://www.kaggle.com/michaelschroter/h5-catsndogs
\nThanks & Best Regards
\nAMJS
Hi, If it takes a few minutes to give an error, I guess it was because the image processing part (i.e.Converted DICOM to JPG) that went wrong. My suggestion is that you need to check your notebook again, compare it with others' notebook. Good luck to you!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2099180, "author_name": "Yuki Sahashi", "author_url": "", "post_date": "2023-01-14T08:40:53.550000", "content": "Actually, I am also having trouble resolving a similar error(Notebook threw exception). Have you already solved the error? If possible, please let me know how you solved it.
", "votes": 0, "replies": [] }, { "id": 2056297, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2022-12-06T01:12:19.527000", "content": "It seems that the problem is in here.
\ntest_image_paths = []\nfor i in range(4):\n img_path = '/kaggle/input/rsna-test-image-in-png/output' + '/' + os.listdir('/kaggle/input/rsna-test-image-in-png/output')[i]\n test_image_paths.append(img_path)\nWe must process 8,000 patients in the hidden test set. Other inference submission notebooks will help.
", "votes": 0, "replies": [ { "id": 2056303, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-06T01:32:52.440000", "content": "I changed the range from 4 to the length of os.listdir('/kaggle/input/rsna-test-image-in-png/output'), and it seems the problem remains
This problem doesn't seem to happen with your commit, but it will most likely happen with your model (because as you said it throws an error when it runs for 2 minutes, so the data should go into the model at exactly this time)
", "votes": 0, "replies": [] } ] }, { "id": 2058819, "author_name": "", "author_url": "", "post_date": "2022-12-08T08:30:34.620000", "content": "", "votes": 0, "replies": [ { "id": 2059058, "author_name": "Lau2664", "author_url": "", "post_date": "2022-12-08T12:48:35.947000", "content": "When you mentioned the \"messages\", do you mean the \"Cleanup Called\" that keeps showing up? I have the same problem and I also solved it by installing the tensorflow 2.4.1, but I only do this in my training notebook. In the submission notebook I used the default version of tensorflow, the message still keeps showing up, but it's unlikely to make the submission throws an exception.
\nIt seems some other reasons cause the problem.
Have you added the image processing part into your notebook?
", "votes": 0, "replies": [] }, { "id": 2062590, "author_name": "", "author_url": "", "post_date": "2022-12-12T08:36:05.507000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2056020": "Hi everyone！I got the **Notebook threw exception** problem while submitting, it occurs within 2 minutes after I submitted.\nIt's really confusing, I've disabled the internet connection. And here is my [notebook](https://www.kaggle.com/code/jay2333/rsna-submission/notebook) I would be so grateful if someone could tell me what causes the problem and how to solve it, Thanks!🙏🙏", "2056035": "Are you making an array of all the test images? I suspect you will run out of memory. Try running your notebook interactively with the train data substituted for the test data.", "2056221": "Also, to read all images in test, you need this (make sure you run this before importing `pydicom`:\n\n```\ntry:\n import pylibjpeg\nexcept:\n !rm -rf /root/.cache/torch/hub/checkpoints/\n !mkdir -p /root/.cache/torch/hub/checkpoints/\n !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}\n``` \n\nalong with the rsna-2022-whl attached to your notebook. I have a public notebook with prediction that I am hoping will pass 😅 (running now). Maybe taking a look at how I go about this there might be helpful 🙂", "2065263": "I am also facing the Submission Error : [My Notebook](https://www.kaggle.com/bibhabasumohapatra/rsna-breast-cancer-part-1-inference) \nI am not getting what I am doing wrong here in the submission. I have trained models but now I'm not able to submit any !! Can anybody help me with this", "2057470": "@jay2333 I have the same problem; but you can try modifying the model", "2162880": "Hi,\nI also have the same error. But the sample_submission gives me a score. Not sure what is the issue. Where should I check. The test image loader converts the .dcm to .jpg. So when scoring the private dataset there should not be a problem right? \nThanks & Best Regards\nAMJS\n", "2099180": "Actually, I am also having trouble resolving a similar error(Notebook threw exception). Have you already solved the error? If possible, please let me know how you solved it.", "2056297": "It seems that the problem is in here.\n```\ntest_image_paths = []\nfor i in range(4):\n img_path = '/kaggle/input/rsna-test-image-in-png/output' + '/' + os.listdir('/kaggle/input/rsna-test-image-in-png/output')[i]\n test_image_paths.append(img_path)\n```\n\nWe must process 8,000 patients in the hidden test set. Other inference submission notebooks will help.", "2058819": "" } }