{
  "id": 189685,
  "title": "'Submission Not Found Error' solved!! Anyone able to do inference without 'try' and 'except'? ",
  "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/189685",
  "author_name": "Jose",
  "post_date": "2020-10-08T09:47:44.352000",
  "votes": 1,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Hey all,<br>\nI created a sample model and when I tried to do inference on the private test data, it was giving me a 'submission not found' error. It had something to do with .dcm files in private test data. My pipeline works for public test data but when I rerun it on public data I am getting a 'submission not found' error within 10 minutes which means there is an error occurring somewhere. <br>\nSo I kept the reading of DCM files with a try and except block as shown below and now it is working fine and I got my current public score by trying this. </p>\n<pre><code>        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n            .\n            .\n            .  \n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n</code></pre>\n<p>But the problem is we don't know how many private images we are skipping. I may be just 1 private image or it can be even all private images. So can someone confirm this is the only way to read private test images? And to organizers, can you please look into this issue and help.</p>\n<p>Thanks </p>",
  "messages": [
    {
      "id": 1052683,
      "postDate": "2020-10-18T06:11:05.960Z",
      "content": "<p>Hi Jose, did you manage to solve this problem yet? Turns out I'm facing the same issue. What's more disturbing is that I added this code block to capture how many exceptions I'm incurring (it will only generate a submission file if the number of exceptions is less than 50). It indeed cause a \"submission not found\" error which means I'm unable to read many more studies in the private test set than I thought.</p>\n<p><code>\nexcept:\nnumExceptions += 1</code></p>\n<p><code>\nif numExceptions &lt; 50:\nsubmissionDF.to_csv('submission.csv', index=False)\n</code></p>",
      "rawMarkdown": "Hi Jose, did you manage to solve this problem yet? Turns out I'm facing the same issue. What's more disturbing is that I added this code block to capture how many exceptions I'm incurring (it will only generate a submission file if the number of exceptions is less than 50). It indeed cause a \"submission not found\" error which means I'm unable to read many more studies in the private test set than I thought.\n\n`\nexcept:\nnumExceptions += 1`\n\n`\nif numExceptions < 50:\nsubmissionDF.to_csv('submission.csv', index=False)\n`",
      "votes": 1,
      "replies": [
        {
          "id": 1052810,
          "postDate": "2020-10-18T09:48:30.190Z",
          "content": "<p>Hey,<br>\nThe error for me was actually due to <code>image = np.asarray(dicom.pixel_array)</code>.<br>\nI used vtk to get the array and it was working fine. You can see it in the notebook <a href=\"https://www.kaggle.com/eladwar/20-seconds-or-less\" target=\"_blank\">here</a><br>\nHope it helps :)</p>",
          "rawMarkdown": "Hey,\nThe error for me was actually due to `image = np.asarray(dicom.pixel_array)`.\nI used vtk to get the array and it was working fine. You can see it in the notebook [here](https://www.kaggle.com/eladwar/20-seconds-or-less)\nHope it helps :)"
        },
        {
          "id": 1055127,
          "postDate": "2020-10-20T13:57:52.957Z",
          "content": "<p><a href=\"https://www.kaggle.com/josealways123\" target=\"_blank\">@josealways123</a> Thanks so much! VTK works.</p>",
          "rawMarkdown": "@josealways123 Thanks so much! VTK works."
        }
      ]
    },
    {
      "id": 1042637,
      "postDate": "2020-10-08T11:25:27.200Z",
      "content": "<p><a href=\"https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet\" target=\"_blank\">https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet</a></p>\n<p>Installing GDCM pydicom is able to read all the private test set without try catch</p>",
      "rawMarkdown": "https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet\n\nInstalling GDCM pydicom is able to read all the private test set without try catch",
      "votes": 1,
      "replies": [
        {
          "id": 1042681,
          "postDate": "2020-10-08T11:51:01.477Z",
          "content": "<p>Strange, I tried with GDCM and got the same error. Are you using all these lines in your inference kernel?</p>\n<pre><code>            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n</code></pre>",
          "rawMarkdown": "Strange, I tried with GDCM and got the same error. Are you using all these lines in your inference kernel?\n```\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n```"
        },
        {
          "id": 1042714,
          "postDate": "2020-10-08T12:23:16.473Z",
          "content": "<p>Yes, the code is that<br>\nAre you able to read all the train set? Because if you are able to read the train set the problem is not gdcm since there are also files in the train set not readable without gdcm</p>",
          "rawMarkdown": "Yes, the code is that\nAre you able to read all the train set? Because if you are able to read the train set the problem is not gdcm since there are also files in the train set not readable without gdcm",
          "votes": 1
        },
        {
          "id": 1044231,
          "postDate": "2020-10-09T15:57:45.243Z",
          "content": "<p>I haven't tried with train dataset. Let me check.</p>",
          "rawMarkdown": "I haven't tried with train dataset. Let me check."
        }
      ]
    },
    {
      "id": 1042538,
      "postDate": "2020-10-08T09:47:44.353Z",
      "content": "<p>Hey all,<br>\nI created a sample model and when I tried to do inference on the private test data, it was giving me a 'submission not found' error. It had something to do with .dcm files in private test data. My pipeline works for public test data but when I rerun it on public data I am getting a 'submission not found' error within 10 minutes which means there is an error occurring somewhere. <br>\nSo I kept the reading of DCM files with a try and except block as shown below and now it is working fine and I got my current public score by trying this. </p>\n<pre><code>        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n            .\n            .\n            .  \n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n</code></pre>\n<p>But the problem is we don't know how many private images we are skipping. I may be just 1 private image or it can be even all private images. So can someone confirm this is the only way to read private test images? And to organizers, can you please look into this issue and help.</p>\n<p>Thanks </p>",
      "rawMarkdown": "Hey all,\nI created a sample model and when I tried to do inference on the private test data, it was giving me a 'submission not found' error. It had something to do with .dcm files in private test data. My pipeline works for public test data but when I rerun it on public data I am getting a 'submission not found' error within 10 minutes which means there is an error occurring somewhere. \nSo I kept the reading of DCM files with a try and except block as shown below and now it is working fine and I got my current public score by trying this. \n```\n        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n            .\n            .\n            .  \n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n```\nBut the problem is we don't know how many private images we are skipping. I may be just 1 private image or it can be even all private images. So can someone confirm this is the only way to read private test images? And to organizers, can you please look into this issue and help.\n\nThanks ",
      "votes": 1
    },
    {
      "id": 1044179,
      "postDate": "2020-10-09T15:20:07.963Z",
      "content": "<p>I thought it is because some test images are not 512 x 512, but I see that you are resizing everything 256 x 256 so this can't be the cause. I guess everyone's problem is a bit different.</p>\n<p>My original reply:</p>\n<blockquote>\n  <p>Are you assuming 512 by 512 resolution for the images? I suspect some test images are not 512 x 512.</p>\n  <p>I added</p>\n  <p><code>if toPred.size()[2] != 512 or toPred.size()[3] != 512:\n                torch.nn.interpolate(toPred,(512,512))\n</code></p>\n  <p>and the errors disappeared for me.</p>\n</blockquote>",
      "rawMarkdown": "I thought it is because some test images are not 512 x 512, but I see that you are resizing everything 256 x 256 so this can't be the cause. I guess everyone's problem is a bit different.\n\nMy original reply:\n> \nAre you assuming 512 by 512 resolution for the images? I suspect some test images are not 512 x 512.\n>\nI added\n>\n`if toPred.size()[2] != 512 or toPred.size()[3] != 512:\n                torch.nn.interpolate(toPred,(512,512))\n`\n>\nand the errors disappeared for me.",
      "votes": 2,
      "replies": [
        {
          "id": 1044234,
          "postDate": "2020-10-09T16:01:02.837Z",
          "content": "<p>Thanks for helping. This is my full try and except block</p>\n<pre><code>        MAX_LENGTH = 256.\n        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            image = np.asarray(dicom.pixel_array)\n            image = image * M\n            image = image + B\n            image_lung = np.expand_dims(window(image, WL=-600, WW=1500), axis=2)\n            image_mediastinal = np.expand_dims(window(image, WL=40, WW=400), axis=2)\n            image_pe_specific = np.expand_dims(window(image, WL=100, WW=700), axis=2)\n            image = np.concatenate([image_mediastinal, image_pe_specific, image_lung], axis=2)\n            rat = MAX_LENGTH / np.max(image.shape)\n            image = zoom(image, [rat,rat,1.], prefilter=False, order=1)\n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n</code></pre>\n<p>Zoom should work the same as interpolate right?</p>",
          "rawMarkdown": "Thanks for helping. This is my full try and except block\n```\n        MAX_LENGTH = 256.\n        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            image = np.asarray(dicom.pixel_array)\n            image = image * M\n            image = image + B\n            image_lung = np.expand_dims(window(image, WL=-600, WW=1500), axis=2)\n            image_mediastinal = np.expand_dims(window(image, WL=40, WW=400), axis=2)\n            image_pe_specific = np.expand_dims(window(image, WL=100, WW=700), axis=2)\n            image = np.concatenate([image_mediastinal, image_pe_specific, image_lung], axis=2)\n            rat = MAX_LENGTH / np.max(image.shape)\n            image = zoom(image, [rat,rat,1.], prefilter=False, order=1)\n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n```\nZoom should work the same as interpolate right?"
        },
        {
          "id": 1045226,
          "postDate": "2020-10-10T12:53:23.040Z",
          "content": "<p>Yes, I think that has the same effect as interpolate; both are functions to resize image.</p>",
          "rawMarkdown": "Yes, I think that has the same effect as interpolate; both are functions to resize image."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1052683,
      "author_name": "Yee Ng",
      "author_url": "",
      "post_date": "2020-10-18T06:11:05.960000",
      "content": "<p>Hi Jose, did you manage to solve this problem yet? Turns out I'm facing the same issue. What's more disturbing is that I added this code block to capture how many exceptions I'm incurring (it will only generate a submission file if the number of exceptions is less than 50). It indeed cause a \"submission not found\" error which means I'm unable to read many more studies in the private test set than I thought.</p>\n<p><code>\nexcept:\nnumExceptions += 1</code></p>\n<p><code>\nif numExceptions &lt; 50:\nsubmissionDF.to_csv('submission.csv', index=False)\n</code></p>",
      "votes": 1,
      "replies": [
        {
          "id": 1052810,
          "author_name": "Jose",
          "author_url": "",
          "post_date": "2020-10-18T09:48:30.190000",
          "content": "<p>Hey,<br>\nThe error for me was actually due to <code>image = np.asarray(dicom.pixel_array)</code>.<br>\nI used vtk to get the array and it was working fine. You can see it in the notebook <a href=\"https://www.kaggle.com/eladwar/20-seconds-or-less\" target=\"_blank\">here</a><br>\nHope it helps :)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1055127,
          "author_name": "Yee Ng",
          "author_url": "",
          "post_date": "2020-10-20T13:57:52.957000",
          "content": "<p><a href=\"https://www.kaggle.com/josealways123\" target=\"_blank\">@josealways123</a> Thanks so much! VTK works.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1042637,
      "author_name": "Marco Stefani",
      "author_url": "",
      "post_date": "2020-10-08T11:25:27.200000",
      "content": "<p><a href=\"https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet\" target=\"_blank\">https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet</a></p>\n<p>Installing GDCM pydicom is able to read all the private test set without try catch</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1042681,
          "author_name": "Jose",
          "author_url": "",
          "post_date": "2020-10-08T11:51:01.477000",
          "content": "<p>Strange, I tried with GDCM and got the same error. Are you using all these lines in your inference kernel?</p>\n<pre><code>            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1042714,
          "author_name": "Marco Stefani",
          "author_url": "",
          "post_date": "2020-10-08T12:23:16.473000",
          "content": "<p>Yes, the code is that<br>\nAre you able to read all the train set? Because if you are able to read the train set the problem is not gdcm since there are also files in the train set not readable without gdcm</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1044231,
          "author_name": "Jose",
          "author_url": "",
          "post_date": "2020-10-09T15:57:45.243000",
          "content": "<p>I haven't tried with train dataset. Let me check.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1044179,
      "author_name": "Yee Ng",
      "author_url": "",
      "post_date": "2020-10-09T15:20:07.963000",
      "content": "<p>I thought it is because some test images are not 512 x 512, but I see that you are resizing everything 256 x 256 so this can't be the cause. I guess everyone's problem is a bit different.</p>\n<p>My original reply:</p>\n<blockquote>\n  <p>Are you assuming 512 by 512 resolution for the images? I suspect some test images are not 512 x 512.</p>\n  <p>I added</p>\n  <p><code>if toPred.size()[2] != 512 or toPred.size()[3] != 512:\n                torch.nn.interpolate(toPred,(512,512))\n</code></p>\n  <p>and the errors disappeared for me.</p>\n</blockquote>",
      "votes": 2,
      "replies": [
        {
          "id": 1044234,
          "author_name": "Jose",
          "author_url": "",
          "post_date": "2020-10-09T16:01:02.837000",
          "content": "<p>Thanks for helping. This is my full try and except block</p>\n<pre><code>        MAX_LENGTH = 256.\n        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            image = np.asarray(dicom.pixel_array)\n            image = image * M\n            image = image + B\n            image_lung = np.expand_dims(window(image, WL=-600, WW=1500), axis=2)\n            image_mediastinal = np.expand_dims(window(image, WL=40, WW=400), axis=2)\n            image_pe_specific = np.expand_dims(window(image, WL=100, WW=700), axis=2)\n            image = np.concatenate([image_mediastinal, image_pe_specific, image_lung], axis=2)\n            rat = MAX_LENGTH / np.max(image.shape)\n            image = zoom(image, [rat,rat,1.], prefilter=False, order=1)\n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n</code></pre>\n<p>Zoom should work the same as interpolate right?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1045226,
          "author_name": "Yee Ng",
          "author_url": "",
          "post_date": "2020-10-10T12:53:23.040000",
          "content": "<p>Yes, I think that has the same effect as interpolate; both are functions to resize image.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1052683": "Hi Jose, did you manage to solve this problem yet? Turns out I'm facing the same issue. What's more disturbing is that I added this code block to capture how many exceptions I'm incurring (it will only generate a submission file if the number of exceptions is less than 50). It indeed cause a \"submission not found\" error which means I'm unable to read many more studies in the private test set than I thought.\n\n`\nexcept:\nnumExceptions += 1`\n\n`\nif numExceptions < 50:\nsubmissionDF.to_csv('submission.csv', index=False)\n`",
    "1042637": "https://www.kaggle.com/richardepstein/load-gdcm-in-notebook-without-internet\n\nInstalling GDCM pydicom is able to read all the private test set without try catch",
    "1042538": "Hey all,\nI created a sample model and when I tried to do inference on the private test data, it was giving me a 'submission not found' error. It had something to do with .dcm files in private test data. My pipeline works for public test data but when I rerun it on public data I am getting a 'submission not found' error within 10 minutes which means there is an error occurring somewhere. \nSo I kept the reading of DCM files with a try and except block as shown below and now it is working fine and I got my current public score by trying this. \n```\n        try:\n            dicom = pydicom.read_file(path)\n            M = float(dicom.RescaleSlope)\n            B = float(dicom.RescaleIntercept)\n            dicom = np.asarray(dicom.pixel_array)\n            .\n            .\n            .  \n        except:\n            print('Exception')\n            image = np.ones(shape=(256,256,3), dtype=np.float32)\n```\nBut the problem is we don't know how many private images we are skipping. I may be just 1 private image or it can be even all private images. So can someone confirm this is the only way to read private test images? And to organizers, can you please look into this issue and help.\n\nThanks ",
    "1044179": "I thought it is because some test images are not 512 x 512, but I see that you are resizing everything 256 x 256 so this can't be the cause. I guess everyone's problem is a bit different.\n\nMy original reply:\n> \nAre you assuming 512 by 512 resolution for the images? I suspect some test images are not 512 x 512.\n>\nI added\n>\n`if toPred.size()[2] != 512 or toPred.size()[3] != 512:\n                torch.nn.interpolate(toPred,(512,512))\n`\n>\nand the errors disappeared for me."
  }
}