{
  "id": 150910,
  "title": "NO test data found",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/150910",
  "author_name": "Jaideep",
  "post_date": "2020-05-13T18:43:40.139000",
  "votes": 0,
  "comment_count": 8,
  "views": 0,
  "content": "<p>if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):</p>\n\n<p>There is no path existing upon commit .</p>\n\n<p>I exactly followed the inference based public kernel that have run successfully. </p>\n\n<p>What m i still missing</p>",
  "messages": [
    {
      "id": 846773,
      "postDate": "2020-05-14T02:04:12.150Z",
      "content": "<p>You need to commit as is. With\n<code>if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):</code>\n<code>do something</code>\n<code>else:</code>\n<code>pick sample</code></p>\n\n<p>Then in the notebook viewer output you need to select your submission.csv (from the sample) and select submit predictions. It will then run your notebook again in the test environment and pass the test above.</p>",
      "rawMarkdown": "You need to commit as is. With\n`if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):`\n`    do something`\n`else:`\n`    pick sample`\n\n Then in the notebook viewer output you need to select your submission.csv (from the sample) and select submit predictions. It will then run your notebook again in the test environment and pass the test above.",
      "replies": [
        {
          "id": 847016,
          "postDate": "2020-05-14T05:57:51.977Z",
          "content": "<p>Thanks i overcame this but now caught in another bad issue.It took 5-10 min to get the issue.</p>\n\n<p>Sub Csv not found\n```\nsub_smple_df = pd.read_csv(SAMPLE)\nif Data.exists()\n   with torch.no_grad():\n        for data loader loop:</p>\n\n<pre><code>          names.append(y)\n          preds.append(p)\n\n   names = np.concatenate(names)\n   final_preds = OptimizedRounder().predict(preds , coefficients)\n\n   preds=np.concatenate(final_preds)\n   sub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\n   sub_df.to_csv('submission.csv', index=False)\n</code></pre>\n\n<p>else:\n    sub_smple_df.to_csv(\"submission.csv\", index=False)\n    print('empty',sub_smple_df.head())\n```</p>",
          "rawMarkdown": "Thanks i overcame this but now caught in another bad issue.It took 5-10 min to get the issue.\n\nSub Csv not found\n```\nsub_smple_df = pd.read_csv(SAMPLE)\nif Data.exists()\n   with torch.no_grad():\n        for data loader loop:\n             \n\n              names.append(y)\n              preds.append(p)\n\n       names = np.concatenate(names)\n       final_preds = OptimizedRounder().predict(preds , coefficients)\n \n       preds=np.concatenate(final_preds)\n       sub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\n       sub_df.to_csv('submission.csv', index=False)\nelse:\n    sub_smple_df.to_csv(\"submission.csv\", index=False)\n    print('empty',sub_smple_df.head())\n```"
        },
        {
          "id": 848445,
          "postDate": "2020-05-15T01:21:24.757Z",
          "content": "<p>Have you verified the code actually produces a submission with the train set for example.</p>",
          "rawMarkdown": "Have you verified the code actually produces a submission with the train set for example."
        },
        {
          "id": 848588,
          "postDate": "2020-05-15T05:05:32.917Z",
          "content": "<p>Yes I did that got thus issue \n.not sure why ingot this issue .during of pre processing train set I dint got any exception \n```</p>\n\n<p>Return [default_collate(samples) for samples in transposed]  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py\", line 55, in default_collate\n    return torch.stack(batch, 0, out=out)\nRuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 14 and 6 in dimension 1 at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensor.cpp:612\n```</p>",
          "rawMarkdown": "Yes I did that got thus issue \n.not sure why ingot this issue .during of pre processing train set I dint got any exception \n```\n\n Return [default_collate(samples) for samples in transposed]  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py\", line 55, in default_collate\n    return torch.stack(batch, 0, out=out)\nRuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 14 and 6 in dimension 1 at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensor.cpp:612\n```"
        },
        {
          "id": 849312,
          "postDate": "2020-05-15T16:39:21.223Z",
          "content": "<p>This means your data processing is not always returning the same number in dimension 1. If this is the tile dimension it means your pipeline is not always outputing the same number of tiles.</p>",
          "rawMarkdown": "This means your data processing is not always returning the same number in dimension 1. If this is the tile dimension it means your pipeline is not always outputing the same number of tiles."
        },
        {
          "id": 849811,
          "postDate": "2020-05-16T04:59:11.383Z",
          "content": "<p><a href=\"/arroqc\">@arroqc</a>  Can you explain further. I'm dealing with same issue.</p>",
          "rawMarkdown": "@arroqc  Can you explain further. I'm dealing with same issue."
        },
        {
          "id": 850557,
          "postDate": "2020-05-16T18:48:02.827Z",
          "content": "<p>The error above is from default_collate. This is the collate function used by pytorch dataloader. It simply is what takes individual element of the dataset and makes a batch out of it. Usually it uses the stack function on the 0 axis. But the tensor must therefore be all of the same size. If you are using tiles dataset then your shape is probably B, T, C, H, W with T the number of tiles. So dimension 1 is the number of tiles. If there is a mismatch on dimension 1 it means not all element of the dataset have the same number of tiles.</p>",
          "rawMarkdown": "The error above is from default_collate. This is the collate function used by pytorch dataloader. It simply is what takes individual element of the dataset and makes a batch out of it. Usually it uses the stack function on the 0 axis. But the tensor must therefore be all of the same size. If you are using tiles dataset then your shape is probably B, T, C, H, W with T the number of tiles. So dimension 1 is the number of tiles. If there is a mismatch on dimension 1 it means not all element of the dataset have the same number of tiles."
        },
        {
          "id": 850757,
          "postDate": "2020-05-17T01:42:45.443Z",
          "content": "<p>Yes so I added padding method but it required  readding of tiles to make up for diff which could result into false prediction of higher grade.\nAny other better way to overcome tile diff issue</p>",
          "rawMarkdown": "Yes so I added padding method but it required  readding of tiles to make up for diff which could result into false prediction of higher grade.\nAny other better way to overcome tile diff issue"
        }
      ]
    },
    {
      "id": 846419,
      "postDate": "2020-05-13T18:43:40.140Z",
      "content": "<p>if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):</p>\n\n<p>There is no path existing upon commit .</p>\n\n<p>I exactly followed the inference based public kernel that have run successfully. </p>\n\n<p>What m i still missing</p>",
      "rawMarkdown": "if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n\nThere is no path existing upon commit .\n\nI exactly followed the inference based public kernel that have run successfully. \n\nWhat m i still missing"
    }
  ],
  "comments": [
    {
      "id": 846773,
      "author_name": "Arnaud Roussel",
      "author_url": "",
      "post_date": "2020-05-14T02:04:12.150000",
      "content": "<p>You need to commit as is. With\n<code>if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):</code>\n<code>do something</code>\n<code>else:</code>\n<code>pick sample</code></p>\n\n<p>Then in the notebook viewer output you need to select your submission.csv (from the sample) and select submit predictions. It will then run your notebook again in the test environment and pass the test above.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 847016,
          "author_name": "Jaideep",
          "author_url": "",
          "post_date": "2020-05-14T05:57:51.977000",
          "content": "<p>Thanks i overcame this but now caught in another bad issue.It took 5-10 min to get the issue.</p>\n\n<p>Sub Csv not found\n```\nsub_smple_df = pd.read_csv(SAMPLE)\nif Data.exists()\n   with torch.no_grad():\n        for data loader loop:</p>\n\n<pre><code>          names.append(y)\n          preds.append(p)\n\n   names = np.concatenate(names)\n   final_preds = OptimizedRounder().predict(preds , coefficients)\n\n   preds=np.concatenate(final_preds)\n   sub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\n   sub_df.to_csv('submission.csv', index=False)\n</code></pre>\n\n<p>else:\n    sub_smple_df.to_csv(\"submission.csv\", index=False)\n    print('empty',sub_smple_df.head())\n```</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 848445,
          "author_name": "Arnaud Roussel",
          "author_url": "",
          "post_date": "2020-05-15T01:21:24.757000",
          "content": "<p>Have you verified the code actually produces a submission with the train set for example.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 848588,
          "author_name": "Jaideep",
          "author_url": "",
          "post_date": "2020-05-15T05:05:32.917000",
          "content": "<p>Yes I did that got thus issue \n.not sure why ingot this issue .during of pre processing train set I dint got any exception \n```</p>\n\n<p>Return [default_collate(samples) for samples in transposed]  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py\", line 55, in default_collate\n    return torch.stack(batch, 0, out=out)\nRuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 14 and 6 in dimension 1 at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensor.cpp:612\n```</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 849312,
          "author_name": "Arnaud Roussel",
          "author_url": "",
          "post_date": "2020-05-15T16:39:21.223000",
          "content": "<p>This means your data processing is not always returning the same number in dimension 1. If this is the tile dimension it means your pipeline is not always outputing the same number of tiles.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 849811,
          "author_name": "William Green",
          "author_url": "",
          "post_date": "2020-05-16T04:59:11.383000",
          "content": "<p><a href=\"/arroqc\">@arroqc</a>  Can you explain further. I'm dealing with same issue.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 850557,
          "author_name": "Arnaud Roussel",
          "author_url": "",
          "post_date": "2020-05-16T18:48:02.827000",
          "content": "<p>The error above is from default_collate. This is the collate function used by pytorch dataloader. It simply is what takes individual element of the dataset and makes a batch out of it. Usually it uses the stack function on the 0 axis. But the tensor must therefore be all of the same size. If you are using tiles dataset then your shape is probably B, T, C, H, W with T the number of tiles. So dimension 1 is the number of tiles. If there is a mismatch on dimension 1 it means not all element of the dataset have the same number of tiles.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 850757,
          "author_name": "Jaideep",
          "author_url": "",
          "post_date": "2020-05-17T01:42:45.443000",
          "content": "<p>Yes so I added padding method but it required  readding of tiles to make up for diff which could result into false prediction of higher grade.\nAny other better way to overcome tile diff issue</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "846773": "You need to commit as is. With\n`if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):`\n`    do something`\n`else:`\n`    pick sample`\n\n Then in the notebook viewer output you need to select your submission.csv (from the sample) and select submit predictions. It will then run your notebook again in the test environment and pass the test above.",
    "846419": "if os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n\nThere is no path existing upon commit .\n\nI exactly followed the inference based public kernel that have run successfully. \n\nWhat m i still missing"
  }
}