{
  "id": 151584,
  "title": "Submission CSV Not Found is so freaking annoying! ",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/151584",
  "author_name": "William Green",
  "post_date": "2020-05-16T05:05:28.864000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Maybe a different set of eyes can see what I'm not seeing. I updated @lafoss inference kernel to:</p>\n\n<p>```</p>\n\n<h1># Prediction</h1>\n\n<p>print(DATA)</p>\n\n<h1>sub = inference(da = feature_engineering(data = test , dir_name = \"test_images\") , dir_path = \"test_images\")</h1>\n\n<p>sub_df = pd.read_csv(SAMPLE)\nif os.path.exists(DATA):\n    print(\"RUN INFERENCE ON TEST DATA\")\n    ds = PandaDataset(DATA,TEST)\n    dl = DataLoader(ds, batch_size=bs, num_workers=nworkers, shuffle=False)\n    names,preds = [],[]</p>\n\n<pre><code>with torch.no_grad():\n    for x,y in tqdm(dl):\n        x = x.cuda()\n        #dihedral TTA\n        x = torch.stack([x,x.flip(-1),x.flip(-2),x.flip(-1,-2),\n          x.transpose(-1,-2),x.transpose(-1,-2).flip(-1),\n          x.transpose(-1,-2).flip(-2),x.transpose(-1,-2).flip(-1,-2)],1)\n        x = x.view(-1,N,3,sz,sz)\n        p = [model(x) for model in models]\n        p = torch.stack(p,1)\n        p = p.view(bs,8*len(models),-1).mean(1).argmax(-1).cpu()\n        names.append(y)\n        preds.append(p)\n\nnames = np.concatenate(names)\npreds = torch.cat(preds).numpy()\nsub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\n</code></pre>\n\n<p>else:\n    sub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    sub.to_csv('submission.csv', index=False)\n    #ub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    #sub_df.to_csv('submission.csv', index=False)\n    sub_df.head()\n    print('SUBMISSION DATAFRAME LENGTH IS: ',len(sub_df.index))\n<code>``\nNow, I get</code>submission csv not found`.  I greatly appreciate anyone who takes the time to tell me what the hech I am doing wrong. 😊 </p>",
  "messages": [
    {
      "id": 850571,
      "postDate": "2020-05-16T19:19:32.513Z",
      "content": "<p>That code doesn't have a .to_csv for the if condition</p>",
      "rawMarkdown": "That code doesn't have a .to_csv for the if condition",
      "votes": 3
    },
    {
      "id": 851466,
      "postDate": "2020-05-17T16:34:42.307Z",
      "content": "<p>I add a check for the existence of the test folder at the lowest level and I load a training image if it does not exist. This allows test all my code prior to submission. </p>\n\n<pre><code>    if os.path.exists(os.path.join(DATA,name+'.tiff')):\n        img = skimage.io.MultiImage(os.path.join(DATA,name+'.tiff'))[1]\n    else:\n        img = skimage.io.MultiImage(os.path.join('../input/prostate-cancer-grade-assessment/train_images','0076bcb66e46fb485f5ba432b9a1fe8a.tiff'))[1]\n</code></pre>",
      "rawMarkdown": "I add a check for the existence of the test folder at the lowest level and I load a training image if it does not exist. This allows test all my code prior to submission. \n\n        if os.path.exists(os.path.join(DATA,name+'.tiff')):\n            img = skimage.io.MultiImage(os.path.join(DATA,name+'.tiff'))[1]\n        else:\n            img = skimage.io.MultiImage(os.path.join('../input/prostate-cancer-grade-assessment/train_images','0076bcb66e46fb485f5ba432b9a1fe8a.tiff'))[1]\n\n",
      "votes": 1
    },
    {
      "id": 849814,
      "postDate": "2020-05-16T05:05:28.863Z",
      "content": "<p>Maybe a different set of eyes can see what I'm not seeing. I updated @lafoss inference kernel to:</p>\n\n<p>```</p>\n\n<h1># Prediction</h1>\n\n<p>print(DATA)</p>\n\n<h1>sub = inference(da = feature_engineering(data = test , dir_name = \"test_images\") , dir_path = \"test_images\")</h1>\n\n<p>sub_df = pd.read_csv(SAMPLE)\nif os.path.exists(DATA):\n    print(\"RUN INFERENCE ON TEST DATA\")\n    ds = PandaDataset(DATA,TEST)\n    dl = DataLoader(ds, batch_size=bs, num_workers=nworkers, shuffle=False)\n    names,preds = [],[]</p>\n\n<pre><code>with torch.no_grad():\n    for x,y in tqdm(dl):\n        x = x.cuda()\n        #dihedral TTA\n        x = torch.stack([x,x.flip(-1),x.flip(-2),x.flip(-1,-2),\n          x.transpose(-1,-2),x.transpose(-1,-2).flip(-1),\n          x.transpose(-1,-2).flip(-2),x.transpose(-1,-2).flip(-1,-2)],1)\n        x = x.view(-1,N,3,sz,sz)\n        p = [model(x) for model in models]\n        p = torch.stack(p,1)\n        p = p.view(bs,8*len(models),-1).mean(1).argmax(-1).cpu()\n        names.append(y)\n        preds.append(p)\n\nnames = np.concatenate(names)\npreds = torch.cat(preds).numpy()\nsub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\n</code></pre>\n\n<p>else:\n    sub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    sub.to_csv('submission.csv', index=False)\n    #ub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    #sub_df.to_csv('submission.csv', index=False)\n    sub_df.head()\n    print('SUBMISSION DATAFRAME LENGTH IS: ',len(sub_df.index))\n<code>``\nNow, I get</code>submission csv not found`.  I greatly appreciate anyone who takes the time to tell me what the hech I am doing wrong. 😊 </p>",
      "rawMarkdown": "Maybe a different set of eyes can see what I'm not seeing. I updated @lafoss inference kernel to:\n\n```\n# # Prediction\n\n\nprint(DATA)\n\n#sub = inference(da = feature_engineering(data = test , dir_name = \"test_images\") , dir_path = \"test_images\")\n\nsub_df = pd.read_csv(SAMPLE)\nif os.path.exists(DATA):\n    print(\"RUN INFERENCE ON TEST DATA\")\n    ds = PandaDataset(DATA,TEST)\n    dl = DataLoader(ds, batch_size=bs, num_workers=nworkers, shuffle=False)\n    names,preds = [],[]\n\n    with torch.no_grad():\n        for x,y in tqdm(dl):\n            x = x.cuda()\n            #dihedral TTA\n            x = torch.stack([x,x.flip(-1),x.flip(-2),x.flip(-1,-2),\n              x.transpose(-1,-2),x.transpose(-1,-2).flip(-1),\n              x.transpose(-1,-2).flip(-2),x.transpose(-1,-2).flip(-1,-2)],1)\n            x = x.view(-1,N,3,sz,sz)\n            p = [model(x) for model in models]\n            p = torch.stack(p,1)\n            p = p.view(bs,8*len(models),-1).mean(1).argmax(-1).cpu()\n            names.append(y)\n            preds.append(p)\n    \n    names = np.concatenate(names)\n    preds = torch.cat(preds).numpy()\n    sub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\nelse:\n    sub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    sub.to_csv('submission.csv', index=False)\n    #ub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    #sub_df.to_csv('submission.csv', index=False)\n    sub_df.head()\n    print('SUBMISSION DATAFRAME LENGTH IS: ',len(sub_df.index))\n```\nNow, I get `submission csv not found`.  I greatly appreciate anyone who takes the time to tell me what the hech I am doing wrong. 😊 ",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 850571,
      "author_name": "Arnaud Roussel",
      "author_url": "",
      "post_date": "2020-05-16T19:19:32.513000",
      "content": "<p>That code doesn't have a .to_csv for the if condition</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 851466,
      "author_name": "Peter Cnudde",
      "author_url": "",
      "post_date": "2020-05-17T16:34:42.307000",
      "content": "<p>I add a check for the existence of the test folder at the lowest level and I load a training image if it does not exist. This allows test all my code prior to submission. </p>\n\n<pre><code>    if os.path.exists(os.path.join(DATA,name+'.tiff')):\n        img = skimage.io.MultiImage(os.path.join(DATA,name+'.tiff'))[1]\n    else:\n        img = skimage.io.MultiImage(os.path.join('../input/prostate-cancer-grade-assessment/train_images','0076bcb66e46fb485f5ba432b9a1fe8a.tiff'))[1]\n</code></pre>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "850571": "That code doesn't have a .to_csv for the if condition",
    "851466": "I add a check for the existence of the test folder at the lowest level and I load a training image if it does not exist. This allows test all my code prior to submission. \n\n        if os.path.exists(os.path.join(DATA,name+'.tiff')):\n            img = skimage.io.MultiImage(os.path.join(DATA,name+'.tiff'))[1]\n        else:\n            img = skimage.io.MultiImage(os.path.join('../input/prostate-cancer-grade-assessment/train_images','0076bcb66e46fb485f5ba432b9a1fe8a.tiff'))[1]\n\n",
    "849814": "Maybe a different set of eyes can see what I'm not seeing. I updated @lafoss inference kernel to:\n\n```\n# # Prediction\n\n\nprint(DATA)\n\n#sub = inference(da = feature_engineering(data = test , dir_name = \"test_images\") , dir_path = \"test_images\")\n\nsub_df = pd.read_csv(SAMPLE)\nif os.path.exists(DATA):\n    print(\"RUN INFERENCE ON TEST DATA\")\n    ds = PandaDataset(DATA,TEST)\n    dl = DataLoader(ds, batch_size=bs, num_workers=nworkers, shuffle=False)\n    names,preds = [],[]\n\n    with torch.no_grad():\n        for x,y in tqdm(dl):\n            x = x.cuda()\n            #dihedral TTA\n            x = torch.stack([x,x.flip(-1),x.flip(-2),x.flip(-1,-2),\n              x.transpose(-1,-2),x.transpose(-1,-2).flip(-1),\n              x.transpose(-1,-2).flip(-2),x.transpose(-1,-2).flip(-1,-2)],1)\n            x = x.view(-1,N,3,sz,sz)\n            p = [model(x) for model in models]\n            p = torch.stack(p,1)\n            p = p.view(bs,8*len(models),-1).mean(1).argmax(-1).cpu()\n            names.append(y)\n            preds.append(p)\n    \n    names = np.concatenate(names)\n    preds = torch.cat(preds).numpy()\n    sub_df = pd.DataFrame({'image_id': names, 'isup_grade': preds})\nelse:\n    sub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    sub.to_csv('submission.csv', index=False)\n    #ub = pd.read_csv(\"/kaggle/input/prostate-cancer-grade-assessment/sample_submission.csv\")\n    #sub_df.to_csv('submission.csv', index=False)\n    sub_df.head()\n    print('SUBMISSION DATAFRAME LENGTH IS: ',len(sub_df.index))\n```\nNow, I get `submission csv not found`.  I greatly appreciate anyone who takes the time to tell me what the hech I am doing wrong. 😊 "
  }
}