{
  "id": 373222,
  "title": "Timed out vs 5hrs",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/373222",
  "author_name": "Kolya Forrat",
  "post_date": "2022-12-20T08:45:24.249000",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hello everyone!</p>\n<p>I submitted 6 times the same code. Difference was only in threshold value. But 3 times inference time ~5hours and 3 times Notebook timeout.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Ff843a5dfff9c05bd936e76537ec89a94%2Fload.png?generation=1671526351935712&amp;alt=media\" alt=\"\"></p>\n<p>Do you have any ideas how to deal with it?</p>",
  "messages": [
    {
      "id": 2070668,
      "postDate": "2022-12-20T08:45:24.250Z",
      "content": "<p>Hello everyone!</p>\n<p>I submitted 6 times the same code. Difference was only in threshold value. But 3 times inference time ~5hours and 3 times Notebook timeout.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Ff843a5dfff9c05bd936e76537ec89a94%2Fload.png?generation=1671526351935712&amp;alt=media\" alt=\"\"></p>\n<p>Do you have any ideas how to deal with it?</p>",
      "rawMarkdown": "Hello everyone!\n\nI submitted 6 times the same code. Difference was only in threshold value. But 3 times inference time ~5hours and 3 times Notebook timeout.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Ff843a5dfff9c05bd936e76537ec89a94%2Fload.png?generation=1671526351935712&alt=media)\n\nDo you have any ideas how to deal with it?",
      "votes": 3
    },
    {
      "id": 2091274,
      "postDate": "2023-01-08T07:40:13.943Z",
      "content": "<p>I don't know what happened, but in the last few days this problem has almost disappeared. The difference now is only +-0.5 hours. Not +-5 hours.<br>\nHas anyone else noticed this?</p>",
      "rawMarkdown": "I don't know what happened, but in the last few days this problem has almost disappeared. The difference now is only +-0.5 hours. Not +-5 hours.\nHas anyone else noticed this?",
      "votes": 1,
      "replies": [
        {
          "id": 2091310,
          "postDate": "2023-01-08T08:35:03.120Z",
          "content": "<p>On my side I still have exactly the same code run twice, one with timeout one with 8h runtime. I got it yesterday again. However, I'm quite sure now it's a bad condition in my Parallel multiprocessing. Something is either not thread safe or locking during some loops. I've removed  the Parallel multiprocessing and runtime is stable now.</p>",
          "rawMarkdown": "On my side I still have exactly the same code run twice, one with timeout one with 8h runtime. I got it yesterday again. However, I'm quite sure now it's a bad condition in my Parallel multiprocessing. Something is either not thread safe or locking during some loops. I've removed  the Parallel multiprocessing and runtime is stable now."
        }
      ]
    },
    {
      "id": 2075813,
      "postDate": "2022-12-25T23:12:27.393Z",
      "content": "<p>The same happens to me.<br>\nI do not know why. I'm saving the submission.csv file in /kaggle/working but the same message appears.<br>\nIf you know the problem, please post the solution here!<br>\nthanks</p>",
      "rawMarkdown": "The same happens to me.\nI do not know why. I'm saving the submission.csv file in /kaggle/working but the same message appears.\nIf you know the problem, please post the solution here!\nthanks",
      "votes": 1
    },
    {
      "id": 2070698,
      "postDate": "2022-12-20T09:19:50.267Z",
      "content": "<p>I met same problem. No idea what is going on.</p>",
      "rawMarkdown": "I met same problem. No idea what is going on.",
      "votes": 1
    },
    {
      "id": 2082587,
      "postDate": "2023-01-01T17:45:45.933Z",
      "content": "<p>Inference time looks not consistent for me too. Same code run twice (threshold change) can different from 1 hour.<br>\nSo either there is a race condition in the <code>Parallel</code> usage in my code or Kaggle environment is loaded.</p>",
      "rawMarkdown": "Inference time looks not consistent for me too. Same code run twice (threshold change) can different from 1 hour.\nSo either there is a race condition in the `Parallel` usage in my code or Kaggle environment is loaded.",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 2091274,
      "author_name": "Kolya Forrat",
      "author_url": "",
      "post_date": "2023-01-08T07:40:13.943000",
      "content": "<p>I don't know what happened, but in the last few days this problem has almost disappeared. The difference now is only +-0.5 hours. Not +-5 hours.<br>\nHas anyone else noticed this?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2091310,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2023-01-08T08:35:03.120000",
          "content": "<p>On my side I still have exactly the same code run twice, one with timeout one with 8h runtime. I got it yesterday again. However, I'm quite sure now it's a bad condition in my Parallel multiprocessing. Something is either not thread safe or locking during some loops. I've removed  the Parallel multiprocessing and runtime is stable now.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2075813,
      "author_name": "manuel neto",
      "author_url": "",
      "post_date": "2022-12-25T23:12:27.393000",
      "content": "<p>The same happens to me.<br>\nI do not know why. I'm saving the submission.csv file in /kaggle/working but the same message appears.<br>\nIf you know the problem, please post the solution here!<br>\nthanks</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2070698,
      "author_name": "dragon zhang",
      "author_url": "",
      "post_date": "2022-12-20T09:19:50.267000",
      "content": "<p>I met same problem. No idea what is going on.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2082587,
      "author_name": "MPWARE",
      "author_url": "",
      "post_date": "2023-01-01T17:45:45.933000",
      "content": "<p>Inference time looks not consistent for me too. Same code run twice (threshold change) can different from 1 hour.<br>\nSo either there is a race condition in the <code>Parallel</code> usage in my code or Kaggle environment is loaded.</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2070668": "Hello everyone!\n\nI submitted 6 times the same code. Difference was only in threshold value. But 3 times inference time ~5hours and 3 times Notebook timeout.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Ff843a5dfff9c05bd936e76537ec89a94%2Fload.png?generation=1671526351935712&alt=media)\n\nDo you have any ideas how to deal with it?",
    "2091274": "I don't know what happened, but in the last few days this problem has almost disappeared. The difference now is only +-0.5 hours. Not +-5 hours.\nHas anyone else noticed this?",
    "2075813": "The same happens to me.\nI do not know why. I'm saving the submission.csv file in /kaggle/working but the same message appears.\nIf you know the problem, please post the solution here!\nthanks",
    "2070698": "I met same problem. No idea what is going on.",
    "2082587": "Inference time looks not consistent for me too. Same code run twice (threshold change) can different from 1 hour.\nSo either there is a race condition in the `Parallel` usage in my code or Kaggle environment is loaded."
  }
}