{
  "id": 427648,
  "title": "Maximum Training Time vs Inference Time",
  "url": "/competitions/asl-fingerspelling/discussion/427648",
  "author_name": "Aaryam Sharma",
  "post_date": "2023-07-28T21:33:19.145000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Personally, I found that if my model takes more than 360 ms per step during training with a batch size of 32, then it times out during inference, but this is not the strongest bound. </p>\n<p>This is on a P100 GPU</p>\n<p>I'm curious about what other people found, and if there are methods of increasing final inference speed.</p>",
  "messages": [
    {
      "id": 2363850,
      "postDate": "2023-07-29T02:23:13.270Z",
      "content": "<p>Yo do not need to care about training time, you could test your infer time on kaggle notebook, as your local machine might be faster or slower.</p>",
      "rawMarkdown": "Yo do not need to care about training time, you could test your infer time on kaggle notebook, as your local machine might be faster or slower.",
      "votes": 1,
      "replies": [
        {
          "id": 2365065,
          "postDate": "2023-07-29T23:04:50.007Z",
          "content": "<p>Hi! Thank you for your response!<br>\nI completely understand, and I should have clarified that these times are from a Kaggle Notebook.</p>\n<p>Just for curiosity, are you using any kind of quantization? If so, which type, and how many parameters you were able to fit :)</p>",
          "rawMarkdown": "Hi! Thank you for your response!\nI completely understand, and I should have clarified that these times are from a Kaggle Notebook.\n\nJust for curiosity, are you using any kind of quantization? If so, which type, and how many parameters you were able to fit :)"
        }
      ]
    },
    {
      "id": 2363706,
      "postDate": "2023-07-28T21:33:19.147Z",
      "content": "<p>Personally, I found that if my model takes more than 360 ms per step during training with a batch size of 32, then it times out during inference, but this is not the strongest bound. </p>\n<p>This is on a P100 GPU</p>\n<p>I'm curious about what other people found, and if there are methods of increasing final inference speed.</p>",
      "rawMarkdown": "Personally, I found that if my model takes more than 360 ms per step during training with a batch size of 32, then it times out during inference, but this is not the strongest bound. \n\nThis is on a P100 GPU\n\nI'm curious about what other people found, and if there are methods of increasing final inference speed.\n",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 2363850,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2023-07-29T02:23:13.270000",
      "content": "<p>Yo do not need to care about training time, you could test your infer time on kaggle notebook, as your local machine might be faster or slower.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2365065,
          "author_name": "Aaryam Sharma",
          "author_url": "",
          "post_date": "2023-07-29T23:04:50.007000",
          "content": "<p>Hi! Thank you for your response!<br>\nI completely understand, and I should have clarified that these times are from a Kaggle Notebook.</p>\n<p>Just for curiosity, are you using any kind of quantization? If so, which type, and how many parameters you were able to fit :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2363850": "Yo do not need to care about training time, you could test your infer time on kaggle notebook, as your local machine might be faster or slower.",
    "2363706": "Personally, I found that if my model takes more than 360 ms per step during training with a batch size of 32, then it times out during inference, but this is not the strongest bound. \n\nThis is on a P100 GPU\n\nI'm curious about what other people found, and if there are methods of increasing final inference speed.\n"
  }
}