{
  "id": 415322,
  "title": "How long does it take for you to submit the answer? Why did I submit it for a long time and finally got 0 points?",
  "url": "/competitions/asl-fingerspelling/discussion/415322",
  "author_name": "Chaton",
  "post_date": "2023-06-06T02:44:43.653000",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>How long does it take to score after submitting answers</p>",
  "messages": [
    {
      "id": 2290969,
      "postDate": "2023-06-07T08:25:46.633Z",
      "content": "<p>Hi Chaton,</p>\n<p>Someone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: <br>\n111111111<br>\non correct phrase:<br>\nlap<br>\nthen the length is 3 and the distance is 9, then the score would be:<br>\n(3-9)/3 = -3<br>\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.</p>\n<p>It can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.</p>",
      "rawMarkdown": "Hi Chaton,\n\nSomeone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: \n111111111\non correct phrase:\nlap\nthen the length is 3 and the distance is 9, then the score would be:\n(3-9)/3 = -3\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.\n\nIt can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.",
      "votes": 1,
      "replies": [
        {
          "id": 2291237,
          "postDate": "2023-06-07T12:30:45.447Z",
          "content": "<p>This is also my interpretation</p>",
          "rawMarkdown": "This is also my interpretation"
        }
      ]
    },
    {
      "id": 2289215,
      "postDate": "2023-06-06T02:44:43.653Z",
      "content": "<p>How long does it take to score after submitting answers</p>",
      "rawMarkdown": "\nHow long does it take to score after submitting answers",
      "votes": 1
    },
    {
      "id": 2289415,
      "postDate": "2023-06-06T06:27:39.200Z",
      "content": "<p>Hi Chaton,</p>\n<p>That really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).</p>\n<p>As for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference</a> , for example, gets a zero score.</p>",
      "rawMarkdown": "Hi Chaton,\n\nThat really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).\n\nAs for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook [https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference](url) , for example, gets a zero score."
    }
  ],
  "comments": [
    {
      "id": 2290969,
      "author_name": "FinnK",
      "author_url": "",
      "post_date": "2023-06-07T08:25:46.633000",
      "content": "<p>Hi Chaton,</p>\n<p>Someone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: <br>\n111111111<br>\non correct phrase:<br>\nlap<br>\nthen the length is 3 and the distance is 9, then the score would be:<br>\n(3-9)/3 = -3<br>\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.</p>\n<p>It can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2291237,
          "author_name": "Mathieu De Coster",
          "author_url": "",
          "post_date": "2023-06-07T12:30:45.447000",
          "content": "<p>This is also my interpretation</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2289415,
      "author_name": "Wondering Alice",
      "author_url": "",
      "post_date": "2023-06-06T06:27:39.200000",
      "content": "<p>Hi Chaton,</p>\n<p>That really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).</p>\n<p>As for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference</a> , for example, gets a zero score.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2290969": "Hi Chaton,\n\nSomeone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: \n111111111\non correct phrase:\nlap\nthen the length is 3 and the distance is 9, then the score would be:\n(3-9)/3 = -3\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.\n\nIt can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.",
    "2289215": "\nHow long does it take to score after submitting answers",
    "2289415": "Hi Chaton,\n\nThat really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).\n\nAs for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook [https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference](url) , for example, gets a zero score."
  }
}