As other competitions does, let's discuss the correlate between CV and LB(public).
\nI used @markwijkhuizen 's pipeline.
\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference
Below is my current status:
\n| Model | \nVal loss | \nVal top1acc | \nLB | \n
|---|---|---|---|
| Baseline | \n1.39561 | \n0.76351 | \n0.642 | \n
| V1 | \n1.34531 | \n0.78055 | \n0.643 | \n
| V2 | \n1.35213 | \n0.78010 | \n0.631 | \n
It seems that I can't find the correlation between CV and LB. :(
", "messages": [ { "id": 2334628, "postDate": "2023-07-07T20:35:36.897Z", "content": "Your Val top1acc it's competition metric?
\nFor me correlation between CV and LB almost perfect. And I'm using (N - D) / N , where N and D global for whole validation dataset.
Like:
\n Levenshtein distance Lev_distance\n ():\n l = (s1)\n lvd = Lev_distance(s1, s2)\n lvd, l\n\nglobal_N, global_D = , \n step, batch pbar:\n logits = model(batch[])\n target_strings = convert_to_strings_function(batch[])\n predict_strings = convert_to_strings_function(logits)\n values = [calculate_N_D(target, predict) target, predict (target_strings, predict_strings)]\n global_D += np.([x[] x values])\n global_N += np.([x[] x values])\n\nmetric_value = np.clip((global_N - global_D) / global_N, a_min=, a_max=)\nHi,kolyaforrat, how do you split your train data, and valid by 5 fold or 10 fold? my cv is 0.78 and pb is just 0.68( use (N - D) / N, random split, 10 fold), I don't know where is the problem.
", "rawMarkdown": "Hi,kolyaforrat, how do you split your train data, and valid by 5 fold or 10 fold? my cv is 0.78 and pb is just 0.68( use (N - D) / N, random split, 10 fold), I don't know where is the problem.", "replies": [ { "id": 2334815, "postDate": "2023-07-08T02:41:15.417Z", "content": "you probably have data leakage in your splitting. I use 95% 5% train/val split, and the difference between my LB score and val score is only 0.5%~1%
", "rawMarkdown": "you probably have data leakage in your splitting. I use 95% 5% train/val split, and the difference between my LB score and val score is only 0.5%~1%", "votes": 1 }, { "id": 2334866, "postDate": "2023-07-08T04:48:38.557Z", "content": "You are right, I will try new ways to split, thank you !
", "rawMarkdown": "You are right, I will try new ways to split, thank you !" }, { "id": 2334867, "postDate": "2023-07-08T04:49:31.523Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2334525, "postDate": "2023-07-07T18:36:07.980Z", "content": "As other competitions does, let's discuss the correlate between CV and LB(public).
\nI used @markwijkhuizen 's pipeline.
\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference
Below is my current status:
\n| Model | \nVal loss | \nVal top1acc | \nLB | \n
|---|---|---|---|
| Baseline | \n1.39561 | \n0.76351 | \n0.642 | \n
| V1 | \n1.34531 | \n0.78055 | \n0.643 | \n
| V2 | \n1.35213 | \n0.78010 | \n0.631 | \n
It seems that I can't find the correlation between CV and LB. :(
", "rawMarkdown": "As other competitions does, let's discuss the correlate between CV and LB(public).\nI used @markwijkhuizen 's pipeline.\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference\n\nBelow is my current status:\n| Model | Val loss | Val top1acc | LB |\n| --- | --- |--- |--- |\n| Baseline | 1.39561 | 0.76351 | 0.642 |\n| V1| 1.34531 | 0.78055 | 0.643 |\n| V2| 1.35213 | 0.78010 | 0.631 |\n\n\nIt seems that I can't find the correlation between CV and LB. :(\n\n" }, { "id": 2337727, "postDate": "2023-07-10T09:59:10.047Z", "content": "The input is a dictionary of frames and phrases. I think phrases cause data leakage. I would consider to remove that parts from network which phrases are used. I also noticed that all the data is used for the mean and std calculation. You can change it in the data process notebook.
", "rawMarkdown": "The input is a dictionary of frames and phrases. I think phrases cause data leakage. I would consider to remove that parts from network which phrases are used. I also noticed that all the data is used for the mean and std calculation. You can change it in the data process notebook." } ], "comments": [ { "id": 2334628, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-07-07T20:35:36.897000", "content": "Your Val top1acc it's competition metric?
\nFor me correlation between CV and LB almost perfect. And I'm using (N - D) / N , where N and D global for whole validation dataset.
Like:
\n Levenshtein distance Lev_distance\n ():\n l = (s1)\n lvd = Lev_distance(s1, s2)\n lvd, l\n\nglobal_N, global_D = , \n step, batch pbar:\n logits = model(batch[])\n target_strings = convert_to_strings_function(batch[])\n predict_strings = convert_to_strings_function(logits)\n values = [calculate_N_D(target, predict) target, predict (target_strings, predict_strings)]\n global_D += np.([x[] x values])\n global_N += np.([x[] x values])\n\nmetric_value = np.clip((global_N - global_D) / global_N, a_min=, a_max=)\nHi,kolyaforrat, how do you split your train data, and valid by 5 fold or 10 fold? my cv is 0.78 and pb is just 0.68( use (N - D) / N, random split, 10 fold), I don't know where is the problem.
", "votes": 0, "replies": [ { "id": 2334815, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-07-08T02:41:15.417000", "content": "you probably have data leakage in your splitting. I use 95% 5% train/val split, and the difference between my LB score and val score is only 0.5%~1%
", "votes": 1, "replies": [] }, { "id": 2334866, "author_name": "Xiang Huang", "author_url": "", "post_date": "2023-07-08T04:48:38.557000", "content": "You are right, I will try new ways to split, thank you !
", "votes": 0, "replies": [] }, { "id": 2334867, "author_name": "", "author_url": "", "post_date": "2023-07-08T04:49:31.523000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2337727, "author_name": "Farukcan Saglam", "author_url": "", "post_date": "2023-07-10T09:59:10.047000", "content": "The input is a dictionary of frames and phrases. I think phrases cause data leakage. I would consider to remove that parts from network which phrases are used. I also noticed that all the data is used for the mean and std calculation. You can change it in the data process notebook.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2334628": "Your `Val top1acc` it's competition metric?\nFor me correlation between CV and LB almost perfect. And I'm using `(N - D) / N` , where N and D global for whole validation dataset.\n\nLike:\n\n```python\n\nfrom Levenshtein import distance as Lev_distance\ndef calculate_N_D(s1, s2):\n l = len(s1)\n lvd = Lev_distance(s1, s2)\n return lvd, l\n\nglobal_N, global_D = 0, 0\nfor step, batch in pbar:\n logits = model(batch['signs'])\n target_strings = convert_to_strings_function(batch['text'])\n predict_strings = convert_to_strings_function(logits)\n values = [calculate_N_D(target, predict) for target, predict in zip(target_strings, predict_strings)]\n global_D += np.sum([x[0] for x in values])\n global_N += np.sum([x[1] for x in values])\n\nmetric_value = np.clip((global_N - global_D) / global_N, a_min=0, a_max=1)\n```", "2334525": "As other competitions does, let's discuss the correlate between CV and LB(public).\nI used @markwijkhuizen 's pipeline.\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference\n\nBelow is my current status:\n| Model | Val loss | Val top1acc | LB |\n| --- | --- |--- |--- |\n| Baseline | 1.39561 | 0.76351 | 0.642 |\n| V1| 1.34531 | 0.78055 | 0.643 |\n| V2| 1.35213 | 0.78010 | 0.631 |\n\n\nIt seems that I can't find the correlation between CV and LB. :(\n\n", "2337727": "The input is a dictionary of frames and phrases. I think phrases cause data leakage. I would consider to remove that parts from network which phrases are used. I also noticed that all the data is used for the mean and std calculation. You can change it in the data process notebook." } }