{ "id": 428572, "title": "Are there duplicate participant_ids in the test dataset?", "url": "/competitions/asl-fingerspelling/discussion/428572", "author_name": "gezi", "post_date": "2023-08-02T01:03:18.174000", "votes": 0, "comment_count": 5, "views": 0, "content": "

I did not found related info from dataset decription.
\nJust wondering wether we need to do cv considering participant_id?

", "messages": [ { "id": 2369717, "postDate": "2023-08-02T01:33:59.090Z", "content": "

In the previous comp, it was confirmed by @sohier that there was no overlap between train/pulic/private. I guess it’s also the case in this competition. see https://www.kaggle.com/competitions/asl-signs/discussion/406364

", "rawMarkdown": "In the previous comp, it was confirmed by @sohier that there was no overlap between train/pulic/private. I guess it’s also the case in this competition. see https://www.kaggle.com/competitions/asl-signs/discussion/406364", "votes": 1, "replies": [ { "id": 2369721, "postDate": "2023-08-02T01:39:17.597Z", "content": "

Thanks! @hoyso48
\nBut I compared local score, if just random split, local score is much more closer to LB. @sohier Could organizer tell if there are duplicates of particpant_ids between train and test?

", "rawMarkdown": "Thanks! @hoyso48\nBut I compared local score, if just random split, local score is much more closer to LB. @sohier Could organizer tell if there are duplicates of particpant_ids between train and test?", "votes": 1, "replies": [ { "id": 2372163, "postDate": "2023-08-03T14:20:08.553Z", "content": "

same for me. The performance on data split according to participant id is not consistent between CV and LB. 0.01 improvement on CV could result in -0.05 on LB.

", "rawMarkdown": "same for me. The performance on data split according to participant id is not consistent between CV and LB. 0.01 improvement on CV could result in -0.05 on LB." }, { "id": 2373896, "postDate": "2023-08-04T13:43:14.690Z", "content": "

Yes personally I would assume the test with same participant ids as train but might have a few new participant ids also. Just like learning-equality competion more epochs produce better result online, but here if too much epochs might not improve LB so that may mean there are new participant ids in the test.

", "rawMarkdown": "Yes personally I would assume the test with same participant ids as train but might have a few new participant ids also. Just like learning-equality competion more epochs produce better result online, but here if too much epochs might not improve LB so that may mean there are new participant ids in the test." } ] } ] }, { "id": 2369697, "postDate": "2023-08-02T01:03:18.173Z", "content": "

I did not found related info from dataset decription.
\nJust wondering wether we need to do cv considering participant_id?

", "rawMarkdown": "I did not found related info from dataset decription. \nJust wondering wether we need to do cv considering participant_id?" }, { "id": 2371954, "postDate": "2023-08-03T11:30:31.113Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2369717, "author_name": "hoyso48", "author_url": "", "post_date": "2023-08-02T01:33:59.090000", "content": "

In the previous comp, it was confirmed by @sohier that there was no overlap between train/pulic/private. I guess it’s also the case in this competition. see https://www.kaggle.com/competitions/asl-signs/discussion/406364

", "votes": 1, "replies": [ { "id": 2369721, "author_name": "gezi", "author_url": "", "post_date": "2023-08-02T01:39:17.597000", "content": "

Thanks! @hoyso48
\nBut I compared local score, if just random split, local score is much more closer to LB. @sohier Could organizer tell if there are duplicates of particpant_ids between train and test?

", "votes": 1, "replies": [ { "id": 2372163, "author_name": "bliao", "author_url": "", "post_date": "2023-08-03T14:20:08.553000", "content": "

same for me. The performance on data split according to participant id is not consistent between CV and LB. 0.01 improvement on CV could result in -0.05 on LB.

", "votes": 0, "replies": [] }, { "id": 2373896, "author_name": "gezi", "author_url": "", "post_date": "2023-08-04T13:43:14.690000", "content": "

Yes personally I would assume the test with same participant ids as train but might have a few new participant ids also. Just like learning-equality competion more epochs produce better result online, but here if too much epochs might not improve LB so that may mean there are new participant ids in the test.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2371954, "author_name": "", "author_url": "", "post_date": "2023-08-03T11:30:31.113000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2369717": "In the previous comp, it was confirmed by @sohier that there was no overlap between train/pulic/private. I guess it’s also the case in this competition. see https://www.kaggle.com/competitions/asl-signs/discussion/406364", "2369697": "I did not found related info from dataset decription. \nJust wondering wether we need to do cv considering participant_id?", "2371954": "" } }