Hi @rodrigotenorio @michaeljwill, I'm trying to construct a reasonable validation set, but I'm not sure if the public leaderboard(24% of the test set) is splited randomly? Some of my experiments demonstrate that my method may have some overfitting issue so I'm afraid that my private score would collapse if the test set is not splited randomly. Looking forward to your reply, thanks in advance.
", "messages": [ { "id": 2049942, "postDate": "2022-11-30T12:46:06.420Z", "content": "Hi @zzzc18, I can confirm the public/private split is indeed random. So whilst the private scores will likely be different, if you're not overfitting to the public score, they should be roughly comparable in most cases.
", "rawMarkdown": "Hi @zzzc18, I can confirm the public/private split is indeed random. So whilst the private scores will likely be different, if you're not overfitting to the public score, they should be roughly comparable in most cases.", "votes": 13, "replies": [ { "id": 2050089, "postDate": "2022-11-30T14:13:35.097Z", "content": "Thanks for replying, got it.
", "rawMarkdown": "Thanks for replying, got it.", "votes": 2 } ] }, { "id": 2049918, "postDate": "2022-11-30T12:24:56.403Z", "content": "Hi @rodrigotenorio @michaeljwill, I'm trying to construct a reasonable validation set, but I'm not sure if the public leaderboard(24% of the test set) is splited randomly? Some of my experiments demonstrate that my method may have some overfitting issue so I'm afraid that my private score would collapse if the test set is not splited randomly. Looking forward to your reply, thanks in advance.
", "rawMarkdown": "Hi @rodrigotenorio @michaeljwill, I'm trying to construct a reasonable validation set, but I'm not sure if the public leaderboard(24% of the test set) is splited randomly? Some of my experiments demonstrate that my method may have some overfitting issue so I'm afraid that my private score would collapse if the test set is not splited randomly. Looking forward to your reply, thanks in advance.", "votes": 8 }, { "id": 2050352, "postDate": "2022-11-30T17:22:26.657Z", "content": "I had the same doubt. Good post!
", "rawMarkdown": "I had the same doubt. Good post!", "votes": 2 } ], "comments": [ { "id": 2049942, "author_name": "Michael J. Williams", "author_url": "", "post_date": "2022-11-30T12:46:06.420000", "content": "Hi @zzzc18, I can confirm the public/private split is indeed random. So whilst the private scores will likely be different, if you're not overfitting to the public score, they should be roughly comparable in most cases.
", "votes": 13, "replies": [ { "id": 2050089, "author_name": "zzzc18", "author_url": "", "post_date": "2022-11-30T14:13:35.097000", "content": "Thanks for replying, got it.
", "votes": 2, "replies": [] } ] }, { "id": 2050352, "author_name": "Zollkron", "author_url": "", "post_date": "2022-11-30T17:22:26.657000", "content": "I had the same doubt. Good post!
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2049942": "Hi @zzzc18, I can confirm the public/private split is indeed random. So whilst the private scores will likely be different, if you're not overfitting to the public score, they should be roughly comparable in most cases.", "2049918": "Hi @rodrigotenorio @michaeljwill, I'm trying to construct a reasonable validation set, but I'm not sure if the public leaderboard(24% of the test set) is splited randomly? Some of my experiments demonstrate that my method may have some overfitting issue so I'm afraid that my private score would collapse if the test set is not splited randomly. Looking forward to your reply, thanks in advance.", "2050352": "I had the same doubt. Good post!" } }