Hello guys, i have various transformer seq-2-seq encoder-only models. I split to 9m train and 1m validation set and all of my models reach arround 0.63 r2 score. Then i split to another 9m train and 1m validation set and the models strugle at 0.56. With so many data i was under the impression that splits should not make much of a difference, i assumed that the sets would be more or less homogenous. Does it make sense to observe such a big differences between different splits? The validation sets share around 100k common rows.
", "messages": [ { "id": 2863389, "postDate": "2024-06-09T11:26:23.463Z", "content": "0.63 is not good and you probably have some negative r2 score targets. Fluctuation could be happening because of that. As your model gets stronger, it will start to get less and less negative r2 scores.
", "rawMarkdown": "0.63 is not good and you probably have some negative r2 score targets. Fluctuation could be happening because of that. As your model gets stronger, it will start to get less and less negative r2 scores. ", "votes": 5, "replies": [ { "id": 2864655, "postDate": "2024-06-10T09:07:51.697Z", "content": "Hey Gunes, i checked my losses per target variable, i have only 1 slightly negative target loss. What i noticed though is that the losses are very tiny in the first layers and they gets bigger in the higher layers, can this be because of the way i feed the data in the transformer? i start from 0 layer up to 59, i think that makes sense. Any idea why the transformer deteriorates as layers progress?
", "rawMarkdown": "Hey Gunes, i checked my losses per target variable, i have only 1 slightly negative target loss. What i noticed though is that the losses are very tiny in the first layers and they gets bigger in the higher layers, can this be because of the way i feed the data in the transformer? i start from 0 layer up to 59, i think that makes sense. Any idea why the transformer deteriorates as layers progress?", "replies": [ { "id": 2864752, "postDate": "2024-06-10T09:59:14.007Z", "content": "'I start from 0 layer up to 59'
\nYour architecture doesn't makes sense to me…sounds more like a decoder
no its an encoder only, my train data are N, 60, 25, oh i see so the order from 0 to 59 in the encoder does not matter i guess. Anyway i observe higher loss in higher layers
", "rawMarkdown": "no its an encoder only, my train data are N, 60, 25, oh i see so the order from 0 to 59 in the encoder does not matter i guess. Anyway i observe higher loss in higher layers\n" } ] } ] }, { "id": 2863329, "postDate": "2024-06-09T10:22:41.393Z", "content": "Hello guys, i have various transformer seq-2-seq encoder-only models. I split to 9m train and 1m validation set and all of my models reach arround 0.63 r2 score. Then i split to another 9m train and 1m validation set and the models strugle at 0.56. With so many data i was under the impression that splits should not make much of a difference, i assumed that the sets would be more or less homogenous. Does it make sense to observe such a big differences between different splits? The validation sets share around 100k common rows.
", "rawMarkdown": "Hello guys, i have various transformer seq-2-seq encoder-only models. I split to 9m train and 1m validation set and all of my models reach arround 0.63 r2 score. Then i split to another 9m train and 1m validation set and the models strugle at 0.56. With so many data i was under the impression that splits should not make much of a difference, i assumed that the sets would be more or less homogenous. Does it make sense to observe such a big differences between different splits? The validation sets share around 100k common rows.", "votes": 2 } ], "comments": [ { "id": 2863389, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-06-09T11:26:23.463000", "content": "0.63 is not good and you probably have some negative r2 score targets. Fluctuation could be happening because of that. As your model gets stronger, it will start to get less and less negative r2 scores.
", "votes": 5, "replies": [ { "id": 2864655, "author_name": "Vasilis", "author_url": "", "post_date": "2024-06-10T09:07:51.697000", "content": "Hey Gunes, i checked my losses per target variable, i have only 1 slightly negative target loss. What i noticed though is that the losses are very tiny in the first layers and they gets bigger in the higher layers, can this be because of the way i feed the data in the transformer? i start from 0 layer up to 59, i think that makes sense. Any idea why the transformer deteriorates as layers progress?
", "votes": 0, "replies": [ { "id": 2864752, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-10T09:59:14.007000", "content": "'I start from 0 layer up to 59'
\nYour architecture doesn't makes sense to me…sounds more like a decoder
no its an encoder only, my train data are N, 60, 25, oh i see so the order from 0 to 59 in the encoder does not matter i guess. Anyway i observe higher loss in higher layers
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2863389": "0.63 is not good and you probably have some negative r2 score targets. Fluctuation could be happening because of that. As your model gets stronger, it will start to get less and less negative r2 scores. ", "2863329": "Hello guys, i have various transformer seq-2-seq encoder-only models. I split to 9m train and 1m validation set and all of my models reach arround 0.63 r2 score. Then i split to another 9m train and 1m validation set and the models strugle at 0.56. With so many data i was under the impression that splits should not make much of a difference, i assumed that the sets would be more or less homogenous. Does it make sense to observe such a big differences between different splits? The validation sets share around 100k common rows." } }