This thread has returned as per tradition.
\nAn updated continuation of the CV/LB thread for the latter half of the competition.
I'll start with mine:
\nArchitecture: transformer based encoder
\nLB: 0.75280
\nCV: 0.66317
\nCV method: 5 fold (single fold), 0.5m samples, clip R2 to (-1, 1), exclude masked values from sample sub.
\nCV curve: (clipped to -0.1, 1)
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3586013%2F130e770c5f8b7b7b2b9c7ebcbb48eeab%2FimageData.png?generation=1718107905493582&alt=media\")
\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.
\nGood luck everyone.
Edit: New LB (similar model) ~0.761
", "messages": [ { "id": 2866146, "postDate": "2024-06-11T06:53:50.650Z", "content": "This thread has returned as per tradition.
\nAn updated continuation of the CV/LB thread for the latter half of the competition.
I'll start with mine:
\nArchitecture: transformer based encoder
\nLB: 0.75280
\nCV: 0.66317
\nCV method: 5 fold (single fold), 0.5m samples, clip R2 to (-1, 1), exclude masked values from sample sub.
\nCV curve: (clipped to -0.1, 1)
\n
\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.
\nGood luck everyone.
Edit: New LB (similar model) ~0.761
", "rawMarkdown": "This thread has returned as per tradition.\nAn updated continuation of the [CV/LB thread](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/498194) for the latter half of the competition.\n\nI'll start with mine:\nArchitecture: transformer based encoder\nLB: 0.75280\nCV: 0.66317\nCV method: 5 fold (single fold), 0.5m samples, clip R2 to (-1, 1), exclude masked values from sample sub.\nCV curve: (clipped to -0.1, 1)\n\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.\nGood luck everyone.\n\nEdit: New LB (similar model) ~0.761", "votes": 18 }, { "id": 2866200, "postDate": "2024-06-11T07:43:54.920Z", "content": "I use 625000 samples for validation. Current best single model, cv: 0.766, lb: 0.769.Still have lots of work to do.
", "rawMarkdown": "I use 625000 samples for validation. Current best single model, cv: 0.766, lb: 0.769.Still have lots of work to do.", "votes": 9, "replies": [ { "id": 2867712, "postDate": "2024-06-12T03:44:25.613Z", "content": "Very impressive, do you still see large rooms for improvement for a single model?
", "rawMarkdown": "Very impressive, do you still see large rooms for improvement for a single model?" } ] }, { "id": 2888643, "postDate": "2024-06-25T02:20:02.397Z", "content": "single model
\nCV: 0.7844
\nNew LB: 0.7794
\n--- update ---
\nCV: 7864
\nNew LB: 0.782
How many parameters in your model?
", "rawMarkdown": "How many parameters in your model?", "votes": 1 } ] }, { "id": 2881771, "postDate": "2024-06-21T00:25:56.913Z", "content": "One fold (out of 5), trained on all data, new test data, submitted just minutes ago. CV=0.7630. LB=0.75535. So LB - CV = -0.77%.
\nThis is concerning, because on old test data my LB was almost identical to CV - i think differences between them were always less than 0.1%.
\nDoes this mean that the new test data is materially different from the train data (and from old test data)? From looking at leaderboard it appears that all the scores are way down from the old test data, so it is probably not just me who has this problem.
\nComments?
", "rawMarkdown": "One fold (out of 5), trained on all data, new test data, submitted just minutes ago. CV=0.7630. LB=0.75535. So LB - CV = -0.77%.\n\nThis is concerning, because on old test data my LB was almost identical to CV - i think differences between them were always less than 0.1%.\n\nDoes this mean that the new test data is materially different from the train data (and from old test data)? From looking at leaderboard it appears that all the scores are way down from the old test data, so it is probably not just me who has this problem.\n\nComments?", "votes": 6, "replies": [ { "id": 2882239, "postDate": "2024-06-21T08:38:52.343Z", "content": "Old-new LB diff should be ~0.003. If your old LB was also 0.763 then your difference is too big, did you remember to apply trick to ptend_q0002 indices 12-14? Top scores are 'way down' (more than ~0.003 diff) because most top teams probably just submitted some high-scoring single models instead of the same ensembles they had at old LB. This is at least my case.
", "rawMarkdown": "Old-new LB diff should be ~0.003. If your old LB was also 0.763 then your difference is too big, did you remember to apply trick to ptend_q0002 indices 12-14? Top scores are 'way down' (more than ~0.003 diff) because most top teams probably just submitted some high-scoring single models instead of the same ensembles they had at old LB. This is at least my case.", "votes": 2, "replies": [ { "id": 2882548, "postDate": "2024-06-21T12:09:20.567Z", "content": "OK, thanks. Maybe it just me - i'll double-check my submission logic.
", "rawMarkdown": "OK, thanks. Maybe it just me - i'll double-check my submission logic." }, { "id": 2884064, "postDate": "2024-06-22T09:29:06.707Z", "content": "My CV/LB gap was 0.005 before the update and now it's 0.015.
", "rawMarkdown": "My CV/LB gap was 0.005 before the update and now it's 0.015." } ] }, { "id": 2884527, "postDate": "2024-06-22T15:13:02.527Z", "content": "Nice scores, thanks for sharing. \"trained on all data\" with this you mean competition training data (downsampled) or entire low-res data?
", "rawMarkdown": "Nice scores, thanks for sharing. \"trained on all data\" with this you mean competition training data (downsampled) or entire low-res data?" } ] }, { "id": 2869754, "postDate": "2024-06-13T08:45:09.420Z", "content": "Validation: 0.7582
\nLB: 0.7534
\nSplit: Single holdout
\nMetric: r2_score(y_true * weights, y_pred * weights)
![\"q1\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2F8e2989afbd29784513a2230c06a5194c%2Fptend_q0001_scores.png?generation=1718268470483041&alt=media\")
![\"q2\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2Fdc4b6c69563e866c2f376ab5c3a5c4e4%2Fptend_q0002_scores.png?generation=1718268480862178&alt=media\")
![\"q3\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2Ff83113185c4befcacb8b8eb11e61bc79%2Fptend_q0003_scores.png?generation=1718268490804572&alt=media\")
![\"t\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2F50fb29066114829cfe8cd24c16ecfd5f%2Fptend_t_scores.png?generation=1718268511712532&alt=media\")
![\"u\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2F19197fab41cd57b51e2a28ede349a483%2Fptend_u_scores.png?generation=1718268521689842&alt=media\")
![\"v\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2F23582a8c9da9ad46f8b75ce154677b7a%2Fptend_v_scores.png?generation=1718268529991291&alt=media\")
![\"single\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2706866%2Fb3757d18a54fe3a6fedda12948f691ec%2Fsingle_target_scores.png?generation=1718268553787515&alt=media\")
How did you avoid large negative r2 scores? I am using a simple model, a CNN, and get ~0.61, but sometimes I get large negative scores for some targets. I am new to ML and I spend a lot of time debugging, so I still haven’t tried a more complex model. Do these errors go away with better/larger models? Or is it related to the scaling that I’m using (Notmalized values)? Thanks
", "rawMarkdown": "How did you avoid large negative r2 scores? I am using a simple model, a CNN, and get ~0.61, but sometimes I get large negative scores for some targets. I am new to ML and I spend a lot of time debugging, so I still haven’t tried a more complex model. Do these errors go away with better/larger models? Or is it related to the scaling that I’m using (Notmalized values)? Thanks", "replies": [ { "id": 2874031, "postDate": "2024-06-16T01:23:07.320Z", "content": "Don’t forget to multiply the predictions by the weights when calculating the r2.
", "rawMarkdown": "Don’t forget to multiply the predictions by the weights when calculating the r2.", "votes": 1 } ] } ] }, { "id": 2884531, "postDate": "2024-06-22T15:15:04.403Z", "content": "Single Model, 7M samples:
\nCV: 0.7697
\nOld LB: 0.772x
\nNew LB: 0.7687
0.70859
\nI am working on Amadeo's model.
\nFirst, I understood every line of the code, and after that, I began to improve it.
\nI think with more epochs (about 20 right now), we can achieve a score of 0.715, maybe 0.72 or even 0.73.
\nBut for a silver or gold medal, we will need a big change in the architecture.
Are you using a 1D cnn (as per Amedeo's notebook)?
\nIf so, you can try transformers, rnns and unets as well!
I am also working on @abiolatti 's notebook, improving the architecture, but cant seem to get past 67 LB… some hint? ;)
", "rawMarkdown": "I am also working on @abiolatti 's notebook, improving the architecture, but cant seem to get past 67 LB... some hint? ;)", "votes": 1 }, { "id": 2867883, "postDate": "2024-06-12T06:05:55.597Z", "content": "Try a 1d cnn with resnet or unet architecture
", "rawMarkdown": "Try a 1d cnn with resnet or unet architecture", "votes": 2 }, { "id": 2868256, "postDate": "2024-06-12T09:18:28.647Z", "content": "Thanks! I will try it
", "rawMarkdown": "Thanks! I will try it", "votes": 1 }, { "id": 2868275, "postDate": "2024-06-12T09:31:04.913Z", "content": "Try a UNet with at least 1M parameters and train longer
", "rawMarkdown": "Try a UNet with at least 1M parameters and train longer", "votes": 4 }, { "id": 2868565, "postDate": "2024-06-12T13:23:57.403Z", "content": "Is a 1D CNN + ResNet like the Amadeo CNN network with additional ResNet layers?
", "rawMarkdown": "Is a 1D CNN + ResNet like the Amadeo CNN network with additional ResNet layers?" }, { "id": 2868572, "postDate": "2024-06-12T13:26:23.953Z", "content": "@abiolatti my current network has 6MM parameters, I think it's very complex for a \"small\" gain (0.07) compared to your notebook.
\nDo you have any examples (may be external to kaggle) of UNet applied to a regression problem?
https://www.kaggle.com/code/ishandandekar/u-net-from-scratch-using-keras-and-tensorflow
", "rawMarkdown": "https://www.kaggle.com/code/ishandandekar/u-net-from-scratch-using-keras-and-tensorflow", "votes": 1 }, { "id": 2869166, "postDate": "2024-06-12T21:56:28.190Z", "content": "There is also this notebook shared by @farisalahmdi : https://www.kaggle.com/code/farisalahmdi/mlp-and-unet-baseline which would be easy to adapt
", "rawMarkdown": "There is also this notebook shared by @farisalahmdi : https://www.kaggle.com/code/farisalahmdi/mlp-and-unet-baseline which would be easy to adapt" }, { "id": 2869420, "postDate": "2024-06-13T05:09:07.023Z", "content": "How did you predict ptend_q0001_12-15 and ptend_q0003_12-15? I failed to get a positive R2 score on these targets by improving the architecture. The notebook shared by @konstantinboyko :https://www.kaggle.com/code/konstantinboyko/keras-baseline-seq2seq , increasing the epochs to 100 based on your notebook, still did not solve the problem.
", "rawMarkdown": "How did you predict ptend_q0001_12-15 and ptend_q0003_12-15? I failed to get a positive R2 score on these targets by improving the architecture. The notebook shared by @konstantinboyko :https://www.kaggle.com/code/konstantinboyko/keras-baseline-seq2seq , increasing the epochs to 100 based on your notebook, still did not solve the problem." }, { "id": 2869661, "postDate": "2024-06-13T07:05:17.953Z", "content": "For me, the start of q1 and q3 were negative before because I calculated R2 batch-wise instead of across the whole validation set, hence the tip in the main post.
", "rawMarkdown": "For me, the start of q1 and q3 were negative before because I calculated R2 batch-wise instead of across the whole validation set, hence the tip in the main post.", "votes": 2 }, { "id": 2869895, "postDate": "2024-06-13T10:41:58.007Z", "content": "is U-net enough to beat LB0.7? I am trying it with 2M parameters and 3hrs of training.
", "rawMarkdown": "is U-net enough to beat LB0.7? I am trying it with 2M parameters and 3hrs of training." }, { "id": 2869938, "postDate": "2024-06-13T10:57:38.620Z", "content": "I got a UNet with 1.3M params to 0.75, but the training was 24+h
", "rawMarkdown": "I got a UNet with 1.3M params to 0.75, but the training was 24+h", "votes": 7 }, { "id": 2870373, "postDate": "2024-06-13T14:50:15.677Z", "content": "0.75 with 1.3M is incredible.
\nI tried a Unet 1D, 4 encoder-4 decoder, 7MM Parameteres, 20 epochs, 0.67 LB.
I also used UNet baseline. I am surprised at how long the training was. Thank you very much for telling me this information.
", "rawMarkdown": "I also used UNet baseline. I am surprised at how long the training was. Thank you very much for telling me this information." }, { "id": 2870436, "postDate": "2024-06-13T15:21:25.340Z", "content": "Hyperparameters may be critical for UNet
", "rawMarkdown": "Hyperparameters may be critical for UNet" }, { "id": 2870439, "postDate": "2024-06-13T15:22:10.900Z", "content": "Is the graph posted correct? Thanks, I will check my work.
", "rawMarkdown": "Is the graph posted correct? Thanks, I will check my work." }, { "id": 2877192, "postDate": "2024-06-18T08:30:57.057Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2869165, "postDate": "2024-06-12T21:50:14.520Z", "content": "In comparing with my own R2 graph, I have a much more substantial dip in scores in the 270-280 range regardless of using cnn1d, MLP, or U-Net. I'm still musing on whether this is really related to the architecture or not, though I haven't tried transformer encoder.
", "rawMarkdown": "In comparing with my own R2 graph, I have a much more substantial dip in scores in the 270-280 range regardless of using cnn1d, MLP, or U-Net. I'm still musing on whether this is really related to the architecture or not, though I haven't tried transformer encoder.", "votes": 1 }, { "id": 2868857, "postDate": "2024-06-12T17:18:00.547Z", "content": "I also use a transformer encoder with 3.6 million parameters and my score is around 0.63 so i am very curious what do i do so wrong. Did you train your model many epochs? i think i have train it for less than 10. I have nhead=8 d_model =256 and 2 encoder layers.
", "rawMarkdown": "I also use a transformer encoder with 3.6 million parameters and my score is around 0.63 so i am very curious what do i do so wrong. Did you train your model many epochs? i think i have train it for less than 10. I have nhead=8 d_model =256 and 2 encoder layers.", "votes": 1, "replies": [ { "id": 2869137, "postDate": "2024-06-12T21:09:50.983Z", "content": "You need more layers, training for longer helps too.
", "rawMarkdown": "You need more layers, training for longer helps too.", "votes": 3 } ] }, { "id": 2866763, "postDate": "2024-06-11T13:48:44.793Z", "content": "Is the LB score truly a single model or the average of the 5 from CV?
", "rawMarkdown": "Is the LB score truly a single model or the average of the 5 from CV?", "votes": 1, "replies": [ { "id": 2866769, "postDate": "2024-06-11T13:55:45.617Z", "content": "Everything shown above is a single fold. I have not trained other folds yet.
", "rawMarkdown": "Everything shown above is a single fold. I have not trained other folds yet.", "votes": 1 } ] }, { "id": 2869505, "postDate": "2024-06-13T06:11:52.540Z", "content": "unet
\ncv 0.7387 lb 0.739
\ntraining all data waste only 2 hours
\ncv: 1 flod
\ni trust my cv.
\ni only impove my cv score
\ni try to expand my model and train time.
Can I ask you the number of model parameters that achieved this score
", "rawMarkdown": "Can I ask you the number of model parameters that achieved this score", "votes": 2, "replies": [ { "id": 2869136, "postDate": "2024-06-12T21:08:58.617Z", "content": "Between 7-8m parameters.
", "rawMarkdown": "Between 7-8m parameters.", "votes": 4 } ] }, { "id": 2885508, "postDate": "2024-06-23T06:17:24.470Z", "content": "Very useful analysis and great notebook @sroger
", "rawMarkdown": "Very useful analysis and great notebook @sroger " }, { "id": 2878107, "postDate": "2024-06-18T18:28:37.527Z", "content": "\n\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.
\n
What happens if you batch the calculations? Do you get a worse result?
", "rawMarkdown": "> BTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.\n\nWhat happens if you batch the calculations? Do you get a worse result?\n\n", "replies": [ { "id": 2883289, "postDate": "2024-06-21T19:50:29.037Z", "content": "It ruins the score for numerous columns (start of q1-3 after sub mask).
", "rawMarkdown": "It ruins the score for numerous columns (start of q1-3 after sub mask)." } ] }, { "id": 2877573, "postDate": "2024-06-18T13:02:03.333Z", "content": "@sroger @motokisatokaggle kaggle for a transformer encoder, is positional encoding necessary?
", "rawMarkdown": "@sroger @motokisatokaggle kaggle for a transformer encoder, is positional encoding necessary?" }, { "id": 2877347, "postDate": "2024-06-18T10:16:12.020Z", "content": "Train only on 1.87M samples. 16 epochs. PyTorch.
\nCV: 0.500 (I'm definitely calculating it incorrectly)
\nLB: 0.679
I am getting something similar. My CV was 0.51 and LB was 0.68, but CV and LB always grow proportionately for me, so at least it serves the purpose of showing the trend.
", "rawMarkdown": "I am getting something similar. My CV was 0.51 and LB was 0.68, but CV and LB always grow proportionately for me, so at least it serves the purpose of showing the trend.", "votes": 1, "replies": [ { "id": 2877960, "postDate": "2024-06-18T16:54:58.733Z", "content": "How many observations did you use to get this result? For me, CV and LB also grow proportionally and this reassure. I'm very interested in what result I will have when increasing the amount of training data by 5 times and number of epochs from 16 to ~50.
", "rawMarkdown": "How many observations did you use to get this result? For me, CV and LB also grow proportionally and this reassure. I'm very interested in what result I will have when increasing the amount of training data by 5 times and number of epochs from 16 to ~50." }, { "id": 2878099, "postDate": "2024-06-18T18:21:26.860Z", "content": "I used only the data on train.csv, but I have my own scaled dataset saved in tfrecords. I held-off ~5% of data for validation (I shuffled everything beforehand). it ran for like 18 epochs before early-stopping (so running longer probably wouldn't help). I haven't used the \"low res Climsim\" data. I am also interested in seeing what would happen if I did. My model is relatively small, Unet-like architecture with ~1.5M params (it trained on my laptop with a 8GB GPU :) ). My assumption is that if I use more data, I would be able to use a larger model.
", "rawMarkdown": "I used only the data on train.csv, but I have my own scaled dataset saved in tfrecords. I held-off ~5% of data for validation (I shuffled everything beforehand). it ran for like 18 epochs before early-stopping (so running longer probably wouldn't help). I haven't used the \"low res Climsim\" data. I am also interested in seeing what would happen if I did. My model is relatively small, Unet-like architecture with ~1.5M params (it trained on my laptop with a 8GB GPU :) ). My assumption is that if I use more data, I would be able to use a larger model. " } ] } ] }, { "id": 2888727, "postDate": "2024-06-25T03:41:27.340Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2888663, "postDate": "2024-06-25T02:53:27.057Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2871028, "postDate": "2024-06-14T03:03:02.130Z", "rawMarkdown": "", "votes": 3, "isDeleted": true, "replies": [ { "id": 2871508, "postDate": "2024-06-14T08:38:57.233Z", "content": "i was thinking about it lately so i tried to log normalise the variables with skew > 1 first. The problem with my case is that i cannot access if that helped. I have never managed to train something more than 10 epochs, it takes too long but i think i did not notice drastic difference. I noticed that some variables have huge skew like 5-6, this probably means that there are some very insane outliers, i was wandering if it makes a difference to remove them. Unfortunatelly i dont have time to test all these, leaving for vacations -_-
", "rawMarkdown": "i was thinking about it lately so i tried to log normalise the variables with skew > 1 first. The problem with my case is that i cannot access if that helped. I have never managed to train something more than 10 epochs, it takes too long but i think i did not notice drastic difference. I noticed that some variables have huge skew like 5-6, this probably means that there are some very insane outliers, i was wandering if it makes a difference to remove them. Unfortunatelly i dont have time to test all these, leaving for vacations -_-", "votes": 1, "replies": [ { "id": 2871909, "postDate": "2024-06-14T13:08:21.943Z", "content": "In terms of epochs, I usually train my model offline for 5 to 10 epochs, and compare it with my previous model, if the results are promising I train it for more epochs, like 20.
", "rawMarkdown": "In terms of epochs, I usually train my model offline for 5 to 10 epochs, and compare it with my previous model, if the results are promising I train it for more epochs, like 20." } ] }, { "id": 2871742, "postDate": "2024-06-14T11:28:35.433Z", "content": "From what you wrote, I would recommend focusing on your model.
", "rawMarkdown": "From what you wrote, I would recommend focusing on your model.", "replies": [ { "id": 2871746, "postDate": "2024-06-14T11:34:18.950Z", "content": "Thank you for your helpful advice.
", "rawMarkdown": "Thank you for your helpful advice." }, { "id": 2872067, "postDate": "2024-06-14T15:00:53.413Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ] } ] } ], "comments": [ { "id": 2866200, "author_name": "Ethan", "author_url": "", "post_date": "2024-06-11T07:43:54.920000", "content": "I use 625000 samples for validation. Current best single model, cv: 0.766, lb: 0.769.Still have lots of work to do.
", "votes": 9, "replies": [ { "id": 2867712, "author_name": "sroger", "author_url": "", "post_date": "2024-06-12T03:44:25.613000", "content": "Very impressive, do you still see large rooms for improvement for a single model?
", "votes": 0, "replies": [] } ] }, { "id": 2888643, "author_name": "phalanx", "author_url": "", "post_date": "2024-06-25T02:20:02.397000", "content": "single model
\nCV: 0.7844
\nNew LB: 0.7794
\n--- update ---
\nCV: 7864
\nNew LB: 0.782
How many parameters in your model?
", "votes": 1, "replies": [] } ] }, { "id": 2881771, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2024-06-21T00:25:56.913000", "content": "One fold (out of 5), trained on all data, new test data, submitted just minutes ago. CV=0.7630. LB=0.75535. So LB - CV = -0.77%.
\nThis is concerning, because on old test data my LB was almost identical to CV - i think differences between them were always less than 0.1%.
\nDoes this mean that the new test data is materially different from the train data (and from old test data)? From looking at leaderboard it appears that all the scores are way down from the old test data, so it is probably not just me who has this problem.
\nComments?
", "votes": 6, "replies": [ { "id": 2882239, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-21T08:38:52.343000", "content": "Old-new LB diff should be ~0.003. If your old LB was also 0.763 then your difference is too big, did you remember to apply trick to ptend_q0002 indices 12-14? Top scores are 'way down' (more than ~0.003 diff) because most top teams probably just submitted some high-scoring single models instead of the same ensembles they had at old LB. This is at least my case.
", "votes": 2, "replies": [ { "id": 2882548, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2024-06-21T12:09:20.567000", "content": "OK, thanks. Maybe it just me - i'll double-check my submission logic.
", "votes": 0, "replies": [] }, { "id": 2884064, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-06-22T09:29:06.707000", "content": "My CV/LB gap was 0.005 before the update and now it's 0.015.
", "votes": 0, "replies": [] } ] }, { "id": 2884527, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-06-22T15:13:02.527000", "content": "Nice scores, thanks for sharing. \"trained on all data\" with this you mean competition training data (downsampled) or entire low-res data?
", "votes": 0, "replies": [] } ] }, { "id": 2869754, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-06-13T08:45:09.420000", "content": "Validation: 0.7582
\nLB: 0.7534
\nSplit: Single holdout
\nMetric: r2_score(y_true * weights, y_pred * weights)
How did you avoid large negative r2 scores? I am using a simple model, a CNN, and get ~0.61, but sometimes I get large negative scores for some targets. I am new to ML and I spend a lot of time debugging, so I still haven’t tried a more complex model. Do these errors go away with better/larger models? Or is it related to the scaling that I’m using (Notmalized values)? Thanks
", "votes": 0, "replies": [ { "id": 2874031, "author_name": "Rob Freeman", "author_url": "", "post_date": "2024-06-16T01:23:07.320000", "content": "Don’t forget to multiply the predictions by the weights when calculating the r2.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2884531, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-06-22T15:15:04.403000", "content": "Single Model, 7M samples:
\nCV: 0.7697
\nOld LB: 0.772x
\nNew LB: 0.7687
0.70859
\nI am working on Amadeo's model.
\nFirst, I understood every line of the code, and after that, I began to improve it.
\nI think with more epochs (about 20 right now), we can achieve a score of 0.715, maybe 0.72 or even 0.73.
\nBut for a silver or gold medal, we will need a big change in the architecture.
Are you using a 1D cnn (as per Amedeo's notebook)?
\nIf so, you can try transformers, rnns and unets as well!
I am also working on @abiolatti 's notebook, improving the architecture, but cant seem to get past 67 LB… some hint? ;)
", "votes": 1, "replies": [] }, { "id": 2867883, "author_name": "Zhuoqun Li", "author_url": "", "post_date": "2024-06-12T06:05:55.597000", "content": "Try a 1d cnn with resnet or unet architecture
", "votes": 2, "replies": [] }, { "id": 2868256, "author_name": "Federico Peccia", "author_url": "", "post_date": "2024-06-12T09:18:28.647000", "content": "Thanks! I will try it
", "votes": 1, "replies": [] }, { "id": 2868275, "author_name": "Amedeo Biolatti", "author_url": "", "post_date": "2024-06-12T09:31:04.913000", "content": "Try a UNet with at least 1M parameters and train longer
", "votes": 4, "replies": [] }, { "id": 2868565, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-12T13:23:57.403000", "content": "Is a 1D CNN + ResNet like the Amadeo CNN network with additional ResNet layers?
", "votes": 0, "replies": [] }, { "id": 2868572, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-12T13:26:23.953000", "content": "@abiolatti my current network has 6MM parameters, I think it's very complex for a \"small\" gain (0.07) compared to your notebook.
\nDo you have any examples (may be external to kaggle) of UNet applied to a regression problem?
https://www.kaggle.com/code/ishandandekar/u-net-from-scratch-using-keras-and-tensorflow
", "votes": 1, "replies": [] }, { "id": 2869166, "author_name": "Rob Freeman", "author_url": "", "post_date": "2024-06-12T21:56:28.190000", "content": "There is also this notebook shared by @farisalahmdi : https://www.kaggle.com/code/farisalahmdi/mlp-and-unet-baseline which would be easy to adapt
", "votes": 0, "replies": [] }, { "id": 2869420, "author_name": "永遠的三號波", "author_url": "", "post_date": "2024-06-13T05:09:07.023000", "content": "How did you predict ptend_q0001_12-15 and ptend_q0003_12-15? I failed to get a positive R2 score on these targets by improving the architecture. The notebook shared by @konstantinboyko :https://www.kaggle.com/code/konstantinboyko/keras-baseline-seq2seq , increasing the epochs to 100 based on your notebook, still did not solve the problem.
", "votes": 0, "replies": [] }, { "id": 2869661, "author_name": "sroger", "author_url": "", "post_date": "2024-06-13T07:05:17.953000", "content": "For me, the start of q1 and q3 were negative before because I calculated R2 batch-wise instead of across the whole validation set, hence the tip in the main post.
", "votes": 2, "replies": [] }, { "id": 2869895, "author_name": "Jotaro", "author_url": "", "post_date": "2024-06-13T10:41:58.007000", "content": "is U-net enough to beat LB0.7? I am trying it with 2M parameters and 3hrs of training.
", "votes": 0, "replies": [] }, { "id": 2869938, "author_name": "Amedeo Biolatti", "author_url": "", "post_date": "2024-06-13T10:57:38.620000", "content": "I got a UNet with 1.3M params to 0.75, but the training was 24+h
", "votes": 7, "replies": [] }, { "id": 2870373, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-13T14:50:15.677000", "content": "0.75 with 1.3M is incredible.
\nI tried a Unet 1D, 4 encoder-4 decoder, 7MM Parameteres, 20 epochs, 0.67 LB.
I also used UNet baseline. I am surprised at how long the training was. Thank you very much for telling me this information.
", "votes": 0, "replies": [] }, { "id": 2870436, "author_name": "Zhuoqun Li", "author_url": "", "post_date": "2024-06-13T15:21:25.340000", "content": "Hyperparameters may be critical for UNet
", "votes": 0, "replies": [] }, { "id": 2870439, "author_name": "永遠的三號波", "author_url": "", "post_date": "2024-06-13T15:22:10.900000", "content": "Is the graph posted correct? Thanks, I will check my work.
", "votes": 0, "replies": [] }, { "id": 2877192, "author_name": "", "author_url": "", "post_date": "2024-06-18T08:30:57.057000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2869165, "author_name": "Rob Freeman", "author_url": "", "post_date": "2024-06-12T21:50:14.520000", "content": "In comparing with my own R2 graph, I have a much more substantial dip in scores in the 270-280 range regardless of using cnn1d, MLP, or U-Net. I'm still musing on whether this is really related to the architecture or not, though I haven't tried transformer encoder.
", "votes": 1, "replies": [] }, { "id": 2868857, "author_name": "Vasilis", "author_url": "", "post_date": "2024-06-12T17:18:00.547000", "content": "I also use a transformer encoder with 3.6 million parameters and my score is around 0.63 so i am very curious what do i do so wrong. Did you train your model many epochs? i think i have train it for less than 10. I have nhead=8 d_model =256 and 2 encoder layers.
", "votes": 1, "replies": [ { "id": 2869137, "author_name": "sroger", "author_url": "", "post_date": "2024-06-12T21:09:50.983000", "content": "You need more layers, training for longer helps too.
", "votes": 3, "replies": [] } ] }, { "id": 2866763, "author_name": "Amedeo Biolatti", "author_url": "", "post_date": "2024-06-11T13:48:44.793000", "content": "Is the LB score truly a single model or the average of the 5 from CV?
", "votes": 1, "replies": [ { "id": 2866769, "author_name": "sroger", "author_url": "", "post_date": "2024-06-11T13:55:45.617000", "content": "Everything shown above is a single fold. I have not trained other folds yet.
", "votes": 1, "replies": [] } ] }, { "id": 2869505, "author_name": "Yousef", "author_url": "", "post_date": "2024-06-13T06:11:52.540000", "content": "unet
\ncv 0.7387 lb 0.739
\ntraining all data waste only 2 hours
\ncv: 1 flod
\ni trust my cv.
\ni only impove my cv score
\ni try to expand my model and train time.
Can I ask you the number of model parameters that achieved this score
", "votes": 2, "replies": [ { "id": 2869136, "author_name": "sroger", "author_url": "", "post_date": "2024-06-12T21:08:58.617000", "content": "Between 7-8m parameters.
", "votes": 4, "replies": [] } ] }, { "id": 2885508, "author_name": "Deepika A", "author_url": "", "post_date": "2024-06-23T06:17:24.470000", "content": "Very useful analysis and great notebook @sroger
", "votes": 0, "replies": [] }, { "id": 2878107, "author_name": "Juan D C F", "author_url": "", "post_date": "2024-06-18T18:28:37.527000", "content": "\n\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.
\n
What happens if you batch the calculations? Do you get a worse result?
", "votes": 0, "replies": [ { "id": 2883289, "author_name": "sroger", "author_url": "", "post_date": "2024-06-21T19:50:29.037000", "content": "It ruins the score for numerous columns (start of q1-3 after sub mask).
", "votes": 0, "replies": [] } ] }, { "id": 2877573, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-18T13:02:03.333000", "content": "@sroger @motokisatokaggle kaggle for a transformer encoder, is positional encoding necessary?
", "votes": 0, "replies": [] }, { "id": 2877347, "author_name": "Dmitry Uarov", "author_url": "", "post_date": "2024-06-18T10:16:12.020000", "content": "Train only on 1.87M samples. 16 epochs. PyTorch.
\nCV: 0.500 (I'm definitely calculating it incorrectly)
\nLB: 0.679
I am getting something similar. My CV was 0.51 and LB was 0.68, but CV and LB always grow proportionately for me, so at least it serves the purpose of showing the trend.
", "votes": 1, "replies": [ { "id": 2877960, "author_name": "Dmitry Uarov", "author_url": "", "post_date": "2024-06-18T16:54:58.733000", "content": "How many observations did you use to get this result? For me, CV and LB also grow proportionally and this reassure. I'm very interested in what result I will have when increasing the amount of training data by 5 times and number of epochs from 16 to ~50.
", "votes": 0, "replies": [] }, { "id": 2878099, "author_name": "Juan D C F", "author_url": "", "post_date": "2024-06-18T18:21:26.860000", "content": "I used only the data on train.csv, but I have my own scaled dataset saved in tfrecords. I held-off ~5% of data for validation (I shuffled everything beforehand). it ran for like 18 epochs before early-stopping (so running longer probably wouldn't help). I haven't used the \"low res Climsim\" data. I am also interested in seeing what would happen if I did. My model is relatively small, Unet-like architecture with ~1.5M params (it trained on my laptop with a 8GB GPU :) ). My assumption is that if I use more data, I would be able to use a larger model.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2888727, "author_name": "", "author_url": "", "post_date": "2024-06-25T03:41:27.340000", "content": "", "votes": 0, "replies": [] }, { "id": 2888663, "author_name": "", "author_url": "", "post_date": "2024-06-25T02:53:27.057000", "content": "", "votes": 0, "replies": [] }, { "id": 2871028, "author_name": "", "author_url": "", "post_date": "2024-06-14T03:03:02.130000", "content": "", "votes": 3, "replies": [ { "id": 2871508, "author_name": "Vasilis", "author_url": "", "post_date": "2024-06-14T08:38:57.233000", "content": "i was thinking about it lately so i tried to log normalise the variables with skew > 1 first. The problem with my case is that i cannot access if that helped. I have never managed to train something more than 10 epochs, it takes too long but i think i did not notice drastic difference. I noticed that some variables have huge skew like 5-6, this probably means that there are some very insane outliers, i was wandering if it makes a difference to remove them. Unfortunatelly i dont have time to test all these, leaving for vacations -_-
", "votes": 1, "replies": [ { "id": 2871909, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-14T13:08:21.943000", "content": "In terms of epochs, I usually train my model offline for 5 to 10 epochs, and compare it with my previous model, if the results are promising I train it for more epochs, like 20.
", "votes": 0, "replies": [] } ] }, { "id": 2871742, "author_name": "sroger", "author_url": "", "post_date": "2024-06-14T11:28:35.433000", "content": "From what you wrote, I would recommend focusing on your model.
", "votes": 0, "replies": [ { "id": 2871746, "author_name": "Moggle", "author_url": "", "post_date": "2024-06-14T11:34:18.950000", "content": "Thank you for your helpful advice.
", "votes": 0, "replies": [] }, { "id": 2872067, "author_name": "", "author_url": "", "post_date": "2024-06-14T15:00:53.413000", "content": "", "votes": -1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2866146": "This thread has returned as per tradition.\nAn updated continuation of the [CV/LB thread](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/498194) for the latter half of the competition.\n\nI'll start with mine:\nArchitecture: transformer based encoder\nLB: 0.75280\nCV: 0.66317\nCV method: 5 fold (single fold), 0.5m samples, clip R2 to (-1, 1), exclude masked values from sample sub.\nCV curve: (clipped to -0.1, 1)\n\nBTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.\nGood luck everyone.\n\nEdit: New LB (similar model) ~0.761", "2866200": "I use 625000 samples for validation. Current best single model, cv: 0.766, lb: 0.769.Still have lots of work to do.", "2888643": "single model\nCV: 0.7844\nNew LB: 0.7794\n--- update ---\nCV: 7864\nNew LB: 0.782", "2881771": "One fold (out of 5), trained on all data, new test data, submitted just minutes ago. CV=0.7630. LB=0.75535. So LB - CV = -0.77%.\n\nThis is concerning, because on old test data my LB was almost identical to CV - i think differences between them were always less than 0.1%.\n\nDoes this mean that the new test data is materially different from the train data (and from old test data)? From looking at leaderboard it appears that all the scores are way down from the old test data, so it is probably not just me who has this problem.\n\nComments?", "2869754": "Validation: 0.7582\nLB: 0.7534\nSplit: Single holdout\nMetric: r2_score(y_true * weights, y_pred * weights)\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "2884531": "Single Model, **7M** samples: \n**CV**: 0.7697 \n**Old LB**: 0.772x \n**New LB**: 0.7687\n", "2867450": "0.70859\n\nI am working on Amadeo's model. \nFirst, I understood every line of the code, and after that, I began to improve it. \nI think with more epochs (about 20 right now), we can achieve a score of 0.715, maybe 0.72 or even 0.73.\nBut for a silver or gold medal, we will need a big change in the architecture.\n", "2869165": "In comparing with my own R2 graph, I have a much more substantial dip in scores in the 270-280 range regardless of using cnn1d, MLP, or U-Net. I'm still musing on whether this is really related to the architecture or not, though I haven't tried transformer encoder.", "2868857": "I also use a transformer encoder with 3.6 million parameters and my score is around 0.63 so i am very curious what do i do so wrong. Did you train your model many epochs? i think i have train it for less than 10. I have nhead=8 d_model =256 and 2 encoder layers.", "2866763": "Is the LB score truly a single model or the average of the 5 from CV?", "2869505": "unet\ncv 0.7387 lb 0.739 \ntraining all data waste only 2 hours \ncv: 1 flod \ni trust my cv. \ni only impove my cv score\ni try to expand my model and train time.", "2867748": "Can I ask you the number of model parameters that achieved this score", "2885508": "Very useful analysis and great notebook @sroger ", "2878107": "> BTW don't make my mistake, calculate R2 across your validation set instead of batching the calculations.\n\nWhat happens if you batch the calculations? Do you get a worse result?\n\n", "2877573": "@sroger @motokisatokaggle kaggle for a transformer encoder, is positional encoding necessary?", "2877347": "Train only on 1.87M samples. 16 epochs. PyTorch.\nCV: 0.500 (I'm definitely calculating it incorrectly)\nLB: 0.679", "2888727": "", "2888663": "", "2871028": "" } }