Hi,
\nI am fairly new to Kaggle, but LEAP in particular has been my first competition where the public notebooks with the highest LB scores were mainly based around NN models as opposed to other competitions being centered around Gradient Boost models.
My noob question is: How can I improve my skills to build better NN models? e.g. knowing how to create better model architecture, etc.
\nI've had some chance to copy and edit some notebooks and tweak them a little to get slightly better LB, but I'm not sure how people make drastic improvements by reconstructing the architecture, or by any means.
\nSorry if my question is vague; but I'm curious to learn your approach to creating better NN models, especially because I feel like there is not much room for trial and error, given the computational intensity.
", "messages": [ { "id": 2917210, "postDate": "2024-07-11T14:44:45.413Z", "content": "It's mainly this: reading public notebooks, tweaking, etc.
\nBut there is a caveat. Public notebooks published during competitions would not reveal the best tricks and architecture. It will only bring you so far.
\nIf you want to move toward SOTA architectures, you must read previous relevant competition-winning solutions. This is where people reveal the TRULY valuable secrets.
\nIn my case I had the luck, when I participated in ASLFR, to build on the winning solution of ISLR. It was enough for me to reach gold.
\nOnce you get used to SOTA techniques, it's easy to keep going strong.
Wow, thank you SO MUCH!
\nSo I guess it would have been better if I had known of any previous NN competitions.
\nI will look into the ASLFR and ISLR solutions for sure! I'm curious to know how you utilized the previous solution and tailored it for another set of data.
I have another question out of curiosity if you don't mind: is it rare for people to build model architectures from scratch on Kaggle?
\nI am far from having the skills/knowledge/experience to do so yet, but I was wondering if the ability to build from scratch is something one should focus on.
Previous good 1D (or semi-1D) NN competition on massive data in last year: iceCube, ISLR, ASLFR, Ribonanza.
\nISLR and ASLFR were a very special case, since ASLFR was kind of 'ISLR 2' with the same kind of data. So it was easy to tailor the solution. This is why I wrote that I had the luck.
\nDefine 'from scratch'…but I guess it depends on the situation and person.
This can be your \"first nn competition\", I'm sure people will share a lot of insights on how to approach these kind of problems
", "rawMarkdown": "This can be your \"first nn competition\", I'm sure people will share a lot of insights on how to approach these kind of problems", "votes": 2 }, { "id": 2917410, "postDate": "2024-07-11T16:14:33.327Z", "content": "Thank you for the list of competitions from last year! I'll try and learn from them.
\nAlso thanks for the clarification with the ISLR and ASLFR! I had no idea.
My idea of \"from scratch\" stems from this notebook from Chris Deotte: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112
\nIn this notebook he writes:
\n\n\nI'm very proud of this transformer. In Kaggle's Vent Comp, there are many pubic notebooks demonstrating how to build RNNs but there are not public notebooks about transformers. I needed to design this network and tune this network by myself. I am very happy with the results.
\n
Hence I was picturing that maybe I will have to be able to design and tune my own neural network without reference if I want to be confident with my skills in the future.
\nAdditionally, I have always wondered how people decided how deep their model is going to be or how many neurons they want to set up in their hidden layers; is there an optimal set up or do people know this from experience?
", "rawMarkdown": "Thank you for the list of competitions from last year! I'll try and learn from them.\nAlso thanks for the clarification with the ISLR and ASLFR! I had no idea.\n\nMy idea of \"from scratch\" stems from this notebook from Chris Deotte: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112\n\nIn this notebook he writes:\n>I'm very proud of this transformer. In Kaggle's Vent Comp, there are many pubic notebooks demonstrating how to build RNNs but there are not public notebooks about transformers. I needed to design this network and tune this network by myself. I am very happy with the results.\n\nHence I was picturing that maybe I will have to be able to design and tune my own neural network without reference if I want to be confident with my skills in the future.\n\nAdditionally, I have always wondered how people decided how deep their model is going to be or how many neurons they want to set up in their hidden layers; is there an optimal set up or do people know this from experience? \n\n" }, { "id": 2917422, "postDate": "2024-07-11T16:19:40.827Z", "content": "Thank you!
\nIt definitely is my first NN competition, and I think I've gained some insights from all the discussions and notebooks. (While of course losing some resources from training💸🥲)
\nSo far I am overwhelmed by how supportive the community is with generous peers sharing their insights and thoughts :)
Yes you will want do design and fine-tune but it's a skill that develops with time.
\n'how people decide': experiments, intuition, experience and then some more experiments.
Thank you! That makes sense. I'm guessing that experiment is key = there is no single pre-defined answer and everyone is looking for the most optimal solution.
\nI will keep trying! :)
Indeed, ML, and in particular DL, is an experimental field by its nature. I would let you in on some advice: In my first deep learning competition (IceCube), I considered each experiment very carefully and tried to save compute (leading to most of the compute being unutilized since I would get to the end of the week without using the Kaggle compute enough).
\nA year later, in this competition, I made more than 300 experiments, some of them totally crazy, some with very surprising results. It basically goes like this: have some crazy idea->there is no chance it's going to work->try it anyway.
\nThe challenges are in constructing a good, fast, and reliable training and validation pipeline. The faster you can test new ideas, the better your results will be.
\nAlso, if you don't have rtx4090 at home- learn to use TPU.
I've been having a strange relationship with Kaggle compute; sometimes I can't push myself to just try and run my script even if it might not work, so your comment is very reassuring. Thank you.
\nThe amount of experiments you've done is truly what I look up to. Hats off!!
\nI will try to seek for ways to cycle through my validation process quickly since I have not yet considered that aspect yet.
To be honest, I am still in the stage where I don't know if the experiment is \"crazy\" or not. I'm just trying things out to see how it performs.
\nCould you give an example of a \"crazy\" experiment? ( I am sorry if I am bombarding you with questions.. there are just so many to ask T_T )
p.s. I will be looking into TPU, or maybe getting an rtx4090 and ditch my MacBook air. lol
", "rawMarkdown": "I've been having a strange relationship with Kaggle compute; sometimes I can't push myself to just try and run my script even if it might not work, so your comment is very reassuring. Thank you.\n\nThe amount of experiments you've done is truly what I look up to. Hats off!! \nI will try to seek for ways to cycle through my validation process quickly since I have not yet considered that aspect yet.\n\nTo be honest, I am still in the stage where I don't know if the experiment is \"crazy\" or not. I'm just trying things out to see how it performs. \nCould you give an example of a \"crazy\" experiment? ( I am sorry if I am bombarding you with questions.. there are just so many to ask T_T )\n\np.s. I will be looking into TPU, or maybe getting an rtx4090 and ditch my MacBook air. lol " }, { "id": 2917627, "postDate": "2024-07-11T17:44:54.947Z", "content": "I will give examples of crazy when I publish my solution to this competition 😛 (of course, most of 'crazy' don't work eventually lol. But even if one works it can be the game changer)
", "rawMarkdown": "I will give examples of crazy when I publish my solution to this competition 😛 (of course, most of 'crazy' don't work eventually lol. But even if one works it can be the game changer)" }, { "id": 2917654, "postDate": "2024-07-11T17:55:23.747Z", "content": "Thank you! I'm looking forward to seeing how you managed through this complicated (at least to my eyes) data. I'm not sure when the sweet solution is coming to the surface given the situation with leakage, but I will have hope :)
", "rawMarkdown": "Thank you! I'm looking forward to seeing how you managed through this complicated (at least to my eyes) data. I'm not sure when the sweet solution is coming to the surface given the situation with leakage, but I will have hope :) " }, { "id": 2917893, "postDate": "2024-07-11T21:00:55.003Z", "content": "One advice I have is to try to build your own model from scratch instead of copying from a public notebook and making changes to it. They may have a good score at times, but are often riddled with bugs or are not clean at all. And often you are not quite sure why they did the things (and often rightfully so). In one code competition, hundreds of participants were dropped from the private LB on a data-rerun because one popular public notebook had a bug which caused the notebook to crash.
\nI usually like to build my own model first, then look at the public notebook. If my model is worse than the public notebook I ask myself why that could be, or what the public notebook does better. And then I try to address these things in my model.
Thank you!
\n\n\nIn one code competition, hundreds of participants were dropped from the private LB on a data-rerun because one popular public notebook had a bug which caused the notebook to crash.
\n
That sounds devastating. :(
\nI'll start to try and work on building my own model!
\nIt sure won't be good from the get-go but hopefully my model building skills will improve with time and experience.
Thanks again! :)
", "rawMarkdown": "Thank you! \n> In one code competition, hundreds of participants were dropped from the private LB on a data-rerun because one popular public notebook had a bug which caused the notebook to crash.\n\nThat sounds devastating. :( \nI'll start to try and work on building my own model! \nIt sure won't be good from the get-go but hopefully my model building skills will improve with time and experience.\n\nThanks again! :)" } ] } ] }, { "id": 2917196, "postDate": "2024-07-11T14:35:07.543Z", "content": "Hi,
\nI am fairly new to Kaggle, but LEAP in particular has been my first competition where the public notebooks with the highest LB scores were mainly based around NN models as opposed to other competitions being centered around Gradient Boost models.
My noob question is: How can I improve my skills to build better NN models? e.g. knowing how to create better model architecture, etc.
\nI've had some chance to copy and edit some notebooks and tweak them a little to get slightly better LB, but I'm not sure how people make drastic improvements by reconstructing the architecture, or by any means.
\nSorry if my question is vague; but I'm curious to learn your approach to creating better NN models, especially because I feel like there is not much room for trial and error, given the computational intensity.
", "rawMarkdown": "Hi, \nI am fairly new to Kaggle, but LEAP in particular has been my first competition where the public notebooks with the highest LB scores were mainly based around NN models as opposed to other competitions being centered around Gradient Boost models.\n\nMy noob question is: How can I improve my skills to build better NN models? e.g. knowing how to create better model architecture, etc.\n\nI've had some chance to copy and edit some notebooks and tweak them a little to get slightly better LB, but I'm not sure how people make drastic improvements by reconstructing the architecture, or by any means.\n\nSorry if my question is vague; but I'm curious to learn your approach to creating better NN models, especially because I feel like there is not much room for trial and error, given the computational intensity.", "votes": 5 } ], "comments": [ { "id": 2917210, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:44:45.413000", "content": "It's mainly this: reading public notebooks, tweaking, etc.
\nBut there is a caveat. Public notebooks published during competitions would not reveal the best tricks and architecture. It will only bring you so far.
\nIf you want to move toward SOTA architectures, you must read previous relevant competition-winning solutions. This is where people reveal the TRULY valuable secrets.
\nIn my case I had the luck, when I participated in ASLFR, to build on the winning solution of ISLR. It was enough for me to reach gold.
\nOnce you get used to SOTA techniques, it's easy to keep going strong.
Wow, thank you SO MUCH!
\nSo I guess it would have been better if I had known of any previous NN competitions.
\nI will look into the ASLFR and ISLR solutions for sure! I'm curious to know how you utilized the previous solution and tailored it for another set of data.
I have another question out of curiosity if you don't mind: is it rare for people to build model architectures from scratch on Kaggle?
\nI am far from having the skills/knowledge/experience to do so yet, but I was wondering if the ability to build from scratch is something one should focus on.
Previous good 1D (or semi-1D) NN competition on massive data in last year: iceCube, ISLR, ASLFR, Ribonanza.
\nISLR and ASLFR were a very special case, since ASLFR was kind of 'ISLR 2' with the same kind of data. So it was easy to tailor the solution. This is why I wrote that I had the luck.
\nDefine 'from scratch'…but I guess it depends on the situation and person.
This can be your \"first nn competition\", I'm sure people will share a lot of insights on how to approach these kind of problems
", "votes": 2, "replies": [] }, { "id": 2917410, "author_name": "professional food critic", "author_url": "", "post_date": "2024-07-11T16:14:33.327000", "content": "Thank you for the list of competitions from last year! I'll try and learn from them.
\nAlso thanks for the clarification with the ISLR and ASLFR! I had no idea.
My idea of \"from scratch\" stems from this notebook from Chris Deotte: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112
\nIn this notebook he writes:
\n\n\nI'm very proud of this transformer. In Kaggle's Vent Comp, there are many pubic notebooks demonstrating how to build RNNs but there are not public notebooks about transformers. I needed to design this network and tune this network by myself. I am very happy with the results.
\n
Hence I was picturing that maybe I will have to be able to design and tune my own neural network without reference if I want to be confident with my skills in the future.
\nAdditionally, I have always wondered how people decided how deep their model is going to be or how many neurons they want to set up in their hidden layers; is there an optimal set up or do people know this from experience?
", "votes": 0, "replies": [] }, { "id": 2917422, "author_name": "professional food critic", "author_url": "", "post_date": "2024-07-11T16:19:40.827000", "content": "Thank you!
\nIt definitely is my first NN competition, and I think I've gained some insights from all the discussions and notebooks. (While of course losing some resources from training💸🥲)
\nSo far I am overwhelmed by how supportive the community is with generous peers sharing their insights and thoughts :)
Yes you will want do design and fine-tune but it's a skill that develops with time.
\n'how people decide': experiments, intuition, experience and then some more experiments.
Thank you! That makes sense. I'm guessing that experiment is key = there is no single pre-defined answer and everyone is looking for the most optimal solution.
\nI will keep trying! :)
Indeed, ML, and in particular DL, is an experimental field by its nature. I would let you in on some advice: In my first deep learning competition (IceCube), I considered each experiment very carefully and tried to save compute (leading to most of the compute being unutilized since I would get to the end of the week without using the Kaggle compute enough).
\nA year later, in this competition, I made more than 300 experiments, some of them totally crazy, some with very surprising results. It basically goes like this: have some crazy idea->there is no chance it's going to work->try it anyway.
\nThe challenges are in constructing a good, fast, and reliable training and validation pipeline. The faster you can test new ideas, the better your results will be.
\nAlso, if you don't have rtx4090 at home- learn to use TPU.
I've been having a strange relationship with Kaggle compute; sometimes I can't push myself to just try and run my script even if it might not work, so your comment is very reassuring. Thank you.
\nThe amount of experiments you've done is truly what I look up to. Hats off!!
\nI will try to seek for ways to cycle through my validation process quickly since I have not yet considered that aspect yet.
To be honest, I am still in the stage where I don't know if the experiment is \"crazy\" or not. I'm just trying things out to see how it performs.
\nCould you give an example of a \"crazy\" experiment? ( I am sorry if I am bombarding you with questions.. there are just so many to ask T_T )
p.s. I will be looking into TPU, or maybe getting an rtx4090 and ditch my MacBook air. lol
", "votes": 0, "replies": [] }, { "id": 2917627, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T17:44:54.947000", "content": "I will give examples of crazy when I publish my solution to this competition 😛 (of course, most of 'crazy' don't work eventually lol. But even if one works it can be the game changer)
", "votes": 0, "replies": [] }, { "id": 2917654, "author_name": "professional food critic", "author_url": "", "post_date": "2024-07-11T17:55:23.747000", "content": "Thank you! I'm looking forward to seeing how you managed through this complicated (at least to my eyes) data. I'm not sure when the sweet solution is coming to the surface given the situation with leakage, but I will have hope :)
", "votes": 0, "replies": [] }, { "id": 2917893, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-11T21:00:55.003000", "content": "One advice I have is to try to build your own model from scratch instead of copying from a public notebook and making changes to it. They may have a good score at times, but are often riddled with bugs or are not clean at all. And often you are not quite sure why they did the things (and often rightfully so). In one code competition, hundreds of participants were dropped from the private LB on a data-rerun because one popular public notebook had a bug which caused the notebook to crash.
\nI usually like to build my own model first, then look at the public notebook. If my model is worse than the public notebook I ask myself why that could be, or what the public notebook does better. And then I try to address these things in my model.
Thank you!
\n\n\nIn one code competition, hundreds of participants were dropped from the private LB on a data-rerun because one popular public notebook had a bug which caused the notebook to crash.
\n
That sounds devastating. :(
\nI'll start to try and work on building my own model!
\nIt sure won't be good from the get-go but hopefully my model building skills will improve with time and experience.
Thanks again! :)
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2917210": "It's mainly this: reading public notebooks, tweaking, etc.\nBut there is a caveat. Public notebooks published during competitions would not reveal the best tricks and architecture. It will only bring you so far.\nIf you want to move toward SOTA architectures, you must read previous relevant competition-winning solutions. This is where people reveal the TRULY valuable secrets.\nIn my case I had the luck, when I participated in ASLFR, to build on the winning solution of ISLR. It was enough for me to reach gold.\nOnce you get used to SOTA techniques, it's easy to keep going strong.", "2917196": "Hi, \nI am fairly new to Kaggle, but LEAP in particular has been my first competition where the public notebooks with the highest LB scores were mainly based around NN models as opposed to other competitions being centered around Gradient Boost models.\n\nMy noob question is: How can I improve my skills to build better NN models? e.g. knowing how to create better model architecture, etc.\n\nI've had some chance to copy and edit some notebooks and tweak them a little to get slightly better LB, but I'm not sure how people make drastic improvements by reconstructing the architecture, or by any means.\n\nSorry if my question is vague; but I'm curious to learn your approach to creating better NN models, especially because I feel like there is not much room for trial and error, given the computational intensity." } }