![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16113%2F41cc2be6401b98616bc3a66f9c4b4a17%2Fk1.JPG?generation=1590331883968756&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16113%2F497848916e3a3bf4413bc4828881484d%2Fk2.JPG?generation=1590331910158875&alt=media\")
Hi everyone,
\n\nI was wondering what your favourite ways to quickly test feasible ideas are. Two questions come to mind.
\n\n1) Do you have any favourite ways to test ideas that might improve your models without training on the full dataset?
\n\n2) An extension to this would be how do you test ideas when CV and LB are not correlated?
", "messages": [ { "id": 857765, "postDate": "2020-05-22T22:38:22.337Z", "content": "Hi everyone,
\n\nI was wondering what your favourite ways to quickly test feasible ideas are. Two questions come to mind.
\n\n1) Do you have any favourite ways to test ideas that might improve your models without training on the full dataset?
\n\n2) An extension to this would be how do you test ideas when CV and LB are not correlated?
", "rawMarkdown": "Hi everyone,\n\nI was wondering what your favourite ways to quickly test feasible ideas are. Two questions come to mind.\n\n1) Do you have any favourite ways to test ideas that might improve your models without training on the full dataset?\n\n2) An extension to this would be how do you test ideas when CV and LB are not correlated?", "votes": 4 }, { "id": 859712, "postDate": "2020-05-24T16:59:42.923Z", "content": "This is a very good question and I wonder how people do it at the top of the leaderboard. I'm stuck at 0.85LB for a while and very few of my ideas end up in an improvement. I found no good way quickly experiment as I need really quite a bit of training to see if there is an improvement. For what its worth here is what I do.
\n\n(all my training is locally on a single 2080Ti). If money would be no blocker I would use a bunch of V100s on AWS to run several experiments in parallel to accelerate things.
\n\n(Edit: I just made a submission that I was sure was going to be my best and hit 0.86 at least, and I got 0.83.... and that was after spending 24 hours of training....)
", "rawMarkdown": "This is a very good question and I wonder how people do it at the top of the leaderboard. I'm stuck at 0.85LB for a while and very few of my ideas end up in an improvement. I found no good way quickly experiment as I need really quite a bit of training to see if there is an improvement. For what its worth here is what I do.\n\n- quick (10 epoch) run on smaller # of tiles, just to see if things more more or less work (converge) and end up 0.70+CV. This takes an hour or less.\n- then I train 1 fold with full data for several hours to see if my CV gets into ballpark. (~0.86). \n- Then validate by submitting to get an LB \n- Then 4 fold training overnight and submit again. \n\n(all my training is locally on a single 2080Ti). If money would be no blocker I would use a bunch of V100s on AWS to run several experiments in parallel to accelerate things.\n\n(Edit: I just made a submission that I was sure was going to be my best and hit 0.86 at least, and I got 0.83.... and that was after spending 24 hours of training....)\n", "votes": 1 }, { "id": 859571, "postDate": "2020-05-24T14:51:04.547Z", "content": "2.) I push the prototype to my gpu-enabled cloud to do N=1600 4-fold cross val. (!1hr all folds to train) and see if modelX performs differently than modelY. I have prewritten graph to visualize differences [0].
\n\n3.) I push succesfully validated experiments to kaggle to do \"full runs\" on all the data. I like to use the kaggle api here where I can simply do kaggle k push and have my nb run, ```kaggle k output stillsut/
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F16113%2F2e8c580afbc556e4d26f3a4a37cdd7ca%2Fk3.JPG?generation=1590331920461835&alt=media\")
In my case, I always take some time to do some basic data analysis, and prepare some visualization. Usually i can give me some ideas on algorithms I want to design.\nThen everytime I select like a very very small sample set. In case of neural network,I even I start to try to overfit one batch, and it help me to change the architecture, then I start to add data.
\n\nIndeed when I add data I will have to change hyper-parameters and then I do it iteratively when its just to get quick ideas. other wise I run a fill experimental design and change parameters and analyze the results to find appropriate hyperparameters.
", "rawMarkdown": "In my case, I always take some time to do some basic data analysis, and prepare some visualization. Usually i can give me some ideas on algorithms I want to design.\nThen everytime I select like a very very small sample set. In case of neural network,I even I start to try to overfit one batch, and it help me to change the architecture, then I start to add data.\n\nIndeed when I add data I will have to change hyper-parameters and then I do it iteratively when its just to get quick ideas. other wise I run a fill experimental design and change parameters and analyze the results to find appropriate hyperparameters." }, { "id": 858939, "postDate": "2020-05-24T00:54:24.753Z", "content": "I'm not expert, but my two perfered methods for quickly testing ideas are taking a sample of the dataset (2k images are typically enough for basic testing as long as there aren't an obsene number of classes) and another method is to use smaller images. Typically you will need some slight tweaks to get the same improvements from those tests on the full unmodified set.
", "rawMarkdown": "I'm not expert, but my two perfered methods for quickly testing ideas are taking a sample of the dataset (2k images are typically enough for basic testing as long as there aren't an obsene number of classes) and another method is to use smaller images. Typically you will need some slight tweaks to get the same improvements from those tests on the full unmodified set." } ], "comments": [ { "id": 859712, "author_name": "Peter Cnudde", "author_url": "", "post_date": "2020-05-24T16:59:42.923000", "content": "This is a very good question and I wonder how people do it at the top of the leaderboard. I'm stuck at 0.85LB for a while and very few of my ideas end up in an improvement. I found no good way quickly experiment as I need really quite a bit of training to see if there is an improvement. For what its worth here is what I do.
\n\n(all my training is locally on a single 2080Ti). If money would be no blocker I would use a bunch of V100s on AWS to run several experiments in parallel to accelerate things.
\n\n(Edit: I just made a submission that I was sure was going to be my best and hit 0.86 at least, and I got 0.83.... and that was after spending 24 hours of training....)
", "votes": 1, "replies": [] }, { "id": 859571, "author_name": "S.U.T.", "author_url": "", "post_date": "2020-05-24T14:51:04.547000", "content": "2.) I push the prototype to my gpu-enabled cloud to do N=1600 4-fold cross val. (!1hr all folds to train) and see if modelX performs differently than modelY. I have prewritten graph to visualize differences [0].
\n\n3.) I push succesfully validated experiments to kaggle to do \"full runs\" on all the data. I like to use the kaggle api here where I can simply do kaggle k push and have my nb run, ```kaggle k output stillsut/
\n
In my case, I always take some time to do some basic data analysis, and prepare some visualization. Usually i can give me some ideas on algorithms I want to design.\nThen everytime I select like a very very small sample set. In case of neural network,I even I start to try to overfit one batch, and it help me to change the architecture, then I start to add data.
\n\nIndeed when I add data I will have to change hyper-parameters and then I do it iteratively when its just to get quick ideas. other wise I run a fill experimental design and change parameters and analyze the results to find appropriate hyperparameters.
", "votes": 0, "replies": [] }, { "id": 858939, "author_name": "Neil Kloper", "author_url": "", "post_date": "2020-05-24T00:54:24.753000", "content": "I'm not expert, but my two perfered methods for quickly testing ideas are taking a sample of the dataset (2k images are typically enough for basic testing as long as there aren't an obsene number of classes) and another method is to use smaller images. Typically you will need some slight tweaks to get the same improvements from those tests on the full unmodified set.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "857765": "Hi everyone,\n\nI was wondering what your favourite ways to quickly test feasible ideas are. Two questions come to mind.\n\n1) Do you have any favourite ways to test ideas that might improve your models without training on the full dataset?\n\n2) An extension to this would be how do you test ideas when CV and LB are not correlated?", "859712": "This is a very good question and I wonder how people do it at the top of the leaderboard. I'm stuck at 0.85LB for a while and very few of my ideas end up in an improvement. I found no good way quickly experiment as I need really quite a bit of training to see if there is an improvement. For what its worth here is what I do.\n\n- quick (10 epoch) run on smaller # of tiles, just to see if things more more or less work (converge) and end up 0.70+CV. This takes an hour or less.\n- then I train 1 fold with full data for several hours to see if my CV gets into ballpark. (~0.86). \n- Then validate by submitting to get an LB \n- Then 4 fold training overnight and submit again. \n\n(all my training is locally on a single 2080Ti). If money would be no blocker I would use a bunch of V100s on AWS to run several experiments in parallel to accelerate things.\n\n(Edit: I just made a submission that I was sure was going to be my best and hit 0.86 at least, and I got 0.83.... and that was after spending 24 hours of training....)\n", "859571": "1. I prototype locally (no gpu) on very small dataset (epoch trains in ~10 seconds, N=22, stratified ). My goal here is: a.) my arch/code actually works, b.) I can run (vscode) debugger to understand exactly what's going on in my libraries.\n\n2.) I push the prototype to my gpu-enabled cloud to do N=1600 4-fold cross val. (!1hr all folds to train) and see if modelX performs differently than modelY. I have prewritten graph to visualize differences [0]. \n\n3.) I push succesfully validated experiments to kaggle to do \"full runs\" on all the data. I like to use the kaggle api here where I can simply do ```kaggle k push``` and have my nb run, ```kaggle k output stillsut/\n\n\n\n\n\n", "859222": "In my case, I always take some time to do some basic data analysis, and prepare some visualization. Usually i can give me some ideas on algorithms I want to design.\nThen everytime I select like a very very small sample set. In case of neural network,I even I start to try to overfit one batch, and it help me to change the architecture, then I start to add data.\n\nIndeed when I add data I will have to change hyper-parameters and then I do it iteratively when its just to get quick ideas. other wise I run a fill experimental design and change parameters and analyze the results to find appropriate hyperparameters.", "858939": "I'm not expert, but my two perfered methods for quickly testing ideas are taking a sample of the dataset (2k images are typically enough for basic testing as long as there aren't an obsene number of classes) and another method is to use smaller images. Typically you will need some slight tweaks to get the same improvements from those tests on the full unmodified set." } }