Hi all,
\nI struggled for a long time to train an Effnet on all the data (Ian Pan's Jpeg dataset) in pytorch. I used the GPUs on kaggle and got it down to 45mins/epoch but thought this was still far too long. I then tried using the TPUs and got it running but the times were the same so I think my bottleneck was the CPU.
\nBut reading in detail through @khyeh0719 training notebooks I see his training took many many GPU hours too and I think he went to GCP to train some parts.
\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?
\nor is everyone using TF and/or TPUs and its just me struggling to get pytorch working that is the problem? [feeling it is probably<---]
", "messages": [ { "id": 1057411, "postDate": "2020-10-22T16:06:29.850Z", "content": "Hi all,
\nI struggled for a long time to train an Effnet on all the data (Ian Pan's Jpeg dataset) in pytorch. I used the GPUs on kaggle and got it down to 45mins/epoch but thought this was still far too long. I then tried using the TPUs and got it running but the times were the same so I think my bottleneck was the CPU.
\nBut reading in detail through @khyeh0719 training notebooks I see his training took many many GPU hours too and I think he went to GCP to train some parts.
\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?
\nor is everyone using TF and/or TPUs and its just me struggling to get pytorch working that is the problem? [feeling it is probably<---]
", "rawMarkdown": "Hi all,\n\nI struggled for a long time to train an Effnet on all the data (Ian Pan's Jpeg dataset) in pytorch. I used the GPUs on kaggle and got it down to 45mins/epoch but thought this was still far too long. I then tried using the TPUs and got it running but the times were the same so I think my bottleneck was the CPU.\n\nBut reading in detail through @khyeh0719 training notebooks I see his training took many many GPU hours too and I think he went to GCP to train some parts.\n\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?\n\nor is everyone using TF and/or TPUs and its just me struggling to get pytorch working that is the problem? [feeling it is probably<---]", "votes": 2 }, { "id": 1057625, "postDate": "2020-10-22T19:58:12.273Z", "content": "\n\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?
\n
You can still use Colab Pro, it's way much cheaper than GCP; That's what I do whenever I need extensive resources
\nAnd for GPU, colab Pro has now Tesla V100-sxm2 option, which is way faster than kaggle P100
Yeah High RAM.
\nOnce I get P100, I keep retinitializing the notebook, until I get V100 ^^
", "rawMarkdown": "Yeah High RAM.\n\nOnce I get P100, I keep retinitializing the notebook, until I get V100 ^^", "votes": 1 }, { "id": 1057705, "postDate": "2020-10-22T22:38:15.227Z", "content": "Colab pro is only available in US, so that option doesn't work for many of us. Is a great advantage for those who can use that opportunity. Just to wait and hope for a release in Europe.
", "rawMarkdown": "Colab pro is only available in US, so that option doesn't work for many of us. Is a great advantage for those who can use that opportunity. Just to wait and hope for a release in Europe." }, { "id": 1057951, "postDate": "2020-10-23T07:20:14.053Z", "content": "Yeah this is kind of my point - if you can only do well in this competition by using external resources (own GPUs, Colab Pro etc) that either cost money or are only available to USA participants is it really a fair competition? Shouldn't it be about knowledge rather than availability of resources?
", "rawMarkdown": "Yeah this is kind of my point - if you can only do well in this competition by using external resources (own GPUs, Colab Pro etc) that either cost money or are only available to USA participants is it really a fair competition? Shouldn't it be about knowledge rather than availability of resources?", "votes": 1 }, { "id": 1057978, "postDate": "2020-10-23T08:09:35.413Z", "content": "It is quite common that $ prize/ gold medal winners trained their models entirely by Kaggle Kernel/ Colab. Hardware can make a difference but more important is your knowledge, experience and \"luck\".
\nI think as long as there is no un-disclosed external data source and private sharing, the competition is considered as fair.
\nWill become a philosophy question if drill too deep about \"fairness\". If both Tom and Jerry are joining this competition but Tom is a machine learning PhD while Jerry is just a chemistry graduate, is this fair even they have the same hardware?
", "rawMarkdown": "It is quite common that $ prize/ gold medal winners trained their models entirely by Kaggle Kernel/ Colab. Hardware can make a difference but more important is your knowledge, experience and \"luck\".\n\nI think as long as there is no un-disclosed external data source and private sharing, the competition is considered as fair. \n\nWill become a philosophy question if drill too deep about \"fairness\". If both Tom and Jerry are joining this competition but Tom is a machine learning PhD while Jerry is just a chemistry graduate, is this fair even they have the same hardware?\n", "votes": 1 }, { "id": 1058031, "postDate": "2020-10-23T09:09:57.507Z", "content": "One can make it simple, in general, and to not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering.
", "rawMarkdown": "One can make it simple, in general, and to not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering." }, { "id": 1058067, "postDate": "2020-10-23T09:53:35.093Z", "content": "\n\nShouldn't it be about knowledge rather than availability of resources?
\n
And why not both ?
\nAs far as i'm concerned, I don't have even own single GPU and Colab Pro is available everywhere not only the USA (I suscribed from Paris in the early days).
\nAs for fairness, it's quite relative . Because you can also say even Kaggle itself is not fair, as it gives +30 hours of GPU/TPU to each team member for people teaming up.
\nAnyway this competition requires extensive resources by its nature and by the size of the dataset. You can't help but finding solutions like teaming up, using online services or buying your own solid infrastructure. Otherwise there are still other competitions that require much less resources
", "rawMarkdown": "> Shouldn't it be about knowledge rather than availability of resources?\n\nAnd why not both ? \n\nAs far as i'm concerned, I don't have even own single GPU and Colab Pro is available everywhere not only the USA (I suscribed from Paris in the early days). \n\nAs for fairness, it's quite relative . Because you can also say even Kaggle itself is not fair, as it gives +30 hours of GPU/TPU to each team member for people teaming up. \n\nAnyway this competition requires extensive resources by its nature and by the size of the dataset. You can't help but finding solutions like teaming up, using online services or buying your own solid infrastructure. Otherwise there are still other competitions that require much less resources", "votes": -1 }, { "id": 1058071, "postDate": "2020-10-23T10:00:31.127Z", "content": "\n\nOne can make it simple, in general, and do not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering.
\n
I propose we all use 2 cores CPU only and build every thing from scratch. Because, even using Pytorch instead of writing own CNN and Backpropagation algorithms, give some advantages.
", "rawMarkdown": "> One can make it simple, in general, and do not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering.\n\nI propose we all use 2 cores CPU only and build every thing from scratch. Because, even using Pytorch instead of writing own CNN and Backpropagation algorithms, give some advantages. \n" }, { "id": 1058111, "postDate": "2020-10-23T10:59:28.167Z", "content": "I thought we were talking about hw. I can only talk for my self, if I had the possible to use more TPU and GPU time through Colab Pro (V100 is really a good option that they have added, using Volta+mixed precision š) I would use it to help tuning and experiment more and wider, but unfortunately it says only available in US and Canada. Would it help me make a better model or getting a better scores? Who knows, that's more a factor of knowledge but I get that knowledge through testing and experimenting, thats why one loves Kaggle and Colab, resources and challenges helping one getting better in different AI areas through helping and contribute solving a problem and with the goal to help and contribute solving problems even better :)
\nI hope they make Colab Pro more available, that's my point.
As I said, Colab Pro is available outside of these countries. Don't mind what is written at the homepage.
\nI'm just giving advices of the most affordable resources to common kagglers in such highly demanding competition. You can be sure many people at (top) LB use much more extensive resources.
\nIt's (nearly) impossible to win such competition by using Kaggle only, just like the annual Youtube-8m video competition (+ 1.5 TB of data last year) and many others google competitions.
\nThat's why kaggle (and sponsors) offer GCP credits on most of these competitions, in particular for those who can't afford high computational power
Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources.
", "rawMarkdown": "Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources. \n\n", "votes": 1 }, { "id": 1059084, "postDate": "2020-10-24T15:40:51.823Z", "content": "Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources.
", "rawMarkdown": "Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources. \n\n" }, { "id": 1059176, "postDate": "2020-10-24T18:04:53.123Z", "content": "Yes and it's a great advice Serigne š So you think it's OK to use outside US and Canada? I have to look into it again then, maybe something has changed :)
\nAnd I'm also greatfull that we can use Kaggle and Colab, opens the door for everyone to learn and contribute. So I don't want to sound ungrateful but it's just that the Colab Pro seems like a great subscribe service and open up for even more powerful testing and training for everyone. They have now added another country to the list, so maybe more will come soon if it is not possible to use.
You can try rapid from nvidia for using gpu power while matrix calculations and preporcessing.
", "rawMarkdown": "You can try rapid from nvidia for using gpu power while matrix calculations and preporcessing." }, { "id": 1057436, "postDate": "2020-10-22T16:32:39.863Z", "content": "We were silver before the public notebook (30th or so). We used Efficientnetb0 trained on tfrecords and TPU, and had an epoch time of about 5 minutes with 5 fold, which was spectacular. Did everything on TPU, but at this point, I think it would be outstandingly difficult to beat the 0.233 baseline with only Kaggle resources. The baseline has a lot to tune, but without insane resources, it is ridiculously difficult to improve it (my teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch). Try out Colab if you're running out of GPU hours though.
", "rawMarkdown": "We *were* silver before the public notebook (30th or so). We used Efficientnetb0 trained on tfrecords and TPU, and had an epoch time of about 5 minutes with 5 fold, which was spectacular. Did everything on TPU, but at this point, I think it would be outstandingly difficult to beat the 0.233 baseline with only Kaggle resources. The baseline has a lot to tune, but without insane resources, it is ridiculously difficult to improve it (my teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch). Try out Colab if you're running out of GPU hours though.", "replies": [ { "id": 1057456, "postDate": "2020-10-22T16:52:56.187Z", "content": "Wow. Great work Stanley. Iām sure pytorch must be able to get close to those speeds on TPU right? Or are TFrecords really so much faster?
", "rawMarkdown": "Wow. Great work Stanley. Iām sure pytorch must be able to get close to those speeds on TPU right? Or are TFrecords really so much faster?", "votes": 1 }, { "id": 1057468, "postDate": "2020-10-22T17:07:52.020Z", "content": "I'm not sure, I really haven't done an insanely large competition like this with pytorch. I believe that they are quite a bit faster than pytorch, since I/O is decreased (reading 30 2gb files) instead of thousands of 1mb files. Preprocessing and stratified validation is already completed, so any CPU bottlenecks are minimized also. All the CPU needs to do is split the tfrecords into groups and read them. As far as I know, Kaggle TPU's are mainly CPU bottlenecked, since feeding 128gb of RAM is quite intensive.
", "rawMarkdown": "I'm not sure, I really haven't done an insanely large competition like this with pytorch. I believe that they are quite a bit faster than pytorch, since I/O is decreased (reading 30 2gb files) instead of thousands of 1mb files. Preprocessing and stratified validation is already completed, so any CPU bottlenecks are minimized also. All the CPU needs to do is split the tfrecords into groups and read them. As far as I know, Kaggle TPU's are mainly CPU bottlenecked, since feeding 128gb of RAM is quite intensive. ", "votes": 1 }, { "id": 1057573, "postDate": "2020-10-22T19:08:31.730Z", "content": "Have you used one stage, or two stage model?
", "rawMarkdown": "Have you used one stage, or two stage model?", "votes": 1 }, { "id": 1057579, "postDate": "2020-10-22T19:13:57.103Z", "content": "This was only with regular CNN trained on image and exam level - stacking with GRU/LSTM would take longer, and separating them would reduce individual training time slightly. The big limitation here is the max TPU time of 3 hours, so even if you had 5 minutes per epoch, you still only get 36 epochs, so 7 epochs for 5 fold, which is not very much. Better to just do each fold separately
", "rawMarkdown": "This was only with regular CNN trained on image and exam level - stacking with GRU/LSTM would take longer, and separating them would reduce individual training time slightly. The big limitation here is the max TPU time of 3 hours, so even if you had 5 minutes per epoch, you still only get 36 epochs, so 7 epochs for 5 fold, which is not very much. Better to just do each fold separately", "votes": 2, "replies": [ { "id": 1057615, "postDate": "2020-10-22T19:52:08.747Z", "content": "I have never used Kaggle TPU for pytorch. Colab Pro has much better config for Pytorch. And of course no time limitation.
\nI am training on TPU with a non effnet based model but bigger than b0, b1, b2.
\nBut I think it's a big mistake. I won't likely be able to do all inference in 9 hours.
\n\n\nmy teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch
\n
TPU is the way to go on colab Pro in this case. As I said , I'm training a bigger model than b0 with less than 2 hour per epoch on all the data and heavy augmentations.
\nBig batch size shared between 8 TPU cores and you have more ram/CPU/disk space on TPU than GPU
I agree with your opinion about TPU -- 3 hours is too little for such a big task.
", "rawMarkdown": "I agree with your opinion about TPU -- 3 hours is too little for such a big task.", "votes": 1 }, { "id": 1057629, "postDate": "2020-10-22T20:01:10.853Z", "content": "Have you tried stacking with LGBM trained on meta features?
", "rawMarkdown": "Have you tried stacking with LGBM trained on meta features?", "votes": 1 }, { "id": 1058557, "postDate": "2020-10-23T21:32:19.370Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1057625, "author_name": "Serigne ", "author_url": "", "post_date": "2020-10-22T19:58:12.273000", "content": "\n\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?
\n
You can still use Colab Pro, it's way much cheaper than GCP; That's what I do whenever I need extensive resources
\nAnd for GPU, colab Pro has now Tesla V100-sxm2 option, which is way faster than kaggle P100
Yeah High RAM.
\nOnce I get P100, I keep retinitializing the notebook, until I get V100 ^^
", "votes": 1, "replies": [] }, { "id": 1057705, "author_name": "Kirderf", "author_url": "", "post_date": "2020-10-22T22:38:15.227000", "content": "Colab pro is only available in US, so that option doesn't work for many of us. Is a great advantage for those who can use that opportunity. Just to wait and hope for a release in Europe.
", "votes": 0, "replies": [] }, { "id": 1057951, "author_name": "Mark P", "author_url": "", "post_date": "2020-10-23T07:20:14.053000", "content": "Yeah this is kind of my point - if you can only do well in this competition by using external resources (own GPUs, Colab Pro etc) that either cost money or are only available to USA participants is it really a fair competition? Shouldn't it be about knowledge rather than availability of resources?
", "votes": 1, "replies": [] }, { "id": 1057978, "author_name": "Salaryman", "author_url": "", "post_date": "2020-10-23T08:09:35.413000", "content": "It is quite common that $ prize/ gold medal winners trained their models entirely by Kaggle Kernel/ Colab. Hardware can make a difference but more important is your knowledge, experience and \"luck\".
\nI think as long as there is no un-disclosed external data source and private sharing, the competition is considered as fair.
\nWill become a philosophy question if drill too deep about \"fairness\". If both Tom and Jerry are joining this competition but Tom is a machine learning PhD while Jerry is just a chemistry graduate, is this fair even they have the same hardware?
", "votes": 1, "replies": [] }, { "id": 1058031, "author_name": "Kirderf", "author_url": "", "post_date": "2020-10-23T09:09:57.507000", "content": "One can make it simple, in general, and to not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering.
", "votes": 0, "replies": [] }, { "id": 1058067, "author_name": "Serigne ", "author_url": "", "post_date": "2020-10-23T09:53:35.093000", "content": "\n\nShouldn't it be about knowledge rather than availability of resources?
\n
And why not both ?
\nAs far as i'm concerned, I don't have even own single GPU and Colab Pro is available everywhere not only the USA (I suscribed from Paris in the early days).
\nAs for fairness, it's quite relative . Because you can also say even Kaggle itself is not fair, as it gives +30 hours of GPU/TPU to each team member for people teaming up.
\nAnyway this competition requires extensive resources by its nature and by the size of the dataset. You can't help but finding solutions like teaming up, using online services or buying your own solid infrastructure. Otherwise there are still other competitions that require much less resources
", "votes": -1, "replies": [] }, { "id": 1058071, "author_name": "Serigne ", "author_url": "", "post_date": "2020-10-23T10:00:31.127000", "content": "\n\nOne can make it simple, in general, and do not include the extremes, having access to unlimited TPU and V100, gives some advantage in machine learning engineering.
\n
I propose we all use 2 cores CPU only and build every thing from scratch. Because, even using Pytorch instead of writing own CNN and Backpropagation algorithms, give some advantages.
", "votes": 0, "replies": [] }, { "id": 1058111, "author_name": "Kirderf", "author_url": "", "post_date": "2020-10-23T10:59:28.167000", "content": "I thought we were talking about hw. I can only talk for my self, if I had the possible to use more TPU and GPU time through Colab Pro (V100 is really a good option that they have added, using Volta+mixed precision š) I would use it to help tuning and experiment more and wider, but unfortunately it says only available in US and Canada. Would it help me make a better model or getting a better scores? Who knows, that's more a factor of knowledge but I get that knowledge through testing and experimenting, thats why one loves Kaggle and Colab, resources and challenges helping one getting better in different AI areas through helping and contribute solving a problem and with the goal to help and contribute solving problems even better :)
\nI hope they make Colab Pro more available, that's my point.
As I said, Colab Pro is available outside of these countries. Don't mind what is written at the homepage.
\nI'm just giving advices of the most affordable resources to common kagglers in such highly demanding competition. You can be sure many people at (top) LB use much more extensive resources.
\nIt's (nearly) impossible to win such competition by using Kaggle only, just like the annual Youtube-8m video competition (+ 1.5 TB of data last year) and many others google competitions.
\nThat's why kaggle (and sponsors) offer GCP credits on most of these competitions, in particular for those who can't afford high computational power
Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources.
", "votes": 1, "replies": [] }, { "id": 1059084, "author_name": "Mark P", "author_url": "", "post_date": "2020-10-24T15:40:51.823000", "content": "Thanks Serigne and all. I think its fair overall. Those at the very top are probably those who have the combination of knowledge and resources plus the technical skills to properly utilise those resources.
", "votes": 0, "replies": [] }, { "id": 1059176, "author_name": "Kirderf", "author_url": "", "post_date": "2020-10-24T18:04:53.123000", "content": "Yes and it's a great advice Serigne š So you think it's OK to use outside US and Canada? I have to look into it again then, maybe something has changed :)
\nAnd I'm also greatfull that we can use Kaggle and Colab, opens the door for everyone to learn and contribute. So I don't want to sound ungrateful but it's just that the Colab Pro seems like a great subscribe service and open up for even more powerful testing and training for everyone. They have now added another country to the list, so maybe more will come soon if it is not possible to use.
You can try rapid from nvidia for using gpu power while matrix calculations and preporcessing.
", "votes": 0, "replies": [] }, { "id": 1057436, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-22T16:32:39.863000", "content": "We were silver before the public notebook (30th or so). We used Efficientnetb0 trained on tfrecords and TPU, and had an epoch time of about 5 minutes with 5 fold, which was spectacular. Did everything on TPU, but at this point, I think it would be outstandingly difficult to beat the 0.233 baseline with only Kaggle resources. The baseline has a lot to tune, but without insane resources, it is ridiculously difficult to improve it (my teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch). Try out Colab if you're running out of GPU hours though.
", "votes": 0, "replies": [ { "id": 1057456, "author_name": "Mark P", "author_url": "", "post_date": "2020-10-22T16:52:56.187000", "content": "Wow. Great work Stanley. Iām sure pytorch must be able to get close to those speeds on TPU right? Or are TFrecords really so much faster?
", "votes": 1, "replies": [] }, { "id": 1057468, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-22T17:07:52.020000", "content": "I'm not sure, I really haven't done an insanely large competition like this with pytorch. I believe that they are quite a bit faster than pytorch, since I/O is decreased (reading 30 2gb files) instead of thousands of 1mb files. Preprocessing and stratified validation is already completed, so any CPU bottlenecks are minimized also. All the CPU needs to do is split the tfrecords into groups and read them. As far as I know, Kaggle TPU's are mainly CPU bottlenecked, since feeding 128gb of RAM is quite intensive.
", "votes": 1, "replies": [] }, { "id": 1057573, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-22T19:08:31.730000", "content": "Have you used one stage, or two stage model?
", "votes": 1, "replies": [] }, { "id": 1057579, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-22T19:13:57.103000", "content": "This was only with regular CNN trained on image and exam level - stacking with GRU/LSTM would take longer, and separating them would reduce individual training time slightly. The big limitation here is the max TPU time of 3 hours, so even if you had 5 minutes per epoch, you still only get 36 epochs, so 7 epochs for 5 fold, which is not very much. Better to just do each fold separately
", "votes": 2, "replies": [ { "id": 1057615, "author_name": "Serigne ", "author_url": "", "post_date": "2020-10-22T19:52:08.747000", "content": "I have never used Kaggle TPU for pytorch. Colab Pro has much better config for Pytorch. And of course no time limitation.
\nI am training on TPU with a non effnet based model but bigger than b0, b1, b2.
\nBut I think it's a big mistake. I won't likely be able to do all inference in 9 hours.
\n\n\nmy teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch
\n
TPU is the way to go on colab Pro in this case. As I said , I'm training a bigger model than b0 with less than 2 hour per epoch on all the data and heavy augmentations.
\nBig batch size shared between 8 TPU cores and you have more ram/CPU/disk space on TPU than GPU
I agree with your opinion about TPU -- 3 hours is too little for such a big task.
", "votes": 1, "replies": [] }, { "id": 1057629, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-22T20:01:10.853000", "content": "Have you tried stacking with LGBM trained on meta features?
", "votes": 1, "replies": [] }, { "id": 1058557, "author_name": "", "author_url": "", "post_date": "2020-10-23T21:32:19.370000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1057411": "Hi all,\n\nI struggled for a long time to train an Effnet on all the data (Ian Pan's Jpeg dataset) in pytorch. I used the GPUs on kaggle and got it down to 45mins/epoch but thought this was still far too long. I then tried using the TPUs and got it running but the times were the same so I think my bottleneck was the CPU.\n\nBut reading in detail through @khyeh0719 training notebooks I see his training took many many GPU hours too and I think he went to GCP to train some parts.\n\nMy question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?\n\nor is everyone using TF and/or TPUs and its just me struggling to get pytorch working that is the problem? [feeling it is probably<---]", "1057625": "> My question is then is everybody in the gold and silver positions having to resort to using compute outside of kaggle to train their models? If so doesn't that rather undermine the purpose of the kaggle platform if we have to do that?\n\n\n\nYou can still use Colab Pro, it's way much cheaper than GCP; That's what I do whenever I need extensive resources\n\nAnd for GPU, colab Pro has now `Tesla V100-sxm2` option, which is way faster than kaggle P100", "1058657": "You can try rapid from nvidia for using gpu power while matrix calculations and preporcessing.", "1057436": "We *were* silver before the public notebook (30th or so). We used Efficientnetb0 trained on tfrecords and TPU, and had an epoch time of about 5 minutes with 5 fold, which was spectacular. Did everything on TPU, but at this point, I think it would be outstandingly difficult to beat the 0.233 baseline with only Kaggle resources. The baseline has a lot to tune, but without insane resources, it is ridiculously difficult to improve it (my teammate tried to train on Colab Pro GPU and it predicted 8 hours per epoch). Try out Colab if you're running out of GPU hours though." } }