Please find the starter pack on github. I have not used kernels yet - if someone would find this useful and would want to transfer it over to a kernel, please feel free to go ahead.
\n\nThis uses the brand new FastAi v1 library. I trained with defaults, including default hyperparameter values. I trained on 1% of data of size 128x128. The training was probably done using SGD (with one cycle policy) but I have not checked, neither have I looked at the batch size. Of course, that means there is a lot of room to improve (also in terms of hw utilization) but wanted this to be as vanilla as it gets. Also wanted to start getting a feel for the use of the library in the way the authors envisioned it.
\n\nTurns out the library supports TTA out of the box so I predicted with TTA using again the provided defaults.
\n\nThis code is based on code from a fast.ai MOOC that will be publicly available in Jan 2019.
\n\nEDIT: The default optimization algorithm with fit_one_cycle turns out to be Adam.
", "messages": [ { "id": 408924, "postDate": "2018-10-23T15:57:45.893Z", "content": "Please find the starter pack on github. I have not used kernels yet - if someone would find this useful and would want to transfer it over to a kernel, please feel free to go ahead.
\n\nThis uses the brand new FastAi v1 library. I trained with defaults, including default hyperparameter values. I trained on 1% of data of size 128x128. The training was probably done using SGD (with one cycle policy) but I have not checked, neither have I looked at the batch size. Of course, that means there is a lot of room to improve (also in terms of hw utilization) but wanted this to be as vanilla as it gets. Also wanted to start getting a feel for the use of the library in the way the authors envisioned it.
\n\nTurns out the library supports TTA out of the box so I predicted with TTA using again the provided defaults.
\n\nThis code is based on code from a fast.ai MOOC that will be publicly available in Jan 2019.
\n\nEDIT: The default optimization algorithm with fit_one_cycle turns out to be Adam.
", "rawMarkdown": "Please find the starter pack on [github][1]. I have not used kernels yet - if someone would find this useful and would want to transfer it over to a kernel, please feel free to go ahead.\n\nThis uses the brand new [FastAi v1 library][2]. I trained with defaults, including default hyperparameter values. I trained on 1% of data of size 128x128. The training was probably done using SGD (with one cycle policy) but I have not checked, neither have I looked at the batch size. Of course, that means there is a lot of room to improve (also in terms of hw utilization) but wanted this to be as vanilla as it gets. Also wanted to start getting a feel for the use of the library in the way the authors envisioned it.\n\nTurns out the library supports TTA out of the box so I predicted with TTA using again the provided defaults.\n\n**This code is based on code from a fast.ai MOOC that will be publicly available in Jan 2019.** \n\nEDIT: The default optimization algorithm with fit_one_cycle turns out to be Adam.\n\n [1]: https://github.com/radekosmulski/quickdraw\n [2]: https://github.com/fastai/fastai", "votes": 55 }, { "id": 412708, "postDate": "2018-10-30T16:27:46.803Z", "content": "The good news is that resnet34 trained on 2 million of 128x128 drawings achieves a score of 0.9. This is just below the bronze medal range.
\n\nAn even better news is that training on same quantity of data, but 256x256, gets a score of 0.907. Unfortunately fully training the model takes > 12 hrs. I could possibly bring the time down to ~8hrs but hard to say if that would have an adverse effect on performance or not.
\n\nEither way, that train time with resnet34 is depressingly long. I am not sure what to try next - it seems that ensembling should work here really well. At the same time bigger models should also give a better score - as expected there are no signs of overfitting with resnet34.
\n\nI just got the results a couple of minutes ago so it might take me a while to figure out what to make of the situation. Nonetheless, I think I'll start another train cycle with a bigger architecture just to see what it will do and will try to trade marginal improvements in performance for shorter train time.
", "rawMarkdown": "The good news is that resnet34 trained on 2 million of 128x128 drawings achieves a score of 0.9. This is just below the bronze medal range.\n\nAn even better news is that training on same quantity of data, but 256x256, gets a score of 0.907. Unfortunately fully training the model takes > 12 hrs. I could possibly bring the time down to ~8hrs but hard to say if that would have an adverse effect on performance or not. \n\nEither way, that train time with resnet34 is depressingly long. I am not sure what to try next - it seems that ensembling should work here really well. At the same time bigger models should also give a better score - as expected there are no signs of overfitting with resnet34. \n\nI just got the results a couple of minutes ago so it might take me a while to figure out what to make of the situation. Nonetheless, I think I'll start another train cycle with a bigger architecture just to see what it will do and will try to trade marginal improvements in performance for shorter train time.", "votes": 9, "replies": [ { "id": 412733, "postDate": "2018-10-30T17:37:27.120Z", "content": "It might be worth to use smaller sized images with larger architectures-might allow room for speedy experimentation.
", "rawMarkdown": "It might be worth to use smaller sized images with larger architectures-might allow room for speedy experimentation.", "votes": 2 }, { "id": 412969, "postDate": "2018-10-31T04:15:12.680Z", "content": "Did you normalize (with imagenet stats) ?
", "rawMarkdown": "Did you normalize (with imagenet stats) ?" }, { "id": 413024, "postDate": "2018-10-31T06:50:22.780Z", "content": "I use stats automatically calculated from one batch (this is what you get when you call db.normalize() without arguments).
", "rawMarkdown": "I use stats automatically calculated from one batch (this is what you get when you call db.normalize() without arguments)." }, { "id": 413210, "postDate": "2018-10-31T13:59:43.150Z", "content": "I tried with resnet34, with 256 size on 5% of data (around 2.5M), it's nowhere close to 0.9. Am i missing something here ? Even tried with Resnet50, still the same problem. Even after training on 10% of the data, it's not helping much.
", "rawMarkdown": "I tried with resnet34, with 256 size on 5% of data (around 2.5M), it's nowhere close to 0.9. Am i missing something here ? Even tried with Resnet50, still the same problem. Even after training on 10% of the data, it's not helping much.", "votes": 1 }, { "id": 413225, "postDate": "2018-10-31T14:26:54.917Z", "content": "Maybe you need to train for longer? You could try to manually train for x epochs with some lr, say something like this:
\n\nlearn.fit(5, 1e-3)
and only lower the training rate once you hit a plateau. This can be done automatically with callbacks (you can find them here: fastai/callbacks/tracker.py), but it can be useful to go through the motions by hand a couple of times.
5 cycles before unfreezing ? How did you call ReduceLROnPlateau? It's not covered in docs yet. Do we pass it like metrics or call on learn ? Also CLR can handle the same functionality ?
Did you try using MobileNetV2 ?
", "rawMarkdown": "Did you try using MobileNetV2 ?" }, { "id": 418455, "postDate": "2018-11-09T23:10:56.737Z", "content": "I have the same question on How you call ReduceLROnPlateau. Fastai's documentation is so hard to read and lack of examples which are using each functions.
", "rawMarkdown": "I have the same question on How you call ReduceLROnPlateau. Fastai's documentation is so hard to read and lack of examples which are using each functions. " }, { "id": 418467, "postDate": "2018-11-10T00:14:39.240Z", "content": "I have figured out how to call ReduceLROnPlateau.\nreduceLR = ReduceLROnPlateauCallback(learn=learn, monitor = 'val_loss', mode = 'auto', patience = 10, factor = 0.2, min_delta = 0)\nlearn.fit_one_cycle(20, max_lr=slice(5e-6,5e-4), callbacks=[reduceLR])
", "rawMarkdown": "I have figured out how to call ReduceLROnPlateau.\nreduceLR = ReduceLROnPlateauCallback(learn=learn, monitor = 'val_loss', mode = 'auto', patience = 10, factor = 0.2, min_delta = 0)\nlearn.fit_one_cycle(20, max_lr=slice(5e-6,5e-4), callbacks=[reduceLR])", "votes": 1 } ] }, { "id": 420494, "postDate": "2018-11-13T17:59:22.183Z", "content": "I updated the starter pack to use the data_block API. You can find the new version here.
\n\nI transitioned to using the data_block API. Drawings are now generated on the fly. Training with 128x128 drawings, 1000 examples per class, now achieves 0.856 on public LB in ~30 minutes of training on a single 1080TI.
", "rawMarkdown": "I updated the starter pack to use the data_block API. You can find the new version [here](https://github.com/radekosmulski/quickdraw).\n\nI transitioned to using the data_block API. Drawings are now generated on the fly. Training with 128x128 drawings, 1000 examples per class, now achieves 0.856 on public LB in ~30 minutes of training on a single 1080TI.", "votes": 7, "replies": [ { "id": 421065, "postDate": "2018-11-14T14:24:45.407Z", "content": "Hi Radek,
\n\nThanks for sharing the code. What is the fastai version you are using? Right now, I am on 1.0.24. I am getting an error on InputList (name 'InputList' is not defined).
\n\nThanks
", "rawMarkdown": "Hi Radek, \n\nThanks for sharing the code. What is the fastai version you are using? Right now, I am on 1.0.24. I am getting an error on InputList (name 'InputList' is not defined).\n\nThanks" }, { "id": 421076, "postDate": "2018-11-14T14:48:20.483Z", "content": "@asanghai You should use 1.0.25, there was a big refactor of the data_block API in that version.
", "rawMarkdown": "@asanghai You should use 1.0.25, there was a big refactor of the data_block API in that version.", "votes": 1 }, { "id": 421130, "postDate": "2018-11-14T16:16:39.780Z", "content": "I just pushed an updated version - this one works with the current fast.ai master.
", "rawMarkdown": "I just pushed an updated version - this one works with the current fast.ai master.", "votes": 1 }, { "id": 421150, "postDate": "2018-11-14T16:45:15.620Z", "content": "Thanks William. I could not find 1.0.25 (https://pypi.org/project/fastai/#history https://anaconda.org/fastai/fastai/files). Can you please let me know how to get 1.0.25?
\n\nThanks Radek. Can you please let me know what fastai version you were using before?
", "rawMarkdown": "Thanks William. I could not find 1.0.25 (https://pypi.org/project/fastai/#history https://anaconda.org/fastai/fastai/files). Can you please let me know how to get 1.0.25?\n\nThanks Radek. Can you please let me know what fastai version you were using before? \n\n\n\n" }, { "id": 421167, "postDate": "2018-11-14T16:56:09.103Z", "content": "I am not sure - it was one from yesterday. For future reference, the version I am on right now is af068ecf9ba98c5c3383b59bb2a7d44b01337297.
@asanghai unfortunately it looks like 1.0.25 hasn’t been published yet. If you don’t want to wait, you can download the GitHub repo and follow the instructions for a local build (which is what I do to always have the latest code).
", "rawMarkdown": "@asanghai unfortunately it looks like 1.0.25 hasn’t been published yet. If you don’t want to wait, you can download the GitHub repo and follow the instructions for a local build (which is what I do to always have the latest code).", "votes": 2 }, { "id": 422159, "postDate": "2018-11-15T21:16:05.400Z", "content": "When I run your v2, I got error \"name 'ItemList' is not defined\". What do you mean of data_block API? Do I need to download something instead of fastai library? Many thanks.
", "rawMarkdown": "When I run your v2, I got error \"name 'ItemList' is not defined\". What do you mean of data_block API? Do I need to download something instead of fastai library? Many thanks.\n" }, { "id": 422162, "postDate": "2018-11-15T21:27:26.257Z", "content": "@William Horton \nWhat do you mean \"local build\"? I couldn't find this item from the instruction of installing fastai library.
", "rawMarkdown": "@William Horton \nWhat do you mean \"local build\"? I couldn't find this item from the instruction of installing fastai library." }, { "id": 422260, "postDate": "2018-11-16T01:36:22.010Z", "content": "@Xu Zhang i meant the section in the fastai README called “Developer Install”. It involves running pip install -e .[dev] within fastai once you’ve cloned it
", "rawMarkdown": "@Xu Zhang i meant the section in the fastai README called “Developer Install”. It involves running pip install -e .[dev] within fastai once you’ve cloned it" }, { "id": 422294, "postDate": "2018-11-16T03:00:23.893Z", "content": "thanks Radek!
\n\ni get error executing this line of code:
\n\n\ncreate_submission(preds, data_bunch.test_dl, name)\n
error:
\n\n\nNameError: name 'classes' is not defined\n
starter pack version: ab4cf72
\n\ni use fastai developer install, fastai version: 672c8c5
", "rawMarkdown": "thanks Radek!\n\ni get error executing this line of code:\n\n```\ncreate_submission(preds, data_bunch.test_dl, name)\n```\n\nerror:\n\n```\nNameError: name 'classes' is not defined\n```\n\nstarter pack version: ab4cf72\n\ni use fastai developer install, fastai version: 672c8c5 " }, { "id": 422381, "postDate": "2018-11-16T06:28:50.170Z", "content": "Hi! The issue was that create_submission required the classes list to be defined. I now made it so that classes needs to be explicitly passed into the function so that should give an easier to understand error message.
The classes list gets loaded in cell #14 - please make sure you execute it before trying to create a submission file.
great thank you!\npossibly would be nice if the subs directory was created first as well...
Hi @radek,
\n\nWhat a fun starter pack to enjoy, again !
\n\nI'm currently on fastai 1.0.28, if not mistaken. Trying to have fun with a dedicated RTX 2080ti, beside my good old 1080Ti.
\n\nWhen I run your latest notebook, in the item_list = ItemList.from_folder(PATH/'train', create_func=create_func) cell, I get this error: any idea ?
Traceback (most recent call last)\n<ipython-input-14-611893917372> in <module>()\n----> 1 item_list = ItemList.from_folder(PATH/'train', create_func=create_func)\n\nAttributeError: type object 'ItemList' has no attribute 'from_folder'\nI think the API changed a bit in 1.0.28 and I haven't had a chance to update the starter pack yet. At this point one of the options would be figuring out what changed, but probably it might be easier to downgrade to 1.0.25 which the current version of the starter pack should work with.
", "rawMarkdown": "I think the API changed a bit in 1.0.28 and I haven't had a chance to update the starter pack yet. At this point one of the options would be figuring out what changed, but probably it might be easier to downgrade to 1.0.25 which the current version of the starter pack should work with.", "votes": 1 }, { "id": 425685, "postDate": "2018-11-22T00:43:07.557Z", "content": "Hi Eric,\nI’ve run into these changes in my human protein atlas starter code as well. The two ways to fix would be:\n1) create a subclass of ItemList that overrides the open method to use the create func\n2) a more hacky approach, once you have your item list variable, do item list.open=create func and that should also work (had to remove underscores because it messes up comment formatting)
", "rawMarkdown": "Hi Eric,\nI’ve run into these changes in my human protein atlas starter code as well. The two ways to fix would be:\n1) create a subclass of ItemList that overrides the open method to use the create func\n2) a more hacky approach, once you have your item list variable, do item list.open=create func and that should also work (had to remove underscores because it messes up comment formatting)", "votes": 2 }, { "id": 426195, "postDate": "2018-11-22T20:48:27.370Z", "content": "Sorry, just realised my post had an incomplete copy/paste of the AttributeError so I updated it with the full version.
\n\n---------------------------------------------------------------------------\nAttributeError Traceback (most recent call last)\n<ipython-input-14-611893917372> in <module>()\n----> 1 item_list = ItemList.from_folder(PATH/'train', create_func=create_func)\n\nAttributeError: type object 'ItemList' has no attribute 'from_folder'\nJust got back to play around with your Starter Pack on 1.0.25: wow !\nNot only is it way way way faster that the original one but the score is really better from scratch \"as it is\".\nNow I need to figure out what is happening :-D\nAnd how to improve the score step by step.
\n\nYou dropped the TTA btw or you are using a different syntax ?
\n\n", "rawMarkdown": "Just got back to play around with your Starter Pack on 1.0.25: wow !\nNot only is it way way way faster that the original one but the score is really better from scratch \"as it is\".\nNow I need to figure out what is happening :-D\nAnd how to improve the score step by step.\n\nYou dropped the TTA btw or you are using a different syntax ?\n\ncc @hortonhearsafoo", "votes": 1 }, { "id": 428557, "postDate": "2018-11-27T13:27:18.850Z", "content": "I dropped the TTA - haven't experimented much with it though, not sure if it is useful or not.
", "rawMarkdown": "I dropped the TTA - haven't experimented much with it though, not sure if it is useful or not." }, { "id": 428749, "postDate": "2018-11-27T20:29:42.903Z", "content": "In theory there is no reason to use augmentation when enough data is available, like in this competition.
\n\nIf you think about it, no matter how clever augmentation can be, a new image from the true population will always be better. (Such a big dataset we have here...! ) :-)
", "rawMarkdown": "In theory there is no reason to use augmentation when enough data is available, like in this competition. \n\nIf you think about it, no matter how clever augmentation can be, a new image from the true population will always be better. (Such a big dataset we have here...! ) :-)", "votes": 4 }, { "id": 429112, "postDate": "2018-11-28T11:04:33.587Z", "content": "Hi I am trying with the starter pack. When i execute \nlabel_lists = item_lists.label_from_folder() - I am getting error IndexError: index 0 is out of bounds for axis 0 with size 0 . I am using fast ai Version: 1.0.29\nHow to download the fast ai version 1.0.25? Thanks for the help.
", "rawMarkdown": "Hi I am trying with the starter pack. When i execute \nlabel_lists = item_lists.label_from_folder() - I am getting error IndexError: index 0 is out of bounds for axis 0 with size 0 . I am using fast ai Version: 1.0.29\nHow to download the fast ai version 1.0.25? Thanks for the help.\n\n" }, { "id": 429134, "postDate": "2018-11-28T11:51:25.970Z", "rawMarkdown": "", "isDeleted": true }, { "id": 429331, "postDate": "2018-11-28T17:39:04.353Z", "content": "I downloaded the version using -
\n\n!pip install fastai==1.0.25
", "rawMarkdown": "I downloaded the version using -\n\n!pip install fastai==1.0.25\n\n", "votes": 1 }, { "id": 429396, "postDate": "2018-11-28T19:54:43.193Z", "content": "(I edited my post as I figured out the cause of the huge increase in epoch time, my own mistake).
\n\nAs I dig deeper into the different parameters, which helps me understand your approach, I'm now looking at your second cell \"NUM-SAMPLES-PER-CLASS = 1000; NUM-VAL = 50 * 340\".
\n\nI think I understand the first one \"Num-samples-per-class = 1000\" as the name is obvious, and easy to change/increase for better accuracy like 2000.
\n\nBut I'm not sure about the second one \"Num-val = 50 * 340\", apart from 340 being the # of categories.\nWhen I look at the later cell \"data_bunch.normalize(batch_stats)\", it shows \"CategoryList(323000 items)\".
\n\nThen if I modify to \"Num-val = 200 * 340\", the later cell shows \"CategoryList(272000 items)\".
\n\nI can't unlock the logic/relation behind 50x340=>323K and 200x340=>272K.
", "rawMarkdown": "*(I edited my post as I figured out the cause of the huge increase in epoch time, my own mistake).*\n\nAs I dig deeper into the different parameters, which helps me understand your approach, I'm now looking at your second cell \"NUM-SAMPLES-PER-CLASS = 1000; NUM-VAL = 50 * 340\".\n\nI think I understand the first one \"Num-samples-per-class = 1000\" as the name is obvious, and easy to change/increase for better accuracy like 2000.\n\nBut I'm not sure about the second one \"Num-val = 50 * 340\", apart from 340 being the # of categories.\nWhen I look at the later cell \"data_bunch.normalize(batch_stats)\", it shows \"CategoryList(323000 items)\".\n\nThen if I modify to \"Num-val = 200 * 340\", the later cell shows \"CategoryList(272000 items)\".\n\nI can't unlock the logic/relation behind 50x340=>323K and 200x340=>272K." }, { "id": 429422, "postDate": "2018-11-28T20:35:21.040Z", "content": "Around 50 examples per class to have in the validation set was just an arbitrary choice.
\n\nThe bigger the val set (200 x 340), the fewer examples will be left to include in the train set.
", "rawMarkdown": "Around 50 examples per class to have in the validation set was just an arbitrary choice.\n\nThe bigger the val set (200 x 340), the fewer examples will be left to include in the train set.", "votes": 1 }, { "id": 429441, "postDate": "2018-11-28T21:12:33.447Z", "content": "Thanks, you rock Sir !
\n\nI learn so much looking at your code and decoding it.
\n\n(BTW, loved your previous profile's picture, looking like a crazy helicopter pilot adjusting his goggles ^!^)
", "rawMarkdown": "Thanks, you rock Sir !\n\nI learn so much looking at your code and decoding it.\n\n(BTW, loved your previous profile's picture, looking like a crazy helicopter pilot adjusting his goggles ^!^)", "votes": 1 }, { "id": 469439, "postDate": "2019-02-11T08:18:25.177Z", "content": "Sorry for the beginners question, but where is the create_submission() function imported from? I'm having issues even recognizing that function.
", "rawMarkdown": "Sorry for the beginners question, but where is the create_submission() function imported from? I'm having issues even recognizing that function." }, { "id": 469652, "postDate": "2019-02-11T15:57:26.500Z", "content": "So to answer my own question, if someone else has the same question (I'm using a Google Cloud jupyter notebook).
\n\nTo get the create_submission function to work, you need to put the utils.py file (also in radeks repository) in the same folder as your notebook file (.ipynb). The create_submission() function is defined in utils.py
", "rawMarkdown": "So to answer my own question, if someone else has the same question (I'm using a Google Cloud jupyter notebook).\n\nTo get the create_submission function to work, you need to put the utils.py file (also in radeks repository) in the same folder as your notebook file (.ipynb). The create_submission() function is defined in utils.py", "votes": 1 } ] }, { "id": 415828, "postDate": "2018-11-05T19:08:59.727Z", "content": "Great way of putting to test new Fastai v1, very nice pipeline, thanks for that Radek! :-)
", "rawMarkdown": "Great way of putting to test new Fastai v1, very nice pipeline, thanks for that Radek! :-)", "votes": 3 }, { "id": 415765, "postDate": "2018-11-05T16:52:50.197Z", "content": "I've created a custom dataset that converts strokes to images on the fly and is compatible with fastai - you can find the baseline here, there is still a lot of room for imrovement.
", "rawMarkdown": "I've created a custom dataset that converts strokes to images on the fly and is compatible with fastai - you can find the baseline [here](https://github.com/adilism/quickdraw/blob/master/fastai-baseline.ipynb), there is still a lot of room for imrovement.", "votes": 3, "replies": [ { "id": 415778, "postDate": "2018-11-05T17:16:08.130Z", "content": "Thanks for sharing it. Was just curious, whenever the next batch is called only then images are being produced and fed or are they produced earlier ? Is this similar to using generator?
", "rawMarkdown": "Thanks for sharing it. Was just curious, whenever the next batch is called only then images are being produced and fed or are they produced earlier ? Is this similar to using generator?", "votes": 1 }, { "id": 415829, "postDate": "2018-11-05T19:09:26.980Z", "content": "I am not sure about a generator, but yes, here the images are produces when the batch is called
", "rawMarkdown": "I am not sure about a generator, but yes, here the images are produces when the batch is called" }, { "id": 416071, "postDate": "2018-11-06T06:21:28.093Z", "content": "I got this RuntimeError: DataLoader worker (pid 2157) is killed by signal: Bus error.
Hm, maybe try with smaller bs? I think it might be an issue with RAM
", "rawMarkdown": "Hm, maybe try with smaller bs? I think it might be an issue with RAM" }, { "id": 416245, "postDate": "2018-11-06T13:17:01.313Z", "content": "No,this issue lies with Pytorch, using num_workers=0 eliminates the issue, but increases the training time by a factor of approx num_of_workers. It happens due to lack of shared memory. It hasn't been resolved on Pytorch issues section as well.
Indeed it seems that training with bigger images helps a bit. Another consideration would be to include unrecognized drawings. Currently, during data creation, I filter those out.
\n\nAlso, we use imagenet stats to normalize the data. If you don't pass the stats into the normalize method, the images will be normalized based on statistics calculated from a single batch.
Once you train more than one model, you can start thinking of ensembling. I haven't tried it yet for this competition, but it is a powerful technique that can move you up a lot of places. Here is a really great blog post summarizing a couple of approaches. It is a good idea to start with averaging results and use this as a baseline - might be more advanced techniques are not necessary or can provide only small gains.
", "rawMarkdown": "Indeed it seems that training with bigger images helps a bit. Another consideration would be to include unrecognized drawings. Currently, during data creation, I filter those out.\n\nAlso, we use imagenet stats to normalize the data. If you don't pass the stats into the `normalize` method, the images will be normalized based on statistics calculated from a single batch. \n\nOnce you train more than one model, you can start thinking of ensembling. I haven't tried it yet for this competition, but it is a powerful technique that can move you up a lot of places. Here is [a really great blog post][1] summarizing a couple of approaches. It is a good idea to start with averaging results and use this as a baseline - might be more advanced techniques are not necessary or can provide only small gains. \n\n\n [1]: https://mlwave.com/kaggle-ensembling-guide", "votes": 4, "replies": [ { "id": 411651, "postDate": "2018-10-28T17:10:18.490Z", "content": "I have trained model on 10% data,but still it can't cross 0.9 on LB. Was thinking of adding custom head or making changes with ResNet. Increasing parameters with well defined arch will definitely help. Also not sure whether loss function being used in ConvLearner is the best choice for this.
Your link to the blog post is broken.
\n\nThanks for sharing and the wonderful kernel. :)
", "rawMarkdown": "Your link to the blog post is broken. \n\nThanks for sharing and the wonderful kernel. :)", "votes": 1 }, { "id": 411950, "postDate": "2018-10-29T09:27:26.627Z", "content": "Thanks :) Should be fixed now.
", "rawMarkdown": "Thanks :) Should be fixed now.", "votes": 1 } ] }, { "id": 424581, "postDate": "2018-11-20T11:03:31.350Z", "content": "To stay true to the 'no sharing outside of teams rule' - and also because I think it might actually be useful to other folks using this starter pack - I share here on the fastai forums how to train on epoch_sizes of arbitrary size.
\n\nIt's more of a technicality on how to use the framework but is a really nice feature and might be of use to some.
\n\nHere is the code:\n```\ndef chunks(l, n):\n \"\"\"Yield successive n-sized chunks from l.\"\"\"\n for i in range(0, len(l), n):\n yield l[i:i + n]
\n\nclass RandomSamplerWithEpochSize(Sampler):\n \"\"\"Yields epochs of specified sizes. Iterates over all examples in a data_source in random\n order. Ensures (nearly) all examples have been trained on before beginning the next iteration\n over the data_source - drops the last epoch that would likely be smaller than epoch_size.\n \"\"\"\n def init(self, data_source, epoch_size):\n self.n = len(data_source)\n self.epoch_size = epoch_size\n self._epochs = []\n def iter(self):\n return iter(self.next_epoch)\n @property\n def next_epoch(self):\n if len(self._epochs) == 0: self.generate_epochs()\n return self._epochs.pop()\n def generate_epochs(self):\n idxs = [i for i in range(self.n)]\n np.random.shuffle(idxs)\n self._epochs = list(chunks(idxs, self.epoch_size))[:-1]\n def len(self):\n return self.epoch_size\n```
\n\nAnd here is how to use it to create the data_bunch:\n```\ntrain_dl = DataLoader(\n label_lists.train,\n num_workers=12,\n batch_sampler=BatchSampler(RandomSamplerWithEpochSize(label_lists.train, 200_000), bs, True)\n)\nvalid_dl = DataLoader(label_lists.valid, 2*bs, False, num_workers=12)\ntest_dl = DataLoader(label_lists.test, 2*bs, False, num_workers=12)
data_bunch = ImageDataBunch(train_dl, valid_dl, test_dl)\n```
", "rawMarkdown": "To stay true to the 'no sharing outside of teams rule' - and also because I think it might actually be useful to other folks using this starter pack - I share here on [the fastai forums](https://forums.fast.ai/t/cpu-memory-usage-keeps-growing-as-training-one-cycle/30879/6?u=radek) how to train on epoch_sizes of arbitrary size.\n\nIt's more of a technicality on how to use the framework but is a really nice feature and might be of use to some.\n\nHere is the code:\n```\ndef chunks(l, n):\n \"\"\"Yield successive n-sized chunks from l.\"\"\"\n for i in range(0, len(l), n):\n yield l[i:i + n]\n\nclass RandomSamplerWithEpochSize(Sampler):\n \"\"\"Yields epochs of specified sizes. Iterates over all examples in a data_source in random\n order. Ensures (nearly) all examples have been trained on before beginning the next iteration\n over the data_source - drops the last epoch that would likely be smaller than epoch_size.\n \"\"\"\n def __init__(self, data_source, epoch_size):\n self.n = len(data_source)\n self.epoch_size = epoch_size\n self._epochs = []\n def __iter__(self):\n return iter(self.next_epoch)\n @property\n def next_epoch(self):\n if len(self._epochs) == 0: self.generate_epochs()\n return self._epochs.pop()\n def generate_epochs(self):\n idxs = [i for i in range(self.n)]\n np.random.shuffle(idxs)\n self._epochs = list(chunks(idxs, self.epoch_size))[:-1]\n def __len__(self):\n return self.epoch_size\n```\n\nAnd here is how to use it to create the `data_bunch`:\n```\ntrain_dl = DataLoader(\n label_lists.train,\n num_workers=12,\n batch_sampler=BatchSampler(RandomSamplerWithEpochSize(label_lists.train, 200_000), bs, True)\n)\nvalid_dl = DataLoader(label_lists.valid, 2*bs, False, num_workers=12)\ntest_dl = DataLoader(label_lists.test, 2*bs, False, num_workers=12)\n\ndata_bunch = ImageDataBunch(train_dl, valid_dl, test_dl)\n```", "votes": 1 }, { "id": 422396, "postDate": "2018-11-16T07:00:16.737Z", "content": "Awesome thanks. I had a go at making it into a kernel but ran out of disk space. work in progress..
\n\nhttps://www.kaggle.com/dromosys/fast-ai-quick-draw/
", "rawMarkdown": "Awesome thanks. I had a go at making it into a kernel but ran out of disk space. work in progress..\n\nhttps://www.kaggle.com/dromosys/fast-ai-quick-draw/", "votes": 1, "replies": [ { "id": 422430, "postDate": "2018-11-16T08:05:45.293Z", "content": "This code uses fastai v1..but on kernel v0.7 is installed. I am afraid you will have to make lot of changes in the code to make it run on kernel.
", "rawMarkdown": "This code uses fastai v1..but on kernel v0.7 is installed. I am afraid you will have to make lot of changes in the code to make it run on kernel.", "votes": 1 }, { "id": 422790, "postDate": "2018-11-16T19:29:42.073Z", "content": "nope fastai version 1.0.26.dev0 is installed via !pip3 install git+https://github.com/fastai/fastai.git
", "rawMarkdown": "nope fastai version 1.0.26.dev0 is installed via !pip3 install git+https://github.com/fastai/fastai.git" }, { "id": 422837, "postDate": "2018-11-16T21:14:38.647Z", "content": "Have you managed to update pytorch as well? The docs say fastai only works with pytorch 1.0. Would love to see fastai 1.0 working on Kernels, I think it’s a great platform for learning.
", "rawMarkdown": "Have you managed to update pytorch as well? The docs say fastai only works with pytorch 1.0. Would love to see fastai 1.0 working on Kernels, I think it’s a great platform for learning." }, { "id": 422971, "postDate": "2018-11-17T06:46:25.610Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 423657, "postDate": "2018-11-18T19:55:43.653Z", "content": "you're right pytorch didn't update automatically. manually update worked with GPU off but not with GPU enabled. https://www.kaggle.com/dromosys/fast-ai-tiny-planet/
", "rawMarkdown": "you're right pytorch didn't update automatically. manually update worked with GPU off but not with GPU enabled. https://www.kaggle.com/dromosys/fast-ai-tiny-planet/" } ] }, { "id": 413238, "postDate": "2018-10-31T14:56:59.807Z", "content": "Hey has anyone able to get convert more than 10% of their dataset to images? It's the max I could convert. Anyone facing the same issue?
\n\nAnd using strokes directly as done in 🐘Greyscale MobileNet [LB=0.892] a better approach?
\n\nThanks, Radek for sharing your code! It was super helpful.
", "rawMarkdown": "Hey has anyone able to get convert more than 10% of their dataset to images? It's the max I could convert. Anyone facing the same issue? \n\nAnd using strokes directly as done in [ 🐘Greyscale MobileNet [LB=0.892]][1] a better approach? \n\nThanks, Radek for sharing your code! It was super helpful.\n\n\n [1]: https://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892#", "votes": 1 }, { "id": 412801, "postDate": "2018-10-30T19:50:12.290Z", "content": "Thanks for sharing. Already looks good from my initial run.
", "rawMarkdown": "Thanks for sharing. Already looks good from my initial run.", "votes": 1, "replies": [ { "id": 413104, "postDate": "2018-10-31T09:18:59.643Z", "content": "Hi, what's the version of fastai, pytorch? I use 1.0.18, but ConvLearner cannot be found for the reason not defined.
", "rawMarkdown": "Hi, what's the version of fastai, pytorch? I use 1.0.18, but ConvLearner cannot be found for the reason not defined." }, { "id": 413108, "postDate": "2018-10-31T09:31:58.997Z", "content": "Recently ConvLearner has become create_cnn. Refer to docs.
Seems that this was introduced in 1.0.15. I think you should be good to use anything prior to this. Also, I believe that just changing ConvLearner(...) to create_cnn might work also.
Please let us know if you get it to work!
", "rawMarkdown": "Seems that this was introduced in `1.0.15`. I think you should be good to use anything prior to this. Also, I believe that just changing `ConvLearner(...)` to `create_cnn` might work also.\n\nPlease let us know if you get it to work!" }, { "id": 413295, "postDate": "2018-10-31T16:35:15.767Z", "content": "I can confirm that just changing ConvLearner(...) and replacing it with create_cnn works just fine everywhere.
Thanks for sharing this! \nI tried submitting with your stater code and with an increase in data I noticed some gain so I decided to train the model on the complete datset after extracting the complete dataset on disk. \nI'm trying to give this approach a shot instead of trying other approaches (About a month of runway until the deadline which is slightly less given the data size)
", "rawMarkdown": "Thanks for sharing this! \nI tried submitting with your stater code and with an increase in data I noticed some gain so I decided to train the model on the complete datset after extracting the complete dataset on disk. \nI'm trying to give this approach a shot instead of trying other approaches (About a month of runway until the deadline which is slightly less given the data size)", "votes": 1, "replies": [ { "id": 411539, "postDate": "2018-10-28T12:15:17.967Z", "content": "May I know how much of the data you use to train?
", "rawMarkdown": "May I know how much of the data you use to train?" }, { "id": 411554, "postDate": "2018-10-28T13:08:30.817Z", "content": "I've tried 1, 5 % so far.
", "rawMarkdown": "I've tried 1, 5 % so far. ", "votes": 1 }, { "id": 412048, "postDate": "2018-10-29T13:16:58.157Z", "content": "I'm not sure what went wrong here. But training the model on the complete data made the model perform worse than the one on 1% data. (Used the same approach, just on 100% of the data)
", "rawMarkdown": "I'm not sure what went wrong here. But training the model on the complete data made the model perform worse than the one on 1% data. (Used the same approach, just on 100% of the data)", "votes": 1 }, { "id": 412050, "postDate": "2018-10-29T13:28:25.417Z", "content": "Was the difference big or only slight? Might be including the unrecognized drawings does not help (maybe they are of very poor quality?)
", "rawMarkdown": "Was the difference big or only slight? Might be including the unrecognized drawings does not help (maybe they are of very poor quality?)", "votes": 1 }, { "id": 412084, "postDate": "2018-10-29T14:48:57.440Z", "rawMarkdown": "", "isDeleted": true }, { "id": 414727, "postDate": "2018-11-03T13:23:28.780Z", "content": "I'm trying with 5% at size=64, it runs all fine like 1% until preds = learn.TTA() in Generate Submission part where it gets an error object of type \"NoneType\" has no len().\nNeed to look into the docs :-)
NVM, it was just a spelling mistake in the path for the ../test_simp... in data=
I added my solution based on the starter pack to the repository. The model in itself is nothing special (it is a combination of 2 cnns, rnn and country code embeddings) but the way the data is generated and fed to the model and a couple of other things (epochs of arbitrary size, changing the classifier during training, etc) might potentially be of interest.
\n\nHere is a link to a Twitter thread where I go into some additional details.
", "rawMarkdown": "I added my solution based on the starter pack to the repository. The model in itself is nothing special (it is a combination of 2 cnns, rnn and country code embeddings) but the way the data is generated and fed to the model and a couple of other things (epochs of arbitrary size, changing the classifier during training, etc) might potentially be of interest.\n\nHere is a [link](https://twitter.com/radekosmulski/status/1071341277453656070) to a Twitter thread where I go into some additional details.", "votes": 2 }, { "id": 413877, "postDate": "2018-11-01T17:33:09.693Z", "content": "Hi Radek !
\n\nJust started your Starter Pack, quite an amazing surprise, many thanks.
\n\nBTW I think I read in one of your posts on Kaggle that you use a Ryzen 5 + 1080Ti, I got almost the same setup with a Ryzen 7 1700X but\n- (1) my epochs are about 10-20% slower than yours and\n- (2) I later had to lower batch size to 48 (default 64) for the unfreeze part to avoid CUDA OOM error.
Cheers,
\n\nEPB
", "rawMarkdown": "\nHi Radek !\n\nJust started your Starter Pack, quite an amazing surprise, many thanks.\n\nBTW I think I read in one of your posts on Kaggle that you use a Ryzen 5 + 1080Ti, I got almost the same setup with a Ryzen 7 1700X but\n- (1) my epochs are about 10-20% slower than yours and\n- (2) I later had to lower batch size to 48 (default 64) for the `unfreeze` part to avoid CUDA OOM error.\n\nCheers,\n\nEPB", "votes": 2, "replies": [ { "id": 413950, "postDate": "2018-11-01T21:10:45.777Z", "content": "Hi Eric!
\n\nNot sure if that is what is happening here, but I think I sometimes see variability in epoch times if my computer is working on more than one thing. I don't mean extreme cases where I max out the CPU cores, but even if it uses some of the capacity for some other task the increase in epoch duration seems disproportionate. Here my computer was just working on this single task and reading data off the NVME drive. A bit of a far stretch but maybe this played into the effect a little bit (could be many other things as well, I think my GPU has slightly higher clock speeds, maybe I just got lucky with the algorithm it chose, etc). I am still on Ryzen 5 and a 1080TI btw.
\n\nAnyhow, glad you are finding this useful :)
\n\nAll the best,
\n\nRadek
", "rawMarkdown": "Hi Eric!\n\nNot sure if that is what is happening here, but I think I sometimes see variability in epoch times if my computer is working on more than one thing. I don't mean extreme cases where I max out the CPU cores, but even if it uses some of the capacity for some other task the increase in epoch duration seems disproportionate. Here my computer was just working on this single task and reading data off the NVME drive. A bit of a far stretch but maybe this played into the effect a little bit (could be many other things as well, I think my GPU has slightly higher clock speeds, maybe I just got lucky with the algorithm it chose, etc). I am still on Ryzen 5 and a 1080TI btw.\n\nAnyhow, glad you are finding this useful :)\n\nAll the best,\n\nRadek ", "votes": 1 }, { "id": 413958, "postDate": "2018-11-01T21:30:58.087Z", "content": "It's an amazing job you did here.
\n\nCrunching the dirty work of preprocessing the data in a clear and transparent way (so we can alter it as well, changing image_size=128 or dataset_size=1pc), to the point of submitting via Kaggle CLI all inclusive: it opens a whole world of options to focus and play with FastaiV1 documentation to finetune the model, the learning rate, the number of epochs and the rest.
\n\nSeriously: bravo !
", "rawMarkdown": "It's an amazing job you did here.\n\nCrunching the dirty work of preprocessing the data in a clear and transparent way (so we can alter it as well, changing image_size=128 or dataset_size=1pc), to the point of submitting via Kaggle CLI all inclusive: it opens a whole world of options to focus and play with FastaiV1 documentation to finetune the model, the learning rate, the number of epochs and the rest.\n\n\nSeriously: bravo !", "votes": 6 } ] }, { "id": 412247, "postDate": "2018-10-29T21:05:34.663Z", "content": "It seems that training with unrecognized images included is better (also, based on validation / LB gap I suspect test includes unrecognized images).
\n\nI was able to get to 0.882 after training resnet34 on 2 million of images of size 64x64. Expect another increase in performance with training on 128x128.
", "rawMarkdown": "It seems that training with unrecognized images included is better (also, based on validation / LB gap I suspect test includes unrecognized images).\n\nI was able to get to 0.882 after training resnet34 on 2 million of images of size 64x64. Expect another increase in performance with training on 128x128. ", "votes": 2 }, { "id": 411946, "postDate": "2018-10-29T09:22:16.227Z", "content": "Hey everybody. Since we are converting the arrays in csv to .png files and saving to disk,if any one of you get out-of-space error, there might be 2 reasons for that.
\n1. Your inodes have reached 2x10^7 entries. In short, just use mkefs /dev/sXX -N 200000000(8 zeros). Suggested by @radek . See full post here - link.
\n2. You are literally out of space. Dude you gotta increase your disk size. :)\nThanks
To add to the useful link-incase you get an error complaining: \"/dev/sXX is mounted; will not make a filesystem here!\"
\n\nUmount it using \"umount /dev/sXX\" and then run again-wont work if the same partition is the where the OS resides.
\n\nEdit: Another caveat worth mentioning is: \nIf the mount is inside your /home/mount path, it may cause problems with mounting.
", "rawMarkdown": "To add to the useful link-incase you get an error complaining: \"/dev/sXX is mounted; will not make a filesystem here!\"\n\nUmount it using \"umount /dev/sXX\" and then run again-wont work if the same partition is the where the OS resides. \n\nEdit: Another caveat worth mentioning is: \nIf the mount is inside your /home/mount path, it may cause problems with mounting. ", "votes": 1 } ] }, { "id": 408955, "postDate": "2018-10-23T16:40:32.973Z", "content": "Beautiful!
\nI am going to create one. Thank for your sharing
What base model are you using?
", "rawMarkdown": "What base model are you using? " } ] }, { "id": 432273, "postDate": "2018-12-03T16:41:32.540Z", "content": "Jeremy recommends using imagenet stats if we use imageNet pretrained models.\nhttps://forums.fast.ai/t/normalize-for-pretrained-false-resnet/28963/16?u=hwasiti
\n\nI tried both its own dataset stats and imagenet stats and the latter gave me a slightly better performance.
", "rawMarkdown": "Jeremy recommends using imagenet stats if we use imageNet pretrained models.\nhttps://forums.fast.ai/t/normalize-for-pretrained-false-resnet/28963/16?u=hwasiti\n\nI tried both its own dataset stats and imagenet stats and the latter gave me a slightly better performance. " }, { "id": 418218, "postDate": "2018-11-09T13:48:12.197Z", "content": "So, is anyone try building the model on gcp? basically with fastai basic installation using gpu p100 instead of p4. If so, how long does training take? Will the training be faster if using ssd instead of using standard hdd?
", "rawMarkdown": "So, is anyone try building the model on gcp? basically with fastai basic installation using gpu p100 instead of p4. If so, how long does training take? Will the training be faster if using ssd instead of using standard hdd?", "replies": [ { "id": 418309, "postDate": "2018-11-09T16:17:47.410Z", "content": "For me, for 10% of the data, it takes around 10-12 hours, but because it's preemptive, there are high chances that your connection may be terminated in the mean time. That's for one cycle. Yes, SSD is way for more faster, and it's definitely going to help. Not sure how much time difference will it make though.
", "rawMarkdown": "For me, for 10% of the data, it takes around 10-12 hours, but because it's preemptive, there are high chances that your connection may be terminated in the mean time. That's for one cycle. Yes, SSD is way for more faster, and it's definitely going to help. Not sure how much time difference will it make though." } ] }, { "id": 417727, "postDate": "2018-11-08T17:38:44.100Z", "content": "I used your starter code with 10% data and size of 64X64. Compared to the error rate of 0.22 using your starter code with 1% data and size of 128X128, I got error rate is about 0.84. Why? More data did not help and got much worse.
", "rawMarkdown": "I used your starter code with 10% data and size of 64X64. Compared to the error rate of 0.22 using your starter code with 1% data and size of 128X128, I got error rate is about 0.84. Why? More data did not help and got much worse. ", "replies": [ { "id": 417752, "postDate": "2018-11-08T18:15:49.057Z", "content": "I have quite consistently seen improvement in performance with increased image size so likely the issue is somewhere else.
\n\nIt could be anything. Too high of a learning rate given smaller batch size, some issues with sampling (maybe not all classes are equally represented), too little training with bigger images, etc.
", "rawMarkdown": "I have quite consistently seen improvement in performance with increased image size so likely the issue is somewhere else. \n\nIt could be anything. Too high of a learning rate given smaller batch size, some issues with sampling (maybe not all classes are equally represented), too little training with bigger images, etc. ", "votes": 1 }, { "id": 417815, "postDate": "2018-11-08T20:19:12.133Z", "content": "I'm guessing maybe you need to train for more epochs considering maybe you model still underfit. More data and larger image size definitely lead to better result.
", "rawMarkdown": "I'm guessing maybe you need to train for more epochs considering maybe you model still underfit. More data and larger image size definitely lead to better result." } ] }, { "id": 415653, "postDate": "2018-11-05T12:58:21.300Z", "content": "Is there any faster way to getting all images, we can either use generator, but not sure if ImageDataBunch supports it yet? Even after multi threading, it's gonna take a lot of time.
Thank you so much for your starter pack. How many epochs do you use for training?
", "rawMarkdown": "Thank you so much for your starter pack. How many epochs do you use for training? ", "replies": [ { "id": 414636, "postDate": "2018-11-03T08:31:37.727Z", "content": "I don't recall now and unable to check as I made some changes to the fastai library locally which would impact this, but I trained my best performing model for ~300 epochs with each epoch being either 50k or 200k images (that is the part I am unsure of).
", "rawMarkdown": "I don't recall now and unable to check as I made some changes to the fastai library locally which would impact this, but I trained my best performing model for ~300 epochs with each epoch being either 50k or 200k images (that is the part I am unsure of).", "votes": 1 }, { "id": 417720, "postDate": "2018-11-08T17:24:29.277Z", "content": "Thank you radek. I am not very clear about your epochs. Do you mean that changing learn.fit_one_cycle(4) to learn.fit_one_cycle(300), or learn.fit(300)? Many thanks.
", "rawMarkdown": "Thank you radek. I am not very clear about your epochs. Do you mean that changing learn.fit_one_cycle(4) to learn.fit_one_cycle(300), or learn.fit(300)? Many thanks.\n" }, { "id": 417818, "postDate": "2018-11-08T20:21:32.817Z", "content": "May I know how long does it take for 300 epochs? Because I try for 10 epochs for 10% of the data at 128x128 and it took nearly one whole day.
", "rawMarkdown": "May I know how long does it take for 300 epochs? Because I try for 10 epochs for 10% of the data at 128x128 and it took nearly one whole day." }, { "id": 417834, "postDate": "2018-11-08T21:06:00.440Z", "content": "My epochs were smaller. I am unable to tell now but each epoch was either 50k or 200k examples. I also progressively resized images as I trained (first trained on 64x64, than 128x128 and finally 256x256) but not sure if this is helping or not.
\n\nWith 200k examples of size 256x256 an epoch takes ~10 minutes for me (I have a single 1080TI)
", "rawMarkdown": "My epochs were smaller. I am unable to tell now but each epoch was either 50k or 200k examples. I also progressively resized images as I trained (first trained on 64x64, than 128x128 and finally 256x256) but not sure if this is helping or not.\n\nWith 200k examples of size 256x256 an epoch takes ~10 minutes for me (I have a single 1080TI)" }, { "id": 417836, "postDate": "2018-11-08T21:09:55.327Z", "content": "Yes, learn.fit_one_cycle(...) takes epoch count as argument. Training on bigger images might require more training and also each batch will most likely take longer to train.
I don't understand your idea of training on resized images progressively. Did you use one dateset, such as 256X256, or you have to prepare three datasets with 64X64, 128X128 and 256X256? Actually, when I learn the fastai course, I didn't fully understand how it works. I posted a question on the forum, but no one answers my question.
", "rawMarkdown": "I don't understand your idea of training on resized images progressively. Did you use one dateset, such as 256X256, or you have to prepare three datasets with 64X64, 128X128 and 256X256? Actually, when I learn the fastai course, I didn't fully understand how it works. I posted a question on the forum, but no one answers my question. " }, { "id": 418426, "postDate": "2018-11-09T22:06:23.153Z", "content": "With the same model, I first trained on 64x64. Once trained, I than trained on 128x128. Finally, I trained the same model on 256x256.
\n\nSuch strategies generally fall under the umbrella of curriculum learning but why they work and to what extent they work on various problems is an open question.
\n\nYou need to create datasets of the sizes you would like to train on, but in reality this approach is just a minor aesthetic detail in the larger scheme of things. It should have very little impact on the end result assuming you can get the major components of training right (how long to train, how to decay the learning rate, etc).
", "rawMarkdown": "With the same model, I first trained on 64x64. Once trained, I than trained on 128x128. Finally, I trained the same model on 256x256. \n\nSuch strategies generally fall under the umbrella of curriculum learning but why they work and to what extent they work on various problems is an open question. \n\nYou need to create datasets of the sizes you would like to train on, but in reality this approach is just a minor aesthetic detail in the larger scheme of things. It should have very little impact on the end result assuming you can get the major components of training right (how long to train, how to decay the learning rate, etc)." }, { "id": 419772, "postDate": "2018-11-12T14:35:28.417Z", "content": "@radek when you do progressive resizing, how do you distribute your training time to the different sizes? Like if you're doing 300 epochs total, do you do 100 epochs on 64x64, 100 on 128x128, and 100 on 256x256? Or do you do more on the larger sizes?
", "rawMarkdown": "@radek when you do progressive resizing, how do you distribute your training time to the different sizes? Like if you're doing 300 epochs total, do you do 100 epochs on 64x64, 100 on 128x128, and 100 on 256x256? Or do you do more on the larger sizes?" }, { "id": 419974, "postDate": "2018-11-12T21:31:24.150Z", "content": "Hey Will!
\n\nI think that the way I trained the model is far from ideal. I used the callbacks for reducing the LR on plateau and for early stopping. I trained for 12 epochs with one cycle and the conv layers frozen. Once I unfroze the model, I trained for 99 epochs on size 64x64, 74 epochs on 128x128 and 91 on 256x256. I think that epochs up to size 128 might have consisted of 200k drawings, but once I moved up to 256 I think I switched to 50k drawings per epoch. Don't have a way of checking this now.
\n\nI am convinced that with one cycle this training could be made much shorter. Training with these callbacks though allows me to gather some information about the LR range to use.
\n\nI have not checked whether progressive resizing helps on this problem or not.
", "rawMarkdown": "Hey Will!\n\nI think that the way I trained the model is far from ideal. I used the callbacks for reducing the LR on plateau and for early stopping. I trained for 12 epochs with one cycle and the conv layers frozen. Once I unfroze the model, I trained for 99 epochs on size 64x64, 74 epochs on 128x128 and 91 on 256x256. I think that epochs up to size 128 might have consisted of 200k drawings, but once I moved up to 256 I think I switched to 50k drawings per epoch. Don't have a way of checking this now.\n\nI am convinced that with one cycle this training could be made much shorter. Training with these callbacks though allows me to gather some information about the LR range to use.\n\nI have not checked whether progressive resizing helps on this problem or not.", "votes": 2 } ] }, { "id": 411837, "postDate": "2018-10-29T05:12:54.590Z", "content": "From this thread, it's clear that size matters by a relatively big margin. So taking 25K images/class, which model would be best to go with (CNN) in your opinion,ResNet isn't helping much ? Also LSTM support can be added afterwards for stoke based classification.
", "rawMarkdown": "From this [thread](https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/69260#latest-410763), it's clear that size matters by a relatively big margin. So taking 25K images/class, which model would be best to go with (CNN) in your opinion,ResNet isn't helping much ? Also LSTM support can be added afterwards for stoke based classification. ", "replies": [ { "id": 412138, "postDate": "2018-10-29T16:30:31.543Z", "content": "MobileNet can achieve an relatively good LB score. There is an kernel by beluga by the same.
", "rawMarkdown": "MobileNet can achieve an relatively good LB score. There is an kernel by beluga by the same." } ] }, { "id": 411732, "postDate": "2018-10-28T22:44:32.830Z", "content": "Nice Work Radek!
\n\nJust out of curiosity: What made you decide to use the ratio 1% instead of using a fixed number of images (for example 1k or 10k) per class?
", "rawMarkdown": "Nice Work Radek!\n\nJust out of curiosity: What made you decide to use the ratio 1% instead of using a fixed number of images (for example 1k or 10k) per class?", "replies": [ { "id": 411768, "postDate": "2018-10-29T01:11:46.943Z", "content": "It's because while creating the subset from actual training data, we pass in a value (r in this case) during sampling. So this is not being done class wise.\nselected = df[df.recognized==True].sample(int(r * df.shape[0]))\nIs there any other way of choosing on class basis instead of fixed number using this approach?
In the case the distribution of classes in the test set follows the one in the train set, I assume this might be marginally better.
", "rawMarkdown": "In the case the distribution of classes in the test set follows the one in the train set, I assume this might be marginally better." } ] }, { "id": 411536, "postDate": "2018-10-28T12:09:16.560Z", "content": "Would love to know you guys opinion on the size of the image itself. Will it give better result training on bigger image? 256 instead of 128?
", "rawMarkdown": "Would love to know you guys opinion on the size of the image itself. Will it give better result training on bigger image? 256 instead of 128?", "replies": [ { "id": 411553, "postDate": "2018-10-28T13:07:08.997Z", "content": "Based on the initial experiments, I noticed some slightly better performance on the 256 image size when compared to 128.
", "rawMarkdown": "Based on the initial experiments, I noticed some slightly better performance on the 256 image size when compared to 128.", "votes": 1 } ] }, { "id": 419143, "postDate": "2018-11-11T11:38:09.880Z", "rawMarkdown": "", "isDeleted": true }, { "id": 409322, "postDate": "2018-10-24T05:58:05.293Z", "content": "Thanks for this
", "rawMarkdown": "Thanks for this", "votes": 1 } ], "comments": [ { "id": 412708, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-30T16:27:46.803000", "content": "The good news is that resnet34 trained on 2 million of 128x128 drawings achieves a score of 0.9. This is just below the bronze medal range.
\n\nAn even better news is that training on same quantity of data, but 256x256, gets a score of 0.907. Unfortunately fully training the model takes > 12 hrs. I could possibly bring the time down to ~8hrs but hard to say if that would have an adverse effect on performance or not.
\n\nEither way, that train time with resnet34 is depressingly long. I am not sure what to try next - it seems that ensembling should work here really well. At the same time bigger models should also give a better score - as expected there are no signs of overfitting with resnet34.
\n\nI just got the results a couple of minutes ago so it might take me a while to figure out what to make of the situation. Nonetheless, I think I'll start another train cycle with a bigger architecture just to see what it will do and will try to trade marginal improvements in performance for shorter train time.
", "votes": 9, "replies": [ { "id": 412733, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-10-30T17:37:27.120000", "content": "It might be worth to use smaller sized images with larger architectures-might allow room for speedy experimentation.
", "votes": 2, "replies": [] }, { "id": 412969, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-31T04:15:12.680000", "content": "Did you normalize (with imagenet stats) ?
", "votes": 0, "replies": [] }, { "id": 413024, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-31T06:50:22.780000", "content": "I use stats automatically calculated from one batch (this is what you get when you call db.normalize() without arguments).
", "votes": 0, "replies": [] }, { "id": 413210, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-31T13:59:43.150000", "content": "I tried with resnet34, with 256 size on 5% of data (around 2.5M), it's nowhere close to 0.9. Am i missing something here ? Even tried with Resnet50, still the same problem. Even after training on 10% of the data, it's not helping much.
", "votes": 1, "replies": [] }, { "id": 413225, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-31T14:26:54.917000", "content": "Maybe you need to train for longer? You could try to manually train for x epochs with some lr, say something like this:
\n\nlearn.fit(5, 1e-3)
and only lower the training rate once you hit a plateau. This can be done automatically with callbacks (you can find them here: fastai/callbacks/tracker.py), but it can be useful to go through the motions by hand a couple of times.
5 cycles before unfreezing ? How did you call ReduceLROnPlateau? It's not covered in docs yet. Do we pass it like metrics or call on learn ? Also CLR can handle the same functionality ?
Did you try using MobileNetV2 ?
", "votes": 0, "replies": [] }, { "id": 418455, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-09T23:10:56.737000", "content": "I have the same question on How you call ReduceLROnPlateau. Fastai's documentation is so hard to read and lack of examples which are using each functions.
", "votes": 0, "replies": [] }, { "id": 418467, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-10T00:14:39.240000", "content": "I have figured out how to call ReduceLROnPlateau.\nreduceLR = ReduceLROnPlateauCallback(learn=learn, monitor = 'val_loss', mode = 'auto', patience = 10, factor = 0.2, min_delta = 0)\nlearn.fit_one_cycle(20, max_lr=slice(5e-6,5e-4), callbacks=[reduceLR])
", "votes": 1, "replies": [] } ] }, { "id": 420494, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-13T17:59:22.183000", "content": "I updated the starter pack to use the data_block API. You can find the new version here.
\n\nI transitioned to using the data_block API. Drawings are now generated on the fly. Training with 128x128 drawings, 1000 examples per class, now achieves 0.856 on public LB in ~30 minutes of training on a single 1080TI.
", "votes": 7, "replies": [ { "id": 421065, "author_name": "sanghai", "author_url": "", "post_date": "2018-11-14T14:24:45.407000", "content": "Hi Radek,
\n\nThanks for sharing the code. What is the fastai version you are using? Right now, I am on 1.0.24. I am getting an error on InputList (name 'InputList' is not defined).
\n\nThanks
", "votes": 0, "replies": [] }, { "id": 421076, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-14T14:48:20.483000", "content": "@asanghai You should use 1.0.25, there was a big refactor of the data_block API in that version.
", "votes": 1, "replies": [] }, { "id": 421130, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-14T16:16:39.780000", "content": "I just pushed an updated version - this one works with the current fast.ai master.
", "votes": 1, "replies": [] }, { "id": 421150, "author_name": "sanghai", "author_url": "", "post_date": "2018-11-14T16:45:15.620000", "content": "Thanks William. I could not find 1.0.25 (https://pypi.org/project/fastai/#history https://anaconda.org/fastai/fastai/files). Can you please let me know how to get 1.0.25?
\n\nThanks Radek. Can you please let me know what fastai version you were using before?
", "votes": 0, "replies": [] }, { "id": 421167, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-14T16:56:09.103000", "content": "I am not sure - it was one from yesterday. For future reference, the version I am on right now is af068ecf9ba98c5c3383b59bb2a7d44b01337297.
@asanghai unfortunately it looks like 1.0.25 hasn’t been published yet. If you don’t want to wait, you can download the GitHub repo and follow the instructions for a local build (which is what I do to always have the latest code).
", "votes": 2, "replies": [] }, { "id": 422159, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-15T21:16:05.400000", "content": "When I run your v2, I got error \"name 'ItemList' is not defined\". What do you mean of data_block API? Do I need to download something instead of fastai library? Many thanks.
", "votes": 0, "replies": [] }, { "id": 422162, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-15T21:27:26.257000", "content": "@William Horton \nWhat do you mean \"local build\"? I couldn't find this item from the instruction of installing fastai library.
", "votes": 0, "replies": [] }, { "id": 422260, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-16T01:36:22.010000", "content": "@Xu Zhang i meant the section in the fastai README called “Developer Install”. It involves running pip install -e .[dev] within fastai once you’ve cloned it
", "votes": 0, "replies": [] }, { "id": 422294, "author_name": "miwojc", "author_url": "", "post_date": "2018-11-16T03:00:23.893000", "content": "thanks Radek!
\n\ni get error executing this line of code:
\n\n\ncreate_submission(preds, data_bunch.test_dl, name)\n
error:
\n\n\nNameError: name 'classes' is not defined\n
starter pack version: ab4cf72
\n\ni use fastai developer install, fastai version: 672c8c5
", "votes": 0, "replies": [] }, { "id": 422381, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-16T06:28:50.170000", "content": "Hi! The issue was that create_submission required the classes list to be defined. I now made it so that classes needs to be explicitly passed into the function so that should give an easier to understand error message.
The classes list gets loaded in cell #14 - please make sure you execute it before trying to create a submission file.
great thank you!\npossibly would be nice if the subs directory was created first as well...
Hi @radek,
\n\nWhat a fun starter pack to enjoy, again !
\n\nI'm currently on fastai 1.0.28, if not mistaken. Trying to have fun with a dedicated RTX 2080ti, beside my good old 1080Ti.
\n\nWhen I run your latest notebook, in the item_list = ItemList.from_folder(PATH/'train', create_func=create_func) cell, I get this error: any idea ?
Traceback (most recent call last)\n<ipython-input-14-611893917372> in <module>()\n----> 1 item_list = ItemList.from_folder(PATH/'train', create_func=create_func)\n\nAttributeError: type object 'ItemList' has no attribute 'from_folder'\nI think the API changed a bit in 1.0.28 and I haven't had a chance to update the starter pack yet. At this point one of the options would be figuring out what changed, but probably it might be easier to downgrade to 1.0.25 which the current version of the starter pack should work with.
", "votes": 1, "replies": [] }, { "id": 425685, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-22T00:43:07.557000", "content": "Hi Eric,\nI’ve run into these changes in my human protein atlas starter code as well. The two ways to fix would be:\n1) create a subclass of ItemList that overrides the open method to use the create func\n2) a more hacky approach, once you have your item list variable, do item list.open=create func and that should also work (had to remove underscores because it messes up comment formatting)
", "votes": 2, "replies": [] }, { "id": 426195, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-22T20:48:27.370000", "content": "Sorry, just realised my post had an incomplete copy/paste of the AttributeError so I updated it with the full version.
\n\n---------------------------------------------------------------------------\nAttributeError Traceback (most recent call last)\n<ipython-input-14-611893917372> in <module>()\n----> 1 item_list = ItemList.from_folder(PATH/'train', create_func=create_func)\n\nAttributeError: type object 'ItemList' has no attribute 'from_folder'\nJust got back to play around with your Starter Pack on 1.0.25: wow !\nNot only is it way way way faster that the original one but the score is really better from scratch \"as it is\".\nNow I need to figure out what is happening :-D\nAnd how to improve the score step by step.
\n\nYou dropped the TTA btw or you are using a different syntax ?
\n\n", "votes": 1, "replies": [] }, { "id": 428557, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-27T13:27:18.850000", "content": "I dropped the TTA - haven't experimented much with it though, not sure if it is useful or not.
", "votes": 0, "replies": [] }, { "id": 428749, "author_name": "miguel perez", "author_url": "", "post_date": "2018-11-27T20:29:42.903000", "content": "In theory there is no reason to use augmentation when enough data is available, like in this competition.
\n\nIf you think about it, no matter how clever augmentation can be, a new image from the true population will always be better. (Such a big dataset we have here...! ) :-)
", "votes": 4, "replies": [] }, { "id": 429112, "author_name": "Bhuvana Kundumani", "author_url": "", "post_date": "2018-11-28T11:04:33.587000", "content": "Hi I am trying with the starter pack. When i execute \nlabel_lists = item_lists.label_from_folder() - I am getting error IndexError: index 0 is out of bounds for axis 0 with size 0 . I am using fast ai Version: 1.0.29\nHow to download the fast ai version 1.0.25? Thanks for the help.
", "votes": 0, "replies": [] }, { "id": 429134, "author_name": "", "author_url": "", "post_date": "2018-11-28T11:51:25.970000", "content": "", "votes": 0, "replies": [] }, { "id": 429331, "author_name": "Bhuvana Kundumani", "author_url": "", "post_date": "2018-11-28T17:39:04.353000", "content": "I downloaded the version using -
\n\n!pip install fastai==1.0.25
", "votes": 1, "replies": [] }, { "id": 429396, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-28T19:54:43.193000", "content": "(I edited my post as I figured out the cause of the huge increase in epoch time, my own mistake).
\n\nAs I dig deeper into the different parameters, which helps me understand your approach, I'm now looking at your second cell \"NUM-SAMPLES-PER-CLASS = 1000; NUM-VAL = 50 * 340\".
\n\nI think I understand the first one \"Num-samples-per-class = 1000\" as the name is obvious, and easy to change/increase for better accuracy like 2000.
\n\nBut I'm not sure about the second one \"Num-val = 50 * 340\", apart from 340 being the # of categories.\nWhen I look at the later cell \"data_bunch.normalize(batch_stats)\", it shows \"CategoryList(323000 items)\".
\n\nThen if I modify to \"Num-val = 200 * 340\", the later cell shows \"CategoryList(272000 items)\".
\n\nI can't unlock the logic/relation behind 50x340=>323K and 200x340=>272K.
", "votes": 0, "replies": [] }, { "id": 429422, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-28T20:35:21.040000", "content": "Around 50 examples per class to have in the validation set was just an arbitrary choice.
\n\nThe bigger the val set (200 x 340), the fewer examples will be left to include in the train set.
", "votes": 1, "replies": [] }, { "id": 429441, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-28T21:12:33.447000", "content": "Thanks, you rock Sir !
\n\nI learn so much looking at your code and decoding it.
\n\n(BTW, loved your previous profile's picture, looking like a crazy helicopter pilot adjusting his goggles ^!^)
", "votes": 1, "replies": [] }, { "id": 469439, "author_name": "Arcile", "author_url": "", "post_date": "2019-02-11T08:18:25.177000", "content": "Sorry for the beginners question, but where is the create_submission() function imported from? I'm having issues even recognizing that function.
", "votes": 0, "replies": [] }, { "id": 469652, "author_name": "Arcile", "author_url": "", "post_date": "2019-02-11T15:57:26.500000", "content": "So to answer my own question, if someone else has the same question (I'm using a Google Cloud jupyter notebook).
\n\nTo get the create_submission function to work, you need to put the utils.py file (also in radeks repository) in the same folder as your notebook file (.ipynb). The create_submission() function is defined in utils.py
", "votes": 1, "replies": [] } ] }, { "id": 415828, "author_name": "miguel perez", "author_url": "", "post_date": "2018-11-05T19:08:59.727000", "content": "Great way of putting to test new Fastai v1, very nice pipeline, thanks for that Radek! :-)
", "votes": 3, "replies": [] }, { "id": 415765, "author_name": "Adilzhan Ismailov", "author_url": "", "post_date": "2018-11-05T16:52:50.197000", "content": "I've created a custom dataset that converts strokes to images on the fly and is compatible with fastai - you can find the baseline here, there is still a lot of room for imrovement.
", "votes": 3, "replies": [ { "id": 415778, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-05T17:16:08.130000", "content": "Thanks for sharing it. Was just curious, whenever the next batch is called only then images are being produced and fed or are they produced earlier ? Is this similar to using generator?
", "votes": 1, "replies": [] }, { "id": 415829, "author_name": "Adilzhan Ismailov", "author_url": "", "post_date": "2018-11-05T19:09:26.980000", "content": "I am not sure about a generator, but yes, here the images are produces when the batch is called
", "votes": 0, "replies": [] }, { "id": 416071, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-06T06:21:28.093000", "content": "I got this RuntimeError: DataLoader worker (pid 2157) is killed by signal: Bus error.
Hm, maybe try with smaller bs? I think it might be an issue with RAM
", "votes": 0, "replies": [] }, { "id": 416245, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-06T13:17:01.313000", "content": "No,this issue lies with Pytorch, using num_workers=0 eliminates the issue, but increases the training time by a factor of approx num_of_workers. It happens due to lack of shared memory. It hasn't been resolved on Pytorch issues section as well.
Indeed it seems that training with bigger images helps a bit. Another consideration would be to include unrecognized drawings. Currently, during data creation, I filter those out.
\n\nAlso, we use imagenet stats to normalize the data. If you don't pass the stats into the normalize method, the images will be normalized based on statistics calculated from a single batch.
Once you train more than one model, you can start thinking of ensembling. I haven't tried it yet for this competition, but it is a powerful technique that can move you up a lot of places. Here is a really great blog post summarizing a couple of approaches. It is a good idea to start with averaging results and use this as a baseline - might be more advanced techniques are not necessary or can provide only small gains.
", "votes": 4, "replies": [ { "id": 411651, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-28T17:10:18.490000", "content": "I have trained model on 10% data,but still it can't cross 0.9 on LB. Was thinking of adding custom head or making changes with ResNet. Increasing parameters with well defined arch will definitely help. Also not sure whether loss function being used in ConvLearner is the best choice for this.
Your link to the blog post is broken.
\n\nThanks for sharing and the wonderful kernel. :)
", "votes": 1, "replies": [] }, { "id": 411950, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-29T09:27:26.627000", "content": "Thanks :) Should be fixed now.
", "votes": 1, "replies": [] } ] }, { "id": 424581, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-20T11:03:31.350000", "content": "To stay true to the 'no sharing outside of teams rule' - and also because I think it might actually be useful to other folks using this starter pack - I share here on the fastai forums how to train on epoch_sizes of arbitrary size.
\n\nIt's more of a technicality on how to use the framework but is a really nice feature and might be of use to some.
\n\nHere is the code:\n```\ndef chunks(l, n):\n \"\"\"Yield successive n-sized chunks from l.\"\"\"\n for i in range(0, len(l), n):\n yield l[i:i + n]
\n\nclass RandomSamplerWithEpochSize(Sampler):\n \"\"\"Yields epochs of specified sizes. Iterates over all examples in a data_source in random\n order. Ensures (nearly) all examples have been trained on before beginning the next iteration\n over the data_source - drops the last epoch that would likely be smaller than epoch_size.\n \"\"\"\n def init(self, data_source, epoch_size):\n self.n = len(data_source)\n self.epoch_size = epoch_size\n self._epochs = []\n def iter(self):\n return iter(self.next_epoch)\n @property\n def next_epoch(self):\n if len(self._epochs) == 0: self.generate_epochs()\n return self._epochs.pop()\n def generate_epochs(self):\n idxs = [i for i in range(self.n)]\n np.random.shuffle(idxs)\n self._epochs = list(chunks(idxs, self.epoch_size))[:-1]\n def len(self):\n return self.epoch_size\n```
\n\nAnd here is how to use it to create the data_bunch:\n```\ntrain_dl = DataLoader(\n label_lists.train,\n num_workers=12,\n batch_sampler=BatchSampler(RandomSamplerWithEpochSize(label_lists.train, 200_000), bs, True)\n)\nvalid_dl = DataLoader(label_lists.valid, 2*bs, False, num_workers=12)\ntest_dl = DataLoader(label_lists.test, 2*bs, False, num_workers=12)
data_bunch = ImageDataBunch(train_dl, valid_dl, test_dl)\n```
", "votes": 1, "replies": [] }, { "id": 422396, "author_name": "dromosys", "author_url": "", "post_date": "2018-11-16T07:00:16.737000", "content": "Awesome thanks. I had a go at making it into a kernel but ran out of disk space. work in progress..
\n\nhttps://www.kaggle.com/dromosys/fast-ai-quick-draw/
", "votes": 1, "replies": [ { "id": 422430, "author_name": "Abhilash Awasthi", "author_url": "", "post_date": "2018-11-16T08:05:45.293000", "content": "This code uses fastai v1..but on kernel v0.7 is installed. I am afraid you will have to make lot of changes in the code to make it run on kernel.
", "votes": 1, "replies": [] }, { "id": 422790, "author_name": "dromosys", "author_url": "", "post_date": "2018-11-16T19:29:42.073000", "content": "nope fastai version 1.0.26.dev0 is installed via !pip3 install git+https://github.com/fastai/fastai.git
", "votes": 0, "replies": [] }, { "id": 422837, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-16T21:14:38.647000", "content": "Have you managed to update pytorch as well? The docs say fastai only works with pytorch 1.0. Would love to see fastai 1.0 working on Kernels, I think it’s a great platform for learning.
", "votes": 0, "replies": [] }, { "id": 422971, "author_name": "", "author_url": "", "post_date": "2018-11-17T06:46:25.610000", "content": "", "votes": 1, "replies": [] }, { "id": 423657, "author_name": "dromosys", "author_url": "", "post_date": "2018-11-18T19:55:43.653000", "content": "you're right pytorch didn't update automatically. manually update worked with GPU off but not with GPU enabled. https://www.kaggle.com/dromosys/fast-ai-tiny-planet/
", "votes": 0, "replies": [] } ] }, { "id": 413238, "author_name": "Nazim Girach", "author_url": "", "post_date": "2018-10-31T14:56:59.807000", "content": "Hey has anyone able to get convert more than 10% of their dataset to images? It's the max I could convert. Anyone facing the same issue?
\n\nAnd using strokes directly as done in 🐘Greyscale MobileNet [LB=0.892] a better approach?
\n\nThanks, Radek for sharing your code! It was super helpful.
", "votes": 1, "replies": [] }, { "id": 412801, "author_name": "Ikenna Chifo", "author_url": "", "post_date": "2018-10-30T19:50:12.290000", "content": "Thanks for sharing. Already looks good from my initial run.
", "votes": 1, "replies": [ { "id": 413104, "author_name": "wqk", "author_url": "", "post_date": "2018-10-31T09:18:59.643000", "content": "Hi, what's the version of fastai, pytorch? I use 1.0.18, but ConvLearner cannot be found for the reason not defined.
", "votes": 0, "replies": [] }, { "id": 413108, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-31T09:31:58.997000", "content": "Recently ConvLearner has become create_cnn. Refer to docs.
Seems that this was introduced in 1.0.15. I think you should be good to use anything prior to this. Also, I believe that just changing ConvLearner(...) to create_cnn might work also.
Please let us know if you get it to work!
", "votes": 0, "replies": [] }, { "id": 413295, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-10-31T16:35:15.767000", "content": "I can confirm that just changing ConvLearner(...) and replacing it with create_cnn works just fine everywhere.
Thanks for sharing this! \nI tried submitting with your stater code and with an increase in data I noticed some gain so I decided to train the model on the complete datset after extracting the complete dataset on disk. \nI'm trying to give this approach a shot instead of trying other approaches (About a month of runway until the deadline which is slightly less given the data size)
", "votes": 1, "replies": [ { "id": 411539, "author_name": "Fadhli", "author_url": "", "post_date": "2018-10-28T12:15:17.967000", "content": "May I know how much of the data you use to train?
", "votes": 0, "replies": [] }, { "id": 411554, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-10-28T13:08:30.817000", "content": "I've tried 1, 5 % so far.
", "votes": 1, "replies": [] }, { "id": 412048, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-10-29T13:16:58.157000", "content": "I'm not sure what went wrong here. But training the model on the complete data made the model perform worse than the one on 1% data. (Used the same approach, just on 100% of the data)
", "votes": 1, "replies": [] }, { "id": 412050, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-29T13:28:25.417000", "content": "Was the difference big or only slight? Might be including the unrecognized drawings does not help (maybe they are of very poor quality?)
", "votes": 1, "replies": [] }, { "id": 412084, "author_name": "", "author_url": "", "post_date": "2018-10-29T14:48:57.440000", "content": "", "votes": 0, "replies": [] }, { "id": 414727, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-03T13:23:28.780000", "content": "I'm trying with 5% at size=64, it runs all fine like 1% until preds = learn.TTA() in Generate Submission part where it gets an error object of type \"NoneType\" has no len().\nNeed to look into the docs :-)
NVM, it was just a spelling mistake in the path for the ../test_simp... in data=
I added my solution based on the starter pack to the repository. The model in itself is nothing special (it is a combination of 2 cnns, rnn and country code embeddings) but the way the data is generated and fed to the model and a couple of other things (epochs of arbitrary size, changing the classifier during training, etc) might potentially be of interest.
\n\nHere is a link to a Twitter thread where I go into some additional details.
", "votes": 2, "replies": [] }, { "id": 413877, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-01T17:33:09.693000", "content": "Hi Radek !
\n\nJust started your Starter Pack, quite an amazing surprise, many thanks.
\n\nBTW I think I read in one of your posts on Kaggle that you use a Ryzen 5 + 1080Ti, I got almost the same setup with a Ryzen 7 1700X but\n- (1) my epochs are about 10-20% slower than yours and\n- (2) I later had to lower batch size to 48 (default 64) for the unfreeze part to avoid CUDA OOM error.
Cheers,
\n\nEPB
", "votes": 2, "replies": [ { "id": 413950, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-01T21:10:45.777000", "content": "Hi Eric!
\n\nNot sure if that is what is happening here, but I think I sometimes see variability in epoch times if my computer is working on more than one thing. I don't mean extreme cases where I max out the CPU cores, but even if it uses some of the capacity for some other task the increase in epoch duration seems disproportionate. Here my computer was just working on this single task and reading data off the NVME drive. A bit of a far stretch but maybe this played into the effect a little bit (could be many other things as well, I think my GPU has slightly higher clock speeds, maybe I just got lucky with the algorithm it chose, etc). I am still on Ryzen 5 and a 1080TI btw.
\n\nAnyhow, glad you are finding this useful :)
\n\nAll the best,
\n\nRadek
", "votes": 1, "replies": [] }, { "id": 413958, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-11-01T21:30:58.087000", "content": "It's an amazing job you did here.
\n\nCrunching the dirty work of preprocessing the data in a clear and transparent way (so we can alter it as well, changing image_size=128 or dataset_size=1pc), to the point of submitting via Kaggle CLI all inclusive: it opens a whole world of options to focus and play with FastaiV1 documentation to finetune the model, the learning rate, the number of epochs and the rest.
\n\nSeriously: bravo !
", "votes": 6, "replies": [] } ] }, { "id": 412247, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-10-29T21:05:34.663000", "content": "It seems that training with unrecognized images included is better (also, based on validation / LB gap I suspect test includes unrecognized images).
\n\nI was able to get to 0.882 after training resnet34 on 2 million of images of size 64x64. Expect another increase in performance with training on 128x128.
", "votes": 2, "replies": [] }, { "id": 411946, "author_name": "VikasSangwan", "author_url": "", "post_date": "2018-10-29T09:22:16.227000", "content": "Hey everybody. Since we are converting the arrays in csv to .png files and saving to disk,if any one of you get out-of-space error, there might be 2 reasons for that.
\n1. Your inodes have reached 2x10^7 entries. In short, just use mkefs /dev/sXX -N 200000000(8 zeros). Suggested by @radek . See full post here - link.
\n2. You are literally out of space. Dude you gotta increase your disk size. :)\nThanks
To add to the useful link-incase you get an error complaining: \"/dev/sXX is mounted; will not make a filesystem here!\"
\n\nUmount it using \"umount /dev/sXX\" and then run again-wont work if the same partition is the where the OS resides.
\n\nEdit: Another caveat worth mentioning is: \nIf the mount is inside your /home/mount path, it may cause problems with mounting.
", "votes": 1, "replies": [] } ] }, { "id": 408955, "author_name": "cab", "author_url": "", "post_date": "2018-10-23T16:40:32.973000", "content": "Beautiful!
\nI am going to create one. Thank for your sharing
What base model are you using?
", "votes": 0, "replies": [] } ] }, { "id": 432273, "author_name": "Haider Alwasiti", "author_url": "", "post_date": "2018-12-03T16:41:32.540000", "content": "Jeremy recommends using imagenet stats if we use imageNet pretrained models.\nhttps://forums.fast.ai/t/normalize-for-pretrained-false-resnet/28963/16?u=hwasiti
\n\nI tried both its own dataset stats and imagenet stats and the latter gave me a slightly better performance.
", "votes": 0, "replies": [] }, { "id": 418218, "author_name": "Fadhli", "author_url": "", "post_date": "2018-11-09T13:48:12.197000", "content": "So, is anyone try building the model on gcp? basically with fastai basic installation using gpu p100 instead of p4. If so, how long does training take? Will the training be faster if using ssd instead of using standard hdd?
", "votes": 0, "replies": [ { "id": 418309, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-09T16:17:47.410000", "content": "For me, for 10% of the data, it takes around 10-12 hours, but because it's preemptive, there are high chances that your connection may be terminated in the mean time. That's for one cycle. Yes, SSD is way for more faster, and it's definitely going to help. Not sure how much time difference will it make though.
", "votes": 0, "replies": [] } ] }, { "id": 417727, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-08T17:38:44.100000", "content": "I used your starter code with 10% data and size of 64X64. Compared to the error rate of 0.22 using your starter code with 1% data and size of 128X128, I got error rate is about 0.84. Why? More data did not help and got much worse.
", "votes": 0, "replies": [ { "id": 417752, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-08T18:15:49.057000", "content": "I have quite consistently seen improvement in performance with increased image size so likely the issue is somewhere else.
\n\nIt could be anything. Too high of a learning rate given smaller batch size, some issues with sampling (maybe not all classes are equally represented), too little training with bigger images, etc.
", "votes": 1, "replies": [] }, { "id": 417815, "author_name": "Fadhli", "author_url": "", "post_date": "2018-11-08T20:19:12.133000", "content": "I'm guessing maybe you need to train for more epochs considering maybe you model still underfit. More data and larger image size definitely lead to better result.
", "votes": 0, "replies": [] } ] }, { "id": 415653, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-05T12:58:21.300000", "content": "Is there any faster way to getting all images, we can either use generator, but not sure if ImageDataBunch supports it yet? Even after multi threading, it's gonna take a lot of time.
Thank you so much for your starter pack. How many epochs do you use for training?
", "votes": 0, "replies": [ { "id": 414636, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-03T08:31:37.727000", "content": "I don't recall now and unable to check as I made some changes to the fastai library locally which would impact this, but I trained my best performing model for ~300 epochs with each epoch being either 50k or 200k images (that is the part I am unsure of).
", "votes": 1, "replies": [] }, { "id": 417720, "author_name": "Xu Zhang", "author_url": "", "post_date": "2018-11-08T17:24:29.277000", "content": "Thank you radek. I am not very clear about your epochs. Do you mean that changing learn.fit_one_cycle(4) to learn.fit_one_cycle(300), or learn.fit(300)? Many thanks.
", "votes": 0, "replies": [] }, { "id": 417818, "author_name": "Fadhli", "author_url": "", "post_date": "2018-11-08T20:21:32.817000", "content": "May I know how long does it take for 300 epochs? Because I try for 10 epochs for 10% of the data at 128x128 and it took nearly one whole day.
", "votes": 0, "replies": [] }, { "id": 417834, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-08T21:06:00.440000", "content": "My epochs were smaller. I am unable to tell now but each epoch was either 50k or 200k examples. I also progressively resized images as I trained (first trained on 64x64, than 128x128 and finally 256x256) but not sure if this is helping or not.
\n\nWith 200k examples of size 256x256 an epoch takes ~10 minutes for me (I have a single 1080TI)
", "votes": 0, "replies": [] }, { "id": 417836, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-08T21:09:55.327000", "content": "Yes, learn.fit_one_cycle(...) takes epoch count as argument. Training on bigger images might require more training and also each batch will most likely take longer to train.
I don't understand your idea of training on resized images progressively. Did you use one dateset, such as 256X256, or you have to prepare three datasets with 64X64, 128X128 and 256X256? Actually, when I learn the fastai course, I didn't fully understand how it works. I posted a question on the forum, but no one answers my question.
", "votes": 0, "replies": [] }, { "id": 418426, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-09T22:06:23.153000", "content": "With the same model, I first trained on 64x64. Once trained, I than trained on 128x128. Finally, I trained the same model on 256x256.
\n\nSuch strategies generally fall under the umbrella of curriculum learning but why they work and to what extent they work on various problems is an open question.
\n\nYou need to create datasets of the sizes you would like to train on, but in reality this approach is just a minor aesthetic detail in the larger scheme of things. It should have very little impact on the end result assuming you can get the major components of training right (how long to train, how to decay the learning rate, etc).
", "votes": 0, "replies": [] }, { "id": 419772, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-12T14:35:28.417000", "content": "@radek when you do progressive resizing, how do you distribute your training time to the different sizes? Like if you're doing 300 epochs total, do you do 100 epochs on 64x64, 100 on 128x128, and 100 on 256x256? Or do you do more on the larger sizes?
", "votes": 0, "replies": [] }, { "id": 419974, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2018-11-12T21:31:24.150000", "content": "Hey Will!
\n\nI think that the way I trained the model is far from ideal. I used the callbacks for reducing the LR on plateau and for early stopping. I trained for 12 epochs with one cycle and the conv layers frozen. Once I unfroze the model, I trained for 99 epochs on size 64x64, 74 epochs on 128x128 and 91 on 256x256. I think that epochs up to size 128 might have consisted of 200k drawings, but once I moved up to 256 I think I switched to 50k drawings per epoch. Don't have a way of checking this now.
\n\nI am convinced that with one cycle this training could be made much shorter. Training with these callbacks though allows me to gather some information about the LR range to use.
\n\nI have not checked whether progressive resizing helps on this problem or not.
", "votes": 2, "replies": [] } ] }, { "id": 411837, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-29T05:12:54.590000", "content": "From this thread, it's clear that size matters by a relatively big margin. So taking 25K images/class, which model would be best to go with (CNN) in your opinion,ResNet isn't helping much ? Also LSTM support can be added afterwards for stoke based classification.
", "votes": 0, "replies": [ { "id": 412138, "author_name": "Rajath", "author_url": "", "post_date": "2018-10-29T16:30:31.543000", "content": "MobileNet can achieve an relatively good LB score. There is an kernel by beluga by the same.
", "votes": 0, "replies": [] } ] }, { "id": 411732, "author_name": "Alexander Schumacher", "author_url": "", "post_date": "2018-10-28T22:44:32.830000", "content": "Nice Work Radek!
\n\nJust out of curiosity: What made you decide to use the ratio 1% instead of using a fixed number of images (for example 1k or 10k) per class?
", "votes": 0, "replies": [ { "id": 411768, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-10-29T01:11:46.943000", "content": "It's because while creating the subset from actual training data, we pass in a value (r in this case) during sampling. So this is not being done class wise.\nselected = df[df.recognized==True].sample(int(r * df.shape[0]))\nIs there any other way of choosing on class basis instead of fixed number using this approach?
In the case the distribution of classes in the test set follows the one in the train set, I assume this might be marginally better.
", "votes": 0, "replies": [] } ] }, { "id": 411536, "author_name": "Fadhli", "author_url": "", "post_date": "2018-10-28T12:09:16.560000", "content": "Would love to know you guys opinion on the size of the image itself. Will it give better result training on bigger image? 256 instead of 128?
", "votes": 0, "replies": [ { "id": 411553, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-10-28T13:07:08.997000", "content": "Based on the initial experiments, I noticed some slightly better performance on the 256 image size when compared to 128.
", "votes": 1, "replies": [] } ] }, { "id": 419143, "author_name": "", "author_url": "", "post_date": "2018-11-11T11:38:09.880000", "content": "", "votes": 0, "replies": [] }, { "id": 409322, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-10-24T05:58:05.293000", "content": "Thanks for this
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "408924": "Please find the starter pack on [github][1]. I have not used kernels yet - if someone would find this useful and would want to transfer it over to a kernel, please feel free to go ahead.\n\nThis uses the brand new [FastAi v1 library][2]. I trained with defaults, including default hyperparameter values. I trained on 1% of data of size 128x128. The training was probably done using SGD (with one cycle policy) but I have not checked, neither have I looked at the batch size. Of course, that means there is a lot of room to improve (also in terms of hw utilization) but wanted this to be as vanilla as it gets. Also wanted to start getting a feel for the use of the library in the way the authors envisioned it.\n\nTurns out the library supports TTA out of the box so I predicted with TTA using again the provided defaults.\n\n**This code is based on code from a fast.ai MOOC that will be publicly available in Jan 2019.** \n\nEDIT: The default optimization algorithm with fit_one_cycle turns out to be Adam.\n\n [1]: https://github.com/radekosmulski/quickdraw\n [2]: https://github.com/fastai/fastai", "412708": "The good news is that resnet34 trained on 2 million of 128x128 drawings achieves a score of 0.9. This is just below the bronze medal range.\n\nAn even better news is that training on same quantity of data, but 256x256, gets a score of 0.907. Unfortunately fully training the model takes > 12 hrs. I could possibly bring the time down to ~8hrs but hard to say if that would have an adverse effect on performance or not. \n\nEither way, that train time with resnet34 is depressingly long. I am not sure what to try next - it seems that ensembling should work here really well. At the same time bigger models should also give a better score - as expected there are no signs of overfitting with resnet34. \n\nI just got the results a couple of minutes ago so it might take me a while to figure out what to make of the situation. Nonetheless, I think I'll start another train cycle with a bigger architecture just to see what it will do and will try to trade marginal improvements in performance for shorter train time.", "420494": "I updated the starter pack to use the data_block API. You can find the new version [here](https://github.com/radekosmulski/quickdraw).\n\nI transitioned to using the data_block API. Drawings are now generated on the fly. Training with 128x128 drawings, 1000 examples per class, now achieves 0.856 on public LB in ~30 minutes of training on a single 1080TI.", "415828": "Great way of putting to test new Fastai v1, very nice pipeline, thanks for that Radek! :-)", "415765": "I've created a custom dataset that converts strokes to images on the fly and is compatible with fastai - you can find the baseline [here](https://github.com/adilism/quickdraw/blob/master/fastai-baseline.ipynb), there is still a lot of room for imrovement.", "411606": "Indeed it seems that training with bigger images helps a bit. Another consideration would be to include unrecognized drawings. Currently, during data creation, I filter those out.\n\nAlso, we use imagenet stats to normalize the data. If you don't pass the stats into the `normalize` method, the images will be normalized based on statistics calculated from a single batch. \n\nOnce you train more than one model, you can start thinking of ensembling. I haven't tried it yet for this competition, but it is a powerful technique that can move you up a lot of places. Here is [a really great blog post][1] summarizing a couple of approaches. It is a good idea to start with averaging results and use this as a baseline - might be more advanced techniques are not necessary or can provide only small gains. \n\n\n [1]: https://mlwave.com/kaggle-ensembling-guide", "424581": "To stay true to the 'no sharing outside of teams rule' - and also because I think it might actually be useful to other folks using this starter pack - I share here on [the fastai forums](https://forums.fast.ai/t/cpu-memory-usage-keeps-growing-as-training-one-cycle/30879/6?u=radek) how to train on epoch_sizes of arbitrary size.\n\nIt's more of a technicality on how to use the framework but is a really nice feature and might be of use to some.\n\nHere is the code:\n```\ndef chunks(l, n):\n \"\"\"Yield successive n-sized chunks from l.\"\"\"\n for i in range(0, len(l), n):\n yield l[i:i + n]\n\nclass RandomSamplerWithEpochSize(Sampler):\n \"\"\"Yields epochs of specified sizes. Iterates over all examples in a data_source in random\n order. Ensures (nearly) all examples have been trained on before beginning the next iteration\n over the data_source - drops the last epoch that would likely be smaller than epoch_size.\n \"\"\"\n def __init__(self, data_source, epoch_size):\n self.n = len(data_source)\n self.epoch_size = epoch_size\n self._epochs = []\n def __iter__(self):\n return iter(self.next_epoch)\n @property\n def next_epoch(self):\n if len(self._epochs) == 0: self.generate_epochs()\n return self._epochs.pop()\n def generate_epochs(self):\n idxs = [i for i in range(self.n)]\n np.random.shuffle(idxs)\n self._epochs = list(chunks(idxs, self.epoch_size))[:-1]\n def __len__(self):\n return self.epoch_size\n```\n\nAnd here is how to use it to create the `data_bunch`:\n```\ntrain_dl = DataLoader(\n label_lists.train,\n num_workers=12,\n batch_sampler=BatchSampler(RandomSamplerWithEpochSize(label_lists.train, 200_000), bs, True)\n)\nvalid_dl = DataLoader(label_lists.valid, 2*bs, False, num_workers=12)\ntest_dl = DataLoader(label_lists.test, 2*bs, False, num_workers=12)\n\ndata_bunch = ImageDataBunch(train_dl, valid_dl, test_dl)\n```", "422396": "Awesome thanks. I had a go at making it into a kernel but ran out of disk space. work in progress..\n\nhttps://www.kaggle.com/dromosys/fast-ai-quick-draw/", "413238": "Hey has anyone able to get convert more than 10% of their dataset to images? It's the max I could convert. Anyone facing the same issue? \n\nAnd using strokes directly as done in [ 🐘Greyscale MobileNet [LB=0.892]][1] a better approach? \n\nThanks, Radek for sharing your code! It was super helpful.\n\n\n [1]: https://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892#", "412801": "Thanks for sharing. Already looks good from my initial run.", "411274": "Thanks for sharing this! \nI tried submitting with your stater code and with an increase in data I noticed some gain so I decided to train the model on the complete datset after extracting the complete dataset on disk. \nI'm trying to give this approach a shot instead of trying other approaches (About a month of runway until the deadline which is slightly less given the data size)", "435574": "I added my solution based on the starter pack to the repository. The model in itself is nothing special (it is a combination of 2 cnns, rnn and country code embeddings) but the way the data is generated and fed to the model and a couple of other things (epochs of arbitrary size, changing the classifier during training, etc) might potentially be of interest.\n\nHere is a [link](https://twitter.com/radekosmulski/status/1071341277453656070) to a Twitter thread where I go into some additional details.", "413877": "\nHi Radek !\n\nJust started your Starter Pack, quite an amazing surprise, many thanks.\n\nBTW I think I read in one of your posts on Kaggle that you use a Ryzen 5 + 1080Ti, I got almost the same setup with a Ryzen 7 1700X but\n- (1) my epochs are about 10-20% slower than yours and\n- (2) I later had to lower batch size to 48 (default 64) for the `unfreeze` part to avoid CUDA OOM error.\n\nCheers,\n\nEPB", "412247": "It seems that training with unrecognized images included is better (also, based on validation / LB gap I suspect test includes unrecognized images).\n\nI was able to get to 0.882 after training resnet34 on 2 million of images of size 64x64. Expect another increase in performance with training on 128x128. ", "411946": "Hey everybody. Since we are converting the arrays in csv to .png files and saving to disk,if any one of you get out-of-space error, there might be 2 reasons for that.