Hi everyone !
\nthis is my very first feature competition, I'm brand new to machine and deep learning, and I'm beginning to realize that this particular competition is a hell of a start… (dataset size, semantic segmentation + time frames, not forgetting the training time limit… and the terribly low number of participants is kind of confirming the size of the challenge)
\nI would really appreciate some piece of advice, at least to be sure I'm not on the wrong track.
\nI'm going with tensorflow, planning on training a Unet (I'll start with the one provided on Keras as an example by F. Chollet). To cut down complexity a bit, I decided not to take into account the 4+3 timeframes around the masked one.
\nRight now, my main concern is the time it takes to train a model :
\nI already have a custom Sequence to load the data in batches of 32 instances, and I'm just fiddling with a one layer model just to see if everything fits together, using a custom Dice loss function.
\nAnd the average training time over one epoch is… almost 2 hours. RAM is almost full, and CPU usage is curiously low.
\nI fear the moment when I'll start training a real Unet model with 30+ layers over 10s of epochs…
Anyway, for all the TF users here, what did you do (plan to do) to boost the training time ?
", "messages": [ { "id": 2276863, "postDate": "2023-05-27T09:11:24.037Z", "content": "Many people including myself started with Keras. Learn from the mistakes of others and switch pytorch. This comp already has a strong baseline torch code LB0.494. Just trying to reduce suffering in this word…
", "rawMarkdown": "Many people including myself started with Keras. Learn from the mistakes of others and switch pytorch. This comp already has a strong baseline torch code LB0.494. Just trying to reduce suffering in this word..." }, { "id": 2260745, "postDate": "2023-05-15T20:44:58.817Z", "content": "Hey iceman,
\nwelcome to this challenge! Currently, I am pursuing a similar approach than you do.
\nI am also training a UNET with tensorflow. For my pipeline, I am using tf.data.Dataset, since it allows to parallelize the data loading and training (If you are not familiar with this, check this link https://www.tensorflow.org/api_docs/python/tf/data/Dataset). I am also only using the masked image, not the ones before and after. My training time per epoch lies between 5 and 15 minutes, depending on the \"depth\" (amount of filters and layers) I use for the UNET.
And what do you mean with the \"preprint\" you mentioned in your comment? Where can i find this information?
\nThanks and good luck!
", "rawMarkdown": "Hey iceman, \nwelcome to this challenge! Currently, I am pursuing a similar approach than you do.\nI am also training a UNET with tensorflow. For my pipeline, I am using tf.data.Dataset, since it allows to parallelize the data loading and training (If you are not familiar with this, check this link https://www.tensorflow.org/api_docs/python/tf/data/Dataset). I am also only using the masked image, not the ones before and after. My training time per epoch lies between 5 and 15 minutes, depending on the \"depth\" (amount of filters and layers) I use for the UNET. \n\nAnd what do you mean with the \"preprint\" you mentioned in your comment? Where can i find this information?\n\nThanks and good luck!", "replies": [ { "id": 2260778, "postDate": "2023-05-15T21:28:07.140Z", "content": "Hey Jan,
\nthanks for your reply !
\nYou have the preprint of the research here : https://arxiv.org/abs/2304.02122 (download the pdf at the top right corner of the page)
\nThey went with a modified ResNet.
\nConcerning your training time, this is what I expected too, but 2 hours for just a single layer, damn ! Something is clearly wrong somewhere. I was thinking about moving to a Dataset, but I don't see how to avoid going through a np.load() to get the images in the shape I want.
\nFor your Unet, you build it from scratch or do you do transfer learning ?
Good luck to you too !
", "rawMarkdown": "Hey Jan,\nthanks for your reply !\nYou have the preprint of the research here : https://arxiv.org/abs/2304.02122 (download the pdf at the top right corner of the page)\nThey went with a modified ResNet.\nConcerning your training time, this is what I expected too, but 2 hours for just a single layer, damn ! Something is clearly wrong somewhere. I was thinking about moving to a Dataset, but I don't see how to avoid going through a np.load() to get the images in the shape I want.\nFor your Unet, you build it from scratch or do you do transfer learning ?\n\nGood luck to you too !", "replies": [ { "id": 2260793, "postDate": "2023-05-15T22:02:36.457Z", "content": "Hey,
\nI used an implementation which was available publicly but trained it from scratch.
\nDid you already consider training on GPU? On CPU, cnn training is quite slow whereas on GPU it is a lot faster!
Well yes I tried, but it didn't seem to work, I had lots of warning telling me something like 'cuda not responding' (I honestly don't recall exactly the warnings)
\nAnd I'm going for a scratch training too ;-)
Hey Jan,
\nI come back to you since I'm lost in implementation hell with these damn tf.dataset. I tried using from_generator, list_file (and then mapping a numpy_function or a py_function), and either this is still very slow, or I lose the shape of the tensors during the process… Could you give me a hint on how you worked your way around ? (congrats for your first results, by the way !)
Hey Iceman,
\nI created the tf Dataset using the paths to the record ids folder and then make a \"from_tensor_sclices\" dataset. Then i mapped a function for loading the bands and masks (you need to use tf pyfunc) and processing them. Hope this helps!
", "rawMarkdown": "Hey Iceman,\n\nI created the tf Dataset using the paths to the record ids folder and then make a \"from_tensor_sclices\" dataset. Then i mapped a function for loading the bands and masks (you need to use tf pyfunc) and processing them. Hope this helps!", "votes": 1 }, { "id": 2269682, "postDate": "2023-05-22T16:13:41.673Z", "content": "Hi Jan,
\nthanks for your reply ! (and sorry for the delay) Yes, it helps, by confirming I was on the right track, but still experiencing long loading time issues. I went with a dataset list_files, which I then map with a numpy_function to extract the data from each .npy file. I might try with a py_func instead, maybe it's more efficient.
Little side note :
\nI just read the preprint (finally), and at least, the intuition of dropping the timeframes seems ok, at least for the beginning, since it boosts there AUC by just ~2 (on a score of ~70). And they conclude that the image BEFORE have more impact on the result than the image AFTER. And it seems that two of the bands are the difference between two couples of other bands. Wouldn't it be worth it of discarding them, then, to avoid processing the 'same' informations ?
Hi everyone !
\nthis is my very first feature competition, I'm brand new to machine and deep learning, and I'm beginning to realize that this particular competition is a hell of a start… (dataset size, semantic segmentation + time frames, not forgetting the training time limit… and the terribly low number of participants is kind of confirming the size of the challenge)
\nI would really appreciate some piece of advice, at least to be sure I'm not on the wrong track.
\nI'm going with tensorflow, planning on training a Unet (I'll start with the one provided on Keras as an example by F. Chollet). To cut down complexity a bit, I decided not to take into account the 4+3 timeframes around the masked one.
\nRight now, my main concern is the time it takes to train a model :
\nI already have a custom Sequence to load the data in batches of 32 instances, and I'm just fiddling with a one layer model just to see if everything fits together, using a custom Dice loss function.
\nAnd the average training time over one epoch is… almost 2 hours. RAM is almost full, and CPU usage is curiously low.
\nI fear the moment when I'll start training a real Unet model with 30+ layers over 10s of epochs…
Anyway, for all the TF users here, what did you do (plan to do) to boost the training time ?
", "rawMarkdown": "Hi everyone !\nthis is my very first feature competition, I'm brand new to machine and deep learning, and I'm beginning to realize that this particular competition is a hell of a start... (dataset size, semantic segmentation + time frames, not forgetting the training time limit... and the terribly low number of participants is kind of confirming the size of the challenge)\nI would really appreciate some piece of advice, at least to be sure I'm not on the wrong track.\nI'm going with tensorflow, planning on training a Unet (I'll start with the one provided on Keras as an example by F. Chollet). To cut down complexity a bit, I decided not to take into account the 4+3 timeframes around the masked one.\nRight now, my main concern is the time it takes to train a model :\nI already have a custom Sequence to load the data in batches of 32 instances, and I'm just fiddling with a one layer model just to see if everything fits together, using a custom Dice loss function.\nAnd the average training time over one epoch is... almost 2 hours. RAM is almost full, and CPU usage is curiously low.\nI fear the moment when I'll start training a real Unet model with 30+ layers over 10s of epochs...\n\nAnyway, for all the TF users here, what did you do (plan to do) to boost the training time ?" } ], "comments": [ { "id": 2276863, "author_name": "dmitrykonovalov", "author_url": "", "post_date": "2023-05-27T09:11:24.037000", "content": "Many people including myself started with Keras. Learn from the mistakes of others and switch pytorch. This comp already has a strong baseline torch code LB0.494. Just trying to reduce suffering in this word…
", "votes": 0, "replies": [] }, { "id": 2260745, "author_name": "Jan H", "author_url": "", "post_date": "2023-05-15T20:44:58.817000", "content": "Hey iceman,
\nwelcome to this challenge! Currently, I am pursuing a similar approach than you do.
\nI am also training a UNET with tensorflow. For my pipeline, I am using tf.data.Dataset, since it allows to parallelize the data loading and training (If you are not familiar with this, check this link https://www.tensorflow.org/api_docs/python/tf/data/Dataset). I am also only using the masked image, not the ones before and after. My training time per epoch lies between 5 and 15 minutes, depending on the \"depth\" (amount of filters and layers) I use for the UNET.
And what do you mean with the \"preprint\" you mentioned in your comment? Where can i find this information?
\nThanks and good luck!
", "votes": 0, "replies": [ { "id": 2260778, "author_name": "iceman273k", "author_url": "", "post_date": "2023-05-15T21:28:07.140000", "content": "Hey Jan,
\nthanks for your reply !
\nYou have the preprint of the research here : https://arxiv.org/abs/2304.02122 (download the pdf at the top right corner of the page)
\nThey went with a modified ResNet.
\nConcerning your training time, this is what I expected too, but 2 hours for just a single layer, damn ! Something is clearly wrong somewhere. I was thinking about moving to a Dataset, but I don't see how to avoid going through a np.load() to get the images in the shape I want.
\nFor your Unet, you build it from scratch or do you do transfer learning ?
Good luck to you too !
", "votes": 0, "replies": [ { "id": 2260793, "author_name": "Jan H", "author_url": "", "post_date": "2023-05-15T22:02:36.457000", "content": "Hey,
\nI used an implementation which was available publicly but trained it from scratch.
\nDid you already consider training on GPU? On CPU, cnn training is quite slow whereas on GPU it is a lot faster!
Well yes I tried, but it didn't seem to work, I had lots of warning telling me something like 'cuda not responding' (I honestly don't recall exactly the warnings)
\nAnd I'm going for a scratch training too ;-)
Hey Jan,
\nI come back to you since I'm lost in implementation hell with these damn tf.dataset. I tried using from_generator, list_file (and then mapping a numpy_function or a py_function), and either this is still very slow, or I lose the shape of the tensors during the process… Could you give me a hint on how you worked your way around ? (congrats for your first results, by the way !)
Hey Iceman,
\nI created the tf Dataset using the paths to the record ids folder and then make a \"from_tensor_sclices\" dataset. Then i mapped a function for loading the bands and masks (you need to use tf pyfunc) and processing them. Hope this helps!
", "votes": 1, "replies": [] }, { "id": 2269682, "author_name": "iceman273k", "author_url": "", "post_date": "2023-05-22T16:13:41.673000", "content": "Hi Jan,
\nthanks for your reply ! (and sorry for the delay) Yes, it helps, by confirming I was on the right track, but still experiencing long loading time issues. I went with a dataset list_files, which I then map with a numpy_function to extract the data from each .npy file. I might try with a py_func instead, maybe it's more efficient.
Little side note :
\nI just read the preprint (finally), and at least, the intuition of dropping the timeframes seems ok, at least for the beginning, since it boosts there AUC by just ~2 (on a score of ~70). And they conclude that the image BEFORE have more impact on the result than the image AFTER. And it seems that two of the bands are the difference between two couples of other bands. Wouldn't it be worth it of discarding them, then, to avoid processing the 'same' informations ?