Hi everyone,
\nI'm working on random data augmentation in TensorFlow.
\nI start with adding a very simple random swapping in my preprocess function like:
tf.random.uniform(()) > .:\n ,lhand = lhand,rhand\n ,lpose = lpose,rpose\nThis increases my training time from approximately 50ms/step to about 80ms/step.
However, if I remove the random condition to like:
\n\nlhand = lhand,rhand\nlpose = lpose,rpose\nThe training time is back to 50ms/step again so I suspect this delay is from tf.random.uniform().
As this delay causes about 60% increasing training time, I'm wondering if everyone has to face this problem or if there is a faster way to do random augmentation?
Hi everyone,
\nI'm working on random data augmentation in TensorFlow.
\nI start with adding a very simple random swapping in my preprocess function like:
tf.random.uniform(()) > .:\n ,lhand = lhand,rhand\n ,lpose = lpose,rpose\nThis increases my training time from approximately 50ms/step to about 80ms/step.
However, if I remove the random condition to like:
\n\nlhand = lhand,rhand\nlpose = lpose,rpose\nThe training time is back to 50ms/step again so I suspect this delay is from tf.random.uniform().
As this delay causes about 60% increasing training time, I'm wondering if everyone has to face this problem or if there is a faster way to do random augmentation?
If you suspect that tf.random is very slow, try to replace it with the np.random equivalent…
", "rawMarkdown": "If you suspect that tf.random is very slow, try to replace it with the np.random equivalent...", "votes": 1, "replies": [ { "id": 2307191, "postDate": "2023-06-18T00:27:28.500Z", "content": "np.random.uniform() works perfectly (at least in terms of speed)✌️but why is tensorflow's random function so slow ….. 🤔🤔🤔
np.random.uniform() works well
", "rawMarkdown": "np.random.uniform() works well" }, { "id": 2315300, "postDate": "2023-06-24T03:30:18.413Z", "content": "Random with numpy or pure python random lib will be make this node to be constant. Ie np.random.randn() is tf.constant and your augmentations are wrong !!!
", "rawMarkdown": "Random with numpy or pure python random lib will be make this node to be constant. Ie np.random.randn() is tf.constant and your augmentations are wrong !!!", "votes": 4 }, { "id": 2315321, "postDate": "2023-06-24T04:06:00.470Z", "content": "Yes, you're right, I just got stuck because of it for 1 week….. 😭😭😭
", "rawMarkdown": "Yes, you're right, I just got stuck because of it for 1 week..... 😭😭😭" }, { "id": 2401731, "postDate": "2023-08-21T18:40:04.213Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2401734, "postDate": "2023-08-21T18:40:58.957Z", "content": "Thank you. This should be pinned.
", "rawMarkdown": "Thank you. This should be pinned." } ] } ] }, { "id": 2401752, "postDate": "2023-08-21T18:50:47.323Z", "content": "i never get tf data pipeline to work. i don't know why.
\neven if i code everything in tf function and add the tf.function decorator, it seems so slow.
the data pipeline is difficult to debug, becuase breakpoint cannot be set (even with tf.experimental.data decorator).
\nworse still the debug code can run differently from the non-debug one (e.g. tensor.numpy() is only available in eagar mode).
\nmy code end up working in debug mode, but not running correctly in runtime
in the end i use pytorch for training and then trasnfer the pytorch weights to a keras model for tflite conversion.
\nmy pytorch frame work is training 10x faster than my tf/keras one.
I found out that I need to manually add tf.data.AUTOTUNE into data loading-related functions.
I followed the previous 1st solution (https://www.kaggle.com/code/hoyso48/1st-place-solution-training):
\n = .data.TFRecordDataset(tfrecords, num_parallel_reads=.data.AUTOTUNE, compression_type=)\n = .(decode_tfrec, .data.AUTOTUNE)\n = .(lambda : preprocess(, augment=augment, max_len=max_len), .data.AUTOTUNE)\n ...\n ...\nMaybe this can help
", "rawMarkdown": "I found out that I need to manually add `tf.data.AUTOTUNE` into data loading-related functions. \n\nI followed the previous 1st solution (https://www.kaggle.com/code/hoyso48/1st-place-solution-training):\n```\n ds = tf.data.TFRecordDataset(tfrecords, num_parallel_reads=tf.data.AUTOTUNE, compression_type='GZIP')\n ds = ds.map(decode_tfrec, tf.data.AUTOTUNE)\n ds = ds.map(lambda x: preprocess(x, augment=augment, max_len=max_len), tf.data.AUTOTUNE)\n ...\n ...\n```\n\nMaybe this can help", "replies": [ { "id": 2401990, "postDate": "2023-08-22T00:21:00.393Z", "content": "how fast does your tf train run?
", "rawMarkdown": "how fast does your tf train run?" }, { "id": 2402039, "postDate": "2023-08-22T00:57:11.793Z", "content": "It depends on the model I use. My 15M parameter model runs ~55s per epoch on kaggle TPU with 256 batch size.
", "rawMarkdown": "It depends on the model I use. My 15M parameter model runs ~55s per epoch on kaggle TPU with 256 batch size. " } ] }, { "id": 2401879, "postDate": "2023-08-21T21:19:21.587Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2402374, "postDate": "2023-08-22T05:52:57.700Z", "content": "@hengck23 Yes, I also found tf much slower then torch(3-4times slower on 1*4090 gpu) for training, not sure why, might due to some driver issues as I found intel cpu is much slower then amd for tensorflow but for torch do not have this issue. Also using torch we could use torch.compile to further speedup. So if not using tpu, torch may be a better choice.
\nFor those using tensorflow I would recommend using https://github.com/alexeytochin/tf_seq2seq_losses as I found the original ctc loss of tensorflow is too slow.
If you suspect that tf.random is very slow, try to replace it with the np.random equivalent…
", "votes": 1, "replies": [ { "id": 2307191, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-06-18T00:27:28.500000", "content": "np.random.uniform() works perfectly (at least in terms of speed)✌️but why is tensorflow's random function so slow ….. 🤔🤔🤔
np.random.uniform() works well
", "votes": 0, "replies": [] }, { "id": 2315300, "author_name": "Just A game on your lips", "author_url": "", "post_date": "2023-06-24T03:30:18.413000", "content": "Random with numpy or pure python random lib will be make this node to be constant. Ie np.random.randn() is tf.constant and your augmentations are wrong !!!
", "votes": 4, "replies": [] }, { "id": 2315321, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-06-24T04:06:00.470000", "content": "Yes, you're right, I just got stuck because of it for 1 week….. 😭😭😭
", "votes": 0, "replies": [] }, { "id": 2401731, "author_name": "", "author_url": "", "post_date": "2023-08-21T18:40:04.213000", "content": "", "votes": 0, "replies": [] }, { "id": 2401734, "author_name": "JoAI", "author_url": "", "post_date": "2023-08-21T18:40:58.957000", "content": "Thank you. This should be pinned.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2401752, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-21T18:50:47.323000", "content": "i never get tf data pipeline to work. i don't know why.
\neven if i code everything in tf function and add the tf.function decorator, it seems so slow.
the data pipeline is difficult to debug, becuase breakpoint cannot be set (even with tf.experimental.data decorator).
\nworse still the debug code can run differently from the non-debug one (e.g. tensor.numpy() is only available in eagar mode).
\nmy code end up working in debug mode, but not running correctly in runtime
in the end i use pytorch for training and then trasnfer the pytorch weights to a keras model for tflite conversion.
\nmy pytorch frame work is training 10x faster than my tf/keras one.
I found out that I need to manually add tf.data.AUTOTUNE into data loading-related functions.
I followed the previous 1st solution (https://www.kaggle.com/code/hoyso48/1st-place-solution-training):
\n = .data.TFRecordDataset(tfrecords, num_parallel_reads=.data.AUTOTUNE, compression_type=)\n = .(decode_tfrec, .data.AUTOTUNE)\n = .(lambda : preprocess(, augment=augment, max_len=max_len), .data.AUTOTUNE)\n ...\n ...\nMaybe this can help
", "votes": 0, "replies": [ { "id": 2401990, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-22T00:21:00.393000", "content": "how fast does your tf train run?
", "votes": 0, "replies": [] }, { "id": 2402039, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-08-22T00:57:11.793000", "content": "It depends on the model I use. My 15M parameter model runs ~55s per epoch on kaggle TPU with 256 batch size.
", "votes": 0, "replies": [] } ] }, { "id": 2401879, "author_name": "", "author_url": "", "post_date": "2023-08-21T21:19:21.587000", "content": "", "votes": 0, "replies": [] }, { "id": 2402374, "author_name": "gezi", "author_url": "", "post_date": "2023-08-22T05:52:57.700000", "content": "@hengck23 Yes, I also found tf much slower then torch(3-4times slower on 1*4090 gpu) for training, not sure why, might due to some driver issues as I found intel cpu is much slower then amd for tensorflow but for torch do not have this issue. Also using torch we could use torch.compile to further speedup. So if not using tpu, torch may be a better choice.
\nFor those using tensorflow I would recommend using https://github.com/alexeytochin/tf_seq2seq_losses as I found the original ctc loss of tensorflow is too slow.