![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2127141%2F1cbd13b7959cb0e261f0a95c0c4fd084%2Fgiphy.gif?generation=1588520450382636&alt=media\")
In this competition, I have observed there are more (almost all) kernels written using Pytorch.\nCan anyone explain why tensorflow is so unpopular in this competition?
", "messages": [ { "id": 829903, "postDate": "2020-05-02T06:54:09.553Z", "content": "In this competition, I have observed there are more (almost all) kernels written using Pytorch.\nCan anyone explain why tensorflow is so unpopular in this competition?
", "rawMarkdown": "In this competition, I have observed there are more (almost all) kernels written using Pytorch.\nCan anyone explain why tensorflow is so unpopular in this competition?", "votes": 14 }, { "id": 831144, "postDate": "2020-05-03T07:16:52.667Z", "content": "I've seen the trend on Kaggle switching to PyTorch over TF. Personally I still use Keras (PyTorch is on my to-do list to learn solely due to it's popularity). There is a lot of power in Keras that shouldn't go underestimated, so you can do a lot in Keras with less effort. Also, TF \\ Keras is still heavily used in industry. As with anything, programming languages come and go. Currently, both have value and both are good to know. I think PyTorch is growing in popularity due to it's push from Fast.ai \\ Jeremy Howard, and it's more Pythonic approach vs. TF.
\n\nOutside of the newly released ZeRO & DeepSpeed by Microsoft, I think TF\\Horovod has stronger distributed GPU support right now; but this is not something you'll see on Kaggle because Kaggle doesn't offer multi-GPU machines. You might see more Horovod if Kaggle offered multi-GPU machines. So for Kaggle, it also depends on the hardware. *(Doesn't mean that people aren't using Horovod or other things off of Kaggle for some competitions that don't require running a kernel on Kaggle itself). *
\n\nIn short, I think TF has the tenure and more features (and more industry tenure). PyTorch is the new guy with less tenure but it's growing super fast due to it's Pythonic design which is very appealing to many programmers.
\n\nEvolution of programming language popularity over time: https://www.youtube.com/watch?v=Og847HVwRSI
\n\nMicrosoft ZeRO & DeepSpeed: https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
\n\nOne of my Keras notebooks for this competition: https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-gpu
", "rawMarkdown": "I've seen the trend on Kaggle switching to PyTorch over TF. Personally I still use Keras (PyTorch is on my to-do list to learn solely due to it's popularity). There is a lot of power in Keras that shouldn't go underestimated, so you can do a lot in Keras with less effort. Also, TF \\ Keras is still heavily used in industry. As with anything, programming languages come and go. Currently, both have value and both are good to know. I think PyTorch is growing in popularity due to it's push from Fast.ai \\ Jeremy Howard, and it's more Pythonic approach vs. TF. \n\nOutside of the newly released ZeRO & DeepSpeed by Microsoft, I think TF\\Horovod has stronger distributed GPU support right now; but this is not something you'll see on Kaggle because Kaggle doesn't offer multi-GPU machines. You might see more Horovod if Kaggle offered multi-GPU machines. So for Kaggle, it also depends on the hardware. *(Doesn't mean that people aren't using Horovod or other things off of Kaggle for some competitions that don't require running a kernel on Kaggle itself). *\n\nIn short, I think TF has the tenure and more features (and more industry tenure). PyTorch is the new guy with less tenure but it's growing super fast due to it's Pythonic design which is very appealing to many programmers.\n\nEvolution of programming language popularity over time: https://www.youtube.com/watch?v=Og847HVwRSI\n\nMicrosoft ZeRO & DeepSpeed: https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/\n\nOne of my Keras notebooks for this competition: https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-gpu", "votes": 8, "replies": [ { "id": 831176, "postDate": "2020-05-03T07:53:47.227Z", "content": "@yeayates21 Thanks for the detailed explanation and for your hope in Tensorflow. \nYour notebook is a very nice baseline in Keras for this competition.
", "rawMarkdown": "@yeayates21 Thanks for the detailed explanation and for your hope in Tensorflow. \nYour notebook is a very nice baseline in Keras for this competition.", "votes": 1 }, { "id": 831813, "postDate": "2020-05-03T16:06:50.087Z", "content": "Np. Also, here is the Google trend chart on TF and PyTorch. As you can see a year ago, PyTorch wasn't even near TF use, but today they're neck and neck. So as others have said, it's not just this competition, it's a general trend that PyTorch is rising up and TF is going down.
\n\nMy guess is that, just like the Hadoop boom, TF won't go away it will just level off and PyTorch will continue to climb (for now). HDFS didn't go away over pure cloud services, it's still heavily used @ Uber and tons of companies, it just leveled off is all and I think TF will do the same.
\n\nAlso, PyTorch and TF are not the only games in town. You also have Chainer, MXNet, etc. and then within TF you have TF for Swift and TF.js ... and who knows what might come out next!
\n\nhttps://trends.google.com/trends/explore?geo=US&q=%2Fg%2F11bwp1s2k3,%2Fg%2F11gd3905v1
", "rawMarkdown": "Np. Also, here is the Google trend chart on TF and PyTorch. As you can see a year ago, PyTorch wasn't even near TF use, but today they're neck and neck. So as others have said, it's not just this competition, it's a general trend that PyTorch is rising up and TF is going down. \n\nMy guess is that, just like the Hadoop boom, TF won't go away it will just level off and PyTorch will continue to climb (for now). HDFS didn't go away over pure cloud services, it's still heavily used @ Uber and tons of companies, it just leveled off is all and I think TF will do the same. \n\nAlso, PyTorch and TF are not the only games in town. You also have Chainer, MXNet, etc. and then within TF you have TF for Swift and TF.js ... and who knows what might come out next!\n\nhttps://trends.google.com/trends/explore?geo=US&q=%2Fg%2F11bwp1s2k3,%2Fg%2F11gd3905v1", "votes": 5 }, { "id": 831923, "postDate": "2020-05-03T17:26:29.497Z", "content": "NumPy arrays and PyTorch tensors\nNumpy is the most widely used library for scientific and numeric programming in Python, and provides very similar functionality and a very similar API to that provided by PyTorch; however, it does not support using the GPU, or calculating gradients, which are both critical for deep learning.
\n\n(Note that fastai adds some features to NumPy and PyTorch to make them a bit more similar to each other. If any code in this book doesn't work on your computer, it's possible that you forgot to include a line at the start of your notebook such as: from fastai.vision.all import *.)
\n\nBut what are arrays and tensors, and why should you care?
\n\nPython is slow compared to many languages. Anything fast in Python, NumPy or PyTorch is likely to be a wrapper to a compiled object written (and optimized) in another language - specifically C. In fact, NumPy arrays and PyTorch tensors can finish computations many thousands of times faster than using pure Python.
\n\nA NumPy array is a multidimensional table of data, with all items of the same type. Since that can be any type at all, they could even be arrays of arrays, with the innermost arrays potentially being different sizes — this is called a \"jagged array\". By \"multidimensional table\" we mean, for instance, a list (dimension of one), a table or matrix (dimension of two), a \"table of tables\" or a \"cube\" (dimension of three), and so forth. If the items are all of some simple type such as an integer or a float then NumPy will store them as a compact C data structure in memory. This is where NumPy shines. Numpy has a wide variety of operators and methods which can run computations on these compact structures at the same speed as optimized C, because they are written in optimized C.
\n\nA PyTorch tensor is nearly the same thing as a NumPy array, but with an additional restriction which unlocks some additional capabilities. It's the same in that it, too, is a multidimensional table of data, with all items of the same type. However, the restriction is that a tensor cannot use just any old type — it has to use a single basic numeric type for all components. As a result, a tensor is not as flexible as a genuine array of arrays, which allows jagged arrays, where the inner arrays could have different sizes. So a PyTorch tensor cannot be jagged. It is always a regularly shaped multidimensional rectangular structure.
\n\nThe vast majority of methods and operators supported by NumPy on these structures are also supported by PyTorch. But PyTorch tensors have additional capabilities. One major capability is that these structures can live on the GPU, in which case their computation will be optimized for the GPU, and can run much faster (given lots of values to work on). In addition, PyTorch can automatically calculate derivatives of these operations, including combinations of operations. As you'll see, it would be impossible to do deep learning in practice without this capability.
\n\ncopied from Fast.ai documentation, but it is the same in PyTorch documentation.
\n\nPlus, MXNET is as Pytorch. pretty much the same API, once you learn one, the other one it is eaiser to get your arms around to it.
", "rawMarkdown": "NumPy arrays and PyTorch tensors\nNumpy is the most widely used library for scientific and numeric programming in Python, and provides very similar functionality and a very similar API to that provided by PyTorch; however, it does not support using the GPU, or calculating gradients, which are both critical for deep learning. \n\n(Note that fastai adds some features to NumPy and PyTorch to make them a bit more similar to each other. If any code in this book doesn't work on your computer, it's possible that you forgot to include a line at the start of your notebook such as: from fastai.vision.all import *.)\n\nBut what are arrays and tensors, and why should you care?\n\nPython is slow compared to many languages. Anything fast in Python, NumPy or PyTorch is likely to be a wrapper to a compiled object written (and optimized) in another language - specifically C. In fact, NumPy arrays and PyTorch tensors can finish computations many thousands of times faster than using pure Python.\n\nA NumPy array is a multidimensional table of data, with all items of the same type. Since that can be any type at all, they could even be arrays of arrays, with the innermost arrays potentially being different sizes — this is called a \"jagged array\". By \"multidimensional table\" we mean, for instance, a list (dimension of one), a table or matrix (dimension of two), a \"table of tables\" or a \"cube\" (dimension of three), and so forth. If the items are all of some simple type such as an integer or a float then NumPy will store them as a compact C data structure in memory. This is where NumPy shines. Numpy has a wide variety of operators and methods which can run computations on these compact structures at the same speed as optimized C, because they are written in optimized C.\n\nA PyTorch tensor is nearly the same thing as a NumPy array, but with an additional restriction which unlocks some additional capabilities. It's the same in that it, too, is a multidimensional table of data, with all items of the same type. However, the restriction is that a tensor cannot use just any old type — it has to use a single basic numeric type for all components. As a result, a tensor is not as flexible as a genuine array of arrays, which allows jagged arrays, where the inner arrays could have different sizes. So a PyTorch tensor cannot be jagged. It is always a regularly shaped multidimensional rectangular structure.\n\nThe vast majority of methods and operators supported by NumPy on these structures are also supported by PyTorch. But PyTorch tensors have additional capabilities. One major capability is that these structures can live on the GPU, in which case their computation will be optimized for the GPU, and can run much faster (given lots of values to work on). In addition, PyTorch can automatically calculate derivatives of these operations, including combinations of operations. As you'll see, it would be impossible to do deep learning in practice without this capability.\n\ncopied from Fast.ai documentation, but it is the same in PyTorch documentation.\n\nPlus, MXNET is as Pytorch. pretty much the same API, once you learn one, the other one it is eaiser to get your arms around to it. ", "votes": 1 }, { "id": 832042, "postDate": "2020-05-03T19:09:00.353Z", "content": "Also, here is a great article on the PyTorch vs. TF conversation with more of an industry trend lens: https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304
", "rawMarkdown": "Also, here is a great article on the PyTorch vs. TF conversation with more of an industry trend lens: https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304\n\n", "votes": 2 } ] }, { "id": 830711, "postDate": "2020-05-02T20:02:53.310Z", "content": "I'm giving my opinion on this in a different way, IMHO If anyone shares high scored kernel in TensorFlow, PyTorch, Fast.ai or whatever, I think that would be popular. 😏
", "rawMarkdown": "I'm giving my opinion on this in a different way, IMHO **If anyone shares high scored kernel in TensorFlow, PyTorch, Fast.ai or whatever, I think that would be popular.** 😏 ", "votes": 8, "replies": [ { "id": 830926, "postDate": "2020-05-03T01:42:44.357Z", "content": "That's true!
", "rawMarkdown": "That's true!", "votes": 1 }, { "id": 831132, "postDate": "2020-05-03T06:57:03.127Z", "content": "Agreed 👍 👌
", "rawMarkdown": "Agreed 👍 👌 ", "votes": 1 }, { "id": 831617, "postDate": "2020-05-03T13:38:25.537Z", "content": "Lol .. You hit the Bulls eye :)
", "rawMarkdown": "Lol .. You hit the Bulls eye :) ", "votes": 2 }, { "id": 831739, "postDate": "2020-05-03T15:25:42.733Z", "content": "😄 😄
", "rawMarkdown": "😄 😄 ", "votes": 1 }, { "id": 831763, "postDate": "2020-05-03T15:41:56.590Z", "content": "
Then look @ the lattes competitions winners.... I wonder why they use Pytorch...
", "rawMarkdown": "Then look @ the lattes competitions winners.... I wonder why they use Pytorch...", "votes": 1 }, { "id": 834022, "postDate": "2020-05-05T08:32:15.680Z", "content": "@oscarrangel If I understand the topic author's question, she has only concern for this competition (I think, she didn't mean in general). I would say PyTorch and TensorFlow both are good, it actually depends on us. @yeayates21 explained really well in his comment and shared some wonderful resources regarding this issue.
\n\nAs you wonder why in previous competition people chose PyTorch (when it may be not that popular), simple, it's their choice, they felt comfortable with this framework. They thought they could do diverse experimentation with these tools focusing on the problem. Switching to a new framework can be costly especially at the time of competition. I feel really weird when someone says something bad (too bad, even worse) about TensorFlow but couldn't explain properly why so. They faced a little but read shortcomings a lot from the web and speaks accordingly.
\n\nHowever, as far as popularity is concern among frameworks in a particular competition, it reminds of the Bengali.AI competition. Yes, same as this comp. there were also strong PyTorch resources and supports. It's only this Keras based kernel which was massively popular, just see its upvote and fork. People loved it and tried on it. I also made a public notebook with Keras but in a different pipeline. At the beginning and mid of this Bengali.AI competition, these two kernels were only the unique Keras/TF based kernel there. Plus, there were open secret and open source cool techniques implemented in PyTorch for boosting accuracy but nor for in Keras, such as MixUp, CutMix, OHEM. Later, two awesome Keras kernels published, one of them based on training strategies and another one was on TPU, COOL! Now there were more people who tried on these kernels and began to share their findings.
\n\nIn short, both PyTorch and TensorFlow are awesome. I think it depends on us. At the same time, Community Support Matters! But we should also remember that these are just tools, we should focus more on the problem. =)
", "rawMarkdown": "@oscarrangel If I understand the topic author's question, she has only concern for **this competition** (I think, she didn't mean in general). I would say **PyTorch** and **TensorFlow** both are good, it actually depends on us. @yeayates21 explained really well in his comment and shared some wonderful resources regarding this issue. \n\nAs you wonder why in previous competition people chose **PyTorch** (when it may be not that popular), simple, it's their choice, they felt comfortable with this framework. They thought they could do diverse experimentation with these tools focusing on the problem. Switching to a new framework can be costly especially at the time of competition. I feel really weird when someone says something bad (too bad, even worse) about **TensorFlow** but couldn't explain properly why so. They faced a little but read shortcomings a lot from the web and speaks accordingly.\n\nHowever, as far as popularity is concern among frameworks in a particular competition, it reminds of the [Bengali.AI](https://www.kaggle.com/c/bengaliai-cv19) competition. Yes, same as this comp. there were also strong PyTorch resources and supports. It's only [this Keras based kernel](https://www.kaggle.com/kaushal2896/bengali-graphemes-starter-eda-multi-output-cnn) which was massively popular, just see its **upvote** and **fork**. People loved it and tried on it. I also made a [public notebook with Keras](https://www.kaggle.com/ipythonx/keras-grapheme-gridmask-augmix-ensemble) but in a different pipeline. At the beginning and mid of this Bengali.AI competition, these two kernels were only the unique **Keras/TF** based kernel there. Plus, there were open secret and open source cool techniques implemented in **PyTorch** for boosting accuracy but nor for in **Keras**, such as [MixUp](https://arxiv.org/pdf/1710.09412.pdf), [CutMix](https://arxiv.org/abs/1905.04899), [OHEM](https://arxiv.org/abs/1604.03540). Later, two awesome **Keras** kernels published, one of them based on [training strategies](https://www.kaggle.com/rsmits/keras-efficientnet-b3-training-inference) and another one was [on TPU](https://www.kaggle.com/seesee/1-create-tfrecords), COOL! Now there were more people who tried on these kernels and began to share their findings.\n\nIn short, both PyTorch and TensorFlow are awesome. I think it depends on us. At the same time, **Community Support Matters**! But we should also remember that these are just tools, we should focus more on the problem. =)", "votes": 3 }, { "id": 834711, "postDate": "2020-05-05T18:07:03.690Z", "content": "@ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.
\n\nBut at the end of the day.
\n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly.
\n\ntaken from fastai https://arxiv.org/abs/2002.04688
\n\nend of the @ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.
\n\nBut at the end of the day.
\n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly.
\n\ntaken from fastai https://arxiv.org/abs/2002.04688
\n\nend of the discussion for me.
", "rawMarkdown": "@ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.\n\nBut at the end of the day. \n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly. \n\ntaken from fastai https://arxiv.org/abs/2002.04688\n\nend of the @ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.\n\nBut at the end of the day. \n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly. \n\ntaken from fastai https://arxiv.org/abs/2002.04688\n\nend of the discussion for me.", "votes": -1 } ] }, { "id": 833242, "postDate": "2020-05-04T17:26:56.493Z", "content": "Simplicity and PyTorch Model Zoo is more than what tf has to offer; IT's very easy to control what's happening here; Plus helpers like fastAI, PyTorch lightning, Catalyst etc makes live easier!
\n\nPlus nearly many re-search papers which i have read, if had implementations, it's also in PyTorch wrt CV nowadays as it's Simply Python!
\n\nIn short Ecosystem is very strong for kagglers if they use PyTorch but that doesn't mean it cannot be done on TF as well; Pretty much they are the same or converging to be one in terms of specs!
", "rawMarkdown": "Simplicity and PyTorch Model Zoo is more than what tf has to offer; IT's very easy to control what's happening here; Plus helpers like fastAI, PyTorch lightning, Catalyst etc makes live easier!\n\nPlus nearly many re-search papers which i have read, if had implementations, it's also in PyTorch wrt CV nowadays as it's Simply Python!\n\nIn short Ecosystem is very strong for kagglers if they use PyTorch but that doesn't mean it cannot be done on TF as well; Pretty much they are the same or converging to be one in terms of specs!", "votes": 4 }, { "id": 830917, "postDate": "2020-05-03T01:11:05.467Z", "content": "I think the improvement in the simplicity of the code tensorflow is improving over time is remarkable but still its pytorch for the best...
\n\nThe reason why no one uses tensorflow is because of its non simplicity for the beginner and less easy to fully customize things, people including me just does not like to use it, however keras still remains the best and the easiest to get a baseline working but over time you will have to switch to either tensorflow or pytorch. Fastai is good as it gives you the power that just wraps around the pytorch providing more custom features than keras.
", "rawMarkdown": "I think the improvement in the simplicity of the code tensorflow is improving over time is remarkable but still its pytorch for the best...\n\nThe reason why no one uses tensorflow is because of its non simplicity for the beginner and less easy to fully customize things, people including me just does not like to use it, however keras still remains the best and the easiest to get a baseline working but over time you will have to switch to either tensorflow or pytorch. Fastai is good as it gives you the power that just wraps around the pytorch providing more custom features than keras.", "votes": 1 }, { "id": 830743, "postDate": "2020-05-02T20:55:14.840Z", "content": "I don't think it has anything to do with this comp, just that people prefer Pytorch over Tensorflow.
", "rawMarkdown": "I don't think it has anything to do with this comp, just that people prefer Pytorch over Tensorflow.", "votes": 1 }, { "id": 830801, "postDate": "2020-05-02T23:13:43.463Z", "content": "in my personal opinion, I have tried both and like better PyTorch, especially if you are treating to migrate from old versions of TensorFlow, with the new 2.xx it is very buggy and lost of things don't run,
\n\nI believe because it is an older framework and they are trying to adapt it to new things and they are having problems doing it.
\n\nPytorch is much less complicated and has much better support.
\n\nThis has been my own personal experience.
", "rawMarkdown": "in my personal opinion, I have tried both and like better PyTorch, especially if you are treating to migrate from old versions of TensorFlow, with the new 2.xx it is very buggy and lost of things don't run, \n\nI believe because it is an older framework and they are trying to adapt it to new things and they are having problems doing it.\n\nPytorch is much less complicated and has much better support.\n\nThis has been my own personal experience.", "replies": [ { "id": 831166, "postDate": "2020-05-03T07:47:52.303Z", "content": "Yeah, I'm starting to realize the pitfalls of TF too. Can't really compare to Pytorch because I have never used it. Seems like it's the right time to start learning Pytorch.
", "rawMarkdown": "Yeah, I'm starting to realize the pitfalls of TF too. Can't really compare to Pytorch because I have never used it. Seems like it's the right time to start learning Pytorch." }, { "id": 831615, "postDate": "2020-05-03T13:36:38.817Z", "content": "Yes, it is a good idea so you save lots of time, it happen t to me, go with PyTorch and then go with Fastai, which sits on top of PyTorch and does most of the boiler code for you, but to do Fastai you need to know Pytorch.
\n\nThe problem with Keras is that it sits on top of Tensorflow so all the Tensorfow bugs are inherited y Keras automatically, beside is not that flexible.....
\n\nGood luck, may the force be with you.
", "rawMarkdown": "Yes, it is a good idea so you save lots of time, it happen t to me, go with PyTorch and then go with Fastai, which sits on top of PyTorch and does most of the boiler code for you, but to do Fastai you need to know Pytorch.\n\nThe problem with Keras is that it sits on top of Tensorflow so all the Tensorfow bugs are inherited y Keras automatically, beside is not that flexible..... \n\nGood luck, may the force be with you.", "votes": -1 }, { "id": 831663, "postDate": "2020-05-03T14:24:55.633Z", "content": "Can you elaborate on the pitfalls or bugs in the latest version of Tensorflow 2.0?
", "rawMarkdown": "Can you elaborate on the pitfalls or bugs in the latest version of Tensorflow 2.0?" }, { "id": 831672, "postDate": "2020-05-03T14:34:34.163Z", "content": "there are too many to mention, but the ones I remember before switching to Pytorh where in Object detection and especially using more than to GPUs.
", "rawMarkdown": "there are too many to mention, but the ones I remember before switching to Pytorh where in Object detection and especially using more than to GPUs. " } ] }, { "id": 1663223, "postDate": "2022-01-24T22:09:14.973Z", "content": "Another great breakdown on TensorFlow vs PyTorch: https://fall2019.fullstackdeeplearning.com/course-content/infrastructure-and-tooling/frameworks-and-distributed-training
", "rawMarkdown": "Another great breakdown on TensorFlow vs PyTorch: https://fall2019.fullstackdeeplearning.com/course-content/infrastructure-and-tooling/frameworks-and-distributed-training" }, { "id": 835244, "postDate": "2020-05-06T06:21:33.470Z", "content": "PyTorch is more like Python Native. TF was designed to support every language. It's mostly a personal choice.
", "rawMarkdown": "PyTorch is more like Python Native. TF was designed to support every language. It's mostly a personal choice." }, { "id": 832458, "postDate": "2020-05-04T06:44:03.977Z", "content": "IMHO, I've tried both and i think that TF is better for stable models or to work in production, but PyTorch is easier to code and try new things because is more \"pythonic\".
", "rawMarkdown": "IMHO, I've tried both and i think that TF is better for stable models or to work in production, but PyTorch is easier to code and try new things because is more \"pythonic\".\n" }, { "id": 834544, "postDate": "2020-05-05T15:44:10.677Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 834413, "postDate": "2020-05-05T14:27:22.193Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 833968, "postDate": "2020-05-05T07:37:59.093Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 831144, "author_name": "Matt Yates", "author_url": "", "post_date": "2020-05-03T07:16:52.667000", "content": "I've seen the trend on Kaggle switching to PyTorch over TF. Personally I still use Keras (PyTorch is on my to-do list to learn solely due to it's popularity). There is a lot of power in Keras that shouldn't go underestimated, so you can do a lot in Keras with less effort. Also, TF \\ Keras is still heavily used in industry. As with anything, programming languages come and go. Currently, both have value and both are good to know. I think PyTorch is growing in popularity due to it's push from Fast.ai \\ Jeremy Howard, and it's more Pythonic approach vs. TF.
\n\nOutside of the newly released ZeRO & DeepSpeed by Microsoft, I think TF\\Horovod has stronger distributed GPU support right now; but this is not something you'll see on Kaggle because Kaggle doesn't offer multi-GPU machines. You might see more Horovod if Kaggle offered multi-GPU machines. So for Kaggle, it also depends on the hardware. *(Doesn't mean that people aren't using Horovod or other things off of Kaggle for some competitions that don't require running a kernel on Kaggle itself). *
\n\nIn short, I think TF has the tenure and more features (and more industry tenure). PyTorch is the new guy with less tenure but it's growing super fast due to it's Pythonic design which is very appealing to many programmers.
\n\nEvolution of programming language popularity over time: https://www.youtube.com/watch?v=Og847HVwRSI
\n\nMicrosoft ZeRO & DeepSpeed: https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
\n\nOne of my Keras notebooks for this competition: https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-gpu
", "votes": 8, "replies": [ { "id": 831176, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-03T07:53:47.227000", "content": "@yeayates21 Thanks for the detailed explanation and for your hope in Tensorflow. \nYour notebook is a very nice baseline in Keras for this competition.
", "votes": 1, "replies": [] }, { "id": 831813, "author_name": "Matt Yates", "author_url": "", "post_date": "2020-05-03T16:06:50.087000", "content": "Np. Also, here is the Google trend chart on TF and PyTorch. As you can see a year ago, PyTorch wasn't even near TF use, but today they're neck and neck. So as others have said, it's not just this competition, it's a general trend that PyTorch is rising up and TF is going down.
\n\nMy guess is that, just like the Hadoop boom, TF won't go away it will just level off and PyTorch will continue to climb (for now). HDFS didn't go away over pure cloud services, it's still heavily used @ Uber and tons of companies, it just leveled off is all and I think TF will do the same.
\n\nAlso, PyTorch and TF are not the only games in town. You also have Chainer, MXNet, etc. and then within TF you have TF for Swift and TF.js ... and who knows what might come out next!
\n\nhttps://trends.google.com/trends/explore?geo=US&q=%2Fg%2F11bwp1s2k3,%2Fg%2F11gd3905v1
", "votes": 5, "replies": [] }, { "id": 831923, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-05-03T17:26:29.497000", "content": "NumPy arrays and PyTorch tensors\nNumpy is the most widely used library for scientific and numeric programming in Python, and provides very similar functionality and a very similar API to that provided by PyTorch; however, it does not support using the GPU, or calculating gradients, which are both critical for deep learning.
\n\n(Note that fastai adds some features to NumPy and PyTorch to make them a bit more similar to each other. If any code in this book doesn't work on your computer, it's possible that you forgot to include a line at the start of your notebook such as: from fastai.vision.all import *.)
\n\nBut what are arrays and tensors, and why should you care?
\n\nPython is slow compared to many languages. Anything fast in Python, NumPy or PyTorch is likely to be a wrapper to a compiled object written (and optimized) in another language - specifically C. In fact, NumPy arrays and PyTorch tensors can finish computations many thousands of times faster than using pure Python.
\n\nA NumPy array is a multidimensional table of data, with all items of the same type. Since that can be any type at all, they could even be arrays of arrays, with the innermost arrays potentially being different sizes — this is called a \"jagged array\". By \"multidimensional table\" we mean, for instance, a list (dimension of one), a table or matrix (dimension of two), a \"table of tables\" or a \"cube\" (dimension of three), and so forth. If the items are all of some simple type such as an integer or a float then NumPy will store them as a compact C data structure in memory. This is where NumPy shines. Numpy has a wide variety of operators and methods which can run computations on these compact structures at the same speed as optimized C, because they are written in optimized C.
\n\nA PyTorch tensor is nearly the same thing as a NumPy array, but with an additional restriction which unlocks some additional capabilities. It's the same in that it, too, is a multidimensional table of data, with all items of the same type. However, the restriction is that a tensor cannot use just any old type — it has to use a single basic numeric type for all components. As a result, a tensor is not as flexible as a genuine array of arrays, which allows jagged arrays, where the inner arrays could have different sizes. So a PyTorch tensor cannot be jagged. It is always a regularly shaped multidimensional rectangular structure.
\n\nThe vast majority of methods and operators supported by NumPy on these structures are also supported by PyTorch. But PyTorch tensors have additional capabilities. One major capability is that these structures can live on the GPU, in which case their computation will be optimized for the GPU, and can run much faster (given lots of values to work on). In addition, PyTorch can automatically calculate derivatives of these operations, including combinations of operations. As you'll see, it would be impossible to do deep learning in practice without this capability.
\n\ncopied from Fast.ai documentation, but it is the same in PyTorch documentation.
\n\nPlus, MXNET is as Pytorch. pretty much the same API, once you learn one, the other one it is eaiser to get your arms around to it.
", "votes": 1, "replies": [] }, { "id": 832042, "author_name": "Matt Yates", "author_url": "", "post_date": "2020-05-03T19:09:00.353000", "content": "Also, here is a great article on the PyTorch vs. TF conversation with more of an industry trend lens: https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304
", "votes": 2, "replies": [] } ] }, { "id": 830711, "author_name": "Innat", "author_url": "", "post_date": "2020-05-02T20:02:53.310000", "content": "I'm giving my opinion on this in a different way, IMHO If anyone shares high scored kernel in TensorFlow, PyTorch, Fast.ai or whatever, I think that would be popular. 😏
", "votes": 8, "replies": [ { "id": 830926, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-05-03T01:42:44.357000", "content": "That's true!
", "votes": 1, "replies": [] }, { "id": 831132, "author_name": "Matt Yates", "author_url": "", "post_date": "2020-05-03T06:57:03.127000", "content": "Agreed 👍 👌
", "votes": 1, "replies": [] }, { "id": 831617, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-05-03T13:38:25.537000", "content": "Lol .. You hit the Bulls eye :)
", "votes": 2, "replies": [] }, { "id": 831739, "author_name": "Innat", "author_url": "", "post_date": "2020-05-03T15:25:42.733000", "content": "😄 😄
", "votes": 1, "replies": [] }, { "id": 831763, "author_name": "SAIF UDDIN", "author_url": "", "post_date": "2020-05-03T15:41:56.590000", "content": "
Then look @ the lattes competitions winners.... I wonder why they use Pytorch...
", "votes": 1, "replies": [] }, { "id": 834022, "author_name": "Innat", "author_url": "", "post_date": "2020-05-05T08:32:15.680000", "content": "@oscarrangel If I understand the topic author's question, she has only concern for this competition (I think, she didn't mean in general). I would say PyTorch and TensorFlow both are good, it actually depends on us. @yeayates21 explained really well in his comment and shared some wonderful resources regarding this issue.
\n\nAs you wonder why in previous competition people chose PyTorch (when it may be not that popular), simple, it's their choice, they felt comfortable with this framework. They thought they could do diverse experimentation with these tools focusing on the problem. Switching to a new framework can be costly especially at the time of competition. I feel really weird when someone says something bad (too bad, even worse) about TensorFlow but couldn't explain properly why so. They faced a little but read shortcomings a lot from the web and speaks accordingly.
\n\nHowever, as far as popularity is concern among frameworks in a particular competition, it reminds of the Bengali.AI competition. Yes, same as this comp. there were also strong PyTorch resources and supports. It's only this Keras based kernel which was massively popular, just see its upvote and fork. People loved it and tried on it. I also made a public notebook with Keras but in a different pipeline. At the beginning and mid of this Bengali.AI competition, these two kernels were only the unique Keras/TF based kernel there. Plus, there were open secret and open source cool techniques implemented in PyTorch for boosting accuracy but nor for in Keras, such as MixUp, CutMix, OHEM. Later, two awesome Keras kernels published, one of them based on training strategies and another one was on TPU, COOL! Now there were more people who tried on these kernels and began to share their findings.
\n\nIn short, both PyTorch and TensorFlow are awesome. I think it depends on us. At the same time, Community Support Matters! But we should also remember that these are just tools, we should focus more on the problem. =)
", "votes": 3, "replies": [] }, { "id": 834711, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-05-05T18:07:03.690000", "content": "@ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.
\n\nBut at the end of the day.
\n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly.
\n\ntaken from fastai https://arxiv.org/abs/2002.04688
\n\nend of the @ipythonx I am answering your initial statement..... I really don't feel like getting in a long debate about the Pytorch vs Tensorflow, I been doing this for some time and I have used TF, Keras, MxNet, Pytorch, and I am stating my experience and opinion.
\n\nBut at the end of the day.
\n\nHowever, it doesn't really matter what software you learn, because it takes only a few days to learn to switch from one library to another. What really matters is learning the deep learning foundations and techniques properly.
\n\ntaken from fastai https://arxiv.org/abs/2002.04688
\n\nend of the discussion for me.
", "votes": -1, "replies": [] } ] }, { "id": 833242, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-05-04T17:26:56.493000", "content": "Simplicity and PyTorch Model Zoo is more than what tf has to offer; IT's very easy to control what's happening here; Plus helpers like fastAI, PyTorch lightning, Catalyst etc makes live easier!
\n\nPlus nearly many re-search papers which i have read, if had implementations, it's also in PyTorch wrt CV nowadays as it's Simply Python!
\n\nIn short Ecosystem is very strong for kagglers if they use PyTorch but that doesn't mean it cannot be done on TF as well; Pretty much they are the same or converging to be one in terms of specs!
", "votes": 4, "replies": [] }, { "id": 830917, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-05-03T01:11:05.467000", "content": "I think the improvement in the simplicity of the code tensorflow is improving over time is remarkable but still its pytorch for the best...
\n\nThe reason why no one uses tensorflow is because of its non simplicity for the beginner and less easy to fully customize things, people including me just does not like to use it, however keras still remains the best and the easiest to get a baseline working but over time you will have to switch to either tensorflow or pytorch. Fastai is good as it gives you the power that just wraps around the pytorch providing more custom features than keras.
", "votes": 1, "replies": [] }, { "id": 830743, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-05-02T20:55:14.840000", "content": "I don't think it has anything to do with this comp, just that people prefer Pytorch over Tensorflow.
", "votes": 1, "replies": [] }, { "id": 830801, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-05-02T23:13:43.463000", "content": "in my personal opinion, I have tried both and like better PyTorch, especially if you are treating to migrate from old versions of TensorFlow, with the new 2.xx it is very buggy and lost of things don't run,
\n\nI believe because it is an older framework and they are trying to adapt it to new things and they are having problems doing it.
\n\nPytorch is much less complicated and has much better support.
\n\nThis has been my own personal experience.
", "votes": 0, "replies": [ { "id": 831166, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-03T07:47:52.303000", "content": "Yeah, I'm starting to realize the pitfalls of TF too. Can't really compare to Pytorch because I have never used it. Seems like it's the right time to start learning Pytorch.
", "votes": 0, "replies": [] }, { "id": 831615, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-05-03T13:36:38.817000", "content": "Yes, it is a good idea so you save lots of time, it happen t to me, go with PyTorch and then go with Fastai, which sits on top of PyTorch and does most of the boiler code for you, but to do Fastai you need to know Pytorch.
\n\nThe problem with Keras is that it sits on top of Tensorflow so all the Tensorfow bugs are inherited y Keras automatically, beside is not that flexible.....
\n\nGood luck, may the force be with you.
", "votes": -1, "replies": [] }, { "id": 831663, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-05-03T14:24:55.633000", "content": "Can you elaborate on the pitfalls or bugs in the latest version of Tensorflow 2.0?
", "votes": 0, "replies": [] }, { "id": 831672, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-05-03T14:34:34.163000", "content": "there are too many to mention, but the ones I remember before switching to Pytorh where in Object detection and especially using more than to GPUs.
", "votes": 0, "replies": [] } ] }, { "id": 1663223, "author_name": "Matt Yates", "author_url": "", "post_date": "2022-01-24T22:09:14.973000", "content": "Another great breakdown on TensorFlow vs PyTorch: https://fall2019.fullstackdeeplearning.com/course-content/infrastructure-and-tooling/frameworks-and-distributed-training
", "votes": 0, "replies": [] }, { "id": 835244, "author_name": "mlpll90", "author_url": "", "post_date": "2020-05-06T06:21:33.470000", "content": "PyTorch is more like Python Native. TF was designed to support every language. It's mostly a personal choice.
", "votes": 0, "replies": [] }, { "id": 832458, "author_name": "Yael Tudela", "author_url": "", "post_date": "2020-05-04T06:44:03.977000", "content": "IMHO, I've tried both and i think that TF is better for stable models or to work in production, but PyTorch is easier to code and try new things because is more \"pythonic\".
", "votes": 0, "replies": [] }, { "id": 834544, "author_name": "", "author_url": "", "post_date": "2020-05-05T15:44:10.677000", "content": "", "votes": -2, "replies": [] }, { "id": 834413, "author_name": "", "author_url": "", "post_date": "2020-05-05T14:27:22.193000", "content": "", "votes": -1, "replies": [] }, { "id": 833968, "author_name": "", "author_url": "", "post_date": "2020-05-05T07:37:59.093000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "829903": "In this competition, I have observed there are more (almost all) kernels written using Pytorch.\nCan anyone explain why tensorflow is so unpopular in this competition?", "831144": "I've seen the trend on Kaggle switching to PyTorch over TF. Personally I still use Keras (PyTorch is on my to-do list to learn solely due to it's popularity). There is a lot of power in Keras that shouldn't go underestimated, so you can do a lot in Keras with less effort. Also, TF \\ Keras is still heavily used in industry. As with anything, programming languages come and go. Currently, both have value and both are good to know. I think PyTorch is growing in popularity due to it's push from Fast.ai \\ Jeremy Howard, and it's more Pythonic approach vs. TF. \n\nOutside of the newly released ZeRO & DeepSpeed by Microsoft, I think TF\\Horovod has stronger distributed GPU support right now; but this is not something you'll see on Kaggle because Kaggle doesn't offer multi-GPU machines. You might see more Horovod if Kaggle offered multi-GPU machines. So for Kaggle, it also depends on the hardware. *(Doesn't mean that people aren't using Horovod or other things off of Kaggle for some competitions that don't require running a kernel on Kaggle itself). *\n\nIn short, I think TF has the tenure and more features (and more industry tenure). PyTorch is the new guy with less tenure but it's growing super fast due to it's Pythonic design which is very appealing to many programmers.\n\nEvolution of programming language popularity over time: https://www.youtube.com/watch?v=Og847HVwRSI\n\nMicrosoft ZeRO & DeepSpeed: https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/\n\nOne of my Keras notebooks for this competition: https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-gpu", "830711": "I'm giving my opinion on this in a different way, IMHO **If anyone shares high scored kernel in TensorFlow, PyTorch, Fast.ai or whatever, I think that would be popular.** 😏 ", "833242": "Simplicity and PyTorch Model Zoo is more than what tf has to offer; IT's very easy to control what's happening here; Plus helpers like fastAI, PyTorch lightning, Catalyst etc makes live easier!\n\nPlus nearly many re-search papers which i have read, if had implementations, it's also in PyTorch wrt CV nowadays as it's Simply Python!\n\nIn short Ecosystem is very strong for kagglers if they use PyTorch but that doesn't mean it cannot be done on TF as well; Pretty much they are the same or converging to be one in terms of specs!", "830917": "I think the improvement in the simplicity of the code tensorflow is improving over time is remarkable but still its pytorch for the best...\n\nThe reason why no one uses tensorflow is because of its non simplicity for the beginner and less easy to fully customize things, people including me just does not like to use it, however keras still remains the best and the easiest to get a baseline working but over time you will have to switch to either tensorflow or pytorch. Fastai is good as it gives you the power that just wraps around the pytorch providing more custom features than keras.", "830743": "I don't think it has anything to do with this comp, just that people prefer Pytorch over Tensorflow.", "830801": "in my personal opinion, I have tried both and like better PyTorch, especially if you are treating to migrate from old versions of TensorFlow, with the new 2.xx it is very buggy and lost of things don't run, \n\nI believe because it is an older framework and they are trying to adapt it to new things and they are having problems doing it.\n\nPytorch is much less complicated and has much better support.\n\nThis has been my own personal experience.", "1663223": "Another great breakdown on TensorFlow vs PyTorch: https://fall2019.fullstackdeeplearning.com/course-content/infrastructure-and-tooling/frameworks-and-distributed-training", "835244": "PyTorch is more like Python Native. TF was designed to support every language. It's mostly a personal choice.", "832458": "IMHO, I've tried both and i think that TF is better for stable models or to work in production, but PyTorch is easier to code and try new things because is more \"pythonic\".\n", "834544": "", "834413": "", "833968": "" } }