![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F95ec86cd2452068bde95ff4b09235f75%2FScreenshot_96.jpg?generation=1685688703795772&alt=media\")
This load function comes from this git repo:
\n :\n ():\n file=(file,)\n header = file.read()\n descr = (header[:], ).replace(,).replace(,)\n shape = ((num) num (header[:], ).replace(, ).replace(, ).replace(, ).split())\n datasize = np.lib..descr_to_dtype(descr).itemsize\n dimension shape:\n datasize *= dimension\n np.ndarray(shape, dtype=descr, buffer=file.read(datasize))\n
my epoch time went from 4mins to 2:30mins with this simple change
", "messages": [ { "id": 2284589, "postDate": "2023-06-02T06:53:55.433Z", "content": "This load function comes from this git repo:
\n :\n ():\n file=(file,)\n header = file.read()\n descr = (header[:], ).replace(,).replace(,)\n shape = ((num) num (header[:], ).replace(, ).replace(, ).replace(, ).split())\n datasize = np.lib..descr_to_dtype(descr).itemsize\n dimension shape:\n datasize *= dimension\n np.ndarray(shape, dtype=descr, buffer=file.read(datasize))\n
my epoch time went from 4mins to 2:30mins with this simple change
", "rawMarkdown": "**This load function comes from [this](https://github.com/divideconcept/fastnumpyio) git repo:**\n\n```python\nclass fastnumpyio:\n def load(file):\n file=open(file,\"rb\")\n header = file.read(128)\n descr = str(header[19:25], 'utf-8').replace(\"'\",\"\").replace(\" \",\"\")\n shape = tuple(int(num) for num in str(header[60:120], 'utf-8').replace(', }', '').replace('(', '').replace(')', '').split(','))\n datasize = np.lib.format.descr_to_dtype(descr).itemsize\n for dimension in shape:\n datasize *= dimension\n return np.ndarray(shape, dtype=descr, buffer=file.read(datasize))\n```\n\n\n\n*my epoch time went from 4mins to 2:30mins with this simple change*", "votes": 41 }, { "id": 2284776, "postDate": "2023-06-02T09:13:31.890Z", "content": "Wow, this appears to be highly useful! Thanks for the post @janmpia!
", "rawMarkdown": "Wow, this appears to be highly useful! Thanks for the post @janmpia!", "votes": 3 }, { "id": 2284592, "postDate": "2023-06-02T06:58:58.500Z", "content": "If you are willing to sacrifice the adaptivness of your code, you can also do the following for a 2x:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F4d2739538ebb540155d2c405fe0798bb%2FScreenshot_97.jpg?generation=1685689109654626&alt=media\")
What do you mean with \"adaptiveness\"?
", "rawMarkdown": "What do you mean with \"adaptiveness\"?\n", "votes": 1, "replies": [ { "id": 2285289, "postDate": "2023-06-02T15:31:03.910Z", "content": "hey @janhuebik,
\nfor this to work, you have a write the dtype and shape of thr array you are loading, meaning that if you want to upscale the images or change to dtype to float16 for example, you will have to change that too, hence the adaptiveness.
\nOn top of that, you will now be working with menmap and not numpy array, which will need be taken care of.
\nHope that makes my point clearer.
this is helpful i think
", "rawMarkdown": "this is helpful i think\n", "votes": 1 }, { "id": 2286653, "postDate": "2023-06-03T16:06:39.527Z", "content": "very useful content @janmpia
", "rawMarkdown": "very useful content @janmpia ", "votes": 1 }, { "id": 2294664, "postDate": "2023-06-10T08:00:38.217Z", "content": "I have made kernel to compare all numpy methods (reading fp16 data) vs JPEG. https://www.kaggle.com/code/melgor/benchmark-numpy-vs-jpeg
\nOptions with scores:
\nTiming vary based on run, but fastnumpy/memmap is enough for this competition.
", "rawMarkdown": "I have made kernel to compare all numpy methods (reading fp16 data) vs JPEG. https://www.kaggle.com/code/melgor/benchmark-numpy-vs-jpeg\n\nOptions with scores:\n- raw numpy: 1:48\n- fastnumpy: 0:50\n- numpy-memmap: 0:49\n- jpeg: 1:13\n\nTiming vary based on run, but fastnumpy/memmap is enough for this competition.", "votes": 2, "replies": [ { "id": 2295114, "postDate": "2023-06-10T15:40:31.357Z", "content": "Hey @melgor,
\nI'm kind of confused by your conclusion after seeing your notebook:
\n\nTiming vary based on run, but numpy is enough for this competition.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F792949947e52943c1478d181384d23cf%2FScreenshot_104.jpg?generation=1686410876904695&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F2266719ca8ebc1c6ead933b1eddbe7ad%2FScreenshot_105.jpg?generation=1686410886826231&alt=media\")
from those screen shots of your notebook, fastnumpy looks much faster.
", "rawMarkdown": "Hey @melgor, \nI'm kind of confused by your conclusion after seeing your notebook:\n>Timing vary based on run, **but numpy is enough for this competition.**\n\n\n\n\nfrom those screen shots of your notebook, fastnumpy looks much faster.", "votes": 2, "replies": [ { "id": 2295356, "postDate": "2023-06-10T20:35:21.087Z", "content": "I should be more specific as I have in my mind Numpy vs JPEG.
\nBut just comparing just Numpy methods, fastnumpy or memmap should be used.
", "rawMarkdown": "I should be more specific as I have in my mind Numpy vs JPEG. \n\nBut just comparing just Numpy methods, fastnumpy or memmap should be used.", "votes": 1 } ] } ] }, { "id": 2317537, "postDate": "2023-06-25T19:04:24.243Z", "content": "Git repo link is broken.
", "rawMarkdown": "Git repo link is broken.", "replies": [ { "id": 2343473, "postDate": "2023-07-13T17:27:02.100Z", "content": "oh thank you for telling me. I updated the topic
", "rawMarkdown": "oh thank you for telling me. I updated the topic" } ] }, { "id": 2286960, "postDate": "2023-06-04T02:47:37.550Z", "content": "(From translator) Does the memory expense increase?
", "rawMarkdown": "(From translator) Does the memory expense increase?", "isDeleted": true, "replies": [ { "id": 2287249, "postDate": "2023-06-04T08:56:17.227Z", "content": "Hey @mozattt,
\nI didn't test that to be honest but my code only sped up without any other side effect.
\nNote that you get a speed up only with numpy arrays that don't have a complex structure, but for this competition there are none !
Thanks. :D
", "rawMarkdown": "Thanks. :D", "isDeleted": true } ] } ] } ], "comments": [ { "id": 2284776, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2023-06-02T09:13:31.890000", "content": "Wow, this appears to be highly useful! Thanks for the post @janmpia!
", "votes": 3, "replies": [] }, { "id": 2284592, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-02T06:58:58.500000", "content": "If you are willing to sacrifice the adaptivness of your code, you can also do the following for a 2x:
\n
What do you mean with \"adaptiveness\"?
", "votes": 1, "replies": [ { "id": 2285289, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-02T15:31:03.910000", "content": "hey @janhuebik,
\nfor this to work, you have a write the dtype and shape of thr array you are loading, meaning that if you want to upscale the images or change to dtype to float16 for example, you will have to change that too, hence the adaptiveness.
\nOn top of that, you will now be working with menmap and not numpy array, which will need be taken care of.
\nHope that makes my point clearer.
this is helpful i think
", "votes": 1, "replies": [] }, { "id": 2286653, "author_name": "Tamanna Akter Swarna", "author_url": "", "post_date": "2023-06-03T16:06:39.527000", "content": "very useful content @janmpia
", "votes": 1, "replies": [] }, { "id": 2294664, "author_name": "Bartek", "author_url": "", "post_date": "2023-06-10T08:00:38.217000", "content": "I have made kernel to compare all numpy methods (reading fp16 data) vs JPEG. https://www.kaggle.com/code/melgor/benchmark-numpy-vs-jpeg
\nOptions with scores:
\nTiming vary based on run, but fastnumpy/memmap is enough for this competition.
", "votes": 2, "replies": [ { "id": 2295114, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-10T15:40:31.357000", "content": "Hey @melgor,
\nI'm kind of confused by your conclusion after seeing your notebook:
\n\nTiming vary based on run, but numpy is enough for this competition.
\n
\n
from those screen shots of your notebook, fastnumpy looks much faster.
", "votes": 2, "replies": [ { "id": 2295356, "author_name": "Bartek", "author_url": "", "post_date": "2023-06-10T20:35:21.087000", "content": "I should be more specific as I have in my mind Numpy vs JPEG.
\nBut just comparing just Numpy methods, fastnumpy or memmap should be used.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2317537, "author_name": "Pankaj Joshi", "author_url": "", "post_date": "2023-06-25T19:04:24.243000", "content": "Git repo link is broken.
", "votes": 0, "replies": [ { "id": 2343473, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-07-13T17:27:02.100000", "content": "oh thank you for telling me. I updated the topic
", "votes": 0, "replies": [] } ] }, { "id": 2286960, "author_name": "", "author_url": "", "post_date": "2023-06-04T02:47:37.550000", "content": "(From translator) Does the memory expense increase?
", "votes": 0, "replies": [ { "id": 2287249, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-04T08:56:17.227000", "content": "Hey @mozattt,
\nI didn't test that to be honest but my code only sped up without any other side effect.
\nNote that you get a speed up only with numpy arrays that don't have a complex structure, but for this competition there are none !
Thanks. :D
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2284589": "**This load function comes from [this](https://github.com/divideconcept/fastnumpyio) git repo:**\n\n```python\nclass fastnumpyio:\n def load(file):\n file=open(file,\"rb\")\n header = file.read(128)\n descr = str(header[19:25], 'utf-8').replace(\"'\",\"\").replace(\" \",\"\")\n shape = tuple(int(num) for num in str(header[60:120], 'utf-8').replace(', }', '').replace('(', '').replace(')', '').split(','))\n datasize = np.lib.format.descr_to_dtype(descr).itemsize\n for dimension in shape:\n datasize *= dimension\n return np.ndarray(shape, dtype=descr, buffer=file.read(datasize))\n```\n\n\n\n*my epoch time went from 4mins to 2:30mins with this simple change*", "2284776": "Wow, this appears to be highly useful! Thanks for the post @janmpia!", "2284592": "**If you are willing to sacrifice the adaptivness of your code, you can also do the following for a 2x:**\n", "2291790": "this is helpful i think\n", "2286653": "very useful content @janmpia ", "2294664": "I have made kernel to compare all numpy methods (reading fp16 data) vs JPEG. https://www.kaggle.com/code/melgor/benchmark-numpy-vs-jpeg\n\nOptions with scores:\n- raw numpy: 1:48\n- fastnumpy: 0:50\n- numpy-memmap: 0:49\n- jpeg: 1:13\n\nTiming vary based on run, but fastnumpy/memmap is enough for this competition.", "2317537": "Git repo link is broken.", "2286960": "(From translator) Does the memory expense increase?" } }