![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F789366%2F02c2b9238524242279f2504ef8821734%2FCapture.PNG?generation=1590151952480407&alt=media\")
\n ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F789366%2F7b9f01aac3c99f0c43d8e5e6c55c2da4%2FCapture1.PNG?generation=1590151953238420&alt=media\")
Here is an image-only dataset with level 1 and level 2 images (cropped), no masks. It's size is 9 GB to download, around 11 GB uncompressed. Link: https://www.kaggle.com/lopuhin/panda-2020-level-1-2
", "messages": [ { "id": 832632, "postDate": "2020-05-04T10:00:04.077Z", "content": "Here is an image-only dataset with level 1 and level 2 images (cropped), no masks. It's size is 9 GB to download, around 11 GB uncompressed. Link: https://www.kaggle.com/lopuhin/panda-2020-level-1-2
", "rawMarkdown": "Here is an image-only dataset with level 1 and level 2 images (cropped), no masks. It's size is 9 GB to download, around 11 GB uncompressed. Link: https://www.kaggle.com/lopuhin/panda-2020-level-1-2", "votes": 36 }, { "id": 871698, "postDate": "2020-06-02T15:22:47.617Z", "content": "thanks for sharing! have you by chance encountered issues related to PIL.Image.DecompressionBombError? I get it when training on 25x256x256 images made from tiles
ok, solved it, for whoever might be interested, you can increase the maximum allowed size: https://stackoverflow.com/questions/51152059/pillow-in-python-wont-let-me-open-image-exceeds-limit
", "rawMarkdown": "ok, solved it, for whoever might be interested, you can increase the maximum allowed size: https://stackoverflow.com/questions/51152059/pillow-in-python-wont-let-me-open-image-exceeds-limit", "votes": 1 }, { "id": 874356, "postDate": "2020-06-04T21:40:15.083Z", "content": "PIL.Image.MAX_IMAGE_PIXELS = None instead of maximum size to skip that DOS verifier.
Hey, thanks for sharing :) Did you get your best results out of this dataset or did you have to use original tiffs?
", "rawMarkdown": "Hey, thanks for sharing :) Did you get your best results out of this dataset or did you have to use original tiffs? ", "replies": [ { "id": 857068, "postDate": "2020-05-22T09:21:59.350Z", "content": "My best submission is trained on this dataset on level 1, I use original tiffs only for submission (the only trick is to re-apply jpeg compression/decompression during submission). I'm yet to train a model on higher resolution.
", "rawMarkdown": "My best submission is trained on this dataset on level 1, I use original tiffs only for submission (the only trick is to re-apply jpeg compression/decompression during submission). I'm yet to train a model on higher resolution.", "votes": 3 }, { "id": 857094, "postDate": "2020-05-22T09:44:10.863Z", "content": "Thanks. As I understand, to submit one has to repeat image_to_jpeg method partially (without saving). And do compression with the same quality (90 in your case). Am I understand that correctly? Can you provide some pseudo code for submission part? The part with compression/decompression is unclear for me ((
Yes, here is the actual code which I use (it's not optimal, I plan to move away from PIL here):
\n\n\n image = crop_white(skimage.io.MultiImage(\n str(self.root / f'{item.image_id}.tiff'))[self.level])\n if self.level != 0:\n # use PIL as jpeg4py is not available on kaggle\n buffer = io.BytesIO()\n Image.fromarray(image).save(buffer, format='jpeg', quality=90)\n image = np.array(Image.open(buffer))\n
image = skimage.io.MultiImage(str(path))[1]\nimage = crop_white(image)\nimage = Image.fromarray(image)\ninmem = io.BytesIO()\nimage.save(inmem, format=\"jpeg\", quality=90)\nimg = Image.open(inmem)\n# continue with preprocessing for your network here\nShould be something like this?
\n\n\nShould be something like this?
\n
yes looks look to me
", "rawMarkdown": "> Should be something like this?\n\nyes looks look to me" }, { "id": 857157, "postDate": "2020-05-22T10:57:46.830Z", "content": "Thanks. Get into the same code, while this page was not updated! )))
", "rawMarkdown": "Thanks. Get into the same code, while this page was not updated! )))", "votes": 1 }, { "id": 857209, "postDate": "2020-05-22T12:17:03.810Z", "content": "@lopuhin Did you compare how much faster is to train with jpeg instead of tiff?
", "rawMarkdown": "@lopuhin Did you compare how much faster is to train with jpeg instead of tiff?" }, { "id": 857240, "postDate": "2020-05-22T12:52:42.263Z", "content": "Quick comparison load tiffs vs jpeg on the kaggle kernel gave me roughly the same time\n
Thanks for the dataset @lopuhin, I use it when running on servers, really helpful! Interestingly, what I've noticed is that, whether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression (I set it to always_apply=True, and the quality range to be 90-90). However, I guess it's such an easy thing to include in the submission code anyways so it's unimportant in that sense.. :-)
", "rawMarkdown": "Thanks for the dataset @lopuhin, I use it when running on servers, really helpful! Interestingly, what I've noticed is that, whether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression (I set it to always_apply=True, and the quality range to be 90-90). However, I guess it's such an easy thing to include in the submission code anyways so it's unimportant in that sense.. :-)", "votes": 1 }, { "id": 858060, "postDate": "2020-05-23T07:33:48.317Z", "content": "\n\n\nI use it when running on servers, really helpful!
\n
Yep that was my motivation as well, to quickly copy it.
\n\n\n\n\nwhether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression
\n
For me the difference was around 0.02 both on LB and validation, but I didn't use this augmentation - probably that's the reason.
\n\n\n\n\nQuick comparison load tiffs vs jpeg on the kaggle kernel gave me roughly the same time
\n
Good point - I don't recall exact numbers, the difference may be not so large. Two notes here: jpeg4py was around 2x faster than cv2 IIRC (I'm not sure if proper turbojpeg is available on kaggle though), and also what matters for dataloader speed is not \"wall time\" but \"total time\", because you're launching multiple workers in any case. Smaller wall time but larger total time means multiple cores are utilized.
", "rawMarkdown": "> I use it when running on servers, really helpful! \n\nYep that was my motivation as well, to quickly copy it.\n\n> whether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression\n\nFor me the difference was around 0.02 both on LB and validation, but I didn't use this augmentation - probably that's the reason.\n\n> Quick comparison load tiffs vs jpeg on the kaggle kernel gave me roughly the same time\n\nGood point - I don't recall exact numbers, the difference may be not so large. Two notes here: jpeg4py was around 2x faster than cv2 IIRC (I'm not sure if proper turbojpeg is available on kaggle though), and also what matters for dataloader speed is not \"wall time\" but \"total time\", because you're launching multiple workers in any case. Smaller wall time but larger total time means multiple cores are utilized.", "votes": 1 }, { "id": 858816, "postDate": "2020-05-23T20:35:12.477Z", "content": "Yep, compared both locally as a part of dataloaders with the same number of workers, still no significant difference.\nAnyways, good to have if need to copy 11GB is better than 300GB )))
", "rawMarkdown": "Yep, compared both locally as a part of dataloaders with the same number of workers, still no significant difference.\nAnyways, good to have if need to copy 11GB is better than 300GB )))", "votes": 1 } ] }, { "id": 833112, "postDate": "2020-05-04T15:57:48.807Z", "content": "Thanks 🙏Will be great help.
", "rawMarkdown": "Thanks 🙏Will be great help." } ], "comments": [ { "id": 871698, "author_name": "Marco Perini", "author_url": "", "post_date": "2020-06-02T15:22:47.617000", "content": "thanks for sharing! have you by chance encountered issues related to PIL.Image.DecompressionBombError? I get it when training on 25x256x256 images made from tiles
ok, solved it, for whoever might be interested, you can increase the maximum allowed size: https://stackoverflow.com/questions/51152059/pillow-in-python-wont-let-me-open-image-exceeds-limit
", "votes": 1, "replies": [] }, { "id": 874356, "author_name": "BachT", "author_url": "", "post_date": "2020-06-04T21:40:15.083000", "content": "PIL.Image.MAX_IMAGE_PIXELS = None instead of maximum size to skip that DOS verifier.
Hey, thanks for sharing :) Did you get your best results out of this dataset or did you have to use original tiffs?
", "votes": 0, "replies": [ { "id": 857068, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-22T09:21:59.350000", "content": "My best submission is trained on this dataset on level 1, I use original tiffs only for submission (the only trick is to re-apply jpeg compression/decompression during submission). I'm yet to train a model on higher resolution.
", "votes": 3, "replies": [] }, { "id": 857094, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-05-22T09:44:10.863000", "content": "Thanks. As I understand, to submit one has to repeat image_to_jpeg method partially (without saving). And do compression with the same quality (90 in your case). Am I understand that correctly? Can you provide some pseudo code for submission part? The part with compression/decompression is unclear for me ((
Yes, here is the actual code which I use (it's not optimal, I plan to move away from PIL here):
\n\n\n image = crop_white(skimage.io.MultiImage(\n str(self.root / f'{item.image_id}.tiff'))[self.level])\n if self.level != 0:\n # use PIL as jpeg4py is not available on kaggle\n buffer = io.BytesIO()\n Image.fromarray(image).save(buffer, format='jpeg', quality=90)\n image = np.array(Image.open(buffer))\n
image = skimage.io.MultiImage(str(path))[1]\nimage = crop_white(image)\nimage = Image.fromarray(image)\ninmem = io.BytesIO()\nimage.save(inmem, format=\"jpeg\", quality=90)\nimg = Image.open(inmem)\n# continue with preprocessing for your network here\nShould be something like this?
\n\n\nShould be something like this?
\n
yes looks look to me
", "votes": 0, "replies": [] }, { "id": 857157, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-05-22T10:57:46.830000", "content": "Thanks. Get into the same code, while this page was not updated! )))
", "votes": 1, "replies": [] }, { "id": 857209, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-05-22T12:17:03.810000", "content": "@lopuhin Did you compare how much faster is to train with jpeg instead of tiff?
", "votes": 0, "replies": [] }, { "id": 857240, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-05-22T12:52:42.263000", "content": "Quick comparison load tiffs vs jpeg on the kaggle kernel gave me roughly the same time\n
Thanks for the dataset @lopuhin, I use it when running on servers, really helpful! Interestingly, what I've noticed is that, whether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression (I set it to always_apply=True, and the quality range to be 90-90). However, I guess it's such an easy thing to include in the submission code anyways so it's unimportant in that sense.. :-)
", "votes": 1, "replies": [] }, { "id": 858060, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2020-05-23T07:33:48.317000", "content": "\n\n\nI use it when running on servers, really helpful!
\n
Yep that was my motivation as well, to quickly copy it.
\n\n\n\n\nwhether or not I compress the .tiff images to jpeg on submission have no (or insignificant) effect on LB.. Note that I use albumentations JpegCompression
\n
For me the difference was around 0.02 both on LB and validation, but I didn't use this augmentation - probably that's the reason.
\n\n\n\n\nQuick comparison load tiffs vs jpeg on the kaggle kernel gave me roughly the same time
\n
Good point - I don't recall exact numbers, the difference may be not so large. Two notes here: jpeg4py was around 2x faster than cv2 IIRC (I'm not sure if proper turbojpeg is available on kaggle though), and also what matters for dataloader speed is not \"wall time\" but \"total time\", because you're launching multiple workers in any case. Smaller wall time but larger total time means multiple cores are utilized.
", "votes": 1, "replies": [] }, { "id": 858816, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-05-23T20:35:12.477000", "content": "Yep, compared both locally as a part of dataloaders with the same number of workers, still no significant difference.\nAnyways, good to have if need to copy 11GB is better than 300GB )))
", "votes": 1, "replies": [] } ] }, { "id": 833112, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-05-04T15:57:48.807000", "content": "Thanks 🙏Will be great help.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "832632": "Here is an image-only dataset with level 1 and level 2 images (cropped), no masks. It's size is 9 GB to download, around 11 GB uncompressed. Link: https://www.kaggle.com/lopuhin/panda-2020-level-1-2", "871698": "thanks for sharing! have you by chance encountered issues related to `PIL.Image.DecompressionBombError`? I get it when training on 25x256x256 images made from tiles", "857061": "Hey, thanks for sharing :) Did you get your best results out of this dataset or did you have to use original tiffs? ", "833112": "Thanks 🙏Will be great help." } }