I've published various datasets in different format to help you get started:
\nThis is a work in progress; I create new notebooks to also generate 256x256 and 1024x1024 images, but this will take time so keep an eye on this thread.
\nOriginal post: I published this notebook showing how to process and resize the training data. You can directly use the notebook as an input to your model, or you can click on the \"+ New Dataset\" button in the output section to create a dataset that you can use.
\nI'm waiting for an answer from the organizer in order to create a public dataset that you can use directly without creating your own; I will update this post if I get a positive answer.
", "messages": [ { "id": 1134324, "postDate": "2021-01-01T06:40:04.917Z", "content": "I've published various datasets in different format to help you get started:
\nThis is a work in progress; I create new notebooks to also generate 256x256 and 1024x1024 images, but this will take time so keep an eye on this thread.
\nOriginal post: I published this notebook showing how to process and resize the training data. You can directly use the notebook as an input to your model, or you can click on the \"+ New Dataset\" button in the output section to create a dataset that you can use.
\nI'm waiting for an answer from the organizer in order to create a public dataset that you can use directly without creating your own; I will update this post if I get a positive answer.
", "rawMarkdown": "I've published various datasets in different format to help you get started:\n\n* [PNG 512px with original aspect ratio](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-png-512px-original-ratio) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-original-ratio)): Lossless and larger, the images will retain their aspect ratios and the largest side will have a size of 512px (so a 1000x500 image will be downsized to 512x256). To use this in a model, you will need to add padding (e.g. [using numpy](https://numpy.org/doc/stable/reference/generated/numpy.pad.html)) to the smaller sides (so 512x256 would be padded to 512x512).\n* [PNG 256x256](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-256x256) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-png-256x256)): Lossless and larger, each image is guaranteed to be 256x256 but will lose their aspect ratios.\n* [PNG 512x512](https://www.kaggle.com/xhlulu/vinbigdata) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-image)): Lossless and larger, each image is guaranteed to be 512x512 but will lose their aspect ratios.\n* [PNG 1024x1024](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-1024x1024) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-png-1024x1024)): Lossless and larger, each image is guaranteed to be 1024x1024 but will lose their aspect ratios.\n* [JPG 512x512](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-jpg-512x512) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-jpg)): Lossy but smaller files, each image is guaranteed to be 512x512 but will lose their aspect ratios.\n\nThis is a work in progress; I create new notebooks to also generate 256x256 and 1024x1024 images, but this will take time so keep an eye on this thread.\n\n---\n\n**Original post**: I published [this notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-image) showing how to process and resize the training data. You can directly use the notebook as an input to your model, or you can click on the \"+ New Dataset\" button in the output section to create a dataset that you can use.\n\nI'm waiting for an answer from the organizer in order to create a public dataset that you can use directly without creating your own; I will update this post if I get a positive answer.", "votes": 138 }, { "id": 1207560, "postDate": "2021-02-17T23:19:23.700Z", "content": "I uploaded original sized image
\nThanks for the datasets, they come in handy since I don't have enough disk space to process the original 200GB
\ndataset.
However, one thing I'd like to suggest: your dim0 and dim1 in metadata files are in unusual order - (height, width), when usually it's (width, height). So, dim0 should correspond to x-axis, and dim1 - to y-axis, but it's the other way around..
\nI've spent some time wondering why my YOLO model is discarding ~100 images as incorrectly labelled, and then WBF also mentioned that some coordinated were > 1.
", "rawMarkdown": "Thanks for the datasets, they come in handy since I don't have enough disk space to process the original 200GB \ndataset.\n\nHowever, one thing I'd like to suggest: your dim0 and dim1 in metadata files are in unusual order - (height, width), when usually it's (width, height). So, dim0 should correspond to x-axis, and dim1 - to y-axis, but it's the other way around..\n\nI've spent some time wondering why my YOLO model is discarding ~100 images as incorrectly labelled, and then WBF also mentioned that some coordinated were > 1.", "votes": 1, "replies": [ { "id": 1242753, "postDate": "2021-03-17T19:07:07.653Z", "content": "Actually it makes sense when you think about it in numpy arrays.. But it is true, depending on the method, module, etc. the order of height/width changes which is confusing some times..
", "rawMarkdown": "Actually it makes sense when you think about it in numpy arrays.. But it is true, depending on the method, module, etc. the order of height/width changes which is confusing some times.." } ] }, { "id": 1142346, "postDate": "2021-01-07T10:27:23.310Z", "content": "I've slightly changed your notebook for faster processing. Here is with maximum image size 1024 with original aspect ration. Version with 640px.
", "rawMarkdown": "I've slightly changed your notebook for faster processing. [Here](https://www.kaggle.com/notvuvko/vinbigdata-process-and-resize-original-ratio?scriptVersionId=51233352) is with maximum image size 1024 with original aspect ration. [Version](https://www.kaggle.com/notvuvko/vinbigdata-process-and-resize-original-ratio?scriptVersionId=51254264) with 640px.", "votes": 2 }, { "id": 1138006, "postDate": "2021-01-04T10:43:31.373Z", "content": "Thank you, upvoted ! I have a (stupid) question, as a beginner in detection. How do I use the original bounding box coordinates with these images ? The original images have different sizes right ? So you would have to get the scaling for each original image to 512*512 to apply it to the boxes coords ?
", "rawMarkdown": "Thank you, upvoted ! I have a (stupid) question, as a beginner in detection. How do I use the original bounding box coordinates with these images ? The original images have different sizes right ? So you would have to get the scaling for each original image to 512*512 to apply it to the boxes coords ?", "votes": 2, "replies": [ { "id": 1138189, "postDate": "2021-01-04T13:32:00.980Z", "content": "Hi, I don't know how images are preprocessed in this particular dataset, but usually in object detection tasks you store coordinates of bounding boxes as numbers within the interval [0, 1] dividing bounding boxes coordinates by image shapes:
x_min = x_coord_min_original_shape / original_image_width\ny_min = y_coord_min_original_shape / original_image_height\nx_max = x_coord_max_original_shape / original_image_width\ny_max = y_coord_max_original_shape / original_image_height\nThus you can compute rescaled bboxes from the computed coordinates ([x_min, y_min, x_max, y_max]) multiplying them by the desired image shape
x_coord_min_resized_shape = x_min * reshaped_image_width\ny_coord_min_resized_shape = y_min * reshaped_image_height\nx_coord_max_resized_shape = x_max * reshaped_image_width\ny_coord_max_resized_shape = y_max * reshaped_image_height\nthey are stored in absolute. So I guess it is no more than that, thanks for the reply. As a matter of fact, the author stored the DICOM images sizes in test_meta.csv
", "rawMarkdown": "they are stored in absolute. So I guess it is no more than that, thanks for the reply. As a matter of fact, the author stored the DICOM images sizes in test_meta.csv", "votes": 1 }, { "id": 1143444, "postDate": "2021-01-07T22:52:57.303Z", "content": "Thank you for sharing the preprocessing code!
\nI could find train_meta.csv, but I could not find test_meta.csv. How we can inference original coordinates?
Hi @corochann : I was wondering the same thing. Did you get an answer?
", "rawMarkdown": "Hi @corochann : I was wondering the same thing. Did you get an answer?", "votes": 1 }, { "id": 1195968, "postDate": "2021-02-11T07:10:39.943Z", "content": "I published the kernel & dataset :)
\nThank you:)
", "rawMarkdown": "Thank you:)", "votes": 1 } ] }, { "id": 1521085, "postDate": "2021-09-22T21:26:59.977Z", "content": "Hi @xhlulu
\nI am trying to train DETR on your 1024x1024 dataset ( https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-1024x1024 )
Its working fine for 512x512 but I am getting below error in case of 1024x1024:
\nValueError: Expected x_max for bbox (0.5952381060633343, 0.578276682732394, 1.0590986667957623, 0.7091712844849098, 3) to be in the range [0.0, 1.0], got 1.0590986667957623.
\nMoreover i am scaling the bounding boxes according to the new resolution.
\nAny idea what i might be missing?
\nThank you.
Well, I am not sure but you might be using the wrong Bounding box annotations. They must be under 1.
", "rawMarkdown": "Well, I am not sure but you might be using the wrong Bounding box annotations. They must be under 1. " }, { "id": 1524238, "postDate": "2021-09-26T09:34:48.277Z", "content": "Thank you @akanshmaurya for the response. You are right, the bbox values should be between 0 and 1.
", "rawMarkdown": "Thank you @akanshmaurya for the response. You are right, the bbox values should be between 0 and 1. " } ] }, { "id": 1241644, "postDate": "2021-03-17T06:17:13.650Z", "content": "Many thanks! It will be very helpful in learning some more about image processing <3
", "rawMarkdown": "Many thanks! It will be very helpful in learning some more about image processing <3" }, { "id": 1134755, "postDate": "2021-01-01T14:52:32.500Z", "content": "Thank you for sharing with us this valuable notebook.
", "rawMarkdown": "Thank you for sharing with us this valuable notebook." }, { "id": 1250601, "postDate": "2021-03-24T06:33:58.463Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1216611, "postDate": "2021-02-24T11:13:00.853Z", "content": "amazing, thank you :)
", "rawMarkdown": "amazing, thank you :)" } ], "comments": [ { "id": 1207560, "author_name": "corochann", "author_url": "", "post_date": "2021-02-17T23:19:23.700000", "content": "I uploaded original sized image
\nThanks for the datasets, they come in handy since I don't have enough disk space to process the original 200GB
\ndataset.
However, one thing I'd like to suggest: your dim0 and dim1 in metadata files are in unusual order - (height, width), when usually it's (width, height). So, dim0 should correspond to x-axis, and dim1 - to y-axis, but it's the other way around..
\nI've spent some time wondering why my YOLO model is discarding ~100 images as incorrectly labelled, and then WBF also mentioned that some coordinated were > 1.
", "votes": 1, "replies": [ { "id": 1242753, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-03-17T19:07:07.653000", "content": "Actually it makes sense when you think about it in numpy arrays.. But it is true, depending on the method, module, etc. the order of height/width changes which is confusing some times..
", "votes": 0, "replies": [] } ] }, { "id": 1142346, "author_name": "vuvko", "author_url": "", "post_date": "2021-01-07T10:27:23.310000", "content": "I've slightly changed your notebook for faster processing. Here is with maximum image size 1024 with original aspect ration. Version with 640px.
", "votes": 2, "replies": [] }, { "id": 1138006, "author_name": "hugoboum", "author_url": "", "post_date": "2021-01-04T10:43:31.373000", "content": "Thank you, upvoted ! I have a (stupid) question, as a beginner in detection. How do I use the original bounding box coordinates with these images ? The original images have different sizes right ? So you would have to get the scaling for each original image to 512*512 to apply it to the boxes coords ?
", "votes": 2, "replies": [ { "id": 1138189, "author_name": "LAZCoder", "author_url": "", "post_date": "2021-01-04T13:32:00.980000", "content": "Hi, I don't know how images are preprocessed in this particular dataset, but usually in object detection tasks you store coordinates of bounding boxes as numbers within the interval [0, 1] dividing bounding boxes coordinates by image shapes:
x_min = x_coord_min_original_shape / original_image_width\ny_min = y_coord_min_original_shape / original_image_height\nx_max = x_coord_max_original_shape / original_image_width\ny_max = y_coord_max_original_shape / original_image_height\nThus you can compute rescaled bboxes from the computed coordinates ([x_min, y_min, x_max, y_max]) multiplying them by the desired image shape
x_coord_min_resized_shape = x_min * reshaped_image_width\ny_coord_min_resized_shape = y_min * reshaped_image_height\nx_coord_max_resized_shape = x_max * reshaped_image_width\ny_coord_max_resized_shape = y_max * reshaped_image_height\nthey are stored in absolute. So I guess it is no more than that, thanks for the reply. As a matter of fact, the author stored the DICOM images sizes in test_meta.csv
", "votes": 1, "replies": [] }, { "id": 1143444, "author_name": "corochann", "author_url": "", "post_date": "2021-01-07T22:52:57.303000", "content": "Thank you for sharing the preprocessing code!
\nI could find train_meta.csv, but I could not find test_meta.csv. How we can inference original coordinates?
Hi @corochann : I was wondering the same thing. Did you get an answer?
", "votes": 1, "replies": [] }, { "id": 1195968, "author_name": "corochann", "author_url": "", "post_date": "2021-02-11T07:10:39.943000", "content": "I published the kernel & dataset :)
\nThank you:)
", "votes": 1, "replies": [] } ] }, { "id": 1521085, "author_name": "Yasir Irfan", "author_url": "", "post_date": "2021-09-22T21:26:59.977000", "content": "Hi @xhlulu
\nI am trying to train DETR on your 1024x1024 dataset ( https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-1024x1024 )
Its working fine for 512x512 but I am getting below error in case of 1024x1024:
\nValueError: Expected x_max for bbox (0.5952381060633343, 0.578276682732394, 1.0590986667957623, 0.7091712844849098, 3) to be in the range [0.0, 1.0], got 1.0590986667957623.
\nMoreover i am scaling the bounding boxes according to the new resolution.
\nAny idea what i might be missing?
\nThank you.
Well, I am not sure but you might be using the wrong Bounding box annotations. They must be under 1.
", "votes": 0, "replies": [] }, { "id": 1524238, "author_name": "Yasir Irfan", "author_url": "", "post_date": "2021-09-26T09:34:48.277000", "content": "Thank you @akanshmaurya for the response. You are right, the bbox values should be between 0 and 1.
", "votes": 0, "replies": [] } ] }, { "id": 1241644, "author_name": "Fernandes", "author_url": "", "post_date": "2021-03-17T06:17:13.650000", "content": "Many thanks! It will be very helpful in learning some more about image processing <3
", "votes": 0, "replies": [] }, { "id": 1134755, "author_name": "Corlulu", "author_url": "", "post_date": "2021-01-01T14:52:32.500000", "content": "Thank you for sharing with us this valuable notebook.
", "votes": 0, "replies": [] }, { "id": 1250601, "author_name": "", "author_url": "", "post_date": "2021-03-24T06:33:58.463000", "content": "", "votes": 0, "replies": [] }, { "id": 1216611, "author_name": "domiziano.stingi95", "author_url": "", "post_date": "2021-02-24T11:13:00.853000", "content": "amazing, thank you :)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1134324": "I've published various datasets in different format to help you get started:\n\n* [PNG 512px with original aspect ratio](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-png-512px-original-ratio) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-original-ratio)): Lossless and larger, the images will retain their aspect ratios and the largest side will have a size of 512px (so a 1000x500 image will be downsized to 512x256). To use this in a model, you will need to add padding (e.g. [using numpy](https://numpy.org/doc/stable/reference/generated/numpy.pad.html)) to the smaller sides (so 512x256 would be padded to 512x512).\n* [PNG 256x256](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-256x256) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-png-256x256)): Lossless and larger, each image is guaranteed to be 256x256 but will lose their aspect ratios.\n* [PNG 512x512](https://www.kaggle.com/xhlulu/vinbigdata) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-image)): Lossless and larger, each image is guaranteed to be 512x512 but will lose their aspect ratios.\n* [PNG 1024x1024](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-1024x1024) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-png-1024x1024)): Lossless and larger, each image is guaranteed to be 1024x1024 but will lose their aspect ratios.\n* [JPG 512x512](https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-jpg-512x512) ([notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-jpg)): Lossy but smaller files, each image is guaranteed to be 512x512 but will lose their aspect ratios.\n\nThis is a work in progress; I create new notebooks to also generate 256x256 and 1024x1024 images, but this will take time so keep an eye on this thread.\n\n---\n\n**Original post**: I published [this notebook](https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-image) showing how to process and resize the training data. You can directly use the notebook as an input to your model, or you can click on the \"+ New Dataset\" button in the output section to create a dataset that you can use.\n\nI'm waiting for an answer from the organizer in order to create a public dataset that you can use directly without creating your own; I will update this post if I get a positive answer.", "1207560": "I uploaded original sized image\n - [vinbigdata-chest-xray-original-png](https://www.kaggle.com/corochann/vinbigdata-chest-xray-original-png) ([notebook](https://www.kaggle.com/corochann/preprocessing-image-original-size-lossless-png) on kaggle fails due to disk limit)", "1242703": "Thanks for the datasets, they come in handy since I don't have enough disk space to process the original 200GB \ndataset.\n\nHowever, one thing I'd like to suggest: your dim0 and dim1 in metadata files are in unusual order - (height, width), when usually it's (width, height). So, dim0 should correspond to x-axis, and dim1 - to y-axis, but it's the other way around..\n\nI've spent some time wondering why my YOLO model is discarding ~100 images as incorrectly labelled, and then WBF also mentioned that some coordinated were > 1.", "1142346": "I've slightly changed your notebook for faster processing. [Here](https://www.kaggle.com/notvuvko/vinbigdata-process-and-resize-original-ratio?scriptVersionId=51233352) is with maximum image size 1024 with original aspect ration. [Version](https://www.kaggle.com/notvuvko/vinbigdata-process-and-resize-original-ratio?scriptVersionId=51254264) with 640px.", "1138006": "Thank you, upvoted ! I have a (stupid) question, as a beginner in detection. How do I use the original bounding box coordinates with these images ? The original images have different sizes right ? So you would have to get the scaling for each original image to 512*512 to apply it to the boxes coords ?", "1521085": "Hi @xhlulu \nI am trying to train DETR on your 1024x1024 dataset ( https://www.kaggle.com/xhlulu/vinbigdata-chest-xray-resized-png-1024x1024 )\n\nIts working fine for 512x512 but I am getting below error in case of 1024x1024:\n\n**ValueError: Expected x_max for bbox (0.5952381060633343, 0.578276682732394, 1.0590986667957623, 0.7091712844849098, 3) to be in the range [0.0, 1.0], got 1.0590986667957623.**\n\nMoreover i am scaling the bounding boxes according to the new resolution.\n\nAny idea what i might be missing?\nThank you.", "1241644": "Many thanks! It will be very helpful in learning some more about image processing <3", "1134755": "Thank you for sharing with us this valuable notebook.", "1250601": "", "1216611": "amazing, thank you :)" } }