![\"Dataset\"](\"https://i.ibb.co/zPBHmT6/br002.jpg\")
I created ROI (Region Of Interest) extracted dataset for this compettion. Hope this improve score a lot.
\nSolution:
\n
Dataset is available here: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
", "messages": [ { "id": 2051239, "postDate": "2022-12-01T09:44:06.773Z", "content": "I created ROI (Region Of Interest) extracted dataset for this compettion. Hope this improve score a lot.
\nSolution:
\n
Dataset is available here: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
", "rawMarkdown": "I created ROI (Region Of Interest) extracted dataset for this compettion. Hope this improve score a lot.\n\nSolution:\n- ROI extractor -> https://www.kaggle.com/code/remekkinas/breast-cancer-roi-brest-extractor\n- ROI is resized to SIZE (I resized only taking into account longer dimension) so no distorision -> you can decide in augumentation step what to do with extracted ROI \n\n\n\nDataset is available here: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images", "votes": 70 }, { "id": 2051735, "postDate": "2022-12-01T15:41:54.020Z", "content": "I can confirm your preprocessing gives a 0.02 CV boost.
", "rawMarkdown": "I can confirm your preprocessing gives a 0.02 CV boost. ", "votes": 3, "replies": [ { "id": 2057744, "postDate": "2022-12-07T10:16:06.470Z", "content": "ROI extracted images boost score. Due to my experiments if we leave crops not distorted by resizing it helps a lot.
", "rawMarkdown": "ROI extracted images boost score. Due to my experiments if we leave crops not distorted by resizing it helps a lot." } ] }, { "id": 2056932, "postDate": "2022-12-06T15:31:00.467Z", "content": "Thank you @remekkinas for sharing!
\nCan you just clarify your pipeline for me please:
\nROI extracted DS
\ndef image_resize(image, width = None, height = None, inter = cv2.INTER_LINEAR):\n\n dim = None\n (h, w) = image.shape[:2]\n\n if width is None and height is None:\n return image\n\n if width is None:\n r = height / float(h)\n dim = (int(w * r), height)\n else:\n r = width / float(w)\n dim = (width, int(h * r))\n resized = cv2.resize(image, dim, interpolation = inter)\n\n return resized\n\nh, w, _ = tmp_im.shape\n\nif w > h:\n img = image_resize(tmp_im, width = size)\n else:\n img = image_resize(tmp_im, height = size)\nToday I updated dataset -> resolution 1280px (longer side of image) and all imagess were processed using windowing (dicomsdl). I checked them in my notebook and works better for training (maybe due to consistent file processing). Now they look better as well : https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
\n![\"\"](\"https://i.ibb.co/2qpHw62/lut.jpg\")
Could you also please share Kaggle notebook where you have windowing is applied on the dicom files
", "rawMarkdown": "Could you also please share Kaggle notebook where you have windowing is applied on the dicom files", "replies": [ { "id": 2108160, "postDate": "2023-01-20T09:52:36.723Z", "content": "Hi,
\nI described it in Dataset discussion part: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images/discussion/378201
Now dataset is updated. I cropped oryginal DS to 1024pix. I can see a lot of improvement using higher cropped resolution. Have a nice Kaggling with this dataset.
\nFor crop I used yolov5 and nano model. Source:
\nThis is ROI extraction test report I made today:
\n![\"\"](\"https://i.ibb.co/wdBy2vk/001.jpg\")
@radek1 now you can see that speed is ok for yolo.
", "rawMarkdown": "This is ROI extraction test report I made today: \n\n\n@radek1 now you can see that speed is ok for yolo.", "votes": 2, "replies": [ { "id": 2061270, "postDate": "2022-12-10T22:26:15.793Z", "content": "Yes, thank you! 🙂 This is blazingly fast! 🔥
", "rawMarkdown": "Yes, thank you! 🙂 This is blazingly fast! 🔥" } ] }, { "id": 2051288, "postDate": "2022-12-01T10:20:55.833Z", "content": "Cool dataset !
\nWatch out for license issues with yolov5 though =)
", "rawMarkdown": "Cool dataset !\n\nWatch out for license issues with yolov5 though =)", "votes": 2, "replies": [ { "id": 2051303, "postDate": "2022-12-01T10:30:54.087Z", "content": "Hi @theoviel …. theank you for your comment. Any issue with yolov5? I know past competition where it was banned. What about this one?
\nWe can convert idea (having labels) to anything else :) (yolov7 e.g. or faster yolov6). No problem.
I have no idea, we have to ask the kaggle team or the hosts, and neither are usually not giving explicit answers.
\nAnyways, I will experiment with your dataset asap.
", "rawMarkdown": "I have no idea, we have to ask the kaggle team or the hosts, and neither are usually not giving explicit answers.\n\nAnyways, I will experiment with your dataset asap." }, { "id": 2057886, "postDate": "2022-12-07T12:23:08.887Z", "content": "Yolo is always tricky. It all depends if copy-left license is OK, only Kaggle/hosts can tell.
\nPersonally I would use other object detection models - there are enough good ones with better licenses.
", "rawMarkdown": "Yolo is always tricky. It all depends if copy-left license is OK, only Kaggle/hosts can tell.\n\nPersonally I would use other object detection models - there are enough good ones with better licenses.", "votes": 3 } ] }, { "id": 2145560, "postDate": "2023-02-15T07:29:57.877Z", "content": "Dear @remekkinas. I'm training my own experiment and tuning hyperparameter but my validation could not improve.Can you give me some advices to improve my score. Thank you very much.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7879577%2F1aaabac308bf66765a023417ca07d5d7%2FScreenshot%202023-02-15%20142601.png?generation=1676445999616375&alt=media\")
Hello, guys can somebody answer me this https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/386518
", "rawMarkdown": "Hello, guys can somebody answer me this https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/386518" }, { "id": 2112999, "postDate": "2023-01-24T02:33:24.570Z", "content": "Here is a dumb question…
\nI just wonder if my input ROI images are l large like 2048*2048. When inferring could I store it locally in workworking/tem/imgfolder? I remember there is a max storage limitation.
\n1536*960 0.5MB for each sample.
", "rawMarkdown": "Here is a dumb question...\n\nI just wonder if my input ROI images are l large like 2048*2048. When inferring could I store it locally in workworking/tem/imgfolder? I remember there is a max storage limitation. \n\n1536*960 0.5MB for each sample.", "replies": [ { "id": 2113051, "postDate": "2023-01-24T03:53:04.090Z", "content": "os.remove(f'{image_id}.png')
\nOkay, I got it.
", "rawMarkdown": "os.remove(f'{image_id}.png')\n\nOkay, I got it. " } ] }, { "id": 2080969, "postDate": "2022-12-30T16:58:20.667Z", "content": "@remekkinas I've tested your YoloV5 model, it looks quite good. About Parallel(n_jobs=2) execution, you might experiment the following issue that happens a bit randomly (it happened to me): https://github.com/ultralytics/yolov5/issues/2824 that prevents ROI detection for a small number of images. It could be a problem during inference that you won't see if you try/catch the ROI model execution that could affect the final score. I'm testing a fix currently …
The error I got for a few images:
\n\n", "rawMarkdown": "@remekkinas I've tested your YoloV5 model, it looks quite good. About `Parallel(n_jobs=2)` execution, you might experiment the following issue that happens a bit randomly (it happened to me): https://github.com/ultralytics/yolov5/issues/2824 that prevents ROI detection for a small number of images. It could be a problem during inference that you won't see if you `try/catch` the ROI model execution that could affect the final score. I'm testing a fix currently ...\n\nThe error I got for a few images:\n>The size of tensor a (104) must match the size of tensor b (112) at non-singleton dimension 3", "replies": [ { "id": 2080983, "postDate": "2022-12-30T17:28:31.027Z", "content": "The size of tensor a (104) must match the size of tensor b (112) at non-singleton dimension 3
\n
A possible idea to try is batch processing, it can probably decrease time since batch processing seems to be faster (see the table below from the yolov5 repo). I think since only one model will be loaded and the CPU threads would process the data, it would alleviate this problem.
\n![\"Yolov5](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5918909%2F547bab5861cee666ce1ed9da7f1f9cf8%2FCapture.PNG?generation=1672421053793971&alt=media\")
(for J2K) Maybe with DALI we can load batch on GPU, preprocess, get ROI, crop and save immediately. I am not sure if anyone has done this before but it seems like it is worth a shot. I'll look into it after my experiments.
", "rawMarkdown": "A possible idea to try is batch processing, it can probably decrease time since batch processing seems to be faster (see the table below from the yolov5 repo). I think since only one model will be loaded and the CPU threads would process the data, it would alleviate this problem.\n\n\n\n(for J2K) Maybe with DALI we can load batch on GPU, preprocess, get ROI, crop and save immediately. I am not sure if anyone has done this before but it seems like it is worth a shot. I'll look into it after my experiments.\n", "votes": 1 }, { "id": 2080996, "postDate": "2022-12-30T17:55:14.497Z", "content": "Thank you for information. I will check my notebook. Let me know when you finish testing.
\nI am sure yolo cropped images improve local score.
", "rawMarkdown": "Thank you for information. I will check my notebook. Let me know when you finish testing.\n\nI am sure yolo cropped images improve local score. ", "votes": 1, "replies": [ { "id": 2081850, "postDate": "2022-12-31T18:14:54.300Z", "content": "The fix is explained here: https://github.com/ultralytics/yolov5/commit/03fae98d86394fe02804d7e9bc20013311bf5837
\nHowever, it's for an old version. For current YoloV5 version, it's quite similar, initialize self.grid and self.anchor_grid in forward() method like:
def __init__(self, nc=80, anchors=(), ch=(), inplace=True): # detection layer\n super().__init__()\n self.nc = nc # number of classes\n self.no = nc + 5 # number of outputs per anchor\n self.nl = len(anchors) # number of detection layers\n self.na = len(anchors[0]) // 2 # number of anchors\n # self.grid = [torch.empty(0) for _ in range(self.nl)] # init grid\n # self.anchor_grid = [torch.empty(0) for _ in range(self.nl)] # init anchor grid\n self.register_buffer('anchors', torch.tensor(anchors).float().view(self.nl, -1, 2)) # shape(nl,na,2)\n self.m = nn.ModuleList(nn.Conv2d(x, self.no * self.na, 1) for x in ch) # output conv\n self.inplace = inplace # use inplace ops (e.g. slice assignment) \n\n\ndef forward(self, x):\n z = [] # inference output\n grid = [torch.empty(0) for _ in range(self.nl)] # init grid\n anchor_grid = [torch.empty(0) for _ in range(self.nl)] # init anchor grid \n for i in range(self.nl):\n x[i] = self.m[i](x[i]) # conv\n bs, _, ny, nx = x[i].shape # x(bs,255,20,20) to x(bs,3,20,20,85)\n x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()\n\n if not self.training: # inference\n # if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:\n # self.grid[i], self.anchor_grid[i] = self._make_grid(nx, ny, i)\n grid[i], anchor_grid[i] = self._make_grid(nx, ny, i)\n .... \nIt worked for me.
\nFor now with ROI 1024 I'm stuck for single fold CV=0.44, LB=0.41
\nUpdate: LB=0.45
Yes, I checked it as well. I subbed without Paraller processing - score is the same. Thank you for checking and trying to help.
\nI have almost the same score for single fold:
\nWhat do you mean by CV? Is this f1prob?
", "rawMarkdown": "Yes, I checked it as well. I subbed without Paraller processing - score is the same. Thank you for checking and trying to help.\n\nI have almost the same score for single fold:\n- img size 1024\n- model - effnet_b2\n\nWhat do you mean by CV? Is this f1prob?\n", "votes": 1 }, { "id": 2081965, "postDate": "2022-12-31T22:13:54.653Z", "content": "Yes, CV is based on valid_pfbeta_agg_opt score
\n@remekkinas Are you able to extract ROI if image is already in GPU?
\n\nresults = model_roi((image*).astype(np.uint8), size=)\nrow_pd = results.pandas().xyxy[]\n\nresults = model_roi(torch_image, size=)\nrow_pd = results.pandas().xyxy[]\nI would to avoid the round trip GPU => CPU => GPU. It looks that YoloV5 have a \"letterbox\" function to pad numpy image correctly before passing to GPU but nothing if image is already in GPU.
", "rawMarkdown": "@remekkinas Are you able to extract ROI if image is already in GPU?\n\n```python\n# Numpy image on CPU\nresults = model_roi((image*255).astype(np.uint8), size=1024)\nrow_pd = results.pandas().xyxy[0]\n```\n\n```python\n# Torch image on GPU\nresults = model_roi(torch_image, size=1024)\nrow_pd = results.pandas().xyxy[0]\n```\n\nI would to avoid the round trip GPU => CPU => GPU. It looks that YoloV5 have a \"letterbox\" function to pad numpy image correctly before passing to GPU but nothing if image is already in GPU." }, { "id": 2083846, "postDate": "2023-01-02T23:21:49.747Z", "content": "@mpware Yolov5 returns different results if the input is a tensor vs numpy/list. Here is the code from the Yolov5 repo. It just results raw result so we need to do some post-processing:
\n\nIf the input is a numpy array, it would take of the preprocessing, so I recommend preprocessing and scaling the input tensor 0-1. The returned tensor is the anchors so you need just to run the postprocessing step.
\nHere is how I did it with another postprocessing step to keeping highest conf bboxes:
\n![\"Yolov5](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5918909%2F7d873f9ef4e3ee1515e4cb84210f61e7%2Fyolov566.PNG?generation=1672701128089222&alt=media\")
You can import it via
\n![\"Yolov5](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5918909%2F6481f0c7437f56f67c9f7a28184a050c%2Fcode.png?generation=1672701242401995&alt=media\")
sys.path.append()\n\n utils.general non_max_suppression\nEDIT: I also got batch preprocessing to work with yolov5 but it very specific to my model which also extracts if the breast is on the left or right side of the image with almost perfect accuracy (this is very helpful, and i will release once I get a E2E solution)
", "rawMarkdown": "@mpware Yolov5 returns different results if the input is a tensor vs numpy/list. Here is the code from the Yolov5 repo. It just results raw result so we need to do some post-processing:\n\n\nSee https://github.com/ultralytics/yolov5/blob/632bf485b4ab2adbaef71f4eced5e6b59ecef7e2/models/common.py#L658\n\nIf the input is a numpy array, it would take of the preprocessing, so I recommend preprocessing and scaling the input tensor 0-1. The returned tensor is the anchors so you need just to run the postprocessing step.\n\nHere is how I did it with another postprocessing step to keeping highest conf bboxes:\n\n\n\nYou can import it via\n\n```python\nsys.path.append('/kaggle/input/yolov5-github-repo-tracker/yolov5')\n\nfrom utils.general import non_max_suppression\n```\n\nEDIT: I also got batch preprocessing to work with yolov5 but it very specific to my model which also extracts if the breast is on the left or right side of the image with almost perfect accuracy (this is very helpful, and i will release once I get a E2E solution)", "votes": 1 }, { "id": 2085321, "postDate": "2023-01-04T04:20:52.230Z", "content": "@remekkinas @mpware As promised,
\nhttps://www.kaggle.com/code/outwrest/yolov5-roi-batch-dali-preprocessing-pipeline
\nI am unsure about the speedup compared without DALI batch pipeline but I believe it is about ~30 minutes slower than Theo's no yolov5 notebook. This means it is probably worse than without a batch pipeline but it is still a good start to what can be done in the future. There are many points where I think there is a bottleneck and I can try to improve. I will keep the weights private (didn't expect it to do well, CV was ~0.3) for now but to use it, one would need to retrain yolov5 to detect the breasts in padded images.
\nHopefully, this helps!
", "rawMarkdown": "@remekkinas @mpware As promised, \n\nhttps://www.kaggle.com/code/outwrest/yolov5-roi-batch-dali-preprocessing-pipeline\n\nI am unsure about the speedup compared without DALI batch pipeline but I believe it is about ~30 minutes slower than [Theo's no yolov5 notebook](https://www.kaggle.com/code/theoviel/rsna-breast-baseline-faster-inference-with-dali). This means it is probably worse than without a batch pipeline but it is still a good start to what can be done in the future. There are many points where I think there is a bottleneck and I can try to improve. I will keep the weights private (didn't expect it to do well, CV was ~0.3) for now but to use it, one would need to retrain yolov5 to detect the breasts in padded images.\n\nHopefully, this helps!", "votes": 2 }, { "id": 2085411, "postDate": "2023-01-04T06:40:08.113Z", "content": "Great! Great! Thank you very much. I will use your ideas in my notebook. 👍
\nI am wondering if you get some score improvement with ROI images. Do you?
", "rawMarkdown": "Great! Great! Thank you very much. I will use your ideas in my notebook. 👍\n\nI am wondering if you get some score improvement with ROI images. Do you?", "votes": 2 }, { "id": 2085452, "postDate": "2023-01-04T07:05:07.033Z", "content": "CV, yes but I am still having trouble keeping a good local setup. I see a lot of variations between folds (0.08+-) and I was even surprised by my LB score. So I am not 100% sure. I haven't submitted any without ROI, this notebook has been my own submission to LB (with some failures 😆).
\nAre your score improvements with ROI big? I've only been experimenting with tf_efficientnet_b0_ns and I only saw major improvements with dataset changes (ROI & other preprocessing), and image size. (the 4fold model I submitted was also an efficientnetb0)
I have the same problem - local validation. This is why I do not submit final solution so far (I have solution which uses only one model). Only just checking on LB different setups and looking for improvements. So far I managed to get 0.39 based on one model only. This is actually not good score.
\nI can see difference in score locally - with and without ROI. ROI is better (AUC_ROC score - I do not use probf1 score for validation).
", "rawMarkdown": "I have the same problem - local validation. This is why I do not submit final solution so far (I have solution which uses only one model). Only just checking on LB different setups and looking for improvements. So far I managed to get 0.39 based on one model only. This is actually not good score.\n\nI can see difference in score locally - with and without ROI. ROI is better (AUC_ROC score - I do not use probf1 score for validation).", "votes": 2 }, { "id": 2085494, "postDate": "2023-01-04T07:40:02.800Z", "content": "That is pretty smart, I feel like I am solely focusing on pF1, but your approach is right. I have doubts about the metric, it is noisy and seems like it is not well-suited for this type of competition (maybe it just needs a bit of luck, or we don't even have enough data).
\nI think the key (as @hengck23 mentions) is to work on different architectures and put them together. I have some ideas to experiment with other datasets but only time will tell.
\nLet me run 4k fold and get back to you on metric changes with/without ROI in my setup
", "rawMarkdown": "That is pretty smart, I feel like I am solely focusing on pF1, but your approach is right. I have doubts about the metric, it is noisy and seems like it is [not well-suited for this type of competition](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/372175#2080720) (maybe it just needs a bit of luck, or [we don't even have enough data](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333#2083921)). \n\nI think the key (as @hengck23 mentions) is to work on different architectures and put them together. I have some ideas to experiment with other datasets but only time will tell.\n\nLet me run 4k fold and get back to you on metric changes with/without ROI in my setup", "votes": 1 }, { "id": 2085509, "postDate": "2023-01-04T07:46:37.303Z", "content": "Thanks for it. I'm also working to improve inference time.
\nUpdate: All CPU/GPU round trips removed, no image saved on disk, ROI (1024) + inference of a single model with 4 folds is around 4h30min now, single loop (no Parallel usage)
I use AUC_ROC and MCC metrics (matthew correlation coefficient). pf1 score I plot as well but only to monitor it.
", "rawMarkdown": "I use AUC_ROC and MCC metrics (matthew correlation coefficient). pf1 score I plot as well but only to monitor it.", "votes": 1 } ] } ] }, { "id": 2074309, "postDate": "2022-12-24T01:20:47.677Z", "content": "Great dataset!
\nIf I understand correctly, you keep the picture long side fixed for 1024, right?
\nAlso I have a question that, for submission, will this process be in 9 hours for 1024pix?
\nBecause we need to read from dcm, yolo, resize, ensemble… you said 7 hours for 768 but how about 1024?
\nThanks!
I tested 1600px submit (ROI extracted - yolo nano (model 001 from my public available dataset) on 3 models (efficientnetv2_s) and each has TTA (only flip). It takes 7h.
", "rawMarkdown": "I tested 1600px submit (ROI extracted - yolo nano (model 001 from my public available dataset) on 3 models (efficientnetv2_s) and each has TTA (only flip). It takes 7h. \n\n" } ] }, { "id": 2061102, "postDate": "2022-12-10T17:30:52.917Z", "content": "I checked speed of pipeline:
\n7h submission time on 1xKaggle GPU
", "rawMarkdown": "I checked speed of pipeline:\n\n1. dicom convert to jpg using dicomsdl\n2. extract roi using yolov5 (model small / nano can be even faster)\n3. convert to 512x512\n4. blend - 3 models (resnet50d) with TTA (standard inference + flip)\n\n7h submission time on 1xKaggle GPU\n" }, { "id": 2051359, "postDate": "2022-12-01T11:15:09.027Z", "content": "I'm getting this error :/
\n\n", "rawMarkdown": "I'm getting this error :/\n> Data not available. This dataset may need it's archive recreated. Contact dataset author.", "replies": [ { "id": 2051373, "postDate": "2022-12-01T11:22:34.073Z", "content": "Data not available. This dataset may need it's archive recreated. Contact dataset author.
\n
@theoviel it is ok -> you have to download it here
\n![\"\"](\"https://i.ibb.co/hMQgqxp/004.jpg\")
I use the kaggle dataset API, but it's working now. Thanks anyways !
", "rawMarkdown": "I use the kaggle dataset API, but it's working now. Thanks anyways !", "votes": 1 } ] }, { "id": 2051349, "postDate": "2022-12-01T11:04:31.443Z", "content": "Dataset is available for 768 resolution. It means that longer dimension is 768pix.
\nThis is baseline dataset. Now you can create 512, 384, 256 … just using cv2.resize or other methods.
https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
", "rawMarkdown": "Dataset is available for 768 resolution. It means that longer dimension is 768pix. \nThis is baseline dataset. Now you can create 512, 384, 256 ... just using cv2.resize or other methods.\n\nhttps://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images " }, { "id": 2051249, "postDate": "2022-12-01T09:49:11.993Z", "content": "This is awesome work! 🙌 Well be a bit hard to run it on test though… 🤔 But if someone will figure out how to do this, that can give a major LB boost…
\nSo many interesting avenues to explore in this competition, so little time in a day!
", "rawMarkdown": "This is awesome work! 🙌 Well be a bit hard to run it on test though... 🤔 But if someone will figure out how to do this, that can give a major LB boost...\n\nSo many interesting avenues to explore in this competition, so little time in a day! ", "replies": [ { "id": 2051260, "postDate": "2022-12-01T09:59:11.207Z", "content": "Not exactly. I am sure it will work without any problem on test. I checked it on my notebook and the most consuming time part is convert dcm file to png file (you have to do anyway). Inference is a tiny amout of processing time (miliseconds).
\nYou are sure. This is absolutely fantastic competition. Not only because of ML and computer vision (my favourite task) but … idea.
", "rawMarkdown": "Not exactly. I am sure it will work without any problem on test. I checked it on my notebook and the most consuming time part is convert dcm file to png file (you have to do anyway). Inference is a tiny amout of processing time (miliseconds). \n\nYou are sure. This is absolutely fantastic competition. Not only because of ML and computer vision (my favourite task) but ... idea. ", "votes": 2 } ] } ], "comments": [ { "id": 2051735, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-12-01T15:41:54.020000", "content": "I can confirm your preprocessing gives a 0.02 CV boost.
", "votes": 3, "replies": [ { "id": 2057744, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-07T10:16:06.470000", "content": "ROI extracted images boost score. Due to my experiments if we leave crops not distorted by resizing it helps a lot.
", "votes": 0, "replies": [] } ] }, { "id": 2056932, "author_name": "Optimo", "author_url": "", "post_date": "2022-12-06T15:31:00.467000", "content": "Thank you @remekkinas for sharing!
\nCan you just clarify your pipeline for me please:
\nROI extracted DS
\ndef image_resize(image, width = None, height = None, inter = cv2.INTER_LINEAR):\n\n dim = None\n (h, w) = image.shape[:2]\n\n if width is None and height is None:\n return image\n\n if width is None:\n r = height / float(h)\n dim = (int(w * r), height)\n else:\n r = width / float(w)\n dim = (width, int(h * r))\n resized = cv2.resize(image, dim, interpolation = inter)\n\n return resized\n\nh, w, _ = tmp_im.shape\n\nif w > h:\n img = image_resize(tmp_im, width = size)\n else:\n img = image_resize(tmp_im, height = size)\nToday I updated dataset -> resolution 1280px (longer side of image) and all imagess were processed using windowing (dicomsdl). I checked them in my notebook and works better for training (maybe due to consistent file processing). Now they look better as well : https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
\n
Could you also please share Kaggle notebook where you have windowing is applied on the dicom files
", "votes": 0, "replies": [ { "id": 2108160, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-20T09:52:36.723000", "content": "Hi,
\nI described it in Dataset discussion part: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images/discussion/378201
Now dataset is updated. I cropped oryginal DS to 1024pix. I can see a lot of improvement using higher cropped resolution. Have a nice Kaggling with this dataset.
\nFor crop I used yolov5 and nano model. Source:
\nThis is ROI extraction test report I made today:
\n
@radek1 now you can see that speed is ok for yolo.
", "votes": 2, "replies": [ { "id": 2061270, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-10T22:26:15.793000", "content": "Yes, thank you! 🙂 This is blazingly fast! 🔥
", "votes": 0, "replies": [] } ] }, { "id": 2051288, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-12-01T10:20:55.833000", "content": "Cool dataset !
\nWatch out for license issues with yolov5 though =)
", "votes": 2, "replies": [ { "id": 2051303, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-01T10:30:54.087000", "content": "Hi @theoviel …. theank you for your comment. Any issue with yolov5? I know past competition where it was banned. What about this one?
\nWe can convert idea (having labels) to anything else :) (yolov7 e.g. or faster yolov6). No problem.
I have no idea, we have to ask the kaggle team or the hosts, and neither are usually not giving explicit answers.
\nAnyways, I will experiment with your dataset asap.
", "votes": 0, "replies": [] }, { "id": 2057886, "author_name": "Psi", "author_url": "", "post_date": "2022-12-07T12:23:08.887000", "content": "Yolo is always tricky. It all depends if copy-left license is OK, only Kaggle/hosts can tell.
\nPersonally I would use other object detection models - there are enough good ones with better licenses.
", "votes": 3, "replies": [] } ] }, { "id": 2145560, "author_name": "Bá Huy Nguyễn", "author_url": "", "post_date": "2023-02-15T07:29:57.877000", "content": "Dear @remekkinas. I'm training my own experiment and tuning hyperparameter but my validation could not improve.Can you give me some advices to improve my score. Thank you very much.
\n
Hello, guys can somebody answer me this https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/386518
", "votes": 0, "replies": [] }, { "id": 2112999, "author_name": "ChenxiangSun@NJU", "author_url": "", "post_date": "2023-01-24T02:33:24.570000", "content": "Here is a dumb question…
\nI just wonder if my input ROI images are l large like 2048*2048. When inferring could I store it locally in workworking/tem/imgfolder? I remember there is a max storage limitation.
\n1536*960 0.5MB for each sample.
", "votes": 0, "replies": [ { "id": 2113051, "author_name": "ChenxiangSun@NJU", "author_url": "", "post_date": "2023-01-24T03:53:04.090000", "content": "os.remove(f'{image_id}.png')
\nOkay, I got it.
", "votes": 0, "replies": [] } ] }, { "id": 2080969, "author_name": "MPWARE", "author_url": "", "post_date": "2022-12-30T16:58:20.667000", "content": "@remekkinas I've tested your YoloV5 model, it looks quite good. About Parallel(n_jobs=2) execution, you might experiment the following issue that happens a bit randomly (it happened to me): https://github.com/ultralytics/yolov5/issues/2824 that prevents ROI detection for a small number of images. It could be a problem during inference that you won't see if you try/catch the ROI model execution that could affect the final score. I'm testing a fix currently …
The error I got for a few images:
\n\n", "votes": 0, "replies": [ { "id": 2080983, "author_name": "outwrest", "author_url": "", "post_date": "2022-12-30T17:28:31.027000", "content": "The size of tensor a (104) must match the size of tensor b (112) at non-singleton dimension 3
\n
A possible idea to try is batch processing, it can probably decrease time since batch processing seems to be faster (see the table below from the yolov5 repo). I think since only one model will be loaded and the CPU threads would process the data, it would alleviate this problem.
\n
(for J2K) Maybe with DALI we can load batch on GPU, preprocess, get ROI, crop and save immediately. I am not sure if anyone has done this before but it seems like it is worth a shot. I'll look into it after my experiments.
", "votes": 1, "replies": [] }, { "id": 2080996, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-30T17:55:14.497000", "content": "Thank you for information. I will check my notebook. Let me know when you finish testing.
\nI am sure yolo cropped images improve local score.
", "votes": 1, "replies": [ { "id": 2081850, "author_name": "MPWARE", "author_url": "", "post_date": "2022-12-31T18:14:54.300000", "content": "The fix is explained here: https://github.com/ultralytics/yolov5/commit/03fae98d86394fe02804d7e9bc20013311bf5837
\nHowever, it's for an old version. For current YoloV5 version, it's quite similar, initialize self.grid and self.anchor_grid in forward() method like:
def __init__(self, nc=80, anchors=(), ch=(), inplace=True): # detection layer\n super().__init__()\n self.nc = nc # number of classes\n self.no = nc + 5 # number of outputs per anchor\n self.nl = len(anchors) # number of detection layers\n self.na = len(anchors[0]) // 2 # number of anchors\n # self.grid = [torch.empty(0) for _ in range(self.nl)] # init grid\n # self.anchor_grid = [torch.empty(0) for _ in range(self.nl)] # init anchor grid\n self.register_buffer('anchors', torch.tensor(anchors).float().view(self.nl, -1, 2)) # shape(nl,na,2)\n self.m = nn.ModuleList(nn.Conv2d(x, self.no * self.na, 1) for x in ch) # output conv\n self.inplace = inplace # use inplace ops (e.g. slice assignment) \n\n\ndef forward(self, x):\n z = [] # inference output\n grid = [torch.empty(0) for _ in range(self.nl)] # init grid\n anchor_grid = [torch.empty(0) for _ in range(self.nl)] # init anchor grid \n for i in range(self.nl):\n x[i] = self.m[i](x[i]) # conv\n bs, _, ny, nx = x[i].shape # x(bs,255,20,20) to x(bs,3,20,20,85)\n x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()\n\n if not self.training: # inference\n # if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:\n # self.grid[i], self.anchor_grid[i] = self._make_grid(nx, ny, i)\n grid[i], anchor_grid[i] = self._make_grid(nx, ny, i)\n .... \nIt worked for me.
\nFor now with ROI 1024 I'm stuck for single fold CV=0.44, LB=0.41
\nUpdate: LB=0.45
Yes, I checked it as well. I subbed without Paraller processing - score is the same. Thank you for checking and trying to help.
\nI have almost the same score for single fold:
\nWhat do you mean by CV? Is this f1prob?
", "votes": 1, "replies": [] }, { "id": 2081965, "author_name": "MPWARE", "author_url": "", "post_date": "2022-12-31T22:13:54.653000", "content": "Yes, CV is based on valid_pfbeta_agg_opt score
\n@remekkinas Are you able to extract ROI if image is already in GPU?
\n\nresults = model_roi((image*).astype(np.uint8), size=)\nrow_pd = results.pandas().xyxy[]\n\nresults = model_roi(torch_image, size=)\nrow_pd = results.pandas().xyxy[]\nI would to avoid the round trip GPU => CPU => GPU. It looks that YoloV5 have a \"letterbox\" function to pad numpy image correctly before passing to GPU but nothing if image is already in GPU.
", "votes": 0, "replies": [] }, { "id": 2083846, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-02T23:21:49.747000", "content": "@mpware Yolov5 returns different results if the input is a tensor vs numpy/list. Here is the code from the Yolov5 repo. It just results raw result so we need to do some post-processing:
\n\nIf the input is a numpy array, it would take of the preprocessing, so I recommend preprocessing and scaling the input tensor 0-1. The returned tensor is the anchors so you need just to run the postprocessing step.
\nHere is how I did it with another postprocessing step to keeping highest conf bboxes:
\n
You can import it via
\nsys.path.append()\n\n utils.general non_max_suppression\nEDIT: I also got batch preprocessing to work with yolov5 but it very specific to my model which also extracts if the breast is on the left or right side of the image with almost perfect accuracy (this is very helpful, and i will release once I get a E2E solution)
", "votes": 1, "replies": [] }, { "id": 2085321, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-04T04:20:52.230000", "content": "@remekkinas @mpware As promised,
\nhttps://www.kaggle.com/code/outwrest/yolov5-roi-batch-dali-preprocessing-pipeline
\nI am unsure about the speedup compared without DALI batch pipeline but I believe it is about ~30 minutes slower than Theo's no yolov5 notebook. This means it is probably worse than without a batch pipeline but it is still a good start to what can be done in the future. There are many points where I think there is a bottleneck and I can try to improve. I will keep the weights private (didn't expect it to do well, CV was ~0.3) for now but to use it, one would need to retrain yolov5 to detect the breasts in padded images.
\nHopefully, this helps!
", "votes": 2, "replies": [] }, { "id": 2085411, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-04T06:40:08.113000", "content": "Great! Great! Thank you very much. I will use your ideas in my notebook. 👍
\nI am wondering if you get some score improvement with ROI images. Do you?
", "votes": 2, "replies": [] }, { "id": 2085452, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-04T07:05:07.033000", "content": "CV, yes but I am still having trouble keeping a good local setup. I see a lot of variations between folds (0.08+-) and I was even surprised by my LB score. So I am not 100% sure. I haven't submitted any without ROI, this notebook has been my own submission to LB (with some failures 😆).
\nAre your score improvements with ROI big? I've only been experimenting with tf_efficientnet_b0_ns and I only saw major improvements with dataset changes (ROI & other preprocessing), and image size. (the 4fold model I submitted was also an efficientnetb0)
I have the same problem - local validation. This is why I do not submit final solution so far (I have solution which uses only one model). Only just checking on LB different setups and looking for improvements. So far I managed to get 0.39 based on one model only. This is actually not good score.
\nI can see difference in score locally - with and without ROI. ROI is better (AUC_ROC score - I do not use probf1 score for validation).
", "votes": 2, "replies": [] }, { "id": 2085494, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-04T07:40:02.800000", "content": "That is pretty smart, I feel like I am solely focusing on pF1, but your approach is right. I have doubts about the metric, it is noisy and seems like it is not well-suited for this type of competition (maybe it just needs a bit of luck, or we don't even have enough data).
\nI think the key (as @hengck23 mentions) is to work on different architectures and put them together. I have some ideas to experiment with other datasets but only time will tell.
\nLet me run 4k fold and get back to you on metric changes with/without ROI in my setup
", "votes": 1, "replies": [] }, { "id": 2085509, "author_name": "MPWARE", "author_url": "", "post_date": "2023-01-04T07:46:37.303000", "content": "Thanks for it. I'm also working to improve inference time.
\nUpdate: All CPU/GPU round trips removed, no image saved on disk, ROI (1024) + inference of a single model with 4 folds is around 4h30min now, single loop (no Parallel usage)
I use AUC_ROC and MCC metrics (matthew correlation coefficient). pf1 score I plot as well but only to monitor it.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2074309, "author_name": "nicehzj", "author_url": "", "post_date": "2022-12-24T01:20:47.677000", "content": "Great dataset!
\nIf I understand correctly, you keep the picture long side fixed for 1024, right?
\nAlso I have a question that, for submission, will this process be in 9 hours for 1024pix?
\nBecause we need to read from dcm, yolo, resize, ensemble… you said 7 hours for 768 but how about 1024?
\nThanks!
I tested 1600px submit (ROI extracted - yolo nano (model 001 from my public available dataset) on 3 models (efficientnetv2_s) and each has TTA (only flip). It takes 7h.
", "votes": 0, "replies": [] } ] }, { "id": 2061102, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-10T17:30:52.917000", "content": "I checked speed of pipeline:
\n7h submission time on 1xKaggle GPU
", "votes": 0, "replies": [] }, { "id": 2051359, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-12-01T11:15:09.027000", "content": "I'm getting this error :/
\n\n", "votes": 0, "replies": [ { "id": 2051373, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-01T11:22:34.073000", "content": "Data not available. This dataset may need it's archive recreated. Contact dataset author.
\n
@theoviel it is ok -> you have to download it here
\n
I use the kaggle dataset API, but it's working now. Thanks anyways !
", "votes": 1, "replies": [] } ] }, { "id": 2051349, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-01T11:04:31.443000", "content": "Dataset is available for 768 resolution. It means that longer dimension is 768pix.
\nThis is baseline dataset. Now you can create 512, 384, 256 … just using cv2.resize or other methods.
https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images
", "votes": 0, "replies": [] }, { "id": 2051249, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-01T09:49:11.993000", "content": "This is awesome work! 🙌 Well be a bit hard to run it on test though… 🤔 But if someone will figure out how to do this, that can give a major LB boost…
\nSo many interesting avenues to explore in this competition, so little time in a day!
", "votes": 0, "replies": [ { "id": 2051260, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-01T09:59:11.207000", "content": "Not exactly. I am sure it will work without any problem on test. I checked it on my notebook and the most consuming time part is convert dcm file to png file (you have to do anyway). Inference is a tiny amout of processing time (miliseconds).
\nYou are sure. This is absolutely fantastic competition. Not only because of ML and computer vision (my favourite task) but … idea.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "2051239": "I created ROI (Region Of Interest) extracted dataset for this compettion. Hope this improve score a lot.\n\nSolution:\n- ROI extractor -> https://www.kaggle.com/code/remekkinas/breast-cancer-roi-brest-extractor\n- ROI is resized to SIZE (I resized only taking into account longer dimension) so no distorision -> you can decide in augumentation step what to do with extracted ROI \n\n\n\nDataset is available here: https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images", "2051735": "I can confirm your preprocessing gives a 0.02 CV boost. ", "2056932": "Thank you @remekkinas for sharing!\n\nCan you just clarify your pipeline for me please:\n- you trained the yolov5 model on resized images of size (768, 768)\n- you inferred the position of the breast on images of size (768, 768)\n- from the detected bounding box resize the bounding box area from (h_long, w_smaller) -> (768, w_smaller*h_long/768) then save ?", "2099972": "Today I updated dataset -> resolution 1280px (longer side of image) and all imagess were processed using windowing (dicomsdl). I checked them in my notebook and works better for training (maybe due to consistent file processing). Now they look better as well : https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images\n\n\n\n", "2066520": "Now dataset is updated. I cropped oryginal DS to 1024pix. I can see a lot of improvement using higher cropped resolution. Have a nice Kaggling with this dataset.\n\nFor crop I used yolov5 and nano model. Source:\n- model -> 001 -> https://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-roi-model\n- training notebook -> https://www.kaggle.com/code/remekkinas/roi-detector-yolov5-training-and-annotations\n- annotations -> https://www.kaggle.com/datasets/remekkinas/rsna-roi-detector-annotations-yolo", "2061165": "This is ROI extraction test report I made today: \n\n\n@radek1 now you can see that speed is ok for yolo.", "2051288": "Cool dataset !\n\nWatch out for license issues with yolov5 though =)", "2145560": "Dear @remekkinas. I'm training my own experiment and tuning hyperparameter but my validation could not improve.Can you give me some advices to improve my score. Thank you very much.\n\n", "2142375": "Hello, guys can somebody answer me this https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/386518", "2112999": "Here is a dumb question...\n\nI just wonder if my input ROI images are l large like 2048*2048. When inferring could I store it locally in workworking/tem/imgfolder? I remember there is a max storage limitation. \n\n1536*960 0.5MB for each sample.", "2080969": "@remekkinas I've tested your YoloV5 model, it looks quite good. About `Parallel(n_jobs=2)` execution, you might experiment the following issue that happens a bit randomly (it happened to me): https://github.com/ultralytics/yolov5/issues/2824 that prevents ROI detection for a small number of images. It could be a problem during inference that you won't see if you `try/catch` the ROI model execution that could affect the final score. I'm testing a fix currently ...\n\nThe error I got for a few images:\n>The size of tensor a (104) must match the size of tensor b (112) at non-singleton dimension 3", "2074309": "Great dataset!\nIf I understand correctly, you keep the picture long side fixed for 1024, right?\nAlso I have a question that, for submission, will this process be in 9 hours for 1024pix?\nBecause we need to read from dcm, yolo, resize, ensemble... you said 7 hours for 768 but how about 1024?\nThanks!", "2061102": "I checked speed of pipeline:\n\n1. dicom convert to jpg using dicomsdl\n2. extract roi using yolov5 (model small / nano can be even faster)\n3. convert to 512x512\n4. blend - 3 models (resnet50d) with TTA (standard inference + flip)\n\n7h submission time on 1xKaggle GPU\n", "2051359": "I'm getting this error :/\n> Data not available. This dataset may need it's archive recreated. Contact dataset author.", "2051349": "Dataset is available for 768 resolution. It means that longer dimension is 768pix. \nThis is baseline dataset. Now you can create 512, 384, 256 ... just using cv2.resize or other methods.\n\nhttps://www.kaggle.com/datasets/remekkinas/rsna-breast-cancer-detection-poi-images ", "2051249": "This is awesome work! 🙌 Well be a bit hard to run it on test though... 🤔 But if someone will figure out how to do this, that can give a major LB boost...\n\nSo many interesting avenues to explore in this competition, so little time in a day! " } }