![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F478989%2F51ad8deb361d5f8c50f2265970a30d3c%2FPicture1.png?generation=1604080349102371&alt=media\")
Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this competition and presenting us this interesting problem. The data size is big and of high quality and there is no shakeup. I’m glad I can win this one and I have learnt a lot during this journey.
\nSpecial thanks to @vaillant for providing the topic introduction and useful input processing code. Also, credits should go to last year’s RSNA winners, lots of their ideas are incorporated in my solution.
\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117242
My solution is described below.
\nCode: https://github.com/GuanshuoXu/RSNA-STR-Pulmonary-Embolism-Detection
\nInference kernel: https://www.kaggle.com/wowfattie/notebook6fff7ff27a?scriptVersionId=45476524
Early after I joined this competition, I noticed that increasing input image size from 512x512 to 640x640 improves the modeling performance. By browsing the training images, I further noticed that the lungs did not occupy large and consistent portions of the images. This is inefficient because we know input size matters and it’s not worthy to waste computing time on irrelevant things in the images, and this could also give the modeling unnecessary difficulty to learn large scale and shift invariance. So, it’s necessary to have a high-quality lung localizer. There are some existing pretrained lung localizer online, I did not try them because according to my observation it’s easy for a CNN to accurately localize the lung area from images as long as we have the bbox labels of the lungs. So, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone. For simplicity I only annotated four images per study. The training and prediction process were also on only four images per study to save time. Some examples of this preprocessing are given below. The localizer is very robust even in some relatively difficult conditions. The idea of preprocessing the input is partly inspired from last year’s 2nd place solution.
\n
Since the provided data are big and of high quality, we don't have to do cross validation, a single training/validation split is reliable enough. In this competition, I randomly set aside 1000 studies for validation and used the rest 6200+ studies for training and hyperparameter tuning. For final LB submission I re-trained my models with the full training set and the optimized hyperparameters.
\nI used the same 2-stage training strategy as in last year’s RSNA competitions. For image-level modeling, the 3-channel input was the PE windows of the current image and its two direct neighbors. Using neighboring images has proved to be effective in last years 1st and 3rd place solutions. My experiments also confirmed that this input setting outperformed single images with 3 types of windows.
\nApart from predicting image-level labels, this year we are given various study-level labels. At first glance it appeared to me that, because the input of the study-level models are image embeddings, we need to use these study-level labels during image-level modeling so that the following study-level model could have sufficient knowledge to model and predict them. But after I tried lots of combinations of them and various loss masking tricks, the best performing model in both the image-level and study-level stages was still the one trained with the image-level labels only. I’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.
\nThe training loss was the vanilla BCE loss with linear lr scheduler. No special data sampling was applied. I found that a single epoch through the train data was the optimal for my settings. The best augmentations were
\nalbumentations.RandomContrast(limit=0.2, p=1.0),\nalbumentations.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=20, border_mode=cv2.BORDER_CONSTANT, p=1.0),\nalbumentations.Cutout(num_holes=2, max_h_size=int(0.4*image_size), max_w_size=int(0.4*image_size), fill_value=0, always_apply=True, p=1.0),\nMy final ensemble were with one serexnext50 and one seresnext101. Their respective validation performance for image-level PE prediction was
\n Loss AUC\nseresnext101 0.079 0.964\nseresnext50 0.080 0.962\nOther good backbones are inception_resnet_v2 and efficientnets. Densenets and resnexts performed a lot worse. Input were resized to 576x576 after the lung localization, this was the largest size the models could finish running in the 9 hours.
\nImage embeddings of dimension 2048 served as the input to a RNN for both image-level and study-level modeling.
\nOne thing we need to handle was that the number of images each study has could vary from 100+ to 1000+. As we don’t know the information of the private test data, it was hard to predefine an input sequence length for our RNN model if we want to predict all the images. Stacking all the images into a 3-D array and resizing it along the z-axis before generating image embeddings is an option, but it was not compatible to my inference pipeline. For convenience, I swapped the order of embedding generation and resizing, in other words, I chose to resize the features instead of images. For example, given a study which has N images, the input feature shape is Nx2048. If the max sequence length limit in the RNN is M, the cv2.resize function is applied to resize features to Mx2048 if N>M, otherwise if N<M zero-padding is used. The image-level labels and the predictions are zoomed in and out in the same way during training and inference. To find the best M, I ran a search in the step size of 32, and M=128 gave the best performance. In the train set, the majority of Ns is in the range of 200-250. This means downsizing across the z-pos first before sequence modeling improves the performance. In my final models, I actually set m=192 because I believed there might be more big Ns in the private test data.
\nInspired from last year's 2nd place solution, I also computed the difference of embeddings between current and the two direct neighbors and concatenate with the current features. So the input size was expanded to 2048x3.
\nThe exact RNN architecture is not very important, I settled down to only a single bidirectional GRU layer, with the study-level labels predicted by a concatenated attention weighted average pooling and max pooling over the sequence. My local validation loss was around 0.18, I have no idea why it is much higher than the LB scores.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F478989%2F5aabec8cb08e9bc19b65b9efd4c0dcdd%2FPicture5.png?generation=1604080838336664&alt=media\")
The main purpose of the postprocessing step is to satisfy the consistency requirement of the labels. Since this consistency requirement agrees with how the data was labeled, a careful postprocessing could improve the performance. In my case, the local validation has a tiny improvement after postprocessing. The brief workflow of the postprocessing is
\nfor each study:\n if the original predictions satisfy the consistency requirement\n do nothing\n else\n change the original predictions into consistent positive predictions, and compute loss between them\n change the original predictions into consistent negative predictions, and compute loss between them\n choose from the positive and negative predictions based on which causes the smaller loss\nThe weights of the loss function is almost same as the competition metric, except that the q_i of image loss weight is replaced by a fixed 0.005 because we don’t have the ground truth of the test data. Code of this postprocessing can be found in my inference kernel.
", "messages": [ { "id": 1065098, "postDate": "2020-10-30T21:07:01.467Z", "content": "Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this competition and presenting us this interesting problem. The data size is big and of high quality and there is no shakeup. I’m glad I can win this one and I have learnt a lot during this journey.
\nSpecial thanks to @vaillant for providing the topic introduction and useful input processing code. Also, credits should go to last year’s RSNA winners, lots of their ideas are incorporated in my solution.
\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117242
My solution is described below.
\nCode: https://github.com/GuanshuoXu/RSNA-STR-Pulmonary-Embolism-Detection
\nInference kernel: https://www.kaggle.com/wowfattie/notebook6fff7ff27a?scriptVersionId=45476524
Early after I joined this competition, I noticed that increasing input image size from 512x512 to 640x640 improves the modeling performance. By browsing the training images, I further noticed that the lungs did not occupy large and consistent portions of the images. This is inefficient because we know input size matters and it’s not worthy to waste computing time on irrelevant things in the images, and this could also give the modeling unnecessary difficulty to learn large scale and shift invariance. So, it’s necessary to have a high-quality lung localizer. There are some existing pretrained lung localizer online, I did not try them because according to my observation it’s easy for a CNN to accurately localize the lung area from images as long as we have the bbox labels of the lungs. So, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone. For simplicity I only annotated four images per study. The training and prediction process were also on only four images per study to save time. Some examples of this preprocessing are given below. The localizer is very robust even in some relatively difficult conditions. The idea of preprocessing the input is partly inspired from last year’s 2nd place solution.
\n
Since the provided data are big and of high quality, we don't have to do cross validation, a single training/validation split is reliable enough. In this competition, I randomly set aside 1000 studies for validation and used the rest 6200+ studies for training and hyperparameter tuning. For final LB submission I re-trained my models with the full training set and the optimized hyperparameters.
\nI used the same 2-stage training strategy as in last year’s RSNA competitions. For image-level modeling, the 3-channel input was the PE windows of the current image and its two direct neighbors. Using neighboring images has proved to be effective in last years 1st and 3rd place solutions. My experiments also confirmed that this input setting outperformed single images with 3 types of windows.
\nApart from predicting image-level labels, this year we are given various study-level labels. At first glance it appeared to me that, because the input of the study-level models are image embeddings, we need to use these study-level labels during image-level modeling so that the following study-level model could have sufficient knowledge to model and predict them. But after I tried lots of combinations of them and various loss masking tricks, the best performing model in both the image-level and study-level stages was still the one trained with the image-level labels only. I’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.
\nThe training loss was the vanilla BCE loss with linear lr scheduler. No special data sampling was applied. I found that a single epoch through the train data was the optimal for my settings. The best augmentations were
\nalbumentations.RandomContrast(limit=0.2, p=1.0),\nalbumentations.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=20, border_mode=cv2.BORDER_CONSTANT, p=1.0),\nalbumentations.Cutout(num_holes=2, max_h_size=int(0.4*image_size), max_w_size=int(0.4*image_size), fill_value=0, always_apply=True, p=1.0),\nMy final ensemble were with one serexnext50 and one seresnext101. Their respective validation performance for image-level PE prediction was
\n Loss AUC\nseresnext101 0.079 0.964\nseresnext50 0.080 0.962\nOther good backbones are inception_resnet_v2 and efficientnets. Densenets and resnexts performed a lot worse. Input were resized to 576x576 after the lung localization, this was the largest size the models could finish running in the 9 hours.
\nImage embeddings of dimension 2048 served as the input to a RNN for both image-level and study-level modeling.
\nOne thing we need to handle was that the number of images each study has could vary from 100+ to 1000+. As we don’t know the information of the private test data, it was hard to predefine an input sequence length for our RNN model if we want to predict all the images. Stacking all the images into a 3-D array and resizing it along the z-axis before generating image embeddings is an option, but it was not compatible to my inference pipeline. For convenience, I swapped the order of embedding generation and resizing, in other words, I chose to resize the features instead of images. For example, given a study which has N images, the input feature shape is Nx2048. If the max sequence length limit in the RNN is M, the cv2.resize function is applied to resize features to Mx2048 if N>M, otherwise if N<M zero-padding is used. The image-level labels and the predictions are zoomed in and out in the same way during training and inference. To find the best M, I ran a search in the step size of 32, and M=128 gave the best performance. In the train set, the majority of Ns is in the range of 200-250. This means downsizing across the z-pos first before sequence modeling improves the performance. In my final models, I actually set m=192 because I believed there might be more big Ns in the private test data.
\nInspired from last year's 2nd place solution, I also computed the difference of embeddings between current and the two direct neighbors and concatenate with the current features. So the input size was expanded to 2048x3.
\nThe exact RNN architecture is not very important, I settled down to only a single bidirectional GRU layer, with the study-level labels predicted by a concatenated attention weighted average pooling and max pooling over the sequence. My local validation loss was around 0.18, I have no idea why it is much higher than the LB scores.
\n
The main purpose of the postprocessing step is to satisfy the consistency requirement of the labels. Since this consistency requirement agrees with how the data was labeled, a careful postprocessing could improve the performance. In my case, the local validation has a tiny improvement after postprocessing. The brief workflow of the postprocessing is
\nfor each study:\n if the original predictions satisfy the consistency requirement\n do nothing\n else\n change the original predictions into consistent positive predictions, and compute loss between them\n change the original predictions into consistent negative predictions, and compute loss between them\n choose from the positive and negative predictions based on which causes the smaller loss\nThe weights of the loss function is almost same as the competition metric, except that the q_i of image loss weight is replaced by a fixed 0.005 because we don’t have the ground truth of the test data. Code of this postprocessing can be found in my inference kernel.
", "rawMarkdown": "Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this competition and presenting us this interesting problem. The data size is big and of high quality and there is no shakeup. I’m glad I can win this one and I have learnt a lot during this journey.\nSpecial thanks to @vaillant for providing the topic introduction and useful input processing code. Also, credits should go to last year’s RSNA winners, lots of their ideas are incorporated in my solution.\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117242\n\nMy solution is described below. \nCode: https://github.com/GuanshuoXu/RSNA-STR-Pulmonary-Embolism-Detection\nInference kernel: https://www.kaggle.com/wowfattie/notebook6fff7ff27a?scriptVersionId=45476524\n\n# Preprocessing\n\nEarly after I joined this competition, I noticed that increasing input image size from 512x512 to 640x640 improves the modeling performance. By browsing the training images, I further noticed that the lungs did not occupy large and consistent portions of the images. This is inefficient because we know input size matters and it’s not worthy to waste computing time on irrelevant things in the images, and this could also give the modeling unnecessary difficulty to learn large scale and shift invariance. So, it’s necessary to have a high-quality lung localizer. There are some existing pretrained lung localizer online, I did not try them because according to my observation it’s easy for a CNN to accurately localize the lung area from images as long as we have the bbox labels of the lungs. So, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone. For simplicity I only annotated four images per study. The training and prediction process were also on only four images per study to save time. Some examples of this preprocessing are given below. The localizer is very robust even in some relatively difficult conditions. The idea of preprocessing the input is partly inspired from last year’s 2nd place solution.\n\n\n\n# Training/validation split\n\nSince the provided data are big and of high quality, we don't have to do cross validation, a single training/validation split is reliable enough. In this competition, I randomly set aside 1000 studies for validation and used the rest 6200+ studies for training and hyperparameter tuning. For final LB submission I re-trained my models with the full training set and the optimized hyperparameters.\n\n# Image-level modeling\n\nI used the same 2-stage training strategy as in last year’s RSNA competitions. For image-level modeling, the 3-channel input was the PE windows of the current image and its two direct neighbors. Using neighboring images has proved to be effective in last years 1st and 3rd place solutions. My experiments also confirmed that this input setting outperformed single images with 3 types of windows.\n\nApart from predicting image-level labels, this year we are given various study-level labels. At first glance it appeared to me that, because the input of the study-level models are image embeddings, we need to use these study-level labels during image-level modeling so that the following study-level model could have sufficient knowledge to model and predict them. But after I tried lots of combinations of them and various loss masking tricks, the best performing model in both the image-level and study-level stages was still the one trained with the image-level labels only. I’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.\n\nThe training loss was the vanilla BCE loss with linear lr scheduler. No special data sampling was applied. I found that a single epoch through the train data was the optimal for my settings. The best augmentations were \n\n```\nalbumentations.RandomContrast(limit=0.2, p=1.0),\nalbumentations.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=20, border_mode=cv2.BORDER_CONSTANT, p=1.0),\nalbumentations.Cutout(num_holes=2, max_h_size=int(0.4*image_size), max_w_size=int(0.4*image_size), fill_value=0, always_apply=True, p=1.0),\n```\n\nMy final ensemble were with one serexnext50 and one seresnext101. Their respective validation performance for image-level PE prediction was\n\n```\n Loss AUC\nseresnext101 0.079 0.964\nseresnext50 0.080 0.962\n```\n\nOther good backbones are inception_resnet_v2 and efficientnets. Densenets and resnexts performed a lot worse. Input were resized to 576x576 after the lung localization, this was the largest size the models could finish running in the 9 hours.\n\n# Study-level modeling\n\nImage embeddings of dimension 2048 served as the input to a RNN for both image-level and study-level modeling. \n\nOne thing we need to handle was that the number of images each study has could vary from 100+ to 1000+. As we don’t know the information of the private test data, it was hard to predefine an input sequence length for our RNN model if we want to predict all the images. Stacking all the images into a 3-D array and resizing it along the z-axis before generating image embeddings is an option, but it was not compatible to my inference pipeline. For convenience, I swapped the order of embedding generation and resizing, in other words, I chose to resize the features instead of images. For example, given a study which has N images, the input feature shape is Nx2048. If the max sequence length limit in the RNN is M, the cv2.resize function is applied to resize features to Mx2048 if N>M, otherwise if NCongratulations Guanshuo. Brilliant and simple solution, well done! I like how you located the lungs, that was smart. My best models used 320x320 center crops from 512x512 which looks like it corresponds to your lung bbox.
\n\n\nI actually set m=192 because I believed there might be more big Ns in the private test data.
\n
This is good intuition. I probed the private LB. The public LB has 0% rows associtated with exams of more than 350 images. The private LB has 10% rows associated with exams of more than 350 images. (The train data had around 8%)
", "rawMarkdown": "Congratulations Guanshuo. Brilliant and simple solution, well done! I like how you located the lungs, that was smart. My best models used 320x320 center crops from 512x512 which looks like it corresponds to your lung bbox.\n\n> I actually set m=192 because I believed there might be more big Ns in the private test data.\n\nThis is good intuition. I probed the private LB. The public LB has 0% rows associtated with exams of more than 350 images. The private LB has 10% rows associated with exams of more than 350 images. (The train data had around 8%)", "votes": 8 }, { "id": 1065117, "postDate": "2020-10-30T21:38:08.553Z", "content": "Congratulations on 1st place and competitions rank 1! Thanks for sharing your solution.
\n\n\nI’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.
\n
This is interesting- I'm also surprised. I used the study-level labels while training the image-level model because I thought they were necessary to encode those features as well, but apparently not.
\nHow did you apply the bounding box predictions to the image? Did you use the largest box across all slices for each slice? Or individual boxes per slice? Some slices don't have lungs at all.
", "rawMarkdown": "Congratulations on 1st place and competitions rank 1! Thanks for sharing your solution. \n\n> I’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.\n\nThis is interesting- I'm also surprised. I used the study-level labels while training the image-level model because I thought they were necessary to encode those features as well, but apparently not. \n\nHow did you apply the bounding box predictions to the image? Did you use the largest box across all slices for each slice? Or individual boxes per slice? Some slices don't have lungs at all. ", "votes": 4, "replies": [ { "id": 1065159, "postDate": "2020-10-30T23:31:21.060Z", "content": "Thanks. Congratulations to you too.
\nI computed the largest possible bbox from the four bboxes for each study/scan. The code from my inference kernel.
xmin = np.round(min([bbox[0,0], bbox[1,0], bbox[2,0], bbox[3,0]])*512)\n ymin = np.round(min([bbox[0,1], bbox[1,1], bbox[2,1], bbox[3,1]])*512)\n xmax = np.round(max([bbox[0,2], bbox[1,2], bbox[2,2], bbox[3,2]])*512)\n ymax = np.round(max([bbox[0,3], bbox[1,3], bbox[2,3], bbox[3,3]])*512)\n bbox_dict[series_id] = [int(max(0, xmin)), int(max(0, ymin)), int(min(512, xmax)), int(min(512, ymax))]\nCongratulations and thanks for sharing your solution! I also started off using lungmasks and then finding the largest bounding box for a given study but it wasn't feasible given the code requirements and I couldn't think of training a new model like you did unfortunately. Do you know how much improvement lung cropping gave in your final solution or was it mainly helpful for using computational resources efficiently?
\nWe also noticed a huge gap (0.03) between our local CV (1456 studies for validation) vs the public/private LB, don't have an idea why it happens.
\nResizing sequence input and then resizing the output back to map the original image number was something I was wondering if possible, thanks for sharing it is!
\nPretty novel and clever postprocessing!
\nWhat kind of attention mechanism is applied to GRU hidden outputs?
", "rawMarkdown": "Congratulations and thanks for sharing your solution! I also started off using lungmasks and then finding the largest bounding box for a given study but it wasn't feasible given the code requirements and I couldn't think of training a new model like you did unfortunately. Do you know how much improvement lung cropping gave in your final solution or was it mainly helpful for using computational resources efficiently? \n\nWe also noticed a huge gap (0.03) between our local CV (1456 studies for validation) vs the public/private LB, don't have an idea why it happens.\n\nResizing sequence input and then resizing the output back to map the original image number was something I was wondering if possible, thanks for sharing it is!\n\nPretty novel and clever postprocessing!\n\nWhat kind of attention mechanism is applied to GRU hidden outputs?", "votes": 2, "replies": [ { "id": 1065161, "postDate": "2020-10-30T23:39:12.047Z", "content": "I only have a result of a naive model, with lung localization the AUC improved from 0.945 to 0.953.
", "rawMarkdown": "I only have a result of a naive model, with lung localization the AUC improved from 0.945 to 0.953.", "votes": 2 }, { "id": 1065194, "postDate": "2020-10-31T02:20:18.283Z", "content": "Thanks that's helpful!
", "rawMarkdown": "Thanks that's helpful!" } ] }, { "id": 3434703, "postDate": "2026-04-03T06:22:18.973Z", "content": "Congratulations Guanshuo. Brilliant and simple solution
", "rawMarkdown": "Congratulations Guanshuo. Brilliant and simple solution" }, { "id": 2456806, "postDate": "2023-09-26T12:25:38.493Z", "content": "this is a very detailed solution, thanks
", "rawMarkdown": "this is a very detailed solution, thanks " }, { "id": 1851358, "postDate": "2022-07-11T07:27:25.697Z", "content": "Would you please provide the trained weights?
", "rawMarkdown": "Would you please provide the trained weights?" }, { "id": 1738699, "postDate": "2022-03-29T13:09:43.133Z", "content": "Hello, author, congratulations on winning the first prize. I tried to reproduce your code content, but the final effect failed to reach your AUC of 0.96. Do you know the reason?
", "rawMarkdown": "Hello, author, congratulations on winning the first prize. I tried to reproduce your code content, but the final effect failed to reach your AUC of 0.96. Do you know the reason?" }, { "id": 1124234, "postDate": "2020-12-23T18:57:17.100Z", "content": "Thank you for sharing, and congratulation for 1st place.
", "rawMarkdown": "Thank you for sharing, and congratulation for 1st place." }, { "id": 1090057, "postDate": "2020-11-25T03:14:37.813Z", "content": "Hi thank you for posting this awesome code! I'm trying to run the lung localizer and getting \"KeyErrors\" like there are missing file names in the lung localization lung_bbox.csv file
\nAny idea what might be going wrong?
", "rawMarkdown": "Hi thank you for posting this awesome code! I'm trying to run the lung localizer and getting \"KeyErrors\" like there are missing file names in the lung localization lung_bbox.csv file\n\nAny idea what might be going wrong?", "replies": [ { "id": 1090158, "postDate": "2020-11-25T06:01:49.080Z", "content": "actually getting this error:
\nKeyError: Caught KeyError in DataLoader worker process 0
Maybe its a memory issue trying to run this large dataset on Kaggle?
", "rawMarkdown": "actually getting this error:\nKeyError: Caught KeyError in DataLoader worker process 0\n\nMaybe its a memory issue trying to run this large dataset on Kaggle?" } ] }, { "id": 1076765, "postDate": "2020-11-12T21:08:09.330Z", "content": "Well done on the achievement and thanks for this amazing breakdown and the accompanying github repo. I have one question (actually I would ask a hundred if I could 😜). Why do you use a max pool instead of average pool over the sequence dimension?
\nmax_pool, _ = torch.max(h_lstm1, 1)\nI was actually working on another project and realised max pool worked better than average in a similar scenario. But would love to get your thoughts.
", "rawMarkdown": "Well done on the achievement and thanks for this amazing breakdown and the accompanying github repo. I have one question (actually I would ask a hundred if I could 😜). Why do you use a max pool instead of average pool over the sequence dimension?\n\n```\nmax_pool, _ = torch.max(h_lstm1, 1)\n```\n\nI was actually working on another project and realised max pool worked better than average in a similar scenario. But would love to get your thoughts.", "replies": [ { "id": 1076888, "postDate": "2020-11-13T02:39:26.723Z", "content": "I used both max and attention pooling. The attention pooling is similar to average pooling.
", "rawMarkdown": "I used both max and attention pooling. The attention pooling is similar to average pooling.", "votes": 2 }, { "id": 1077099, "postDate": "2020-11-13T08:37:02.870Z", "content": "Thank you! Yes, your work has finally convinced me to take some time to learn about attention. Nevertheless, any logic or anecdotal experience behind the choice? I ask because I've been on this topic for some days now
", "rawMarkdown": "Thank you! Yes, your work has finally convinced me to take some time to learn about attention. Nevertheless, any logic or anecdotal experience behind the choice? I ask because [I've been on this topic for some days now](https://stats.stackexchange.com/questions/495954/what-are-ways-to-learn-a-classifier-for-labelling-a-series-of-images-rather-than)" }, { "id": 1077534, "postDate": "2020-11-13T17:30:11.877Z", "content": "Sequence model may not be suitable for your problem. You can take a lot at some previous kaggle competitions, for example
\nhttps://www.kaggle.com/c/yelp-restaurant-photo-classification
\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge
Nice work. Congrats on 1st place
", "rawMarkdown": "Nice work. Congrats on 1st place" }, { "id": 1070619, "postDate": "2020-11-06T00:57:06.480Z", "content": "Nice solution
", "rawMarkdown": "Nice solution" }, { "id": 1069890, "postDate": "2020-11-05T04:03:02.087Z", "content": "Congrats! Very good work.
", "rawMarkdown": "Congrats! Very good work." }, { "id": 1068578, "postDate": "2020-11-03T14:29:59.087Z", "content": "Amazing.Stuff
", "rawMarkdown": "Amazing.Stuff\n" }, { "id": 1068313, "postDate": "2020-11-03T09:11:52.827Z", "content": "congratulations and thanks for sharing the solution. Thanks for the guidance.
", "rawMarkdown": "congratulations and thanks for sharing the solution. Thanks for the guidance." }, { "id": 1068237, "postDate": "2020-11-03T07:46:54.387Z", "content": "Nice! Congratulations on the first place :)
", "rawMarkdown": "Nice! Congratulations on the first place :)" }, { "id": 1067998, "postDate": "2020-11-03T00:51:23.720Z", "content": "Congratulations! Thanks for sharing your solution.
", "rawMarkdown": "Congratulations! Thanks for sharing your solution." }, { "id": 1067049, "postDate": "2020-11-02T10:10:34.107Z", "content": "\n\nSo, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone.
\n
Awesome !!
", "rawMarkdown": "> So, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone.\n\nAwesome !! " }, { "id": 1065177, "postDate": "2020-10-31T00:49:47.710Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1738701, "postDate": "2022-03-29T13:11:48.393Z", "content": "Thank you for your reply!
", "rawMarkdown": "Thank you for your reply!" } ], "comments": [ { "id": 1065212, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-10-31T03:11:41.330000", "content": "Really waited for your solution. Thanks for sharing your details solution and code. Congrats on 1st place and 1 st global competitions ranking @wowfattie !
", "votes": 7, "replies": [] }, { "id": 1065229, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-31T04:00:23.793000", "content": "Congratulations Guanshuo. Brilliant and simple solution, well done! I like how you located the lungs, that was smart. My best models used 320x320 center crops from 512x512 which looks like it corresponds to your lung bbox.
\n\n\nI actually set m=192 because I believed there might be more big Ns in the private test data.
\n
This is good intuition. I probed the private LB. The public LB has 0% rows associtated with exams of more than 350 images. The private LB has 10% rows associated with exams of more than 350 images. (The train data had around 8%)
", "votes": 8, "replies": [] }, { "id": 1065117, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-30T21:38:08.553000", "content": "Congratulations on 1st place and competitions rank 1! Thanks for sharing your solution.
\n\n\nI’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.
\n
This is interesting- I'm also surprised. I used the study-level labels while training the image-level model because I thought they were necessary to encode those features as well, but apparently not.
\nHow did you apply the bounding box predictions to the image? Did you use the largest box across all slices for each slice? Or individual boxes per slice? Some slices don't have lungs at all.
", "votes": 4, "replies": [ { "id": 1065159, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-10-30T23:31:21.060000", "content": "Thanks. Congratulations to you too.
\nI computed the largest possible bbox from the four bboxes for each study/scan. The code from my inference kernel.
xmin = np.round(min([bbox[0,0], bbox[1,0], bbox[2,0], bbox[3,0]])*512)\n ymin = np.round(min([bbox[0,1], bbox[1,1], bbox[2,1], bbox[3,1]])*512)\n xmax = np.round(max([bbox[0,2], bbox[1,2], bbox[2,2], bbox[3,2]])*512)\n ymax = np.round(max([bbox[0,3], bbox[1,3], bbox[2,3], bbox[3,3]])*512)\n bbox_dict[series_id] = [int(max(0, xmin)), int(max(0, ymin)), int(min(512, xmax)), int(min(512, ymax))]\nCongratulations and thanks for sharing your solution! I also started off using lungmasks and then finding the largest bounding box for a given study but it wasn't feasible given the code requirements and I couldn't think of training a new model like you did unfortunately. Do you know how much improvement lung cropping gave in your final solution or was it mainly helpful for using computational resources efficiently?
\nWe also noticed a huge gap (0.03) between our local CV (1456 studies for validation) vs the public/private LB, don't have an idea why it happens.
\nResizing sequence input and then resizing the output back to map the original image number was something I was wondering if possible, thanks for sharing it is!
\nPretty novel and clever postprocessing!
\nWhat kind of attention mechanism is applied to GRU hidden outputs?
", "votes": 2, "replies": [ { "id": 1065161, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-10-30T23:39:12.047000", "content": "I only have a result of a naive model, with lung localization the AUC improved from 0.945 to 0.953.
", "votes": 2, "replies": [] }, { "id": 1065194, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-31T02:20:18.283000", "content": "Thanks that's helpful!
", "votes": 0, "replies": [] } ] }, { "id": 3434703, "author_name": "Mahmoud Alyosify", "author_url": "", "post_date": "2026-04-03T06:22:18.973000", "content": "Congratulations Guanshuo. Brilliant and simple solution
", "votes": 0, "replies": [] }, { "id": 2456806, "author_name": "Abdulquadri Adegbiji", "author_url": "", "post_date": "2023-09-26T12:25:38.493000", "content": "this is a very detailed solution, thanks
", "votes": 0, "replies": [] }, { "id": 1851358, "author_name": "Hi-AI", "author_url": "", "post_date": "2022-07-11T07:27:25.697000", "content": "Would you please provide the trained weights?
", "votes": 0, "replies": [] }, { "id": 1738699, "author_name": "li huatao", "author_url": "", "post_date": "2022-03-29T13:09:43.133000", "content": "Hello, author, congratulations on winning the first prize. I tried to reproduce your code content, but the final effect failed to reach your AUC of 0.96. Do you know the reason?
", "votes": 0, "replies": [] }, { "id": 1124234, "author_name": "Ammar Nassr Mohammed", "author_url": "", "post_date": "2020-12-23T18:57:17.100000", "content": "Thank you for sharing, and congratulation for 1st place.
", "votes": 0, "replies": [] }, { "id": 1090057, "author_name": "rivesis", "author_url": "", "post_date": "2020-11-25T03:14:37.813000", "content": "Hi thank you for posting this awesome code! I'm trying to run the lung localizer and getting \"KeyErrors\" like there are missing file names in the lung localization lung_bbox.csv file
\nAny idea what might be going wrong?
", "votes": 0, "replies": [ { "id": 1090158, "author_name": "rivesis", "author_url": "", "post_date": "2020-11-25T06:01:49.080000", "content": "actually getting this error:
\nKeyError: Caught KeyError in DataLoader worker process 0
Maybe its a memory issue trying to run this large dataset on Kaggle?
", "votes": 0, "replies": [] } ] }, { "id": 1076765, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-11-12T21:08:09.330000", "content": "Well done on the achievement and thanks for this amazing breakdown and the accompanying github repo. I have one question (actually I would ask a hundred if I could 😜). Why do you use a max pool instead of average pool over the sequence dimension?
\nmax_pool, _ = torch.max(h_lstm1, 1)\nI was actually working on another project and realised max pool worked better than average in a similar scenario. But would love to get your thoughts.
", "votes": 0, "replies": [ { "id": 1076888, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-11-13T02:39:26.723000", "content": "I used both max and attention pooling. The attention pooling is similar to average pooling.
", "votes": 2, "replies": [] }, { "id": 1077099, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-11-13T08:37:02.870000", "content": "Thank you! Yes, your work has finally convinced me to take some time to learn about attention. Nevertheless, any logic or anecdotal experience behind the choice? I ask because I've been on this topic for some days now
", "votes": 0, "replies": [] }, { "id": 1077534, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-11-13T17:30:11.877000", "content": "Sequence model may not be suitable for your problem. You can take a lot at some previous kaggle competitions, for example
\nhttps://www.kaggle.com/c/yelp-restaurant-photo-classification
\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge
Nice work. Congrats on 1st place
", "votes": 0, "replies": [] }, { "id": 1070619, "author_name": "Nguyen Nhat Minh", "author_url": "", "post_date": "2020-11-06T00:57:06.480000", "content": "Nice solution
", "votes": 0, "replies": [] }, { "id": 1069890, "author_name": "David Llanio", "author_url": "", "post_date": "2020-11-05T04:03:02.087000", "content": "Congrats! Very good work.
", "votes": 0, "replies": [] }, { "id": 1068578, "author_name": "BipanGharti", "author_url": "", "post_date": "2020-11-03T14:29:59.087000", "content": "Amazing.Stuff
", "votes": 0, "replies": [] }, { "id": 1068313, "author_name": "keerthi pakka", "author_url": "", "post_date": "2020-11-03T09:11:52.827000", "content": "congratulations and thanks for sharing the solution. Thanks for the guidance.
", "votes": 0, "replies": [] }, { "id": 1068237, "author_name": "kagglebaggledaggle", "author_url": "", "post_date": "2020-11-03T07:46:54.387000", "content": "Nice! Congratulations on the first place :)
", "votes": 0, "replies": [] }, { "id": 1067998, "author_name": "Alin Cijov", "author_url": "", "post_date": "2020-11-03T00:51:23.720000", "content": "Congratulations! Thanks for sharing your solution.
", "votes": 0, "replies": [] }, { "id": 1067049, "author_name": "Vee", "author_url": "", "post_date": "2020-11-02T10:10:34.107000", "content": "\n\nSo, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone.
\n
Awesome !!
", "votes": 0, "replies": [] }, { "id": 1065177, "author_name": "", "author_url": "", "post_date": "2020-10-31T00:49:47.710000", "content": "", "votes": 0, "replies": [] }, { "id": 1738701, "author_name": "li huatao", "author_url": "", "post_date": "2022-03-29T13:11:48.393000", "content": "Thank you for your reply!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1065098": "Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this competition and presenting us this interesting problem. The data size is big and of high quality and there is no shakeup. I’m glad I can win this one and I have learnt a lot during this journey.\nSpecial thanks to @vaillant for providing the topic introduction and useful input processing code. Also, credits should go to last year’s RSNA winners, lots of their ideas are incorporated in my solution.\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117242\n\nMy solution is described below. \nCode: https://github.com/GuanshuoXu/RSNA-STR-Pulmonary-Embolism-Detection\nInference kernel: https://www.kaggle.com/wowfattie/notebook6fff7ff27a?scriptVersionId=45476524\n\n# Preprocessing\n\nEarly after I joined this competition, I noticed that increasing input image size from 512x512 to 640x640 improves the modeling performance. By browsing the training images, I further noticed that the lungs did not occupy large and consistent portions of the images. This is inefficient because we know input size matters and it’s not worthy to waste computing time on irrelevant things in the images, and this could also give the modeling unnecessary difficulty to learn large scale and shift invariance. So, it’s necessary to have a high-quality lung localizer. There are some existing pretrained lung localizer online, I did not try them because according to my observation it’s easy for a CNN to accurately localize the lung area from images as long as we have the bbox labels of the lungs. So, I annotated the train data and built a lung localizer with the bboxes and Efficientnet-b0 as the backbone. For simplicity I only annotated four images per study. The training and prediction process were also on only four images per study to save time. Some examples of this preprocessing are given below. The localizer is very robust even in some relatively difficult conditions. The idea of preprocessing the input is partly inspired from last year’s 2nd place solution.\n\n\n\n# Training/validation split\n\nSince the provided data are big and of high quality, we don't have to do cross validation, a single training/validation split is reliable enough. In this competition, I randomly set aside 1000 studies for validation and used the rest 6200+ studies for training and hyperparameter tuning. For final LB submission I re-trained my models with the full training set and the optimized hyperparameters.\n\n# Image-level modeling\n\nI used the same 2-stage training strategy as in last year’s RSNA competitions. For image-level modeling, the 3-channel input was the PE windows of the current image and its two direct neighbors. Using neighboring images has proved to be effective in last years 1st and 3rd place solutions. My experiments also confirmed that this input setting outperformed single images with 3 types of windows.\n\nApart from predicting image-level labels, this year we are given various study-level labels. At first glance it appeared to me that, because the input of the study-level models are image embeddings, we need to use these study-level labels during image-level modeling so that the following study-level model could have sufficient knowledge to model and predict them. But after I tried lots of combinations of them and various loss masking tricks, the best performing model in both the image-level and study-level stages was still the one trained with the image-level labels only. I’m a little puzzled how the image embeddings are encoded with the study-level labels, for example, the exact position labels (center, left, right) and the more refined acute and chronic, when the image-level models were not trained using any of those labels.\n\nThe training loss was the vanilla BCE loss with linear lr scheduler. No special data sampling was applied. I found that a single epoch through the train data was the optimal for my settings. The best augmentations were \n\n```\nalbumentations.RandomContrast(limit=0.2, p=1.0),\nalbumentations.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=20, border_mode=cv2.BORDER_CONSTANT, p=1.0),\nalbumentations.Cutout(num_holes=2, max_h_size=int(0.4*image_size), max_w_size=int(0.4*image_size), fill_value=0, always_apply=True, p=1.0),\n```\n\nMy final ensemble were with one serexnext50 and one seresnext101. Their respective validation performance for image-level PE prediction was\n\n```\n Loss AUC\nseresnext101 0.079 0.964\nseresnext50 0.080 0.962\n```\n\nOther good backbones are inception_resnet_v2 and efficientnets. Densenets and resnexts performed a lot worse. Input were resized to 576x576 after the lung localization, this was the largest size the models could finish running in the 9 hours.\n\n# Study-level modeling\n\nImage embeddings of dimension 2048 served as the input to a RNN for both image-level and study-level modeling. \n\nOne thing we need to handle was that the number of images each study has could vary from 100+ to 1000+. As we don’t know the information of the private test data, it was hard to predefine an input sequence length for our RNN model if we want to predict all the images. Stacking all the images into a 3-D array and resizing it along the z-axis before generating image embeddings is an option, but it was not compatible to my inference pipeline. For convenience, I swapped the order of embedding generation and resizing, in other words, I chose to resize the features instead of images. For example, given a study which has N images, the input feature shape is Nx2048. If the max sequence length limit in the RNN is M, the cv2.resize function is applied to resize features to Mx2048 if N>M, otherwise if N