{ "id": 195865, "title": "6th place solution", "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/195865", "author_name": "OsciiArt", "post_date": "2020-11-07T23:14:28.043000", "votes": 21, "comment_count": 3, "views": 0, "content": "

Congratulations to all the winners. Thank you to Kaggle and RSNA for hosting this competition.
\nI'm happy about my result because I'm a medical doctor and therefore, RSNA competitions are the most important competitions for me. In RSNA 2018, I did my best but I failed to get gold. I couldn't participate in RSNA 2019 because I had to prepare for the national exam for medical doctors. In RSNA 2020, I've got the solo gold finally. I'm so sad that I can't attend the RSNA conference in place because it becomes an online conference.

\n

Here I describe my solution. The overview is shown below. Actually, there is nothing special; 2D-CNN for image-level feature extraction and 1D-CNN for exam-level classification.

\n

\"Figure

\n

Stage 1: 2D-CNN for feature extraction

\n

First, I trained 2D-CNN (EfficientNet B0 or B2) with trainable 3 windows (WSO, following Yuval's solution) on 2D images with the condition shown below. To save memory and time, I trained models with mixed-precision.

\n\n

Training takes 3 hours x 5 folds with P100. I extracted 2D-CNN's feature, the output of the global average pooling layer, and used it as input for stage 2. Feature extraction takes 2. 5 hours x 5 folds.

\n

Stage 2: 1D-CNN for exam-level classification

\n

Next, I trained 1D-CNN for exam-level classification. As input, I used feature sequences extracted by 1st-stage 2D-CNN. Feature pooling, Skip connection, and SE-module are employed. For image-level prediction, I used U-Net-like upconv architectures. The training conditions are shown below.

\n\n

Postprocessing

\n

I averaged the predictions of B0 and B2. I did postprocessing to solve the conflict of label consistency with minimal modification. The 5-fold CV score is shown below.

\n
negative_exam_for_pe           bce: 0.345601, auc: 0.895471\nindeterminate                  bce: 0.084615, auc: 0.812207\nchronic_pe                     bce: 0.158846, auc: 0.682470\nacute_and_chronic_pe           bce: 0.081380, auc: 0.842780\ncentral_pe                     bce: 0.111196, auc: 0.949153\nleftsided_pe                   bce: 0.287363, auc: 0.900652\nrightsided_pe                  bce: 0.298690, auc: 0.911001\nrv_lv_ratio_gte_1              bce: 0.228942, auc: 0.902246\nrv_lv_ratio_lt_1               bce: 0.342175, auc: 0.835369\nexam-level score               bce: 0.196321\nq_i weighted_image_bce         bce: 0.206749, auc: 0.965775\ntotal_score                    bce: 0.201473\n
\n

Final submission

\n

As final submissions, I selected

\n
    \n
  1. B0 model with 5-fold averaging: public: 0.161, private: 0.157
    2. \n
  2. B0 and B2 model with 5-fold averaging and model averaging: public: 0.160, private: 0.156
    3. \n
\n

Comparison test

\n

Stage 1

\n
                 exam BCE   exam AUC   image BCE   image AUC\nB0 final model   0.314768   0.709304   0.101278    0.951922\n num step X2     0.316794   0.708131   0.100873    0.951224\n initial LR=1e-4 0.321354   0.698805   0.113893    0.938055\n BatchSize=16    0.320763   0.687397   0.101027    0.950439\n w/o CutOut      0.316906   0.703339   0.100654    0.950882\n InputSize=256   0.328765   0.681211   0.134463    0.917054\n w/o WSO         0.315638   0.703576   0.103514    0.947023\n with MixUp      0.317532   0.693032   0.103800    0.944957\n InputSize=640   0.321643   0.703420   0.108685    0.953675\nB2 final model   0.315517   0.707378   0.099449    0.950943\n
\n

I did comparison tests after the deadline. scores are calculated without any weights. 0.5 epoch is enough to converge. InputSize=512 is better than InputSize=256. InputSize=640 may be better than InputSize=512. LR=1e-3 is better than LR=1e-4. BatchSize=80 may be better than BatchSize=16. Cutout, WSO, and MixUp may not be necessary.

\n

Stage 2

\n
                 exam BCE   exam AUC   image BCE   image AUC\nB0 final model   0.200486   0.867666   0.222240    0.964814\n w/o pred-2      0.200840   0.867577   0.221665    0.963847\n LSTM            0.203616   0.864214   0.216788    0.963465\n GRU             0.195599   0.873768   0.219781    0.963367\nB2 final model   0.196107   0.871995   0.220244    0.962613\n
\n

Scores are calculated with weights of competition metrics. Training with pred-2 may not be necessary. LSTM and GRU may be able to achieve the same performance as CNN.

\n

What I had to do in this competition was obvious because this competition is very similar to the last year's one and Yuval, the last year's winner, was at the top of the LB. Actually, What I did was just implementing the last year's solution and making a custom loss that minimizes the competition metric directly. It is a very baseline model. I think my solution would be in the middle of silver medals in usual competitions. This competition is a little bit harder than usual competitions, because of a large dataset, complicated metric, short span, and notebook competition. Maybe that's why this baseline model can get gold.

\n

All the codes are available here.

", "messages": [ { "id": 1072220, "postDate": "2020-11-07T23:14:28.043Z", "content": "

Congratulations to all the winners. Thank you to Kaggle and RSNA for hosting this competition.
\nI'm happy about my result because I'm a medical doctor and therefore, RSNA competitions are the most important competitions for me. In RSNA 2018, I did my best but I failed to get gold. I couldn't participate in RSNA 2019 because I had to prepare for the national exam for medical doctors. In RSNA 2020, I've got the solo gold finally. I'm so sad that I can't attend the RSNA conference in place because it becomes an online conference.

\n

Here I describe my solution. The overview is shown below. Actually, there is nothing special; 2D-CNN for image-level feature extraction and 1D-CNN for exam-level classification.

\n

\"Figure

\n

Stage 1: 2D-CNN for feature extraction

\n

First, I trained 2D-CNN (EfficientNet B0 or B2) with trainable 3 windows (WSO, following Yuval's solution) on 2D images with the condition shown below. To save memory and time, I trained models with mixed-precision.

\n\n

Training takes 3 hours x 5 folds with P100. I extracted 2D-CNN's feature, the output of the global average pooling layer, and used it as input for stage 2. Feature extraction takes 2. 5 hours x 5 folds.

\n

Stage 2: 1D-CNN for exam-level classification

\n

Next, I trained 1D-CNN for exam-level classification. As input, I used feature sequences extracted by 1st-stage 2D-CNN. Feature pooling, Skip connection, and SE-module are employed. For image-level prediction, I used U-Net-like upconv architectures. The training conditions are shown below.

\n\n

Postprocessing

\n

I averaged the predictions of B0 and B2. I did postprocessing to solve the conflict of label consistency with minimal modification. The 5-fold CV score is shown below.

\n
negative_exam_for_pe           bce: 0.345601, auc: 0.895471\nindeterminate                  bce: 0.084615, auc: 0.812207\nchronic_pe                     bce: 0.158846, auc: 0.682470\nacute_and_chronic_pe           bce: 0.081380, auc: 0.842780\ncentral_pe                     bce: 0.111196, auc: 0.949153\nleftsided_pe                   bce: 0.287363, auc: 0.900652\nrightsided_pe                  bce: 0.298690, auc: 0.911001\nrv_lv_ratio_gte_1              bce: 0.228942, auc: 0.902246\nrv_lv_ratio_lt_1               bce: 0.342175, auc: 0.835369\nexam-level score               bce: 0.196321\nq_i weighted_image_bce         bce: 0.206749, auc: 0.965775\ntotal_score                    bce: 0.201473\n
\n

Final submission

\n

As final submissions, I selected

\n
    \n
  1. B0 model with 5-fold averaging: public: 0.161, private: 0.157
    2. \n
  2. B0 and B2 model with 5-fold averaging and model averaging: public: 0.160, private: 0.156
    3. \n
\n

Comparison test

\n

Stage 1

\n
                 exam BCE   exam AUC   image BCE   image AUC\nB0 final model   0.314768   0.709304   0.101278    0.951922\n num step X2     0.316794   0.708131   0.100873    0.951224\n initial LR=1e-4 0.321354   0.698805   0.113893    0.938055\n BatchSize=16    0.320763   0.687397   0.101027    0.950439\n w/o CutOut      0.316906   0.703339   0.100654    0.950882\n InputSize=256   0.328765   0.681211   0.134463    0.917054\n w/o WSO         0.315638   0.703576   0.103514    0.947023\n with MixUp      0.317532   0.693032   0.103800    0.944957\n InputSize=640   0.321643   0.703420   0.108685    0.953675\nB2 final model   0.315517   0.707378   0.099449    0.950943\n
\n

I did comparison tests after the deadline. scores are calculated without any weights. 0.5 epoch is enough to converge. InputSize=512 is better than InputSize=256. InputSize=640 may be better than InputSize=512. LR=1e-3 is better than LR=1e-4. BatchSize=80 may be better than BatchSize=16. Cutout, WSO, and MixUp may not be necessary.

\n

Stage 2

\n
                 exam BCE   exam AUC   image BCE   image AUC\nB0 final model   0.200486   0.867666   0.222240    0.964814\n w/o pred-2      0.200840   0.867577   0.221665    0.963847\n LSTM            0.203616   0.864214   0.216788    0.963465\n GRU             0.195599   0.873768   0.219781    0.963367\nB2 final model   0.196107   0.871995   0.220244    0.962613\n
\n

Scores are calculated with weights of competition metrics. Training with pred-2 may not be necessary. LSTM and GRU may be able to achieve the same performance as CNN.

\n

What I had to do in this competition was obvious because this competition is very similar to the last year's one and Yuval, the last year's winner, was at the top of the LB. Actually, What I did was just implementing the last year's solution and making a custom loss that minimizes the competition metric directly. It is a very baseline model. I think my solution would be in the middle of silver medals in usual competitions. This competition is a little bit harder than usual competitions, because of a large dataset, complicated metric, short span, and notebook competition. Maybe that's why this baseline model can get gold.

\n

All the codes are available here.

", "rawMarkdown": "Congratulations to all the winners. Thank you to Kaggle and RSNA for hosting this competition.\nI'm happy about my result because I'm a medical doctor and therefore, RSNA competitions are the most important competitions for me. In RSNA 2018, I did my best but I failed to get gold. I couldn't participate in RSNA 2019 because I had to prepare for the national exam for medical doctors. In RSNA 2020, I've got the solo gold finally. I'm so sad that I can't attend the RSNA conference in place because it becomes an online conference.\n\nHere I describe my solution. The overview is shown below. Actually, there is nothing special; 2D-CNN for image-level feature extraction and 1D-CNN for exam-level classification.\n\n![Figure 1](https://storage.googleapis.com/kaggle-forum-message-attachments/1072220/17414/figure1.png)\n# Stage 1: 2D-CNN for feature extraction\nFirst, I trained 2D-CNN (EfficientNet B0 or B2) with trainable 3 windows (WSO, [following Yuval's solution](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117480)) on 2D images with the condition shown below. To save memory and time, I trained models with mixed-precision.\n- Loss: BCE with the weights reflecting the competitions matric weights\n- Window: initialized with [Ian's windows](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930)\n- Image size: 512 (law image)\n- BatchSize: 80 for B0/50 for B2\n- Training Steps: 8192 (= 0.5 epoch)\n- Optimizer: Adam\n- LR: 1e-3 decline to 1e-4 with cosine annealing\n- Augmentation: ShiftScaleRotate, BrightnessContrast, Crop (448 x 448), CutOut\n\nTraining takes 3 hours x 5 folds with P100. I extracted 2D-CNN's feature, the output of the global average pooling layer, and used it as input for stage 2. Feature extraction takes 2. 5 hours x 5 folds.\n\n# Stage 2: 1D-CNN for exam-level classification\nNext, I trained 1D-CNN for exam-level classification. As input, I used feature sequences extracted by 1st-stage 2D-CNN. Feature pooling, Skip connection, and SE-module are employed. For image-level prediction, I used U-Net-like upconv architectures. The training conditions are shown below.\n- Exam-level loss: BCE with the weights reflecting the competitions metric weights\n- Image-level loss: BCE and BCE with the q_i weights\n- BatchSize: 64\n- Epoch: 16\n- Optimizer: Adam\n- LR: 1e-4 decline to 1e-5 with cosine annealing\n- Augmentation: Crop (128 slices), Flip\nTraining takes 6 minutes x 5 fold.\n\n# Postprocessing\nI averaged the predictions of B0 and B2. I did postprocessing to solve the conflict of label consistency with minimal modification. The 5-fold CV score is shown below.\n```\nnegative_exam_for_pe bce: 0.345601, auc: 0.895471\nindeterminate bce: 0.084615, auc: 0.812207\nchronic_pe bce: 0.158846, auc: 0.682470\nacute_and_chronic_pe bce: 0.081380, auc: 0.842780\ncentral_pe bce: 0.111196, auc: 0.949153\nleftsided_pe bce: 0.287363, auc: 0.900652\nrightsided_pe bce: 0.298690, auc: 0.911001\nrv_lv_ratio_gte_1 bce: 0.228942, auc: 0.902246\nrv_lv_ratio_lt_1 bce: 0.342175, auc: 0.835369\nexam-level score bce: 0.196321\nq_i weighted_image_bce bce: 0.206749, auc: 0.965775\ntotal_score bce: 0.201473\n```\n# Final submission\nAs final submissions, I selected\n1. B0 model with 5-fold averaging: public: 0.161, private: 0.157\n2. B0 and B2 model with 5-fold averaging and model averaging: public: 0.160, private: 0.156\n\n# Comparison test\nStage 1\n```\n exam BCE exam AUC image BCE image AUC\nB0 final model 0.314768 0.709304 0.101278 0.951922\n num step X2 0.316794 0.708131 0.100873 0.951224\n initial LR=1e-4 0.321354 0.698805 0.113893 0.938055\n BatchSize=16 0.320763 0.687397 0.101027 0.950439\n w/o CutOut 0.316906 0.703339 0.100654 0.950882\n InputSize=256 0.328765 0.681211 0.134463 0.917054\n w/o WSO 0.315638 0.703576 0.103514 0.947023\n with MixUp 0.317532 0.693032 0.103800 0.944957\n InputSize=640 0.321643 0.703420 0.108685 0.953675\nB2 final model 0.315517 0.707378 0.099449 0.950943\n```\nI did comparison tests after the deadline. scores are calculated without any weights. 0.5 epoch is enough to converge. InputSize=512 is better than InputSize=256. InputSize=640 may be better than InputSize=512. LR=1e-3 is better than LR=1e-4. BatchSize=80 may be better than BatchSize=16. Cutout, WSO, and MixUp may not be necessary.\n\n\nStage 2\n```\n exam BCE exam AUC image BCE image AUC\nB0 final model 0.200486 0.867666 0.222240 0.964814\n w/o pred-2 0.200840 0.867577 0.221665 0.963847\n LSTM 0.203616 0.864214 0.216788 0.963465\n GRU 0.195599 0.873768 0.219781 0.963367\nB2 final model 0.196107 0.871995 0.220244 0.962613\n```\nScores are calculated with weights of competition metrics. Training with pred-2 may not be necessary. LSTM and GRU may be able to achieve the same performance as CNN. \n\n\nWhat I had to do in this competition was obvious because this competition is very similar to the last year's one and Yuval, the last year's winner, was at the top of the LB. Actually, What I did was just implementing the last year's solution and making a custom loss that minimizes the competition metric directly. It is a very baseline model. I think my solution would be in the middle of silver medals in usual competitions. This competition is a little bit harder than usual competitions, because of a large dataset, complicated metric, short span, and notebook competition. Maybe that's why this baseline model can get gold.\n\n[All the codes are available here.](https://github.com/OsciiArt/Kaggle_RSNA2020_6th_Solution)", "votes": 21 }, { "id": 1072342, "postDate": "2020-11-08T05:28:22.930Z", "content": "

@osciiart Congratulations Doctor for bagging SOLO GOLD in RSNA 2020.

", "rawMarkdown": "@osciiart Congratulations Doctor for bagging **SOLO GOLD** in RSNA 2020. ", "votes": 2 }, { "id": 1073466, "postDate": "2020-11-09T14:45:52.030Z", "content": "

Congratulations ! 👍👍👍

\n

One question : have you made something to avoid impossible outputs like pe-present and negative-for-pe in the training process ?

", "rawMarkdown": "Congratulations ! 👍👍👍\n\nOne question : have you made something to avoid impossible outputs like pe-present and negative-for-pe in the training process ?", "replies": [ { "id": 1075335, "postDate": "2020-11-11T15:46:52.503Z", "content": "

I did rule-based postprocessing to solve the conflicts. The 14th cell in https://github.com/OsciiArt/Kaggle_RSNA2020_6th_Solution/blob/main/notebook/postprocess.ipynb

", "rawMarkdown": "I did rule-based postprocessing to solve the conflicts. The 14th cell in https://github.com/OsciiArt/Kaggle_RSNA2020_6th_Solution/blob/main/notebook/postprocess.ipynb ", "votes": 1 } ] } ], "comments": [ { "id": 1072342, "author_name": "Solve for Fun", "author_url": "", "post_date": "2020-11-08T05:28:22.930000", "content": "

@osciiart Congratulations Doctor for bagging SOLO GOLD in RSNA 2020.

", "votes": 2, "replies": [] }, { "id": 1073466, "author_name": "Tanguy Perennec", "author_url": "", "post_date": "2020-11-09T14:45:52.030000", "content": "

Congratulations ! 👍👍👍

\n

One question : have you made something to avoid impossible outputs like pe-present and negative-for-pe in the training process ?

", "votes": 0, "replies": [ { "id": 1075335, "author_name": "OsciiArt", "author_url": "", "post_date": "2020-11-11T15:46:52.503000", "content": "

I did rule-based postprocessing to solve the conflicts. The 14th cell in https://github.com/OsciiArt/Kaggle_RSNA2020_6th_Solution/blob/main/notebook/postprocess.ipynb

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1072220": "Congratulations to all the winners. Thank you to Kaggle and RSNA for hosting this competition.\nI'm happy about my result because I'm a medical doctor and therefore, RSNA competitions are the most important competitions for me. In RSNA 2018, I did my best but I failed to get gold. I couldn't participate in RSNA 2019 because I had to prepare for the national exam for medical doctors. In RSNA 2020, I've got the solo gold finally. I'm so sad that I can't attend the RSNA conference in place because it becomes an online conference.\n\nHere I describe my solution. The overview is shown below. Actually, there is nothing special; 2D-CNN for image-level feature extraction and 1D-CNN for exam-level classification.\n\n![Figure 1](https://storage.googleapis.com/kaggle-forum-message-attachments/1072220/17414/figure1.png)\n# Stage 1: 2D-CNN for feature extraction\nFirst, I trained 2D-CNN (EfficientNet B0 or B2) with trainable 3 windows (WSO, [following Yuval's solution](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117480)) on 2D images with the condition shown below. To save memory and time, I trained models with mixed-precision.\n- Loss: BCE with the weights reflecting the competitions matric weights\n- Window: initialized with [Ian's windows](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930)\n- Image size: 512 (law image)\n- BatchSize: 80 for B0/50 for B2\n- Training Steps: 8192 (= 0.5 epoch)\n- Optimizer: Adam\n- LR: 1e-3 decline to 1e-4 with cosine annealing\n- Augmentation: ShiftScaleRotate, BrightnessContrast, Crop (448 x 448), CutOut\n\nTraining takes 3 hours x 5 folds with P100. I extracted 2D-CNN's feature, the output of the global average pooling layer, and used it as input for stage 2. Feature extraction takes 2. 5 hours x 5 folds.\n\n# Stage 2: 1D-CNN for exam-level classification\nNext, I trained 1D-CNN for exam-level classification. As input, I used feature sequences extracted by 1st-stage 2D-CNN. Feature pooling, Skip connection, and SE-module are employed. For image-level prediction, I used U-Net-like upconv architectures. The training conditions are shown below.\n- Exam-level loss: BCE with the weights reflecting the competitions metric weights\n- Image-level loss: BCE and BCE with the q_i weights\n- BatchSize: 64\n- Epoch: 16\n- Optimizer: Adam\n- LR: 1e-4 decline to 1e-5 with cosine annealing\n- Augmentation: Crop (128 slices), Flip\nTraining takes 6 minutes x 5 fold.\n\n# Postprocessing\nI averaged the predictions of B0 and B2. I did postprocessing to solve the conflict of label consistency with minimal modification. The 5-fold CV score is shown below.\n```\nnegative_exam_for_pe bce: 0.345601, auc: 0.895471\nindeterminate bce: 0.084615, auc: 0.812207\nchronic_pe bce: 0.158846, auc: 0.682470\nacute_and_chronic_pe bce: 0.081380, auc: 0.842780\ncentral_pe bce: 0.111196, auc: 0.949153\nleftsided_pe bce: 0.287363, auc: 0.900652\nrightsided_pe bce: 0.298690, auc: 0.911001\nrv_lv_ratio_gte_1 bce: 0.228942, auc: 0.902246\nrv_lv_ratio_lt_1 bce: 0.342175, auc: 0.835369\nexam-level score bce: 0.196321\nq_i weighted_image_bce bce: 0.206749, auc: 0.965775\ntotal_score bce: 0.201473\n```\n# Final submission\nAs final submissions, I selected\n1. B0 model with 5-fold averaging: public: 0.161, private: 0.157\n2. B0 and B2 model with 5-fold averaging and model averaging: public: 0.160, private: 0.156\n\n# Comparison test\nStage 1\n```\n exam BCE exam AUC image BCE image AUC\nB0 final model 0.314768 0.709304 0.101278 0.951922\n num step X2 0.316794 0.708131 0.100873 0.951224\n initial LR=1e-4 0.321354 0.698805 0.113893 0.938055\n BatchSize=16 0.320763 0.687397 0.101027 0.950439\n w/o CutOut 0.316906 0.703339 0.100654 0.950882\n InputSize=256 0.328765 0.681211 0.134463 0.917054\n w/o WSO 0.315638 0.703576 0.103514 0.947023\n with MixUp 0.317532 0.693032 0.103800 0.944957\n InputSize=640 0.321643 0.703420 0.108685 0.953675\nB2 final model 0.315517 0.707378 0.099449 0.950943\n```\nI did comparison tests after the deadline. scores are calculated without any weights. 0.5 epoch is enough to converge. InputSize=512 is better than InputSize=256. InputSize=640 may be better than InputSize=512. LR=1e-3 is better than LR=1e-4. BatchSize=80 may be better than BatchSize=16. Cutout, WSO, and MixUp may not be necessary.\n\n\nStage 2\n```\n exam BCE exam AUC image BCE image AUC\nB0 final model 0.200486 0.867666 0.222240 0.964814\n w/o pred-2 0.200840 0.867577 0.221665 0.963847\n LSTM 0.203616 0.864214 0.216788 0.963465\n GRU 0.195599 0.873768 0.219781 0.963367\nB2 final model 0.196107 0.871995 0.220244 0.962613\n```\nScores are calculated with weights of competition metrics. Training with pred-2 may not be necessary. LSTM and GRU may be able to achieve the same performance as CNN. \n\n\nWhat I had to do in this competition was obvious because this competition is very similar to the last year's one and Yuval, the last year's winner, was at the top of the LB. Actually, What I did was just implementing the last year's solution and making a custom loss that minimizes the competition metric directly. It is a very baseline model. I think my solution would be in the middle of silver medals in usual competitions. This competition is a little bit harder than usual competitions, because of a large dataset, complicated metric, short span, and notebook competition. Maybe that's why this baseline model can get gold.\n\n[All the codes are available here.](https://github.com/OsciiArt/Kaggle_RSNA2020_6th_Solution)", "1072342": "@osciiart Congratulations Doctor for bagging **SOLO GOLD** in RSNA 2020. ", "1073466": "Congratulations ! 👍👍👍\n\nOne question : have you made something to avoid impossible outputs like pe-present and negative-for-pe in the training process ?" } }