![\"\"](\"https://i.ibb.co/6WtqnNv/batch-images-ep7.jpg\")
First of all congratulations to all participants. Congratulations to dream teams from gold zone. I’m impressed by your consistency in winning Kaggle competition. Waiting to learn from your solution.
\nThank you my team mate Andrij @aikhmelnytskyy We had great collaboration 👍👍👍 - I feel that from first minute we played in one team having one goal - find better solution.
\nGold in competition was dream for me. Last year we (with @christofhenkel ) were #1 in sliver (#12 solution in Image Matching Challange 2021). This year I decided to work hard to experience gold zone and finally become competition master. Even there is no official LB finalized …. we are #9 and in gold! :) and I am …… extremely happy! 😁😁😍😜
\nThis competition was really great for testing many different computer vision techniques. Three months passed very quickly. The first phase of the competition was difficult. We performed a large number of different tests that did not gave us results higher than 0.3 (LB score). It was very frustrating. We were unable to find any correlation between local CV and LB. Then we setup good training pipeline – main our success point are:
\nLast two weeks of competition were hard to me – I caught covid and had to pause (during recovery time I coded using iPad). But we cooperate all the time and we finally managed to jump to TOP10 and 0.63 (public LB). Finally we were closing gold zone on #13 (public LB) and #9 in private LB!
\nModels score summary
\n• best public LB: 0.63 (ensemble) / 0.57 (single model) / local CV (0.48)
\n• private lb: 0.50 (max: 0.50)
Competition achievements
\n• new experience in Kaggle competition - a lot of good discussion
\n• 1x gold medal - dataset (I am very happy - my first one)
\n• 2x gold medals - notebook
\n• 1x gold medal - competition-> extremely happy! 😁😁😍😜
Our final selection
\nWe selected two different solution which based on the same model setup.
\n• 3 models - ensemble average model prediction probabilities -> LB: 0.63 (PL: 0.47)
\n• 3 models - voting strategy and then score average (on votes score) -> LB: 0.62 (PL: 0.5)
\nAfter many tests we had strong feeling that our second choice (even score was lower on LB that many of our rest solution) is more stable (was less sensitive on th) than other solution. So we closed eye and trusted in our test rather then LB score.
Solution description in 4 steps
\nOur solution is very simple. We tried different ways to predict breast cancer but finally it appeared that simples solution wors for us the best (both local CV and LB).
Dataset
\n• Image resolution: 1536x768
\n• ROI cropped - cv2.connectedComponentsWithStats method (we started with yolov5 for prototyping phase but then we used cv2 - since licence regulations)
\n• 4-GroupFold: on patient_id
\n• image pixel scaling: div by 255.
\n• dicom2png – proposed by @hengck23 (https://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example)
\n• we do not use external dataset – all models were trained on competition data
\n• we processed data in two steps (to avoid problem with file storage capacity – during the tests we exported images to different resolutions – max 2400px):
fast and reliable crop roi function we used during competition (cretits to @vslaykovsky : \"RSNA: Cut Off Empty Space from Images\")
\ndef crop_roi(img, photometric_interpretation):\n # it can be improved \n Y = img\n xmin = Y.min()\n xmax = Y.max()\n\n norm = np.empty_like(Y, dtype=np.uint8)\n\n dicomsdl.util.convert_to_uint8(Y, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n\n X = norm\n X = X[5:-5, 5:-5]\n\n output= cv2.connectedComponentsWithStats((X > 10).astype(np.uint8)[:, :], 8, cv2.CV_32S) #\n stats = output[2]\n\n idx = stats[1:, 4].argmax() + 1\n x1, y1, w, h = stats[idx][:4]\n x2 = x1 + w\n y2 = y1 + h\n\n X_out = Y[y1: y2, x1: x2]\n\n return X_out\nAugumentation
\n• Albumentations (we tried Kornia and works great but we need more time to rewrite functions we used in competition).
transformation = [\n A.OneOf([\n A.RandomBrightnessContrast(always_apply=False, p=.5, brightness_limit=(-1, 1.0), contrast_limit=(-1, 1.0), brightness_by_max=True),\n A.RandomGamma(always_apply=False, p=.5, gamma_limit=(60, 120), eps=None),\n ], p = 0.5),\n\n A.Rotate(limit=5, p=0.5),\n A.Affine(rotate = 5, translate_percent=0.1, scale=[0.9,1.5], shear=0, p=0.5),\n\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.Resize(im_size[0], im_size[1]),\n A.ShiftScaleRotate(always_apply=False, p=.2,\n shift_limit_x=(-1.0, 1.0),\n shift_limit_y=(-1.0, 1.0),\n scale_limit=(-0.1, 0.1),\n rotate_limit=(-5, 5),\n interpolation=0,\n border_mode=3,\n value=(0, 0, 0),\n mask_value=None,\n rotate_method='largest_box'),\n\n A.OneOf([\n GridDropoutv2(always_apply=False, p=.2, ratio = .25, unit_size_min = 100, unit_size_max = 400, holes_number_x=100, holes_number_y=100),\n CoarseDropoutv2(always_apply=False, p=.2, max_holes=12, max_height=250, max_width=100, min_holes=3, min_height=50, min_width=50, mask_fill_value=None)], p=0.25),\n ToTensorV2()\n ]\nGridDropoutv2 and CoarseDropoutv2 is our modification of Albumentations function. It generates cuts in random greyscale.
\n
Training
\n• Framework: Pytorch scripts (multi gpu support: DDP)
\n• Optimizer: RAdam
\n• Lookahead: on
\n• Scheduler: OneCycle - no warm up
\n• Weight decay: 1e-2
\n• Dataset each epoch was seeded by different seed.
\n• Batch size: 16
\n• Mixed precision (AMP): on
\n• Gradient clipping: on
\n• Loss function: BCEWithLogitsLoss with pos_weight = 1.0 - 1.25
\n• Epochs: 7 (best models are from 4-5 epochs) - we decided to take early stopping (best probf1score) models instead of the last one.
\n• Sampler (thanks to @hengck23): custom sequential sampler (sequence -> positive sample / negative sample = 8) – small changes compare to the original one.
class Balancer(torch.utils.data.Sampler):\n\n def __init__(self, pos_cases, neg_cases, ratio = 3):\n self.r = ratio - 1\n self.pos_index = pos_cases\n self.neg_index = neg_cases\n\n self.length = self.r * int(np.floor(len(self.neg_index)/self.r))\n self.ds_len = self.length + (self.length // self.r)\n\n def __iter__(self):\n pos_index = self.pos_index\n neg_index = self.neg_index\n\n np.random.shuffle(pos_index)\n np.random.shuffle(neg_index)\n\n neg_index = neg_index[:self.length].reshape(-1,self.r)\n #pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\n pos_index_len = len(pos_index)\n\n pos_index = np.tile(pos_index, ((len(neg_index) // pos_index_len) + 1, 1))\n pos_index = np.apply_along_axis(np.random.permutation, 1, pos_index)\n pos_index = pos_index.reshape(-1,1)[:len(neg_index)]\n\n index = np.concatenate([pos_index,neg_index], -1).reshape(-1)\n return iter(index)\n\n def __len__(self):\n return self.ds_len\nModels
\nConvNext_v1 small (timm - checkpoint: convnext_small.fb_in22k_ft_in1k_384). We tried different architectures but this one works in our solution the best.
\n• avg pooling or
\n• GEM pooling (trainable parameters set to True)
\n• drop_path_rate=0.2 and drop_rate=0.05 (we tested different settings but these works the best for our setup)
\n• input size: (1536, 768), 3 channels (b&w images)
Local validation
\n• Metrics: probf1, ROC_AUC, prec/recall and MCC (Matthews’s correlation coefficient)
\n• For local validation we used tool provided by @hengck23 - https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521
\n• For each traing epoch we additionaly plot: prediction dynamic, confusion matrix.
\n• We looked into prediction and performed visual analysis of missed or incorrectly classified samples.
Inference
\n• Three models – ensemble (voting or model probabilities averaging):
Ensembling
\n• Model probabilities ensembling using better then median function
def better_than_median(inputs, axis):\n \"\"\"Compute the mean of the predictions if there are no outliers,\n or the median if there are outliers.\n\n Parameter: inputs = ndarray of shape (n_samples, n_folds)\"\"\"\n spread = inputs.max(axis=axis) - inputs.min(axis=axis) \n spread_lim = 0.6\n print(f\"Inliers: {(spread > spread_lim).sum():7} -> compute mean\")\n print(f\"Outliers: {(spread <= spread_lim).sum():7} -> compute median\")\n print(f\"Total: {len(inputs):7}\")\n return np.where(spread > spread_lim,\n np.mean(inputs, axis=axis),\n np.median(inputs, axis=axis))\n• Voting and then averaging
\ndef voting_confidence(x):\n if x<=0.25:\n return 0.0\n elif x<=0.45:\n return 0.35\n elif x<=0.55:\n return x\n elif x<=0.75:\n return 0.65\n else:\n return 1.0\nCPU/GPU
\n• Training: 1-2xA6000 or A100 GPU-80GB on Cloud
Experiment tracking
\n• Weights & Biases
Things we tested during competition and did not improve our score (probably more tests are needed)
\n• Full dataset training, site_1/site_2 separate models, up-sampling (by sample frac), down-sampling.
\n• Stochastic Weight Averaging and Exponential Moving Average
\n• Optimizers – AdamW, SGD, NAdam, Lion, Lamb
\n• Training with layer freeze (different levels)
\n• Loss function: Focal loss, LDAM, LMFLoss, Label smoothing
\n• Models: Effnet, NextVit (simmilar score but harder to train and slower), DenseNet, Inception_v3, maxvit
\n• Effnet + pixel-wise self attention, Effnet + cross attention
\n• Convnextv1 + max and avg concatenated pooling
\n• Synthetic dataset: synthetic breast cancer generator (experienced radiologist is required to evaluate solution).
\n• PatchGD - https://arxiv.org/pdf/2301.13817.pdf
\n• Training attitudes - Pair training – MLO / CC – patient_id and side (L/R)
\n• Metamodel - SVM and XGBoost on image enbeddings.
\n• Pseudolabeling - using new samples during submission time.
\n• Selecting best samples for cases where patient has more then 2 views
sub_s1 = sub_s1[['prediction_id', 'site_id', 'patient_id', 'laterality', 'cancer']].\\\n groupby(['patient_id','laterality']).\\\n apply(lambda x: x.nlargest(3,'cancer')).reset_index(drop=True)\nSpecial thanks to
\n• @Andrij - for great cooperation during the competition. For very good and substantive talks aimed at solving the problem and improving results.
\n• @hengck23 - Thanks for the great activity on the forum and sharing knowledge. I implemented many of your ideas, many of them improved our score. Thank you!
Source code will be available soon (in week) on [Remek github](https://github.com/rkinas
", "messages": [ { "id": 2162073, "postDate": "2023-02-28T01:10:34.900Z", "content": "First of all congratulations to all participants. Congratulations to dream teams from gold zone. I’m impressed by your consistency in winning Kaggle competition. Waiting to learn from your solution.
\nThank you my team mate Andrij @aikhmelnytskyy We had great collaboration 👍👍👍 - I feel that from first minute we played in one team having one goal - find better solution.
\nGold in competition was dream for me. Last year we (with @christofhenkel ) were #1 in sliver (#12 solution in Image Matching Challange 2021). This year I decided to work hard to experience gold zone and finally become competition master. Even there is no official LB finalized …. we are #9 and in gold! :) and I am …… extremely happy! 😁😁😍😜
\nThis competition was really great for testing many different computer vision techniques. Three months passed very quickly. The first phase of the competition was difficult. We performed a large number of different tests that did not gave us results higher than 0.3 (LB score). It was very frustrating. We were unable to find any correlation between local CV and LB. Then we setup good training pipeline – main our success point are:
\nLast two weeks of competition were hard to me – I caught covid and had to pause (during recovery time I coded using iPad). But we cooperate all the time and we finally managed to jump to TOP10 and 0.63 (public LB). Finally we were closing gold zone on #13 (public LB) and #9 in private LB!
\nModels score summary
\n• best public LB: 0.63 (ensemble) / 0.57 (single model) / local CV (0.48)
\n• private lb: 0.50 (max: 0.50)
Competition achievements
\n• new experience in Kaggle competition - a lot of good discussion
\n• 1x gold medal - dataset (I am very happy - my first one)
\n• 2x gold medals - notebook
\n• 1x gold medal - competition-> extremely happy! 😁😁😍😜
Our final selection
\nWe selected two different solution which based on the same model setup.
\n• 3 models - ensemble average model prediction probabilities -> LB: 0.63 (PL: 0.47)
\n• 3 models - voting strategy and then score average (on votes score) -> LB: 0.62 (PL: 0.5)
\nAfter many tests we had strong feeling that our second choice (even score was lower on LB that many of our rest solution) is more stable (was less sensitive on th) than other solution. So we closed eye and trusted in our test rather then LB score.
Solution description in 4 steps
\nOur solution is very simple. We tried different ways to predict breast cancer but finally it appeared that simples solution wors for us the best (both local CV and LB).
Dataset
\n• Image resolution: 1536x768
\n• ROI cropped - cv2.connectedComponentsWithStats method (we started with yolov5 for prototyping phase but then we used cv2 - since licence regulations)
\n• 4-GroupFold: on patient_id
\n• image pixel scaling: div by 255.
\n• dicom2png – proposed by @hengck23 (https://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example)
\n• we do not use external dataset – all models were trained on competition data
\n• we processed data in two steps (to avoid problem with file storage capacity – during the tests we exported images to different resolutions – max 2400px):
fast and reliable crop roi function we used during competition (cretits to @vslaykovsky : \"RSNA: Cut Off Empty Space from Images\")
\ndef crop_roi(img, photometric_interpretation):\n # it can be improved \n Y = img\n xmin = Y.min()\n xmax = Y.max()\n\n norm = np.empty_like(Y, dtype=np.uint8)\n\n dicomsdl.util.convert_to_uint8(Y, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n\n X = norm\n X = X[5:-5, 5:-5]\n\n output= cv2.connectedComponentsWithStats((X > 10).astype(np.uint8)[:, :], 8, cv2.CV_32S) #\n stats = output[2]\n\n idx = stats[1:, 4].argmax() + 1\n x1, y1, w, h = stats[idx][:4]\n x2 = x1 + w\n y2 = y1 + h\n\n X_out = Y[y1: y2, x1: x2]\n\n return X_out\nAugumentation
\n• Albumentations (we tried Kornia and works great but we need more time to rewrite functions we used in competition).
transformation = [\n A.OneOf([\n A.RandomBrightnessContrast(always_apply=False, p=.5, brightness_limit=(-1, 1.0), contrast_limit=(-1, 1.0), brightness_by_max=True),\n A.RandomGamma(always_apply=False, p=.5, gamma_limit=(60, 120), eps=None),\n ], p = 0.5),\n\n A.Rotate(limit=5, p=0.5),\n A.Affine(rotate = 5, translate_percent=0.1, scale=[0.9,1.5], shear=0, p=0.5),\n\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.Resize(im_size[0], im_size[1]),\n A.ShiftScaleRotate(always_apply=False, p=.2,\n shift_limit_x=(-1.0, 1.0),\n shift_limit_y=(-1.0, 1.0),\n scale_limit=(-0.1, 0.1),\n rotate_limit=(-5, 5),\n interpolation=0,\n border_mode=3,\n value=(0, 0, 0),\n mask_value=None,\n rotate_method='largest_box'),\n\n A.OneOf([\n GridDropoutv2(always_apply=False, p=.2, ratio = .25, unit_size_min = 100, unit_size_max = 400, holes_number_x=100, holes_number_y=100),\n CoarseDropoutv2(always_apply=False, p=.2, max_holes=12, max_height=250, max_width=100, min_holes=3, min_height=50, min_width=50, mask_fill_value=None)], p=0.25),\n ToTensorV2()\n ]\nGridDropoutv2 and CoarseDropoutv2 is our modification of Albumentations function. It generates cuts in random greyscale.
\n
Training
\n• Framework: Pytorch scripts (multi gpu support: DDP)
\n• Optimizer: RAdam
\n• Lookahead: on
\n• Scheduler: OneCycle - no warm up
\n• Weight decay: 1e-2
\n• Dataset each epoch was seeded by different seed.
\n• Batch size: 16
\n• Mixed precision (AMP): on
\n• Gradient clipping: on
\n• Loss function: BCEWithLogitsLoss with pos_weight = 1.0 - 1.25
\n• Epochs: 7 (best models are from 4-5 epochs) - we decided to take early stopping (best probf1score) models instead of the last one.
\n• Sampler (thanks to @hengck23): custom sequential sampler (sequence -> positive sample / negative sample = 8) – small changes compare to the original one.
class Balancer(torch.utils.data.Sampler):\n\n def __init__(self, pos_cases, neg_cases, ratio = 3):\n self.r = ratio - 1\n self.pos_index = pos_cases\n self.neg_index = neg_cases\n\n self.length = self.r * int(np.floor(len(self.neg_index)/self.r))\n self.ds_len = self.length + (self.length // self.r)\n\n def __iter__(self):\n pos_index = self.pos_index\n neg_index = self.neg_index\n\n np.random.shuffle(pos_index)\n np.random.shuffle(neg_index)\n\n neg_index = neg_index[:self.length].reshape(-1,self.r)\n #pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\n pos_index_len = len(pos_index)\n\n pos_index = np.tile(pos_index, ((len(neg_index) // pos_index_len) + 1, 1))\n pos_index = np.apply_along_axis(np.random.permutation, 1, pos_index)\n pos_index = pos_index.reshape(-1,1)[:len(neg_index)]\n\n index = np.concatenate([pos_index,neg_index], -1).reshape(-1)\n return iter(index)\n\n def __len__(self):\n return self.ds_len\nModels
\nConvNext_v1 small (timm - checkpoint: convnext_small.fb_in22k_ft_in1k_384). We tried different architectures but this one works in our solution the best.
\n• avg pooling or
\n• GEM pooling (trainable parameters set to True)
\n• drop_path_rate=0.2 and drop_rate=0.05 (we tested different settings but these works the best for our setup)
\n• input size: (1536, 768), 3 channels (b&w images)
Local validation
\n• Metrics: probf1, ROC_AUC, prec/recall and MCC (Matthews’s correlation coefficient)
\n• For local validation we used tool provided by @hengck23 - https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521
\n• For each traing epoch we additionaly plot: prediction dynamic, confusion matrix.
\n• We looked into prediction and performed visual analysis of missed or incorrectly classified samples.
Inference
\n• Three models – ensemble (voting or model probabilities averaging):
Ensembling
\n• Model probabilities ensembling using better then median function
def better_than_median(inputs, axis):\n \"\"\"Compute the mean of the predictions if there are no outliers,\n or the median if there are outliers.\n\n Parameter: inputs = ndarray of shape (n_samples, n_folds)\"\"\"\n spread = inputs.max(axis=axis) - inputs.min(axis=axis) \n spread_lim = 0.6\n print(f\"Inliers: {(spread > spread_lim).sum():7} -> compute mean\")\n print(f\"Outliers: {(spread <= spread_lim).sum():7} -> compute median\")\n print(f\"Total: {len(inputs):7}\")\n return np.where(spread > spread_lim,\n np.mean(inputs, axis=axis),\n np.median(inputs, axis=axis))\n• Voting and then averaging
\ndef voting_confidence(x):\n if x<=0.25:\n return 0.0\n elif x<=0.45:\n return 0.35\n elif x<=0.55:\n return x\n elif x<=0.75:\n return 0.65\n else:\n return 1.0\nCPU/GPU
\n• Training: 1-2xA6000 or A100 GPU-80GB on Cloud
Experiment tracking
\n• Weights & Biases
Things we tested during competition and did not improve our score (probably more tests are needed)
\n• Full dataset training, site_1/site_2 separate models, up-sampling (by sample frac), down-sampling.
\n• Stochastic Weight Averaging and Exponential Moving Average
\n• Optimizers – AdamW, SGD, NAdam, Lion, Lamb
\n• Training with layer freeze (different levels)
\n• Loss function: Focal loss, LDAM, LMFLoss, Label smoothing
\n• Models: Effnet, NextVit (simmilar score but harder to train and slower), DenseNet, Inception_v3, maxvit
\n• Effnet + pixel-wise self attention, Effnet + cross attention
\n• Convnextv1 + max and avg concatenated pooling
\n• Synthetic dataset: synthetic breast cancer generator (experienced radiologist is required to evaluate solution).
\n• PatchGD - https://arxiv.org/pdf/2301.13817.pdf
\n• Training attitudes - Pair training – MLO / CC – patient_id and side (L/R)
\n• Metamodel - SVM and XGBoost on image enbeddings.
\n• Pseudolabeling - using new samples during submission time.
\n• Selecting best samples for cases where patient has more then 2 views
sub_s1 = sub_s1[['prediction_id', 'site_id', 'patient_id', 'laterality', 'cancer']].\\\n groupby(['patient_id','laterality']).\\\n apply(lambda x: x.nlargest(3,'cancer')).reset_index(drop=True)\nSpecial thanks to
\n• @Andrij - for great cooperation during the competition. For very good and substantive talks aimed at solving the problem and improving results.
\n• @hengck23 - Thanks for the great activity on the forum and sharing knowledge. I implemented many of your ideas, many of them improved our score. Thank you!
Source code will be available soon (in week) on [Remek github](https://github.com/rkinas
", "rawMarkdown": "First of all congratulations to all participants. Congratulations to dream teams from gold zone. I’m impressed by your consistency in winning Kaggle competition. Waiting to learn from your solution.\n\nThank you my team mate Andrij @aikhmelnytskyy We had great collaboration 👍👍👍 - I feel that from first minute we played in one team having one goal - find better solution.\n\nGold in competition was dream for me. Last year we (with @christofhenkel ) were #1 in sliver (#12 solution in Image Matching Challange 2021). This year I decided to work hard to experience gold zone and finally become competition master. Even there is no official LB finalized .... we are #9 and in gold! :) and I am ...... extremely happy! 😁😁😍😜\n\nThis competition was really great for testing many different computer vision techniques. Three months passed very quickly. The first phase of the competition was difficult. We performed a large number of different tests that did not gave us results higher than 0.3 (LB score). It was very frustrating. We were unable to find any correlation between local CV and LB. Then we setup good training pipeline – main our success point are:\n-\tSampling strategy and positive class balancing\n-\tAugumentation\n-\tModel selection\n- Postprocessing\n\nLast two weeks of competition were hard to me – I caught covid and had to pause (during recovery time I coded using iPad). But we cooperate all the time and we finally managed to jump to TOP10 and 0.63 (public LB). Finally we were closing gold zone on #13 (public LB) and #9 in private LB!\n\n**Models score summary**\n•\tbest public LB: 0.63 (ensemble) / 0.57 (single model) / local CV (0.48)\n•\tprivate lb: 0.50 (max: 0.50)\n\n\n**Competition achievements**\n•\tnew experience in Kaggle competition - a lot of good discussion \n•\t1x gold medal - dataset (I am very happy - my first one)\n•\t2x gold medals - notebook \n•\t1x gold medal - competition-> extremely happy! 😁😁😍😜\n\n**Our final selection**\nWe selected two different solution which based on the same model setup.\n•\t3 models - ensemble average model prediction probabilities -> LB: 0.63 (PL: 0.47)\n•\t3 models - voting strategy and then score average (on votes score) -> LB: 0.62 (PL: 0.5)\nAfter many tests we had strong feeling that our second choice (even score was lower on LB that many of our rest solution) is more stable (was less sensitive on th) than other solution. So we closed eye and trusted in our test rather then LB score.\n\n**Solution description in 4 steps**\nOur solution is very simple. We tried different ways to predict breast cancer but finally it appeared that simples solution wors for us the best (both local CV and LB).\n\n1.\tProcess dicom files to png (windowing).\n2.\tInference – 3 convnext (v1) models with TTA\n3.\tEnsemble – probabilities average or voting\n4.\tThresholding – th ~0.5 -> final prediction result 0|1\n\n\n**Dataset**\n•\tImage resolution: 1536x768\n•\tROI cropped - cv2.connectedComponentsWithStats method (we started with yolov5 for prototyping phase but then we used cv2 - since licence regulations)\n•\t4-GroupFold: on patient_id\n•\timage pixel scaling: div by 255.\n•\tdicom2png – proposed by @hengck23 (https://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example)\n•\twe do not use external dataset – all models were trained on competition data\n•\twe processed data in two steps (to avoid problem with file storage capacity – during the tests we exported images to different resolutions – max 2400px):\n1. process j2k files format (dicom2png and crop to 1536xW - we do not resize it in this step to final resolution)->inference->delete files \n2. process nonj2k files->inference->delete files\n\nfast and reliable crop roi function we used during competition (cretits to @vslaykovsky : [\"RSNA: Cut Off Empty Space from Images\"](https://www.kaggle.com/code/vslaykovsky/rsna-cut-off-empty-space-from-images))\n```\ndef crop_roi(img, photometric_interpretation):\n # it can be improved \n Y = img\n xmin = Y.min()\n xmax = Y.max()\n\n norm = np.empty_like(Y, dtype=np.uint8)\n\n dicomsdl.util.convert_to_uint8(Y, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n \n X = norm\n X = X[5:-5, 5:-5]\n\n output= cv2.connectedComponentsWithStats((X > 10).astype(np.uint8)[:, :], 8, cv2.CV_32S) #\n stats = output[2]\n \n idx = stats[1:, 4].argmax() + 1\n x1, y1, w, h = stats[idx][:4]\n x2 = x1 + w\n y2 = y1 + h\n \n X_out = Y[y1: y2, x1: x2]\n \n return X_out\n```\n\n**Augumentation** \n•\tAlbumentations (we tried Kornia and works great but we need more time to rewrite functions we used in competition).\n\n```\ntransformation = [\n \tA.OneOf([\n \tA.RandomBrightnessContrast(always_apply=False, p=.5, brightness_limit=(-1, 1.0), contrast_limit=(-1, 1.0), brightness_by_max=True),\n \tA.RandomGamma(always_apply=False, p=.5, gamma_limit=(60, 120), eps=None),\n \t], p = 0.5),\n \t \n \tA.Rotate(limit=5, p=0.5),\n \tA.Affine(rotate = 5, translate_percent=0.1, scale=[0.9,1.5], shear=0, p=0.5),\n \t \n \tA.HorizontalFlip(p=0.5),\n \tA.VerticalFlip(p=0.5),\n \tA.Resize(im_size[0], im_size[1]),\n \tA.ShiftScaleRotate(always_apply=False, p=.2,\n \tshift_limit_x=(-1.0, 1.0),\n \tshift_limit_y=(-1.0, 1.0),\n \tscale_limit=(-0.1, 0.1),\n \trotate_limit=(-5, 5),\n \tinterpolation=0,\n \tborder_mode=3,\n \tvalue=(0, 0, 0),\n \tmask_value=None,\n \trotate_method='largest_box'),\n \t \n \tA.OneOf([\n \tGridDropoutv2(always_apply=False, p=.2, ratio = .25, unit_size_min = 100, unit_size_max = 400, holes_number_x=100, holes_number_y=100),\n \tCoarseDropoutv2(always_apply=False, p=.2, max_holes=12, max_height=250, max_width=100, min_holes=3, min_height=50, min_width=50, mask_fill_value=None)], p=0.25),\n \tToTensorV2()\n \t]\n```\n\nGridDropoutv2 and CoarseDropoutv2 is our modification of Albumentations function. It generates cuts in random greyscale.\n\n\n \n\n**Training**\n•\tFramework: Pytorch scripts (multi gpu support: DDP)\n•\tOptimizer: RAdam \n•\tLookahead: on\n•\tScheduler: OneCycle - no warm up\n•\tWeight decay: 1e-2\n•\tDataset each epoch was seeded by different seed.\n•\tBatch size: 16\n•\tMixed precision (AMP): on\n•\tGradient clipping: on\n•\tLoss function: BCEWithLogitsLoss with pos_weight = 1.0 - 1.25\n•\tEpochs: 7 (best models are from 4-5 epochs) - we decided to take early stopping (best probf1score) models instead of the last one.\n•\tSampler (thanks to @hengck23): custom sequential sampler (sequence -> positive sample / negative sample = 8) – small changes compare to the original one.\n\n```\nclass Balancer(torch.utils.data.Sampler):\n\n\tdef __init__(self, pos_cases, neg_cases, ratio = 3):\n \tself.r = ratio - 1\n \tself.pos_index = pos_cases\n \tself.neg_index = neg_cases\n\n \tself.length = self.r * int(np.floor(len(self.neg_index)/self.r))\n \tself.ds_len = self.length + (self.length // self.r)\n\n\tdef __iter__(self):\n \tpos_index = self.pos_index\n \tneg_index = self.neg_index\n \t \n \tnp.random.shuffle(pos_index)\n \tnp.random.shuffle(neg_index)\n\n \tneg_index = neg_index[:self.length].reshape(-1,self.r)\n \t#pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\n \tpos_index_len = len(pos_index)\n \t \n \tpos_index = np.tile(pos_index, ((len(neg_index) // pos_index_len) + 1, 1))\n \tpos_index = np.apply_along_axis(np.random.permutation, 1, pos_index)\n \tpos_index = pos_index.reshape(-1,1)[:len(neg_index)]\n\n \tindex = np.concatenate([pos_index,neg_index], -1).reshape(-1)\n \treturn iter(index)\n\n\tdef __len__(self):\n \treturn self.ds_len\n```\n \n**Models**\nConvNext_v1 small (timm - checkpoint: convnext_small.fb_in22k_ft_in1k_384). We tried different architectures but this one works in our solution the best.\n•\tavg pooling or \n•\tGEM pooling (trainable parameters set to True)\n•\tdrop_path_rate=0.2 and drop_rate=0.05 (we tested different settings but these works the best for our setup)\n•\tinput size: (1536, 768), 3 channels (b&w images)\n\n**Local validation**\n•\tMetrics: probf1, ROC_AUC, prec/recall and MCC (Matthews’s correlation coefficient)\n•\tFor local validation we used tool provided by @hengck23 - https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521\n•\tFor each traing epoch we additionaly plot: prediction dynamic, confusion matrix.\n•\tWe looked into prediction and performed visual analysis of missed or incorrectly classified samples. \n\n**Inference**\n•\tThree models – ensemble (voting or model probabilities averaging):\n1. avg pooling - fold 1 (trained on 1536 x 768) -> probf1: 0.52 (local CV) / 0.57 (LB)\n2. GEM pooling - fold 2 (trained on 1536 x 768) -> probf1: 0.44 (local CV) (we do not test it on LB)\n3. GEM pooling - fold 3 (trained on 1536 x 768) -> probf1: 0.46 (local CV) (-)\n•\tTTA: h_flip only (TTA weighted - 0.6 original image + 0.4 flipped image) - > it inceased our score by 0.03\n\n\n**Ensembling**\n•\tModel probabilities ensembling using better then median function\n\n```\ndef better_than_median(inputs, axis):\n \"\"\"Compute the mean of the predictions if there are no outliers,\n or the median if there are outliers.\n\n Parameter: inputs = ndarray of shape (n_samples, n_folds)\"\"\"\n spread = inputs.max(axis=axis) - inputs.min(axis=axis) \n spread_lim = 0.6\n print(f\"Inliers: {(spread > spread_lim).sum():7} -> compute mean\")\n print(f\"Outliers: {(spread <= spread_lim).sum():7} -> compute median\")\n print(f\"Total: {len(inputs):7}\")\n return np.where(spread > spread_lim,\n np.mean(inputs, axis=axis),\n np.median(inputs, axis=axis))\n```\n\n•\tVoting and then averaging\n\n```\ndef voting_confidence(x):\n if x<=0.25:\n return 0.0\n elif x<=0.45:\n return 0.35\n elif x<=0.55:\n return x\n elif x<=0.75:\n return 0.65\n else:\n return 1.0\n```\n\n\n**CPU/GPU**\n•\tTraining: 1-2xA6000 or A100 GPU-80GB on Cloud \n\n\n**Experiment tracking**\n•\tWeights & Biases \n\n**Things we tested during competition and did not improve our score (probably more tests are needed)**\n•\tFull dataset training, site_1/site_2 separate models, up-sampling (by sample frac), down-sampling.\n•\tStochastic Weight Averaging and Exponential Moving Average\n•\tOptimizers – AdamW, SGD, NAdam, Lion, Lamb\n•\tTraining with layer freeze (different levels)\n•\tLoss function: Focal loss, LDAM, LMFLoss, Label smoothing\n•\tModels: Effnet, NextVit (simmilar score but harder to train and slower), DenseNet, Inception_v3, maxvit\n•\tEffnet + pixel-wise self attention, Effnet + cross attention\n•\tConvnextv1 + max and avg concatenated pooling\n•\tSynthetic dataset: synthetic breast cancer generator (experienced radiologist is required to evaluate solution).\n•\tPatchGD - https://arxiv.org/pdf/2301.13817.pdf\n•\tTraining attitudes - Pair training – MLO / CC – patient_id and side (L/R) \n•\tMetamodel - SVM and XGBoost on image enbeddings.\n•\tPseudolabeling - using new samples during submission time.\n•\tSelecting best samples for cases where patient has more then 2 views \n\n```\nsub_s1 = sub_s1[['prediction_id', 'site_id', 'patient_id', 'laterality', 'cancer']].\\\n groupby(['patient_id','laterality']).\\\n apply(lambda x: x.nlargest(3,'cancer')).reset_index(drop=True)\n```\n\n**Special thanks to**\n•\t@Andrij - for great cooperation during the competition. For very good and substantive talks aimed at solving the problem and improving results.\n•\t@hengck23 - Thanks for the great activity on the forum and sharing knowledge. I implemented many of your ideas, many of them improved our score. Thank you!\n\nSource code will be available soon (in week) on [Remek github](https://github.com/rkinas", "votes": 89 }, { "id": 2162827, "postDate": "2023-02-28T12:31:55.093Z", "content": "Congrats @remekkinas and @aikhmelnytskyy ! Great solution, thanks for sharing.
", "rawMarkdown": "Congrats @remekkinas and @aikhmelnytskyy ! Great solution, thanks for sharing.", "votes": 3, "replies": [ { "id": 2162837, "postDate": "2023-02-28T12:42:36.267Z", "content": "@titericz I would like to thank you very much for your inspiration. This is true story. I was really devastated by this competition :) The more I wanted give up the harder I worked thanks to this message:
\n![\"\"](\"https://i.ibb.co/Bn4m9L1/twitt.jpg\")
Congrats Remek for your 1st gold medal in competition and master title!
", "rawMarkdown": "Congrats Remek for your 1st gold medal in competition and master title! ", "votes": 1, "replies": [ { "id": 2166436, "postDate": "2023-03-02T19:51:28.190Z", "content": "Thank you @michau96 - and as a bonus: our team did not use any pixel from external dataset 😁
", "rawMarkdown": "Thank you @michau96 - and as a bonus: our team did not use any pixel from external dataset 😁", "votes": 1 } ] }, { "id": 2165824, "postDate": "2023-03-02T13:33:38.910Z", "content": "Congratulation for being competition master!
", "rawMarkdown": "Congratulation for being competition master!", "votes": 1, "replies": [ { "id": 2166439, "postDate": "2023-03-02T19:53:05.540Z", "content": "@haqishen thank you very much. I am really, really happy. Still reading solution and working on some improvements. Still learning and have fun during competition. See you soon.
", "rawMarkdown": "@haqishen thank you very much. I am really, really happy. Still reading solution and working on some improvements. Still learning and have fun during competition. See you soon.", "votes": 2 } ] }, { "id": 2164585, "postDate": "2023-03-01T17:17:18.473Z", "content": "Congrats, thanks for this write-up and all the things you shared during this competition.
", "rawMarkdown": "Congrats, thanks for this write-up and all the things you shared during this competition.", "votes": 1 }, { "id": 2164545, "postDate": "2023-03-01T16:38:56.193Z", "content": "Congrats @remekkinas and @aikhmelnytskyy! A well-deserved gold medal indeed & surely, the first of many for @remekkinas 💯
\nA great write-up above. I learned a lot from @hengck23's posts as well. Looking forward to the finer details in your code 🔥
", "rawMarkdown": "Congrats @remekkinas and @aikhmelnytskyy! A well-deserved gold medal indeed & surely, the first of many for @remekkinas 💯\n\nA great write-up above. I learned a lot from @hengck23's posts as well. Looking forward to the finer details in your code 🔥", "votes": 1, "replies": [ { "id": 2164758, "postDate": "2023-03-01T19:17:29.267Z", "content": "Hi, thank you. Code will be available soon - I am working on some improvements (testing ideas provided by rest of participants).
", "rawMarkdown": "Hi, thank you. Code will be available soon - I am working on some improvements (testing ideas provided by rest of participants). ", "votes": 1 } ] }, { "id": 2163958, "postDate": "2023-03-01T07:54:12.853Z", "content": "Congratulations! You gave me a lot of help in this competition, especially the roi extraction of the image dataset, wonderful work!
", "rawMarkdown": "Congratulations! You gave me a lot of help in this competition, especially the roi extraction of the image dataset, wonderful work!", "votes": 1 }, { "id": 2163926, "postDate": "2023-03-01T07:20:15.507Z", "content": "Congrats @remekkinas and team. A lot of works and well deserved!
", "rawMarkdown": "Congrats @remekkinas and team. A lot of works and well deserved!", "votes": 1 }, { "id": 2163653, "postDate": "2023-03-01T01:25:38.160Z", "content": "Congratulations! 🎉
\nYou mentioned that \"Training with layer freeze\" did not help. Does that mean your best models were trained with all layers unfrozen? Do you know roughly how big of an effect this was on the pF1 score?
\nThank you!
", "rawMarkdown": "Congratulations! 🎉\n\nYou mentioned that \"Training with layer freeze\" did not help. Does that mean your best models were trained with all layers unfrozen? Do you know roughly how big of an effect this was on the pF1 score?\n\nThank you!", "votes": 1, "replies": [ { "id": 2164769, "postDate": "2023-03-01T19:26:47.303Z", "content": "0.5 score is from unfrozen model layers (during training). We tried to increase batch size and decrease resources needed for training. This is why we decided to freeze some layers (we used this method only for effnet and resnet - first month of competition). Strategy was simple:
\nThe best in our case was unfrozen case. 75% unfrozen layers was slightly worse but gave us larger batch size. First month of competition I used for all experiment frozen setup to speed up.
\nFor more information about Effnet training procedure please see this tutorial: https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/
", "rawMarkdown": "0.5 score is from unfrozen model layers (during training). We tried to increase batch size and decrease resources needed for training. This is why we decided to freeze some layers (we used this method only for effnet and resnet - first month of competition). Strategy was simple:\n- test unfrozen model \n- test frozen backbone - head unfrozen \n- test 1/3 unfrozen backbone (only low level layers were trainable) - head unfrozen \n- increase to 2/3 unfrozen ...\n\nThe best in our case was unfrozen case. 75% unfrozen layers was slightly worse but gave us larger batch size. First month of competition I used for all experiment frozen setup to speed up. \n\nFor more information about Effnet training procedure please see this tutorial: https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/", "replies": [ { "id": 2164983, "postDate": "2023-03-01T22:41:26.517Z", "content": "That's very helpful, thanks for your response!
", "rawMarkdown": "That's very helpful, thanks for your response!" } ] } ] }, { "id": 2163642, "postDate": "2023-03-01T01:10:49.840Z", "content": "Fascinating solution! Thanks for the detailed write up @remekkinas
", "rawMarkdown": "Fascinating solution! Thanks for the detailed write up @remekkinas", "votes": 1 }, { "id": 2162911, "postDate": "2023-02-28T13:24:48.153Z", "content": "Congratulations @remekkinas. I learned a lot from you during the competition. Thank you for all of the hard work and for sharing.
", "rawMarkdown": "Congratulations @remekkinas. I learned a lot from you during the competition. Thank you for all of the hard work and for sharing.", "votes": 1 }, { "id": 2162906, "postDate": "2023-02-28T13:22:23.990Z", "content": "You are great Remek! congratulation for the result and above all for the support you always provide here.
", "rawMarkdown": "You are great Remek! congratulation for the result and above all for the support you always provide here.", "votes": 1 }, { "id": 2162767, "postDate": "2023-02-28T12:01:56.743Z", "content": "Congratulations @remekkinas !!!
\nIn this comp. you really helped me !!! I want to say thank you.
I like the part of GridDropoutv2 and CoarseDropoutv2 that generates cuts in random grayscale.
\nHow did you come up with the idea?
\nAcutually I don't know much about the image recognition field, so please let me know.
Thank you very much. The source of inspiration was from Trivial Augument: https://github.com/automl/trivialaugment
", "rawMarkdown": "Thank you very much. The source of inspiration was from Trivial Augument: https://github.com/automl/trivialaugment ", "votes": 3, "replies": [ { "id": 2162863, "postDate": "2023-02-28T13:01:46.447Z", "content": "Interesting! I must check this paper.
\nafter the competition, you also give me tips
\nThank you very much!!
I am open to:
\nAI is not only job for me but passion 😍 I really love do it … although it can devastate and frustrate me 😂 It's a complicated relationship.
", "rawMarkdown": "I am open to:\n- learn\n- share\n\nAI is not only job for me but passion 😍 I really love do it ... although it can devastate and frustrate me 😂 It's a complicated relationship.", "votes": 1 } ] } ] }, { "id": 2162510, "postDate": "2023-02-28T08:49:51.377Z", "content": "Congratulations @remekkinas, all your work helped me during this competition and I really wanted to thank you.
\nI have been very excited to see your solution.
\nKeep going! 😊
Thank you very much! 👍
", "rawMarkdown": "Thank you very much! 👍" } ] }, { "id": 2162359, "postDate": "2023-02-28T06:40:17.217Z", "content": "Congratulations @remekkinas! Looks like giving a lot is also a key to success. 😃
", "rawMarkdown": "Congratulations @remekkinas! Looks like giving a lot is also a key to success. 😃", "votes": 1 }, { "id": 2162104, "postDate": "2023-02-28T01:59:52.767Z", "content": "Congratulations! I have a question, I don't understand the ensembling part very well. Did you use 3 models as a each model of 3-fold and use different pooling and tta for each fold?
", "rawMarkdown": "Congratulations! I have a question, I don't understand the ensembling part very well. Did you use 3 models as a each model of 3-fold and use different pooling and tta for each fold?", "votes": 1, "replies": [ { "id": 2162715, "postDate": "2023-02-28T11:30:26.173Z", "content": "Thank you!
\n0.5 (priv LB)
\nCongratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 2162092, "postDate": "2023-02-28T01:33:41.063Z", "content": "Thank in advance for sharing your experience, congratulations!!!
", "rawMarkdown": "Thank in advance for sharing your experience, congratulations!!!", "votes": 1 }, { "id": 2162089, "postDate": "2023-02-28T01:30:18.053Z", "content": "Thank you and congratulations for the competition achievements!
", "rawMarkdown": "Thank you and congratulations for the competition achievements!", "votes": 1 }, { "id": 2162088, "postDate": "2023-02-28T01:29:11.943Z", "content": "Thank you for the very detailed summary. I have definitely learned something here.
", "rawMarkdown": "Thank you for the very detailed summary. I have definitely learned something here. ", "votes": 1 }, { "id": 2178412, "postDate": "2023-03-12T12:11:03.333Z", "content": "Congrats Remek, eagerly waiting for your source code.
", "rawMarkdown": "Congrats Remek, eagerly waiting for your source code.", "votes": 2 }, { "id": 2162846, "postDate": "2023-02-28T12:52:11.053Z", "content": "Congrats @remekkinas and @aikhmelnytskyy Well deserved gold medal indeed.
\nWe also struggled a lot during the competition (CV-LB inconsistencies, finding the right amount of augmentation, not to overfit the train data in 3 epochs :D). It helped a lot to read your posts to look for ideas and see that others are struggling as well :)
", "rawMarkdown": "Congrats @remekkinas and @aikhmelnytskyy Well deserved gold medal indeed.\n\nWe also struggled a lot during the competition (CV-LB inconsistencies, finding the right amount of augmentation, not to overfit the train data in 3 epochs :D). It helped a lot to read your posts to look for ideas and see that others are struggling as well :)", "votes": 2, "replies": [ { "id": 2162855, "postDate": "2023-02-28T12:58:05.453Z", "content": "We had the same problem. I read a paper about augumentation techniques in breast cancer detection/classification. Tested most of them. Setup I described works the best for us … but I am waiting for participants solution description and will apply new things to see if we can progress.
\nNow I can see that we had \"partial\" CV/LB consistency <=> no consistency 😂
", "rawMarkdown": "We had the same problem. I read a paper about augumentation techniques in breast cancer detection/classification. Tested most of them. Setup I described works the best for us ... but I am waiting for participants solution description and will apply new things to see if we can progress. \n\nNow I can see that we had \"partial\" CV/LB consistency <=> no consistency 😂", "votes": 1 } ] }, { "id": 2162381, "postDate": "2023-02-28T06:58:52.520Z", "content": "Congrats Remek, amazing grit going all the way through this comp, putting so much energy and surviving the shake up!
", "rawMarkdown": "Congrats Remek, amazing grit going all the way through this comp, putting so much energy and surviving the shake up!", "votes": 2 }, { "id": 2162739, "postDate": "2023-02-28T11:50:11.647Z", "content": "thanks for sharing.
", "rawMarkdown": "thanks for sharing.", "votes": 1 }, { "id": 2162091, "postDate": "2023-02-28T01:33:30.100Z", "content": "Congratulations! Thanks for sharing.
", "rawMarkdown": "Congratulations! Thanks for sharing.", "votes": 1 } ], "comments": [ { "id": 2162827, "author_name": "Giba", "author_url": "", "post_date": "2023-02-28T12:31:55.093000", "content": "Congrats @remekkinas and @aikhmelnytskyy ! Great solution, thanks for sharing.
", "votes": 3, "replies": [ { "id": 2162837, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-28T12:42:36.267000", "content": "@titericz I would like to thank you very much for your inspiration. This is true story. I was really devastated by this competition :) The more I wanted give up the harder I worked thanks to this message:
\n
Congrats Remek for your 1st gold medal in competition and master title!
", "votes": 1, "replies": [ { "id": 2166436, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-03-02T19:51:28.190000", "content": "Thank you @michau96 - and as a bonus: our team did not use any pixel from external dataset 😁
", "votes": 1, "replies": [] } ] }, { "id": 2165824, "author_name": "Qishen Ha", "author_url": "", "post_date": "2023-03-02T13:33:38.910000", "content": "Congratulation for being competition master!
", "votes": 1, "replies": [ { "id": 2166439, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-03-02T19:53:05.540000", "content": "@haqishen thank you very much. I am really, really happy. Still reading solution and working on some improvements. Still learning and have fun during competition. See you soon.
", "votes": 2, "replies": [] } ] }, { "id": 2164585, "author_name": "FabienDaniel", "author_url": "", "post_date": "2023-03-01T17:17:18.473000", "content": "Congrats, thanks for this write-up and all the things you shared during this competition.
", "votes": 1, "replies": [] }, { "id": 2164545, "author_name": "Mayank Bhaskar", "author_url": "", "post_date": "2023-03-01T16:38:56.193000", "content": "Congrats @remekkinas and @aikhmelnytskyy! A well-deserved gold medal indeed & surely, the first of many for @remekkinas 💯
\nA great write-up above. I learned a lot from @hengck23's posts as well. Looking forward to the finer details in your code 🔥
", "votes": 1, "replies": [ { "id": 2164758, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-03-01T19:17:29.267000", "content": "Hi, thank you. Code will be available soon - I am working on some improvements (testing ideas provided by rest of participants).
", "votes": 1, "replies": [] } ] }, { "id": 2163958, "author_name": "Jijie Li", "author_url": "", "post_date": "2023-03-01T07:54:12.853000", "content": "Congratulations! You gave me a lot of help in this competition, especially the roi extraction of the image dataset, wonderful work!
", "votes": 1, "replies": [] }, { "id": 2163926, "author_name": "KhanhVD", "author_url": "", "post_date": "2023-03-01T07:20:15.507000", "content": "Congrats @remekkinas and team. A lot of works and well deserved!
", "votes": 1, "replies": [] }, { "id": 2163653, "author_name": "Julia Nickerson", "author_url": "", "post_date": "2023-03-01T01:25:38.160000", "content": "Congratulations! 🎉
\nYou mentioned that \"Training with layer freeze\" did not help. Does that mean your best models were trained with all layers unfrozen? Do you know roughly how big of an effect this was on the pF1 score?
\nThank you!
", "votes": 1, "replies": [ { "id": 2164769, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-03-01T19:26:47.303000", "content": "0.5 score is from unfrozen model layers (during training). We tried to increase batch size and decrease resources needed for training. This is why we decided to freeze some layers (we used this method only for effnet and resnet - first month of competition). Strategy was simple:
\nThe best in our case was unfrozen case. 75% unfrozen layers was slightly worse but gave us larger batch size. First month of competition I used for all experiment frozen setup to speed up.
\nFor more information about Effnet training procedure please see this tutorial: https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/
", "votes": 0, "replies": [ { "id": 2164983, "author_name": "Julia Nickerson", "author_url": "", "post_date": "2023-03-01T22:41:26.517000", "content": "That's very helpful, thanks for your response!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2163642, "author_name": "Ravi Shah", "author_url": "", "post_date": "2023-03-01T01:10:49.840000", "content": "Fascinating solution! Thanks for the detailed write up @remekkinas
", "votes": 1, "replies": [] }, { "id": 2162911, "author_name": "kemp", "author_url": "", "post_date": "2023-02-28T13:24:48.153000", "content": "Congratulations @remekkinas. I learned a lot from you during the competition. Thank you for all of the hard work and for sharing.
", "votes": 1, "replies": [] }, { "id": 2162906, "author_name": "Fili73", "author_url": "", "post_date": "2023-02-28T13:22:23.990000", "content": "You are great Remek! congratulation for the result and above all for the support you always provide here.
", "votes": 1, "replies": [] }, { "id": 2162767, "author_name": "taruto", "author_url": "", "post_date": "2023-02-28T12:01:56.743000", "content": "Congratulations @remekkinas !!!
\nIn this comp. you really helped me !!! I want to say thank you.
I like the part of GridDropoutv2 and CoarseDropoutv2 that generates cuts in random grayscale.
\nHow did you come up with the idea?
\nAcutually I don't know much about the image recognition field, so please let me know.
Thank you very much. The source of inspiration was from Trivial Augument: https://github.com/automl/trivialaugment
", "votes": 3, "replies": [ { "id": 2162863, "author_name": "taruto", "author_url": "", "post_date": "2023-02-28T13:01:46.447000", "content": "Interesting! I must check this paper.
\nafter the competition, you also give me tips
\nThank you very much!!
I am open to:
\nAI is not only job for me but passion 😍 I really love do it … although it can devastate and frustrate me 😂 It's a complicated relationship.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2162510, "author_name": "Paul Bacher", "author_url": "", "post_date": "2023-02-28T08:49:51.377000", "content": "Congratulations @remekkinas, all your work helped me during this competition and I really wanted to thank you.
\nI have been very excited to see your solution.
\nKeep going! 😊
Thank you very much! 👍
", "votes": 0, "replies": [] } ] }, { "id": 2162359, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-28T06:40:17.217000", "content": "Congratulations @remekkinas! Looks like giving a lot is also a key to success. 😃
", "votes": 1, "replies": [] }, { "id": 2162104, "author_name": "hoyso48", "author_url": "", "post_date": "2023-02-28T01:59:52.767000", "content": "Congratulations! I have a question, I don't understand the ensembling part very well. Did you use 3 models as a each model of 3-fold and use different pooling and tta for each fold?
", "votes": 1, "replies": [ { "id": 2162715, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-28T11:30:26.173000", "content": "Thank you!
\n0.5 (priv LB)
\nCongratulations!
", "votes": 1, "replies": [] }, { "id": 2162092, "author_name": "Pablo Larrosa", "author_url": "", "post_date": "2023-02-28T01:33:41.063000", "content": "Thank in advance for sharing your experience, congratulations!!!
", "votes": 1, "replies": [] }, { "id": 2162089, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-02-28T01:30:18.053000", "content": "Thank you and congratulations for the competition achievements!
", "votes": 1, "replies": [] }, { "id": 2162088, "author_name": "joejeo1", "author_url": "", "post_date": "2023-02-28T01:29:11.943000", "content": "Thank you for the very detailed summary. I have definitely learned something here.
", "votes": 1, "replies": [] }, { "id": 2178412, "author_name": "tharun_01", "author_url": "", "post_date": "2023-03-12T12:11:03.333000", "content": "Congrats Remek, eagerly waiting for your source code.
", "votes": 2, "replies": [] }, { "id": 2162846, "author_name": "beluga", "author_url": "", "post_date": "2023-02-28T12:52:11.053000", "content": "Congrats @remekkinas and @aikhmelnytskyy Well deserved gold medal indeed.
\nWe also struggled a lot during the competition (CV-LB inconsistencies, finding the right amount of augmentation, not to overfit the train data in 3 epochs :D). It helped a lot to read your posts to look for ideas and see that others are struggling as well :)
", "votes": 2, "replies": [ { "id": 2162855, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-28T12:58:05.453000", "content": "We had the same problem. I read a paper about augumentation techniques in breast cancer detection/classification. Tested most of them. Setup I described works the best for us … but I am waiting for participants solution description and will apply new things to see if we can progress.
\nNow I can see that we had \"partial\" CV/LB consistency <=> no consistency 😂
", "votes": 1, "replies": [] } ] }, { "id": 2162381, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2023-02-28T06:58:52.520000", "content": "Congrats Remek, amazing grit going all the way through this comp, putting so much energy and surviving the shake up!
", "votes": 2, "replies": [] }, { "id": 2162739, "author_name": "dragon zhang", "author_url": "", "post_date": "2023-02-28T11:50:11.647000", "content": "thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 2162091, "author_name": "ChenxiangSun@NJU", "author_url": "", "post_date": "2023-02-28T01:33:30.100000", "content": "Congratulations! Thanks for sharing.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2162073": "First of all congratulations to all participants. Congratulations to dream teams from gold zone. I’m impressed by your consistency in winning Kaggle competition. Waiting to learn from your solution.\n\nThank you my team mate Andrij @aikhmelnytskyy We had great collaboration 👍👍👍 - I feel that from first minute we played in one team having one goal - find better solution.\n\nGold in competition was dream for me. Last year we (with @christofhenkel ) were #1 in sliver (#12 solution in Image Matching Challange 2021). This year I decided to work hard to experience gold zone and finally become competition master. Even there is no official LB finalized .... we are #9 and in gold! :) and I am ...... extremely happy! 😁😁😍😜\n\nThis competition was really great for testing many different computer vision techniques. Three months passed very quickly. The first phase of the competition was difficult. We performed a large number of different tests that did not gave us results higher than 0.3 (LB score). It was very frustrating. We were unable to find any correlation between local CV and LB. Then we setup good training pipeline – main our success point are:\n-\tSampling strategy and positive class balancing\n-\tAugumentation\n-\tModel selection\n- Postprocessing\n\nLast two weeks of competition were hard to me – I caught covid and had to pause (during recovery time I coded using iPad). But we cooperate all the time and we finally managed to jump to TOP10 and 0.63 (public LB). Finally we were closing gold zone on #13 (public LB) and #9 in private LB!\n\n**Models score summary**\n•\tbest public LB: 0.63 (ensemble) / 0.57 (single model) / local CV (0.48)\n•\tprivate lb: 0.50 (max: 0.50)\n\n\n**Competition achievements**\n•\tnew experience in Kaggle competition - a lot of good discussion \n•\t1x gold medal - dataset (I am very happy - my first one)\n•\t2x gold medals - notebook \n•\t1x gold medal - competition-> extremely happy! 😁😁😍😜\n\n**Our final selection**\nWe selected two different solution which based on the same model setup.\n•\t3 models - ensemble average model prediction probabilities -> LB: 0.63 (PL: 0.47)\n•\t3 models - voting strategy and then score average (on votes score) -> LB: 0.62 (PL: 0.5)\nAfter many tests we had strong feeling that our second choice (even score was lower on LB that many of our rest solution) is more stable (was less sensitive on th) than other solution. So we closed eye and trusted in our test rather then LB score.\n\n**Solution description in 4 steps**\nOur solution is very simple. We tried different ways to predict breast cancer but finally it appeared that simples solution wors for us the best (both local CV and LB).\n\n1.\tProcess dicom files to png (windowing).\n2.\tInference – 3 convnext (v1) models with TTA\n3.\tEnsemble – probabilities average or voting\n4.\tThresholding – th ~0.5 -> final prediction result 0|1\n\n\n**Dataset**\n•\tImage resolution: 1536x768\n•\tROI cropped - cv2.connectedComponentsWithStats method (we started with yolov5 for prototyping phase but then we used cv2 - since licence regulations)\n•\t4-GroupFold: on patient_id\n•\timage pixel scaling: div by 255.\n•\tdicom2png – proposed by @hengck23 (https://www.kaggle.com/code/hengck23/3hr-tensorrt-nextvit-example)\n•\twe do not use external dataset – all models were trained on competition data\n•\twe processed data in two steps (to avoid problem with file storage capacity – during the tests we exported images to different resolutions – max 2400px):\n1. process j2k files format (dicom2png and crop to 1536xW - we do not resize it in this step to final resolution)->inference->delete files \n2. process nonj2k files->inference->delete files\n\nfast and reliable crop roi function we used during competition (cretits to @vslaykovsky : [\"RSNA: Cut Off Empty Space from Images\"](https://www.kaggle.com/code/vslaykovsky/rsna-cut-off-empty-space-from-images))\n```\ndef crop_roi(img, photometric_interpretation):\n # it can be improved \n Y = img\n xmin = Y.min()\n xmax = Y.max()\n\n norm = np.empty_like(Y, dtype=np.uint8)\n\n dicomsdl.util.convert_to_uint8(Y, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n \n X = norm\n X = X[5:-5, 5:-5]\n\n output= cv2.connectedComponentsWithStats((X > 10).astype(np.uint8)[:, :], 8, cv2.CV_32S) #\n stats = output[2]\n \n idx = stats[1:, 4].argmax() + 1\n x1, y1, w, h = stats[idx][:4]\n x2 = x1 + w\n y2 = y1 + h\n \n X_out = Y[y1: y2, x1: x2]\n \n return X_out\n```\n\n**Augumentation** \n•\tAlbumentations (we tried Kornia and works great but we need more time to rewrite functions we used in competition).\n\n```\ntransformation = [\n \tA.OneOf([\n \tA.RandomBrightnessContrast(always_apply=False, p=.5, brightness_limit=(-1, 1.0), contrast_limit=(-1, 1.0), brightness_by_max=True),\n \tA.RandomGamma(always_apply=False, p=.5, gamma_limit=(60, 120), eps=None),\n \t], p = 0.5),\n \t \n \tA.Rotate(limit=5, p=0.5),\n \tA.Affine(rotate = 5, translate_percent=0.1, scale=[0.9,1.5], shear=0, p=0.5),\n \t \n \tA.HorizontalFlip(p=0.5),\n \tA.VerticalFlip(p=0.5),\n \tA.Resize(im_size[0], im_size[1]),\n \tA.ShiftScaleRotate(always_apply=False, p=.2,\n \tshift_limit_x=(-1.0, 1.0),\n \tshift_limit_y=(-1.0, 1.0),\n \tscale_limit=(-0.1, 0.1),\n \trotate_limit=(-5, 5),\n \tinterpolation=0,\n \tborder_mode=3,\n \tvalue=(0, 0, 0),\n \tmask_value=None,\n \trotate_method='largest_box'),\n \t \n \tA.OneOf([\n \tGridDropoutv2(always_apply=False, p=.2, ratio = .25, unit_size_min = 100, unit_size_max = 400, holes_number_x=100, holes_number_y=100),\n \tCoarseDropoutv2(always_apply=False, p=.2, max_holes=12, max_height=250, max_width=100, min_holes=3, min_height=50, min_width=50, mask_fill_value=None)], p=0.25),\n \tToTensorV2()\n \t]\n```\n\nGridDropoutv2 and CoarseDropoutv2 is our modification of Albumentations function. It generates cuts in random greyscale.\n\n\n \n\n**Training**\n•\tFramework: Pytorch scripts (multi gpu support: DDP)\n•\tOptimizer: RAdam \n•\tLookahead: on\n•\tScheduler: OneCycle - no warm up\n•\tWeight decay: 1e-2\n•\tDataset each epoch was seeded by different seed.\n•\tBatch size: 16\n•\tMixed precision (AMP): on\n•\tGradient clipping: on\n•\tLoss function: BCEWithLogitsLoss with pos_weight = 1.0 - 1.25\n•\tEpochs: 7 (best models are from 4-5 epochs) - we decided to take early stopping (best probf1score) models instead of the last one.\n•\tSampler (thanks to @hengck23): custom sequential sampler (sequence -> positive sample / negative sample = 8) – small changes compare to the original one.\n\n```\nclass Balancer(torch.utils.data.Sampler):\n\n\tdef __init__(self, pos_cases, neg_cases, ratio = 3):\n \tself.r = ratio - 1\n \tself.pos_index = pos_cases\n \tself.neg_index = neg_cases\n\n \tself.length = self.r * int(np.floor(len(self.neg_index)/self.r))\n \tself.ds_len = self.length + (self.length // self.r)\n\n\tdef __iter__(self):\n \tpos_index = self.pos_index\n \tneg_index = self.neg_index\n \t \n \tnp.random.shuffle(pos_index)\n \tnp.random.shuffle(neg_index)\n\n \tneg_index = neg_index[:self.length].reshape(-1,self.r)\n \t#pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\n \tpos_index_len = len(pos_index)\n \t \n \tpos_index = np.tile(pos_index, ((len(neg_index) // pos_index_len) + 1, 1))\n \tpos_index = np.apply_along_axis(np.random.permutation, 1, pos_index)\n \tpos_index = pos_index.reshape(-1,1)[:len(neg_index)]\n\n \tindex = np.concatenate([pos_index,neg_index], -1).reshape(-1)\n \treturn iter(index)\n\n\tdef __len__(self):\n \treturn self.ds_len\n```\n \n**Models**\nConvNext_v1 small (timm - checkpoint: convnext_small.fb_in22k_ft_in1k_384). We tried different architectures but this one works in our solution the best.\n•\tavg pooling or \n•\tGEM pooling (trainable parameters set to True)\n•\tdrop_path_rate=0.2 and drop_rate=0.05 (we tested different settings but these works the best for our setup)\n•\tinput size: (1536, 768), 3 channels (b&w images)\n\n**Local validation**\n•\tMetrics: probf1, ROC_AUC, prec/recall and MCC (Matthews’s correlation coefficient)\n•\tFor local validation we used tool provided by @hengck23 - https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521\n•\tFor each traing epoch we additionaly plot: prediction dynamic, confusion matrix.\n•\tWe looked into prediction and performed visual analysis of missed or incorrectly classified samples. \n\n**Inference**\n•\tThree models – ensemble (voting or model probabilities averaging):\n1. avg pooling - fold 1 (trained on 1536 x 768) -> probf1: 0.52 (local CV) / 0.57 (LB)\n2. GEM pooling - fold 2 (trained on 1536 x 768) -> probf1: 0.44 (local CV) (we do not test it on LB)\n3. GEM pooling - fold 3 (trained on 1536 x 768) -> probf1: 0.46 (local CV) (-)\n•\tTTA: h_flip only (TTA weighted - 0.6 original image + 0.4 flipped image) - > it inceased our score by 0.03\n\n\n**Ensembling**\n•\tModel probabilities ensembling using better then median function\n\n```\ndef better_than_median(inputs, axis):\n \"\"\"Compute the mean of the predictions if there are no outliers,\n or the median if there are outliers.\n\n Parameter: inputs = ndarray of shape (n_samples, n_folds)\"\"\"\n spread = inputs.max(axis=axis) - inputs.min(axis=axis) \n spread_lim = 0.6\n print(f\"Inliers: {(spread > spread_lim).sum():7} -> compute mean\")\n print(f\"Outliers: {(spread <= spread_lim).sum():7} -> compute median\")\n print(f\"Total: {len(inputs):7}\")\n return np.where(spread > spread_lim,\n np.mean(inputs, axis=axis),\n np.median(inputs, axis=axis))\n```\n\n•\tVoting and then averaging\n\n```\ndef voting_confidence(x):\n if x<=0.25:\n return 0.0\n elif x<=0.45:\n return 0.35\n elif x<=0.55:\n return x\n elif x<=0.75:\n return 0.65\n else:\n return 1.0\n```\n\n\n**CPU/GPU**\n•\tTraining: 1-2xA6000 or A100 GPU-80GB on Cloud \n\n\n**Experiment tracking**\n•\tWeights & Biases \n\n**Things we tested during competition and did not improve our score (probably more tests are needed)**\n•\tFull dataset training, site_1/site_2 separate models, up-sampling (by sample frac), down-sampling.\n•\tStochastic Weight Averaging and Exponential Moving Average\n•\tOptimizers – AdamW, SGD, NAdam, Lion, Lamb\n•\tTraining with layer freeze (different levels)\n•\tLoss function: Focal loss, LDAM, LMFLoss, Label smoothing\n•\tModels: Effnet, NextVit (simmilar score but harder to train and slower), DenseNet, Inception_v3, maxvit\n•\tEffnet + pixel-wise self attention, Effnet + cross attention\n•\tConvnextv1 + max and avg concatenated pooling\n•\tSynthetic dataset: synthetic breast cancer generator (experienced radiologist is required to evaluate solution).\n•\tPatchGD - https://arxiv.org/pdf/2301.13817.pdf\n•\tTraining attitudes - Pair training – MLO / CC – patient_id and side (L/R) \n•\tMetamodel - SVM and XGBoost on image enbeddings.\n•\tPseudolabeling - using new samples during submission time.\n•\tSelecting best samples for cases where patient has more then 2 views \n\n```\nsub_s1 = sub_s1[['prediction_id', 'site_id', 'patient_id', 'laterality', 'cancer']].\\\n groupby(['patient_id','laterality']).\\\n apply(lambda x: x.nlargest(3,'cancer')).reset_index(drop=True)\n```\n\n**Special thanks to**\n•\t@Andrij - for great cooperation during the competition. For very good and substantive talks aimed at solving the problem and improving results.\n•\t@hengck23 - Thanks for the great activity on the forum and sharing knowledge. I implemented many of your ideas, many of them improved our score. Thank you!\n\nSource code will be available soon (in week) on [Remek github](https://github.com/rkinas", "2162827": "Congrats @remekkinas and @aikhmelnytskyy ! Great solution, thanks for sharing.", "2165894": "Congrats Remek for your 1st gold medal in competition and master title! ", "2165824": "Congratulation for being competition master!", "2164585": "Congrats, thanks for this write-up and all the things you shared during this competition.", "2164545": "Congrats @remekkinas and @aikhmelnytskyy! A well-deserved gold medal indeed & surely, the first of many for @remekkinas 💯\n\nA great write-up above. I learned a lot from @hengck23's posts as well. Looking forward to the finer details in your code 🔥", "2163958": "Congratulations! You gave me a lot of help in this competition, especially the roi extraction of the image dataset, wonderful work!", "2163926": "Congrats @remekkinas and team. A lot of works and well deserved!", "2163653": "Congratulations! 🎉\n\nYou mentioned that \"Training with layer freeze\" did not help. Does that mean your best models were trained with all layers unfrozen? Do you know roughly how big of an effect this was on the pF1 score?\n\nThank you!", "2163642": "Fascinating solution! Thanks for the detailed write up @remekkinas", "2162911": "Congratulations @remekkinas. I learned a lot from you during the competition. Thank you for all of the hard work and for sharing.", "2162906": "You are great Remek! congratulation for the result and above all for the support you always provide here.", "2162767": "Congratulations @remekkinas !!!\nIn this comp. you really helped me !!! I want to say thank you.\n\n\nI like the part of GridDropoutv2 and CoarseDropoutv2 that generates cuts in random grayscale.\nHow did you come up with the idea? \nAcutually I don't know much about the image recognition field, so please let me know.", "2162510": "Congratulations @remekkinas, all your work helped me during this competition and I really wanted to thank you.\nI have been very excited to see your solution.\nKeep going! 😊 ", "2162359": "Congratulations @remekkinas! Looks like giving a lot is also a key to success. 😃", "2162104": "Congratulations! I have a question, I don't understand the ensembling part very well. Did you use 3 models as a each model of 3-fold and use different pooling and tta for each fold?", "2162094": "Congratulations!", "2162092": "Thank in advance for sharing your experience, congratulations!!!", "2162089": "Thank you and congratulations for the competition achievements!", "2162088": "Thank you for the very detailed summary. I have definitely learned something here. ", "2178412": "Congrats Remek, eagerly waiting for your source code.", "2162846": "Congrats @remekkinas and @aikhmelnytskyy Well deserved gold medal indeed.\n\nWe also struggled a lot during the competition (CV-LB inconsistencies, finding the right amount of augmentation, not to overfit the train data in 3 epochs :D). It helped a lot to read your posts to look for ideas and see that others are struggling as well :)", "2162381": "Congrats Remek, amazing grit going all the way through this comp, putting so much energy and surviving the shake up!", "2162739": "thanks for sharing.", "2162091": "Congratulations! Thanks for sharing." } }