First of all I would like to thank the organisers and Kaggle for this great competition. It was an amazing learning experience, I got to learn a lot of things. Congratulations to all the winners!
\nI trained on false color images and used soft labels. This gave me a big boost in both cv and lb score as compared to when training on the pixel masks. I think this was one of the most important tricks for this competiton. It was very surprising for me to see that the model was able to capture the uncertainty depicted by the probabilities of the pixels. As individual labels were not available for the valid set, so I didnt include it in training. I used it as hold set.
\nThis was a bit tricky due to the presence of many duplicates in the training set. So, to build a robust local validation, I first sorted the training set by date and time which was available in train.json and then split the sorted data into 5 folds without shuffling. After this, I clipped the valid folds during training like this:
\n self.fold == :\n df = df[:-]\n self.fold == :\n df = df[:]\n:\n df = df[:-]\nThe cv score was not at all correlated with the leaderboard score, but still I trusted my cv in the end to select my submissions.
\nI tried many encoders but used efficientnet-b5, regnety_120 and seresnextaa101d_32x8d in the end as they gave the best cv score. I only used Unet for all the encoders.
\nTo reduce training time, I tried to remove some images during training. Removing all the non-contrail images reduced the local cv score. Then I predicted the train set with one of my best model and made a column with the highest pixel probability predicted by the model for the images. Then I removed the images which had max pixel probability lower than 0.05 and had no contrail.
\ndf = df[~((df['max_prob_pred'] < 0.05) & (df['contrail'] == 0))]
Using this I was able to remove about 4000 images from training, reduce training time a lot and it didn't effect the cv score at all.
\nI used my cv folds to get the ensemble weights for my final models.
\nWeighted ensembling: 0.3172[eb5] + 0.2928[regty120] + 0.39[sr101].
For final prediction, first I ensembled the different models separately for each fold. Then I converted the 5 fold predictions to binary predictions using the thresholds I got using my valid sets. The thresholds were 0.49, 0.47, 0.47, 0.48, 0.46 for the folds from 0 to 4. Then I took the average of the 5 folds and used 0.3 as the final threshold (which I got using the valid hold set).
\nMy teammate's account was deleted just after I teamed up with him. I don't know if he did anything wrong or not. Whatever be the case, I wish him best of luck for his future.
\nI really hope I won't be removed from the leaderboard after verification. Let's see what happens ☺️
First of all I would like to thank the organisers and Kaggle for this great competition. It was an amazing learning experience, I got to learn a lot of things. Congratulations to all the winners!
\nI trained on false color images and used soft labels. This gave me a big boost in both cv and lb score as compared to when training on the pixel masks. I think this was one of the most important tricks for this competiton. It was very surprising for me to see that the model was able to capture the uncertainty depicted by the probabilities of the pixels. As individual labels were not available for the valid set, so I didnt include it in training. I used it as hold set.
\nThis was a bit tricky due to the presence of many duplicates in the training set. So, to build a robust local validation, I first sorted the training set by date and time which was available in train.json and then split the sorted data into 5 folds without shuffling. After this, I clipped the valid folds during training like this:
\n self.fold == :\n df = df[:-]\n self.fold == :\n df = df[:]\n:\n df = df[:-]\nThe cv score was not at all correlated with the leaderboard score, but still I trusted my cv in the end to select my submissions.
\nI tried many encoders but used efficientnet-b5, regnety_120 and seresnextaa101d_32x8d in the end as they gave the best cv score. I only used Unet for all the encoders.
\nTo reduce training time, I tried to remove some images during training. Removing all the non-contrail images reduced the local cv score. Then I predicted the train set with one of my best model and made a column with the highest pixel probability predicted by the model for the images. Then I removed the images which had max pixel probability lower than 0.05 and had no contrail.
\ndf = df[~((df['max_prob_pred'] < 0.05) & (df['contrail'] == 0))]
Using this I was able to remove about 4000 images from training, reduce training time a lot and it didn't effect the cv score at all.
\nI used my cv folds to get the ensemble weights for my final models.
\nWeighted ensembling: 0.3172[eb5] + 0.2928[regty120] + 0.39[sr101].
For final prediction, first I ensembled the different models separately for each fold. Then I converted the 5 fold predictions to binary predictions using the thresholds I got using my valid sets. The thresholds were 0.49, 0.47, 0.47, 0.48, 0.46 for the folds from 0 to 4. Then I took the average of the 5 folds and used 0.3 as the final threshold (which I got using the valid hold set).
\nMy teammate's account was deleted just after I teamed up with him. I don't know if he did anything wrong or not. Whatever be the case, I wish him best of luck for his future.
\nI really hope I won't be removed from the leaderboard after verification. Let's see what happens ☺️
Congrats Shashwat,
\nWell deserved. The solution is clean and very intuitive. Hope we can team up in the future.
Thank you very much @thejavanka
", "rawMarkdown": "Thank you very much @thejavanka" } ] }, { "id": 2385339, "postDate": "2023-08-11T09:21:45.687Z", "content": "Congrats!
\nCool trick with removing 4000 images for extra speed up.
\nWhen you talk about weights ensembling do you mean that ensemble prediction masks outputs with different weights?
\nDid you plan to release your solution?
Thank you @kuddai
\nYes I mean that only. I will publish my code in a few days.
Very nice and clean solution!
\nHow are you able to do so much training and testing on your CV? Do you have your own GPU or did Kaggle's suffice? I ask because running your train notebook took me an hour every time and ran out of GPU hours.
\nCould you also elaborate on your cross-validation as well? Did you clip that way because of the duplicates?
", "rawMarkdown": "Very nice and clean solution!\n\nHow are you able to do so much training and testing on your CV? Do you have your own GPU or did Kaggle's suffice? I ask because running your train notebook took me an hour every time and ran out of GPU hours.\n\nCould you also elaborate on your cross-validation as well? Did you clip that way because of the duplicates?", "votes": 1, "replies": [ { "id": 2384777, "postDate": "2023-08-11T04:34:16.250Z", "content": "Thank you @echen333us,
\nI used JavisLabs' A5000 GPU for all my experiments. It worked really well for me. Kaggle is also great but I find the 30 hour GPU limit very low.
\nYes. There were many duplicates present in the training set. A satellite image from the same location after a few hours can be very similar to the ones before. So sorting, splitting and clipping like this ensured that training and validation are far from each other with respect to time.
Congratulations for 26th place!
\nFingers crossed that you won't get removed from the leaderboard due to bad team-up! I like that you have cared about the image similarities in the data when setting up your validation folds. We experimented with it as well and were able to find a very good correlation between CV and LB by grouping similar images into same folds. If you are interested in it we published a notebook about it.
\nI also like the trick of removing empty masks… have to try it out with our balanced CV set as well to check whether our score would still have been almost the same as well.
\nDo you remember how much boost you got by using soft labels with bce loss?
", "rawMarkdown": "Congratulations for 26th place! \n\nFingers crossed that you won't get removed from the leaderboard due to bad team-up! I like that you have cared about the image similarities in the data when setting up your validation folds. We experimented with it as well and were able to find a very good correlation between CV and LB by grouping similar images into same folds. If you are interested in it we published a notebook about it. \n\nI also like the trick of removing empty masks... have to try it out with our balanced CV set as well to check whether our score would still have been almost the same as well.\n\nDo you remember how much boost you got by using soft labels with bce loss?", "votes": 2, "replies": [ { "id": 2384748, "postDate": "2023-08-11T04:20:41.337Z", "content": "Thank you very much @allunia
\nCongratulations for 52nd Place!
\nSure, I will take a look at the notebook. Thanks!
\nAfter using soft labels, the score boost was around 0.02-0.03, if I remember correctly!
Congrats Shashwat,
\nWell deserved. The solution is clean and very intuitive. Hope we can team up in the future.
Thank you very much @thejavanka
", "votes": 0, "replies": [] } ] }, { "id": 2385339, "author_name": "iadduk", "author_url": "", "post_date": "2023-08-11T09:21:45.687000", "content": "Congrats!
\nCool trick with removing 4000 images for extra speed up.
\nWhen you talk about weights ensembling do you mean that ensemble prediction masks outputs with different weights?
\nDid you plan to release your solution?
Thank you @kuddai
\nYes I mean that only. I will publish my code in a few days.
Very nice and clean solution!
\nHow are you able to do so much training and testing on your CV? Do you have your own GPU or did Kaggle's suffice? I ask because running your train notebook took me an hour every time and ran out of GPU hours.
\nCould you also elaborate on your cross-validation as well? Did you clip that way because of the duplicates?
", "votes": 1, "replies": [ { "id": 2384777, "author_name": "Shashwat Raman", "author_url": "", "post_date": "2023-08-11T04:34:16.250000", "content": "Thank you @echen333us,
\nI used JavisLabs' A5000 GPU for all my experiments. It worked really well for me. Kaggle is also great but I find the 30 hour GPU limit very low.
\nYes. There were many duplicates present in the training set. A satellite image from the same location after a few hours can be very similar to the ones before. So sorting, splitting and clipping like this ensured that training and validation are far from each other with respect to time.
Congratulations for 26th place!
\nFingers crossed that you won't get removed from the leaderboard due to bad team-up! I like that you have cared about the image similarities in the data when setting up your validation folds. We experimented with it as well and were able to find a very good correlation between CV and LB by grouping similar images into same folds. If you are interested in it we published a notebook about it.
\nI also like the trick of removing empty masks… have to try it out with our balanced CV set as well to check whether our score would still have been almost the same as well.
\nDo you remember how much boost you got by using soft labels with bce loss?
", "votes": 2, "replies": [ { "id": 2384748, "author_name": "Shashwat Raman", "author_url": "", "post_date": "2023-08-11T04:20:41.337000", "content": "Thank you very much @allunia
\nCongratulations for 52nd Place!
\nSure, I will take a look at the notebook. Thanks!
\nAfter using soft labels, the score boost was around 0.02-0.03, if I remember correctly!