Thanks everybody for the competition! The task was really interesting and had huge amounts of data. Thanks very much to my teammate who writes really perfect code on Python. You should understand me, he writes comments to functions and uses types in Python! Also thanks to @scitator for his perfect training ML framework Catalyst, all models were trained with the help of it.
\n\nNow I’d like to tell you about my and Artyom Palvelev solution using simplified data only.
\n\nPictures of size 128x128 gave the best combination of score and training time. I don’t have time information, so I encoded the following data in three channels:
\n\nFirstly, I tried models from forum like MobileNetV2, but they showed poor performance, so I trained something deeper. My first good model was SE_ResNext50 0.942 LB. I used CosineAnnealingLR and averaged top4 checkpoints.\nThen we merged with Artyom and tried different model architectures. Blend with his models gave 0.944 LB. In the final submission we had:
\n\nWe also used gradient accumulation to increase batch size to 1024 because data was noisy and bigger batch size gave better performance.
\n\nWe decided to use LGBM to ensemble our models because it is fast and usually gives better results than other methods. We used same idea as Pavel Ostyakov described in his 5th place Cdiscount solution. We predicted top10 classes by our best network, concatenated probabilities of other networks, added class_id feature and gave binary label: whether the class_id is correct or not. This method resulted in validation score 0.001 higher than a simple average. So, if you have enough time, always try LGBM to ensemble.\nFinal submit with LGBM -> 0.948 LB
\n\nThanks everybody for the competition! The task was really interesting and had huge amounts of data. Thanks very much to my teammate who writes really perfect code on Python. You should understand me, he writes comments to functions and uses types in Python! Also thanks to @scitator for his perfect training ML framework Catalyst, all models were trained with the help of it.
\n\nNow I’d like to tell you about my and Artyom Palvelev solution using simplified data only.
\n\nPictures of size 128x128 gave the best combination of score and training time. I don’t have time information, so I encoded the following data in three channels:
\n\nFirstly, I tried models from forum like MobileNetV2, but they showed poor performance, so I trained something deeper. My first good model was SE_ResNext50 0.942 LB. I used CosineAnnealingLR and averaged top4 checkpoints.\nThen we merged with Artyom and tried different model architectures. Blend with his models gave 0.944 LB. In the final submission we had:
\n\nWe also used gradient accumulation to increase batch size to 1024 because data was noisy and bigger batch size gave better performance.
\n\nWe decided to use LGBM to ensemble our models because it is fast and usually gives better results than other methods. We used same idea as Pavel Ostyakov described in his 5th place Cdiscount solution. We predicted top10 classes by our best network, concatenated probabilities of other networks, added class_id feature and gave binary label: whether the class_id is correct or not. This method resulted in validation score 0.001 higher than a simple average. So, if you have enough time, always try LGBM to ensemble.\nFinal submit with LGBM -> 0.948 LB
\n\nthanks for sharing.
\n\nfor CosineAnnealingLR, I wonder what parameters you use? I tried to use but I can't decide what to use for number of cycles, and iterations.
", "rawMarkdown": "thanks for sharing. \n\nfor CosineAnnealingLR, I wonder what parameters you use? I tried to use but I can't decide what to use for number of cycles, and iterations. ", "votes": 1, "replies": [ { "id": 437233, "postDate": "2018-12-11T15:29:54.807Z", "content": "Hi! I have used SGD optimizer, starting from 0.1 LR. Scheduler params were: T_max=7, eta_min=0.001. I divided the whole dataset on 70 epochs for annealing stage.
", "rawMarkdown": "Hi! I have used SGD optimizer, starting from 0.1 LR. Scheduler params were: T_max=7, eta_min=0.001. I divided the whole dataset on 70 epochs for annealing stage." }, { "id": 437425, "postDate": "2018-12-11T22:24:30.270Z", "content": "Thanks for sharing. how do you use etamin in your implementation? do you use min(etamin, eta)?
\n\nalso, min LR 0.001 seems really big..
", "rawMarkdown": "Thanks for sharing. how do you use etamin in your implementation? do you use min(etamin, eta)? \n\nalso, min LR 0.001 seems really big.. \n\n" } ] }, { "id": 434283, "postDate": "2018-12-06T06:44:34.363Z", "content": "Hi, thanks for your sharing. Would you please explain your LGBM method in detail? How did you get training data for \n LGBM? How did you do in the inference stage? Thanks.
", "rawMarkdown": "Hi, thanks for your sharing. Would you please explain your LGBM method in detail? How did you get training data for \n LGBM? How did you do in the inference stage? Thanks. ", "votes": 1, "replies": [ { "id": 437238, "postDate": "2018-12-11T15:37:28.493Z", "content": "Hi! Sorry for such a long answer. At first, I took my best network and predicted validation with it. Then, I took only top10 classes by probability of prediction. I concatenated predictions of other networks and added class_id as a feature. Target feature was is it true, that target_id is class_id. So I got binary classification problem and for every sample from validation and test I got 10 samples. In the inference stage I just took top3 probabilities of predictions of my LGBM.
", "rawMarkdown": "Hi! Sorry for such a long answer. At first, I took my best network and predicted validation with it. Then, I took only top10 classes by probability of prediction. I concatenated predictions of other networks and added **class_id** as a feature. Target feature was **is it true, that target_id is class_id**. So I got binary classification problem and for every sample from validation and test I got 10 samples. In the inference stage I just took top3 probabilities of predictions of my LGBM.", "votes": 1 } ] }, { "id": 434344, "postDate": "2018-12-06T08:30:01.230Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1 } ], "comments": [ { "id": 435080, "author_name": "yimacs", "author_url": "", "post_date": "2018-12-07T13:19:20.077000", "content": "thanks for sharing.
\n\nfor CosineAnnealingLR, I wonder what parameters you use? I tried to use but I can't decide what to use for number of cycles, and iterations.
", "votes": 1, "replies": [ { "id": 437233, "author_name": "Pavel Tsai", "author_url": "", "post_date": "2018-12-11T15:29:54.807000", "content": "Hi! I have used SGD optimizer, starting from 0.1 LR. Scheduler params were: T_max=7, eta_min=0.001. I divided the whole dataset on 70 epochs for annealing stage.
", "votes": 0, "replies": [] }, { "id": 437425, "author_name": "yimacs", "author_url": "", "post_date": "2018-12-11T22:24:30.270000", "content": "Thanks for sharing. how do you use etamin in your implementation? do you use min(etamin, eta)?
\n\nalso, min LR 0.001 seems really big..
", "votes": 0, "replies": [] } ] }, { "id": 434283, "author_name": "good good study", "author_url": "", "post_date": "2018-12-06T06:44:34.363000", "content": "Hi, thanks for your sharing. Would you please explain your LGBM method in detail? How did you get training data for \n LGBM? How did you do in the inference stage? Thanks.
", "votes": 1, "replies": [ { "id": 437238, "author_name": "Pavel Tsai", "author_url": "", "post_date": "2018-12-11T15:37:28.493000", "content": "Hi! Sorry for such a long answer. At first, I took my best network and predicted validation with it. Then, I took only top10 classes by probability of prediction. I concatenated predictions of other networks and added class_id as a feature. Target feature was is it true, that target_id is class_id. So I got binary classification problem and for every sample from validation and test I got 10 samples. In the inference stage I just took top3 probabilities of predictions of my LGBM.
", "votes": 1, "replies": [] } ] }, { "id": 434344, "author_name": "Tommy Jiang", "author_url": "", "post_date": "2018-12-06T08:30:01.230000", "content": "Thanks for sharing!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "433716": "Thanks everybody for the competition! The task was really interesting and had huge amounts of data. Thanks very much to my teammate who writes really perfect code on Python. You should understand me, he writes **comments** to functions and uses **types** in Python! Also thanks to [@scitator][1] for his perfect training ML framework [Catalyst][2], all models were trained with the help of it.\n \nNow I’d like to tell you about my and [Artyom Palvelev][3] solution using simplified data only.\n#data preprocessing#\nPictures of size 128x128 gave the best combination of score and training time. I don’t have time information, so I encoded the following data in three channels:\n\n 1. The index of line (linearly from 10 for the first line to 255 for the last one)\n 2. The number of strokes in the line\n 3. Just constant 255\n#Our models#\nFirstly, I tried models from forum like MobileNetV2, but they showed poor performance, so I trained something deeper. My first good model was SE_ResNext50 0.942 LB. I used CosineAnnealingLR and averaged top4 checkpoints.\nThen we merged with Artyom and tried different model architectures. Blend with his models gave 0.944 LB. In the final submission we had:\n\n1. SE_ResNext50 (~0.942LB)\n2. SE_ResNext101 (~0.944LB)\n3. NASNet-A-Large (~0.944LB)\n4. SENet154 (~0.945LB)\n5. CBAM_ResNet50 (~0.941LB)\n\nWe also used *gradient accumulation* to increase batch size to 1024 because data was noisy and bigger batch size gave better performance.\n#LGBM#\nWe decided to use LGBM to ensemble our models because it is fast and usually gives better results than other methods. We used same idea as [Pavel Ostyakov][4] described in his [5th place Cdiscount solution][5]. We predicted top10 classes by our best network, concatenated probabilities of other networks, added class_id feature and gave binary label: whether the class_id is correct or not. This method resulted in validation score 0.001 higher than a simple average. So, if you have enough time, always try LGBM to ensemble.\nFinal submit with LGBM -> 0.948 LB\n#What we tried and didn’t work#\n\n- CatBoost, RF, XGBoost, ensemble of 3rd level\n- Tuning models on clean data. We predicted the train dataset with our best models and dropped out pictures with small probability for the correct class (about 1M samples). It gave small boost on validation, but we didn’t have enough time, so we only trained a couple of models for about 2-3 epochs. So, I suppose, it is also a good idea for noisy data.\n\n#What we didn’t try but it worked#\n\n1. We didn’t noticed that the test is balanced (we could use same technique as in [Camera Model Identification][6] and get gold) Others say, it give plus ~0.7% score to any submission.\n2. TTA with deleting 20% strokes (people on forum said it also improved score). I tried just light augmentations like flips and shift_scale_rotate but they made score on validation even worse.\n\n\n [1]: https://www.kaggle.com/scitator\n [2]: https://github.com/Scitator/catalyst\n [3]: https://www.kaggle.com/artyomp\n [4]: https://www.kaggle.com/pavelost\n [5]: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733\n [6]: https://www.kaggle.com/c/sp-society-camera-model-identification", "435080": "thanks for sharing. \n\nfor CosineAnnealingLR, I wonder what parameters you use? I tried to use but I can't decide what to use for number of cycles, and iterations. ", "434283": "Hi, thanks for your sharing. Would you please explain your LGBM method in detail? How did you get training data for \n LGBM? How did you do in the inference stage? Thanks. ", "434344": "Thanks for sharing!" } }