{ "id": 73803, "title": "Predictions balancing: source code", "url": "/competitions/quickdraw-doodle-recognition/discussion/73803", "author_name": "Pavel Ostyakov", "post_date": "2018-12-05T16:52:34.264000", "votes": 76, "comment_count": 18, "views": 0, "content": "

Hey guys,

\n\n

I've just published my code for balancing predictions: https://github.com/PavelOstyakov/predictions_balancing

\n\n

It gives ~0.7% boost for any submission. Feel free to use it in future competitions :)

\n\n

\"enter

", "messages": [ { "id": 433924, "postDate": "2018-12-05T16:52:34.263Z", "content": "

Hey guys,

\n\n

I've just published my code for balancing predictions: https://github.com/PavelOstyakov/predictions_balancing

\n\n

It gives ~0.7% boost for any submission. Feel free to use it in future competitions :)

\n\n

\"enter

", "rawMarkdown": "Hey guys,\n\nI've just published my code for balancing predictions: https://github.com/PavelOstyakov/predictions_balancing\n\nIt gives ~0.7% boost for any submission. Feel free to use it in future competitions :)\n\n![enter image description here][1]\n\n\n [1]: https://sd.keepcalm-o-matic.co.uk/i-w600/keep-calm-and-balance-your-predictions.jpg", "votes": 76 }, { "id": 434108, "postDate": "2018-12-05T22:41:43.257Z", "content": "

thanks for the code!

\n\n

any idea what keyword i should use for goggle search for similar methods?

", "rawMarkdown": "thanks for the code!\n\nany idea what keyword i should use for goggle search for similar methods? ", "votes": 3 }, { "id": 434505, "postDate": "2018-12-06T14:14:34.537Z", "content": "

Hi,\nI am new to competitive data science and Kaggle. Could anyone explain what balancing of predictions mean?

", "rawMarkdown": "Hi,\nI am new to competitive data science and Kaggle. Could anyone explain what balancing of predictions mean?", "votes": 4, "replies": [ { "id": 435494, "postDate": "2018-12-08T05:34:20.063Z", "content": "

from https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73738

\n\n
\n

The algorithm behind postprocessing is the following: for the most popular class decrease all the probabilities iteratively by the same small value until it is no longer the most popular, repeat this procedure until all classes become equal. This technique was also used in one of the previous competitions (see github link for the code).

\n
\n\n

This is the implementation of the explanation above. \nA variable alpha is the one he mentioned as \"the same small value\". In this case, they use 0.001 as initial alpha and iteratively halve the value when the score no longer improves until it gets 0.0001.

\n\n

Without balancing, for example if you directly use the prediction of your model, one class might be presented a few times more than minor classes. This results in worse score if the distribution of your prediction much differs from the class distribution of test set. To counter this, you can balance the probability of your prediction so that all classes are equally presented.

\n\n

This kind of balancing works when you know the distribution of test set as a prior knowledge which is the case for this competition. According to inversion, the claases were actually not evenly balanced but I think it's still fairly balanced and that's why it worked very well.

", "rawMarkdown": "from https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73738\n\n> The algorithm behind postprocessing is the following: for the most popular class decrease all the probabilities iteratively by the same small value until it is no longer the most popular, repeat this procedure until all classes become equal. This technique was also used in one of the previous competitions (see github link for the code).\n\nThis is the implementation of the explanation above. \nA variable alpha is the one he mentioned as \"the same small value\". In this case, they use 0.001 as initial alpha and iteratively halve the value when the score no longer improves until it gets 0.0001.\n\nWithout balancing, for example if you directly use the prediction of your model, one class might be presented a few times more than minor classes. This results in worse score if the distribution of your prediction much differs from the class distribution of test set. To counter this, you can balance the probability of your prediction so that all classes are equally presented.\n\nThis kind of balancing works when you know the distribution of test set as a prior knowledge which is the case for this competition. According to inversion, the claases were actually not evenly balanced but I think it's still fairly balanced and that's why it worked very well.\n", "votes": 6 }, { "id": 435646, "postDate": "2018-12-08T13:04:20.973Z", "content": "

Thanks for the nice explanation. By \"all classes become equal\" does it mean that we know all classes are balanced (i.e. equal probabilities)? So if in our prior knowledge the classes are not balanced, we also tune the output distributions to match this unbalanced distribution?

", "rawMarkdown": "Thanks for the nice explanation. By \"all classes become equal\" does it mean that we know all classes are balanced (i.e. equal probabilities)? So if in our prior knowledge the classes are not balanced, we also tune the output distributions to match this unbalanced distribution? ", "votes": 1 }, { "id": 435906, "postDate": "2018-12-09T03:27:36.423Z", "content": "

Yes. Modifying a few lines such as torch.ones() which address equally balanced label distribution to something else should achive what you want.

", "rawMarkdown": "Yes. Modifying a few lines such as torch.ones() which address equally balanced label distribution to something else should achive what you want.", "votes": 2 }, { "id": 436499, "postDate": "2018-12-10T12:33:02.967Z", "content": "

Thanks a lot for that great explanation!!

", "rawMarkdown": "Thanks a lot for that great explanation!!\n\n", "votes": 1 }, { "id": 437915, "postDate": "2018-12-12T18:48:46.837Z", "content": "

thanks for the explanation, this is extremely helpful

", "rawMarkdown": "thanks for the explanation, this is extremely helpful" } ] }, { "id": 434158, "postDate": "2018-12-06T01:24:19.117Z", "content": "

Awesome!! After applying this algorithm, my lb score boosts from Private LB 0.94974 to 0.95467.\nThanks for the code! It will be very useful for the future competitions.

", "rawMarkdown": "Awesome!! After applying this algorithm, my lb score boosts from **Private LB 0.94974** to **0.95467**.\nThanks for the code! It will be very useful for the future competitions.", "votes": 2, "replies": [ { "id": 434357, "postDate": "2018-12-06T09:13:29.817Z", "content": "

You are welcome!

", "rawMarkdown": "You are welcome!" } ] }, { "id": 434114, "postDate": "2018-12-05T22:50:53.877Z", "content": "

here is probably one related paper:\n\"Probabilistic n-Choose-k Models for Classification and Ranking\"

\n\n

http://www.cs.toronto.edu/~kswersky/wp-content/uploads/pnck.pdf

\n\n

\"Instead, we focus on work related to the main novelty in this paper,\nthe explicit modeling of structure on label counts. That is, given that we have prior knowledge of\nlabel count structure, or are modeling a domain that exhibits such structure, the question is how can\nthe structure be leveraged to improve a model.\"

", "rawMarkdown": "here is probably one related paper:\n\"Probabilistic n-Choose-k Models for Classification and Ranking\"\n\nhttp://www.cs.toronto.edu/~kswersky/wp-content/uploads/pnck.pdf\n\n\"Instead, we focus on work related to the main novelty in this paper,\nthe explicit modeling of structure on label counts. That is, given that we have prior knowledge of\nlabel count structure, or are modeling a domain that exhibits such structure, the question is how can\nthe structure be leveraged to improve a model.\"", "votes": 2, "replies": [ { "id": 434359, "postDate": "2018-12-06T09:13:54.677Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing " } ] }, { "id": 438270, "postDate": "2018-12-13T11:49:21.193Z", "content": "

GOOD JOB!

", "rawMarkdown": "GOOD JOB!" }, { "id": 435233, "postDate": "2018-12-07T18:08:51.103Z", "content": "

For what it's worth, classes in the test set were not evenly balanced. But, obviously, they were not highly skewed either. :-)

", "rawMarkdown": "For what it's worth, classes in the test set were not evenly balanced. But, obviously, they were not highly skewed either. :-)", "replies": [ { "id": 435236, "postDate": "2018-12-07T18:12:20.763Z", "content": "

That's interesting! Could I ask you to provide a distribution of classes in the test set? I may try to improve my script for this competition :)

", "rawMarkdown": "That's interesting! Could I ask you to provide a distribution of classes in the test set? I may try to improve my script for this competition :)" } ] }, { "id": 434869, "postDate": "2018-12-07T04:47:04.233Z", "content": "

Amazing and extremely simple. I'm really wondering how it works? Thanks for your explanation in advance.

", "rawMarkdown": "Amazing and extremely simple. I'm really wondering how it works? Thanks for your explanation in advance." }, { "id": 434523, "postDate": "2018-12-06T14:33:06.130Z", "content": "

good job!

", "rawMarkdown": "good job!" }, { "id": 434194, "postDate": "2018-12-06T03:22:54.950Z", "rawMarkdown": "", "votes": 8, "isDeleted": true }, { "id": 434199, "postDate": "2018-12-06T03:31:02.783Z", "content": "

Many Thanks Pavel!

", "rawMarkdown": "Many Thanks Pavel!", "votes": 7 } ], "comments": [ { "id": 434108, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-05T22:41:43.257000", "content": "

thanks for the code!

\n\n

any idea what keyword i should use for goggle search for similar methods?

", "votes": 3, "replies": [] }, { "id": 434505, "author_name": "Sreyan Ghosh", "author_url": "", "post_date": "2018-12-06T14:14:34.537000", "content": "

Hi,\nI am new to competitive data science and Kaggle. Could anyone explain what balancing of predictions mean?

", "votes": 4, "replies": [ { "id": 435494, "author_name": "Appian", "author_url": "", "post_date": "2018-12-08T05:34:20.063000", "content": "

from https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73738

\n\n
\n

The algorithm behind postprocessing is the following: for the most popular class decrease all the probabilities iteratively by the same small value until it is no longer the most popular, repeat this procedure until all classes become equal. This technique was also used in one of the previous competitions (see github link for the code).

\n
\n\n

This is the implementation of the explanation above. \nA variable alpha is the one he mentioned as \"the same small value\". In this case, they use 0.001 as initial alpha and iteratively halve the value when the score no longer improves until it gets 0.0001.

\n\n

Without balancing, for example if you directly use the prediction of your model, one class might be presented a few times more than minor classes. This results in worse score if the distribution of your prediction much differs from the class distribution of test set. To counter this, you can balance the probability of your prediction so that all classes are equally presented.

\n\n

This kind of balancing works when you know the distribution of test set as a prior knowledge which is the case for this competition. According to inversion, the claases were actually not evenly balanced but I think it's still fairly balanced and that's why it worked very well.

", "votes": 6, "replies": [] }, { "id": 435646, "author_name": "Shaohua Li", "author_url": "", "post_date": "2018-12-08T13:04:20.973000", "content": "

Thanks for the nice explanation. By \"all classes become equal\" does it mean that we know all classes are balanced (i.e. equal probabilities)? So if in our prior knowledge the classes are not balanced, we also tune the output distributions to match this unbalanced distribution?

", "votes": 1, "replies": [] }, { "id": 435906, "author_name": "Appian", "author_url": "", "post_date": "2018-12-09T03:27:36.423000", "content": "

Yes. Modifying a few lines such as torch.ones() which address equally balanced label distribution to something else should achive what you want.

", "votes": 2, "replies": [] }, { "id": 436499, "author_name": "Sreyan Ghosh", "author_url": "", "post_date": "2018-12-10T12:33:02.967000", "content": "

Thanks a lot for that great explanation!!

", "votes": 1, "replies": [] }, { "id": 437915, "author_name": "Stan Chen", "author_url": "", "post_date": "2018-12-12T18:48:46.837000", "content": "

thanks for the explanation, this is extremely helpful

", "votes": 0, "replies": [] } ] }, { "id": 434158, "author_name": "pudae", "author_url": "", "post_date": "2018-12-06T01:24:19.117000", "content": "

Awesome!! After applying this algorithm, my lb score boosts from Private LB 0.94974 to 0.95467.\nThanks for the code! It will be very useful for the future competitions.

", "votes": 2, "replies": [ { "id": 434357, "author_name": "Pavel Ostyakov", "author_url": "", "post_date": "2018-12-06T09:13:29.817000", "content": "

You are welcome!

", "votes": 0, "replies": [] } ] }, { "id": 434114, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-05T22:50:53.877000", "content": "

here is probably one related paper:\n\"Probabilistic n-Choose-k Models for Classification and Ranking\"

\n\n

http://www.cs.toronto.edu/~kswersky/wp-content/uploads/pnck.pdf

\n\n

\"Instead, we focus on work related to the main novelty in this paper,\nthe explicit modeling of structure on label counts. That is, given that we have prior knowledge of\nlabel count structure, or are modeling a domain that exhibits such structure, the question is how can\nthe structure be leveraged to improve a model.\"

", "votes": 2, "replies": [ { "id": 434359, "author_name": "Pavel Ostyakov", "author_url": "", "post_date": "2018-12-06T09:13:54.677000", "content": "

Thanks for sharing

", "votes": 0, "replies": [] } ] }, { "id": 438270, "author_name": "Bear Guan", "author_url": "", "post_date": "2018-12-13T11:49:21.193000", "content": "

GOOD JOB!

", "votes": 0, "replies": [] }, { "id": 435233, "author_name": "inversion", "author_url": "", "post_date": "2018-12-07T18:08:51.103000", "content": "

For what it's worth, classes in the test set were not evenly balanced. But, obviously, they were not highly skewed either. :-)

", "votes": 0, "replies": [ { "id": 435236, "author_name": "Pavel Ostyakov", "author_url": "", "post_date": "2018-12-07T18:12:20.763000", "content": "

That's interesting! Could I ask you to provide a distribution of classes in the test set? I may try to improve my script for this competition :)

", "votes": 0, "replies": [] } ] }, { "id": 434869, "author_name": "Shaohua Li", "author_url": "", "post_date": "2018-12-07T04:47:04.233000", "content": "

Amazing and extremely simple. I'm really wondering how it works? Thanks for your explanation in advance.

", "votes": 0, "replies": [] }, { "id": 434523, "author_name": "天大狂徒", "author_url": "", "post_date": "2018-12-06T14:33:06.130000", "content": "

good job!

", "votes": 0, "replies": [] }, { "id": 434194, "author_name": "", "author_url": "", "post_date": "2018-12-06T03:22:54.950000", "content": "", "votes": 8, "replies": [] }, { "id": 434199, "author_name": "", "author_url": "", "post_date": "2018-12-06T03:31:02.783000", "content": "

Many Thanks Pavel!

", "votes": 7, "replies": [] } ], "raw_markdown_by_id": { "433924": "Hey guys,\n\nI've just published my code for balancing predictions: https://github.com/PavelOstyakov/predictions_balancing\n\nIt gives ~0.7% boost for any submission. Feel free to use it in future competitions :)\n\n![enter image description here][1]\n\n\n [1]: https://sd.keepcalm-o-matic.co.uk/i-w600/keep-calm-and-balance-your-predictions.jpg", "434108": "thanks for the code!\n\nany idea what keyword i should use for goggle search for similar methods? ", "434505": "Hi,\nI am new to competitive data science and Kaggle. Could anyone explain what balancing of predictions mean?", "434158": "Awesome!! After applying this algorithm, my lb score boosts from **Private LB 0.94974** to **0.95467**.\nThanks for the code! It will be very useful for the future competitions.", "434114": "here is probably one related paper:\n\"Probabilistic n-Choose-k Models for Classification and Ranking\"\n\nhttp://www.cs.toronto.edu/~kswersky/wp-content/uploads/pnck.pdf\n\n\"Instead, we focus on work related to the main novelty in this paper,\nthe explicit modeling of structure on label counts. That is, given that we have prior knowledge of\nlabel count structure, or are modeling a domain that exhibits such structure, the question is how can\nthe structure be leveraged to improve a model.\"", "438270": "GOOD JOB!", "435233": "For what it's worth, classes in the test set were not evenly balanced. But, obviously, they were not highly skewed either. :-)", "434869": "Amazing and extremely simple. I'm really wondering how it works? Thanks for your explanation in advance.", "434523": "good job!", "434194": "", "434199": "Many Thanks Pavel!" } }