Most of the shared 'mean' notebooks are using weights that take the predictions what seems like a long way from the training means.
\nFor example,
\nThis one seems to suggest that extravasation injuries will be 66% in the test set vs the mean from training of 6%.
\nOne of the early shared mean notebooks had extravasation at 51% of the test set.
\nDoes anyone believe the test set will have such high levels of extravasation?
\nI don't believe it - so does this mean that the metric has been poorly designed? Or are we just chasing noise with weighted mean notebook submissions?
", "messages": [ { "id": 2420498, "postDate": "2023-09-02T16:34:43.877Z", "content": "Most of the shared 'mean' notebooks are using weights that take the predictions what seems like a long way from the training means.
\nFor example,
\nThis one seems to suggest that extravasation injuries will be 66% in the test set vs the mean from training of 6%.
\nOne of the early shared mean notebooks had extravasation at 51% of the test set.
\nDoes anyone believe the test set will have such high levels of extravasation?
\nI don't believe it - so does this mean that the metric has been poorly designed? Or are we just chasing noise with weighted mean notebook submissions?
", "rawMarkdown": "Most of the shared 'mean' notebooks are using weights that take the predictions what seems like a long way from the training means.\n\nFor example, \n\n[This one](https://www.kaggle.com/code/jasonheesanglee/rsna23-weighted-mean-baseline-57c31e) seems to suggest that extravasation injuries will be 66% in the test set vs the mean from training of 6%. \n\nOne of the [early shared mean notebooks](https://www.kaggle.com/code/vishakkbhat/rsna23-weighted-mean-baseline/output) had extravasation at 51% of the test set.\n\nDoes anyone believe the test set will have such high levels of extravasation?\n\nI don't believe it - so does this mean that the metric has been poorly designed? Or are we just chasing noise with weighted mean notebook submissions?\n\n", "votes": 3 }, { "id": 2428891, "postDate": "2023-09-08T08:24:57.670Z", "content": "The optimal constant prediction (independently of all other targets) for extravasation_injury, given the training data, and given the weighted cross entropy metric, is approx 0.29 (with extravasation_healthy approx 0.71).
What complicates matters is that the any_injury class is generated from the max of the (non-healthy) class predictions. If extravasation_injury is scaled up, then this will help contribute to the any_injury predictions (that class is 1 when extravasation_injury is 1 by definition).
I don't know whether everyone attempting to parameter-tune the weighted mean to the public LB believes that there will be high levels of extravasation, but, up to a point it makes sense to scale it beyond its propensity.
", "rawMarkdown": "The optimal constant prediction (independently of all other targets) for `extravasation_injury`, given the training data, and given the weighted cross entropy metric, is approx 0.29 (with `extravasation_healthy` approx 0.71).\n\nWhat complicates matters is that the `any_injury` class is generated from the max of the (non-healthy) class predictions. If `extravasation_injury` is scaled up, then this will help contribute to the `any_injury` predictions (that class is 1 when `extravasation_injury` is 1 by definition).\n\nI don't know whether everyone attempting to parameter-tune the weighted mean to the public LB believes that there will be high levels of extravasation, but, up to a point it makes sense to scale it beyond its propensity." } ], "comments": [ { "id": 2428891, "author_name": "jagofc", "author_url": "", "post_date": "2023-09-08T08:24:57.670000", "content": "The optimal constant prediction (independently of all other targets) for extravasation_injury, given the training data, and given the weighted cross entropy metric, is approx 0.29 (with extravasation_healthy approx 0.71).
What complicates matters is that the any_injury class is generated from the max of the (non-healthy) class predictions. If extravasation_injury is scaled up, then this will help contribute to the any_injury predictions (that class is 1 when extravasation_injury is 1 by definition).
I don't know whether everyone attempting to parameter-tune the weighted mean to the public LB believes that there will be high levels of extravasation, but, up to a point it makes sense to scale it beyond its propensity.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2420498": "Most of the shared 'mean' notebooks are using weights that take the predictions what seems like a long way from the training means.\n\nFor example, \n\n[This one](https://www.kaggle.com/code/jasonheesanglee/rsna23-weighted-mean-baseline-57c31e) seems to suggest that extravasation injuries will be 66% in the test set vs the mean from training of 6%. \n\nOne of the [early shared mean notebooks](https://www.kaggle.com/code/vishakkbhat/rsna23-weighted-mean-baseline/output) had extravasation at 51% of the test set.\n\nDoes anyone believe the test set will have such high levels of extravasation?\n\nI don't believe it - so does this mean that the metric has been poorly designed? Or are we just chasing noise with weighted mean notebook submissions?\n\n", "2428891": "The optimal constant prediction (independently of all other targets) for `extravasation_injury`, given the training data, and given the weighted cross entropy metric, is approx 0.29 (with `extravasation_healthy` approx 0.71).\n\nWhat complicates matters is that the `any_injury` class is generated from the max of the (non-healthy) class predictions. If `extravasation_injury` is scaled up, then this will help contribute to the `any_injury` predictions (that class is 1 when `extravasation_injury` is 1 by definition).\n\nI don't know whether everyone attempting to parameter-tune the weighted mean to the public LB believes that there will be high levels of extravasation, but, up to a point it makes sense to scale it beyond its propensity." } }