I've analyzed the prediction errors. I've noticed that many submission files, even though they contain the correct column values and data types, return \"failed\" during scoring. Do you perhaps know why this is the case?
\nI've noticed in the metric code:
\nlabel_group_losses.append(\n sklearn.metrics.log_loss(\n y_true=solution[col_group].values,\n y_pred=submission[col_group].values,\n sample_weight=solution[].values))\nthat the log loss is calculated for 'col_groups'. This variable, however, is derived as follows:
\nbinary_targets = [, ]\ntriple_level_targets = [, , ]\nall_target_categories = binary_targets + triple_level_targets\nlabel_group_losses = []\n category all_target_categories:\n category binary_targets:\n col_group = [, ]\n :\n col_group = [, , ]\nSo, we have the following values for col_group:
\n['bowel_healthy', 'bowel_injury']
\n['extravasation_healthy', 'extravasation_injury']
\n['kidney_healthy', 'kidney_low', 'kidney_high']
\n['liver_healthy', 'liver_low', 'liver_high']
\n['spleen_healthy', 'spleen_low', 'spleen_high']
With sklearn.metrics.log_loss, you can't compute the log loss for multiple columns at once. We get the error: ValueError: Multioutput target data is not supported with label binarization. Could the reason be somewhere else? Why are the averaged results often okay, but the ones obtained from the model result in 'failed'?
", "messages": [ { "id": 2468307, "postDate": "2023-10-05T11:42:47.790Z", "content": "I get the correct scoring, for example, on randomly generated data.
\nWhy does it work here on data that isn't averaged for all patients?
", "rawMarkdown": "I get the correct scoring, for example, on randomly generated data.\n\nWhy does it work here on data that isn't averaged for all patients?", "votes": 3, "replies": [ { "id": 2469687, "postDate": "2023-10-06T14:16:09.783Z", "content": "Same issue here. I'm kinda stuck in a dead-end. I made predictions from the models for 3 test cases. Even when I upload the csv file in a separate notebook it still gives a calculation error. My columns are exact match to what RSNA gives as submission_sample and all the probabilities for each organ sums to 1. Still don't understand where I'm doing wrong.
", "rawMarkdown": "Same issue here. I'm kinda stuck in a dead-end. I made predictions from the models for 3 test cases. Even when I upload the csv file in a separate notebook it still gives a calculation error. My columns are exact match to what RSNA gives as submission_sample and all the probabilities for each organ sums to 1. Still don't understand where I'm doing wrong.", "replies": [ { "id": 2469709, "postDate": "2023-10-06T14:45:06.913Z", "content": "@noelyoda I believe I resolved it. You need to do a calcualtion of the result (whether it is simply an average or a prediction from the model) in the notebook in which you post the result. In addition (of this I am not sure), perhaps a sample_submission should be loaded. See here: https://www.kaggle.com/code/jedrzejak/rsna-results
", "rawMarkdown": "@noelyoda I believe I resolved it. You need to do a calcualtion of the result (whether it is simply an average or a prediction from the model) in the notebook in which you post the result. In addition (of this I am not sure), perhaps a sample_submission should be loaded. See here: https://www.kaggle.com/code/jedrzejak/rsna-results", "votes": 4 }, { "id": 2471804, "postDate": "2023-10-06T16:32:03.073Z", "content": "Thank you so much for your return! Somehow that worked. I'm so confused what all I did is:
\nsample_submission.set_index(, inplace = )\nmy_original_submission.set_index(, inplace = )\n\nsample_submission.update(my_original_submission)\nsample_submission.reset_index(inplace = )\n\nsample_submission.to_csv(, index = )\nI still don't understand where I did wrong initially but anyway thanks to your notebook I can submit a score. I appreciate your help.
", "rawMarkdown": "Thank you so much for your return! Somehow that worked. I'm so confused what all I did is:\n\n```python\nsample_submission.set_index('patient_id', inplace = True)\nmy_original_submission.set_index('patient_id', inplace = True)\n\nsample_submission.update(my_original_submission)\nsample_submission.reset_index(inplace = True)\n\nsample_submission.to_csv('submission.csv', index = False)\n```\n\nI still don't understand where I did wrong initially but anyway thanks to your notebook I can submit a score. I appreciate your help." }, { "id": 2471927, "postDate": "2023-10-06T18:51:06.093Z", "content": "If I understand correctly, we need to provide a notebook that, after the competition ends:
\nAn additional problem is that we need to load the test set dcm images in a universal way from the path: /kaggle/input/rsna-2023-abdominal-trauma-detection/test_images/ (there will probably be many more images in dcm format there).
\nI modified the functions in the above notebook to be universal (without hard-coded paths).
", "rawMarkdown": "If I understand correctly, we need to provide a notebook that, after the competition ends:\n\n* will make a prediction on the test set (enlarged with hidden cases),\n* will load the sample_submission (enlarged with hidden cases),\n* will replace the results in the place of sample_submission.\n\nAn additional problem is that we need to load the test set dcm images in a universal way from the path: /kaggle/input/rsna-2023-abdominal-trauma-detection/test_images/ (there will probably be many more images in dcm format there).\n\nI modified the functions in the above notebook to be universal (without hard-coded paths).", "votes": 3 } ] } ] }, { "id": 2468227, "postDate": "2023-10-05T10:16:56.947Z", "content": "I've analyzed the prediction errors. I've noticed that many submission files, even though they contain the correct column values and data types, return \"failed\" during scoring. Do you perhaps know why this is the case?
\nI've noticed in the metric code:
\nlabel_group_losses.append(\n sklearn.metrics.log_loss(\n y_true=solution[col_group].values,\n y_pred=submission[col_group].values,\n sample_weight=solution[].values))\nthat the log loss is calculated for 'col_groups'. This variable, however, is derived as follows:
\nbinary_targets = [, ]\ntriple_level_targets = [, , ]\nall_target_categories = binary_targets + triple_level_targets\nlabel_group_losses = []\n category all_target_categories:\n category binary_targets:\n col_group = [, ]\n :\n col_group = [, , ]\nSo, we have the following values for col_group:
\n['bowel_healthy', 'bowel_injury']
\n['extravasation_healthy', 'extravasation_injury']
\n['kidney_healthy', 'kidney_low', 'kidney_high']
\n['liver_healthy', 'liver_low', 'liver_high']
\n['spleen_healthy', 'spleen_low', 'spleen_high']
With sklearn.metrics.log_loss, you can't compute the log loss for multiple columns at once. We get the error: ValueError: Multioutput target data is not supported with label binarization. Could the reason be somewhere else? Why are the averaged results often okay, but the ones obtained from the model result in 'failed'?
", "rawMarkdown": "I've analyzed the prediction errors. I've noticed that many submission files, even though they contain the correct column values and data types, return \"failed\" during scoring. Do you perhaps know why this is the case?\n\nI've noticed in the metric code:\n```python\nlabel_group_losses.append(\n sklearn.metrics.log_loss(\n y_true=solution[col_group].values,\n y_pred=submission[col_group].values,\n sample_weight=solution[f'{category}_weight'].values))\n```\n\nthat the log loss is calculated for 'col_groups'. This variable, however, is derived as follows:\n\n```python\nbinary_targets = ['bowel', 'extravasation']\ntriple_level_targets = ['kidney', 'liver', 'spleen']\nall_target_categories = binary_targets + triple_level_targets\nlabel_group_losses = []\n```\n```python\nfor category in all_target_categories:\n if category in binary_targets:\n col_group = [f'{category}_healthy', f'{category}_injury']\n else:\n col_group = [f'{category}_healthy', f'{category}_low', f'{category}_high']\n```\n\nSo, we have the following values for col_group:\n\n['bowel_healthy', 'bowel_injury']\n['extravasation_healthy', 'extravasation_injury']\n['kidney_healthy', 'kidney_low', 'kidney_high']\n['liver_healthy', 'liver_low', 'liver_high']\n['spleen_healthy', 'spleen_low', 'spleen_high']\n\nWith sklearn.metrics.log_loss, you can't compute the log loss for multiple columns at once. We get the error: ValueError: Multioutput target data is not supported with label binarization. Could the reason be somewhere else? Why are the averaged results often okay, but the ones obtained from the model result in 'failed'?", "votes": 4 }, { "id": 2469061, "postDate": "2023-10-06T05:42:16.473Z", "content": "You may want to look at Single corrupt image in the hidden test set
\nNot sure any fix on the data has happened so you may need to cater for it in your submissions if you have not already.
For the metric - Jake Brusca's notebook Weighted Mean Baseline might help to evaluate some of your models predictions.
", "rawMarkdown": "You may want to look at [Single corrupt image in the hidden test set](https://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/discussion/435815)\nNot sure any fix on the data has happened so you may need to cater for it in your submissions if you have not already.\n\nFor the metric - Jake Brusca's notebook [Weighted Mean Baseline](https://www.kaggle.com/code/jakebrusca/rsna23-weighted-mean-baseline) might help to evaluate some of your models predictions. ", "votes": 1, "replies": [ { "id": 2469429, "postDate": "2023-10-06T10:48:32.200Z", "content": "Ok, thanks, in fact, I was not aware of this damaged photo. I'll look it over.
", "rawMarkdown": "Ok, thanks, in fact, I was not aware of this damaged photo. I'll look it over.", "votes": 3 } ] }, { "id": 2468310, "postDate": "2023-10-05T11:46:32.703Z", "content": "I had a lot of failed submissions, but I suspect that most of them occurred while converting the dicom images to jpeg. I used the try except statement while converting and then I was able to successfully submit.
", "rawMarkdown": "I had a lot of failed submissions, but I suspect that most of them occurred while converting the dicom images to jpeg. I used the try except statement while converting and then I was able to successfully submit.", "votes": 1, "replies": [ { "id": 2468312, "postDate": "2023-10-05T11:54:07.993Z", "content": "Ok, but the result (from model) calculates correctly. We have 3 photos in test dataset, I have correct results for them. I simply attach a CSV file, even in a separate notebook. And what does this have to do with converting DICOM to PNG?
", "rawMarkdown": "Ok, but the result (from model) calculates correctly. We have 3 photos in test dataset, I have correct results for them. I simply attach a CSV file, even in a separate notebook. And what does this have to do with converting DICOM to PNG?\n\n", "votes": 1 } ] }, { "id": 2471814, "postDate": "2023-10-06T16:42:57.307Z", "content": "The error you're encountering with sklearn.metrics.log_loss is due to its limitation in handling multi-output data with label binarization. Averaging results may mitigate this issue, but understanding the root cause of model-specific failures is crucial. Appreciate an upvote if you found this insight helpful for troubleshooting.
", "rawMarkdown": "The error you're encountering with sklearn.metrics.log_loss is due to its limitation in handling multi-output data with label binarization. Averaging results may mitigate this issue, but understanding the root cause of model-specific failures is crucial. Appreciate an upvote if you found this insight helpful for troubleshooting.", "votes": 2, "replies": [ { "id": 2471819, "postDate": "2023-10-06T16:50:26.697Z", "content": "I tried that too but I actually can't understand what I should average. Do you mean that assigning only one variable to all triple class columns? Like:
\ntriple_class_labels = [, , ]\n\nsubmission[triple_class_labels] = submission[triple_class_labels].mean().tolist()\nI get the correct scoring, for example, on randomly generated data.
\nWhy does it work here on data that isn't averaged for all patients?
", "votes": 3, "replies": [ { "id": 2469687, "author_name": "Okan Ince", "author_url": "", "post_date": "2023-10-06T14:16:09.783000", "content": "Same issue here. I'm kinda stuck in a dead-end. I made predictions from the models for 3 test cases. Even when I upload the csv file in a separate notebook it still gives a calculation error. My columns are exact match to what RSNA gives as submission_sample and all the probabilities for each organ sums to 1. Still don't understand where I'm doing wrong.
", "votes": 0, "replies": [ { "id": 2469709, "author_name": "Kacper Jędrczak", "author_url": "", "post_date": "2023-10-06T14:45:06.913000", "content": "@noelyoda I believe I resolved it. You need to do a calcualtion of the result (whether it is simply an average or a prediction from the model) in the notebook in which you post the result. In addition (of this I am not sure), perhaps a sample_submission should be loaded. See here: https://www.kaggle.com/code/jedrzejak/rsna-results
", "votes": 4, "replies": [] }, { "id": 2471804, "author_name": "Okan Ince", "author_url": "", "post_date": "2023-10-06T16:32:03.073000", "content": "Thank you so much for your return! Somehow that worked. I'm so confused what all I did is:
\nsample_submission.set_index(, inplace = )\nmy_original_submission.set_index(, inplace = )\n\nsample_submission.update(my_original_submission)\nsample_submission.reset_index(inplace = )\n\nsample_submission.to_csv(, index = )\nI still don't understand where I did wrong initially but anyway thanks to your notebook I can submit a score. I appreciate your help.
", "votes": 0, "replies": [] }, { "id": 2471927, "author_name": "Kacper Jędrczak", "author_url": "", "post_date": "2023-10-06T18:51:06.093000", "content": "If I understand correctly, we need to provide a notebook that, after the competition ends:
\nAn additional problem is that we need to load the test set dcm images in a universal way from the path: /kaggle/input/rsna-2023-abdominal-trauma-detection/test_images/ (there will probably be many more images in dcm format there).
\nI modified the functions in the above notebook to be universal (without hard-coded paths).
", "votes": 3, "replies": [] } ] } ] }, { "id": 2469061, "author_name": "something4kag", "author_url": "", "post_date": "2023-10-06T05:42:16.473000", "content": "You may want to look at Single corrupt image in the hidden test set
\nNot sure any fix on the data has happened so you may need to cater for it in your submissions if you have not already.
For the metric - Jake Brusca's notebook Weighted Mean Baseline might help to evaluate some of your models predictions.
", "votes": 1, "replies": [ { "id": 2469429, "author_name": "Kacper Jędrczak", "author_url": "", "post_date": "2023-10-06T10:48:32.200000", "content": "Ok, thanks, in fact, I was not aware of this damaged photo. I'll look it over.
", "votes": 3, "replies": [] } ] }, { "id": 2468310, "author_name": "Aakash Rana", "author_url": "", "post_date": "2023-10-05T11:46:32.703000", "content": "I had a lot of failed submissions, but I suspect that most of them occurred while converting the dicom images to jpeg. I used the try except statement while converting and then I was able to successfully submit.
", "votes": 1, "replies": [ { "id": 2468312, "author_name": "Kacper Jędrczak", "author_url": "", "post_date": "2023-10-05T11:54:07.993000", "content": "Ok, but the result (from model) calculates correctly. We have 3 photos in test dataset, I have correct results for them. I simply attach a CSV file, even in a separate notebook. And what does this have to do with converting DICOM to PNG?
", "votes": 1, "replies": [] } ] }, { "id": 2471814, "author_name": "Bhavesh Padharia", "author_url": "", "post_date": "2023-10-06T16:42:57.307000", "content": "The error you're encountering with sklearn.metrics.log_loss is due to its limitation in handling multi-output data with label binarization. Averaging results may mitigate this issue, but understanding the root cause of model-specific failures is crucial. Appreciate an upvote if you found this insight helpful for troubleshooting.
", "votes": 2, "replies": [ { "id": 2471819, "author_name": "Okan Ince", "author_url": "", "post_date": "2023-10-06T16:50:26.697000", "content": "I tried that too but I actually can't understand what I should average. Do you mean that assigning only one variable to all triple class columns? Like:
\ntriple_class_labels = [, , ]\n\nsubmission[triple_class_labels] = submission[triple_class_labels].mean().tolist()\n