The link to the dataset takes us to this article:
\n\n\nAssociation Between Clot Composition and Stroke Origin in Mechanical Thrombectomy Patients: Analysis of the Stroke Thromboembolism Registry of Imaging and Pathology
\n
Based on my understanding of this paper, the authors effectively worked on the same task: try to identify the etiology of the blood clot (classify into LAA or CE) from the histological data. They use Orbit image analysis to extract features and H2O.ai for ML classification.
\nHere are some relevant quotes from the paper:
\n\n\nOn ROC analysis, the AUC for RBC density in differentiating CE from LAA clots was 0.55; the AUC for WBC density was 0.50; the AUC for fibrin density was 0.52; the AUC for platelet density was 0.55.
\nThe stacked ensemble for the classification algorithm differentiating CE from LAA had a 5-fold cross-validated AUC of 0.55 (Area under the precision-recall curve of 0.33).
\nOn ROC analysis, it seems that identification of a reliable threshold with a high AUC for differentiating clots from these two etiologies based on composition analysis alone is not possible. The lack of a reliable histological biomarker on MSB between these two stroke etiologies suggests that conventional histological analyses looking at cellular composition does not provide insights into stroke etiology in cryptogenic cases.
\nOverall, the findings from our study and the disparate results from multiple prior studies suggest that routine histological staining with H&E or MSB probably will not allow us to differentiate clots of different etiologies
\n
From this, my understanding is that the study failed to classify between CE and LAA clots and the study itself suggests that future work should focus on other approaches (like molecular testing and immunohistochemistry).
\nSo it would be great to get this confirmed from the organizers:
\nIs there any strong evidence that there is actual signal in the data?
It's very much possible there is. Being someone who is familiar with computational pathology literature, I know there is a surprising amount of information you can get from histology with the help of deep learning. But it is also possible that no such signal is available in this type of data, and appropriate biomarkers only show up in other sort of tests and analyses (again, most likely molecular testing or immunohistochemistry).
\nIn any case, it would be helpful for participants to know if it has been previously been established or not if signal can be found in this dataset…
\nMaybe I am misunderstanding something, in which case I would appreciate a correction… Hope to get some answers from the organizers :)
", "messages": [ { "id": 1850228, "postDate": "2022-07-10T08:25:31.067Z", "content": "The link to the dataset takes us to this article:
\n\n\nAssociation Between Clot Composition and Stroke Origin in Mechanical Thrombectomy Patients: Analysis of the Stroke Thromboembolism Registry of Imaging and Pathology
\n
Based on my understanding of this paper, the authors effectively worked on the same task: try to identify the etiology of the blood clot (classify into LAA or CE) from the histological data. They use Orbit image analysis to extract features and H2O.ai for ML classification.
\nHere are some relevant quotes from the paper:
\n\n\nOn ROC analysis, the AUC for RBC density in differentiating CE from LAA clots was 0.55; the AUC for WBC density was 0.50; the AUC for fibrin density was 0.52; the AUC for platelet density was 0.55.
\nThe stacked ensemble for the classification algorithm differentiating CE from LAA had a 5-fold cross-validated AUC of 0.55 (Area under the precision-recall curve of 0.33).
\nOn ROC analysis, it seems that identification of a reliable threshold with a high AUC for differentiating clots from these two etiologies based on composition analysis alone is not possible. The lack of a reliable histological biomarker on MSB between these two stroke etiologies suggests that conventional histological analyses looking at cellular composition does not provide insights into stroke etiology in cryptogenic cases.
\nOverall, the findings from our study and the disparate results from multiple prior studies suggest that routine histological staining with H&E or MSB probably will not allow us to differentiate clots of different etiologies
\n
From this, my understanding is that the study failed to classify between CE and LAA clots and the study itself suggests that future work should focus on other approaches (like molecular testing and immunohistochemistry).
\nSo it would be great to get this confirmed from the organizers:
\nIs there any strong evidence that there is actual signal in the data?
It's very much possible there is. Being someone who is familiar with computational pathology literature, I know there is a surprising amount of information you can get from histology with the help of deep learning. But it is also possible that no such signal is available in this type of data, and appropriate biomarkers only show up in other sort of tests and analyses (again, most likely molecular testing or immunohistochemistry).
\nIn any case, it would be helpful for participants to know if it has been previously been established or not if signal can be found in this dataset…
\nMaybe I am misunderstanding something, in which case I would appreciate a correction… Hope to get some answers from the organizers :)
", "rawMarkdown": "The link to the dataset takes us to [this article](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9069417/):\n> Association Between Clot Composition and Stroke Origin in Mechanical Thrombectomy Patients: Analysis of the Stroke Thromboembolism Registry of Imaging and Pathology\n\nBased on my understanding of this paper, the authors effectively worked on the same task: try to identify the etiology of the blood clot (classify into LAA or CE) from the histological data. They use Orbit image analysis to extract features and H2O.ai for ML classification.\n\nHere are some relevant quotes from the paper:\n> On ROC analysis, the AUC for RBC density in differentiating CE from LAA clots was 0.55; the AUC for WBC density was 0.50; the AUC for fibrin density was 0.52; the AUC for platelet density was 0.55.\n\n> The stacked ensemble for the classification algorithm differentiating CE from LAA had a 5-fold cross-validated AUC of 0.55 (Area under the precision-recall curve of 0.33).\n\n> On ROC analysis, it seems that identification of a reliable threshold with a high AUC for differentiating clots from these two etiologies based on composition analysis alone is not possible. The lack of a reliable histological biomarker on MSB between these two stroke etiologies suggests that conventional histological analyses looking at cellular composition does not provide insights into stroke etiology in cryptogenic cases. \n\n> Overall, the findings from our study and the disparate results from multiple prior studies suggest that routine histological staining with H&E or MSB probably will not allow us to differentiate clots of different etiologies\n\nFrom this, my understanding is that the study **failed** to classify between CE and LAA clots and the study itself suggests that future work should focus on other approaches (like molecular testing and immunohistochemistry).\n\nSo it would be great to get this confirmed from the organizers:\n**Is there any strong evidence that there is actual signal in the data?**\n\nIt's very much possible there is. Being someone who is familiar with computational pathology literature, I know there is a surprising amount of information you can get from histology with the help of deep learning. But it is also possible that no such signal is available in this type of data, and appropriate biomarkers only show up in other sort of tests and analyses (again, most likely molecular testing or immunohistochemistry). \n\nIn any case, it would be helpful for participants to know if it has been previously been established or not if signal can be found in this dataset...\n\nMaybe I am misunderstanding something, in which case I would appreciate a correction... Hope to get some answers from the organizers :)", "votes": 58 }, { "id": 1853720, "postDate": "2022-07-13T05:08:54.180Z", "content": "Please note that in the study mentioned, which was conducted by the Mayo Neurovascular Lab, the input used to classify the clots into LAA vs. CE was not the images themselves but rather percentages of the clot components (mainly percentages of RBC, WBC, Fibrin, and Platelets) quantified using Orbit software. The machine learning model used was a packaged ensemble model applied on the tabular data to make the classification, which proved inconclusive in this study.
\nThe rationale for this competition is based on the fact that there is a difference in clot formation mechanism between LAA and CE clots, and the architecture and overall organization of the clot's components could point to the origin. We believe implementing histology-related deep learning methods and digital pathology image processing has the potential of picking such structural signature and help in the etiology classification.
\nI hope this is helpful. Thank you for your question and participation!
\nAll the best! :)
", "rawMarkdown": "Please note that in the study mentioned, which was conducted by the Mayo Neurovascular Lab, the input used to classify the clots into LAA vs. CE was not the images themselves but rather percentages of the clot components (mainly percentages of RBC, WBC, Fibrin, and Platelets) quantified using Orbit software. The machine learning model used was a packaged ensemble model applied on the tabular data to make the classification, which proved inconclusive in this study.\n\nThe rationale for this competition is based on the fact that there is a difference in clot formation mechanism between LAA and CE clots, and the architecture and overall organization of the clot's components could point to the origin. We believe implementing histology-related deep learning methods and digital pathology image processing has the potential of picking such structural signature and help in the etiology classification.\n\nI hope this is helpful. Thank you for your question and participation!\n\nAll the best! :)\n", "votes": 13, "replies": [ { "id": 1854766, "postDate": "2022-07-14T02:01:03.573Z", "content": "Thank you for your clarification, very helpful! :)
", "rawMarkdown": "Thank you for your clarification, very helpful! :)", "votes": 4 } ] }, { "id": 1850552, "postDate": "2022-07-10T13:48:16.330Z", "content": "AUC 0.55… Is it a random game again, like RSNA-MICCAI Brain Tumor Radiogenomic Classification?
", "rawMarkdown": "AUC 0.55... Is it a random game again, like [RSNA-MICCAI Brain Tumor Radiogenomic Classification](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification)?", "votes": 10 }, { "id": 1851890, "postDate": "2022-07-11T15:55:35.313Z", "content": "My short answer would be yes. Have you segmented your images? Are you considering the fact that the data is collected from 18 institutions? For the differences in staining, are you applying any stain color normalization methods?
", "rawMarkdown": "My short answer would be yes. Have you segmented your images? Are you considering the fact that the data is collected from 18 institutions? For the differences in staining, are you applying any stain color normalization methods?", "votes": 5 }, { "id": 1850501, "postDate": "2022-07-10T13:03:48.107Z", "content": "I asked a similar not as thorough question on the organizer welcome thread - https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/335564
\nNo reply as of yet but I will update you if anything changes there. Thanks for putting this out there!
", "rawMarkdown": "I asked a similar not as thorough question on the organizer welcome thread - https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/335564\n\nNo reply as of yet but I will update you if anything changes there. Thanks for putting this out there!", "votes": 1 }, { "id": 1850678, "postDate": "2022-07-10T16:03:08.267Z", "content": "The research article tells us that the study using machine learning models failed to classify between CE and LAA clots, the question should not be weather it can be done by machine learning, because there are too many ways to process this type of data, the question should be, Can a trained professional human identify between the clots using only the histological data, if yes, then what are the things he/she is looking for to make the prediction. If not, then it is probably gonna be a random game again…
", "rawMarkdown": "The research article tells us that the study using machine learning models failed to classify between CE and LAA clots, the question should not be weather it can be done by machine learning, because there are too many ways to process this type of data, the question should be, Can a trained professional human identify between the clots using only the histological data, if yes, then what are the things he/she is looking for to make the prediction. If not, then it is probably gonna be a random game again...", "votes": 2, "replies": [ { "id": 1850872, "postDate": "2022-07-10T21:28:53.260Z", "content": "I actually disagree… Just because a pathologist cannot identify the clot etiology doesn't mean that it's impossible to do so with AI.
\nA hot topic in computational pathology is predicting molecular features from the H&E. For example, predicting the presence of a genetic mutation from the slides without having to do any genetic/molecular testing! The pathologists have no clue how to do this, but numerous studies have shown this is very much possible with deep learning (high AUCs of like 0.8-0.9 have been reported).
\nI suspect this might be a similar motivation for the competition organizers. But in this case, it's not clear if there is signal to begin with or not, and there is a chance that unlike some of these other molecular features, it is impossible to predict the clot etiology from just the MSB WSIs alone. We just don't know yet… And the fact that the previous study failed to predict clot etiology is just further evidence supporting the possibility that it may be impossible.
\nBut hopefully the organizers can provide more clarification…
", "rawMarkdown": "I actually disagree... Just because a pathologist cannot identify the clot etiology doesn't mean that it's impossible to do so with AI.\n\nA hot topic in computational pathology is predicting molecular features from the H&E. For example, predicting the presence of a genetic mutation from the slides without having to do any genetic/molecular testing! The pathologists have no clue how to do this, but numerous studies have shown this is very much possible with deep learning (high AUCs of like 0.8-0.9 have been reported).\n\nI suspect this might be a similar motivation for the competition organizers. But in this case, it's not clear if there is signal to begin with or not, and there is a chance that unlike some of these other molecular features, it is impossible to predict the clot etiology from just the MSB WSIs alone. We just don't know yet... And the fact that the previous study failed to predict clot etiology is just further evidence supporting the possibility that it may be impossible.\n\nBut hopefully the organizers can provide more clarification...", "votes": 2 }, { "id": 1850886, "postDate": "2022-07-10T22:06:15.573Z", "content": "I agree with the first statement, a lot of times what pathologists can not do, can be done with AI, totally agreed, but you asked for strong evidence from the organizer, in which case, strong evidence would be human specialists able to do it.. if there was any current paper on models being able to do it, I think you probably have had read it by now. Also, currently I have 67.5% accuracy on my balanced validation set, and 0.63+ ROC_AUC_SCORE and still improving…, If I had to make a guess right now, this data is not completely bound by random seeds.
", "rawMarkdown": "I agree with the first statement, a lot of times what pathologists can not do, can be done with AI, totally agreed, but you asked for strong evidence from the organizer, in which case, strong evidence would be human specialists able to do it.. if there was any current paper on models being able to do it, I think you probably have had read it by now. Also, currently I have 67.5% accuracy on my balanced validation set, and 0.63+ ROC_AUC_SCORE and still improving..., If I had to make a guess right now, this data is not completely bound by random seeds.", "votes": 2 }, { "id": 1850902, "postDate": "2022-07-10T22:47:28.507Z", "content": "Yeah makes sense…
\nGood to hear you have seen some promising results locally…
I’m looking forward to seeing you conquer the sample submission :)
", "rawMarkdown": "I’m looking forward to seeing you conquer the sample submission :)", "votes": 4 }, { "id": 1888429, "postDate": "2022-08-07T14:51:53.740Z", "content": "@dschettler8845 evaluation uses the weighted log loss, do we know the wts for each class ?
", "rawMarkdown": "@dschettler8845 evaluation uses the weighted log loss, do we know the wts for each class ?" } ] }, { "id": 1859650, "postDate": "2022-07-17T20:16:53.147Z", "content": "Can you include datasets for blood chemistries and hematology for each of
\nif the histology sample?
Please note that in the study mentioned, which was conducted by the Mayo Neurovascular Lab, the input used to classify the clots into LAA vs. CE was not the images themselves but rather percentages of the clot components (mainly percentages of RBC, WBC, Fibrin, and Platelets) quantified using Orbit software. The machine learning model used was a packaged ensemble model applied on the tabular data to make the classification, which proved inconclusive in this study.
\nThe rationale for this competition is based on the fact that there is a difference in clot formation mechanism between LAA and CE clots, and the architecture and overall organization of the clot's components could point to the origin. We believe implementing histology-related deep learning methods and digital pathology image processing has the potential of picking such structural signature and help in the etiology classification.
\nI hope this is helpful. Thank you for your question and participation!
\nAll the best! :)
", "votes": 13, "replies": [ { "id": 1854766, "author_name": "ilovescience", "author_url": "", "post_date": "2022-07-14T02:01:03.573000", "content": "Thank you for your clarification, very helpful! :)
", "votes": 4, "replies": [] } ] }, { "id": 1850552, "author_name": "Leon", "author_url": "", "post_date": "2022-07-10T13:48:16.330000", "content": "AUC 0.55… Is it a random game again, like RSNA-MICCAI Brain Tumor Radiogenomic Classification?
", "votes": 10, "replies": [] }, { "id": 1851890, "author_name": "Barbaros", "author_url": "", "post_date": "2022-07-11T15:55:35.313000", "content": "My short answer would be yes. Have you segmented your images? Are you considering the fact that the data is collected from 18 institutions? For the differences in staining, are you applying any stain color normalization methods?
", "votes": 5, "replies": [] }, { "id": 1850501, "author_name": "Darien Schettler", "author_url": "", "post_date": "2022-07-10T13:03:48.107000", "content": "I asked a similar not as thorough question on the organizer welcome thread - https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/335564
\nNo reply as of yet but I will update you if anything changes there. Thanks for putting this out there!
", "votes": 1, "replies": [] }, { "id": 1850678, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-07-10T16:03:08.267000", "content": "The research article tells us that the study using machine learning models failed to classify between CE and LAA clots, the question should not be weather it can be done by machine learning, because there are too many ways to process this type of data, the question should be, Can a trained professional human identify between the clots using only the histological data, if yes, then what are the things he/she is looking for to make the prediction. If not, then it is probably gonna be a random game again…
", "votes": 2, "replies": [ { "id": 1850872, "author_name": "ilovescience", "author_url": "", "post_date": "2022-07-10T21:28:53.260000", "content": "I actually disagree… Just because a pathologist cannot identify the clot etiology doesn't mean that it's impossible to do so with AI.
\nA hot topic in computational pathology is predicting molecular features from the H&E. For example, predicting the presence of a genetic mutation from the slides without having to do any genetic/molecular testing! The pathologists have no clue how to do this, but numerous studies have shown this is very much possible with deep learning (high AUCs of like 0.8-0.9 have been reported).
\nI suspect this might be a similar motivation for the competition organizers. But in this case, it's not clear if there is signal to begin with or not, and there is a chance that unlike some of these other molecular features, it is impossible to predict the clot etiology from just the MSB WSIs alone. We just don't know yet… And the fact that the previous study failed to predict clot etiology is just further evidence supporting the possibility that it may be impossible.
\nBut hopefully the organizers can provide more clarification…
", "votes": 2, "replies": [] }, { "id": 1850886, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-07-10T22:06:15.573000", "content": "I agree with the first statement, a lot of times what pathologists can not do, can be done with AI, totally agreed, but you asked for strong evidence from the organizer, in which case, strong evidence would be human specialists able to do it.. if there was any current paper on models being able to do it, I think you probably have had read it by now. Also, currently I have 67.5% accuracy on my balanced validation set, and 0.63+ ROC_AUC_SCORE and still improving…, If I had to make a guess right now, this data is not completely bound by random seeds.
", "votes": 2, "replies": [] }, { "id": 1850902, "author_name": "ilovescience", "author_url": "", "post_date": "2022-07-10T22:47:28.507000", "content": "Yeah makes sense…
\nGood to hear you have seen some promising results locally…
I’m looking forward to seeing you conquer the sample submission :)
", "votes": 4, "replies": [] }, { "id": 1888429, "author_name": "Jaideep", "author_url": "", "post_date": "2022-08-07T14:51:53.740000", "content": "@dschettler8845 evaluation uses the weighted log loss, do we know the wts for each class ?
", "votes": 0, "replies": [] } ] }, { "id": 1859650, "author_name": "John Hasty", "author_url": "", "post_date": "2022-07-17T20:16:53.147000", "content": "Can you include datasets for blood chemistries and hematology for each of
\nif the histology sample?