This other discussion prompted me to look at the performance of the 17 different radiologists. I found something interesting (link to notebook): To my surprise several of them had a 100% match in terms of class_id values assigned to each image with the other three radiologists looking a the same image.
It turns out that radiologists that 100% agree with other radiologists only ever assigned class_id = 14 (=\"no finding\"). I'd guess X-rays that were a-priori identified (how?) to be without a finding and were sent to a subset of radiologists, especially R1 to R7 (and sometimes others, but , R8, R9 and R10 mostly did not get those images).
None of this is mentioned in the paper. So I'm wondering whether there's something I overlooked and whether there's other things I did miss about the training process.
\nIt also makes me think how much we should trust the \"No finding\" images… If someone has to go through 1500 to 3000 images, one after another and there's just never anything there, do they get tired and sloppy (and their colleagues, too)? I would have thought that mixing things up would be a good idea to make sure the radiologists are kept engaged/interested/\"on their toes\" (or however else you want to describe it).
\n| rad_id | \nImages with no finding | \nPercent with no finding | \n
|---|---|---|
| R1 | \n1995 | \n100% | \n
| R2 | \n3118 | \n99.9% | \n
| R3 | \n2285 | \n100% | \n
| R4 | \n1513 | \n100% | \n
| R5 | \n2783 | \n100% | \n
| R6 | \n2041 | \n100% | \n
| R7 | \n1733 | \n100% | \n
| R8 | \n2436 | \n19.97% | \n
| R9 | \n1979 | \n12.6% | \n
| R10 | \n2321 | \n17.46% | \n
| R11 | \n1413 | \n84.61% | \n
| R12 | \n1580 | \n91.38% | \n
| R13 | \n1505 | \n82.51% | \n
| R14 | \n1300 | \n80.05% | \n
| R15 | \n1508 | \n82.72% | \n
| R16 | \n1565 | \n88.77% | \n
| R17 | \n743 | \n91.5% | \n
This other discussion prompted me to look at the performance of the 17 different radiologists. I found something interesting (link to notebook): To my surprise several of them had a 100% match in terms of class_id values assigned to each image with the other three radiologists looking a the same image.
It turns out that radiologists that 100% agree with other radiologists only ever assigned class_id = 14 (=\"no finding\"). I'd guess X-rays that were a-priori identified (how?) to be without a finding and were sent to a subset of radiologists, especially R1 to R7 (and sometimes others, but , R8, R9 and R10 mostly did not get those images).
None of this is mentioned in the paper. So I'm wondering whether there's something I overlooked and whether there's other things I did miss about the training process.
\nIt also makes me think how much we should trust the \"No finding\" images… If someone has to go through 1500 to 3000 images, one after another and there's just never anything there, do they get tired and sloppy (and their colleagues, too)? I would have thought that mixing things up would be a good idea to make sure the radiologists are kept engaged/interested/\"on their toes\" (or however else you want to describe it).
\n| rad_id | \nImages with no finding | \nPercent with no finding | \n
|---|---|---|
| R1 | \n1995 | \n100% | \n
| R2 | \n3118 | \n99.9% | \n
| R3 | \n2285 | \n100% | \n
| R4 | \n1513 | \n100% | \n
| R5 | \n2783 | \n100% | \n
| R6 | \n2041 | \n100% | \n
| R7 | \n1733 | \n100% | \n
| R8 | \n2436 | \n19.97% | \n
| R9 | \n1979 | \n12.6% | \n
| R10 | \n2321 | \n17.46% | \n
| R11 | \n1413 | \n84.61% | \n
| R12 | \n1580 | \n91.38% | \n
| R13 | \n1505 | \n82.51% | \n
| R14 | \n1300 | \n80.05% | \n
| R15 | \n1508 | \n82.72% | \n
| R16 | \n1565 | \n88.77% | \n
| R17 | \n743 | \n91.5% | \n
@bjoernholzhauer,
\nthe normal daily routine is different in every hospital, there are houses where the junior radiologists send each and every report to second reading and places where only the \"questionable\" ones are sent to second reader.
I think that the \"normal\" tagged images come from a screening ( young ppl must have a chest x-ray before starting work to exclude pulmonary TB ) and these are with high % without finding.
", "rawMarkdown": "@bjoernholzhauer,\nthe normal daily routine is different in every hospital, there are houses where the junior radiologists send each and every report to second reading and places where only the \"questionable\" ones are sent to second reader.\n\nI think that the \"normal\" tagged images come from a screening ( young ppl must have a chest x-ray before starting work to exclude pulmonary TB ) and these are with high % without finding.", "votes": 1, "replies": [ { "id": 1211861, "postDate": "2021-02-20T16:09:58.047Z", "content": "Thanks, that sounds plausible. It was not clearly described in the paper on the dataset. Largely a good job on those, given that no findings seems to then have mostly been confirmed. But, probably raises the risk that a subtle finding was missed originally and that the reviewers for this dataset got some fatigue after clicking though one\"no finding\" after another and missed it, too, I'd guess?
", "rawMarkdown": "Thanks, that sounds plausible. It was not clearly described in the paper on the dataset. Largely a good job on those, given that no findings seems to then have mostly been confirmed. But, probably raises the risk that a subtle finding was missed originally and that the reviewers for this dataset got some fatigue after clicking though one\"no finding\" after another and missed it, too, I'd guess?" }, { "id": 1211891, "postDate": "2021-02-20T16:29:04.213Z", "content": "It is also just guessing from my side…
\nSure, it is always a possibility to have subtle finding on them though!
", "rawMarkdown": "It is also just guessing from my side... \n\nSure, it is always a possibility to have subtle finding on them though!\n" } ] }, { "id": 1184022, "postDate": "2021-02-03T11:05:01.830Z", "content": "Interesting!
\nDisregarding the >99% no finding radiologists, it seems unlikely that R8, R9, R10 get no findings 80% of the time.
\nUnless the xrays were distributed non-randomly.
", "rawMarkdown": "Interesting!\n\nDisregarding the >99% no finding radiologists, it seems unlikely that R8, R9, R10 get no findings <20% of the time while R11-R17 get no findings >80% of the time.\n\nUnless the xrays were distributed non-randomly.", "votes": 1, "replies": [ { "id": 1184065, "postDate": "2021-02-03T11:24:54.730Z", "content": "Yes, I think it's pretty clear that the x-rays were assigned to radiologists in a non-random way. I'm guessing that they split by the original assigned diagnosis at the two hospitals where they got it from (or perhaps they ran some existing algorithm over it) and then somehow systematically assigned \"no findings\" x-rays mostly to one set of reviewers and ones with findings to others (or something like that).
", "rawMarkdown": "Yes, I think it's pretty clear that the x-rays were assigned to radiologists in a non-random way. I'm guessing that they split by the original assigned diagnosis at the two hospitals where they got it from (or perhaps they ran some existing algorithm over it) and then somehow systematically assigned \"no findings\" x-rays mostly to one set of reviewers and ones with findings to others (or something like that).", "votes": 2 }, { "id": 1184083, "postDate": "2021-02-03T11:33:09.537Z", "content": "It's odd that from what I can see, all radiologists agree when there's \"no finding.\"
", "rawMarkdown": "It's odd that from what I can see, all radiologists agree when there's \"no finding.\"", "votes": 1 } ] } ], "comments": [ { "id": 1211830, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-02-20T15:39:43.717000", "content": "@bjoernholzhauer,
\nthe normal daily routine is different in every hospital, there are houses where the junior radiologists send each and every report to second reading and places where only the \"questionable\" ones are sent to second reader.
I think that the \"normal\" tagged images come from a screening ( young ppl must have a chest x-ray before starting work to exclude pulmonary TB ) and these are with high % without finding.
", "votes": 1, "replies": [ { "id": 1211861, "author_name": "Björn", "author_url": "", "post_date": "2021-02-20T16:09:58.047000", "content": "Thanks, that sounds plausible. It was not clearly described in the paper on the dataset. Largely a good job on those, given that no findings seems to then have mostly been confirmed. But, probably raises the risk that a subtle finding was missed originally and that the reviewers for this dataset got some fatigue after clicking though one\"no finding\" after another and missed it, too, I'd guess?
", "votes": 0, "replies": [] }, { "id": 1211891, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-02-20T16:29:04.213000", "content": "It is also just guessing from my side…
\nSure, it is always a possibility to have subtle finding on them though!
", "votes": 0, "replies": [] } ] }, { "id": 1184022, "author_name": "quillio", "author_url": "", "post_date": "2021-02-03T11:05:01.830000", "content": "Interesting!
\nDisregarding the >99% no finding radiologists, it seems unlikely that R8, R9, R10 get no findings 80% of the time.
\nUnless the xrays were distributed non-randomly.
", "votes": 1, "replies": [ { "id": 1184065, "author_name": "Björn", "author_url": "", "post_date": "2021-02-03T11:24:54.730000", "content": "Yes, I think it's pretty clear that the x-rays were assigned to radiologists in a non-random way. I'm guessing that they split by the original assigned diagnosis at the two hospitals where they got it from (or perhaps they ran some existing algorithm over it) and then somehow systematically assigned \"no findings\" x-rays mostly to one set of reviewers and ones with findings to others (or something like that).
", "votes": 2, "replies": [] }, { "id": 1184083, "author_name": "quillio", "author_url": "", "post_date": "2021-02-03T11:33:09.537000", "content": "It's odd that from what I can see, all radiologists agree when there's \"no finding.\"
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1181501": "This [other discussion](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/215444) prompted me to look at the performance of the 17 different radiologists. I found something interesting ([link to notebook](https://www.kaggle.com/bjoernholzhauer/eda-dicom-reading-vinbigdata-chest-x-ray/#1.-Labelling-process-that-created-the-training-and-test-data)): To my surprise several of them had a 100% match in terms of `class_id` values assigned to each image with the other three radiologists looking a the same image.\n\nIt turns out that radiologists that 100% agree with other radiologists only ever assigned `class_id = 14` (=\"no finding\"). I'd guess X-rays that were a-priori identified (how?) to be without a finding and were sent to a subset of radiologists, especially `R1` to `R7` (and sometimes others, but , `R8`, `R9` and `R10` mostly did not get those images).\n\nNone of this is mentioned in [the paper](https://arxiv.org/pdf/2012.15029.pdf). So I'm wondering whether there's something I overlooked and whether there's other things I did miss about the training process. \n\nIt also makes me think how much we should trust the \"No finding\" images... If someone has to go through 1500 to 3000 images, one after another and there's just never anything there, do they get tired and sloppy (and their colleagues, too)? I would have thought that mixing things up would be a good idea to make sure the radiologists are kept engaged/interested/\"on their toes\" (or however else you want to describe it).\n\n| rad_id | Images with no finding | Percent with no finding |\n| --- | --- |\n|R1 |\t1995 |100% |\n|R2\t| 3118 | 99.9% |\n|R3\t| 2285 | 100% |\n|R4\t| 1513 | 100% |\n|R5\t| 2783 | 100% |\n|R6\t| 2041 | 100% |\n|R7\t| 1733 | 100% |\n|R8\t| 2436 | 19.97% |\n|R9\t| 1979 | 12.6% |\n|R10\t| 2321 | 17.46% |\n|R11 | 1413 | 84.61% |\n|R12|1580 | 91.38% |\n|R13\t|1505 | 82.51% |\n|R14\t|1300 | 80.05% |\n|R15\t|1508 | 82.72% |\n|R16\t|1565 | 88.77% |\n|R17 | 743 | 91.5% |\n", "1211830": "@bjoernholzhauer,\nthe normal daily routine is different in every hospital, there are houses where the junior radiologists send each and every report to second reading and places where only the \"questionable\" ones are sent to second reader.\n\nI think that the \"normal\" tagged images come from a screening ( young ppl must have a chest x-ray before starting work to exclude pulmonary TB ) and these are with high % without finding.", "1184022": "Interesting!\n\nDisregarding the >99% no finding radiologists, it seems unlikely that R8, R9, R10 get no findings <20% of the time while R11-R17 get no findings >80% of the time.\n\nUnless the xrays were distributed non-randomly." } }