![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F6df074abd85c8314486ac2c95a42fac6%2Fpp.png?generation=1686487021251390&alt=media\")
We have to pay attention to post-processing and the impact on score. Improving average dice score per image is not the same as improving global dice score.
\nHereafter an example on my model. I'm trying to remove small predicted items (less than a number of pixels) and see how it impacts score on validation dataset:
\n
Average dice score per image improves (orange with scale on right) if I drop area below 11 pixels but not global dice score (blue with scale on left).
\nIt's due that dice score jumps from 0.0 to 1.0 on each image with bad small predictions like this:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F30530edcab649466446e7c9bcb61cc47%2Fpp-dice.png?generation=1686486647669080&alt=media\")
\nBut impact of this fix on global dice is not the same (much lower as considered just as small improvement globally).
And of course removing small items could be counter-productive like on this example:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F0af8c37f445b2a17a4bee28080c94b33%2Fpp-bad-dice.png?generation=1686486863144935&alt=media\")
At the end, at least for this model, dropping small items in post-processing is a bad idea.
\nAny feedback is welcome on your post-processing options. On my side I've tried the following and it fails to improve validation score:
\nnp.sum(mask) < Nnp.sum(mask))On the Contrails paper they're using OpenCV’s LineSegmentDetector.
", "messages": [ { "id": 2295971, "postDate": "2023-06-11T12:47:27.183Z", "content": "We have to pay attention to post-processing and the impact on score. Improving average dice score per image is not the same as improving global dice score.
\nHereafter an example on my model. I'm trying to remove small predicted items (less than a number of pixels) and see how it impacts score on validation dataset:
\n
Average dice score per image improves (orange with scale on right) if I drop area below 11 pixels but not global dice score (blue with scale on left).
\nIt's due that dice score jumps from 0.0 to 1.0 on each image with bad small predictions like this:
\n
\nBut impact of this fix on global dice is not the same (much lower as considered just as small improvement globally).
And of course removing small items could be counter-productive like on this example:
\n
At the end, at least for this model, dropping small items in post-processing is a bad idea.
\nAny feedback is welcome on your post-processing options. On my side I've tried the following and it fails to improve validation score:
\nnp.sum(mask) < Nnp.sum(mask))On the Contrails paper they're using OpenCV’s LineSegmentDetector.
", "rawMarkdown": "We have to pay attention to post-processing and the impact on score. Improving average dice score per image is not the same as improving global dice score.\n\nHereafter an example on my model. I'm trying to remove small predicted items (less than a number of pixels) and see how it impacts score on validation dataset:\n\n\n\nAverage dice score per image improves (orange with scale on right) if I drop area below 11 pixels but not global dice score (blue with scale on left).\n\nIt's due that dice score jumps from 0.0 to 1.0 on each image with bad small predictions like this:\n\nBut impact of this fix on global dice is not the same (much lower as considered just as small improvement globally).\n\nAnd of course removing small items could be counter-productive like on this example:\n\n\nAt the end, at least for this model, dropping small items in post-processing is a bad idea.\n\nAny feedback is welcome on your post-processing options. On my side I've tried the following and it fails to improve validation score:\n- Drop mask with `np.sum(mask) < N`\n- Detect and drop small items (whatever `np.sum(mask)`)\n- [Morphology](https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html) operators (dilatation, opening, closing, tophat)\n\nOn the Contrails paper they're using OpenCV’s LineSegmentDetector.", "votes": 24 }, { "id": 2296012, "postDate": "2023-06-11T13:24:18.930Z", "content": "Thank you for initiating the postprocessing discussion.
\nI've tried the same ideas:
\nOn the Contrails paper they're using OpenCV’s LineSegmentDetector. - you can calculate the line angles, lines lengths, etc. Then we need to find the parallel lines, then to connect them and draw something out of it. I went really deep, but so far no big gains. Even if you look closely to what the researchers generated out of LineSegmentDetector in the end of the preprint (the way they connect the beginning of the contrail and the end), It does not look precise.
", "rawMarkdown": "Thank you for initiating the postprocessing discussion.\n\nI've tried the same ideas:\n* Drop mask with np.sum(mask) < N,\n - **< 11 pxl ** gives **0.001** improvement, though any other thresholds make the score unstable.\n* Morphology operators (dilatation, opening, closing, tophat)\n - I tried above-mentioned (different kernel sizes, different kernel shapes [ellipse, rectangle, etc]). it makes my score much worse, only for some bins there is an infinitesimal increase.\n \nOn the Contrails paper they're using OpenCV’s LineSegmentDetector. - you can calculate the line angles, lines lengths, etc. Then we need to find the parallel lines, then to connect them and draw something out of it. I went really deep, but so far no big gains. Even if you look closely to what the researchers generated out of LineSegmentDetector in the end of the preprint (the way they connect the beginning of the contrail and the end), It does not look precise.", "votes": 6, "replies": [ { "id": 2296473, "postDate": "2023-06-11T21:47:09.947Z", "content": "I got that with np.sum(mask) < N, stable but no improvement. But now I see it looks safe to use below 9px.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fb1b28ccea2ba0ff7a71eeb812777324f%2Fpp-min.png?generation=1686519887916878&alt=media\")
Hi @mpware thanks for your analysis. I also tried this idea with randomly pick a threshold, say 15 pixels, but my results were not stable. How about 9? Does it work on LB?
\nThank you.
", "rawMarkdown": "Hi @mpware thanks for your analysis. I also tried this idea with randomly pick a threshold, say 15 pixels, but my results were not stable. How about 9? Does it work on LB?\n\nThank you.", "votes": 1 }, { "id": 2320899, "postDate": "2023-06-28T06:26:37.670Z", "content": "I'm not using any post-processing for now on my submissions.
", "rawMarkdown": "I'm not using any post-processing for now on my submissions." }, { "id": 2320984, "postDate": "2023-06-28T07:53:04.887Z", "content": "OK， thank you for the reply.
", "rawMarkdown": "OK， thank you for the reply. " } ] } ] }, { "id": 2295998, "postDate": "2023-06-11T13:06:42.747Z", "content": "I did the same post-processing a week ago and have similiar result. but In my case, when I tried to drop mask with np.sum(mask) < N,
\nmy cv can boost ~0.001, and lb didn't change…😂I tested it just twice.
Thanks for the feedback!
", "rawMarkdown": "Thanks for the feedback!" } ] }, { "id": 2344982, "postDate": "2023-07-15T01:03:58.590Z", "content": "cv2.connectedComponents failed as well, even removing one pixel drops the CV
I tried post processing with cv2 connected components . But there were lot of small true masks and at times just one small line in whole image . It was removing those as well . Therefore I was not sure, what is the best way to do it and dropped post processing idea for now . For me same for threshold. Many of my models are bot sensitive to threshold at all ..I don't get any benefit by changing thresholds .
", "rawMarkdown": "I tried post processing with cv2 connected components . But there were lot of small true masks and at times just one small line in whole image . It was removing those as well . Therefore I was not sure, what is the best way to do it and dropped post processing idea for now . For me same for threshold. Many of my models are bot sensitive to threshold at all ..I don't get any benefit by changing thresholds . ", "votes": 1 }, { "id": 2296013, "postDate": "2023-06-11T13:24:29.487Z", "content": "I tried to find positive clusters using scipy ndimage and remove the ones that had fewer than 10px as well but CV dropped, now I really get why, thx for this post
", "rawMarkdown": "I tried to find positive clusters using scipy ndimage and remove the ones that had fewer than 10px as well but CV dropped, now I really get why, thx for this post", "votes": 1 }, { "id": 2296145, "postDate": "2023-06-11T15:15:18.223Z", "content": "maybe some outlier detection tools (remove -1 from hdbscan ?)
", "rawMarkdown": "maybe some outlier detection tools (remove -1 from hdbscan ?)" } ], "comments": [ { "id": 2296012, "author_name": "SSS", "author_url": "", "post_date": "2023-06-11T13:24:18.930000", "content": "Thank you for initiating the postprocessing discussion.
\nI've tried the same ideas:
\nOn the Contrails paper they're using OpenCV’s LineSegmentDetector. - you can calculate the line angles, lines lengths, etc. Then we need to find the parallel lines, then to connect them and draw something out of it. I went really deep, but so far no big gains. Even if you look closely to what the researchers generated out of LineSegmentDetector in the end of the preprint (the way they connect the beginning of the contrail and the end), It does not look precise.
", "votes": 6, "replies": [ { "id": 2296473, "author_name": "MPWARE", "author_url": "", "post_date": "2023-06-11T21:47:09.947000", "content": "I got that with np.sum(mask) < N, stable but no improvement. But now I see it looks safe to use below 9px.
Hi @mpware thanks for your analysis. I also tried this idea with randomly pick a threshold, say 15 pixels, but my results were not stable. How about 9? Does it work on LB?
\nThank you.
", "votes": 1, "replies": [] }, { "id": 2320899, "author_name": "MPWARE", "author_url": "", "post_date": "2023-06-28T06:26:37.670000", "content": "I'm not using any post-processing for now on my submissions.
", "votes": 0, "replies": [] }, { "id": 2320984, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2023-06-28T07:53:04.887000", "content": "OK， thank you for the reply.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2295998, "author_name": "lyu", "author_url": "", "post_date": "2023-06-11T13:06:42.747000", "content": "I did the same post-processing a week ago and have similiar result. but In my case, when I tried to drop mask with np.sum(mask) < N,
\nmy cv can boost ~0.001, and lb didn't change…😂I tested it just twice.
Thanks for the feedback!
", "votes": 0, "replies": [] } ] }, { "id": 2344982, "author_name": "william.wu", "author_url": "", "post_date": "2023-07-15T01:03:58.590000", "content": "cv2.connectedComponents failed as well, even removing one pixel drops the CV
I tried post processing with cv2 connected components . But there were lot of small true masks and at times just one small line in whole image . It was removing those as well . Therefore I was not sure, what is the best way to do it and dropped post processing idea for now . For me same for threshold. Many of my models are bot sensitive to threshold at all ..I don't get any benefit by changing thresholds .
", "votes": 1, "replies": [] }, { "id": 2296013, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-11T13:24:29.487000", "content": "I tried to find positive clusters using scipy ndimage and remove the ones that had fewer than 10px as well but CV dropped, now I really get why, thx for this post
", "votes": 1, "replies": [] }, { "id": 2296145, "author_name": "Lucas Morin", "author_url": "", "post_date": "2023-06-11T15:15:18.223000", "content": "maybe some outlier detection tools (remove -1 from hdbscan ?)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2295971": "We have to pay attention to post-processing and the impact on score. Improving average dice score per image is not the same as improving global dice score.\n\nHereafter an example on my model. I'm trying to remove small predicted items (less than a number of pixels) and see how it impacts score on validation dataset:\n\n\n\nAverage dice score per image improves (orange with scale on right) if I drop area below 11 pixels but not global dice score (blue with scale on left).\n\nIt's due that dice score jumps from 0.0 to 1.0 on each image with bad small predictions like this:\n\nBut impact of this fix on global dice is not the same (much lower as considered just as small improvement globally).\n\nAnd of course removing small items could be counter-productive like on this example:\n\n\nAt the end, at least for this model, dropping small items in post-processing is a bad idea.\n\nAny feedback is welcome on your post-processing options. On my side I've tried the following and it fails to improve validation score:\n- Drop mask with `np.sum(mask) < N`\n- Detect and drop small items (whatever `np.sum(mask)`)\n- [Morphology](https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html) operators (dilatation, opening, closing, tophat)\n\nOn the Contrails paper they're using OpenCV’s LineSegmentDetector.", "2296012": "Thank you for initiating the postprocessing discussion.\n\nI've tried the same ideas:\n* Drop mask with np.sum(mask) < N,\n - **< 11 pxl ** gives **0.001** improvement, though any other thresholds make the score unstable.\n* Morphology operators (dilatation, opening, closing, tophat)\n - I tried above-mentioned (different kernel sizes, different kernel shapes [ellipse, rectangle, etc]). it makes my score much worse, only for some bins there is an infinitesimal increase.\n \nOn the Contrails paper they're using OpenCV’s LineSegmentDetector. - you can calculate the line angles, lines lengths, etc. Then we need to find the parallel lines, then to connect them and draw something out of it. I went really deep, but so far no big gains. Even if you look closely to what the researchers generated out of LineSegmentDetector in the end of the preprint (the way they connect the beginning of the contrail and the end), It does not look precise.", "2295998": "I did the same post-processing a week ago and have similiar result. but In my case, when I tried to drop mask with `np.sum(mask) < N `,\nmy cv can boost ~0.001, and lb didn't change...😂I tested it just twice.", "2344982": "`cv2.connectedComponents` failed as well, even removing one pixel drops the CV", "2296096": "I tried post processing with cv2 connected components . But there were lot of small true masks and at times just one small line in whole image . It was removing those as well . Therefore I was not sure, what is the best way to do it and dropped post processing idea for now . For me same for threshold. Many of my models are bot sensitive to threshold at all ..I don't get any benefit by changing thresholds . ", "2296013": "I tried to find positive clusters using scipy ndimage and remove the ones that had fewer than 10px as well but CV dropped, now I really get why, thx for this post", "2296145": "maybe some outlier detection tools (remove -1 from hdbscan ?)" } }