Based on discussion and kernal by iafoss, I have created two datasets from intermediate tiff layer.
\n\nFor tiles selection I used mask instead of images used by lafoss.\n1. 15x256x256- https://www.kaggle.com/gaur128/panda-256x256-tiles\n2. 5x512x512- https://www.kaggle.com/gaur128/panda-512x512-tiles
\n\nDue to kaggle dataset size limit these are the maximum no. of tiles I can create from intermediate tiff layer. \nHope it helps.
", "messages": [ { "id": 872403, "postDate": "2020-06-03T06:53:42.120Z", "content": "Based on discussion and kernal by iafoss, I have created two datasets from intermediate tiff layer.
\n\nFor tiles selection I used mask instead of images used by lafoss.\n1. 15x256x256- https://www.kaggle.com/gaur128/panda-256x256-tiles\n2. 5x512x512- https://www.kaggle.com/gaur128/panda-512x512-tiles
\n\nDue to kaggle dataset size limit these are the maximum no. of tiles I can create from intermediate tiff layer. \nHope it helps.
", "rawMarkdown": "Based on [discussion](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/146855) and [kernal](https://www.kaggle.com/iafoss/panda-16x128x128-tiles) by iafoss, I have created two datasets from intermediate tiff layer. \n\nFor tiles selection I used mask instead of images used by lafoss.\n1. 15x256x256- https://www.kaggle.com/gaur128/panda-256x256-tiles\n2. 5x512x512- https://www.kaggle.com/gaur128/panda-512x512-tiles\n\nDue to kaggle dataset size limit these are the maximum no. of tiles I can create from intermediate tiff layer. \nHope it helps. ", "votes": 5 }, { "id": 883662, "postDate": "2020-06-12T20:11:56.667Z", "content": "A good way to check if your tile choice makes sense you can first open all images and compute the tissue area (pixel with values below say 240) and then divide that number by your tile size (squared). This should give you the approximate number of tiles that a slice can acomodate. Get the mean and std of this to get some insight for a good choice of tile size. For example on the intermediary level I get for 224*224 tiles approximately a mean of 30 tiles and std 10 from memory. Meaning that 15 x 256 x256 is likely too small...
", "rawMarkdown": "A good way to check if your tile choice makes sense you can first open all images and compute the tissue area (pixel with values below say 240) and then divide that number by your tile size (squared). This should give you the approximate number of tiles that a slice can acomodate. Get the mean and std of this to get some insight for a good choice of tile size. For example on the intermediary level I get for 224*224 tiles approximately a mean of 30 tiles and std 10 from memory. Meaning that 15 x 256 x256 is likely too small...", "replies": [ { "id": 885350, "postDate": "2020-06-14T05:58:52.137Z", "content": "@arroqc Thanks for the response.\nIt seems I overlooked this. But if it as you said it is difficult to create a tiles dataset to be used on kaggle for intermediate tiff layer. As I want to use kaggle tpu and I'm not getting any success in reading tiff at runtime with tensorflow dataset and TPU.
", "rawMarkdown": "@arroqc Thanks for the response.\nIt seems I overlooked this. But if it as you said it is difficult to create a tiles dataset to be used on kaggle for intermediate tiff layer. As I want to use kaggle tpu and I'm not getting any success in reading tiff at runtime with tensorflow dataset and TPU." } ] }, { "id": 882685, "postDate": "2020-06-12T04:07:51.777Z", "content": "@iafoss Can you help me out if you got any time.\nI wanted to work with dataset created by me to get more understanding about dataset. So I created the tiles dataset taking your kernal as base. I used density of mask pixels for tiles selection.
\n\nCan you just go through my dataset and kernal a bit and see what exactly I did wrong as my model is not learning anything in both separate tiles(Keras/GPU) and concating tiles as single image(Keras/TPU). \nI'm able to get some result if I used others dataset if it is single image or tiles.
\n\nWould be grateful if you can help out a bit as I don't want to switch to another dataset without understanding what I did wrong.\nThanks.
", "rawMarkdown": "@iafoss Can you help me out if you got any time.\nI wanted to work with dataset created by me to get more understanding about dataset. So I created the tiles dataset taking your kernal as base. I used density of mask pixels for tiles selection.\n\nCan you just go through my dataset and kernal a bit and see what exactly I did wrong as my model is not learning anything in both separate tiles(Keras/GPU) and concating tiles as single image(Keras/TPU). \nI'm able to get some result if I used others dataset if it is single image or tiles.\n\nWould be grateful if you can help out a bit as I don't want to switch to another dataset without understanding what I did wrong.\nThanks.", "replies": [ { "id": 883558, "postDate": "2020-06-12T18:18:58.707Z", "content": "If you are working with intermediate res, there could be some convergence issues because of too small bs. In addition, 15x256x256 seems to be not enough to cover the tissue area for intermediate res layer. Probably, \"other\" datasets u are referring to are made of low res layer, and therefore everything works well. You should also be careful with tile selection based on masks because they are not available for test set. If you want to use such an approach, you could consider training a segmentation model first that can produce masks, and then use the generated masks to build the dataset. If u do mask based selection, make sure that the model can capture the relative area of different components. Check how ISUP grade is assigned in the additional information provided to this competition and ask yourself if you could assign it based on the tiles u generate if your were able to recognized Gleason patterns, or something is missing in your data.
", "rawMarkdown": "If you are working with intermediate res, there could be some convergence issues because of too small bs. In addition, 15x256x256 seems to be not enough to cover the tissue area for intermediate res layer. Probably, \"other\" datasets u are referring to are made of low res layer, and therefore everything works well. You should also be careful with tile selection based on masks because they are not available for test set. If you want to use such an approach, you could consider training a segmentation model first that can produce masks, and then use the generated masks to build the dataset. If u do mask based selection, make sure that the model can capture the relative area of different components. Check how ISUP grade is assigned in the additional information provided to this competition and ask yourself if you could assign it based on the tiles u generate if your were able to recognized Gleason patterns, or something is missing in your data. " }, { "id": 885346, "postDate": "2020-06-14T05:51:06.927Z", "content": "@iafoss Thanks for the reply\nThe main aim for using mask to select tiles is have tiles which contain infected cells. For both providers marking is different but they do set the pixel values in mask. I thought of using those pixel values to select tiles. \nIs it possible that this approach might be selecting tiles from same component like if we talk about Radbound provider this approach is selecting tiles from region of values 4 and 5 only instead of capturing of different components as you mentioned above.\nIf it is so then in tiles approach isn't no. of tiles affects learning a lot.\nThanks
", "rawMarkdown": "@iafoss Thanks for the reply\nThe main aim for using mask to select tiles is have tiles which contain infected cells. For both providers marking is different but they do set the pixel values in mask. I thought of using those pixel values to select tiles. \nIs it possible that this approach might be selecting tiles from same component like if we talk about Radbound provider this approach is selecting tiles from region of values 4 and 5 only instead of capturing of different components as you mentioned above.\nIf it is so then in tiles approach isn't no. of tiles affects learning a lot.\nThanks " } ] } ], "comments": [ { "id": 883662, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-12T20:11:56.667000", "content": "A good way to check if your tile choice makes sense you can first open all images and compute the tissue area (pixel with values below say 240) and then divide that number by your tile size (squared). This should give you the approximate number of tiles that a slice can acomodate. Get the mean and std of this to get some insight for a good choice of tile size. For example on the intermediary level I get for 224*224 tiles approximately a mean of 30 tiles and std 10 from memory. Meaning that 15 x 256 x256 is likely too small...
", "votes": 0, "replies": [ { "id": 885350, "author_name": "Gaurav Yadav", "author_url": "", "post_date": "2020-06-14T05:58:52.137000", "content": "@arroqc Thanks for the response.\nIt seems I overlooked this. But if it as you said it is difficult to create a tiles dataset to be used on kaggle for intermediate tiff layer. As I want to use kaggle tpu and I'm not getting any success in reading tiff at runtime with tensorflow dataset and TPU.
", "votes": 0, "replies": [] } ] }, { "id": 882685, "author_name": "Gaurav Yadav", "author_url": "", "post_date": "2020-06-12T04:07:51.777000", "content": "@iafoss Can you help me out if you got any time.\nI wanted to work with dataset created by me to get more understanding about dataset. So I created the tiles dataset taking your kernal as base. I used density of mask pixels for tiles selection.
\n\nCan you just go through my dataset and kernal a bit and see what exactly I did wrong as my model is not learning anything in both separate tiles(Keras/GPU) and concating tiles as single image(Keras/TPU). \nI'm able to get some result if I used others dataset if it is single image or tiles.
\n\nWould be grateful if you can help out a bit as I don't want to switch to another dataset without understanding what I did wrong.\nThanks.
", "votes": 0, "replies": [ { "id": 883558, "author_name": "Iafoss", "author_url": "", "post_date": "2020-06-12T18:18:58.707000", "content": "If you are working with intermediate res, there could be some convergence issues because of too small bs. In addition, 15x256x256 seems to be not enough to cover the tissue area for intermediate res layer. Probably, \"other\" datasets u are referring to are made of low res layer, and therefore everything works well. You should also be careful with tile selection based on masks because they are not available for test set. If you want to use such an approach, you could consider training a segmentation model first that can produce masks, and then use the generated masks to build the dataset. If u do mask based selection, make sure that the model can capture the relative area of different components. Check how ISUP grade is assigned in the additional information provided to this competition and ask yourself if you could assign it based on the tiles u generate if your were able to recognized Gleason patterns, or something is missing in your data.
", "votes": 0, "replies": [] }, { "id": 885346, "author_name": "Gaurav Yadav", "author_url": "", "post_date": "2020-06-14T05:51:06.927000", "content": "@iafoss Thanks for the reply\nThe main aim for using mask to select tiles is have tiles which contain infected cells. For both providers marking is different but they do set the pixel values in mask. I thought of using those pixel values to select tiles. \nIs it possible that this approach might be selecting tiles from same component like if we talk about Radbound provider this approach is selecting tiles from region of values 4 and 5 only instead of capturing of different components as you mentioned above.\nIf it is so then in tiles approach isn't no. of tiles affects learning a lot.\nThanks
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "872403": "Based on [discussion](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/146855) and [kernal](https://www.kaggle.com/iafoss/panda-16x128x128-tiles) by iafoss, I have created two datasets from intermediate tiff layer. \n\nFor tiles selection I used mask instead of images used by lafoss.\n1. 15x256x256- https://www.kaggle.com/gaur128/panda-256x256-tiles\n2. 5x512x512- https://www.kaggle.com/gaur128/panda-512x512-tiles\n\nDue to kaggle dataset size limit these are the maximum no. of tiles I can create from intermediate tiff layer. \nHope it helps. ", "883662": "A good way to check if your tile choice makes sense you can first open all images and compute the tissue area (pixel with values below say 240) and then divide that number by your tile size (squared). This should give you the approximate number of tiles that a slice can acomodate. Get the mean and std of this to get some insight for a good choice of tile size. For example on the intermediary level I get for 224*224 tiles approximately a mean of 30 tiles and std 10 from memory. Meaning that 15 x 256 x256 is likely too small...", "882685": "@iafoss Can you help me out if you got any time.\nI wanted to work with dataset created by me to get more understanding about dataset. So I created the tiles dataset taking your kernal as base. I used density of mask pixels for tiles selection.\n\nCan you just go through my dataset and kernal a bit and see what exactly I did wrong as my model is not learning anything in both separate tiles(Keras/GPU) and concating tiles as single image(Keras/TPU). \nI'm able to get some result if I used others dataset if it is single image or tiles.\n\nWould be grateful if you can help out a bit as I don't want to switch to another dataset without understanding what I did wrong.\nThanks." } }