{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Quick Overview\n\nIn this challenge, the objective is to **detect and classify the severity of prostate cancer on images of prostate tissue samples**. In practice, tissue samples are examined and scored by pathologists according to the so-called Gleason grading system which is later converted to an ISUP grade. But we will get into this in a bit. Let's have a quick look at the data and get an overview."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport skimage.io # Loading images\nimport cv2 # Resizing images\nfrom tqdm.notebook import tqdm # Visualizing progress\n\nimport matplotlib.pyplot as plt\nimport matplotlib.colors\n\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nPATH = \"../input/prostate-cancer-grade-assessment/\"\n\ndf_train = pd.read_csv(f'{PATH}train.csv')\ndf_test = pd.read_csv(f'{PATH}test.csv')\n\ndf_train.head().style.set_caption('Quick Overview of train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"Number of training data: {len(df_train)}\\n\")\n\nprint(f\"Unique data_providers: {df_train.data_provider.unique()}\\n\")\nprint(f\"Unique isup_grade: {df_train.isup_grade.unique()}\\n\")\nprint(f\"Unique gleason_score: {df_train.gleason_score.unique()}\\n\")\n\nprint(f\"Missing data:\\n{df_train.isna().any()}\\n\")\n\nmasks = os.listdir(PATH + 'train_label_masks/')\nimages = os.listdir(PATH + 'train_images/')\n\ndf_masks = pd.Series(masks).to_frame()\ndf_masks.columns = ['mask_file_name']\ndf_masks['image_id'] = df_masks.mask_file_name.apply(lambda x: x.split('_')[0])\ndf_train = pd.merge(df_train, df_masks, on='image_id', how='outer')\ndel df_masks\nprint(f\"There are {len(df_train[df_train.mask_file_name.isna()])} images without a mask.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings from Quick Overview\nAt first glance, we found 100 images without masks. For further analysis, we will drop the 100 images without a mask.\n\nHowever, for your model **it might be a good idea to use these test cases for validation. All suspicious test cases found in this EDA are summarized in a .csv file at the end of this kernel.**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Train data shape before reduction: {len(df_train)}\")\ndf_train_red = df_train[~df_train.mask_file_name.isna()]\nprint(f\"Train data shape after reduction: {len(df_train_red)}\")\n\nno_masks = df_train[df_train.mask_file_name.isna()][['image_id']]\nno_masks['Suspicious_because'] = 'No Mask'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Gleason Score and ISUP Grade\n\n> The grading process consists of finding and **classifying cancer tissue into so-called Gleason patterns** (3, 4, or 5)[...]. After the biopsy is assigned a Gleason score, it is **converted into an ISUP grade** on a 1-5 scale. [...] However, **the system suffers from significant inter-observer variability between pathologists**, limiting its usefulness for individual patients. This variability in ratings could lead to unnecessary treatment, or worse, missing a severe diagnosis. \n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_train_red.groupby('isup_grade').gleason_score.unique().to_frame().style.set_caption('Mapping of ISUP Grade to Gleason Score')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## One Mislabeled Image?\nIn the above dataframe it looks like one image might have been converted to a wrong ISUP grade."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_train_red[(df_train_red.isup_grade == 2) & (df_train_red.gleason_score != '3+4')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the explanation see [this discussion topic](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/145194#816268)\n\n> [...] All the Karolinska images in the training data is graded by the same pathologist. However, for the test set we used several pathologists who each labeled the images using ISUP (not Gleason) and derived a consensus label. The mislabeled image was one of those images but was later moved to the training set. So, the **Gleason score is the original label by the study pathologist** and the **ISUP is the consensus label by the pathologists who graded the test set**. [...]\n\nWe will not drop this training data at this point but keep this in mind."},{"metadata":{},"cell_type":"markdown","source":"# Differences Between Data Providers\nThere are two data providers. \n> They used different scanners with slightly different maximum microscope resolutions and worked with different pathologists for labeling their images."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"providers = df_train_red.data_provider.unique()\n\nfig = plt.figure(figsize=(6,4))\nax = sns.countplot(x=\"isup_grade\", hue=\"data_provider\", data=df_train_red)\nplt.title(\"ISUP Grade Count by Data Provider\", fontsize=14)\nplt.xlabel(\"ISUP Grade\", fontsize=14)\nplt.ylabel(\"Count\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_train_red[\"height\"] = 0\ndf_train_red[\"width\"] = 0\ndf_train_red[0] = 0\ndf_train_red[1] = 0\ndf_train_red[2] = 0\ndf_train_red[3] = 0\ndf_train_red[4] = 0\ndf_train_red[5] = 0\n\ndef get_image_data(row):\n    biopsy = skimage.io.MultiImage(PATH + 'train_label_masks/' + row.image_id + '_mask.tiff')\n    temp = biopsy[-1][:, :, 0]\n    counts = pd.Series(temp.reshape(-1)).value_counts()\n    row.height = temp.shape[0]\n    row.width = temp.shape[1]\n    row.update(counts)\n    return row\n\ndf_train_red = df_train_red.apply(lambda row: get_image_data(row), axis=1)\n\ndf_train_red['pixels'] = df_train_red.height * df_train_red.width\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, [ax1, ax2] = plt.subplots(nrows=1, ncols=2, figsize=(16, 6))\n\n\"\"\"\nInspired by something similiar I saw here https://www.kaggle.com/dhananjay3/panda-eda-all-you-need-to-know\n\"\"\"\nsns.scatterplot(data=df_train_red, x='width', y='height', marker='.',hue='data_provider', ax=ax1)\nax1.set_title(\"Image Sizes by Data Provider\", fontsize=14)\nax1.set_xlabel(\"Image Width\", fontsize=14)\nax1.set_ylabel(\"Image Height\", fontsize=14)\n\nsns.kdeplot(df_train_red[df_train_red.data_provider == 'karolinska'].pixels, label='karolinska', ax=ax2)\nsns.kdeplot(df_train_red[df_train_red.data_provider == 'radboud'].pixels, label= 'radboud', ax=ax2)\n\nax2.set_title(\"Image Sizes by Data Provider\", fontsize=14)\nax2.set_ylabel(\"Pixels per Image\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Quick Check of Masks and ISUP Grade\nLet's do a quick check of the provided masks."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"empty_masks = df_train_red[(df_train_red[1] == 0) & (df_train_red[2] ==0)& (df_train_red[3] ==0) & (df_train_red[4] ==0) & (df_train_red[5] ==0)]\nprint(f\"There are {len(empty_masks)} masks that only contain background pixels.\")\nempty_masks[['image_id', 'data_provider', 'isup_grade', 'gleason_score', 0, 1, 2, 3, 4, 5]]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in empty_masks.image_id:\n    biopsy = skimage.io.MultiImage(PATH + 'train_label_masks/' + i + '_mask.tiff')\n    # Check whether all three channels are empty\n    no_mask = True\n    for j in range(3): \n        if biopsy[-1][:,:, j].max() > 0:\n            print(f\"Found mask for image {i} in channel {j}\")\n            no_mask = False\n        \n    if no_mask == True:\n        print(f\"Couldn't find mask for image {i} in other channels.\")\n        \nempty_masks = empty_masks[['image_id']]\nempty_masks['Suspicious_because'] = 'Background only'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For an ISUP Grade to be higher than 0, I would expect some cancerous tissue to be marked in the masks. Let's have a look whether that is the case:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_train_red[(df_train_red.data_provider == 'karolinska') & (df_train_red.isup_grade > 0) & (df_train_red[2] ==0)][['image_id', 'data_provider', 'isup_grade', 'gleason_score', 0, 1, 2, 3, 4, 5]].style.set_caption('Suspicious Masks provided by Karolinska')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no_cancerous_tissue = df_train_red[(df_train_red.data_provider == 'radboud') & (df_train_red.isup_grade > 0) & (df_train_red[3] ==0) & (df_train_red[4] ==0) & (df_train_red[5] ==0)]\nno_cancerous_tissue[['image_id', 'data_provider', 'isup_grade', 'gleason_score', 0, 1, 2, 3, 4, 5]].style.set_caption('Suspicious Masks provided by Radboud')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Summary of Quick Check of Masks and ISUP Grade\nFor further analysis, we will drop the following test cases:\n* 4 test cases with masks that only contain background and no tissue (contains 3 test cases marked with ISUP Grade > 0 but not cancerous tissue)\n* 85 test cases marked with a ISUP Grade > 0 but not cancerous tissue\n\nHowever, for your model **it might be a good idea to use these test cases for validation. All suspicious test cases found in this EDA are summarized in a .csv file at the end of this kernel.**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Train data shape before second reduction: {len(df_train_red)}\")\ndf_train_red = df_train_red[(~df_train_red.image_id.isin(empty_masks.image_id)) & (~df_train_red.image_id.isin(no_cancerous_tissue.image_id))]\nprint(f\"Train data shape after second reduction: {len(df_train_red)}\")\n\nno_cancerous_tissue = no_cancerous_tissue[['image_id']]\nno_cancerous_tissue['Suspicious_because'] = 'No cancerous tissue but ISUP Grade > 0'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Image and Mask Samples"},{"metadata":{},"cell_type":"markdown","source":"Let's have a quick first look at the differences between the data providers in regards to the original images and the masks.\n\n> **Radboud**: Prostate glands are individually labelled. Valid values are:\n* 0: background (non tissue) or unknown\n* 1: <span style='background :gray' >stroma (connective tissue, non-epithelium tissue)</span> \n* 2: <span style='background :green' >healthy (benign) epithelium</span> \n* 3: <span style='background :orange' >cancerous epithelium (Gleason 3)</span> \n* 4: <span style='background :red' >cancerous epithelium (Gleason 4)</span> \n* 5: <span style='background :darkred' >cancerous epithelium (Gleason 5)</span> \n\n>**Karolinska**: Regions are labelled. Valid values are:\n* [0]: background (non tissue) or unknown\n* [1]: <span style='background :green' >benign tissue (stroma and epithelium combined)</span> \n* [2]: <span style='background :red' >cancerous tissue (stroma and epithelium combined)</span> \n\n(For Karolinska the description actually says values 1 through 3 but in the masks it is 0 through 2.)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_train_red['tissue'] = df_train_red[1] + df_train_red[2] + df_train_red[3] + df_train_red[4] + df_train_red[5]\n\nfig, [ax1, ax2] = plt.subplots(nrows=1, ncols=2, figsize=(16, 6))\n\n\nkarolinska = df_train_red[df_train_red.data_provider == 'karolinska'].groupby('isup_grade')[[1, 2, 'tissue']].mean()\nkarolinska[1] = karolinska[1] /karolinska['tissue']*100\nkarolinska[2] = karolinska[2] /karolinska['tissue'] *100\nkarolinska = karolinska.drop(['tissue'], axis=1)\n\nkarolinska.plot(kind='bar', stacked=True, ax=ax1, cmap = matplotlib.colors.ListedColormap(['green','red']))\nax1.set_title(\"Tissue Labels by Karolinska\", fontsize=14)\nax1.set_xlabel(\"ISUP Grade\", fontsize=14)\nax1.set_ylabel(\"Percentage of Labeled Tissue\", fontsize=14)\n\n\nradboud = df_train_red[df_train_red.data_provider == 'radboud'].groupby('isup_grade')[[1, 2, 3, 4, 5, 'tissue']].mean()\nradboud[1] = radboud[1] /radboud['tissue']*100\nradboud[2] = radboud[2] /radboud['tissue'] *100\nradboud[3] = radboud[3] /radboud['tissue'] *100\nradboud[4] = radboud[4] /radboud['tissue'] *100\nradboud[5] = radboud[5] /radboud['tissue'] *100\n\nradboud = radboud.drop(['tissue'], axis=1)\n\nradboud.plot(kind='bar', stacked=True, ax=ax2, cmap = matplotlib.colors.ListedColormap(['lightgrey', 'green', 'orange', 'red', 'darkred']))\nax2.set_title(\"Tissue Labels by Radboud\", fontsize=14)\nax2.set_xlabel(\"ISUP Grade\", fontsize=14)\nax2.set_ylabel(\"Percentage of Labeled Tissue\", fontsize=14)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on this first comparison of pixel values, we can already see:\n* As expected ISUP Grade 0 has no cancerous tissue\n* Radboud labels most of the tissue as non-epithelium tissue. Karolinska does not have this label. In Karolinska's case these would be marked as benign tissue.\n* Karolinska labels larger areas a cancerous tissue\n* Interestingly, the **percentage of cancerous tissue decreases between ISUP Grade 3 and 4** in both data provided by Karolinska and Radboud. In Radboud's case ISUP grade 5 also has a lower percentage of cancerous tissue than ISUP Grade. However, the severity of the marked cancerous tissue shows an increase in the data provided by Radboud.\n\nNow let's look at some tissue samples and their corresponding masks:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def load_and_resize_image(img_id):\n    \"\"\"\n    Edited from https://www.kaggle.com/xhlulu/panda-resize-and-save-train-data\n    \"\"\"\n    path = PATH + 'train_images/' + img_id + '.tiff'\n    biopsy = skimage.io.MultiImage(path)\n    return cv2.resize(biopsy[-1], (512, 512))\n\ndef load_and_resize_mask(img_id):\n    \"\"\"\n    Edited from https://www.kaggle.com/xhlulu/panda-resize-and-save-train-data\n    \"\"\"\n    path = PATH + 'train_label_masks/' + img_id + '_mask.tiff'\n    biopsy = skimage.io.MultiImage(path)\n    return cv2.resize(biopsy[-1], (512, 512))[:,:,0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"labels = []\nfor grade in range(df_train_red.isup_grade.nunique()):\n    fig, ax = plt.subplots(nrows=4, ncols=4, figsize=(12, 12))\n\n    for i, row in enumerate(ax):\n        idx = i//2\n        temp = df_train_red[(df_train_red.isup_grade == grade) & (df_train_red.data_provider == providers[idx])].image_id.head(4).reset_index(drop=True)\n        \n        if i%2 < 1:\n            labels.append(f'{providers[idx]} (image)')\n            for j, col in enumerate(row):\n                col.imshow(load_and_resize_image(temp[j]))\n        else:\n            labels.append(f'{providers[idx]} (mask)')\n            for j, col in enumerate(row):\n                if providers[idx] == 'radboud':\n                    col.imshow(load_and_resize_mask(temp[j]), \n                               cmap = matplotlib.colors.ListedColormap(['white', 'lightgrey', 'green', 'orange', 'red', 'darkred']), \n                               norm = matplotlib.colors.Normalize(vmin=0, vmax=5, clip=True))\n                else:\n                    col.imshow(load_and_resize_mask(temp[j]), \n                           cmap = matplotlib.colors.ListedColormap(['white', 'green', 'red']), \n                           norm = matplotlib.colors.Normalize(vmin=0, vmax=2, clip=True))\n\n    for row, r in zip(ax[:,0], labels):\n        row.set_ylabel(r, rotation=90, size='large', fontsize=14)\n\n    plt.suptitle(f'ISUP Grade {grade}', fontsize=14)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at some samples when we use a common mask:\n* 0: background (non tissue) or unknown\n* 1: <span style='background :green' >benign tissue (stroma and epithelium combined)</span> \n* 2: <span style='background :red' >cancerous tissue (stroma and epithelium combined)</span> \n\nWe can see that the labeling between the two data providers still seems to differ: While the masks provided by Karolinska seem to be covering larger areas, the masks provided by Radboud seem to be covering smaller, more specific areas."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#import matplotlib.cm as cm\ncommon_cmap = matplotlib.colors.ListedColormap(['white', 'green', 'red'])\nnorm = matplotlib.colors.Normalize(vmin=0, vmax=2, clip=True)\n#mapper = cm.ScalarMappable(norm=norm, cmap=common_cmap)\n\ncommon_mask_dict = { 0 : 0, #background (non tissue) or unknown\n                    1 : 1, # benign tissue (stroma and epithelium combined)\n                    2 : 1, # benign tissue (stroma and epithelium combined),\n                    3 : 2, # cancerous tissue (stroma and epithelium combined)\n                    4 : 2, # cancerous tissue (stroma and epithelium combined)\n                    5 : 2, # cancerous tissue (stroma and epithelium combined)\n                   }\ndef load_and_resize_mask_with_common_mask(img_id):\n    path = PATH + 'train_label_masks/' + img_id + '_mask.tiff'\n    biopsy = skimage.io.MultiImage(path)\n    mask = cv2.resize(biopsy[-1], (512, 512))[:,:,0]\n    \n    return np.array([common_mask_dict[letter] for letter in mask.reshape(-1)]).reshape(512, 512)\n   \nlabels = []\nfor grade in range(df_train_red.isup_grade.nunique()):\n    fig, ax = plt.subplots(nrows=4, ncols=4, figsize=(12, 12))\n\n    for i, row in enumerate(ax):\n        idx = i//2\n        temp = df_train_red[(df_train_red.isup_grade == grade) & (df_train_red.data_provider == providers[idx])].image_id.head(4).reset_index(drop=True)\n        \n        if i%2 < 1:\n            labels.append(f'{providers[idx]} (image)')\n            for j, col in enumerate(row):\n                col.imshow(load_and_resize_image(temp[j]))\n        else:\n            labels.append(f'{providers[idx]} (mask)')\n            for j, col in enumerate(row):\n                if providers[idx] == 'radboud':\n                    col.imshow(load_and_resize_mask_with_common_mask(temp[j]), cmap = common_cmap,norm=norm)\n                else: \n                    col.imshow(load_and_resize_mask(temp[j]), cmap = common_cmap,norm=norm)\n    for row, r in zip(ax[:,0], labels):\n        row.set_ylabel(r, rotation=90, size='large', fontsize=14)\n\n    plt.suptitle(f'ISUP Grade {grade}', fontsize=14)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Summary of Suspicious Test Cases\nDuring the EDA, we found a few susupicious test cases:\n* Missing mask\n* Mask only contains background\n* Mask does not contain tissue marked as cancerous although ISUP grade > 0\n\nInstead of dropping them, it might be a good idea to use them for validation since I assume only the masks are suspicious but the ISUP grade might be trustworthy. \n\nBelow I have summarized the suspicious test cases and **wrote them to a  file called suspicious_test_cases.csv. Feel free to use it if you like.**\n\nAdditionally, we found one image that has inconsistent ISUP grade and Gleason score. However, this is not critical."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"suspicious_test_cases = no_masks.reset_index(drop=True)\nsuspicious_test_cases = suspicious_test_cases.append(empty_masks.reset_index(drop=True))\nsuspicious_test_cases = suspicious_test_cases.append(no_cancerous_tissue.reset_index(drop=True))\nsuspicious_test_cases.to_csv(\"suspicious_test_cases.csv\",index=False)\nsuspicious_test_cases","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# One Last Thing Before You Go\nThe **test set only contains 3 images** divided into two images provided by Radboud and one image provided by Karolinska. In the training set however, we have more image provided by Karolinska than by Radboud."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(16, 6))\n\nsns.countplot(df_train.data_provider, ax=ax[0])\n\nax[0].set_title(\"Image Sizes by Data Provider\", fontsize=14)\nax[0].set_title(\"Orginial Train Set (Wtih Suspicious Data)\", fontsize=14)\nax[0].set_xlabel(\"Data Provider\", fontsize=14)\nax[0].set_ylabel(\"Number of Image in Set\", fontsize=14)\n\nsns.countplot(df_train_red.data_provider, ax=ax[1])\nax[1].set_title(\"Reduced Train Set (Without Suspicious Data)\", fontsize=14)\nax[1].set_xlabel(\"Data Provider\", fontsize=14)\nax[1].set_ylabel(\"Number of Image in Set\", fontsize=14)\n\nsns.countplot(df_test.data_provider, ax=ax[2], order=df_test.data_provider.value_counts().sort_values(ascending=True).index)\nax[2].set_title(\"Test Set\", fontsize=14)\nax[2].set_xlabel(\"Data Provider\", fontsize=14)\nax[2].set_ylabel(\"Number of Image in Set\", fontsize=14)\nax[2].set_ylim([0, 20])\nplt.show()\n\nsample_submission = pd.read_csv(f'{PATH}sample_submission.csv')\nsample_submission.style.set_caption('Sample Submission')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code References\n\nThanks to [Xhlulu](https://www.kaggle.com/xhlulu) for this cool notebook on the most efficient image loading techniques: https://www.kaggle.com/xhlulu/panda-resize-and-save-train-data\n\nThanks to [Dhananjay Raut](https://www.kaggle.com/dhananjay3) for the cool idea about using scatter plots to show image width and heights: https://www.kaggle.com/dhananjay3/panda-eda-all-you-need-to-know"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}