{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis Kernel objective is to explore the dataset for Prostate cANcer graDe Assessment (PANDA) Challenge.  \n\nProstate cancer is the second most common cancer among males worldwide that results in more than 350k deaths annually. The key to decreasing mortality is developing more precise screeening procedures and diagnostics. Diagnosis of PCa is based on the grading of prostate tissue biopsies. These tissue samples are examined by a pathologist and scored according to the **Gleason grading system**. In the next Figure we show the principle of Gleason grading system.\n\n![](https://murtagh.mhmedical.com/data/books/2471/murtagh7e_c114_f04.png)\n\nThe grading process consists of finding and classifying cancer tissue into so-called Gleason patterns (3, 4, or 5) based on the architectural growth patterns of the tumor (see Figure below). Based on presence of various formations, the Gleason score is given for majority (first digit in the score) and minority Gleason score (the second digit). After the biopsy is assigned a Gleason score (a combination of the two digits), it is converted into an ISUP grade on a 1-5 scale, using the correspondence matrix shown in the next Figure. \n\n![](https://storage.googleapis.com/kaggle-media/competitions/PANDA/Screen%20Shot%202020-04-08%20at%202.03.53%20PM.png)\n"},{"metadata":{},"cell_type":"markdown","source":"# Prepare the Data Analysis\n\n## Load Packages"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd \nimport numpy as np\nimport os\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook\nfrom matplotlib.patches import Rectangle\nimport seaborn as sns\nimport openslide\n%matplotlib inline \nPATH = \"/kaggle/input/prostate-cancer-grade-assessment/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the Data\n\nLet's load the tabular data. There are three files:\n\n* Sample submission;\n* Train;\n* Test.\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sample_submission_df = pd.read_csv(os.path.join(PATH,'sample_submission.csv'))\ntrain_df = pd.read_csv(os.path.join(PATH,'train.csv'))\ntest_df = pd.read_csv(os.path.join(PATH,'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"sample submission shape: {sample_submission_df.shape}\")\nprint(f\"train shape: {train_df.shape}\")\nprint(f\"test shape: {test_df.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sample_submission_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test and sample submission csv files have only 3 rows (samples).\n\nIn train data there is a column (gleason_score) that is not present either in test (with image_id & data_provider) or in sample submission (image_id & isup_grade) csv.\n\nThe objective is to predict isup_grade for each image in test set.\n\n\nLet's check now the train data in train_images and train_label masks folders."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_image_list = os.listdir(os.path.join(PATH, 'train_images'))\ntrain_label_masks_list = os.listdir(os.path.join(PATH, 'train_label_masks'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"train image_id list: {train_df.image_id.nunique()}\")\nprint(f\"train image list: {len(train_image_list)}\")\nprint(f\"train label masks list: {len(train_label_masks_list)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can observe that there are totally 10616 train images, with same number 10616 corresponding images and only 10516 mask images.  \n\n\nConsequently there are 100 images with missing masks."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"sample of image_id list: {train_df.image_id.values[0:3]}\")\nprint(f\"sample of image list: {train_image_list[0:3]}\")\nprint(f\"sample of label masks list: {train_label_masks_list[0:3]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Missing Data\n\nLet's check for missing values in these lists. \n\nFor this, we will remove the suffix and extension from label masks and will remove the extension from image list."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trimmed_image_list = []\nfor img in train_image_list:\n    trimmed_image_list.append(img.split('.tiff')[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trimmed_label_masks_list = []\nfor img in train_label_masks_list:\n    trimmed_label_masks_list.append(img.split('_mask.tiff')[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compare now the intersections of sets of resulted lists."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"intersect_i_m = (set(trimmed_image_list) & set(trimmed_label_masks_list))\nintersect_id_m = (set(train_df.image_id.unique()) & set(trimmed_label_masks_list))\nintersect_id_i = (set(train_df.image_id.unique()) & set(trimmed_image_list))\n\nprint(f\"image (tiff) & label masks: {len(intersect_i_m)}\")\nprint(f\"image_id (train) & label masks: {len(intersect_id_m)}\")\nprint(f\"image_id (train) & image (tiff): {len(intersect_id_i)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both intersections with masks set cardinality is equal with the mask set data dimmensions. And intersection between train image ids and image equals train cardinality.   \n\nThat means that, besides the missing 100 mask images, is no missing data: all images indexed in train data has corresponding tiff images and only 100 tiff label masks are missing. \n\n\nLet's see what images have missing masks."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"missing_masks  = np.setdiff1d(trimmed_image_list,trimmed_label_masks_list)\nprint(f'missing masks: {len(missing_masks)} images (press output button to see the list)')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"print(list(missing_masks))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration\n\n\nLet's check distribution of data_provider, isup_grade and gleason_score in train data."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_count(df, feature, title='', size=2):\n    f, ax = plt.subplots(1,1, figsize=(3*size,2*size))\n    total = float(len(df))\n    sns.countplot(df[feature],order = df[feature].value_counts().index, palette='Set3')\n    plt.title(title)\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(100*height/total),\n                ha=\"center\") \n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_count(train_df, 'data_provider', 'Data provider - data count and percent')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Two research sources are present: Karolinska Institute and Radboud University."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_count(train_df, 'isup_grade','ISUP grade - data count and percent', size=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In 2012, the International Society of Urologic Pathologists (ISUP) proposed a novel, validated grading system for clear cell renal cell carcinoma (ccRCC) and papillary renal cell carcinoma (pRCC) that has been implemented by the World Health Organization (WHO).This system is based primarily on the nucleoli assessment of the tumors, as follows [1]:\n\n* Grade 1: Inconspicuous nucleoli at ×400 magnification and basophilic\n* Grade 2: Clearly visible nucleoli at ×400 magnification and eosinophilic\n* Grade 3: Clearly visible nucleoli at ×100 magnification\n* Grade 4: Extreme pleomorphism or rhabdoid and/or sarcomatoid morphology  \n\n\nMajority of data samples in train set have ISUP grade values 0 or 1 (total > 50%) and the rest of the data samples have associated ISUP grades from 2 to 5 with all ranging in the 11-12% each. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_count(train_df, 'gleason_score', 'Gleason score - data count and percent', size=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is unbalanced with respect of Gleason score values distribution. \n\nLet's check now relative distribution of ISUP grade anf Gleason score values."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6))\ntmp = train_df.groupby('isup_grade')['gleason_score'].value_counts()\ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index()\nsns.barplot(ax=ax,x = 'isup_grade', y='Exams',hue='gleason_score',data=df, palette='Set1')\nplt.title(\"Number of examinations grouped on ISUP grade and Gleason score\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nDiscussion:\n* All exams with ISUP grade = 0 have Gleason score 0+0 or negative.\n* All exams with ISUP grade = 1 have Gleason score 3+3.\n* All exams with ISUP grade = 2 have Gleason score 3+4.\n* All exams with ISUP grade = 3 have Gleason score 4+3.\n* All exams with ISUP grade = 4 have Gleason score 4+4 (majority), 3+5 or 5+3.\n* All exams with ISUP grade = 5 have Gleason score 4+5 (majority), 5+4 or 5+5.\n\nThis data distribution can be better visualized using a heatmap.\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(8,8))\nheatmap_data = pd.pivot_table(df, values='Exams', index=['isup_grade'], columns='gleason_score')\nsns.heatmap(heatmap_data, cmap=\"YlGnBu\",linewidth=0.5, linecolor='blue')\nplt.title('Number of examinations grouped on ISUP grade and Gleason score')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This seems to be aligned with the correspondence matrix as shown in the following table:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import HTML, display\n\ndata = [[\"Gleason Score\", \"ISUP Grade\"],\n        [\"0+0\", \"0\"], [\"negative\", \"0\"],\n        [\"3+3\", \"1\"], [\"3+4\", \"2\"], [\"4+3\", \"3\"], \n        [\"4+4\", \"4\"], [\"3+5\", \"4\"], [\"5+3\", \"4\"],\n        [\"4+5\", \"5\"], [\"5+4\", \"5\"], [\"5+5\", \"5\"],\n        ]\n\ndisplay(HTML(\n   '<table><tr>{}</tr></table>'.format(\n       '</tr><tr>'.join(\n           '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in data)\n       )\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The only misalignment is in the fact that for ISUP Grade 2, in the data we also have Gleason score 4+3 (which also appears for ISUP Grade 3).  \n\nLet's see how Gleason score is grouped by Data source."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \ntmp = train_df.groupby('data_provider')['gleason_score'].value_counts() \ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index() \nsns.barplot(ax=ax,x = 'data_provider', y='Exams',hue='gleason_score',data=df, palette='Set1') \nplt.title(\"Number of examinations grouped on Data provider and Gleason score\") \nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can observe that all of the 0+0 Gleason score data samples are from Karolinska while from Radboud we have most of negative data.\n\nFor Karolinska, next (in terms of frequency) are samples with Gleason score 3+3, 3+4, 4+4.\n\nFor Radboud, next (in terms of frequency) most frequent are samples with Gleason score 4+3, 3+3, 3+4, 4+4, 4+5.\n\n\nLet's see how ISUP grade is distributed with respect of Data provider."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \ntmp = train_df.groupby('data_provider')['isup_grade'].value_counts() \ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index() \nsns.barplot(ax=ax,x = 'data_provider', y='Exams',hue='isup_grade',data=df, palette='Set1') \nplt.title(\"Number of examinations grouped on Data provider and ISUP Grade\") \nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the 0 & 1 ISUP Grade samples are originated from Karolinska.  \n\nMost of the 2-5 ISUP Grade samples are originated from Radboud."},{"metadata":{},"cell_type":"markdown","source":"## Image samples\n\n\nLet's plot some image samples from train_images.  \n\n\nFor this visualization, I reused the code from this Kernel: https://www.kaggle.com/rohitsingh9990/panda-eda-better-visualization"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_images(df, read_region=(1780,1950)):\n    data = df\n    f, ax = plt.subplots(3,3, figsize=(16,18))\n    for i,data_row in enumerate(data.iterrows()):\n        image = str(data_row[1][0])+'.tiff'\n        image_path = os.path.join(PATH,\"train_images\",image)\n        image = openslide.OpenSlide(image_path)\n        spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n        patch = image.read_region(read_region, 0, (256, 256))\n        ax[i//3, i%3].imshow(patch) \n        image.close()       \n        ax[i//3, i%3].axis('off')\n        ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"images = [\n    '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n    '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n    '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad']   \ndata_sample = train_df.loc[train_df.image_id.isin(images)]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Masks samples\n\nLet's show now the masks for the same images."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def display_masks(df, read_region=(0,0)):\n    data = df\n    f, ax = plt.subplots(3,3, figsize=(16,18))\n    for i,data_row in enumerate(data.iterrows()):\n        image = str(data_row[1][0])+'_mask.tiff'\n        image_path = os.path.join(PATH,\"train_label_masks\",image)\n        mask = openslide.OpenSlide(image_path)\n        \n        mask_data = mask.read_region(read_region, mask.level_count - 1, mask.level_dimensions[-1])\n        cmap = matplotlib.colors.ListedColormap(['black', 'lightgray', 'darkgreen', 'yellow', 'orange', 'red'])\n        ax[i//3, i%3].imshow(np.asarray(mask_data)[:,:,0], cmap=cmap, interpolation='nearest', vmin=0, vmax=5) \n        mask.close()       \n        ax[i//3, i%3].axis('off')\n        ax[i//3, i%3].axis('off')\n        ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n        \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_masks(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Gleason score = 5+5\n\nLet's look now to few images and associated masks for samples with Gleason score (5+5)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_images = list(train_df.loc[train_df.gleason_score==\"5+5\", \"image_id\"])\nprint(f\"total samples (Gleason score=5+5): {len(sample_images)}\")\nsample_images = [ '08459aaedfda0679aab403ababbd6ece','0a848ccbbb065ef5ee59dd01710f8531', '0bbbb6734f721f4df4d2ba60ade0ed15', \n                 '0bd231c85b2695e2cf021299e67a6afc',  '0efdb66c93d6b474d93dfe41e40be6ca', '1364c10e1e7f1ad0457f649a44d74888', \n                 '1e644a98460e4f7ea50717720a001efd',  '1fb65315d7ded63d688194863a1b123e', '244d9617bd58fa1db73ab4c1f40d298e']\ndata_sample = train_df.loc[train_df.image_id.isin(sample_images)]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_masks(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Gleason score = 4+5\n\nLet's look now to few images and associated masks for samples with Gleason score (4+5)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_images = list(train_df.loc[train_df.gleason_score==\"4+5\", \"image_id\"])\nprint(f\"total samples (Gleason score=4+5): {len(sample_images)}\")\nsample_images = ['010670e9572e67e5a7e00fb791a343ef', '02f35b793b4fe3032ad6d91f181e391c', '0373ed8a095e0a283da690de360ccc21',\n                 '03b3788b5dca6fdae323d0f1a03c03f6', '046b35ae95374bfb48cdca8d7c83233f', '0550b23f29085f41b10d165a46ad4371', \n                 '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad',   '07aa24f15ce062d65979b6a8bc7eb3f0']\ndata_sample = train_df.loc[train_df.image_id.isin(sample_images)]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_masks(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Gleason score = 3+4\n\nLet's look now to images with Gleason score 3+4."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_images = list(train_df.loc[train_df.gleason_score==\"3+4\", \"image_id\"])\nprint(f\"total samples (Gleason score=3+4): {len(sample_images)}\")\nsample_images =[\n   '022544d1446c2c44f8ca8ff53262dc5b', '061054331a952dd1c9df45c283d756b0', '06bf945aaacb9a67d9f2439d9a7d73ea', \n    '06fb67ffa126811dad6ffb6ecdb8558b', '07697ea97bfbbac071d41a375c3ad036', '083ab9e2c95fb0ea2b999c592fb41653', \n    '08f055372c7b8a7e1df97c6586542ac8', '08f12f69f71b3b4c3c45eafbd710b156', '0a107c91216d62b2543122a46eb26541']\ndata_sample = train_df.loc[train_df.image_id.isin(sample_images[0:9])]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_masks(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Gleason score = 3+3\n\nLet's look now to images with Gleason score 3+3.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_images = list(train_df.loc[train_df.gleason_score==\"3+3\", \"image_id\"])\nprint(f\"total samples (Gleason score=3+3): {len(sample_images)}\")\nsample_images =[\n '004dd32d9cd167d9cc31c13b704498af', '00c46b336b5b06423fcdec1b4d5bee06', '00e6511435645e50673991768a713c66',\n '00ee879798782aca1248baa9132d7307', '0280f8b612771801229e2dde52371141', '03849b0243900d79446bb27849dc0bb2', \n '0dccbb58add854759951038d5ee736ab', '0dfaba8f37fac34150fb70ca8e425141',  '0e347ad243e4019fad579d3282a730f9']\ndata_sample = train_df.loc[train_df.image_id.isin(sample_images[0:9])]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_masks(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Extract image characteristics\n\nLet's parse all images for train data to extract image characteristics."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import time\nstart_time = time.time()\nslide_dimensions, spacings, level_counts = [], [], []\n\nfor image_id in train_df.image_id:\n    image = str(image_id)+'.tiff'\n    image_path = os.path.join(PATH,\"train_images\",image)\n    slide = openslide.OpenSlide(image_path)\n    spacing = 1 / (float(slide.properties['tiff.XResolution']) / 10000)\n    slide_dimensions.append(slide.dimensions)\n    spacings.append(spacing)\n    level_counts.append(slide.level_count)\n    slide.close()\n    del slide\n\ntrain_df['width']  = [i[0] for i in slide_dimensions]\ntrain_df['height'] = [i[1] for i in slide_dimensions]\ntrain_df['spacing'] = spacings\ntrain_df['level_count'] = level_counts\n\nend_time = time.time()\nprint(f\"Total processing time: {round(end_time - start_time,2)} sec.\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now represent the distribution of image dimmensions (width, height), the spacing and level_count, also related to the other features, namely data_provide, isup_grade and gleason_score."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\" level count: {train_df.level_count.nunique()}\")\nprint(f\" spacing: {train_df.spacing.nunique()}\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \ntmp = train_df.groupby('data_provider')['spacing'].value_counts() \ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index() \nsns.barplot(ax=ax,x = 'data_provider', y='Exams',hue='spacing',data=df, palette='Set1') \nplt.title(\"Number of examinations grouped on Data provider and Spacing\") \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \ntmp = train_df.groupby('isup_grade')['spacing'].value_counts() \ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index() \nsns.barplot(ax=ax,x = 'isup_grade', y='Exams',hue='spacing',data=df, palette='Set1') \nplt.title(\"Number of examinations grouped on ISUP Grade and Spacing\") \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \ntmp = train_df.groupby('gleason_score')['spacing'].value_counts() \ndf = pd.DataFrame(data={'Exams': tmp.values}, index=tmp.index).reset_index() \nsns.barplot(ax=ax,x = 'gleason_score', y='Exams',hue='spacing',data=df, palette='Set1') \nplt.title(\"Number of examinations grouped on Gleason Score and Spacing\") \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \nsns.distplot(train_df['width'], kde=True, label='width')\nsns.distplot(train_df['height'], kde=True, label='height')\nplt.xlabel('dimension')\nplt.title('Images Width and Height distribution')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_distribution_grouped(feature, feature_group, hist_flag=True):\n    fig, ax = plt.subplots(nrows=1,figsize=(12,6)) \n    for f in train_df[feature_group].unique():\n        df = train_df.loc[train_df[feature_group] == f]\n        sns.distplot(df[feature], hist=hist_flag, label=f)\n    plt.title(f'Images {feature} distribution, grouped by {feature_group}')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('width', 'data_provider')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('height', 'data_provider')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('width', 'isup_grade', False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('height', 'isup_grade', False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('width', 'gleason_score', False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_distribution_grouped('height', 'gleason_score', False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n\n[1] YiFen Zhang, MD, PhD, Pathology Grading of Renal Cell Carcinoma, https://emedicine.medscape.com/article/1612022-overview  \n[2] Rohit Singh, PANDA - EDA + Better Visualization, https://www.kaggle.com/rohitsingh9990/panda-eda-better-visualization  \n[3] Dhananjay Raut, PANDA: EDA All you need to know, https://www.kaggle.com/dhananjay3/panda-eda-all-you-need-to-know\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}