{"cells":[{"metadata":{"_uuid":"fc84fd93-736b-49ce-90c9-ae7597574959","_cell_guid":"60f6e4c2-02b4-4fc3-8c1a-3b1d408d2f7a","trusted":true},"cell_type":"code","source":"%%html\n

prostate-cancer-grade-assessment

","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7926317-6e88-423e-8a95-69676d5fc970","_cell_guid":"d02c153a-ccd9-4170-9e17-5f700ba7e63e","trusted":true},"cell_type":"markdown","source":"# Sommaire\n1. Objectifs\n2. Comprendre la base de données\n * Comprendre la base de données\n3. Préparation de la base de données\n * Visualisation de données\n * Fixer quelques problèmes dans la base de données\n * Images sans masque\n * ISUP = 2 Gleason score = 4 + 3 \n * remplacer \"négatif\" par \"0+0\"\n * Quelques problèmes dans la base de données","execution_count":null},{"metadata":{"_uuid":"5854a53e-93fc-421d-9b1d-17cf0a96942c","_cell_guid":"28b619f3-7bb6-42df-a1b7-6f74e0b38a35","trusted":true},"cell_type":"markdown","source":"# 1. Objectifs\nDétecter et classer la gravité du cancer de la prostate sur des images d'échantillons de tissus prostatiques.\n\nEn pratique, les échantillons de tissus sont examinés et notés par les pathologistes selon le système de notation dit de Gleason, qui est ensuite converti en grade ISUP.","execution_count":null},{"metadata":{"_uuid":"496d6537-6a94-46de-908b-e964c8f45c05","_cell_guid":"09157560-043a-4833-bf4b-90b95ee6eb32","trusted":true},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_uuid":"5a0b9abd-2132-44c9-9ec7-f8c63e8a24c5","_cell_guid":"3e2a287f-5e22-414d-a5dd-8c8508889d77","trusted":true},"cell_type":"markdown","source":"# 2.Comprendre la base de données\n\n\ntrain.csv et test.csv:\n\n* image_id: Code d'identification de l'image.\n\n* data_provider: Le nom de l'institution qui a fourni les données. L'Institut **Karolinska** et le Centre médical universitaire **Radboud** \n\n\n\n* uniquement dans train.csv\n\n* isup_grade: La gravité du cancer sur une échelle de 0 à 5.\n\n* gleason_score: Un système alternatif d'évaluation de la gravité du cancer avec plus de niveaux que l'échelle ISUP. \n\n* train_images:\n* 10616 images de type .tiff \n * Karolinska=5455 images\n * Radboud=5060 images\n* test_images:\n3 images de type .tiff\n\ntrain_label_masks: Segmentation masks showing which parts of the image led to the ISUP grade. Not all training images have label masks, and there may be false positives or false negatives in the label masks for a variety of reasons. These masks are provided to assist with the development of strategies for selecting the most useful subsamples of the images. The mask values depend on the data provider:","execution_count":null},{"metadata":{"_uuid":"10a84676-7471-469b-bfe2-82453e9a8142","_cell_guid":"7cea7dc2-21cd-4fb7-b9f8-dd56eaa53376","trusted":true},"cell_type":"markdown","source":"# 3.Préparation de la base de données","execution_count":null},{"metadata":{"_uuid":"2b058ab1-d122-4774-bca2-a70717e8ea6c","_cell_guid":"c6096c9f-29ed-43a4-a0ca-c79c71477e68","trusted":true},"cell_type":"markdown","source":"## Visualisation de données","execution_count":null},{"metadata":{"_uuid":"d8e22e80-c430-419c-998c-e64565fa9422","_cell_guid":"34a5dd93-f673-45fd-bf29-cc5eb028a4ae","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib\nimport seaborn as sns\nimport openslide\nimport os\nimport cv2\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.vgg16 import VGG16,preprocess_input\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Dense, Dropout, Input, Flatten,BatchNormalization,Activation\nfrom keras.layers import GlobalMaxPooling2D,GlobalAveragePooling2D\nfrom keras.models import Model\nfrom keras.optimizers import Adam, SGD, RMSprop\nfrom keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\nfrom keras.callbacks.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom sklearn.metrics import cohen_kappa_score\nimport tensorflow as tf\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.metrics import *\ntrain_df = pd.read_csv(\"../input/prostate-cancer-grade-assessment/train.csv\")\nimage_path = \"../input/prostate-cancer-grade-assessment/train_images/\"\nPATH = \"../input/prostate-cancer-grade-assessment/\"\ntrain_df = pd.read_csv(os.path.join(PATH,'train.csv'))\ntest_df = pd.read_csv(os.path.join(PATH,'test.csv'))\ntrain_img_path = '../input/prostate-cancer-grade-assessment/train_images'\ntrain_read_img= pd.read_csv(PATH+\"train.csv\")\nmasks = '../input/prostate-cancer-grade-assessment/train_label_masks'\nimages_train_list = os.listdir(os.path.join(PATH, 'train_images'))\nmasks_list = os.listdir(os.path.join(PATH, 'train_label_masks'))\nsns.set_style(\"darkgrid\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eed03eb4-27e1-42aa-b9f6-5e7e93cf08f1","_cell_guid":"e61b4ef5-26b2-4b29-9bb1-6f604e214554","trusted":true},"cell_type":"code","source":"print(train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c39acbe-1eaa-4a6a-9af5-7a6ddcf144ea","_cell_guid":"9491ae4e-aa01-4411-8775-b87f3ddd3984","trusted":true},"cell_type":"code","source":"print(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfbf294e-eede-41aa-bad8-3eeeb4f577a8","_cell_guid":"2cf4d1c1-134b-4b16-b366-a1b869c2224a","trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31883da2-5cb8-42be-8837-6611ab72cb56","_cell_guid":"f27adfb7-514f-4a4a-9d3b-53d8a5ce837c","trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d347d94b-5cfb-46ec-af10-3f64bc8cc461","_cell_guid":"cc0e83d5-b6ac-4fb2-b367-cbefc19ef47a","trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,5))\nsns.countplot(ax=ax1, x=\"data_provider\", data=train_df)\nax1.set_title(\"distribution de data_provider dans training data\")\nsns.countplot(ax=ax2, x=\"data_provider\", data=test_df)\nax2.set_title(\"distribution de data_provider dans test data\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c83e5464-cc80-40d3-8513-34d1dd5b44ba","_cell_guid":"4b9f6180-34bd-4650-9e45-97e58fdc546a","trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,5))\nsns.countplot(ax=ax1, x=\"isup_grade\", data=train_df)\nax1.set_title(\"ISUP Grade distribution dans Data Provider\")\nsns.countplot(ax=ax2, x=\"gleason_score\", data=train_df)\nax2.set_title(\"Gleason_Score distribution dans Data Provider\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c188b42-ba94-4fa1-b6f0-9f9ac114bffb","_cell_guid":"9d9cb83d-246f-4f55-8535-2ef5d95f3311","trusted":true},"cell_type":"code","source":"from tqdm import tqdm\n\nimg_dim= []\n\nfor i,row in tqdm(train_df.iterrows()):\n slide = openslide.OpenSlide(os.path.join(train_img_path, train_df.image_id.iloc[i]+'.tiff'))\n img_dim.append(slide.dimensions)\n slide.close()\n \nwidth = [dimensions[0] for dimensions in img_dim] \nheight = [dimensions[1] for dimensions in img_dim] \n\ntrain_df['width'] = width\ntrain_df['height'] = height","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"671764f9-dbab-45df-bd9c-416046d48636","_cell_guid":"198dcbc6-6deb-4232-8b54-31504f949aab","trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(20,5))\nax = sns.scatterplot(x='width', y='height', data=train_df, hue='data_provider', alpha=0.70)\nax.tick_params(labelsize=10)\n\nplt.title('Dimensions des images')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5bb7f5a-6451-48f0-bd4a-14bd3f05d350","_cell_guid":"ff0284eb-3253-482d-a074-110e5eb0ecb6","trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2)\nfig.set_size_inches(20, 5)\n\nsns.stripplot(train_df['width'],train_df['data_provider'],ax=ax[0],jitter=True)\nsns.stripplot(train_df['height'],train_df['data_provider'],ax=ax[1],jitter=True)\n\nax[0].tick_params(labelsize=10)\nax[1].tick_params(labelsize=10)\nax[0].tick_params(labelrotation=90)\nax[1].tick_params(labelrotation=90)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0934906-e512-43c7-9864-89b185ae60ce","_cell_guid":"861b375a-388b-4cf8-b464-ecdc6fb060ae","trusted":true},"cell_type":"code","source":"data_file_masks = pd.Series(masks_list).to_frame()\ndata_file_masks.columns = ['mask_file_name']\ndata_file_masks.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29d987ba-0ecd-462e-b8de-707944fd8fc3","_cell_guid":"c9a78487-dba3-4fe9-a8c3-c45a68ee4179","trusted":true},"cell_type":"code","source":"data_file_masks['image_id'] =data_file_masks.mask_file_name.apply(lambda x: x.split('_')[0])\ndata_file_masks.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc9e81b2-f12e-4baf-9786-1abefdbba3b1","_cell_guid":"134464bc-cc99-46a6-85c1-fd6b1caace6b","trusted":true},"cell_type":"code","source":"train_df = pd.merge(train_df, data_file_masks, on='image_id', how='outer')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6aaabafc-739c-4429-ad61-1955294f2174","_cell_guid":"65e970df-3123-4076-909b-dc8ff33f80fd","trusted":true},"cell_type":"markdown","source":"## Fixer quelques problèmes dans la base de données","execution_count":null},{"metadata":{"_uuid":"11815ae2-1c3d-4b8b-bfd5-205304411079","_cell_guid":"a3cc9f44-13f5-47d7-ac9d-e2db3a900f2f","trusted":true},"cell_type":"markdown","source":"# Images sans masque\nil y a des images sans masque dans la base de ddonnées","execution_count":null},{"metadata":{"_uuid":"784062dc-f571-426f-b207-0c77a3230d55","_cell_guid":"d10c6d24-baf0-43ff-b42e-3e5668343ac0","trusted":true},"cell_type":"code","source":"del data_file_masks\nprint(f\"Il y a {len(train_df[train_df.mask_file_name.isna()])} images sans masque.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7f779e8-8526-4af9-b9d2-8c340d6af71a","_cell_guid":"ed96ac2a-27b2-437d-aace-ef7baf143ed9","trusted":true},"cell_type":"code","source":"print(f\"Train data avant la réduction: {len(train_df)}\")\ndf_train_reduction= train_df[~train_df.mask_file_name.isna()]\nprint(f\"Train data après la réduction: {len(df_train_reduction)}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7181047-b6af-4806-ad4f-c06a3b5f1133","_cell_guid":"5ee95c78-e1f3-4799-b35b-d2979b476b21","trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(1,2,figsize=(20,5))\ntrain_df['data_provider'].value_counts().plot.pie(autopct='%1.1f%%',ax=ax[0])\nax[0].set_ylabel('')\ndf_train_reduction['data_provider'].value_counts().plot.pie(autopct='%1.1f%%',ax=ax[1])\nax[1].set_ylabel('')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"601a1ff6-6ca4-4fac-a0f3-2581f6b5b079","_cell_guid":"2e349714-50e1-4d06-8563-57647ca8950f","trusted":true},"cell_type":"markdown","source":"* le test data contient uniquement 3 images , donc je vais créer un autre fichier new_test.csv avec les 100 images que j'ai supprimé (images sans masque)","execution_count":null},{"metadata":{"_uuid":"cde152f7-004e-4890-9a48-c8102125dcc1","_cell_guid":"e47a89b8-d55c-4818-a6dc-c706b50ab84e","trusted":true},"cell_type":"code","source":"images_without_masks=train_df[train_df.mask_file_name.isna()]\nwithout_masks=images_without_masks.groupby('image_id').data_provider.unique().to_frame()\nwithout_masks.to_csv(\"new_test.csv\",index=False)\nwithout_masks\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c067b7e0-3550-4ba0-928a-5b32fe0f6ef2","_cell_guid":"a2f011f2-5619-4963-afbb-e51682097637","trusted":true},"cell_type":"markdown","source":"inspiré de : [Links](https://medium.com/@kvnamipara/a-better-visualisation-of-pie-charts-by-matplotlib-935b7667d77f)","execution_count":null},{"metadata":{"_uuid":"1d0b5a1a-9db9-48a2-9399-30b46d7923ab","_cell_guid":"796d1630-c672-40e1-9bcd-0b483fe245e5","trusted":true},"cell_type":"markdown","source":"* 1. ISUP grade = 0 Gleason score 0+0 or negative.\n* 1. ISUP grade = 1 Gleason score 3+3.\n* 1. ISUP grade = 2 Gleason score 3+4.\n* 1. ISUP grade = 3 Gleason score 4+3.\n* 1. ISUP grade = 4 Gleason score 4+4 (majority), 3+5 or 5+3.\n* 1. ISUP grade = 5 Gleason score 4+5 (majority), 5+4 or 5+5.","execution_count":null},{"metadata":{"_uuid":"5c42e882-b445-4362-8a08-e366b881ec76","_cell_guid":"73f37bc4-4843-4dc9-92fd-a8ccca5e88b1","trusted":true},"cell_type":"code","source":"df_train_reduction.groupby('isup_grade').gleason_score.unique().to_frame()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3601aa0e-6511-4f6b-b8ba-87ff8178f086","_cell_guid":"9863a6ab-c13c-48ba-a8f2-75c07b2c26d2","trusted":true},"cell_type":"markdown","source":"# ISUP = 2 Gleason score = 4 + 3 \n** Il n'y a pas de ISUP = 2 , Gleason score = 4+3 dans le système de notation Gleason + il n'y a qu'une seule image de ce type et elle semble être une erreur, je vais donc la supprimer.**","execution_count":null},{"metadata":{"_uuid":"624075a6-ef37-4f82-9b06-25f2d9d4771a","_cell_guid":"ecf15552-20f4-4b9e-8e26-3de4e414529b","trusted":true},"cell_type":"code","source":"df_train_reduction[(df_train_reduction.isup_grade == 2) & (df_train_reduction.gleason_score == '4+3')].reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d683692-be9c-409f-b721-ae36ff1b38d9","_cell_guid":"e01cf395-e1d8-464a-9909-f70c2c11292b","trusted":true},"cell_type":"code","source":"df_train_reduction.reset_index(inplace=True)\ndf_train_reduction = df_train_reduction[df_train_reduction.image_id !='b0a92a74cb53899311acc30b7405e101']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"292c975d-e085-467c-a34e-0227fffb3e12","_cell_guid":"162a2b78-cd84-4f04-ad87-61a5e53ed566","trusted":true},"cell_type":"code","source":"df_train_reduction[(df_train_reduction.isup_grade == 2) & (df_train_reduction.gleason_score == '4+3')].reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfbda04a-6a7d-42da-8087-9088ff992970","_cell_guid":"5f731547-1abb-44a9-a298-405a008bf089","trusted":true},"cell_type":"code","source":"df_train_reduction.groupby('isup_grade').gleason_score.unique().to_frame()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9ecca22-df38-4e17-8bcb-a311c3a850d3","_cell_guid":"0a7c6dbd-4d8f-401e-9a42-ce6597f71f2d","trusted":true},"cell_type":"code","source":"temp = df_train_reduction.groupby('isup_grade').count()['image_id'].reset_index().sort_values(by='image_id',ascending=False)\ntemp.style.background_gradient(cmap='Purples')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23bee60e-dbd3-4a74-8c2e-af66c5e4483e","_cell_guid":"6418279c-6dd1-461e-bc29-703b443f47ea","trusted":true},"cell_type":"code","source":"temp = df_train_reduction.groupby('gleason_score').count()['image_id'].reset_index().sort_values(by='image_id',ascending=False)\ntemp.style.background_gradient(cmap='Reds')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ab5a6dd-dacd-4994-a0f4-7ed7b17abb91","_cell_guid":"e213a378-758d-405c-8661-f0eb818cb6d6","trusted":true},"cell_type":"markdown","source":"# remplacer \"negative\" par \"0+0\"","execution_count":null},{"metadata":{"_uuid":"90164be3-c3c6-4563-91c4-79736766506a","_cell_guid":"f5d16a30-c095-4735-827c-b068863c163f","trusted":true},"cell_type":"code","source":"df_train_reduction[(df_train_reduction.isup_grade == 0) & (df_train_reduction.gleason_score =='negative')].reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8e04c3d-f802-4653-9d7b-70c4e8fbd0cb","_cell_guid":"7af08ef0-2394-4ae6-ab1b-8ffc59f89705","trusted":true},"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nfig= plt.subplots(figsize=(20,5))\nsns.countplot(x='gleason_score', hue=\"data_provider\", data=df_train_reduction)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d76c482-5a44-4ed3-9b78-4f9cc46acb80","_cell_guid":"67414289-068d-4b46-8a18-5417bfbb87b6","trusted":true},"cell_type":"markdown","source":"* nous pouvons voir que radboud n'a pas de valeurs \"0+0\" alors que karolinska n'a pas de valeurs \"negative\".\n* conclusion : \"negative\" correspond à la façon dont le radbound représente \"0+0\" (c'est-à-dire l'absence de cancer) ; il serait donc plus logique de remplacer \"negative\" par \"0+0\".","execution_count":null},{"metadata":{"_uuid":"ab58d7de-ec4a-4680-a7c4-dabb55505cf9","_cell_guid":"8cda8234-2b8f-4c01-8464-db9eac3e22de","trusted":true},"cell_type":"code","source":"df_train_reduction[\"gleason_score\"]= df_train_reduction[\"gleason_score\"].replace(\"negative\", \"0+0\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b981cec-9b44-4bde-a1c9-61d24059d56d","_cell_guid":"bd909551-c1a1-4f2e-8994-4d145d56f71a","trusted":true},"cell_type":"code","source":"df_train_reduction.groupby('isup_grade').gleason_score.unique().to_frame()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0543c469-e456-4c65-a96a-759eaa22c4ce","_cell_guid":"070b14ad-0e34-4a0c-9187-165cfeabd004","trusted":true},"cell_type":"code","source":"temp = df_train_reduction.groupby('gleason_score').count()['image_id'].reset_index().sort_values(by='image_id',ascending=False)\ntemp.style.background_gradient(cmap='Reds')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97d8f9a0-877f-45b0-9e29-3b28c79b8287","_cell_guid":"797363e9-e421-4a27-929f-71c1bffd3037","trusted":true},"cell_type":"markdown","source":"# Affichage de quelques images","execution_count":null},{"metadata":{"_uuid":"78dc548c-0b82-4031-8153-3bc63a145a82","_cell_guid":"b2bba995-1c81-4b64-b15d-c056588b038d","trusted":true},"cell_type":"code","source":"def show_images(df, read_region=(1780,1950)):\n \n data = df\n f, ax = plt.subplots(3,3, figsize=(20,20))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join(PATH,\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n ax[i//3, i%3].imshow(patch) \n image.close() \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title('ID: {}\\nSource: {} ISUP: {} Gleason: {}'.format(\n data_row[1][0], data_row[1][1], data_row[1][2], data_row[1][3]))\n\n plt.show()\n \nimages = [\n '07a7ef0ba3bb0d6564a73f4f3e1c2293',\n '037504061b9fba71ef6e24c48c6df44d',\n '035b1edd3d1aeeffc77ce5d248a01a53',\n '059cbf902c5e42972587c8d17d49efed',\n '06a0cbd8fd6320ef1aa6f19342af2e68',\n '06eda4a6faca84e84a781fee2d5f47e1',\n '0a4b7a7499ed55c71033cefb0765e93d',\n '0838c82917cd9af681df249264d2769c',\n '046b35ae95374bfb48cdca8d7c83233f'\n]\ndata_sample = train_df.loc[train_df.image_id.isin(images)]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6df6796a-6835-4de9-87fe-fe2e529a15b8","_cell_guid":"dcb28666-fa8e-44cb-aa13-cd8208e7ecdb","trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,5))\nsns.countplot(ax=ax1, x=\"isup_grade\", data=df_train_reduction)\nax1.set_title(\"ISUP Grade Count by Data Provider\")\nsns.countplot(ax=ax2, x=\"gleason_score\", data=df_train_reduction)\nax2.set_title(\"Gleason_Score Count by Data Provider\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1409558-2654-4d40-9865-bf791905bead","_cell_guid":"db52a2c1-94fe-4d13-b383-1bc7d7a04e91","trusted":true},"cell_type":"markdown","source":"# Affichage de quelques masques pour loacaliser le cancer et comprendre chaque grade de la maladie","execution_count":null},{"metadata":{"_uuid":"949108ce-98ce-4cca-86d9-846f0373c8ef","_cell_guid":"73c77ec6-7d20-4ffd-bc1a-dae9cf0b9e3c","trusted":true},"cell_type":"code","source":"def show_masks(slides): \n f, ax = plt.subplots(5,3, figsize=(18,22))\n for i, slide in enumerate(slides):\n mask = openslide.OpenSlide(os.path.join(mask_dir, f'{slide}_mask.tiff'))\n mask_data = mask.read_region((0,0), mask.level_count - 1, mask.level_dimensions[-1])\n cmap = matplotlib.colors.ListedColormap(['black', 'gray', 'green', 'yellow', 'orange', 'red'])\n ax[i//3, i%3].imshow(np.asarray(mask_data)[:,:,0], cmap=cmap, interpolation='nearest', vmin=0, vmax=5) \n mask.close() \n ax[i//3, i%3].axis('off') \n image_id = slide\n data_provider = data_sample_mask.loc[slide, 'data_provider']\n isup_grade = data_sample_mask.loc[slide, 'isup_grade']\n gleason_score = data_sample_mask.loc[slide, 'gleason_score']\n ax[i//3, i%3].set_title(f\"ID: {image_id}\\nSource: {data_provider} ISUP: {isup_grade} Gleason: {gleason_score}\")\n f.tight_layout()\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"45eadf49-44e2-489d-a351-866ec3b43892","_cell_guid":"3f5ccf67-6214-487c-953c-7082a0735946","trusted":true},"cell_type":"code","source":"images_mask = [\n '07a7ef0ba3bb0d6564a73f4f3e1c2293',\n '037504061b9fba71ef6e24c48c6df44d',\n '035b1edd3d1aeeffc77ce5d248a01a53',\n '059cbf902c5e42972587c8d17d49efed',\n '06a0cbd8fd6320ef1aa6f19342af2e68',\n '06eda4a6faca84e84a781fee2d5f47e1',\n '0a4b7a7499ed55c71033cefb0765e93d',\n '0838c82917cd9af681df249264d2769c',\n '028098c36eb49a8c6aa6e76e365dd055',\n '0280f8b612771801229e2dde52371141',\n '028dc05d52d1dd336952a437f2852a0a',\n '02a2dcd6ad8bc1d9ad7fdc04ffb6dff3',\n '049031b0ea0dede1ca1e5ca470c1332d',\n '05f4e9415af9fdabc19109c980daf5ad',\n '07fd8d4f02f9b95d86da4bc89563e077'\n]\n\nmask_dir = os.path.join(PATH,\"train_label_masks\")\ndata_sample_mask = df_train_reduction.set_index('image_id')\nshow_masks(images_mask)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"820bd10d-b282-416b-a4f4-c1461db241b0","_cell_guid":"9d2f78a5-a625-4237-b7da-586b39c01308","trusted":true},"cell_type":"markdown","source":"# Affichage de quelques images et leurs masques","execution_count":null},{"metadata":{"_uuid":"77b06b2b-7d17-4749-be71-2aa61da9bbbc","_cell_guid":"226a87ab-9451-4c75-8db0-f80595459638","trusted":true},"cell_type":"code","source":"def mask_img(image,max_size=(600,400)):\n slide = openslide.OpenSlide(os.path.join(train_img_path, f'{image}.tiff'))\n mask = openslide.OpenSlide(os.path.join(mask_dir, f'{image}_mask.tiff'))\n f,ax = plt.subplots(1,2 ,figsize=(18,22))\n spacing = 1 / (float(slide.properties['tiff.XResolution']) / 10000)\n img = slide.get_thumbnail(size=(600,400)) \n mask_data = mask.read_region((0,0), mask.level_count - 1, mask.level_dimensions[-1])\n cmap = matplotlib.colors.ListedColormap(['black', 'gray', 'green', 'yellow', 'orange', 'red'])\n \n \n ax[0].imshow(img)\n ax[1].imshow(np.asarray(mask_data)[:,:,0], cmap=cmap, interpolation='nearest', vmin=0, vmax=5) \n \n image_id = image\n data_provider = data_sample_mask.loc[image, 'data_provider']\n isup_grade = data_sample_mask.loc[image, 'isup_grade']\n gleason_score = data_sample_mask.loc[image, 'gleason_score']\n ax[0].set_title(f\"ID: {image_id}\\nSource: {data_provider} ISUP: {isup_grade} Gleason: {gleason_score} IMAGE\")\n ax[1].set_title(f\"ID: {image_id}\\nSource: {data_provider} ISUP: {isup_grade} Gleason: {gleason_score} IMAGE_MASK\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b0947c4-f83a-46e4-b166-dc4a54c61f90","_cell_guid":"882daf6d-b17f-4e1f-bd59-9b7b191f5166","trusted":true},"cell_type":"code","source":"images1= [\n '08ab45297bfe652cc0397f4b37719ba1',\n '090a77c517a7a2caa23e443a77a78bc7',\n '07fd8d4f02f9b95d86da4bc89563e077'\n]\n\nfor image in images1:\n mask_img(image)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f916ce1-8671-4f64-8c63-32d14ccd39e0","_cell_guid":"3a706eef-8377-4f71-9c73-8b78687f56e8","trusted":true},"cell_type":"markdown","source":"\npanda-resized-train-data-512x512 , code source : [Links](https://www.kaggle.com/xhlulu/panda-resize-and-save-train-data)","execution_count":null},{"metadata":{"_uuid":"ca644467-25c8-4218-9eb4-ac27b7be3665","_cell_guid":"7023c390-3f0e-4133-b35b-0a92d0430da8","trusted":true},"cell_type":"code","source":"train_df=df_train_reduction\nAccuracies_list=[]\nlabels=[]\ndata=[]\ndata_dir='../input/panda-resized-train-data-512x512/train_images/train_images/'\nfor i in range(train_df.shape[0]):\n data.append(data_dir + train_df['image_id'].iloc[i]+'.png')\n labels.append(train_df['isup_grade'].iloc[i])\ndf=pd.DataFrame(data)\ndf.columns=['images']\ndf['isup_grade']=labels","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d28ea76d-0c1d-428d-81aa-3289e45a6c20","_cell_guid":"84c2fd8b-97ab-4c4e-9b15-c97081429c1b","trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4122294-0102-4217-8a49-3cdc6b86c6bc","_cell_guid":"9ed48424-5a57-4e3f-8175-afc69738cd0d","trusted":true},"cell_type":"code","source":"print(len(df))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f97c959-e766-4b1d-8807-9343b04b4d8e","_cell_guid":"684d2250-d4be-4ba5-a42a-11c12aa30d9d","trusted":true},"cell_type":"code","source":"print(labels)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3eb63ff5-1481-40c3-8312-6014e2bdde5f","_cell_guid":"47164b19-c4eb-48f8-94d1-c510af4e45bf","trusted":true},"cell_type":"markdown","source":"### diviser notre data set","execution_count":null},{"metadata":{"_uuid":"022b3481-a6d8-4741-a90e-be6d993f552e","_cell_guid":"29370d66-250c-4dc8-a45a-89feb45c31b3","trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(df['images'],df['isup_grade'], test_size=0.1, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"232d7974-6af6-45b6-96fe-83dda17f70e5","_cell_guid":"5003ce3e-986f-49e0-bace-da583c15ea93","trusted":true},"cell_type":"code","source":"train=pd.DataFrame(X_train)\ntrain.columns=['images']\ntrain['isup_grade']=y_train\n\nvalidation=pd.DataFrame(X_val)\nvalidation.columns=['images']\nvalidation['isup_grade']=y_val\n\ntrain['isup_grade']=train['isup_grade'].astype(str)\nvalidation['isup_grade']=validation['isup_grade'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c08843c3-1ac0-49d5-b84c-f22f8d9e1a59","_cell_guid":"3f2e137b-c040-4db9-919a-25787a50750b","trusted":true},"cell_type":"code","source":"print(\"train size \",len(train))\nprint(\"validation size \",len(validation))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c83f0b1-6464-4db1-a56c-a42b635f838a","_cell_guid":"f3a53e99-d472-4cbe-813a-b3f8dc72eec6","trusted":true},"cell_type":"code","source":"print(train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a94fea0a-5b35-4a07-9564-946fa5158521","_cell_guid":"ba3864f6-a492-4878-a47b-6a9f3b0fb4ee","trusted":true},"cell_type":"code","source":"print(validation)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09a269ee-79a2-4b99-a7ea-72a947a2d1e0","_cell_guid":"ea150e5a-16b7-41a5-a703-9185bd4492d1","trusted":true},"cell_type":"markdown","source":"### après le divisiment de notre data set","execution_count":null},{"metadata":{"_uuid":"0a35a989-7f73-40a2-8527-d4778138c5bc","_cell_guid":"6500d7b6-7c57-4ba1-906b-bd2e145511f4","trusted":true},"cell_type":"code","source":"sns.set(style=\"darkgrid\")\na = ['TRAIN DATA ','TEST DATA ']\nb = [len((train)),len((validation))]\nax = sns.barplot(x=a, y=b)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"257269b8-b1af-407b-99bb-62b2b9b2ad4c","_cell_guid":"ec669219-29d5-47a6-bfca-af145be41ba5","trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,5))\nsns.countplot(ax=ax1, x=\"isup_grade\", data=train)\nax1.set_title(\"distribution de Grade ISUP dans le TRAIN DATA après le divisiment\")\nsns.countplot(ax=ax2, x=\"isup_grade\", data=validation)\nax2.set_title(\"distribution de Grade ISUP dans le TEST DATA après le divisiment\")\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}