{"cells":[{"metadata":{},"cell_type":"markdown","source":"# PANDA: A rather simple EDA\n\n---\n\nHello everyone :-)\n\nI hope you are having a good time while in lockdown, so I decided to make a constructive use of my time by brushing up on my linear algebra and working on more EDAs. Since I got positive feedback on my last EDA, i decided to make one here.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Contents:\n\n1. Introduction
\n 1-1. General Exploration
\n 1-2. Discrepancies between data providers\n2. Preprocessing
\n 2-1. Background subtractor
\n 2-2. Gaussian Blur
\n 2-3. Grayscale
\n 2-4. Circle crop","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" **1. Introduction**
","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt; import seaborn as sns\nplt.style.use('seaborn-whitegrid')\nimport openslide\nimport os\nimport cv2\nimport torch\ntrain = pd.read_csv('../input/prostate-cancer-grade-assessment/train.csv')\ngpu = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ngpu","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can view images with Gabriel's simple function:","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_images(df, read_region=(1780,1950)):\n data = df\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join('../input/prostate-cancer-grade-assessment',\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n ax[i//3, i%3].imshow(patch) \n image.close() \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n plt.show()\nimages = [\n '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad'] \ndata_sample = train.loc[train.image_id.isin(images)]\nshow_images(data_sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"General exploration
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Alright great! Let's now take a look at our metadata - I am sure there's a lot we can find.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will see the distribution of data providers:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nsns.countplot(train.data_provider);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have two data providers:\n+ **Karolinska Institute:** It is one of the leading cancer research centers in the world and is located in Sweden. It covers a lot of the biological fields of study in its research.\n+ **Radboud University:** It is located in the Netherlands (Nijmegen to be specific)\n\nLet's look at `isup_grade` (our target variable BTW):","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nsns.countplot(train.isup_grade);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Oho... look we have a skewed distribution! Our variables are clustered towards 0 and 1! This is interesting now. Or as a meme would put it:\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Gleason score distributions?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nsns.countplot(train.gleason_score);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have quite a strange distribution here. All this does, is make the problem much much more interesting. BTW, if you are interested in what the Gleason score is check this out:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo(\"1Q7ERNtLcvk\", height=500, width=700)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That was rather informative!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Discrepancies between data providers
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I am not exactly 100 percent sure that the data provided by each is exactly similar, so I will have to see for myself about this suspicious business with the providers. The providers are Karolinska Institue (located in Sweden) and Radboud University (located in the Netherlands)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train[train['data_provider'] == \"karolinska\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['data_provider'] == \"radboud\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Already I feel that the discrepancies in gleason scores are visible IMHO. We will need to see for ourselves as always:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nsns.countplot(train[train['data_provider'] == \"radboud\"].gleason_score, color=\"red\");\nplt.legend()\nplt.title(\"Gleason score(s) of Radboud University's Data\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So it seems like higher gleason scores are present with the Radboud University's data. What about Karolinska Institutute?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nsns.countplot(train[train['data_provider'] == \"karolinska\"].gleason_score, color=\"blue\");\nplt.legend()\nplt.title(\"Gleason score(s) of Karolinska University's Data\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Karolinska Institue and Radboud University both have significant edge cases.\n+ **For Karolinska Institute it is 0+0 gleason score.**\n+ **For Radboud University it is negative gleason score.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We can also use isup_grade as an example now for the discrepancies in the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nsns.countplot(train[train['data_provider'] == \"karolinska\"].isup_grade, color=\"blue\");\nplt.legend()\nplt.title(\"Grade(s) of Karolinska University's Data\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the grades are clustered towards 0 and 1. What about Radboud's data?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nsns.countplot(train[train['data_provider'] == \"radboud\"].isup_grade, color=\"red\");\nplt.legend()\nplt.title(\"Grade(s) of Radboud University's Data\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All the classes have approximately the same counts (with exception of 2 of course). ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now that we have established the data discrepancies, let's move on to preprocessing.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Preprocessing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Background subtractor
","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import cv2; fgbg = cv2.createBackgroundSubtractorMOG2()\n\ndef show_images(df, read_region=(1780,1950)):\n data = df\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join('../input/prostate-cancer-grade-assessment',\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n patch = np.array(patch)\n image = cv2.resize(patch, (256, 256))\n image= fgbg.apply(patch)\n ax[i//3, i%3].imshow(image) \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n plt.show()\nimages = [\n '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad'] \ndata_sample = train.loc[train.image_id.isin(images)]\nshow_images(data_sample)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like we should not use background subtraction. Why? Well, we lose **entire images** using background subtraction which is 100 percent horrible for our model - loss of data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Gaussian Blur
","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def show_images(df, read_region=(1780,1950)):\n data = df\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join('../input/prostate-cancer-grade-assessment',\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n patch = np.array(patch)\n image = cv2.resize(patch, (256, 256))\n image=cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , 256/10) ,-4 ,128)\n ax[i//3, i%3].imshow(image) \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n plt.show()\nimages = [\n '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad'] \ndata_sample = train.loc[train.image_id.isin(images)]\nshow_images(data_sample)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Gaussian Blur does not seem to damage our images too much, Next, we move on to grayscale.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Grayscale images
","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def show_images(df, read_region=(1780,1950)):\n data = df\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join('../input/prostate-cancer-grade-assessment',\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n patch = np.array(patch)\n image = cv2.resize(patch, (256, 256))\n image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n ax[i//3, i%3].imshow(image) \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n plt.show()\nimages = [\n '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad'] \ndata_sample = train.loc[train.image_id.isin(images)]\nshow_images(data_sample)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These help us to visualize the images and the distinctions within each image more clearly. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Circle crop
","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def crop_image_from_gray(img,tol=7):\n if img.ndim ==2:\n mask = img>tol\n return img[np.ix_(mask.any(1),mask.any(0))]\n elif img.ndim==3:\n gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n mask = gray_img>tol\n \n check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n if (check_shape == 0): # image is too dark so that we crop out everything,\n return img # return original image\n else:\n img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n img = np.stack([img1,img2,img3],axis=-1)\n return img\n \ndef circle_crop(img, sigmaX=10): \n \"\"\"\n Create circular crop around image centre \n \"\"\" \n \n img = crop_image_from_gray(img) \n \n height, width, depth = img.shape \n \n x = int(width/2)\n y = int(height/2)\n r = np.amin((x,y))\n \n circle_img = np.zeros((height, width), np.uint8)\n cv2.circle(circle_img, (x,y), int(r), 1, thickness=-1)\n img = cv2.bitwise_and(img, img, mask=circle_img)\n img = crop_image_from_gray(img)\n img=cv2.addWeighted ( img,4, cv2.GaussianBlur( img , (0,0) , sigmaX) ,-4 ,128)\n return img \n\ndef show_images(df, read_region=(1780,1950)):\n data = df\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(data.iterrows()):\n image = str(data_row[1][0])+'.tiff'\n image_path = os.path.join('../input/prostate-cancer-grade-assessment',\"train_images\",image)\n image = openslide.OpenSlide(image_path)\n spacing = 1 / (float(image.properties['tiff.XResolution']) / 10000)\n patch = image.read_region(read_region, 0, (256, 256))\n patch = np.array(patch)\n image = cv2.resize(patch, (256, 256))\n image = circle_crop(image)\n ax[i//3, i%3].imshow(image) \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f'ID: {data_row[1][0]}\\nSource: {data_row[1][1]} ISUP: {data_row[1][2]} Gleason: {data_row[1][3]}')\n\n plt.show()\nimages = [\n '059cbf902c5e42972587c8d17d49efed', '06a0cbd8fd6320ef1aa6f19342af2e68', '06eda4a6faca84e84a781fee2d5f47e1',\n '037504061b9fba71ef6e24c48c6df44d', '035b1edd3d1aeeffc77ce5d248a01a53', '046b35ae95374bfb48cdca8d7c83233f',\n '074c3e01525681a275a42282cd21cbde', '05abe25c883d508ecc15b6e857e59f32', '05f4e9415af9fdabc19109c980daf5ad'] \ndata_sample = train.loc[train.image_id.isin(images)]\nshow_images(data_sample)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like circle crop could be a feasible option, but I am not exactly sure of using circle crop on our images. Why? Well, cropping has a few disadvantages:\n+ **Loss of information**: Losing the possibly helpful information in the corners of our image is very damaging (potentially, yes) because of the issue of potientially cropping away the information.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n\n# Work in progress\n\n---","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}