{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n \n

RSNA-BREAST CANCER DETECTION EDA and Image Analysis in different spectrum

\n\n
\n \"Heat\n
","metadata":{}},{"cell_type":"markdown","source":"

TABLE OF CONTENTS

\n\n\n\n0. **[History and challenges in breast cancer ](#HCBC)**\n\n1. **[Requirments for development](#LR)**
\n 1a.[Importing of python library for operations](#Ipy)
\n 1b.[Analysing Images as they are in DCM formate hence we need these requires for viewing it ](#AID)
\n\n2. **[Post and Pre Processing for DATA from CSV](#PPD)**
\n 2a.[Data Frame View](#DFV)
\n 2b.[Identification of the NA and filling unknown values](#NA)
\n 2c.[No duplicate values found](#ND)
\n 2d.[Drop of unwanted data](#DWD)
\n 2e.[Corrleation found in data ](#Corr)
\n \n3. **[Taking into account of data for image classification](#TAK)**
\n 3a.[Loding the address of the image into a array](#LAI)
\n 3b.[Generating DataFrame for just training in csv ](#GDF)
\n 3c.[Discovery made via count of data](#DVD)
\n\n4. **[EDA Data Visualization ](#EDA)**
\n\n5. **[Image Analysis](#IA)**
\n 5a.[To store DATA for train and testing in png as seperate folder](#DTT)
\n 5b.[Access DATA for train and testing from CSV](#DTC)
\n\n6. **[Modeling and prediction ](#MOD)**
\n\n \n","metadata":{}},{"cell_type":"markdown","source":"
\n\n \n

History and challenges in breast cancer

\n

Determine the presence of cancer through Image processing and Neural Networks

\n\nInitial efforts were focused on developing computer-aided diagnosis (CAD) systems that could assist radiologists in detecting breast tumors in mammograms. These systems used machine learning algorithms to analyze the images and identify regions of interest (ROIs) that might contain tumors.\n
\"Heat
\nOver the years, advances in AI and machine learning techniques have led to the development of more sophisticated systems that can accurately classify breast tumors based on various imaging and clinical features. There has also been a growing interest in using AI for personalized treatment planning for breast cancer patients, with the goal of improving outcomes and reducing treatment-related side effects. \n \n

Challenges we face for the detection of tumors in these images:

\n1. When the Tumors are located near the Lobules their identification is much complex
\n2.The inconsistency of the screening data is also a concurn that needs to be identified
\n
\n","metadata":{}},{"cell_type":"markdown","source":"
Loading in the requirements
\n
\n\n### Importing of python library for operations ","metadata":{}},{"cell_type":"code","source":"import os\nfrom collections import Counter\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nimport statistics\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Activation, Conv2D, Flatten, Dropout, MaxPooling2D, BatchNormalization\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom keras import regularizers, optimizers\nimport tensorflow as tf\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:02:10.148329Z","iopub.execute_input":"2023-02-12T05:02:10.148978Z","iopub.status.idle":"2023-02-12T05:02:16.790712Z","shell.execute_reply.started":"2023-02-12T05:02:10.148896Z","shell.execute_reply":"2023-02-12T05:02:16.789626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analysing Images as they are in DCM formate hence we need these requires for viewing it \n\n* Done to avoid GDCM error reset ones these codes are execited","metadata":{}},{"cell_type":"code","source":"'''\n!python -m pip install pillow\n!conda install -c conda-forge pillow -y\n!conda install -c conda-forge pydicom -y\n!conda install -c conda-forge gdcm -y\n\n'''","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-02-12T05:02:16.792874Z","iopub.execute_input":"2023-02-12T05:02:16.793628Z","iopub.status.idle":"2023-02-12T05:02:16.802252Z","shell.execute_reply.started":"2023-02-12T05:02:16.793589Z","shell.execute_reply":"2023-02-12T05:02:16.801127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''!pip install pylibjpeg pylibjpeg-libjpeg\n!pip install pylibjpeg pylibjpeg-libjpeg pylibjpeg-openjpeg'''\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-02-12T05:02:16.804348Z","iopub.execute_input":"2023-02-12T05:02:16.804765Z","iopub.status.idle":"2023-02-12T05:02:16.824412Z","shell.execute_reply.started":"2023-02-12T05:02:16.804718Z","shell.execute_reply":"2023-02-12T05:02:16.823095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Post and Pre Processing for DATA from CSV
","metadata":{}},{"cell_type":"code","source":"train_images = Path(\"/kaggle/input/rsna-breast-cancer-detection/train_images\")\ntest_images = Path(\"/kaggle/input/rsna-breast-cancer-detection/test_images\")","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:02:16.827007Z","iopub.execute_input":"2023-02-12T05:02:16.828093Z","iopub.status.idle":"2023-02-12T05:02:16.835081Z","shell.execute_reply.started":"2023-02-12T05:02:16.828051Z","shell.execute_reply":"2023-02-12T05:02:16.833852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\nsample_submission_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:02:16.836445Z","iopub.execute_input":"2023-02-12T05:02:16.836868Z","iopub.status.idle":"2023-02-12T05:02:16.937979Z","shell.execute_reply.started":"2023-02-12T05:02:16.83684Z","shell.execute_reply":"2023-02-12T05:02:16.936728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission_df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:02:16.939118Z","iopub.execute_input":"2023-02-12T05:02:16.93942Z","iopub.status.idle":"2023-02-12T05:02:16.954424Z","shell.execute_reply.started":"2023-02-12T05:02:16.939395Z","shell.execute_reply":"2023-02-12T05:02:16.953689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Frame View ","metadata":{}},{"cell_type":"code","source":"train_df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:07.045232Z","iopub.execute_input":"2023-02-12T05:03:07.045632Z","iopub.status.idle":"2023-02-12T05:03:07.072855Z","shell.execute_reply.started":"2023-02-12T05:03:07.045599Z","shell.execute_reply":"2023-02-12T05:03:07.071827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.info()) ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:07.945527Z","iopub.execute_input":"2023-02-12T05:03:07.946566Z","iopub.status.idle":"2023-02-12T05:03:07.97489Z","shell.execute_reply.started":"2023-02-12T05:03:07.946525Z","shell.execute_reply":"2023-02-12T05:03:07.973969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Identification of the NA and filling unknown values ","metadata":{}},{"cell_type":"code","source":"train_df.isnull().values.any()\ntrain_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:09.375596Z","iopub.execute_input":"2023-02-12T05:03:09.376227Z","iopub.status.idle":"2023-02-12T05:03:09.399651Z","shell.execute_reply.started":"2023-02-12T05:03:09.376189Z","shell.execute_reply":"2023-02-12T05:03:09.398584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### No duplicate values found ","metadata":{}},{"cell_type":"code","source":"print('Total Row we have before drop:',len(train_df))\ntrain_df.drop_duplicates(inplace = True)\nprint('Total Row we have after drop: ',len(train_df)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:11.761796Z","iopub.execute_input":"2023-02-12T05:03:11.76216Z","iopub.status.idle":"2023-02-12T05:03:11.797067Z","shell.execute_reply.started":"2023-02-12T05:03:11.76213Z","shell.execute_reply":"2023-02-12T05:03:11.796111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**As data NA for age is the least this can be filled with the help of mean median or mode from the data provided
\n1.mean: 59
\n2.median: 59
\n3.mode: 0 - 50.0
\nhence from the above calculation we take mean as a replacement for NA values**","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\nmean = train_df[\"age\"].mean()\nmean = mean.round()\nprint('The value',mean,' has replaced NA in age')\ntrain_df[\"age\"].fillna(mean, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:13.459548Z","iopub.execute_input":"2023-02-12T05:03:13.459922Z","iopub.status.idle":"2023-02-12T05:03:13.514211Z","shell.execute_reply.started":"2023-02-12T05:03:13.459888Z","shell.execute_reply":"2023-02-12T05:03:13.51311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Drop of unwanted data \n\n**As all the data in these 2 column are missing a lot of values we will be droping them and they are not required for classification also**","metadata":{}},{"cell_type":"code","source":"del train_df['BIRADS']\ndel train_df['density'] \n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:15.201261Z","iopub.execute_input":"2023-02-12T05:03:15.201626Z","iopub.status.idle":"2023-02-12T05:03:15.20701Z","shell.execute_reply.started":"2023-02-12T05:03:15.201597Z","shell.execute_reply":"2023-02-12T05:03:15.206087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Corrleation found in data \n**As from the heat map we can say biopsy and invasive are the only fields that are highlt related to cancer as these are not provided for us during testing this is a data which only gives an insight for how much chance is that the cancer is valid but not for its estimation of cancer to be predicted towards future by image classification is our only option.**\n \n**If we implement a ML model it signifies attribute for all have found to be less but givin it a try for prediction without any images is a \"Highly-unrated\" approch hence this is not to be selected**","metadata":{}},{"cell_type":"code","source":"fig , ax = plt.subplots(figsize = (16,13))\ndataplot = sns.heatmap(train_df.corr(), cmap=\"YlGnBu\", annot=True)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-12T05:03:16.725576Z","iopub.execute_input":"2023-02-12T05:03:16.725919Z","iopub.status.idle":"2023-02-12T05:03:17.331341Z","shell.execute_reply.started":"2023-02-12T05:03:16.725891Z","shell.execute_reply":"2023-02-12T05:03:17.330671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Taking into account of data for image classification
\n\n**The main aim of this process it to add a new attribute into account of the path of each image and their respective data into a new data frame which then is to be assigned to image classification of output attribute to input**","metadata":{}},{"cell_type":"code","source":"design0_df_train = train_df.filter(['biopsy','laterality','invasive','cancer']) \ndesign0_df_test = test_df.filter(['biopsy','laterality','invasive','cancer']) ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:18.247163Z","iopub.execute_input":"2023-02-12T05:03:18.247741Z","iopub.status.idle":"2023-02-12T05:03:18.25658Z","shell.execute_reply.started":"2023-02-12T05:03:18.247708Z","shell.execute_reply":"2023-02-12T05:03:18.255258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"design1_df = train_df.filter(['laterality','patient_id','image_id','cancer'])\ndesign1_df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:19.199379Z","iopub.execute_input":"2023-02-12T05:03:19.199769Z","iopub.status.idle":"2023-02-12T05:03:19.214901Z","shell.execute_reply.started":"2023-02-12T05:03:19.199738Z","shell.execute_reply":"2023-02-12T05:03:19.213549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loading the address of the image into a array ","metadata":{}},{"cell_type":"code","source":"for i in range(54707):\n x = design1_df.iloc[:i, 0:-1].values\nlen(x)\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:03:20.60923Z","iopub.execute_input":"2023-02-12T05:03:20.609639Z","iopub.status.idle":"2023-02-12T05:06:26.327861Z","shell.execute_reply.started":"2023-02-12T05:03:20.609609Z","shell.execute_reply":"2023-02-12T05:06:26.326605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"direction =[]\ndict1 = []\nj=0\nbasepath = '/kaggle/input/rsna-breast-cancer-detection/train_images/'\nfor j in range(54706):\n value = x[j]\n patient_id = str(value[1])\n image_id = str(value[2])\n path = basepath+patient_id+'/'+image_id+'.dcm'\n pathflow = image_id+'.png'\n direction.append(path)\n dict1.append(pathflow)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:29.720391Z","iopub.execute_input":"2023-02-12T05:06:29.720817Z","iopub.status.idle":"2023-02-12T05:06:29.809643Z","shell.execute_reply.started":"2023-02-12T05:06:29.720782Z","shell.execute_reply":"2023-02-12T05:06:29.807837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(direction)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:30.384522Z","iopub.execute_input":"2023-02-12T05:06:30.384899Z","iopub.status.idle":"2023-02-12T05:06:30.392553Z","shell.execute_reply.started":"2023-02-12T05:06:30.384869Z","shell.execute_reply":"2023-02-12T05:06:30.391085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"design1_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:31.575235Z","iopub.execute_input":"2023-02-12T05:06:31.57562Z","iopub.status.idle":"2023-02-12T05:06:31.589922Z","shell.execute_reply.started":"2023-02-12T05:06:31.57559Z","shell.execute_reply":"2023-02-12T05:06:31.588477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generating DataFrame for just training in csv ","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame()\ndf['path']= direction #dcm\nDF = pd.DataFrame()\nDF['spath'] = dict1 #png","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:34.521281Z","iopub.execute_input":"2023-02-12T05:06:34.521642Z","iopub.status.idle":"2023-02-12T05:06:34.550522Z","shell.execute_reply.started":"2023-02-12T05:06:34.521613Z","shell.execute_reply":"2023-02-12T05:06:34.549364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"use = pd.DataFrame(design1_df)\nuse","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:35.294302Z","iopub.execute_input":"2023-02-12T05:06:35.294711Z","iopub.status.idle":"2023-02-12T05:06:35.309768Z","shell.execute_reply.started":"2023-02-12T05:06:35.294676Z","shell.execute_reply":"2023-02-12T05:06:35.307945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:36.923855Z","iopub.execute_input":"2023-02-12T05:06:36.924222Z","iopub.status.idle":"2023-02-12T05:06:36.935521Z","shell.execute_reply.started":"2023-02-12T05:06:36.924191Z","shell.execute_reply":"2023-02-12T05:06:36.934214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = pd.concat([use, df], axis=1)\nresult.to_csv('/kaggle/working/out.csv')","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:38.199096Z","iopub.execute_input":"2023-02-12T05:06:38.199464Z","iopub.status.idle":"2023-02-12T05:06:38.35588Z","shell.execute_reply.started":"2023-02-12T05:06:38.199438Z","shell.execute_reply":"2023-02-12T05:06:38.354778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = result.filter(['path','cancer'])\nresult = result.sort_values(by=['cancer'], ascending = True)\nresult","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:39.632978Z","iopub.execute_input":"2023-02-12T05:06:39.633415Z","iopub.status.idle":"2023-02-12T05:06:39.654041Z","shell.execute_reply.started":"2023-02-12T05:06:39.633385Z","shell.execute_reply":"2023-02-12T05:06:39.653155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Discovery of the uneven split of data
","metadata":{}},{"cell_type":"code","source":"cancer_count_data=result.loc[:,'cancer']\ncancer_count_data","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:43.337592Z","iopub.execute_input":"2023-02-12T05:06:43.338744Z","iopub.status.idle":"2023-02-12T05:06:43.346359Z","shell.execute_reply.started":"2023-02-12T05:06:43.338692Z","shell.execute_reply":"2023-02-12T05:06:43.34569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#verification\ng=0\nj=0\ny=0\nfor k in range(len(cancer_count_data)):\n valuek = cancer_count_data[k]\n if valuek ==0:\n g=g+1\n elif valuek ==1:\n j=j+1\n else:\n y=y+1\nprint(\"NO Cancer:\",g,\"\\nCancer:\",j,\"\\n NA:\",y)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:44.119749Z","iopub.execute_input":"2023-02-12T05:06:44.120253Z","iopub.status.idle":"2023-02-12T05:06:44.35921Z","shell.execute_reply.started":"2023-02-12T05:06:44.120221Z","shell.execute_reply":"2023-02-12T05:06:44.358248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data ratio is one of the most important part for training any model as seen above we understand the ratio between these 2 classes or binary data is found to have a huge difference \n\nNO cancer : 53548\nCancer : 1158\n### More data the better but an uneven split would lead for biasing hence we drop certain data from the row to bring down the 2:1 ratio ","metadata":{}},{"cell_type":"code","source":"DFnc=result[result['cancer'] == 1]\nDFc = result.head(3316)\nprint(\"NO cancer:\",len(DFnc))\nprint(\"Cancer :\",len(DFc))","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:45.356819Z","iopub.execute_input":"2023-02-12T05:06:45.357339Z","iopub.status.idle":"2023-02-12T05:06:45.363234Z","shell.execute_reply.started":"2023-02-12T05:06:45.357292Z","shell.execute_reply":"2023-02-12T05:06:45.362326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DF_final = pd.concat([DFnc, DFc], axis=0)\nlen(DF_final)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:45.908914Z","iopub.execute_input":"2023-02-12T05:06:45.909224Z","iopub.status.idle":"2023-02-12T05:06:45.916926Z","shell.execute_reply.started":"2023-02-12T05:06:45.909199Z","shell.execute_reply":"2023-02-12T05:06:45.916077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cancer_count_DF=DF_final.loc[:,'cancer']\ncancer_count_DF","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:46.44862Z","iopub.execute_input":"2023-02-12T05:06:46.448991Z","iopub.status.idle":"2023-02-12T05:06:46.455782Z","shell.execute_reply.started":"2023-02-12T05:06:46.448956Z","shell.execute_reply":"2023-02-12T05:06:46.455137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#verification\ng=0\nj=0\ny=0 \nfor L in range(len(DF_final)):\n valueL = cancer_count_DF.iloc[L]\n if valueL ==0:\n g=g+1\n elif valueL ==1:\n j=j+1\n else:\n y=y+1\nprint(\"NO Cancer:\",g,\"\\nCancer:\",j,\"\\n NA:\",y)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:47.051467Z","iopub.execute_input":"2023-02-12T05:06:47.05198Z","iopub.status.idle":"2023-02-12T05:06:47.08954Z","shell.execute_reply.started":"2023-02-12T05:06:47.05195Z","shell.execute_reply":"2023-02-12T05:06:47.088293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.utils import shuffle\nDF_final = shuffle(DF_final)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:47.605391Z","iopub.execute_input":"2023-02-12T05:06:47.605716Z","iopub.status.idle":"2023-02-12T05:06:47.641087Z","shell.execute_reply.started":"2023-02-12T05:06:47.60569Z","shell.execute_reply":"2023-02-12T05:06:47.640295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DF_final.to_csv('/kaggle/working/DF_final.csv')\nDF_final","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:48.206235Z","iopub.execute_input":"2023-02-12T05:06:48.206611Z","iopub.status.idle":"2023-02-12T05:06:48.23017Z","shell.execute_reply.started":"2023-02-12T05:06:48.206583Z","shell.execute_reply":"2023-02-12T05:06:48.229227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ","metadata":{}},{"cell_type":"code","source":"DFC = DF_final.loc[:,'path']\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:49.403264Z","iopub.execute_input":"2023-02-12T05:06:49.404511Z","iopub.status.idle":"2023-02-12T05:06:49.409101Z","shell.execute_reply.started":"2023-02-12T05:06:49.40446Z","shell.execute_reply":"2023-02-12T05:06:49.407974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i=0\ndictor = []\nimport os\nfor i in range(len(DF_final)):\n pathi = DFC.iloc[i]\n path = os.path.basename(os.path.normpath(pathi))\n separator = '.'\n image_num = path.split(separator, 1)[0]\n pathflow = image_num+'.png'\n dictor.append(pathflow)\nDF_png = pd.DataFrame()\nDF_png['path'] = dictor ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:50.008217Z","iopub.execute_input":"2023-02-12T05:06:50.008579Z","iopub.status.idle":"2023-02-12T05:06:50.071328Z","shell.execute_reply.started":"2023-02-12T05:06:50.008553Z","shell.execute_reply":"2023-02-12T05:06:50.070371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
EDA
","metadata":{}},{"cell_type":"code","source":"#train_df\ntrain_df = train_df.sort_values(by=['cancer'], ascending = True)\nTDFnc=train_df[train_df['cancer'] == 1]\nTDFc = train_df.head(3316)\ntrain_final = pd.concat([TDFnc, TDFc], axis=0)\ntrain_final\nprint(\"Overall Data:\",len(train_df))\nprint(\"NO cancer:\",len(TDFnc))\nprint(\"Cancer :\",len(TDFc))\nprint(\"Sampled Data:\",len(train_final))","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:50.757346Z","iopub.execute_input":"2023-02-12T05:06:50.757733Z","iopub.status.idle":"2023-02-12T05:06:50.777207Z","shell.execute_reply.started":"2023-02-12T05:06:50.757701Z","shell.execute_reply":"2023-02-12T05:06:50.77502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_df, x=train_df[\"implant\"])\nplt.title(\"Unsampled Data count of implant\") \n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:51.306749Z","iopub.execute_input":"2023-02-12T05:06:51.307655Z","iopub.status.idle":"2023-02-12T05:06:51.519396Z","shell.execute_reply.started":"2023-02-12T05:06:51.30762Z","shell.execute_reply":"2023-02-12T05:06:51.518628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_final, x=train_final[\"implant\"])\nplt.title(\"Sampled Data count of implant\") \n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:52.479521Z","iopub.execute_input":"2023-02-12T05:06:52.480701Z","iopub.status.idle":"2023-02-12T05:06:52.636191Z","shell.execute_reply.started":"2023-02-12T05:06:52.480648Z","shell.execute_reply":"2023-02-12T05:06:52.635386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nsns.set_theme(style=\"whitegrid\")\n\nsns.violinplot(data=train_final, x=\"age\", y=\"implant\",\n split=True, inner=\"quart\",\n linewidth=1, \n )\nsns.despine(left=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:53.067074Z","iopub.execute_input":"2023-02-12T05:06:53.068203Z","iopub.status.idle":"2023-02-12T05:06:54.229463Z","shell.execute_reply.started":"2023-02-12T05:06:53.068154Z","shell.execute_reply":"2023-02-12T05:06:54.226015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(20,10))\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_final, x=train_final[\"site_id\"],hue=\"cancer\")\nplt.title(\"Sampled Data count\") \n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:55.300945Z","iopub.execute_input":"2023-02-12T05:06:55.301354Z","iopub.status.idle":"2023-02-12T05:06:55.5236Z","shell.execute_reply.started":"2023-02-12T05:06:55.301303Z","shell.execute_reply":"2023-02-12T05:06:55.522375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4a. The Most Used Machine?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_df, x=train_df[\"machine_id\"])\n\nplt.title(\"unsampled DATA\") \n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:55.781107Z","iopub.execute_input":"2023-02-12T05:06:55.783197Z","iopub.status.idle":"2023-02-12T05:06:56.039991Z","shell.execute_reply.started":"2023-02-12T05:06:55.783121Z","shell.execute_reply":"2023-02-12T05:06:56.038016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\n\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_final, x=train_final[\"machine_id\"])\n\nplt.title(\"Sampled DATA\") \n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:56.04259Z","iopub.execute_input":"2023-02-12T05:06:56.043269Z","iopub.status.idle":"2023-02-12T05:06:56.308258Z","shell.execute_reply.started":"2023-02-12T05:06:56.043221Z","shell.execute_reply":"2023-02-12T05:06:56.306421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4b. The distribution of age?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\n\nsns.set_theme(style=\"dark\")\n\nsns.countplot(data = train_df, x=train_df[\"age\"])\n\nplt.title(\"Unsampled DATA\") \n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:56.334988Z","iopub.execute_input":"2023-02-12T05:06:56.335402Z","iopub.status.idle":"2023-02-12T05:06:57.568267Z","shell.execute_reply.started":"2023-02-12T05:06:56.335373Z","shell.execute_reply":"2023-02-12T05:06:57.566467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nsns.set_theme(style=\"whitegrid\")\n\nsns.violinplot(data=train_final, x=\"age\", y=\"implant\", hue=\"cancer\",\n split=True, inner=\"quart\",\n linewidth=1, \n )\nsns.despine(left=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:57.571028Z","iopub.execute_input":"2023-02-12T05:06:57.571498Z","iopub.status.idle":"2023-02-12T05:06:58.874591Z","shell.execute_reply.started":"2023-02-12T05:06:57.571458Z","shell.execute_reply":"2023-02-12T05:06:58.870141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_final","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:58.878666Z","iopub.execute_input":"2023-02-12T05:06:58.880188Z","iopub.status.idle":"2023-02-12T05:06:58.901869Z","shell.execute_reply.started":"2023-02-12T05:06:58.880113Z","shell.execute_reply":"2023-02-12T05:06:58.900163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Image Analysis
\n\n
\n The different spectral for the understanding of the appearance of detection to the cancer is a must hence we are to try these spectral ideas:
\n1. gray
\n2. jet
\n3. gist_stern
\n4. gist_ncar
\n5. nipy_spectral
\n6. twilight
\n
\n ","metadata":{}},{"cell_type":"code","source":"'''\nimport pydicom as dicom\nimport pydicom.data\nplt.figure(figsize=(20, 20))\nfor i in range(15):\n ax = plt.subplot(5, 3, i + 1)\n pathi= DF_final.iat[i,0]\n ds = pydicom.dcmread(pathi)\n plt.imshow(ds.pixel_array, cmap=plt.cm.gist_stern) \n datagiven = 'cancer: '+str(DF_final.iat[i,1])\n plt.title(datagiven)\n plt.axis(\"off\")\n \n'''\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:58.903773Z","iopub.execute_input":"2023-02-12T05:06:58.904648Z","iopub.status.idle":"2023-02-12T05:06:58.916653Z","shell.execute_reply.started":"2023-02-12T05:06:58.904608Z","shell.execute_reply":"2023-02-12T05:06:58.915145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gray_path=\"/kaggle/input/processed-image/147232633.png\"\ntwilight_path=\"/kaggle/input/processed-image/2425734.png\"\ngist_stern_path=\"/kaggle/input/processed-image/Screenshot 2023-02-06 111441.png\"","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:58.991611Z","iopub.execute_input":"2023-02-12T05:06:58.992691Z","iopub.status.idle":"2023-02-12T05:06:59.002981Z","shell.execute_reply.started":"2023-02-12T05:06:58.992598Z","shell.execute_reply":"2023-02-12T05:06:58.999596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.image as mpimg\nplt.figure(figsize=(20,10))\nimg = mpimg.imread(gist_stern_path)\nimgplot = plt.imshow(img)\nplt.axis(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:59.419745Z","iopub.execute_input":"2023-02-12T05:06:59.420243Z","iopub.status.idle":"2023-02-12T05:06:59.623028Z","shell.execute_reply.started":"2023-02-12T05:06:59.420199Z","shell.execute_reply":"2023-02-12T05:06:59.621644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.image as mpimg\nplt.figure(figsize=(20,10))\nimg = mpimg.imread(twilight_path)\nimgplot = plt.imshow(img)\nplt.axis(False)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:06:59.937105Z","iopub.execute_input":"2023-02-12T05:06:59.93778Z","iopub.status.idle":"2023-02-12T05:07:00.303704Z","shell.execute_reply.started":"2023-02-12T05:06:59.937722Z","shell.execute_reply":"2023-02-12T05:07:00.299716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.image as mpimg\nplt.figure(figsize=(20,10))\nimg = mpimg.imread(gray_path)\nimgplot = plt.imshow(img)\nplt.axis(False)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:00.409938Z","iopub.execute_input":"2023-02-12T05:07:00.410635Z","iopub.status.idle":"2023-02-12T05:07:00.729283Z","shell.execute_reply.started":"2023-02-12T05:07:00.410563Z","shell.execute_reply":"2023-02-12T05:07:00.727669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
To store DATA for train and testing in png as seperate folder
","metadata":{}},{"cell_type":"code","source":"'''\nos.mkdir('/kaggle/working/Normal')\nos.mkdir('/kaggle/working/Abnormal')'''\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:01.659274Z","iopub.execute_input":"2023-02-12T05:07:01.659772Z","iopub.status.idle":"2023-02-12T05:07:01.667138Z","shell.execute_reply.started":"2023-02-12T05:07:01.659733Z","shell.execute_reply":"2023-02-12T05:07:01.664746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#os.remove(\"/kaggle/working/file.zip\")","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:02.088386Z","iopub.execute_input":"2023-02-12T05:07:02.088844Z","iopub.status.idle":"2023-02-12T05:07:02.093813Z","shell.execute_reply.started":"2023-02-12T05:07:02.08881Z","shell.execute_reply":"2023-02-12T05:07:02.092705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#import shutil\n#shutil.rmtree(\"/kaggle/working/Normal\")\n#shutil.rmtree(\"/kaggle/working/Abnormal\")\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:02.621871Z","iopub.execute_input":"2023-02-12T05:07:02.622391Z","iopub.status.idle":"2023-02-12T05:07:02.629832Z","shell.execute_reply.started":"2023-02-12T05:07:02.622345Z","shell.execute_reply":"2023-02-12T05:07:02.627539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#can_list=DF_final.loc[:,'cancer']","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:02.960903Z","iopub.execute_input":"2023-02-12T05:07:02.961403Z","iopub.status.idle":"2023-02-12T05:07:02.965234Z","shell.execute_reply.started":"2023-02-12T05:07:02.961365Z","shell.execute_reply":"2023-02-12T05:07:02.96454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''else:\n strpath = '/kaggle/working/Normal/'+addpath\n store = 'normal'\n\n''' ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:03.305074Z","iopub.execute_input":"2023-02-12T05:07:03.305857Z","iopub.status.idle":"2023-02-12T05:07:03.316003Z","shell.execute_reply.started":"2023-02-12T05:07:03.305802Z","shell.execute_reply":"2023-02-12T05:07:03.31453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nimport cv2\nimport pydicom as dicom\nimport pydicom.data\n\nfor i in range(len(DF_final)):\n can = can_list.iloc[i]\n pathi= DF_final.iat[i,0]\n addpath = DF_png.iat[i,0]\n if can == 0:\n strpath = '/kaggle/working/Normal/'+addpath\n store = 'normal'\n \n imagedata= pydicom.dcmread(pathi)\n img =imagedata.pixel_array\n cmap=plt.cm.twilight \n plt.figure(figsize=(15,15))\n name = pathi.split('/')[-1][:-4]\n plt.imshow(img,cmap)\n plt.axis(\"off\")\n plt.savefig(strpath) \nprint(\"Done\") \n'''\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:04.756085Z","iopub.execute_input":"2023-02-12T05:07:04.757055Z","iopub.status.idle":"2023-02-12T05:07:04.76484Z","shell.execute_reply.started":"2023-02-12T05:07:04.757018Z","shell.execute_reply":"2023-02-12T05:07:04.763101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!zip -r file.zip /kaggle/working","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:05.871832Z","iopub.execute_input":"2023-02-12T05:07:05.872299Z","iopub.status.idle":"2023-02-12T05:07:05.877445Z","shell.execute_reply.started":"2023-02-12T05:07:05.87227Z","shell.execute_reply":"2023-02-12T05:07:05.876391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Getting the model for detection
","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import load_model\nmodel=load_model('/kaggle/input/model-v2-for-bcd/val2.h5')","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:07.01506Z","iopub.execute_input":"2023-02-12T05:07:07.016433Z","iopub.status.idle":"2023-02-12T05:07:08.233762Z","shell.execute_reply.started":"2023-02-12T05:07:07.016374Z","shell.execute_reply":"2023-02-12T05:07:08.232004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:08.23573Z","iopub.execute_input":"2023-02-12T05:07:08.236119Z","iopub.status.idle":"2023-02-12T05:07:08.250877Z","shell.execute_reply.started":"2023-02-12T05:07:08.236084Z","shell.execute_reply":"2023-02-12T05:07:08.249474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:08.25247Z","iopub.execute_input":"2023-02-12T05:07:08.252796Z","iopub.status.idle":"2023-02-12T05:07:08.262369Z","shell.execute_reply.started":"2023-02-12T05:07:08.252767Z","shell.execute_reply":"2023-02-12T05:07:08.261365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:09.143431Z","iopub.execute_input":"2023-02-12T05:07:09.143841Z","iopub.status.idle":"2023-02-12T05:07:09.1589Z","shell.execute_reply.started":"2023-02-12T05:07:09.143808Z","shell.execute_reply":"2023-02-12T05:07:09.157645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.DataFrame()\nsub = test_df\nsub","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:09.864249Z","iopub.execute_input":"2023-02-12T05:07:09.864629Z","iopub.status.idle":"2023-02-12T05:07:09.877763Z","shell.execute_reply.started":"2023-02-12T05:07:09.864597Z","shell.execute_reply":"2023-02-12T05:07:09.876462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = sub.drop(['image_id'],axis=1)\nsub = sub.iloc[:,1:3]\nsub","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:10.640379Z","iopub.execute_input":"2023-02-12T05:07:10.641246Z","iopub.status.idle":"2023-02-12T05:07:10.654072Z","shell.execute_reply.started":"2023-02-12T05:07:10.641214Z","shell.execute_reply":"2023-02-12T05:07:10.652775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfor n in range(0,4):\n p = sub.iloc[:n, :2].values \np ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:11.496479Z","iopub.execute_input":"2023-02-12T05:07:11.496853Z","iopub.status.idle":"2023-02-12T05:07:11.50732Z","shell.execute_reply.started":"2023-02-12T05:07:11.496822Z","shell.execute_reply":"2023-02-12T05:07:11.505858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"formsub = []\nfor g in range(0,3):\n valuee = p[g]\n patient_ide = str(valuee[0])\n side = str(valuee[1])\n pathe = patient_ide+'_'+side\n formsub.append(pathe)\nformsub ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:13.077916Z","iopub.execute_input":"2023-02-12T05:07:13.078293Z","iopub.status.idle":"2023-02-12T05:07:13.08748Z","shell.execute_reply.started":"2023-02-12T05:07:13.078261Z","shell.execute_reply":"2023-02-12T05:07:13.086053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test_df","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:13.586506Z","iopub.execute_input":"2023-02-12T05:07:13.586862Z","iopub.status.idle":"2023-02-12T05:07:13.590682Z","shell.execute_reply.started":"2023-02-12T05:07:13.586837Z","shell.execute_reply":"2023-02-12T05:07:13.590105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = test_df.drop(['site_id'],axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:14.333567Z","iopub.execute_input":"2023-02-12T05:07:14.333927Z","iopub.status.idle":"2023-02-12T05:07:14.339372Z","shell.execute_reply.started":"2023-02-12T05:07:14.3339Z","shell.execute_reply":"2023-02-12T05:07:14.338575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(test_df)):\n y = test_df.iloc[:i, :2].values\n\nprint(y)","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:14.974042Z","iopub.execute_input":"2023-02-12T05:07:14.974733Z","iopub.status.idle":"2023-02-12T05:07:14.980288Z","shell.execute_reply.started":"2023-02-12T05:07:14.974706Z","shell.execute_reply":"2023-02-12T05:07:14.979688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:15.478474Z","iopub.execute_input":"2023-02-12T05:07:15.478814Z","iopub.status.idle":"2023-02-12T05:07:15.485557Z","shell.execute_reply.started":"2023-02-12T05:07:15.478788Z","shell.execute_reply":"2023-02-12T05:07:15.484599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"directiony =[]\ndict1y=[]\nbasepathy = '/kaggle/input/testout/testout/'\nfor o in range(0,3):\n valuey = y[o]\n patient_idy = str(valuey[0])\n image_idy = str(valuey[1])\n pathy = basepathy+patient_idy+'/'+image_idy+'.dcm'\n pathflowy = image_idy+'.png'\n directiony.append(pathy)\n dict1y.append(pathflowy)\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:16.38233Z","iopub.execute_input":"2023-02-12T05:07:16.382708Z","iopub.status.idle":"2023-02-12T05:07:16.389191Z","shell.execute_reply.started":"2023-02-12T05:07:16.382675Z","shell.execute_reply":"2023-02-12T05:07:16.388103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dict1y","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:17.213923Z","iopub.execute_input":"2023-02-12T05:07:17.214292Z","iopub.status.idle":"2023-02-12T05:07:17.220374Z","shell.execute_reply.started":"2023-02-12T05:07:17.214259Z","shell.execute_reply":"2023-02-12T05:07:17.219677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#os.mkdir('/kaggle/working/testout')\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:17.811425Z","iopub.execute_input":"2023-02-12T05:07:17.812732Z","iopub.status.idle":"2023-02-12T05:07:17.817212Z","shell.execute_reply.started":"2023-02-12T05:07:17.812678Z","shell.execute_reply":"2023-02-12T05:07:17.815996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#dict1y","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:19.153235Z","iopub.execute_input":"2023-02-12T05:07:19.153654Z","iopub.status.idle":"2023-02-12T05:07:19.157691Z","shell.execute_reply.started":"2023-02-12T05:07:19.153623Z","shell.execute_reply":"2023-02-12T05:07:19.156879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''import cv2\nimport pydicom as dicom\nimport pydicom.data\nfor u in range(0,3):\n addpath = dict1y[u]\n strpath = '/kaggle/working/testout/'+addpath\n imagedata= pydicom.dcmread(directiony[u])\n img =imagedata.pixel_array\n cmap=plt.cm.twilight \n plt.figure(figsize=(15,15))\n name = pathi.split('/')[-1][:-4]\n plt.imshow(img,cmap)\n plt.axis(\"off\")\n plt.savefig(strpath) \nprint(\"Done\") '''","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:19.861502Z","iopub.execute_input":"2023-02-12T05:07:19.861873Z","iopub.status.idle":"2023-02-12T05:07:19.868969Z","shell.execute_reply.started":"2023-02-12T05:07:19.861841Z","shell.execute_reply":"2023-02-12T05:07:19.867748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ypath=[]\nfor u in range(0,3):\n addpath = dict1y[u]\n strin = '/kaggle/input/testout/testout/'+addpath\n ypath.append(strin)\nypath ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:20.321724Z","iopub.execute_input":"2023-02-12T05:07:20.322077Z","iopub.status.idle":"2023-02-12T05:07:20.329503Z","shell.execute_reply.started":"2023-02-12T05:07:20.322046Z","shell.execute_reply":"2023-02-12T05:07:20.328369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_names =['Implant', 'Abnormal', 'Normal']","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:21.004881Z","iopub.execute_input":"2023-02-12T05:07:21.005258Z","iopub.status.idle":"2023-02-12T05:07:21.010539Z","shell.execute_reply.started":"2023-02-12T05:07:21.005226Z","shell.execute_reply":"2023-02-12T05:07:21.009112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\ndef resize_images(src_folder, dst_folder, size):\n for filename in os.listdir(src_folder):\n with Image.open(src_folder + filename) as im:\n im.thumbnail(size)\n im.save(dst_folder + filename)\n\nresize_images('/kaggle/working/testout/', '/kaggle/working/testout/', (324, 324))\n'''\n","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:21.770844Z","iopub.execute_input":"2023-02-12T05:07:21.771405Z","iopub.status.idle":"2023-02-12T05:07:21.781141Z","shell.execute_reply.started":"2023-02-12T05:07:21.771346Z","shell.execute_reply":"2023-02-12T05:07:21.77964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pcan=[]\nfor f in range(len(ypath)): \n img = tf.keras.utils.load_img(\n ypath[f], target_size=(324, 324)\n )\n img_array = tf.keras.utils.img_to_array(img)\n img_array = tf.expand_dims(img_array, 0) # Create a batch\n\n predictions = model.predict(img_array)\n score = tf.nn.softmax(predictions[0])\n ID = np.argmax(score)\n if ID == 2:\n pc = 1 - np.max(score)\n \n else:\n pc = np.max(score)\n pcan.append(pc)\n\npcan","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:22.277021Z","iopub.execute_input":"2023-02-12T05:07:22.277652Z","iopub.status.idle":"2023-02-12T05:07:22.858471Z","shell.execute_reply.started":"2023-02-12T05:07:22.277599Z","shell.execute_reply":"2023-02-12T05:07:22.857062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'prediction_id':formsub, 'cancer':pcan})\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:23.16187Z","iopub.execute_input":"2023-02-12T05:07:23.162302Z","iopub.status.idle":"2023-02-12T05:07:23.176215Z","shell.execute_reply.started":"2023-02-12T05:07:23.162266Z","shell.execute_reply":"2023-02-12T05:07:23.174474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('/kaggle/working/submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-02-12T05:07:35.233012Z","iopub.execute_input":"2023-02-12T05:07:35.233469Z","iopub.status.idle":"2023-02-12T05:07:35.241297Z","shell.execute_reply.started":"2023-02-12T05:07:35.233435Z","shell.execute_reply":"2023-02-12T05:07:35.240172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}