Importing Libraries
","metadata":{}},{"cell_type":"code","source":"! pip install pylibjpeg ","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:29.725598Z","iopub.execute_input":"2022-12-12T21:10:29.726069Z","iopub.status.idle":"2022-12-12T21:10:44.483826Z","shell.execute_reply.started":"2022-12-12T21:10:29.725977Z","shell.execute_reply":"2022-12-12T21:10:44.481806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport warnings\nimport pylibjpeg \nimport pandas as pd\nimport seaborn as sns\nfrom glob import glob\nimport plotly.io as pio\nimport pydicom as dicom\nfrom pydicom import dcmread\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom matplotlib import pyplot as plt\nfrom mpl_toolkits.axes_grid1 import ImageGrid\n\npio.renderers.default = 'iframe'\nwarnings.simplefilter(action='ignore', category=FutureWarning)","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:44.487006Z","iopub.execute_input":"2022-12-12T21:10:44.487442Z","iopub.status.idle":"2022-12-12T21:10:46.766133Z","shell.execute_reply.started":"2022-12-12T21:10:44.487403Z","shell.execute_reply":"2022-12-12T21:10:46.764502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Loading Dataset
","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/train.csv')\ntest = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\nprint(f\"train.shape = {train.shape} \\ntest.shape = {test.shape}\")\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:46.768945Z","iopub.execute_input":"2022-12-12T21:10:46.770294Z","iopub.status.idle":"2022-12-12T21:10:46.960578Z","shell.execute_reply.started":"2022-12-12T21:10:46.770217Z","shell.execute_reply":"2022-12-12T21:10:46.959043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Data Description
\n* site_id - ID code for the source hospital\n* patient_id - ID code for the patient.\n* image_id - ID code for the image.\n* laterality - Whether the image is of the left or right breast.\n* view - The orientation of the image. The default for a screening exam is to capture two views per breast.\n* age - The patient's age in years.\n* implant - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.\n* density - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult.\n* machine_id - An ID code for the imaging device.\n* cancer - The target value. Only provided for train.\n* biopsy - Whether or not a follow-up biopsy was performed on the breast.\ninvasive - If the breast is positive for cancer, whether or not the cancer proved to be invasive.\n* BI-RADS - One can essentially think of 0 as \"abnormal\" and 1 and 2 as \"normal.\" There can be subjectivity in assigning BI-RADS 1 or 2. For example, if there are stable findings that are almost certainly benign breast cysts, the mammogram may be assigned 1 or 2 depending on the radiologist.\n** 0 - Need additional imaging evaluation\n** 1 - Negative\n** 2 - Benign\n* difficult_negative_case - True if the case was unusually difficult. Only provided for train.","metadata":{}},{"cell_type":"markdown","source":"Test features\n - age\n - image_id\n - implant\n - laterality\n - machine_id\n - patient_id\n - prediction_id\n - site_id\n - view","metadata":{}},{"cell_type":"code","source":"train.groupby([\"BIRADS\", \"biopsy\"]).cancer.value_counts().to_frame()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:46.964088Z","iopub.execute_input":"2022-12-12T21:10:46.964564Z","iopub.status.idle":"2022-12-12T21:10:46.999867Z","shell.execute_reply.started":"2022-12-12T21:10:46.964512Z","shell.execute_reply":"2022-12-12T21:10:46.998336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby([\"BIRADS\", \"biopsy\", \"implant\", \"laterality\", \"view\"]).cancer.value_counts().to_frame()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:47.00173Z","iopub.execute_input":"2022-12-12T21:10:47.002124Z","iopub.status.idle":"2022-12-12T21:10:47.047038Z","shell.execute_reply.started":"2022-12-12T21:10:47.002091Z","shell.execute_reply":"2022-12-12T21:10:47.045529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.patient_id.nunique(), train.machine_id.nunique(), train.groupby(\"patient_id\")[\"image_id\"].agg(\"count\").reset_index()[\"image_id\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:47.048514Z","iopub.execute_input":"2022-12-12T21:10:47.048876Z","iopub.status.idle":"2022-12-12T21:10:47.066517Z","shell.execute_reply.started":"2022-12-12T21:10:47.048846Z","shell.execute_reply":"2022-12-12T21:10:47.064904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Number of unique patients - 11913.\n* Number of unique machine_id - 10.\n* In the average, each patient has 4-5 images.","metadata":{}},{"cell_type":"markdown","source":"\n#DCM File Structure
","metadata":{}},{"cell_type":"code","source":"image_gen = glob('/kaggle/input/rsna-breast-cancer-detection/train_images/10**/*.dcm', recursive = True)","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:47.068614Z","iopub.execute_input":"2022-12-12T21:10:47.06905Z","iopub.status.idle":"2022-12-12T21:10:50.841618Z","shell.execute_reply.started":"2022-12-12T21:10:47.06901Z","shell.execute_reply":"2022-12-12T21:10:50.839946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = dcmread(image_gen[0])\n\nprint(f\"Patient ID => {sample.PatientID}\")\nprint(f\"Image size => {sample.Rows} x {sample.Columns}\")\nprint(f\"Instance No => {sample.InstanceNumber }\")","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:50.844806Z","iopub.execute_input":"2022-12-12T21:10:50.845373Z","iopub.status.idle":"2022-12-12T21:10:50.910818Z","shell.execute_reply.started":"2022-12-12T21:10:50.845319Z","shell.execute_reply":"2022-12-12T21:10:50.909365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dcmread(image_gen[0]).file_meta.TransferSyntaxUID","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:50.912598Z","iopub.execute_input":"2022-12-12T21:10:50.913832Z","iopub.status.idle":"2022-12-12T21:10:50.927164Z","shell.execute_reply.started":"2022-12-12T21:10:50.913784Z","shell.execute_reply":"2022-12-12T21:10:50.925117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"uid = [dcmread(path).file_meta.TransferSyntaxUID for path in image_gen]\nlen(uid)","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:10:50.931902Z","iopub.execute_input":"2022-12-12T21:10:50.932817Z","iopub.status.idle":"2022-12-12T21:11:44.486076Z","shell.execute_reply.started":"2022-12-12T21:10:50.932775Z","shell.execute_reply":"2022-12-12T21:11:44.484657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"uid_df = pd.DataFrame({\"uid\": uid})\nuid_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:11:44.488113Z","iopub.execute_input":"2022-12-12T21:11:44.489667Z","iopub.status.idle":"2022-12-12T21:11:44.501973Z","shell.execute_reply.started":"2022-12-12T21:11:44.489602Z","shell.execute_reply":"2022-12-12T21:11:44.500307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"uid_df.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:11:44.503999Z","iopub.execute_input":"2022-12-12T21:11:44.504515Z","iopub.status.idle":"2022-12-12T21:11:44.529071Z","shell.execute_reply.started":"2022-12-12T21:11:44.504477Z","shell.execute_reply":"2022-12-12T21:11:44.527633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#DCM to PNG
","metadata":{}},{"cell_type":"code","source":"def process(path_to_image, size=512):\n dicom = dcmread(path_to_image)\n img = dicom.pixel_array\n\n img = (img - img.min()) / (img.max() - img.min())\n img = cv2.resize(img, (size, size))\n \n return img\n\nimgs = [process(path_to_image) for path_to_image in image_gen[200: 212]]","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:11:44.531219Z","iopub.execute_input":"2022-12-12T21:11:44.531793Z","iopub.status.idle":"2022-12-12T21:12:02.053099Z","shell.execute_reply.started":"2022-12-12T21:11:44.531739Z","shell.execute_reply":"2022-12-12T21:12:02.051798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Example image samples
","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(20., 15.))\ngrid = ImageGrid(fig, 111, # similar to subplot(111)\n nrows_ncols=(2, 6), # creates 2x2 grid of axes\n axes_pad=0.25, # pad between axes in inch.\n)\n\nfor ax, im in zip(grid, imgs):\n # Iterating over the grid returns the Axes.\n ax.imshow(im)\n ax.set_xticks([])\n ax.set_yticks([])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:02.055207Z","iopub.execute_input":"2022-12-12T21:12:02.055635Z","iopub.status.idle":"2022-12-12T21:12:03.39772Z","shell.execute_reply.started":"2022-12-12T21:12:02.055598Z","shell.execute_reply":"2022-12-12T21:12:03.396638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Data distributions
","metadata":{}},{"cell_type":"code","source":"count_images_df = train.groupby(\"patient_id\")\\\n .agg({\"image_id\": \"count\"})\\\n .reset_index()[\"image_id\"]\\\n .value_counts()\\\n .rename_axis('number_of_image_per_patient')\\\n .reset_index(name='counts')\n\ncount_images_df[\"percent\"] = (100 * count_images_df.counts / count_images_df.counts.sum()).round(2).astype(str) + \" %\"\ncount_images_df.T","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:03.399027Z","iopub.execute_input":"2022-12-12T21:12:03.399562Z","iopub.status.idle":"2022-12-12T21:12:03.430222Z","shell.execute_reply.started":"2022-12-12T21:12:03.399513Z","shell.execute_reply":"2022-12-12T21:12:03.429042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"go.Figure(go.Bar(\n x=count_images_df['number_of_image_per_patient'],\n y=count_images_df['counts'],\n text=count_images_df['percent']),\n layout=go.Layout(\n title=go.layout.Title(text=\"Number of images per patient\")\n )\n)","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:03.431957Z","iopub.execute_input":"2022-12-12T21:12:03.432466Z","iopub.status.idle":"2022-12-12T21:12:03.521474Z","shell.execute_reply.started":"2022-12-12T21:12:03.432434Z","shell.execute_reply":"2022-12-12T21:12:03.520215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Most cases have 4 images per patient, but there are a few cases with more than 10 images*","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,10))\n\nsns.countplot(train.laterality, ax=ax[0,0], palette='Greens_r')\nsns.countplot(train.view, ax=ax[0,1], palette='Reds_r')\nsns.countplot(train.implant, ax=ax[1,0], palette='Blues_r')\nsns.countplot(train.density, ax=ax[1,1], palette='Purples_r');","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:03.523113Z","iopub.execute_input":"2022-12-12T21:12:03.523718Z","iopub.status.idle":"2022-12-12T21:12:04.238992Z","shell.execute_reply.started":"2022-12-12T21:12:03.523671Z","shell.execute_reply":"2022-12-12T21:12:04.237381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Laterality is almost balanced\n* The most common views are CC and MLO and given these 2 views for the right and left breast we end up with 4 images that most of the patients show.\n* There are almost no patients with implants","metadata":{}},{"cell_type":"code","source":"df_machine_id = train.groupby(by=[\"patient_id\",'cancer','machine_id']).size().reset_index(name=\"counts\")\ndf_machine_id[\"machine_id\"] = df_machine_id.machine_id.apply(lambda x: f\"Machine ~ {x}\")\n\ndf_machine_id.sample(3)","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:04.24049Z","iopub.execute_input":"2022-12-12T21:12:04.24087Z","iopub.status.idle":"2022-12-12T21:12:04.275229Z","shell.execute_reply.started":"2022-12-12T21:12:04.240832Z","shell.execute_reply":"2022-12-12T21:12:04.273989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"machine_distribution = train.groupby(['machine_id','cancer'])['cancer'].sum().reset_index(name='cancer_count')\nfig = px.scatter(machine_distribution, x=\"machine_id\", y=\"cancer_count\", size=\"cancer_count\", color=\"machine_id\",\n hover_name=\"machine_id\",labels ='machine_id', log_x=True, size_max=60,width=1000)\nfig.update_coloraxes(showscale=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:04.27697Z","iopub.execute_input":"2022-12-12T21:12:04.277601Z","iopub.status.idle":"2022-12-12T21:12:05.371422Z","shell.execute_reply.started":"2022-12-12T21:12:04.277528Z","shell.execute_reply":"2022-12-12T21:12:05.37001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.pie(values=df_machine_id.counts,\n names=df_machine_id.machine_id, \n facet_col=df_machine_id.cancer,\n color_discrete_sequence=px.colors.sequential.Burg_r, \n title='The distribution of machines that used to take image'\n)\nfig.for_each_annotation(lambda x: x.update(text=\"diagnosed with cancer = \"+ x.text.split(\"=\")[1]))\nfig.update_traces(textposition='inside', textinfo='percent + label')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:05.37313Z","iopub.execute_input":"2022-12-12T21:12:05.37358Z","iopub.status.idle":"2022-12-12T21:12:05.527527Z","shell.execute_reply.started":"2022-12-12T21:12:05.373514Z","shell.execute_reply":"2022-12-12T21:12:05.526144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_pie(train: pd.DataFrame, column: str) -> None:\n labels = train[column].value_counts().keys().tolist()\n counts = train[column].value_counts().values.tolist()\n explode = (0.1, 0)\n plt.pie(counts, explode=explode, labels=labels, autopct='%1.1f%%',\n shadow=True, startangle=60) \n plt.legend()\n plt.title(column)\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:05.529393Z","iopub.execute_input":"2022-12-12T21:12:05.529945Z","iopub.status.idle":"2022-12-12T21:12:05.538888Z","shell.execute_reply.started":"2022-12-12T21:12:05.529898Z","shell.execute_reply":"2022-12-12T21:12:05.537651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column in [\"cancer\", \"biopsy\", \"laterality\", \"implant\"]:\n plot_pie(train, column)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:05.54044Z","iopub.execute_input":"2022-12-12T21:12:05.541444Z","iopub.status.idle":"2022-12-12T21:12:06.775907Z","shell.execute_reply.started":"2022-12-12T21:12:05.541408Z","shell.execute_reply":"2022-12-12T21:12:06.774256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_patints = train.drop_duplicates(['patient_id'], keep='first')\n\ncancer_0 = unique_patints.query(\"cancer == 0\")['age'].value_counts().rename_axis('age').reset_index(name='counts')\ncancer_1 = unique_patints.query(\"cancer == 1\")['age'].value_counts().rename_axis('age').reset_index(name='counts')\n\ncancer_0['percent'] = ((cancer_0['counts'] / cancer_0['counts'].sum()) * 100).round(2).astype(str) + '%'\ncancer_1['percent'] = ((cancer_1['counts'] / cancer_1['counts'].sum()) * 100).round(2).astype(str) + '%'","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:06.778054Z","iopub.execute_input":"2022-12-12T21:12:06.77894Z","iopub.status.idle":"2022-12-12T21:12:06.81483Z","shell.execute_reply.started":"2022-12-12T21:12:06.778873Z","shell.execute_reply":"2022-12-12T21:12:06.813502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure(data=[\n go.Bar(x=cancer_0['counts'], y=cancer_0['age'], text=cancer_0['percent'], orientation='h', name=\"healthy\"),\n go.Bar(x=cancer_1['counts'], y=cancer_1['age'], text=cancer_1['percent'], orientation='h', name=\"sick\")\n])\n\nfig.update_layout(showlegend=True,\n barmode='stack',\n plot_bgcolor='white', \n margin=dict(pad=20),\n width=800,\n height=1200,\n xaxis={'showticklabels': False},\n yaxis_range = [24, 90],\n yaxis_title=None,\n xaxis_title=None,\n yaxis={'categoryorder':'total ascending'},\n title_text=\"Age distribution\",\n title_x=0.5,\n font=dict(family=\"serif\", size=17, color='#000000'),\n title_font_size=30)\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:06.816799Z","iopub.execute_input":"2022-12-12T21:12:06.817341Z","iopub.status.idle":"2022-12-12T21:12:06.880553Z","shell.execute_reply.started":"2022-12-12T21:12:06.81729Z","shell.execute_reply":"2022-12-12T21:12:06.878938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.box(train.groupby('patient_id')['age','cancer'].max().reset_index(), \n y=\"age\",\n template=\"simple_white\",\n x=\"cancer\",\n title= \"Distribution of patients age based on whether have cancer or not\",\n labels={'x':'Cancer or not (1 = Cancer, 0 = Noncancer)', 'y':\"Distribution of patients age\"},\n color_discrete_sequence=px.colors.sequential.Burg_r)\n\nfig.update_layout(coloraxis_showscale=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:06.882485Z","iopub.execute_input":"2022-12-12T21:12:06.882908Z","iopub.status.idle":"2022-12-12T21:12:07.067661Z","shell.execute_reply.started":"2022-12-12T21:12:06.882872Z","shell.execute_reply":"2022-12-12T21:12:07.066121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(x=train.query(\"difficult_negative_case == True\")[\"age\"], shade=True, color=\"#0066ff\", legend=True)\nsns.kdeplot(x=train.query(\"difficult_negative_case == False\")[\"age\"], shade=True, color=\"#cc0000\", legend=True)\n\nplt.legend(labels = [\"difficult_negative_case: True\", \"difficult_negative_case: False\"])\nplt.title(\"Age of women -- difficult_negative_case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:07.070002Z","iopub.execute_input":"2022-12-12T21:12:07.070484Z","iopub.status.idle":"2022-12-12T21:12:07.625476Z","shell.execute_reply.started":"2022-12-12T21:12:07.070376Z","shell.execute_reply":"2022-12-12T21:12:07.623905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in ['cancer', 'biopsy', 'invasive']:\n sns.kdeplot(x=train.query(f\"{i} == 1\")[\"age\"], shade=True, color=\"#cc0000\", legend=True)\n sns.kdeplot(x=train.query(f\"{i} == 0\")[\"age\"], shade=True, color=\"#0066ff\", legend=True)\n \n plt.legend(labels=[f\"{i}: True\", f\"{i}: False\"])\n plt.title(f\"Age of women -- {i}\")\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-12T21:12:07.627217Z","iopub.execute_input":"2022-12-12T21:12:07.627617Z","iopub.status.idle":"2022-12-12T21:12:09.090081Z","shell.execute_reply.started":"2022-12-12T21:12:07.627576Z","shell.execute_reply":"2022-12-12T21:12:09.088759Z"},"trusted":true},"execution_count":null,"outputs":[]}]}