{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA Screening Mammography Breast Cancer Detection","metadata":{}},{"cell_type":"markdown","source":"### Table of contents:\n[0. Paths to explore](#Paths-to-explore)  \n[1. Introduction](#Introduction)  \n[2. Import modules](#Import-modules)  \n[3. Checking the GPU](#Introduction)  \n[4. Data overview](#Data-overview)  \n[5. Exploratory Data Analysis](#Exploratory-Data-Analysis)  \n[6.DICOM image overview](#DICOM-image-overview)  \n[7.Data Preparation](#Data-preparation)  \n[8.Metric](#Metric)  \n[9.Pytorch Model from Scratch](#Pytorch-model-from-scratch)  ","metadata":{}},{"cell_type":"markdown","source":"## **Paths to explore:**","metadata":{}},{"cell_type":"markdown","source":"## **Introduction:**\n**Context:** Breast cancer is the most commonly occurring cancer worldwide. Early detection and treatment are critical to reducing cancer fatalities and machine learning can be useful in the process of evaluating screening mamograms.  \n**Goal:** Identifying cases of breast cancer out of mamograms from screening exams.  \n**Dataset:** The dataset for this challenge contains radiographic breast images of female subjects.  \n**File Format:** DICOM Digital imaging and communications in medicine  \n**Evaluation:** Submissions are evaluated using the probabilistic F1 score (pF1). This extension of the traditional F score accepts probabilities instead of binary classifications.  \nWith pX as the probabilistic version of X:  \n$$pF1 = 2\\frac{pPrecision.pRecall}{pPrecision + pRecall}$$\nwhere:\n$$pPrecision = \\frac{pTP}{pTP + pFP}$$\n$$pRecall = \\frac{pTP}{TP + FN}$$\n\n    \n\n  ","metadata":{}},{"cell_type":"markdown","source":"## **Import modules:**","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pydicom\nimport glob\nimport cv2\nimport gc\nimport datetime as dtime\nfrom tqdm import tqdm\nfrom joblib import Parallel, delayed\nfrom sklearn.preprocessing import LabelEncoder\nfrom pathlib import Path\nfrom time import time\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import FloatTensor, LongTensor\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.io import read_image\nfrom torchvision.models import resnet50\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold\nfrom sklearn.metrics import accuracy_score, roc_auc_score, confusion_matrix","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:42.337162Z","iopub.execute_input":"2023-01-12T18:08:42.337835Z","iopub.status.idle":"2023-01-12T18:08:45.618669Z","shell.execute_reply.started":"2023-01-12T18:08:42.337751Z","shell.execute_reply":"2023-01-12T18:08:45.617665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Data overview:**\n[train/test].csv Metadata for each patient and image. Only the first few rows of the test set are available for download.  \n**site_id** - ID code for the source hospital.  \n**patient_id** - ID code for the patient.  \n**image_id** - ID code for the image.  \n**laterality** - Whether the image is of the left or right breast.  \n**view** - The orientation of the image. The default for a screening exam is to capture two views per breast.  \n**age** - The patient's age in years.  \n**implant** - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.  \n**density** - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult. Only provided for train.  \n**machine_id** - An ID code for the imaging device.  \n**cancer** - Whether or not the breast was positive for malignant cancer. The target value. Only provided for train.  \n**biopsy** - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.  \n**invasive** - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.  \n**BIRADS** - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.  \n**prediction_id** - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.  \n**difficult_negative_case** - True if the case was unusually difficult. Only provided for train.  \n","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/train.csv')\ntest_data = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:45.624729Z","iopub.execute_input":"2023-01-12T18:08:45.627259Z","iopub.status.idle":"2023-01-12T18:08:45.773535Z","shell.execute_reply.started":"2023-01-12T18:08:45.627209Z","shell.execute_reply":"2023-01-12T18:08:45.772466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Shape of training data: \\n - {} rows \\n - {} columns\".format(train_data.shape[0], train_data.shape[1]))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:46.780657Z","iopub.execute_input":"2023-01-12T18:08:46.781004Z","iopub.status.idle":"2023-01-12T18:08:46.787521Z","shell.execute_reply.started":"2023-01-12T18:08:46.780975Z","shell.execute_reply":"2023-01-12T18:08:46.78648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"number of unique patients: \\n - {} patients\".format(train_data.patient_id.nunique()))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:47.134233Z","iopub.execute_input":"2023-01-12T18:08:47.13478Z","iopub.status.idle":"2023-01-12T18:08:47.150349Z","shell.execute_reply.started":"2023-01-12T18:08:47.134747Z","shell.execute_reply":"2023-01-12T18:08:47.149356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"mean number of images per patient: \\n - {:.2f} image/patient\".format(train_data[['patient_id','image_id']].groupby(['patient_id']).size().mean()))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:47.789608Z","iopub.execute_input":"2023-01-12T18:08:47.790445Z","iopub.status.idle":"2023-01-12T18:08:47.803835Z","shell.execute_reply.started":"2023-01-12T18:08:47.790411Z","shell.execute_reply":"2023-01-12T18:08:47.802817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"number of unique sites: \\n - {} sites\".format(train_data.site_id.nunique()))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:48.161999Z","iopub.execute_input":"2023-01-12T18:08:48.162477Z","iopub.status.idle":"2023-01-12T18:08:48.17012Z","shell.execute_reply.started":"2023-01-12T18:08:48.162438Z","shell.execute_reply":"2023-01-12T18:08:48.168989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"number of unique machines: \\n - {} machines\".format(train_data.machine_id.nunique()))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:48.580798Z","iopub.execute_input":"2023-01-12T18:08:48.581175Z","iopub.status.idle":"2023-01-12T18:08:48.587612Z","shell.execute_reply.started":"2023-01-12T18:08:48.581144Z","shell.execute_reply":"2023-01-12T18:08:48.586552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There are **54706 unique images** in the training set. \n* There are **11913 total patients**. \n* Each patient has an **average of 4.59 images**. \n* There are **2 hospitals** from where the records were gathered and a total of **10 machines** used.","metadata":{}},{"cell_type":"code","source":"ls /kaggle/input/rsna-breast-cancer-detection/train_images | head -1 -n 5","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:49.629595Z","iopub.execute_input":"2023-01-12T18:08:49.629941Z","iopub.status.idle":"2023-01-12T18:08:51.435793Z","shell.execute_reply.started":"2023-01-12T18:08:49.629914Z","shell.execute_reply":"2023-01-12T18:08:51.434592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls /kaggle/input/rsna-breast-cancer-detection/train_images/10006","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:51.441188Z","iopub.execute_input":"2023-01-12T18:08:51.441594Z","iopub.status.idle":"2023-01-12T18:08:52.415801Z","shell.execute_reply.started":"2023-01-12T18:08:51.441554Z","shell.execute_reply":"2023-01-12T18:08:52.414683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/test.csv')\ntest_data","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:52.611816Z","iopub.execute_input":"2023-01-12T18:08:52.612893Z","iopub.status.idle":"2023-01-12T18:08:52.636708Z","shell.execute_reply.started":"2023-01-12T18:08:52.612853Z","shell.execute_reply":"2023-01-12T18:08:52.635644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The images are divided into directories by patient id, with each directory having multiple images.","metadata":{}},{"cell_type":"code","source":"sample_submission = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv')\nsample_submission","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:08:53.810446Z","iopub.execute_input":"2023-01-12T18:08:53.810822Z","iopub.status.idle":"2023-01-12T18:08:53.826199Z","shell.execute_reply.started":"2023-01-12T18:08:53.810791Z","shell.execute_reply":"2023-01-12T18:08:53.825361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each patient and for each breast, we need to predict if cancer is present.","metadata":{}},{"cell_type":"markdown","source":"## **Exploratory Data Analysis:**","metadata":{}},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:02.870843Z","iopub.execute_input":"2023-01-12T18:09:02.871215Z","iopub.status.idle":"2023-01-12T18:09:02.889379Z","shell.execute_reply.started":"2023-01-12T18:09:02.871185Z","shell.execute_reply":"2023-01-12T18:09:02.888388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:03.293304Z","iopub.execute_input":"2023-01-12T18:09:03.293662Z","iopub.status.idle":"2023-01-12T18:09:03.325096Z","shell.execute_reply.started":"2023-01-12T18:09:03.293633Z","shell.execute_reply":"2023-01-12T18:09:03.3242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Number of images per patient:**","metadata":{}},{"cell_type":"code","source":"image_count = train_data[['patient_id','image_id']].groupby(['patient_id']).size().reset_index(name='counts')","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:04.345389Z","iopub.execute_input":"2023-01-12T18:09:04.345751Z","iopub.status.idle":"2023-01-12T18:09:04.357671Z","shell.execute_reply.started":"2023-01-12T18:09:04.345722Z","shell.execute_reply":"2023-01-12T18:09:04.356685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_count","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:04.772529Z","iopub.execute_input":"2023-01-12T18:09:04.773265Z","iopub.status.idle":"2023-01-12T18:09:04.787825Z","shell.execute_reply.started":"2023-01-12T18:09:04.773225Z","shell.execute_reply":"2023-01-12T18:09:04.786882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counts = image_count['counts'].value_counts().reset_index(name='total_counts')\ncounts['percentage'] = (100 * counts['total_counts'] / train_data.patient_id.nunique()).round(2)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:05.407442Z","iopub.execute_input":"2023-01-12T18:09:05.407799Z","iopub.status.idle":"2023-01-12T18:09:05.417556Z","shell.execute_reply.started":"2023-01-12T18:09:05.40777Z","shell.execute_reply":"2023-01-12T18:09:05.416389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counts","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:05.852373Z","iopub.execute_input":"2023-01-12T18:09:05.853548Z","iopub.status.idle":"2023-01-12T18:09:05.865586Z","shell.execute_reply.started":"2023-01-12T18:09:05.853506Z","shell.execute_reply":"2023-01-12T18:09:05.864409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,8))\ncounts_distribution = plt.bar(counts['index'],counts.total_counts)\nfor index, bar in enumerate(counts_distribution):\n    x, y = bar.get_xy()\n    width, height= bar.get_width(), bar.get_height()\n    plt.text(x, y + height*1.02, str(counts.percentage[index])+'%')\nplt.title('Distribution of the number of images per patient')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:06.488242Z","iopub.execute_input":"2023-01-12T18:09:06.489346Z","iopub.status.idle":"2023-01-12T18:09:06.898169Z","shell.execute_reply.started":"2023-01-12T18:09:06.489286Z","shell.execute_reply":"2023-01-12T18:09:06.897224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Most patients have 4 images, some have more.**","metadata":{}},{"cell_type":"markdown","source":"#### **Image Features:**","metadata":{}},{"cell_type":"markdown","source":"density, invasive, biopsy, cancer, BIRADS, difficult_negative_case are only provided for train.  \nLet's look at the distribution of some of the features provided for both train and test: site_id, laterality, view, implant, machine_id, age","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2, 2, figsize=(16, 10))\nsns.countplot(x = train_data.site_id, ax=ax[0, 0])\nsns.countplot(x = train_data.laterality, ax=ax[0, 1])\nsns.countplot(x = train_data.view, ax=ax[1, 0])\nsns.countplot(x = train_data.implant, ax=ax[1, 1])\nfor _ in ax:\n    for element in _:\n        element.bar_label(element.containers[0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:08.348139Z","iopub.execute_input":"2023-01-12T18:09:08.34851Z","iopub.status.idle":"2023-01-12T18:09:08.857187Z","shell.execute_reply.started":"2023-01-12T18:09:08.348477Z","shell.execute_reply":"2023-01-12T18:09:08.856247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"machine_per_site = train_data.groupby(['site_id']).agg({'machine_id': lambda x: list(x)})","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:08.859214Z","iopub.execute_input":"2023-01-12T18:09:08.859828Z","iopub.status.idle":"2023-01-12T18:09:08.875354Z","shell.execute_reply.started":"2023-01-12T18:09:08.859791Z","shell.execute_reply":"2023-01-12T18:09:08.874181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 3, figsize=(24, 5))\nsns.countplot(x = train_data.machine_id, ax=ax[0])\nsns.countplot(x = machine_per_site['machine_id'][1], ax=ax[1])\nsns.countplot(x = machine_per_site['machine_id'][2], ax=ax[2])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:09.008199Z","iopub.execute_input":"2023-01-12T18:09:09.008505Z","iopub.status.idle":"2023-01-12T18:09:09.515159Z","shell.execute_reply.started":"2023-01-12T18:09:09.008478Z","shell.execute_reply":"2023-01-12T18:09:09.514215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the distribution of the machines in site 2 is balanced while the machine 49 is the one mostly used in site 1. The two sites don't share the same machines.","metadata":{}},{"cell_type":"markdown","source":"Let's look at some age statistics:","metadata":{}},{"cell_type":"code","source":"print(f\"The minimum age is: {min(train_data['age'])}\")\nprint(f\"The maxiumum age is: {max(train_data['age'])}\")\nprint(f\"The mean age is: {np.mean(train_data['age']):.2f}\")\nprint(f\"The standard deviation of age is: {np.std(train_data['age']):.2f}\")","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:10.485351Z","iopub.execute_input":"2023-01-12T18:09:10.485747Z","iopub.status.idle":"2023-01-12T18:09:10.505898Z","shell.execute_reply.started":"2023-01-12T18:09:10.485714Z","shell.execute_reply":"2023-01-12T18:09:10.504599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [3, 1]}, figsize=(16, 8))\nsns.kdeplot(x = train_data.age,ax=a0)\nsns.boxenplot(x=train_data.age, ax=a1)\nsns.despine(right=True, top=True, left=True);\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:10.819674Z","iopub.execute_input":"2023-01-12T18:09:10.82036Z","iopub.status.idle":"2023-01-12T18:09:11.351174Z","shell.execute_reply.started":"2023-01-12T18:09:10.8203Z","shell.execute_reply":"2023-01-12T18:09:11.350253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take a look at some features available only for train:","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2, 2, figsize=(16, 10))\nsns.countplot(x = train_data.BIRADS, ax=ax[0, 0])\nsns.countplot(x = train_data.biopsy, ax=ax[0, 1])\nsns.countplot(x = train_data.invasive, ax=ax[1, 0])\nsns.countplot(x = train_data.density, ax=ax[1, 1])\nfor _ in ax:\n    for element in _:\n        element.bar_label(element.containers[0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:11.567764Z","iopub.execute_input":"2023-01-12T18:09:11.569802Z","iopub.status.idle":"2023-01-12T18:09:12.056201Z","shell.execute_reply.started":"2023-01-12T18:09:11.56977Z","shell.execute_reply":"2023-01-12T18:09:12.055244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(x = train_data[train_data.difficult_negative_case == True].age)\nsns.kdeplot(x = train_data[train_data.difficult_negative_case == False].age)\nplt.legend(labels = [\"difficult_negative_case: True\", \"difficult_negative_case: False\"] )\nplt.title(\"Age - difficult_negative_case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:12.058908Z","iopub.execute_input":"2023-01-12T18:09:12.059369Z","iopub.status.idle":"2023-01-12T18:09:12.49326Z","shell.execute_reply.started":"2023-01-12T18:09:12.05934Z","shell.execute_reply":"2023-01-12T18:09:12.492259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Target feature:**","metadata":{}},{"cell_type":"markdown","source":"As a patient may have cancer in one breast only we count them as having cancer if the sum of cancer is not 0.","metadata":{}},{"cell_type":"code","source":"cancer_grouped_by_id = train_data.groupby(['patient_id']).agg({'cancer': lambda x: x.unique().sum()})\ncancer_counts = cancer_grouped_by_id['cancer'].value_counts().reset_index(name='cancer_counts')\ncancer_counts['percentage'] = (100 * cancer_counts['cancer_counts'] / train_data.patient_id.nunique()).round(2)\ncancer_counts","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:12.888684Z","iopub.execute_input":"2023-01-12T18:09:12.890858Z","iopub.status.idle":"2023-01-12T18:09:13.399547Z","shell.execute_reply.started":"2023-01-12T18:09:12.890814Z","shell.execute_reply":"2023-01-12T18:09:13.39846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,8))\ncancer_distribution = plt.bar(cancer_counts['index'],cancer_counts.cancer_counts)\nfor index, bar in enumerate(cancer_distribution):\n    x, y = bar.get_xy()\n    width, height= bar.get_width(), bar.get_height()\n    plt.text(x, y + height*1.02, str(cancer_counts.percentage[index])+'%')\nplt.title('Distribution of cancer among patients')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:13.401689Z","iopub.execute_input":"2023-01-12T18:09:13.402063Z","iopub.status.idle":"2023-01-12T18:09:13.602218Z","shell.execute_reply.started":"2023-01-12T18:09:13.402028Z","shell.execute_reply":"2023-01-12T18:09:13.601202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cancer_grouped_by_breast = train_data.groupby(['patient_id','laterality']).agg({'cancer': lambda x: x.unique().sum()})\ncancer_counts =cancer_grouped_by_breast['cancer'].value_counts().reset_index(name='cancer_counts')\ncancer_counts['percentage'] = (100 * cancer_counts['cancer_counts'] / len(train_data.groupby(['patient_id','laterality']))).round(2)\ncancer_counts","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:13.603841Z","iopub.execute_input":"2023-01-12T18:09:13.604113Z","iopub.status.idle":"2023-01-12T18:09:14.907457Z","shell.execute_reply.started":"2023-01-12T18:09:13.604088Z","shell.execute_reply":"2023-01-12T18:09:14.906271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,8))\ncancer_distribution = plt.bar(cancer_counts['index'],cancer_counts.cancer_counts)\nfor index, bar in enumerate(cancer_distribution):\n    x, y = bar.get_xy()\n    width, height= bar.get_width(), bar.get_height()\n    plt.text(x, y + height*1.02, str(cancer_counts.percentage[index])+'%')\nplt.title('Distribution of cancer for breasts')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:14.909604Z","iopub.execute_input":"2023-01-12T18:09:14.910002Z","iopub.status.idle":"2023-01-12T18:09:15.105699Z","shell.execute_reply.started":"2023-01-12T18:09:14.909964Z","shell.execute_reply":"2023-01-12T18:09:15.104802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(x = train_data[train_data.cancer == True].age)\nsns.kdeplot(x = train_data[train_data.cancer == False].age)\nplt.legend(labels = [\"Cancer: True\", \"Cancer: False\"] )\nplt.title(\"Age - Cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:15.107373Z","iopub.execute_input":"2023-01-12T18:09:15.107799Z","iopub.status.idle":"2023-01-12T18:09:15.527228Z","shell.execute_reply.started":"2023-01-12T18:09:15.107763Z","shell.execute_reply":"2023-01-12T18:09:15.526355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Feature Correlation:**","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,12))\nsns.heatmap(train_data.corr(),annot = True,ax = ax, mask = np.triu(train_data.corr()))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:16.020964Z","iopub.execute_input":"2023-01-12T18:09:16.021562Z","iopub.status.idle":"2023-01-12T18:09:16.715326Z","shell.execute_reply.started":"2023-01-12T18:09:16.021525Z","shell.execute_reply":"2023-01-12T18:09:16.714212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Missing values:**","metadata":{}},{"cell_type":"code","source":"train_data.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:16.717025Z","iopub.execute_input":"2023-01-12T18:09:16.717509Z","iopub.status.idle":"2023-01-12T18:09:16.737153Z","shell.execute_reply.started":"2023-01-12T18:09:16.717473Z","shell.execute_reply":"2023-01-12T18:09:16.736268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are missing values for age, BIRADS and Density. We may think of imputing the missing values for the age column","metadata":{}},{"cell_type":"code","source":"train_data['age'].mean()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:17.50353Z","iopub.execute_input":"2023-01-12T18:09:17.503886Z","iopub.status.idle":"2023-01-12T18:09:17.51186Z","shell.execute_reply.started":"2023-01-12T18:09:17.503856Z","shell.execute_reply":"2023-01-12T18:09:17.510794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data['age'] = train_data['age'].fillna(58)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:17.882063Z","iopub.execute_input":"2023-01-12T18:09:17.882688Z","iopub.status.idle":"2023-01-12T18:09:17.889164Z","shell.execute_reply.started":"2023-01-12T18:09:17.882647Z","shell.execute_reply":"2023-01-12T18:09:17.888152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **DICOM image overview:**","metadata":{}},{"cell_type":"markdown","source":"Let's first take a look at the metadata from one DICOM image.","metadata":{}},{"cell_type":"code","source":"pydicom.dcmread('/kaggle/input/rsna-breast-cancer-detection/train_images/10006/462822612.dcm')","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:18.794451Z","iopub.execute_input":"2023-01-12T18:09:18.795408Z","iopub.status.idle":"2023-01-12T18:09:18.888206Z","shell.execute_reply.started":"2023-01-12T18:09:18.795364Z","shell.execute_reply":"2023-01-12T18:09:18.887181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def rescale_img_to_hu(dcm_ds):\n    \"\"\"Rescales the image to Hounsfield unit.\"\"\"\n    data = dcm_ds.pixel_array\n    if dcm_ds.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    return data * dcm_ds.RescaleSlope + dcm_ds.RescaleIntercept\ndef show_images_for_patient(patient_id):\n    patient_dir = os.path.join('../input/rsna-breast-cancer-detection/train_images', str(patient_id))\n    num_images = len(glob.glob(f\"{patient_dir}/*\"))\n    print(f\"Number of images for patient: {num_images}\")\n    fig, axs = plt.subplots(2, 2, figsize=(24,15))\n    axs = axs.flatten()\n    for i, img_path in enumerate(list(Path(patient_dir).iterdir())):\n        ds = pydicom.dcmread(img_path)\n        axs[i].imshow(rescale_img_to_hu(ds), cmap=\"bone\")\nshow_images_for_patient(10006)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:19.138376Z","iopub.execute_input":"2023-01-12T18:09:19.139829Z","iopub.status.idle":"2023-01-12T18:09:37.337821Z","shell.execute_reply.started":"2023-01-12T18:09:19.139788Z","shell.execute_reply":"2023-01-12T18:09:37.336841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Data Preparation:**","metadata":{}},{"cell_type":"code","source":"image_folder = \"/kaggle/input/rsna-breast-cancer-512-pngs/\"\ntrain_data['image_path'] = train_data.apply(lambda x:image_folder +  str(x.patient_id) + '_' + str(x.image_id) + '.png', axis = 1)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:37.339874Z","iopub.execute_input":"2023-01-12T18:09:37.340483Z","iopub.status.idle":"2023-01-12T18:09:38.525486Z","shell.execute_reply.started":"2023-01-12T18:09:37.340444Z","shell.execute_reply":"2023-01-12T18:09:38.524389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"le_laterality = LabelEncoder()\nle_view = LabelEncoder()\ntrain_data['laterality'] = le_laterality.fit_transform(train_data['laterality'])\ntrain_data['view'] = le_view.fit_transform(train_data['view'])\ntrain_data['cancer'] = train_data['cancer'].astype('str')\ntrain_data = train_data[[\"patient_id\", \"image_id\", \"image_path\", \"laterality\", \"view\", \"age\", \"implant\", \"cancer\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:38.526958Z","iopub.execute_input":"2023-01-12T18:09:38.527496Z","iopub.status.idle":"2023-01-12T18:09:38.625427Z","shell.execute_reply.started":"2023-01-12T18:09:38.527459Z","shell.execute_reply":"2023-01-12T18:09:38.624389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:38.628079Z","iopub.execute_input":"2023-01-12T18:09:38.628778Z","iopub.status.idle":"2023-01-12T18:09:38.646744Z","shell.execute_reply.started":"2023-01-12T18:09:38.628741Z","shell.execute_reply":"2023-01-12T18:09:38.645585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.to_csv(\"train_data.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:38.648595Z","iopub.execute_input":"2023-01-12T18:09:38.649419Z","iopub.status.idle":"2023-01-12T18:09:38.818893Z","shell.execute_reply.started":"2023-01-12T18:09:38.649383Z","shell.execute_reply":"2023-01-12T18:09:38.817834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Metric:**","metadata":{}},{"cell_type":"code","source":"def pfbeta_torch(labels, preds, beta=1):\n    preds = preds.clip(0, 1)\n    y_true_count = labels.sum()\n    ctp = preds[labels==1].sum()\n    cfp = preds[labels==0].sum()\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / y_true_count\n    if (c_precision > 0 and c_recall > 0):\n        result = (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n        return result\n    else:\n        return 0.0","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:38.820529Z","iopub.execute_input":"2023-01-12T18:09:38.820915Z","iopub.status.idle":"2023-01-12T18:09:38.962464Z","shell.execute_reply.started":"2023-01-12T18:09:38.820878Z","shell.execute_reply":"2023-01-12T18:09:38.961118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Pytorch model from scratch:**","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv(\"/kaggle/input/train-data-rsna/train_data.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:38.963977Z","iopub.execute_input":"2023-01-12T18:09:38.964604Z","iopub.status.idle":"2023-01-12T18:09:39.125175Z","shell.execute_reply.started":"2023-01-12T18:09:38.964567Z","shell.execute_reply":"2023-01-12T18:09:39.124205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:39.126728Z","iopub.execute_input":"2023-01-12T18:09:39.127064Z","iopub.status.idle":"2023-01-12T18:09:40.124481Z","shell.execute_reply.started":"2023-01-12T18:09:39.12703Z","shell.execute_reply":"2023-01-12T18:09:40.123257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Device available now:', DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:40.126811Z","iopub.execute_input":"2023-01-12T18:09:40.127236Z","iopub.status.idle":"2023-01-12T18:09:40.198378Z","shell.execute_reply.started":"2023-01-12T18:09:40.127194Z","shell.execute_reply":"2023-01-12T18:09:40.19735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output_size = 1","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:40.20197Z","iopub.execute_input":"2023-01-12T18:09:40.202617Z","iopub.status.idle":"2023-01-12T18:09:40.209031Z","shell.execute_reply.started":"2023-01-12T18:09:40.202577Z","shell.execute_reply":"2023-01-12T18:09:40.208175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RSNAImageDataset(Dataset):\n    \n    def __init__(self, dataframe, is_train=True):\n        self.dataframe, self.is_train = dataframe, is_train\n                 \n    def __len__(self):\n        return len(self.dataframe)\n    \n    def __getitem__(self, index):\n        \n        image_path = self.dataframe['image_path'][index]\n        image = read_image(image_path)*(1./255)\n        transf_image = np.concatenate([image, image, image], axis=0)\n        \n        if self.is_train:\n            return {\"image\": transf_image, \n                    \"target\": self.dataframe['cancer'][index]}\n        else:\n            return {\"image\": transf_image}","metadata":{"execution":{"iopub.status.busy":"2023-01-12T18:09:41.272147Z","iopub.execute_input":"2023-01-12T18:09:41.273119Z","iopub.status.idle":"2023-01-12T18:09:41.278448Z","shell.execute_reply.started":"2023-01-12T18:09:41.273075Z","shell.execute_reply":"2023-01-12T18:09:41.277235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = train_data.head(6)\n\ndataset = RSNAImageDataset(sample_df, is_train=True)\n\ndataloader = DataLoader(dataset, batch_size=3, shuffle=False)\n\nfor k, data in enumerate(dataloader):\n    image,targets = data.values()\n    image,targets = image.to(DEVICE), targets.to(DEVICE)\n    print( f\"Batch: {k}\" \"\\n\" +\n           \"Image:\", image.shape, \"\\n\" +\n           \"Targets:\" , targets, \"\\n\" +\n          \"=\"*50)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:38:23.032864Z","iopub.execute_input":"2023-01-09T17:38:23.033217Z","iopub.status.idle":"2023-01-09T17:38:26.242093Z","shell.execute_reply.started":"2023-01-09T17:38:23.033187Z","shell.execute_reply":"2023-01-09T17:38:26.240921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CNNNetwork(nn.Module):\n    def __init__(self, output_size):\n        super().__init__()\n        self.output_size = output_size\n\n        self.features = nn.Sequential(\n            nn.Conv2d(3, 128, kernel_size = 3, padding = 1),\n            nn.ReLU(),\n            nn.Conv2d(128,64, kernel_size = 3, stride = 1, padding = 1),\n            nn.ReLU(),\n            nn.MaxPool2d(2,2),\n            nn.Conv2d(64, 32, kernel_size = 3, stride = 1, padding = 1),\n            nn.ReLU(),\n            nn.MaxPool2d(2,2),\n            nn.Conv2d(32, 16, kernel_size = 3, stride = 1, padding = 1),\n            nn.ReLU(),\n            nn.MaxPool2d(2,2),\n            nn.Flatten(),\n            nn.Linear(65536,1024),\n            nn.ReLU(),\n            nn.Linear(1024,512),\n            nn.ReLU(),\n            nn.Linear(512,output_size)\n        )\n        \n        \n    def forward(self, image, prints=False):\n        if prints: print('Input Image shape:', image.shape, '\\n')\n        \n        # Image CNN\n        out = self.features(image)\n        if prints: print('Features Image shape:', image.shape)\n        \n        return out","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:39:53.689074Z","iopub.execute_input":"2023-01-09T17:39:53.689459Z","iopub.status.idle":"2023-01-09T17:39:53.698319Z","shell.execute_reply.started":"2023-01-09T17:39:53.689427Z","shell.execute_reply":"2023-01-09T17:39:53.697374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_example = CNNNetwork(output_size=output_size).to(DEVICE)\n\nout = model_example(image, prints=True)\n\ncriterion_example = nn.BCEWithLogitsLoss()\nloss = criterion_example(out, targets.unsqueeze(1).float()) \nprint(\"=\"*50)\nprint('Loss:', loss.item())","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:39:58.243442Z","iopub.execute_input":"2023-01-09T17:39:58.243799Z","iopub.status.idle":"2023-01-09T17:39:59.923504Z","shell.execute_reply.started":"2023-01-09T17:39:58.243769Z","shell.execute_reply":"2023-01-09T17:39:59.922509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_folds(model, train_original):\n    \n    group_fold = GroupKFold(n_splits = FOLDS)\n    k_folds = group_fold.split(X = np.zeros(len(train_original)), \n                               y = train_original['cancer'], \n                               groups = train_original['patient_id'].tolist())\n    \n    for i, (train_index, valid_index) in enumerate(k_folds):\n        \n        print(f\"---------- Fold: {i+1} ----------\")\n        best_roc = None\n        patience_f = PATIENCE\n\n        optimizer = torch.optim.Adam(model.parameters(), lr = LR, \n                                     weight_decay=WD)\n        scheduler = ReduceLROnPlateau(optimizer=optimizer, mode='max', \n                                      patience=LR_PATIENCE, verbose=True, factor=LR_FACTOR)\n        criterion = nn.BCEWithLogitsLoss()\n        train_data = train_original.iloc[train_index].reset_index(drop=True)\n        valid_data = train_original.iloc[valid_index].reset_index(drop=True)\n\n        train = RSNAImageDataset(train_data, is_train=True)\n        valid = RSNAImageDataset(valid_data, is_train=True)\n\n        train_loader = DataLoader(train, batch_size=BATCH_SIZE1, \n                                  shuffle=True, num_workers=WORKERS)\n        valid_loader = DataLoader(valid, batch_size=BATCH_SIZE2, \n                                  shuffle=False, num_workers=WORKERS)\n\n\n        for epoch in range(EPOCHS):\n            start_time = time()\n            correct = 0\n            train_losses = 0\n            model.train()\n\n            for k, data in tqdm(enumerate(train_loader)):\n                image, targets = data.values()\n                image, targets = image.to(DEVICE), targets.to(DEVICE)\n                optimizer.zero_grad()\n                out = model(image)\n                loss = criterion(out, targets.unsqueeze(1).float())\n                loss.backward()\n                optimizer.step()\n\n                train_losses += loss.item()\n                train_preds = torch.round(torch.sigmoid(out)) # 0 and 1\n                correct += (train_preds.cpu() == targets.cpu().unsqueeze(1)).sum().item()\n\n            train_acc = correct / len(train_index)\n            model.eval()\n            valid_preds = torch.zeros(size = (len(valid_index), 1), \n                                      device=DEVICE, dtype=torch.float32)\n            with torch.no_grad():\n                for k, data in tqdm(enumerate(valid_loader)):\n                    image, targets = data.values()\n                    image, targets = image.to(DEVICE), targets.to(DEVICE)\n                    out = model(image)\n                    pred = torch.sigmoid(out)\n                    valid_preds[k*image.shape[0] : k*image.shape[0] + image.shape[0]] = pred\n\n                # Calculate accuracy\n                valid_acc = accuracy_score(valid_data['cancer'].values, \n                                           torch.round(valid_preds.cpu()))\n                valid_roc = roc_auc_score(valid_data['cancer'].values, \n                                          valid_preds.cpu())\n                # Calculate time on Train + Eval\n                duration = str(dtime.timedelta(seconds=time() - start_time))[:7]\n\n\n                # PRINT INFO\n                final_logs = '{} | Epoch: {}/{} | Loss: {:.4} | Acc_tr: {:.3} | Acc_vd: {:.3} | ROC: {:.3}'.\\\n                                format(duration, epoch+1, EPOCHS, \n                                       train_losses, train_acc, valid_acc, valid_roc)\n                print(final_logs)\n\n\n\n                scheduler.step(valid_roc)\n                # Name the model\n                model_name = f\"Fold{i+1}_Epoch{epoch+1}_ValidAcc{valid_acc:.3f}_ROC{valid_roc:.3f}.pth\"\n\n                # Update best_roc\n                if not best_roc: # If best_roc = None\n                    best_roc = valid_roc\n                    torch.save(model.state_dict(), model_name)\n                    continue\n\n                if valid_roc > best_roc:\n                    best_roc = valid_roc\n                    # Reset patience (because we have improvement)\n                    patience_f = PATIENCE\n                    torch.save(model.state_dict(), model_name)\n                else:\n                    # Decrease patience (no improvement in ROC)\n                    patience_f = patience_f - 1\n                    if patience_f == 0:\n                        stop_logs = 'Early stopping (no improvement since 3 models) | Best ROC: {}'.\\\n                                    format(best_roc)\n                        print(stop_logs)\n                        break\n\n        del train, valid, train_loader, valid_loader, image, targets\n        gc.collect()\n        ","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:40:03.029991Z","iopub.execute_input":"2023-01-09T17:40:03.030493Z","iopub.status.idle":"2023-01-09T17:40:03.274471Z","shell.execute_reply.started":"2023-01-09T17:40:03.030452Z","shell.execute_reply":"2023-01-09T17:40:03.273174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FOLDS = 3\nEPOCHS = 1\nPATIENCE = 3\nWORKERS = 2\nLR = 0.0005\nWD = 0.0\nLR_PATIENCE = 1            # 1 model not improving until lr is decreasing\nLR_FACTOR = 0.4            # by how much the lr is decreasing\n\nBATCH_SIZE1 = 2           # for train\nBATCH_SIZE2 = 2           # for valid\n\nVERSION = 'v1'\nMODEL = 'CNN'\nscratch = CNNNetwork(output_size=output_size).to(DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:40:05.108787Z","iopub.execute_input":"2023-01-09T17:40:05.109314Z","iopub.status.idle":"2023-01-09T17:40:06.039229Z","shell.execute_reply.started":"2023-01-09T17:40:05.109252Z","shell.execute_reply":"2023-01-09T17:40:06.038265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_folds(model=scratch, train_original=train_data)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T17:40:09.338079Z","iopub.execute_input":"2023-01-09T17:40:09.338484Z","iopub.status.idle":"2023-01-09T18:46:40.779888Z","shell.execute_reply.started":"2023-01-09T17:40:09.33845Z","shell.execute_reply":"2023-01-09T18:46:40.778819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scratch.eval()\ntrain = RSNAImageDataset(train_data, is_train=True)\ntrain_loader = DataLoader(train, batch_size=BATCH_SIZE1, \n                                  shuffle=False, num_workers=WORKERS)\npreds = np.empty((0,  1))\nwith torch.no_grad():\n    for k, data in enumerate(train_loader):\n        image, targets = data.values()\n        pred = scratch(image.to(DEVICE))\n        pred = pred.sigmoid()\n        preds = np.concatenate([preds, pred.cpu().numpy()])\npreds = preds[:,0]\nlabels = train_data['cancer']\nprint(pfbeta_torch(labels, preds, beta=1))","metadata":{"execution":{"iopub.status.busy":"2023-01-09T18:47:14.675833Z","iopub.execute_input":"2023-01-09T18:47:14.676199Z","iopub.status.idle":"2023-01-09T18:55:46.098765Z","shell.execute_reply.started":"2023-01-09T18:47:14.676167Z","shell.execute_reply":"2023-01-09T18:55:46.096672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Pytorch reference model:**","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv(\"/kaggle/input/train-data-rsna/train_data.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:29.610978Z","iopub.execute_input":"2023-01-09T19:11:29.611324Z","iopub.status.idle":"2023-01-09T19:11:29.688675Z","shell.execute_reply.started":"2023-01-09T19:11:29.611295Z","shell.execute_reply":"2023-01-09T19:11:29.687701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:29.961999Z","iopub.execute_input":"2023-01-09T19:11:29.962574Z","iopub.status.idle":"2023-01-09T19:11:30.998245Z","shell.execute_reply.started":"2023-01-09T19:11:29.96254Z","shell.execute_reply":"2023-01-09T19:11:30.996817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Device available now:', DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:31.001719Z","iopub.execute_input":"2023-01-09T19:11:31.002239Z","iopub.status.idle":"2023-01-09T19:11:31.009512Z","shell.execute_reply.started":"2023-01-09T19:11:31.002189Z","shell.execute_reply":"2023-01-09T19:11:31.008184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"csv_columns = ['laterality', 'view', 'age', 'implant']\nno_columns = len(csv_columns)\noutput_size = 1","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:31.011811Z","iopub.execute_input":"2023-01-09T19:11:31.012213Z","iopub.status.idle":"2023-01-09T19:11:31.018788Z","shell.execute_reply.started":"2023-01-09T19:11:31.012172Z","shell.execute_reply":"2023-01-09T19:11:31.017655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RSNADataset(Dataset):\n    \n    def __init__(self, dataframe, is_train=True):\n        self.dataframe, self.is_train = dataframe, is_train\n                 \n    def __len__(self):\n        return len(self.dataframe)\n    \n    def __getitem__(self, index):\n        \n        # Select path and read image\n        image_path = self.dataframe['image_path'][index]\n        #transf_image = cv2.imread(image_path).permute(1,2,0).astype(np.float32)\n        image = read_image(image_path)*(1./255)\n        \n        csv_data = np.array(self.dataframe.iloc[index][csv_columns].values, \n                            dtype=np.float32)\n\n        transf_image = np.concatenate([image, image, image], axis=0)\n        \n        # Return info\n        if self.is_train:\n            return {\"image\": transf_image, \n                    \"meta\": csv_data, \n                    \"target\": self.dataframe['cancer'][index]}\n        else:\n            return {\"image\": transf_image, \n                    \"meta\": csv_data}","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:31.120248Z","iopub.execute_input":"2023-01-09T19:11:31.120644Z","iopub.status.idle":"2023-01-09T19:11:31.129257Z","shell.execute_reply.started":"2023-01-09T19:11:31.120612Z","shell.execute_reply":"2023-01-09T19:11:31.128116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_to_device(data):  \n    image, metadata, targets = data.values()\n    return image.to(DEVICE), metadata.to(DEVICE), targets.to(DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:31.540452Z","iopub.execute_input":"2023-01-09T19:11:31.540804Z","iopub.status.idle":"2023-01-09T19:11:31.54597Z","shell.execute_reply.started":"2023-01-09T19:11:31.540776Z","shell.execute_reply":"2023-01-09T19:11:31.544983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sanity Check:**","metadata":{}},{"cell_type":"code","source":"# Sample data\nsample_df = train_data.head(6)\n\n# Instantiate Dataset object\ndataset = RSNADataset(sample_df, is_train=True)\n# The Dataloader\ndataloader = DataLoader(dataset, batch_size=3, shuffle=False)\n\n# Output of the Dataloader\nfor k, data in enumerate(dataloader):\n    image, meta, targets = data_to_device(data)\n    print( f\"Batch: {k}\" \"\\n\" +\n           \"Image:\", image.shape, \"\\n\" +\n           \"Meta:\" , meta, \"\\n\" +\n           \"Targets:\" , targets, \"\\n\" +\n          \"=\"*50)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:11:32.910259Z","iopub.execute_input":"2023-01-09T19:11:32.910945Z","iopub.status.idle":"2023-01-09T19:11:33.007529Z","shell.execute_reply.started":"2023-01-09T19:11:32.910909Z","shell.execute_reply":"2023-01-09T19:11:33.006187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ResNet50Network(nn.Module):\n    def __init__(self, output_size, no_columns):\n        super().__init__()\n        self.no_columns, self.output_size = no_columns, output_size\n        \n        # Define Feature part (IMAGE)\n        self.features = resnet50(pretrained=True) # 1000 neurons out\n        # (metadata)\n        self.csv = nn.Sequential(nn.Linear(self.no_columns, 500),\n                                 nn.BatchNorm1d(500),\n                                 nn.ReLU(),\n                                 nn.Dropout(p=0.2))\n        \n        # Define Classification part\n        self.classification = nn.Linear(1000 + 500, output_size)\n        \n        \n    def forward(self, image, meta, prints=False):\n        if prints: print('Input Image shape:', image.shape, '\\n'+\n                         'Input metadata shape:', meta.shape)\n        \n        # Image CNN\n        image = self.features(image)\n        if prints: print('Features Image shape:', image.shape)\n        \n        # CSV FNN\n        meta = self.csv(meta)\n        if prints: print('Meta Data:', meta.shape)\n            \n        # Concatenate layers from image with layers from csv_data\n        image_meta_data = torch.cat((image, meta), dim=1)\n        if prints: print('Concatenated Data:', image_meta_data.shape)\n        \n        # CLASSIF\n        out = self.classification(image_meta_data)\n        if prints: print('Out shape:', out.shape)\n        \n        return out","metadata":{"execution":{"iopub.status.busy":"2023-01-10T00:06:32.310664Z","iopub.execute_input":"2023-01-10T00:06:32.311078Z","iopub.status.idle":"2023-01-10T00:06:32.328689Z","shell.execute_reply.started":"2023-01-10T00:06:32.311046Z","shell.execute_reply":"2023-01-10T00:06:32.327994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" **Sanity Check:**","metadata":{}},{"cell_type":"code","source":"model_example = ResNet50Network(output_size=output_size, no_columns=no_columns).to(DEVICE)\n\nout = model_example(image, meta, prints=True)\n\ncriterion_example = nn.BCEWithLogitsLoss()\nloss = criterion_example(out, targets.unsqueeze(1).float()) \nprint(\"=\"*50)\nprint('Loss:', loss.item())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_folds(model, train_original):\n    \n    group_fold = GroupKFold(n_splits = FOLDS)\n    k_folds = group_fold.split(X = np.zeros(len(train_original)), \n                               y = train_original['cancer'], \n                               groups = train_original['patient_id'].tolist())\n    \n    for i, (train_index, valid_index) in enumerate(k_folds):\n        \n        print(f\"---------- Fold: {i+1} ----------\")\n        best_roc = None\n        patience_f = PATIENCE\n\n        optimizer = torch.optim.Adam(model.parameters(), lr = LR, \n                                     weight_decay=WD)\n        scheduler = ReduceLROnPlateau(optimizer=optimizer, mode='max', \n                                      patience=LR_PATIENCE, verbose=True, factor=LR_FACTOR)\n        criterion = nn.BCEWithLogitsLoss()\n        train_data = train_original.iloc[train_index].reset_index(drop=True)\n        valid_data = train_original.iloc[valid_index].reset_index(drop=True)\n\n        train = RSNADataset(train_data, is_train=True)\n        valid = RSNADataset(valid_data, is_train=True)\n\n        train_loader = DataLoader(train, batch_size=BATCH_SIZE1, \n                                  shuffle=True, num_workers=WORKERS)\n        valid_loader = DataLoader(valid, batch_size=BATCH_SIZE2, \n                                  shuffle=False, num_workers=WORKERS)\n\n\n        for epoch in range(EPOCHS):\n            start_time = time()\n            correct = 0\n            train_losses = 0\n            model.train()\n\n            for k, data in tqdm(enumerate(train_loader)):\n                image, meta, targets = data_to_device(data)\n                optimizer.zero_grad()\n                out = model(image, meta)\n                loss = criterion(out, targets.unsqueeze(1).float())\n                loss.backward()\n                optimizer.step()\n\n                train_losses += loss.item()\n                train_preds = torch.round(torch.sigmoid(out)) # 0 and 1\n                correct += (train_preds.cpu() == targets.cpu().unsqueeze(1)).sum().item()\n\n            train_acc = correct / len(train_index)\n            model.eval()\n            valid_preds = torch.zeros(size = (len(valid_index), 1), \n                                      device=DEVICE, dtype=torch.float32)\n            with torch.no_grad():\n                for k, data in tqdm(enumerate(valid_loader)):\n                    image, meta, targets = data_to_device(data)\n                    out = model(image, meta)\n                    pred = torch.sigmoid(out)\n                    valid_preds[k*image.shape[0] : k*image.shape[0] + image.shape[0]] = pred\n\n                # Calculate accuracy\n                valid_acc = accuracy_score(valid_data['cancer'].values, \n                                           torch.round(valid_preds.cpu()))\n                valid_roc = roc_auc_score(valid_data['cancer'].values, \n                                          valid_preds.cpu())\n                # Calculate time on Train + Eval\n                duration = str(dtime.timedelta(seconds=time() - start_time))[:7]\n\n\n                # PRINT INFO\n                final_logs = '{} | Epoch: {}/{} | Loss: {:.4} | Acc_tr: {:.3} | Acc_vd: {:.3} | ROC: {:.3}'.\\\n                                format(duration, epoch+1, EPOCHS, \n                                       train_losses, train_acc, valid_acc, valid_roc)\n                print(final_logs)\n\n\n\n                scheduler.step(valid_roc)\n                # Name the model\n                model_name = f\"Fold{i+1}_Epoch{epoch+1}_ValidAcc{valid_acc:.3f}_ROC{valid_roc:.3f}.pth\"\n\n                # Update best_roc\n                if not best_roc: # If best_roc = None\n                    best_roc = valid_roc\n                    torch.save(model, model_name)\n                    continue\n\n                if valid_roc > best_roc:\n                    best_roc = valid_roc\n                    # Reset patience (because we have improvement)\n                    patience_f = PATIENCE\n                    torch.save(model, model_name)\n                else:\n                    # Decrease patience (no improvement in ROC)\n                    patience_f = patience_f - 1\n                    if patience_f == 0:\n                        stop_logs = 'Early stopping (no improvement since 3 models) | Best ROC: {}'.\\\n                                    format(best_roc)\n                        print(stop_logs)\n                        break\n\n        del train, valid, train_loader, valid_loader, image, targets\n        gc.collect()\n        ","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:25:51.618822Z","iopub.execute_input":"2023-01-09T19:25:51.619213Z","iopub.status.idle":"2023-01-09T19:25:51.641684Z","shell.execute_reply.started":"2023-01-09T19:25:51.619177Z","shell.execute_reply":"2023-01-09T19:25:51.640771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FOLDS = 3\nEPOCHS = 7\nPATIENCE = 3\nWORKERS = 2\nLR = 0.0005\nWD = 0.0\nLR_PATIENCE = 1            # 1 model not improving until lr is decreasing\nLR_FACTOR = 0.4            # by how much the lr is decreasing\n\nBATCH_SIZE1 = 16           # for train\nBATCH_SIZE2 = 16           # for valid\n\nVERSION = 'v1'\nMODEL = 'resnet50'\nmodel1 = ResNet50Network(output_size=output_size, no_columns=no_columns).to(DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T20:03:29.432003Z","iopub.execute_input":"2023-01-09T20:03:29.432409Z","iopub.status.idle":"2023-01-09T20:03:29.952956Z","shell.execute_reply.started":"2023-01-09T20:03:29.432374Z","shell.execute_reply":"2023-01-09T20:03:29.951975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_folds(model=model1, train_original=train_data)","metadata":{"execution":{"iopub.status.busy":"2023-01-09T20:03:30.004537Z","iopub.execute_input":"2023-01-09T20:03:30.005003Z","iopub.status.idle":"2023-01-10T00:05:14.4475Z","shell.execute_reply.started":"2023-01-09T20:03:30.004973Z","shell.execute_reply":"2023-01-10T00:05:14.444637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.save(model1, 'test_model2.pth')","metadata":{"execution":{"iopub.status.busy":"2023-01-10T00:07:21.525305Z","iopub.execute_input":"2023-01-10T00:07:21.525904Z","iopub.status.idle":"2023-01-10T00:07:21.547169Z","shell.execute_reply.started":"2023-01-10T00:07:21.52587Z","shell.execute_reply":"2023-01-10T00:07:21.546026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = RSNADataset(train_data, is_train=True)\ntrain_loader = DataLoader(train, batch_size=BATCH_SIZE1, \n                                  shuffle=False, num_workers=WORKERS)\npreds = np.empty((0,  1))\nwith torch.no_grad():\n    for k, data in enumerate(train_loader):\n        image, meta, targets = data.values()\n        pred = model(image.to(DEVICE), meta.to(DEVICE))\n        pred = pred.sigmoid()\n        preds = np.concatenate([preds, pred.cpu().numpy()])\npreds = preds[:,0]\nlabels = train_data['cancer']\nprint(pfbeta_torch(labels, preds, beta=1))","metadata":{"execution":{"iopub.status.busy":"2023-01-10T00:07:25.613638Z","iopub.execute_input":"2023-01-10T00:07:25.613989Z","iopub.status.idle":"2023-01-10T00:16:10.193119Z","shell.execute_reply.started":"2023-01-10T00:07:25.613958Z","shell.execute_reply":"2023-01-10T00:16:10.19106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Submission**","metadata":{}},{"cell_type":"code","source":"test_images = glob.glob(\"/kaggle/input/rsna-breast-cancer-detection/test_images/*/*.dcm\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of images :\", len(test_images))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAVE_FOLDER = \"/kaggle/tmp/output/\"\nSIZE = 512\nEXTENSION = \"png\"\n\nos.makedirs(SAVE_FOLDER, exist_ok=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def process(f, size=512, save_folder=\"\", extension=\"png\"):\n    patient = f.split('/')[-2]\n    image = f.split('/')[-1][:-4]\n\n    dicom = pydicom.dcmread(f)\n    img = dicom.pixel_array\n\n    img = (img - img.min()) / (img.max() - img.min())\n\n    if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        img = 1 - img\n\n    img = cv2.resize(img, (size, size))\n\n    cv2.imwrite(save_folder + f\"{patient}_{image}.{extension}\", (img * 255).astype(np.uint8))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_ = Parallel(n_jobs=2)(\n    delayed(process)(uid, size=SIZE, save_folder=SAVE_FOLDER, extension=EXTENSION)\n    for uid in tqdm(test_images)\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\ndf['cancer'] = 0\ndf['image_path'] = SAVE_FOLDER + df[\"patient_id\"].astype(str) + \"_\" + df[\"image_id\"].astype(str) + \".png\"\ndf['laterality'] = le_laterality.fit_transform(df['laterality'])\ndf['view'] = le_view.fit_transform(df['view'])\ndf['cancer'] = df['cancer'].astype('str')\ndf = df[[\"patient_id\", \"image_id\", \"image_path\", \"laterality\", \"view\", \"age\", \"implant\", \"cancer\"]]\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = torch.load('/kaggle/input/rsna-models/test_model.pth')\nmodel.eval()\n","metadata":{"execution":{"iopub.status.busy":"2023-01-09T19:09:58.394384Z","iopub.execute_input":"2023-01-09T19:09:58.394735Z","iopub.status.idle":"2023-01-09T19:09:59.496441Z","shell.execute_reply.started":"2023-01-09T19:09:58.394703Z","shell.execute_reply":"2023-01-09T19:09:59.495367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = RSNADataset(df, is_train=False)\ntest_loader = DataLoader(test, batch_size=4, \n                                shuffle=False, num_workers=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = np.empty((0,  1))\nwith torch.no_grad():\n    for k, data in enumerate(test_loader):\n        image, meta = data.values()\n        pred = model(image.to(DEVICE), meta.to(DEVICE))\n        pred = pred.sigmoid()\n        preds = np.concatenate([preds, pred.cpu().numpy()])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\ndf['cancer'] = preds[:,0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['prediction_id'] = df['patient_id'].astype(str) + \"_\" + df['laterality']\n\nsub = df[['prediction_id', 'cancer']].groupby(\"prediction_id\").mean().reset_index()\n\nsub.to_csv('/kaggle/working/submission.csv', index=False)\n\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission = pd.DataFrame(data={'prediction_id': test_data['prediction_id'], 'cancer': np.random.rand(test_data.shape[0])}).drop_duplicates(subset='prediction_id')\n# submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **References:**","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}