{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Prostate cANcer graDe Assessment (PANDA) Challenge\n\nThis notebook will be used by the CIS 6115 students for Unit 5. \n\nStart by reading about the [Prostate cANcer graDe Assessment (PANDA) Challenge](https://www.kaggle.com/c/prostate-cancer-grade-assessment) in Kaggle. It is often good to read the top voted notebooks for the challenge also, which is Rohit Singh's [PANDA - EDA + Better Visualization+Simple Baseline](https://www.kaggle.com/rohitsingh9990/panda-eda-better-visualization-simple-baseline). This notebook is actually based on [PANDA DenseNet Keras Starter GPU](https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-gpu) by Matt Yates\n\n## Chapter 10 & 11 \n\n- Chapter 10: Introduction to Artificial Neural Networks with Keras - Part 2\n- Chapter 11: Training Deep Neural Networks\n\n\n\n\n## Writeup\n\nUse the the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing) to answer the questions posed in this notebook****\n\n## Walkthrough Video\n​\nWatch the [Unit 5 Walkthough Video by Tom](https://youtu.be/A5Oh95Yfw1E)\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Setting up Python tools\n\nWe'll use three libraries for this tutorial: \n- [pandas](http://pandas.pydata.org/) : dataframes for spreadsheet-like data analysis, reading CSV files, time series\n- [numpy](http://www.numpy.org/) : for multidimensional data and linear algebra tools\n- [matplotlib](http://matplotlib.org/) : Simple plotting and graphing\n- [seaborn](http://stanford.edu/~mwaskom/software/seaborn/) : more advanced graphing\n- [scikit-learn](https://scikit-learn.org/stable/) : provides many machine learning algorithms and tools to training and test.\n- [tensorflow](https://www.tensorflow.org/) : Google's backend library for neural networks and other machine learning\n- [keras](https://keras.io/) : High level machine learning API that we run on top of Tensorflow\n\n\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# First, we'll import pandas and numpy, two data processing libraries\nimport pandas as pd\nimport numpy as np\n\n# We'll also import seaborn and matplot, twp Python graphing libraries\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\n\n# Import the needed sklearn libraries\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler\nfrom sklearn import datasets\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import LabelEncoder\n\n# The Keras library provides support for neural networks and deep learning\n# Use the updated Keras library from Tensorflow -- provides support for neural networks and deep learning\nimport tensorflow as tf\nimport tensorflow.keras as keras\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Input, Dense, Dropout, Activation, Lambda, Flatten, LSTM, concatenate\nfrom tensorflow.keras.layers import Conv2D, Convolution2D, MaxPooling2D, Flatten, AveragePooling2D, GlobalAveragePooling2D\nfrom tensorflow.keras.applications import DenseNet121\n\nfrom tensorflow.keras.optimizers import Adam, RMSprop\nfrom tensorflow.keras.utils import to_categorical\n#from keras.utils import np_utils\nAUTO = tf.data.experimental.AUTOTUNE          # Needed for tensorflow image operations\n\nimport os\nimport PIL\nfrom PIL import Image\n# There are two ways to load the data from the PANDA dataset:\n# Option 1: Load images using openslide\nimport openslide\n# Option 2: Load images using skimage (requires that tifffile is installed)\nimport skimage.io\n\n# We will turn off some warns in this notebook to make it easier to read for new students\nimport warnings\nwarnings.filterwarnings('ignore')\n\nprint (\"Libraries imported\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setting up the GPU or TPU\n\nFor an good overview of GPUs and TPUs in machine learning, read [Central Processing Unit (CPU) vs Graphics Processing Unit (GPU) vs Tensor Processing Unit (TPU](https://iq.opengenus.org/cpu-vs-gpu-vs-tpu/)) from opengenus.org\n\nTo enable the GPU, select it as the accelerator from the setting on the right-side panel.\n\n## Task 1: What is a GPU and a TPU\n\nAnswer the following questions in the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing).\n\n### Question 1.1: GPU vs TPU\n\nIn your own words, describe how a GPU is different from a CPU. Then describe what a TPU is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Variable Constants","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 36 * strategy.num_replicas_in_sync\nEPOCHS = 20\n# Data access\n# GCS_DS_PATH = KaggleDatasets().get_gcs_path()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Target & ID Loading","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/prostate-cancer-grade-assessment/train.csv')\ntest_df = pd.read_csv('../input/prostate-cancer-grade-assessment/test.csv')\nprint(\"Size of training data : \", train_df.shape)\nprint(\"Size of testing data : \",test_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Loading & Pre-processing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Note: This is where the size of the images is set. Default is 224x224\n\ndef preprocess_image(image_path, desired_size=224):\n    biopsy = openslide.OpenSlide(image_path)\n    im = np.array(biopsy.get_thumbnail(size=(desired_size,desired_size)))\n    im = Image.fromarray(im)\n    im = im.resize((desired_size,desired_size)) \n    im = np.array(im) / 255\n    return im","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set number of images to process\n\nThe code below loads in the images. Since this can take many minutes, the initial code only reads in 500 of the 10,000+ image available. You should comment out the line \n> N = 500\n\nand uncomment the line \n> N = train_df.shape[0]  \n\nto use all the images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# This could take some time, so show the progress time\n\n\n# get the number of training images from the target\\id dataset\nN = 2000                     # ===== Use only 500 images to test the model\n#N = train_df.shape[0]     # ===== This to run on all data\nprint (\"Running with a sample of the images with N = \", N)\n\n# create an empty matrix for storing the images\nx_train = np.empty((N, 224, 224, 3), dtype=np.float32)\n# loop through the images from the images ids from the target\\id dataset\n# then grab the cooresponding image from disk, pre-process, and store in matrix in memory\nfor i, image_id in enumerate(tqdm(train_df['image_id'])):\n    x_train[i, :, :, :] = preprocess_image(\n        f'../input/prostate-cancer-grade-assessment/train_images/{image_id}.tiff'\n    )\n    # if sampling\n    if i >= N-1:\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pre-processing the target (i.e. one-hot encoding the target)\ny_train = pd.get_dummies(train_df['isup_grade']).values.astype(np.int32)[0:N]\n\nprint(x_train.shape)\nprint(y_train.shape)\nprint(\"Size of training input : \", x_train.shape)\nprint(\"Size of training output : \",y_train.shape)\nif os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n    print(\"Size of test input : \",x_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train & Validation Split","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_val, y_train, y_val = train_test_split(\n    x_train, \n    y_train,\n    test_size=0.20, \n)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Image Augmentation Generator\n\nWe discuss options for image augmentation in this week's mini-project.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Modify data_augment to change how training images are adjusted during training\n# See tf.image for documentation -- https://www.tensorflow.org/api_docs/python/tf/image\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    \n    image = tf.image.convert_image_dtype(image, tf.float32)             # Cast  \n#    image = tf.image.random_flip_left_right(image)\n#    image = tf.image.random_flip_up_down(image)\n#    image = tf.image.random_brightness(image, 0.1)\n#    image = tf.image.random_contrast(image, 0.9, 1.0)\n#    image = tf.image.random_hue(image, 0.1)\n#    image = tf.image.random_contrast(image, 0.1)\n#    image = tf.image.random_saturation(image, 0.9, 1.0)\n#    image = tf.image.random_jpeg_quality(image, 85, 100)\n\n    #width = INPUT_SHAPE[0]\n    # These next four lines will randomly crop the images\n    #large_width = math.floor(width * 1.2)                                        # increase images sizes by 10% before random crop\n    #print (\"image width = \", width, \" resized to \", large_width)\n    #image = tf.image.resize(image, [large_width, large_width])\n    #image = tf.image.random_crop(image, [width, width, 3])\n    \n    #image = tf.image.random_saturation(image, 0, 2)\n    #image = tf.clip_by_value(image, clip_value_min=0., clip_value_max=1.)      # Change the values from 0-255 to 0.0-1.0\n    return image, label   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()                                  # the training dataset must repeat for several epochs\n#    .shuffle(2048)                            # put images in random order\n    .map(data_augment, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_val, y_val))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Model\n\n## Reusing Pretrained Layers\nFrom *Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems*, 2nd Edition by Aurélien Géron, ISBN-13: 978-1492032649 \n\n> It is generally not a good idea to train a very large DNN from scratch: instead, you\nshould always try to find an existing neural network that accomplishes a similar task\nto the one you are trying to tackle,... \nthen reuse the lower layers of this network. This technique is called transfer learning. (page 345)\n\n## Task 2: Using a Pretrained Model\n\nAnswer the following questions in the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing).\n\n### Question 1.2: Using Pretrained Models\n\nReview the four methods below that each create a NN model. Describe which use pre-trained models. Is just the network structure used or are pretrained weights loaded also? Finally, what does the following line determine:\n> pretrained_model.trainable = True ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT_SHAPE = (224,224,3)\nOUTPUT_SHAPE = 6\n\ndef build_model_custom():\n\n    # Set up the Neural Network\n\n    NN = Sequential()\n    NN.add(Conv2D(8, kernel_size=(3, 3), activation='relu', input_shape=INPUT_SHAPE))\n    NN.add(Conv2D(8, (3, 3), activation='relu'))\n    NN.add(MaxPooling2D(pool_size=(2, 2)))\n    NN.add(Conv2D(16, (3, 3), activation='relu'))\n    NN.add(Conv2D(16, (3, 3), activation='relu'))\n    NN.add(MaxPooling2D(pool_size=(2, 2)))\n    NN.add(Conv2D(32, (3, 3), activation='relu'))\n    NN.add(MaxPooling2D(pool_size=(2, 2)))\n    NN.add(Conv2D(64, (3, 3), activation='relu'))\n    NN.add(MaxPooling2D(pool_size=(2, 2)))\n    NN.add(Flatten())\n    NN.add(Dense(256, activation='relu'))\n    NN.add(Dense(OUTPUT_SHAPE, activation='softmax'))\n    #NN.add(Dense(OUTPUT_SHAPE, activation='sigmoid'))\n\n    print (\"Neural Network Model created\")\n    # NN.summary()\n    return NN\n \ndef build_model_VGG(weight_setting='imagenet'):\n    # Set up the Neural Network   \n\n    # ==== VGG === works with 224x224 size images\n    #pretrained_model = tf.keras.applications.VGG16(weights=None, include_top=False ,input_shape=INPUT_SHAPE)         # Using random initial weights\n    pretrained_model = tf.keras.applications.VGG16(weights=weight_setting, include_top=False ,input_shape=INPUT_SHAPE)   # Using pretrained weights from Imagenet\n\n    # Set the model so that all the weights are trainable with the new whale images\n    pretrained_model.trainable = True      # False = transfer learning, True = fine-tuning\n\n    # Here the sample from page 461 of the textbook\n    model = Sequential([\n        pretrained_model,                                 # Include layers in pretrained model from above\n        GlobalAveragePooling2D(),\n        #Dense(1024, activation=\"relu\"),     # Can add optional additional layers here\n        #Dense(200, activation=\"relu\"),      # Can add optional additional layers here\n        Dense(OUTPUT_SHAPE, activation='softmax')\n        #Dense(OUTPUT_SHAPE, activation='sigmoid')\n    ])\n    print (\"Neural Network Model created\")\n    print (\"=== Pretrained Model =========================================================================\")\n    pretrained_model.summary()   # print layers in pretrained model\n    print (\"=== Final Model =========================================================================\")\n    model.summary()              # print final model\n    return model\n\ndef build_model_Xception(weight_setting='imagenet'):    \n    # Set up the Neural Network   \n\n    # ==== Xception === \n    # by default Xception expects images of size 299x299 pixels\n    #pretrained_model = tf.keras.applications.Xception(weights=None, include_top=False ,input_shape=INPUT_SHAPE)            # Using random initial weights\n    pretrained_model = tf.keras.applications.Xception(weights=weight_setting, include_top=False ,input_shape=INPUT_SHAPE)      # Using pretrained weights from Imagenet\n\n    # Set the model so that all the weights are trainable with the new whale images\n    pretrained_model.trainable = True      # False = transfer learning, True = fine-tuning\n\n    # Here the sample from page 461 of the textbook\n    model = Sequential([\n        pretrained_model,                                 # Include layers in pretrained model from above\n        GlobalAveragePooling2D(),\n        #Dense(1024, activation=\"relu\"),     # Can add optional additional layers here\n        #Dense(200, activation=\"relu\"),      # Can add optional additional layers here\n        Dense(OUTPUT_SHAPE, activation='softmax')\n        #Dense(OUTPUT_SHAPE, activation='sigmoid')\n    ])\n    print (\"Neural Network Model created\")\n    print (\"=== Pretrained Model =========================================================================\")\n    pretrained_model.summary()   # print layers in pretrained model\n    print (\"=== Final Model =========================================================================\")\n    model.summary()              # print final model\n    return model\n\ndef build_model_DenseNet():\n    densenet = DenseNet121(\n    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',\n    include_top=False,\n    input_shape=(224,224,3)\n    )\n    model = Sequential()\n    model.add(densenet)\n    model.add(GlobalAveragePooling2D())\n    model.add(Dropout(0.50))\n    #model.add(Dense(6, activation='sigmoid'))\n    model.add(Dense(6, activation='softmax'))\n\n    print (\"Neural Network Model created\")\n    return model\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Complex Models Using the Functional API\n\nThe following inception model uses the functional API to define the model and is another illustration from the example shown in the textbook. This is the layout of a typical inception module--note that the connections are not sequential\n![](https://www.researchgate.net/profile/Bo_Zhao48/publication/312515254/figure/fig3/AS:489373281067012@1493687090916/nception-module-of-GoogLeNet-This-figure-is-from-the-original-paper-10.png)\n\nFrom *Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems*, 2nd Edition by Aurélien Géron, ISBN-13: 978-1492032649 \n\n> Once you have built the Keras model, everything is exactly like earlier, so there’s no\nneed to repeat it here: you must compile the model, train it, evaluate it, and use it to\nmake predictions. (page 310)\n\n## Task 3: Keras Functional API\n\nAnswer the following questions in the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing).\n\n### Question 1.3: Functional API \n\nThe code below uses the functional API to build a network. Describe how the Functional API is different from the Sequential API. What are some times when the Functional API is needed over the Sequential API?\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model_Inception(): \n    input_img = Input(shape = INPUT_SHAPE)\n\n    pre_1 = Conv2D(64, (3,3), strides=(2,2), activation='relu', name='pre_1')(input_img)\n    pre_2 = MaxPooling2D(pool_size=(3,3), strides=(2,2), name='pre_2')(pre_1)\n\n    #First Inception module\n    size = 32\n    incep_1_1x1_a = Conv2D(64, (1,1), padding='same', activation='relu', name='incep_1_1x1_a')(pre_2)\n    incep_1_1x1_b = Conv2D(96, (1,1), padding='same', activation='relu', name='incep_1_1x1_b')(pre_2)\n    incep_1_3x3_b = Conv2D(128, (3,3), strides=(1,1), padding='same', activation='relu', name='incep_1_3x3_b')(incep_1_1x1_b)\n    incep_1_1x1_c = Conv2D(16, (1,1), padding='same', activation='relu', name='incep_1_1x1_c')(pre_2)\n    incep_1_5x5_c = Convolution2D(32, (5,5), strides=(1,1), padding='same', activation='relu', name='incep_1_5x5_c')(incep_1_1x1_c)\n    incep_1_pool_d = MaxPooling2D(pool_size=(3,3), strides=(1,1), padding='same', name='incep_1_pool_d')(pre_2)\n    incep_1_1x1_d = Convolution2D(32, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_1_1x1_d')(incep_1_pool_d)\n\n    incep_1_output = concatenate([incep_1_1x1_a, incep_1_3x3_b, incep_1_5x5_c, incep_1_1x1_d], axis = 3, name='incep_1_output')\n\n\n    #Second Inception module\n    #size = 64\n    incep_2_1x1_a = Conv2D(128, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_2_1x1_a')(incep_1_output)\n    incep_2_1x1_b = Conv2D(128, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_2_1x1_b')(incep_1_output)\n    incep_2_3x3_b = Conv2D(192, (3,3), strides=(1,1), padding='same', activation='relu', name='incep_2_3x3_b')(incep_2_1x1_b)\n    incep_2_1x1_c = Conv2D(32, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_2_1x1_c')(incep_1_output)\n    incep_2_5x5_c = Convolution2D(96, (5,5), strides=(1,1), padding='same', activation='relu', name='incep_2_5x5_c')(incep_2_1x1_c)\n    incep_2_pool_d = MaxPooling2D(pool_size=(3,3), strides=(1,1), padding='same', name='incep_2_pool_d')(incep_1_output)\n    incep_2_1x1_d = Convolution2D(64, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_2_1x1_d')(incep_2_pool_d)\n\n    incep_2_output = concatenate([incep_2_1x1_a, incep_2_3x3_b, incep_2_5x5_c, incep_2_1x1_d], axis = 3, name='incep_2_output')\n    pool_2 = MaxPooling2D(pool_size=(3,3), strides=(2,2), padding=\"same\", name='pool_2')(incep_2_output)\n\n\n    #Third Inception module\n    #size = 64\n    incep_3_1x1_a = Conv2D(192, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_3_1x1_a')(pool_2)\n    incep_3_1x1_b = Conv2D(96, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_3_1x1_b')(pool_2)\n    incep_3_3x3_b = Conv2D(208, (3,3), strides=(1,1), padding='same', activation='relu', name='incep_3_3x3_b')(incep_3_1x1_b)\n    incep_3_1x1_c = Conv2D(16, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_3_1x1_c')(pool_2)\n    incep_3_5x5_c = Convolution2D(48, (5,5), strides=(1,1), padding='same', activation='relu', name='incep_3_5x5_c')(incep_3_1x1_c)\n    incep_3_pool_d = MaxPooling2D(pool_size=(3,3), strides=(1,1), padding=\"same\", name='incep_3_pool_d')(pool_2)\n    incep_3_1x1_d = Convolution2D(64, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_3_1x1_d')(incep_3_pool_d)\n\n    incep_3_output = concatenate([incep_3_1x1_a, incep_3_3x3_b, incep_3_5x5_c, incep_3_1x1_d], axis = 3, name='incep_3_output')\n\n\n    #Fourth Inception module\n    #size = 128\n    incep_4_1x1_a = Conv2D(160, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_4_1x1_a')(incep_3_output)\n    incep_4_1x1_b = Conv2D(112, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_4_1x1_b')(incep_3_output)\n    incep_4_3x3_b = Conv2D(224, (3,3), strides=(1,1), padding='same', activation='relu', name='incep_4_3x3_b')(incep_4_1x1_b)\n    incep_4_1x1_c = Conv2D(24, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_4_1x1_c')(incep_3_output)\n    incep_4_5x5_c = Convolution2D(64, (5,5), strides=(1,1), padding='same', activation='relu', name='incep_4_5x5_c')(incep_4_1x1_c)\n    incep_4_pool_d = MaxPooling2D(pool_size=(3,3), strides=(1,1), padding=\"same\", name='incep_4_pool_d')(incep_3_output)\n    incep_4_1x1_d = Convolution2D(64, (1,1), strides=(1,1), padding='same', activation='relu', name='incep_4_1x1_d')(incep_4_pool_d)\n\n    incep_4_output = concatenate([incep_4_1x1_a, incep_4_3x3_b, incep_4_5x5_c, incep_4_1x1_d], axis = 3, name='incep_4_output')\n    pool_3 = MaxPooling2D(pool_size=(3,3), strides=(2,2), name='pool_5')(incep_3_output)\n\n    loss3_flat = AveragePooling2D(pool_size=(7,7), strides=(1,1))(pool_3)\n    loss4_flat = Flatten()(loss3_flat)\n\n    loss3_classifier_0 = Dense(1024, name='loss3_classifier_0', activation='relu')(loss4_flat)\n    loss3_classifier_act = Dense(200, name='loss3_classifier_act', activation=\"relu\")(loss3_classifier_0)\n    final_output = Dense(6, name='final_output', activation='sigmoid')(loss3_classifier_act)\n    model = Model(inputs = input_img, outputs = final_output)\n    print (\"Neural Network Model created\")\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning Optimizers\n##  [RMSProp](https://keras.io/api/optimizers/rmsprop/)\n\n> As we’ve seen, AdaGrad runs the risk of slowing down a bit too fast and never con‐\nverging to the global optimum. The RMSProp algorithm16 fixes this by accumulating\nonly the gradients from the most recent iterations (page 355)\n\n## [Adam](https://keras.io/api/optimizers/adam/)\n> Adam which stands for adaptive moment estimation, combines the ideas of momen‐\ntum optimization and RMSProp: just like momentum optimization, it keeps track of\nan exponentially decaying average of past gradients; and just like RMSProp, it keeps\ntrack of an exponentially decaying average of past squared gradients (page 356)\n\n## Task 4: Optimizers\n\nAnswer the following questions in the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing).\n\n### Question 1.4: Functional API \n\nRead about RMSprop and Adam learning optimizers and how the adjust the learning weights as we train a network. Then select one of these to use below. Describe why you selected the one you did.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== Optimizer === We will study these options in a future unit. For not, just leave as RMSprop\n# Some sample weight optimizer settings\n#RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\noptimizer_RMSprop = RMSprop(lr=0.00001, epsilon=1e-08)\n#optimizer_Adam = Adam(learning_rate=0.001) # default learning rate\noptimizer_Adam = Adam(learning_rate=0.0001)\n\nwith strategy.scope():\n    #model = build_model_DenseNet()\n    #model = build_model_Inception()\n    model = build_model_Xception(None)\n    #model = build_model_Xception('imagenet')\n\n    \n    model.compile(\n        # loss='binary_crossentropy',\n        loss='categorical_crossentropy',\n        optimizer=optimizer_Adam,\n        metrics=['accuracy']\n    )\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Callbacks\n\nFrom Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, 2nd Edition by Aurélien Géron, ISBN-13: 978-1492032649\n\n> The fit() method accepts a callbacks argument that lets you specify a list of objects\nthat Keras will call at the start and end of training, at the start and end of each epoch,\nand even before and after processing each batch. For example, the ModelCheckpoint\ncallback saves checkpoints of your model at regular intervals during training, by\ndefault at the end of each epoch: (page 315)\n\nHere is the documentation of the callbacks:\n\n-  [ReduceLROnPlateau](https://keras.io/callbacks/#reducelronplateau). \n\n-  [EarlyStopping callback](https://keras.io/callbacks/#earlystopping) \n\n-  [ModelCheckpoint callback](https://keras.io/callbacks/#modelcheckpoint) \n\n## Task 5: Callbacks\n\nAnswer the following questions in the [Unit 5 Writeup](https://docs.google.com/document/d/1SytQJv4XEn67uc9c4IVcsJb_neW4F1y4LZAVQh6WBLY/edit?usp=sharing).\n\n### Question 1.5: Callbacks \n\nSelect one of the callbacks below and write a description of what it does and why we use it.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint\n\n# reduce the learning rate by 50%. Make patience smaller to change the rate more often\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_loss', \n                                            patience=5, \n                                            verbose=2, \n                                            factor=0.5,                                            \n                                            min_lr=0.000001)\n\n# Stop training if not improvement after a while. Make patience smaller to have model stop faster\nearly_stops = EarlyStopping(monitor='val_loss', \n                            min_delta=0, \n                            patience=20, \n                            verbose=2, \n                            mode='auto')\nimport datetime\n\n# Save the model. Saves only the best results and saves only the wieghts. \n# File name will include the epoch number and the validation accuracy\ncheckpointer = ModelCheckpoint(filepath = 'cis6115_PANDA.{epoch:02d}-{accuracy:.6f}.hdf5',\n                               verbose=2,\n                               save_best_only=True, \n                               save_weights_only = True)\n\n# Set up Tensorboard to monitor the training progress\nlogdir = os.path.join(\"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\ntensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using TensorBoard for Visualization\nFrom Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, 2nd Edition by Aurélien Géron, ISBN-13: 978-1492032649\n\n> TensorBoard is a great interactive visualization tool that you can use to view the\nlearning curves during training, compare learning curves between multiple runs, vis‐\nualize the computation graph, analyze training statistics, view images generated by\nyour model, visualize complex multidimensional data projected down to 3D and\nautomatically clustered for you, and more! This tool is installed automatically when\nyou install TensorFlow, so you already have it. (page 317)\n\nI don't know if tensorboard will work in the Kaggle environment, so I have commented this code out for now. It should work in Colab.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#  %%tensorboard --logdir logs/scalars\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Model\n\nInitially the epochs are only set to 20 to keep training time down. Later consider increasing this, particularly if you are using the early_stops callback which will stop training once it plateaus. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = x_train.shape[0] // BATCH_SIZE\n\nhistory = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    #callbacks=[learning_rate_reduction, early_stops, tensorboard_callback],\n    callbacks=[learning_rate_reduction, early_stops],\n    validation_data=valid_dataset,\n    #epochs=EPOCHS\n    epochs=20\n\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Saving and Restoring a Model\n\nFrom Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, 2nd Edition by Aurélien Géron, ISBN-13: 978-1492032649\n\n> Keras will use the HDF5 format to save both the model’s architecture (including every\nlayer’s hyperparameters) and the values of all the model parameters for every layer\n(e.g., connection weights and biases). It also saves the optimizer (including its hyper‐\nparameters and any state it may have). (page 314)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code sample to save the model, including the weights, or just the weights\n# model.save(\"CIS6115_model.h5\")            # save the model structure\n# model.save_weights('CIS6115_weights.h5')  # always save your weights after training or during training\n\n#Code sample to load the saved model or just the model weights\n# ==== Generally this code should be at the beginning of the notebook once the model is define but before it is trained.\n# model = keras.models.load_model(\"CIS6115_model.h5\")\n# model.load_weights('CIS6115_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot the Training History\nWe store the performance during training is a variable named 'history'. The x-axis is the training time or number of epochs.\n\n- Accuracy: Accuracy of the predictions, hopefully this is increasing to near 1.0\n- Loss: How close the output is to the desired output, this should decrease to near 0.0","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# We will display the loss and the accuracy of the model for each epoch\n# NOTE: this is a little fancy display than is shown in the textbook\ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])\n\n# We store the performance during training in a variable named 'history'. The x-axis is the training time or number of epochs.\n#    Accuracy: Accuracy of the predictions; hopefully this is increasing to near 1.0\n#    Loss: How close the output is to the desired output; this should decrease to near 0.0\ndisplay_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history.history['accuracy'], history.history['val_accuracy'], 'accuracy', 212)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission\n\nKaggle does not provide sample test images for running the notebook. These are only available once you submit your notebook for scoring. So, the code below checks if the test_images are available. If they are, we know we are doing an official submission, otherwise we fake it.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loop through the images from the images ids from the target\\id dataset\n# then grab the cooresponding image from disk, pre-process, and store in matrix in memory\nif os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n    print (\"Test images found\")\n    # do the same thing as the last cell but on the test\\holdout set\n    N = test_df.shape[0]\n    x_test = np.empty((N, 224, 224, 3), dtype=np.float32)\n    for i, image_id in enumerate(tqdm(test_df['image_id'])):\n        x_test[i, :, :, :] = preprocess_image(\n            f'../input/prostate-cancer-grade-assessment/test_images/{image_id}.tiff'\n        )\n        \n    test_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices(x_test)\n        .batch(BATCH_SIZE)\n    )\nelse: \n    print (\"===== Problem ==== No test images found\")\n\n    \nsample_test_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_train)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Debug code\nThis is just some code I was using to debug the submission process. It should be removed enventually.\n\ny_sample = model.predict(sample_test_dataset)\n\nprint (y_sample)\nprint (type(y_sample))\ny_max = np.argmax(y_sample, axis=1)     # Select index with the highest probability for each test image\nprint (y_max)\nprint (train_df['isup_grade'][0:50])","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\n\nif os.path.exists('../input/prostate-cancer-grade-assessment/test_images'):\n    print (\"found test data\")\n    y_test = model.predict(test_dataset)\n    results = np.argmax(y_test, axis=1)     # Select index with the highest probability for each test image\n    test_df['isup_grade'] = results\n    test_df['isup_grade'] = test_df['isup_grade'].astype(int)\n    test_df.to_csv('submission.csv', index=False)\nelse: # if test is not available, just submit some random values\n    print (\"No test data found, using some random values\")  \n    random.seed(42)\n    submission = pd.read_csv('../input/prostate-cancer-grade-assessment/sample_submission.csv')\n    results = np.random.randint(0,6,len(submission))\n    submission['isup_grade'] = results\n    submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# No mini-project this week so you can start the final project\n\n## Mini Project: Improve this notebook\n\n## Mini-Project Writeup 1: Prostrate Cancer Challenge Summary\nRead about the [Prostate cANcer graDe Assessment (PANDA) Challenge](https://www.kaggle.com/c/prostate-cancer-grade-assessment) in Kaggle.\n\nWrite a short summary of the challenge:\n\n- What is the goal of the challenge?\n- What types of images are provided?\n- What should the model predict? What is an ISUP grade?\n\n*Make sure you put this in your own words--do not just cut/paste from Kaggle*\n\n## Mini-Project Writeup 2: Image Augmentation\n\nIn the code above, the `data_augment(image, label)` method can be used to modify the images during training. This uses the tensorflow methods such as \n> image = tf.image.random_flip_up_down(image)\n\nSee the tf.image for documentation -- https://www.tensorflow.org/api_docs/python/tf/image\n\n\n\nImplement some sort of image augmentation in your model by adjusting the `data_augment(image, label)` method. Train the network with this method and describe your results.\n\n*Feel free to post your results to the course discussion area.*\n\nThen review the first portion of Iafoss's notebook,[PANDA concat tile pooling starter](https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb) and describe the image augmenation he suggests. \n\n# Mini-Project Writeup 3: Modify the Model \n\nTry at least one other adjustment to this notebook. This could include changing the network structure, learning epochs, optimizer, etc.\n\nWrite a paragraph summarizing your results and analyze why you think they turned out the way they did.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}