{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this Notebook\n\nThis is a quick start introduction to trainig notebooks using TPU with TensorFlow for this competition using EfficientNet architecture and Densenet. Please consume this with moderation (only 30h per week!).\nRunning this notebooks cost you about 4 dollars.\n\n\n#### **This notebook is build on top of this [kernel](https://www.kaggle.com/yeayates21/panda-densenet-keras-starter-tpu/) by [yeayates21](https://www.kaggle.com/yeayates21/)**\n\n### <font color='green'>If you find this kernel helpful please consider upvoting 😊. Also dont forget to upvote the original kernel.</font>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### What are TPU's? <a class=\"tpu\" id=\"prepare\"></a>\n\n\n\nTPU's are holy grail of computers for any Machine Learning Practitioners! A tensor processing unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google specifically for neural network machine learning. \nTPUs are hardware accelerators specialized in deep learning tasks. In this code lab, you will see how to use them with Keras and Tensorflow 2. Cloud TPUs are available in a base configuration with 8 cores and also in larger configurations called \"TPU pods\" of up to 2048 cores. The extra hardware can be used to accelerate training by increasing the training batch size.\n\n\n### Why TPUs?\n\nModern GPUs are organized around programmable \"cores\", a very flexible architecture that allows them to handle a variety of tasks such as 3D rendering, deep learning, physical simulations, etc.. TPUs on the other hand pair a classic vector processor with a dedicated matrix multiply unit and excel at any task where large matrix multiplications dominate, such as neural networks.\n\n<html>\n<body>\n\n<p><font size=\"4\" color=\"red\"> The following video from Kaggle explains the main components of TPU like systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times </font></p>\n</body>\n</html>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo(\"JC84GCU7zqA\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import efficientnet.tfkeras as efn\nimport gc\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport openslide\nimport os\nimport tensorflow as tf\n\n\n\nfrom random import randint\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.optimizers import Adam\nfrom sklearn.model_selection import train_test_split\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score, confusion_matrix\nfrom tqdm import tqdm\n%matplotlib inline\n\nprint(tf.__version__)\nprint(tf.keras.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('panda-resized-train-data-512x512')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Looking at the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/prostate-cancer-grade-assessment/train.csv')\nprint(train_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training dataset has 4 columns ie:\n\n- image_id\n- data_provider: the lab in which data was annotated ie (karolinska or radbound)\n- isup_grade: \n- gleason_score: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(plt.imread('/kaggle/input/panda-resized-train-data-512x512/train_images/train_images/'\n                      + train_df.iloc[0]['image_id'] + '.png'))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# from https://www.kaggle.com/ateplyuk/panda-tpu-starter-train/\nmsk = np.random.rand(len(train_df)) < 0.85\ntrain = train_df[msk]\nvalid = train_df[~msk]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape) \nprint(valid.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train_paths = train[\"image_id\"].apply(lambda x: GCS_DS_PATH + '/train_images/train_images/' + x + '.png').values\nvalid_paths = valid[\"image_id\"].apply(lambda x: GCS_DS_PATH + '/train_images/train_images/' + x + '.png').values","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train_labels = pd.get_dummies(train['isup_grade']).astype('int32').values\nvalid_labels = pd.get_dummies(valid['isup_grade']).astype('int32').values\n\nprint(train_labels.shape) \nprint(valid_labels.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configurations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE= 8 * strategy.num_replicas_in_sync\nimg_size = 512\nEPOCHS = 15\nnb_classes = 6","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_START = 0.00001\nLR_MAX = 0.0001 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 1\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(filename, label=None, image_size=(img_size, img_size)):\n    bits = tf.io.read_file(filename)\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, image_size)\n    if label is None:\n        return image\n    else:\n        return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating tf.data objects","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((train_paths, train_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .repeat()\n    .cache()\n    .shuffle(512)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((valid_paths, valid_labels))\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNet Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_emodel():\n    with strategy.scope():\n        en =efn.EfficientNetB3(weights='imagenet', include_top=False, input_shape=(img_size, img_size, 3))\n        en.trainable = True\n\n        model = tf.keras.Sequential([\n            en,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(6, activation='softmax')\n        ])\n        opt = Adam(learning_rate=1e-3)\n        model.compile(optimizer = opt,\n            loss = 'categorical_crossentropy',\n            metrics=['accuracy']\n        )\n        print(model.summary())\n        return model\n    \nmodel = get_emodel()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.listdir('../input/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nCheckpoint=tf.keras.callbacks.ModelCheckpoint(f\"Enet_model.h5\", monitor='val_accuracy', verbose=1, save_best_only=True,\n       save_weights_only=True,mode='max')\nmodel.load_weights('../input/enetb3prostate/Enet_model.h5')\ntrain_history1 = model.fit(\n            train_dataset, \n            validation_data = valid_dataset, \n            steps_per_epoch=train_labels.shape[0] // BATCH_SIZE,            \n            validation_steps=valid_labels.shape[0] // BATCH_SIZE,            \n            callbacks=[lr_callback, Checkpoint],\n            epochs=EPOCHS,\n            verbose=2\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training accuracy and loss plots","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_training(H):\n\t# construct a plot that plots and saves the training history\n\twith plt.xkcd():\n\t\tplt.figure()\n\t\tplt.plot(H.history[\"loss\"], label=\"train_loss\")\n\t\tplt.plot(H.history[\"val_loss\"], label=\"val_loss\")\n\t\tplt.plot(H.history[\"accuracy\"], label=\"train_acc\")\n\t\tplt.plot(H.history[\"val_accuracy\"], label=\"val_acc\")\n\t\tplt.title(\"Training Loss and Accuracy\")\n\t\tplt.xlabel(\"Epoch #\")\n\t\tplt.ylabel(\"Loss/Accuracy\")\n\t\tplt.legend(loc=\"lower left\")\n\t\tplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_training(train_history1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(train_history1.history['accuracy'])\nplt.plot(train_history1.history['val_accuracy'])\nplt.title('Accuracy throug epochs')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='best')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Densenet model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_densenet_model():\n    with strategy.scope():\n        rnet = tf.keras.applications.DenseNet201(\n                input_shape=(img_size, img_size, 3),\n                weights='imagenet',\n                include_top=False\n            )\n\n        model2 = tf.keras.Sequential([\n                rnet,\n                tf.keras.layers.GlobalAveragePooling2D(),\n                tf.keras.layers.Dense(6, activation='softmax')\n            ])\n\n        model2.compile(\n            optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n            loss = 'categorical_crossentropy',\n            metrics=['accuracy']\n        )\n        model2.summary()\n        return model2\n    \nmodel2 = get_densenet_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nCheckpoint=tf.keras.callbacks.ModelCheckpoint(f\"Dnet_basic_model.h5\", monitor='val_accuracy', verbose=1, save_best_only=True,\n       save_weights_only=True,mode='max')\nmodel2.load_weights('../input/enetb3prostate/Dnet_basic_model.h5')\ntrain_history2 = model2.fit(\n            train_dataset, \n            validation_data = valid_dataset, \n            steps_per_epoch=train_labels.shape[0] // BATCH_SIZE,            \n            validation_steps=valid_labels.shape[0] // BATCH_SIZE,            \n            callbacks=[lr_callback, Checkpoint],\n            epochs=EPOCHS,\n            verbose=2\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualising accuracy and training losses","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_training(train_history2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(train_history2.history['accuracy'])\nplt.plot(train_history2.history['val_accuracy'])\nplt.title('Accuracy throug epochs')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='best')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fin..\n\n<p><font size=\"4\" color=\"green\"> If you like this notebook leave a upvote  ⬆️</font></p>\n\n- This notebook will be updated with KFold training, augemntations on TPU and  finetuning models and much more soon ....","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}