{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":false,"collapsed":true},"cell_type":"code","source":"import numpy as np\n# Uninstalling tensorflow 2.0 and downgrading to tensorflow 1.14.0 as tf.log works\n# with tf version 1.14. For version 2.0 use tf.math.log.\n!pip install 'tensorflow==1.14.0'\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### What is Focal Loss? \n\nTraining over large class imbalance data encounters overwhelming cross entropy loss. Easily classified negatives comprise the majority of loss and dominate the gradient. Focal loss propose to reshape loss function to down weight the easy examples and thus focus training on hard negatives. Details about focal loss can be found in below paper:\nhttps://arxiv.org/abs/1708.02002\n\n### Original source\nSource of original code for Multilabel Focal Loss Function is at below link. It is written using tensorflow.\nhttps://www.kaggle.com/allunia/rsna-ih-detection-baseline\n\n\n[@Allunia](https://www.kaggle.com/allunia) Thank you for wonderful kernel. I came to know about multilabel focal loss via your kernel only.\n\n**I rewrote the function using numpy and found it was quite fast. So, sharing with the community. I hope it helps.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def multiclass_focal_log_loss(y_true, y_pred, class_weights = None, alpha = 0.5, gamma = 2):\n    \"\"\"\n    Numpy version of the Focal Loss\n    \"\"\"\n    # epsilon \n    eps = 1e-12\n    # If actual value is true, keep pt value as y_pred otherwise (1-y_pred)\n    pt = np.where(y_true == 1, y_pred, 1-y_pred)\n    # If actual value is true, keep alpha_t value as alpha otherwise (1-alpha)\n    alpha_t = np.where(y_true == 1, alpha, 1-alpha)\n    # Clip values below epsilon and above 1-epsilon\n    pt = np.clip(pt, eps, 1-eps)\n    # FL = -alpha_t(1-pt)^gamma log(pt)\n    focal_loss = -np.mean(np.multiply(np.multiply(alpha_t,np.power(1-pt,gamma)),np.log(pt)), axis=0)\n    if class_weights is None:\n        focal_loss = np.mean(focal_loss)\n    else:\n        focal_loss = np.sum(np.multiply(focal_loss, class_weights))\n    print(focal_loss)\n\n\ndef get_raw_xentropies(y_true, y_pred):\n    y_pred = tf.clip_by_value(y_pred, 1e-7, 1-1e-7)\n    xentropies = y_true * tf.log(y_pred) + (1-y_true) * tf.log(1-y_pred)\n    return -xentropies\n\n# multilabel focal loss equals multilabel loss in case of alpha=0.5 and gamma=0 \ndef mutlilabel_focal_loss_inner(y_true, y_pred,class_weights=None, alpha=0.5, gamma=2):\n    \"\"\"\n    Tensorflow version of the Focal Loss\n    \"\"\"\n    y_true = tf.cast(y_true, tf.float32)\n    y_pred = tf.cast(y_pred, tf.float32)\n\n    xentropies = get_raw_xentropies(y_true, y_pred)\n\n    # compute pred_t:\n    y_t = tf.where(tf.equal(y_true,1), y_pred, 1.-y_pred)\n    alpha_t = tf.where(tf.equal(y_true, 1), alpha * tf.ones_like(y_true), (1-alpha) * tf.ones_like(y_true))\n\n    # compute focal loss contributions\n    focal_loss_contributions =  tf.multiply(tf.multiply(tf.pow(1-y_t, gamma), xentropies), alpha_t) \n\n    # our focal loss contributions have shape (n_samples, s_classes), we need to reduce with mean over samples:\n    focal_loss_per_class = tf.reduce_mean(focal_loss_contributions, axis=0)\n\n    # compute the overall loss if class weights are None (equally weighted):\n    if class_weights is None:\n        focal_loss_result = tf.reduce_mean(focal_loss_per_class)\n    else:\n        # weight the single class losses and compute the overall loss\n        weights = tf.constant(class_weights, dtype=tf.float32)\n        focal_loss_result = tf.reduce_sum(tf.multiply(weights, focal_loss_per_class))\n    with tf.Session() as sess:\n        print(focal_loss_result.eval())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dummy matrix to test new function\ny_true = np.array([[0,0,0,0,1],[0,0,0,1,0],[0,0,0,0,1]])\ny_pred = np.array([[0.22,0.13,0.12,0.90,0.32],[0.11,0.33,0.32,0.45,0.89],[0.32,0.22,0.11,0.16,0.97]])\nclass_weight = [.5,.15,.15,.1,.1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Numpy version\nmulticlass_focal_log_loss(y_true, y_pred, class_weights=class_weight)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Tensorflow version\nmutlilabel_focal_loss_inner(y_true, y_pred, class_weights=class_weight)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Numpy version returns result in micro seconds while tensorflow version returns in milli seconds."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}