{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# **Simple Explanation of Quadratic Weighted Kappa**\n\n\n\n\n\n## **Introduction**\n\n\n\nKaggle has recently launched a competition - Prostate cANcer graDe Assessment (PANDA) Challenge. \nWe can find the competition description [here](https://www.kaggle.com/c/prostate-cancer-grade-assessment/overview) and the cometition evaluation page can be found [here](https://www.kaggle.com/c/prostate-cancer-grade-assessment/overview/evaluation). \n\n\n\nSubmissions in this competition are scored based on the **Quadratic Weighted Kappa (QWK)**. So, in this notebook, I will try to provide an intuitive explanation of **Quadratic Weighted Kappa (QWK)**. I have tried to find out the meaning of Quadratic Weighted Kappa (QWK). I have found few excellent resources which explains QWK. These are -\n\n\n- [CPMP's discussion Fast QWK Computation](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/145105).\n\n- [Ultra Fast QWK Calc Method](https://www.kaggle.com/cpmpml/ultra-fast-qwk-calc-method)\n\n- [qwk: cupy vs numpy vs numba](https://www.kaggle.com/jiweiliu/qwk-cupy-vs-numpy-vs-numba)\n\n- [Understanding the Quadratic Weighted Kappa by reigHns](https://www.kaggle.com/reighns/understanding-the-quadratic-weighted-kappa)\n\n\n- The codes have been adapted from Ben Hamner's github repository : https://github.com/benhamner/Metrics\n\n\nSo, let's get started.\n"},{"metadata":{},"cell_type":"markdown","source":"**I hope you find this notebook useful and your <font color=\"red\"><b>UPVOTES</b></font> would be very much appreciated**\n\n"},{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n# **Table of Contents**\n\n\n1.\t[Introduction to Quadraticc Weighted Kappa (QWK)](#1)\n2.\t[How to calculate QWK](#2)\n3.\t[Implement QWK Calculation](#3)\n4.  [Interpretation of QWK Value](#4)\n5.  [Conclusion](#5)\n"},{"metadata":{},"cell_type":"markdown","source":"# **1. Introduction to Quadratic Weighted Kappa (QWK)** <a class=\"anchor\" id=\"1\"></a>\n[Table of Contents](#0.1)\n\n\n\n- **Quadratic Weighted Kappa (QWK)** is described on the evaluation page, which we can find [here](https://www.kaggle.com/c/prostate-cancer-grade-assessment/overview/evaluation).\n\n- **Quadratic Weighted Kappa (QWK)** measures the agreement between two outcomes. We can interpret QWK as the amount of agreement between an algorithm's predictions and true labels.\n\n- This metric typically varies from 0 (random agreement) to 1 (complete agreement). In the event that there is less agreement than expected by chance, the metric may go below 0 i.e. it may become negative.\n\n- So, we can present it in tabular form as below -\n\n    - -1 : Complete disagreement\n    - 0 : Agreement by chance\n    - 0-0.2 ; Poor agreement\n    - 0.2-0.4 : Moderate agreement\n    - 0.4-0.6 : Good agreement\n    - 0.6-0.8 : Very good agreement\n    - 0.8-1 : Perfect agreement \n    - 1 : Complete agreement   \n    \n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# **2. How to calculate QWK** <a class=\"anchor\" id=\"2\"></a>\n\n[Table of Contents](#0.1)\n\n\nThe QWK calculation is a 4 step process as described in the evaluation page. These steps are explained below-\n\n\n- **Step 1** : First, an N x N histogram matrix O is constructed , such that Oi,j corresponds to the number of predicted labels that have a rating of i (actual) that received a predicted value j.\n\n\n- **Step 2** : An N-by-N matrix of weights, w, is calculated based on the difference between actual and predicted values. This matrix is given as \n\n\n$$w_{i,j} = \\dfrac{(i-j)^2}{(N-1)^2}$$\n\n\n- **Step 3** : An N-by-N histogram matrix of expected ratings, E, is calculated, assuming that there is no correlation between rating scores. This is calculated as the outer product between the actual rating's histogram vector of ratings and the predicted rating's histogram vector of ratings, normalized such that E and O have the same sum.\n\n\n- **Step 4** : From these three matrices, the quadratic weighted kappa is calculated as follows -\n\n\n$$\\kappa = 1 - \\dfrac{\\sum_{i,j}\\text{w}_{i,j}O_{i,j}}{\\sum_{i,j}\\text{w}_{i,j}E_{i,j}}$$\n"},{"metadata":{},"cell_type":"markdown","source":"# **3. Implement QWK Calculation** <a class=\"anchor\" id=\"3\"></a>\n\n[Table of Contents](#0.1)\n\n\n- Now, we will demonstrate the calculation of QWK.\n"},{"metadata":{},"cell_type":"markdown","source":"## **Step 1 : Create the N x N histogram matrix O** \n\n\n- To create the N x N histogram matrix O, we have to define the confusion matrix. So, let's do it."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.metrics import confusion_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the purpose of explaination, we will assume are actual and preds labels to be the following."},{"metadata":{"trusted":true},"cell_type":"code","source":"# define the actual and predicted labels\nX_actual = pd.Series([1,1,1,2,3,4,4,4,4,4]) \nX_pred   = pd.Series([1,1,1,2,3,4,4,4,4,0]) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create the histogram matrix O\nO = confusion_matrix(X_actual, X_pred)\nO","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we will calculate N which is the number of labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"N = len(X_actual)\nN","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **Step 2** : An N-by-N matrix of weights (w) is calculated\n\n\n- Now, an N-by-N matrix of weights, w, is calculated based on the difference between actual and predicted values."},{"metadata":{"trusted":true},"cell_type":"code","source":"w = np.zeros((5,5))\nw","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(w)):\n    for j in range(len(w)):\n        w[i][j] = float(((i-j)**2)/((N-1)**2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"w","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **Step 3** : Calculation of N-by-N histogram matrix of expected scores(E)\n\n\n- An N-by-N histogram matrix of expected outcomes(E) is calculated assuming that there is no correlation between values.\n\n- This is calculated as the outer product between the actual histogram vector of outcomes and the predicted histogram vector, normalized such that E and O have the same sum."},{"metadata":{"trusted":true},"cell_type":"code","source":"N = 5\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculation of actual histogram vector\nX_actual_hist=np.zeros([N]) \nfor i in X_actual: \n    X_actual_hist[i]+=1    \n\nprint('Actuals value counts : {}'.format(X_actual_hist))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculation of predicted histogram vector\nX_pred_hist=np.zeros([N]) \nfor i in X_pred: \n    X_pred_hist[i]+=1    \n\nprint('Predicted value counts : {}'.format(X_pred_hist))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Now, we calculate the expected matrix E.\n\n- Expected matrix (E) is calculated as the outer product between the actual values histogram vector and the predicted values histogram vector."},{"metadata":{"trusted":true},"cell_type":"code","source":"E = np.outer(X_actual_hist, X_pred_hist)\nE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Now, we normalize E and O.\n\n- E and O are normalized such that E and O have the same sum."},{"metadata":{"trusted":true},"cell_type":"code","source":"E = E/E.sum()\nE.sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"O = O/O.sum()\nO.sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Let's print the matrix E and O."},{"metadata":{"trusted":true},"cell_type":"code","source":"E","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"O","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **Step 4** : Final Step : QWK Calculation\n\n\n- Now, we come to the final step. In this step, we calculate the **Quadratic Weighted Kappa (QWK)**. It is calculated as follows -\n\n\n$$\\kappa = 1 - \\dfrac{\\sum_{i,j}\\text{w}_{i,j}O_{i,j}}{\\sum_{i,j}\\text{w}_{i,j}E_{i,j}}$$\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"Num=0\nDen=0\n\nfor i in range(len(w)):\n    for j in range(len(w)):\n        Num+=w[i][j]*O[i][j]\n        Den+=w[i][j]*E[i][j]\n        \nRes = Num/Den\n \nQWK = (1 - Res)\nprint('The QWK value is {}'.format(round(QWK,4)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **4. Interpretation of QWK Value** <a class=\"anchor\" id=\"4\"></a>\n\n[Table of Contents](#0.1)\n\n\n- Our QWK value has found to be 0.6154.\n\n- It means that the actual and predicted values have very good agreement."},{"metadata":{},"cell_type":"markdown","source":"# **5. Conclusion** <a class=\"anchor\" id=\"5\"></a>\n\n[Table of Contents](#0.1)\n\n\n- In this notebook, we have explained the meaning of **QWK**.\n\n- We have also demonstrated how to calculate **QWK**.\n\n- We have taken sample actual and predicted values and found the QWK value to be 0.6154.\n\n- So, we can conclude that the actual and predicted values have every good agreement between them."},{"metadata":{},"cell_type":"markdown","source":"Thus, we come to the end of this notebook.\n\nI hope you find this notebook useful and enjoyable.\n\nYour comments and feedback are most welcome.\n\nThank you\n"},{"metadata":{},"cell_type":"markdown","source":"[Go to Top](#0)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}