{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

Weighted multi-class logarithmic loss

\n\nSubmissions are evaluated using a **weighted multi-class logarithmic loss**. The overall effect is such that each class is roughly equally important for the final score.\n\nEach image has been labeled with an etiology class, either CE or LAA. For each image, you must submit a probability for each class. The formula is then:\n\n$$ Log\\:Loss = - \\left( \\frac{\\sum_{i=1}^{M} w_{i} . \\sum_{j=1}^{N_{i}} \\frac{y_{ij}}{N_{i}}.\\ln p_{ij}}{\\sum_{i=1}^{M} w_{i}} \\right) $$\n\nWhere:\n\n- **N:** number of images in the class set\n- **M:** number of classes\n- **Ln:** natural logarithm\n- **yij:** 1 if observation belongs to class *j* and 0 otherwise.\n- **pij:** predicted probability that image *i* belongs to class *j*.\n\n\nThe submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum). In order to avoid the extremes of the log function, each predicted probability 𝑝 is replaced with $max(min(\\rho,1−10^{-15}),10^{−15})$.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"### Import libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-07-11T21:46:20.680792Z","iopub.execute_input":"2022-07-11T21:46:20.681249Z","iopub.status.idle":"2022-07-11T21:46:22.563563Z","shell.execute_reply.started":"2022-07-11T21:46:20.681215Z","shell.execute_reply":"2022-07-11T21:46:22.56228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Example 👨🏻‍🏫!!!

\n\nLet's say we have 5 images: 2 of class \"A\" and 3 of class \"B\".\n- Then N:\n```\nN = {\n \"A\": 2,\n \"B\": 3\n }\n```\n- Since we have two classes then `W=2`. \n- According to this [discussion](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336511) weights `w` are `[0.5,0.5]`.\n- Finally let's say we have made a prediction that:\n```\ny_pred_A = 0.9\ny_pred_B = 0.2\n```\n\n

Let's calculate Weighted Multi-class LogLoss 💪🏼

\n\nAssume that for this prediction `y_true=A`\n\n1. First let's rescale our probabilities: \n - `y_pred_A_scaled` = 0.9/1.1 = 0.8181\n - `y_pred_B_scaled` = 0.2/1.1 = 0.1818\n2. Apply Natural logarithm:\n - `y_pred_A_log` = $\\ln(y_{Ascaled})$ = $\\ln(0.8181)$ = -0.201\n - `y_pred_B_log` = $\\ln(y_{Bscaled})$ = $\\ln(0.1818)$ = -1.705\n3. Divide by $\\frac{y_{ij}}{N_{i}}$:\n - Since `y_true = A`\n - $\\frac{y_{1A}}{N_{1}}$ = 1/2\n - $\\frac{y_{1B}}{N_{1}}$ = 0/3\n$$ Log\\:Loss = - \\left( \\frac{0.5*-0.201*\\frac{1}{2} + 0.5*-1.705*\\frac{0}{3}}{1} \\right) = 0.05016 $$\n\n### Excellent 👏🏼 we have calculated Weighted Multi-class LogLoss for one image! \n\nIn this competition we will have to compute it for every image in the dataset, so in reality we will have a sum of lots of these individual results.\n\n

Time to code ⏳

\n\nLet's code this written example!","metadata":{}},{"cell_type":"markdown","source":"### Utility functions","metadata":{}},{"cell_type":"code","source":"def get_params(df, class_column):\n \"\"\"\n This functions returns a dictionary with counts for every class in a ground truth column.\n :param df: a pandas dataframe with a column of ground truth predictions.\n :param class_column: string with the name of the ground truth column.\n :return : a dictionary where keys are classes and values are counts.\n \"\"\"\n N = dict(df[class_column].value_counts())\n return N\n\n\ndef get_random_weights(size, equal_weight=False):\n \"\"\"\n This functions returns a numpy array with weights that sum to 1.\n :param size: how many weights to have.\n :param equal_weight: if True all weights are equal in value.\n :return : a numpy array with weights between 0 and 1 which sum to 1.\n \"\"\"\n if equal_weight:\n w = np.ones(size)\n w_sum = sum(w)\n w = w/w_sum\n else:\n w = np.random.randint(low=0, high=10, size=size)\n w_sum = sum(w)\n w = w/w_sum\n return w\n \n\ndef scale_predicted_probs(y_pred_1, y_pred_2):\n \"\"\"\n Given two numbers this function returns their weighted values.\n \"\"\"\n sum_ = y_pred_1 + y_pred_2\n y_pred_1, y_pred_2 = y_pred_1/sum_, y_pred_2/sum_\n return y_pred_1, y_pred_2\n\n\ndef cap_prob(prob):\n \"\"\"\n This function caps values. If it is 1 it returns a very close number to 1 (e.g: 0.999...) and if it is 0\n it returns a number very close to 0 (e.g: 0.000...1). This helps prevent extreme values when using a logarithm.\n \"\"\"\n p = max(min(prob, 1-10**-15),10**-15)\n return p\n\n\ndef sep():\n print(\"-\"*100)","metadata":{"execution":{"iopub.status.busy":"2022-07-11T21:48:50.135214Z","iopub.execute_input":"2022-07-11T21:48:50.136145Z","iopub.status.idle":"2022-07-11T21:48:50.148034Z","shell.execute_reply.started":"2022-07-11T21:48:50.136102Z","shell.execute_reply":"2022-07-11T21:48:50.146777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Weighted multi-class logarithmic loss function","metadata":{}},{"cell_type":"code","source":"def multi_class_logarithmic_loss(y_true, y_pred_1, y_pred_0, N, w):\n # Scale row predictions for each class\n y_pred_1, y_pred_2 = scale_predicted_probs(y_pred_1, y_pred_0)\n if y_true == 1:\n # Cap predicted probabilities\n y_pred_1 = cap_prob(y_pred_1)\n # Apply natural log to each probability \n y_pred_1_log = np.log(y_pred_1)\n # Get y_N\n y_N = 1/N[\"A\"]\n mcll = w * y_N * y_pred_1_log\n else:\n y_pred_0 = cap_prob(y_pred_0)\n y_pred_0_log = np.log(y_pred_0)\n y_N = 1/N[\"B\"]\n mcll = (1-w) * y_N * y_pred_0_log\n return -mcll","metadata":{"execution":{"iopub.status.busy":"2022-07-11T21:50:11.793098Z","iopub.execute_input":"2022-07-11T21:50:11.793448Z","iopub.status.idle":"2022-07-11T21:50:11.802304Z","shell.execute_reply.started":"2022-07-11T21:50:11.79342Z","shell.execute_reply":"2022-07-11T21:50:11.801155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create an artificial dataframe","metadata":{}},{"cell_type":"code","source":"y_true = [\"A\",\"A\",\"B\",\"B\",\"B\"]\ny_pred_A = [0.90, 0.11, 0.21, 0.48, 0.18]\ny_pred_B = [0.20, 0.80, 0.23, 0.42, 0.10]\ndf = pd.DataFrame([y_true, y_pred_A, y_pred_B]).T\ndf.columns = [\"y_true\", \"y_pred_A\", \"y_pred_B\"]\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-11T21:46:41.048307Z","iopub.execute_input":"2022-07-11T21:46:41.048723Z","iopub.status.idle":"2022-07-11T21:46:41.080596Z","shell.execute_reply.started":"2022-07-11T21:46:41.048683Z","shell.execute_reply":"2022-07-11T21:46:41.079703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N = get_params(df, \"y_true\")\nprint(f\"N is: {N}\")\nsep()\nw = get_random_weights(2, equal_weight=True)\nprint(f\"Weights are: {w}\")\nsep()\n# Assume A=1 and B=0\nmcll = multi_class_logarithmic_loss(1, 0.9, 0.2, N, 0.5)\nprint(f\"Weighted multi-class logarithmic loss: {mcll}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-11T21:50:25.532404Z","iopub.execute_input":"2022-07-11T21:50:25.532798Z","iopub.status.idle":"2022-07-11T21:50:25.541615Z","shell.execute_reply.started":"2022-07-11T21:50:25.532766Z","shell.execute_reply":"2022-07-11T21:50:25.540576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Hope you liked it 😉👍🏼

","metadata":{}}]}