{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Table of Contents\n### Exploratory Data Analysis\n* [0. Importing Libraries](#c0)\n* [1. Loading Original Dataset](#c1)\n* [2. Exploration and data cleaning](#c2)\n    * [2.1 Understanding the features](#s21)\n    * [2.2 Identifying Duplicated and Null Values](#s22)\n    * [2.3 Handling Missing Values and Removing Irrelevant Information](#s23)\n* [3. Univariate Analysis](#c3)\n    * [3.1 Dividing our dataset into categorial and numerical.](#s31)\n    * [3.2 Categorical Variable Analysis](#s32) \n    * [3.3 Numerical Variable Analysis](#s33)\n* [4. Multivariate Analysis](#c4)\n    * [4.1 Encoding Categorical Values and Saving JSON files](#s41)\n    * [4.2 Numerical-Categorical Analysis (Correlation Analysis)](#s42)\n* [5. Feature Engineering](#c5)\n    * [5.1 Outlier Analysis](#s51)\n    * [5.2 Split train/test of both Data Frames](#s52)","metadata":{}},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA) \n# 0. Importing Libraries <a class=\"anchor\" id=\"c0\"></a>","metadata":{}},{"cell_type":"code","source":"import pandas as pd \nimport numpy as np \nimport seaborn as sns \nimport matplotlib.pyplot as plt\nimport math\n\nimport json\nfrom pickle import dump\n\nfrom sklearn.model_selection import train_test_split, RandomizedSearchCV, GridSearchCV\nfrom sklearn.metrics import r2_score, mean_absolute_error\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\nfrom xgboost import XGBRegressor\n\nimport warnings\ndef warn(*args, **kwargs):\n    pass\nwarnings.warn = warn\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\npd.set_option('display.max_columns', None)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:55:49.403794Z","iopub.execute_input":"2025-02-10T12:55:49.404121Z","iopub.status.idle":"2025-02-10T12:55:53.057421Z","shell.execute_reply.started":"2025-02-10T12:55:49.404084Z","shell.execute_reply":"2025-02-10T12:55:53.056521Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# 1. Loading Original Dataset <a class=\"anchor\" id=\"c1\"></a>","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/playground-series-s4e12/train.csv\")\ndf.head(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:55:53.059473Z","iopub.execute_input":"2025-02-10T12:55:53.059929Z","iopub.status.idle":"2025-02-10T12:56:00.213339Z","shell.execute_reply.started":"2025-02-10T12:55:53.059897Z","shell.execute_reply":"2025-02-10T12:56:00.212164Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# 2. Exploration and data cleaning <a class=\"anchor\" id=\"c2\"></a>\n## 2.1 Understanding the dataset <a id=\"s21\"></a>","metadata":{}},{"cell_type":"code","source":"df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:00.214703Z","iopub.execute_input":"2025-02-10T12:56:00.215108Z","iopub.status.idle":"2025-02-10T12:56:00.92862Z","shell.execute_reply.started":"2025-02-10T12:56:00.215058Z","shell.execute_reply":"2025-02-10T12:56:00.927312Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- ```Id```: A unique identifier assigned to each customer or record. -> float\n- ```Age```: The age of the customer. -> float\n- ```Gender```: The gender of the customer (e.g., male or female). -> str\n- ```Annual Income```: The yearly income of the customer. -> float\n- ```Marital Status```: The marital status of the customer (e.g., single, married). -> str\n- ```Number of Dependents```: The number of people financially dependent on the customer. -> float\n- ```Education Level```: The highest level of education attained by the customer. -> str\n- ```Occupation```: The profession or job category of the customer. -> str\n- ```Health Score```: A numerical representation of the customer’s health status. -> float\n- ```Location```: The geographical area where the customer resides. -> str\n- ```Policy Type```: The category or type of insurance policy (e.g., health, vehicle). -> str\n- ```Previous Claims```: The number or details of prior insurance claims made by the customer. -> float\n- ```Vehicle Age```: The age of the customer's vehicle (if applicable). -> float\n- ```Credit Score```: A numerical score representing the customer’s creditworthiness. -> float\n- ```Insurance Duration```: The length of time the customer has been insured. -> float\n- ```Policy Start Date```: The date when the insurance policy commenced. -> datetime\n- ```Customer Feedback```: Input or reviews provided by the customer about the services. -> str\n- ```Smoking Status```: Indicates whether the customer smokes or not. -> str\n- ```Exercise Frequency```: How often the customer engages in physical exercise. -> str\n- ```Property Type```: The category of property owned (e.g., apartment, house). -> str\n- ```Premium Amount```: The cost paid by the customer for the insurance policy. -> float","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.histplot(df['Premium Amount'], kde=True, bins=50)\nplt.title('Premium Amount Distribution')\nplt.xlabel(\"Premium Amount\")\nplt.ylabel(\"Frequency\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:00.930061Z","iopub.execute_input":"2025-02-10T12:56:00.930517Z","iopub.status.idle":"2025-02-10T12:56:06.703988Z","shell.execute_reply.started":"2025-02-10T12:56:00.930463Z","shell.execute_reply":"2025-02-10T12:56:06.702987Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print('Our dataframe contains {} rows and it has {} features.'.format(len(df), df.shape[1]))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:06.705596Z","iopub.execute_input":"2025-02-10T12:56:06.706043Z","iopub.status.idle":"2025-02-10T12:56:06.712225Z","shell.execute_reply.started":"2025-02-10T12:56:06.705995Z","shell.execute_reply":"2025-02-10T12:56:06.71118Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Key Insights\n- **Small Premiums Are Common**: A significant proportion of premiums (about 5% of the dataset) are below 50.0, indicating a noticeable group of customers with very low premium amounts. This might represent a specific segment (e.g., entry-level policies or minimal coverage).\n\n- **Clustered Distribution in Lower Ranges**:Beyond the spike at the low end (20-50), the main distribution centers around 500 to 2000, suggesting that this range is typical for most customers.\n\n- **Right-Skewed Distribution**: The overall data is still right-skewed, with very few customers having higher premiums exceeding 3000.\n\n- **Potential Outliers at High Premiums**: The tail of the distribution extends up to 5000, which could represent high-value customers or policies. Further analysis may reveal if these values are true outliers or represent unique customer groups.\n\n- **Importance of Segmentation**: The premiums below 50.0 and those in the higher ranges likely correspond to distinct customer groups or insurance products. Segmenting these groups for further analysis may yield valuable insights.","metadata":{}},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 2.2 Identifying Duplicated and Null Values <a id=\"s22\"></a>","metadata":{}},{"cell_type":"code","source":"# Duplicated values\nprint(f'We have {df.duplicated().sum()} duplicated values')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:06.713802Z","iopub.execute_input":"2025-02-10T12:56:06.714225Z","iopub.status.idle":"2025-02-10T12:56:08.549981Z","shell.execute_reply.started":"2025-02-10T12:56:06.714183Z","shell.execute_reply":"2025-02-10T12:56:08.548629Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Null values \nprint('Features with Null Values')\nprint(df.isna().sum()[df.isna().sum()>0])\n\nplt.figure(figsize=(12,10))\nsns.heatmap(df.isnull(), cbar=False, cmap='Blues')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:08.554096Z","iopub.execute_input":"2025-02-10T12:56:08.554468Z","iopub.status.idle":"2025-02-10T12:56:34.352872Z","shell.execute_reply.started":"2025-02-10T12:56:08.554417Z","shell.execute_reply":"2025-02-10T12:56:34.351662Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"####  Key Observations from the Null Value Heatmap:\n- **Widespread Missing Values**: Several columns exhibit a significant number of null values, indicated by the large vertical stripes. Notably, columns like:\n    - ```Annual Income```\n    - ```Occupation```\n    - ```Previous Claims```\n    - ```Credit Score```\n\n- **Columns with Minimal or No Missing Data**: Columns such as ```Id```, ```Premium Amount```, and ```Policy Type``` appear to have no or minimal missing values, which indicates they can be used reliably without additional preprocessing.\n\n- **Clusters of Missing Values**: Certain rows have missing data across multiple columns, suggesting these records may represent specific types of customers or issues during data collection.\n\n- **Specific Columns with High Null Values**: ```Previous Claims```: This column has significant missing values. It could indicate customers with no prior claims, which may need to be filled with a default value like 0. ```Credit Score```: Missing values here could affect predictive models and may need careful imputation, possibly using related columns like ```Annual Income``` or ```Occupation```.\n  \n- **Patterns of Missingness**: The null values appear to be distributed unevenly across the dataset, with some columns showing scattered missing values, while others (e.g., ```Occupation```) have entire sections missing.","metadata":{}},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 2.3 Handling Missing Values and Removing Irrelevant Information <a id=\"s23\"></a>","metadata":{}},{"cell_type":"code","source":"print(df.isna().sum()[df.isna().sum()>0])\n\ndf = df.drop(columns='id')\n# Originally, the \"Policy Start Date\" was an object. Therefore, it has to be converted to DateTime\ndf['Policy Start Date'] = pd.to_datetime(df['Policy Start Date'])\n\n# Simple process of filling null values \ndf['Marital Status'].fillna('Unknown', inplace=True)\ndf['Occupation'].fillna('Unknown', inplace=True)\ndf['Previous Claims'].fillna(0, inplace=True)\ndf['Vehicle Age'].fillna(df['Vehicle Age'].median(), inplace=True)\ndf['Insurance Duration'].fillna(df['Insurance Duration'].median(), inplace=True)\ndf['Customer Feedback'].fillna('No Feedback', inplace=True)\n\n# Filling 'Age' null values using groupings \nage_gcols = ['Gender', 'Occupation', 'Marital Status']\ndf['Age'] = df.groupby(age_gcols)['Age'].transform(lambda x: x.fillna(x.median()))\n\n# Filling 'Health Score' null values using groupings\n# df['Smoking_enc'] = df['Smoking Status'].transform(lambda x: 0 if x=='No' else 1)\nhealth_gcols = ['Age', 'Gender', 'Smoking Status', 'Exercise Frequency']\ndf['Health Score'] = df.groupby(health_gcols)['Health Score'].transform(lambda x: x.fillna(x.median()))\n# df.drop(columns='Smoking_enc', inplace=True)\n\n# Filling 'Number of Dependents' null values using groupings \ndependents_gcols = ['Age', 'Gender', 'Location', 'Marital Status', 'Education Level', 'Occupation', 'Property Type']\ndf['Number of Dependents'] = df.groupby(dependents_gcols)['Number of Dependents'].transform(lambda x: x.fillna(x.median()))\n\n# Filling 'Number of Dependents' null values using groupings -> 2nd time, reducing groups\ndependents_gcols2 = ['Age', 'Gender', 'Marital Status', 'Education Level', 'Property Type']\ndf['Number of Dependents'] = df.groupby(dependents_gcols2)['Number of Dependents'].transform(lambda x: x.fillna(x.median()).astype(int))\n\n# Filling 'Credit Score' null values using groupings \ncredit_gcols = ['Age', 'Education Level', 'Occupation', 'Marital Status', 'Policy Type']\ndf['Credit Score'] = df.groupby(credit_gcols)['Credit Score'].transform(lambda x: x.fillna(x.mean()))\n\n# Filling 'Credit Score' null values using groupings -> 2nd time, reducing groups\ncredit_gcols2 = ['Age', 'Education Level', 'Occupation', 'Policy Type']\ndf['Credit Score'] = df.groupby(credit_gcols2)['Credit Score'].transform(lambda x: x.fillna(x.mean()))\n\n# Filling 'Annual Income' null values using groupings \nincome_gcols = ['Age', 'Education Level', 'Occupation', 'Policy Type']\ndf['Annual Income'] = df.groupby(income_gcols)['Annual Income'].transform(lambda x: x.fillna(int(x.mean())))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:56:34.354102Z","iopub.execute_input":"2025-02-10T12:56:34.354464Z","iopub.status.idle":"2025-02-10T12:57:02.851113Z","shell.execute_reply.started":"2025-02-10T12:56:34.354395Z","shell.execute_reply":"2025-02-10T12:57:02.850007Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Null values \nprint('Now, our features with Null Values are:')\nprint(df.isna().sum()[df.isna().sum()>0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:02.852906Z","iopub.execute_input":"2025-02-10T12:57:02.853338Z","iopub.status.idle":"2025-02-10T12:57:04.102746Z","shell.execute_reply.started":"2025-02-10T12:57:02.853195Z","shell.execute_reply":"2025-02-10T12:57:04.101517Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# 3. Univariate Analysis <a class=\"anchor\" id=\"c3\"></a>\n## 3.1 Dividing our dataset into categorial and numerical. <a id=\"s31\"></a>","metadata":{}},{"cell_type":"code","source":"# Understanding how many type of features we have\ndf.dtypes.unique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:04.104324Z","iopub.execute_input":"2025-02-10T12:57:04.104786Z","iopub.status.idle":"2025-02-10T12:57:04.113521Z","shell.execute_reply.started":"2025-02-10T12:57:04.104739Z","shell.execute_reply":"2025-02-10T12:57:04.112359Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Dividing our dataframe by numerical and categorical features\nnum = ['float64', 'int64']\ncat = ['O']\n\ndf_num = df.select_dtypes(num)\ndf_cat = df.select_dtypes(cat)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:04.115097Z","iopub.execute_input":"2025-02-10T12:57:04.115569Z","iopub.status.idle":"2025-02-10T12:57:04.729465Z","shell.execute_reply.started":"2025-02-10T12:57:04.115524Z","shell.execute_reply":"2025-02-10T12:57:04.728486Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 3.2 Categorical Variable Analysis <a id=\"s32\"></a>","metadata":{}},{"cell_type":"code","source":"cat_features = df_cat.columns\n\ncols = 2\nrows = math.ceil(len(cat_features) / cols)  \nfig, axes = plt.subplots(rows, cols, figsize=(12, (rows*4)), squeeze=False)\n\nfor ax, feature in zip(axes.flat, cat_features):\n    sns.countplot(ax=ax, data=df_cat, x=feature, hue=feature, palette='Set2')\n    if ax.legend_:  \n        ax.legend_.remove()\n    ax.set_xticklabels(ax.get_xticklabels(), fontsize=8)\n    ax.set_title(feature)  \n\nfor ax in axes.flat[len(cat_features):]:  \n    ax.set_visible(False)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:04.730861Z","iopub.execute_input":"2025-02-10T12:57:04.731306Z","iopub.status.idle":"2025-02-10T12:57:19.565684Z","shell.execute_reply.started":"2025-02-10T12:57:04.731265Z","shell.execute_reply":"2025-02-10T12:57:19.564632Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 3.3 Numerical Variable Analysis <a id=\"s33\"></a>","metadata":{}},{"cell_type":"code","source":"num_features = df_num.columns\nn_cols = 2\nn_rows = math.ceil(len(num_features) / n_cols)\n\nfig, ax = plt.subplots(n_rows * 2, n_cols, figsize=(12, n_rows * 4), gridspec_kw={'height_ratios': [6, 1] * n_rows})\nif n_rows == 1:\n    ax = ax.reshape(-1, n_cols)\n\nfor i, feature in enumerate(num_features):\n    col = i % n_cols\n    row = i // n_cols\n    sns.histplot(ax=ax[row * 2, col], data=df_num, x=feature, palette='Set2').set(ylabel='Frequency', title=f\"Histogram of {feature}\")\n    sns.boxplot(ax=ax[row * 2 + 1, col], data=df_num, x=feature).set(xlabel=None, ylabel='')\n\nfor row in range(n_rows * 2):\n    for col in range(n_cols):\n        if row // 2 * n_cols + col >= len(num_features):\n            ax[row, col].set_visible(False)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:19.567639Z","iopub.execute_input":"2025-02-10T12:57:19.568081Z","iopub.status.idle":"2025-02-10T12:57:28.280673Z","shell.execute_reply.started":"2025-02-10T12:57:19.568034Z","shell.execute_reply":"2025-02-10T12:57:28.279542Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# 4. Multivariate Analysis <a class=\"anchor\" id=\"c4\"></a>\n## 4.1 Encoding Categorical Values and Saving JSON files <a id=\"s41\"></a>","metadata":{}},{"cell_type":"code","source":"df_enc = df.copy()\n\n# Creating encoders for categorical features and saving them as JSON files. All files prefixed with 'enc'\n# contain the encoding dictionaries for each categorical feature.\nfor column in df_cat.columns:\n    unique_values = list(df_cat[column].unique())\n    globals()[f\"{column}_enc\"] = dict(zip(unique_values, range(len(unique_values))))\n\n    json.dump(globals()[f\"{column}_enc\"], open(f'/kaggle/working/enc_{column}.json', 'w'))\n\n# Replacing the values in our categorical features to our encoded values (numerical)\nfor column in df_cat.columns:\n    df_enc[column] = df_enc[column].map(json.load(open(f'/kaggle/working/enc_{column}.json')))\n\ndf_enc.head(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:28.282221Z","iopub.execute_input":"2025-02-10T12:57:28.282569Z","iopub.status.idle":"2025-02-10T12:57:30.293321Z","shell.execute_reply.started":"2025-02-10T12:57:28.282535Z","shell.execute_reply":"2025-02-10T12:57:30.292252Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 4.2 Numerical-Categorical Analysis (Correlation Analysis) <a id=\"s42\"></a>","metadata":{}},{"cell_type":"code","source":"features = [feature for feature in df.columns if feature != 'Policy Start Date']\ntarget = 'Premium Amount'\n\nmax_categories = 25\nfeatures = [feature for feature in features if df[feature].nunique() <= max_categories]\n\nrows = math.ceil(len(features) / cols)  \n\nfig, axes = plt.subplots(rows, cols, figsize=(12, rows * 4), squeeze=False)\naxes = axes.flatten()  \n\nfor i, feature in enumerate(features):\n    try:\n        sns.countplot(ax=axes[i * 2], data=df, x=feature, palette='Set2')\n        axes[i * 2].set_title(f'Distribution of {feature}', fontsize=12)\n        axes[i * 2].set_xticklabels(axes[i * 2].get_xticklabels(), fontsize=8)\n        sns.despine(ax=axes[i * 2])\n\n        sns.boxplot(ax=axes[i * 2 + 1], data=df, x=feature, y=target, palette='Set2')\n        axes[i * 2 + 1].set_title(f'{feature} vs {target}', fontsize=12)\n        sns.despine(ax=axes[i * 2 + 1])\n    except Exception as e:\n        continue\n\nfor ax in axes[len(features) * 2:]:\n    ax.set_visible(False)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:30.294646Z","iopub.execute_input":"2025-02-10T12:57:30.295056Z","iopub.status.idle":"2025-02-10T12:57:41.970191Z","shell.execute_reply.started":"2025-02-10T12:57:30.295018Z","shell.execute_reply":"2025-02-10T12:57:41.968934Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Distributions of Features and Their Relationship with Premium Amount (Summary):\n- ```Gender```: Premium amounts are slightly higher for males, but distributions are similar overall.\n- ```Marital Status```: Premiums are consistent across statuses, with only minor differences in medians.\n- ```Number of Dependents```: Premiums tend to increase slightly as the number of dependents grows.\n- ```Education Level```: Higher education levels show marginally higher premiums but with minimal variation overall.\n- ```Occupation & Location```: Premium amounts are evenly distributed across categories, indicating little influence.\n- ```Policy Type```: Slight differences in premiums across types, with one type showing a slightly higher median.\n\n***Conclusion***: Most categorical features show minimal variation or influence on premium amounts. Only minor trends are observed for dependents and policy type.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(14,10))\nsns.heatmap(df_enc.corr().round(2), annot=True, mask=np.triu(df_enc.corr()))\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:41.971705Z","iopub.execute_input":"2025-02-10T12:57:41.97211Z","iopub.status.idle":"2025-02-10T12:57:45.616512Z","shell.execute_reply.started":"2025-02-10T12:57:41.972066Z","shell.execute_reply":"2025-02-10T12:57:45.615501Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Key Observations on the Correlation Heatmap:\n- **Low Correlations**: Most features exhibit very low correlations with ```Premium Amount```. The strongest positive correlation is observed for ```Credit Score``` (~0.19), indicating individuals with better credit scores may pay higher premiums.\n  \n- **Negligible Relationships**: Features like ```Age```, ```Gender```, and ```Marital Status``` have virtually no linear correlation with ```Premium Amount```.\n\n- **Other Features**: ```Previous Claims```, ```Customer Feedback```, and ```Insurance Duration``` have weak but slightly notable correlations, which could be explored further.","metadata":{}},{"cell_type":"markdown","source":"-------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# 5. Feature Engineering <a class=\"anchor\" id=\"c5\"></a>\n## 5.1 Outlier Analysis <a id=\"s51\"></a>\nPerform an outlier analysis exclusively on continuous features; outliers in discrete features should be addressed separately without changing their data. Since a few features are floats and should be ints, we will change them.","metadata":{}},{"cell_type":"code","source":"df_enc['Age'] = df_enc['Age'].apply(lambda x: int(x))\ndf_enc['Annual Income'] = df_enc['Annual Income'].apply(lambda x: int(x))\ndf_enc['Previous Claims'] = df_enc['Previous Claims'].apply(lambda x: int(x))\ndf_enc['Vehicle Age'] = df_enc['Vehicle Age'].apply(lambda x: int(x))\ndf_enc['Credit Score'] = df_enc['Credit Score'].apply(lambda x: int(x))\ndf_enc['Insurance Duration'] = df_enc['Insurance Duration'].apply(lambda x: int(x))\n\ncont_features = df_enc.select_dtypes('float64')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:45.617738Z","iopub.execute_input":"2025-02-10T12:57:45.618224Z","iopub.status.idle":"2025-02-10T12:57:48.800051Z","shell.execute_reply.started":"2025-02-10T12:57:45.618172Z","shell.execute_reply":"2025-02-10T12:57:48.799047Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(10, 5))\n\ncol = 0\nfor feature in cont_features:\n    sns.boxplot(ax = ax[col], data = df_enc, x=feature)\n    ax[col].set_title(f'Boxplot of {feature}', fontsize=12)\n    col += 1\n    if col == len(list(cont_features)):\n        break\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:48.801185Z","iopub.execute_input":"2025-02-10T12:57:48.801503Z","iopub.status.idle":"2025-02-10T12:57:49.354085Z","shell.execute_reply.started":"2025-02-10T12:57:48.801472Z","shell.execute_reply":"2025-02-10T12:57:49.352953Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Based on the chart above, we can see that the feature \"Premium Amount\" has outliers, which we will be removing for our second Data Frame. ","metadata":{}},{"cell_type":"code","source":"# Creating a copy of our df to remove outliers \ndf_enc_no = df_enc.copy()\n\n# This function returns our new df without outliers and the features' limits.  \ndef remove_outliers(x, feature_name, allow_neg=True):\n    q1, q3 = x.quantile([0.25, 0.75])\n    iqr = q3 - q1\n    upper_lim = q3 + (iqr*1.5)\n    lower_lim = q1 - (iqr*1.5) if allow_neg else max(0, q1 - (iqr * 1.5))\n\n    x = x.apply(lambda x: upper_lim if (x > upper_lim) else (lower_lim if (x < lower_lim) else x))\n\n    filename = f'/kaggle/working/outliers_lims_{feature_name}.json'\n    json.dump({'upper_lim': upper_lim, 'lower_lim': lower_lim}, open(filename, 'w'))\n\n    return x\n\n# Removing outliers in df_no\nfor feature in cont_features:\n    df_enc_no[feature] = remove_outliers(df_enc_no[feature], feature, allow_neg=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:49.355832Z","iopub.execute_input":"2025-02-10T12:57:49.356285Z","iopub.status.idle":"2025-02-10T12:57:50.287104Z","shell.execute_reply.started":"2025-02-10T12:57:49.356238Z","shell.execute_reply":"2025-02-10T12:57:50.285986Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_enc['Policy Day'] = df_enc['Policy Start Date'].dt.day\ndf_enc['Policy Month'] = df_enc['Policy Start Date'].dt.month\ndf_enc['Policy Year'] = df_enc['Policy Start Date'].dt.year\ndf_enc.drop(columns='Policy Start Date', inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:50.288596Z","iopub.execute_input":"2025-02-10T12:57:50.289012Z","iopub.status.idle":"2025-02-10T12:57:50.614858Z","shell.execute_reply.started":"2025-02-10T12:57:50.288971Z","shell.execute_reply":"2025-02-10T12:57:50.613504Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"-------------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"## 5.2 Split train/test of both Data Frames <a id=\"s52\"></a>","metadata":{}},{"cell_type":"code","source":"def split(target, df, reference: str, test_size=0.2, random_state=123):\n    X = df.drop(columns=target)\n    y = df[target]\n\n    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=random_state)\n\n    X_train.to_csv(f'/kaggle/working/X_train_{reference}.csv', index=False)\n    X_test.to_csv(f'/kaggle/working/X_test_{reference}.csv', index=False)\n    y_train.to_csv('/kaggle/working/y_train.csv', index=False)\n    y_test.to_csv('/kaggle/working/y_test.csv', index=False)\n    \n    return X_train, X_test, y_train, y_test\n\n# Splitting X_train, X_test with/without outliers and y_train, y_test\nX_train_with_outliers, X_test_with_outliers, y_train, y_test = split('Premium Amount', df_enc, 'with_outliers')\nX_train_without_outliers, X_test_without_outliers, _, _ = split('Premium Amount', df_enc_no, 'without_outliers')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:57:50.616404Z","iopub.execute_input":"2025-02-10T12:57:50.617848Z","iopub.status.idle":"2025-02-10T12:58:10.798833Z","shell.execute_reply.started":"2025-02-10T12:57:50.617783Z","shell.execute_reply":"2025-02-10T12:58:10.797691Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dfs_train, dfs_test = {}, {}\n\nfor label, (X_train, X_test) in {\n    \"with_outliers\": (X_train_with_outliers, X_test_with_outliers),\n    \"without_outliers\": (X_train_without_outliers, X_test_without_outliers)\n}.items():                    \n    dfs_train[f'X_train_{label}'] = X_train\n    dfs_test[f'X_test_{label}'] = X_test\n\ntrain, test = [], []\n\nfor name, df in dfs_train.items():\n    train.append(df)\nfor name, df in dfs_test.items():\n    test.append(df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:58:10.802807Z","iopub.execute_input":"2025-02-10T12:58:10.803153Z","iopub.status.idle":"2025-02-10T12:58:10.848716Z","shell.execute_reply.started":"2025-02-10T12:58:10.803122Z","shell.execute_reply":"2025-02-10T12:58:10.8475Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# *** For some reason, the code below produced an error on Kaggle, even though it ran without issues on my local machine. \n# I'm keeping it here so you can test it locally and see that XGBRegressor using outliers is the best-performing model — that's why I use it in the final implementation. ***\n\n# results = []\n# models = [DecisionTreeRegressor(random_state=123), \n#           RandomForestRegressor(random_state=123), \n#           GradientBoostingRegressor(random_state=123), \n#           XGBRegressor(random_state=123)]\n\n# for index in range(len(train)):\n#     for model in models:\n#         model = model\n#         train_df = train[index]\n#         model.fit(train_df, y_train)\n#         y_pred = model.predict(test[index])\n    \n#         results.append(\n#             {\n#                 'Index': index,\n#                 'Model': model,\n#                 'MAE': mean_absolute_error(y_test, y_pred).round(2),\n#                 'R2_score': round(r2_score(y_test, y_pred),5), \n#                 'R2_score_abs': abs(round(r2_score(y_test, y_pred),5))\n#             }\n#         )\n\n# results = sorted(results, key=lambda x: x['R2_score_abs'])\n# b_model = results[0]['Model']\n# r2_best = results[0]['R2_score_abs']\n# print(f'Our best model is {b_model}, with an R2 Score of {r2_best}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:58:10.850504Z","iopub.execute_input":"2025-02-10T12:58:10.850808Z","iopub.status.idle":"2025-02-10T12:58:10.863865Z","shell.execute_reply.started":"2025-02-10T12:58:10.850779Z","shell.execute_reply":"2025-02-10T12:58:10.862589Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = GradientBoostingRegressor(random_state=123)\nmodel.fit(X_train_with_outliers, y_train)\ny_pred = model.predict(X_test_with_outliers)\nprint(f'MAE: {mean_absolute_error(y_test, y_pred).round(2)}')\nprint(f'R2_score: {round(r2_score(y_test, y_pred),5)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T12:58:10.865458Z","iopub.execute_input":"2025-02-10T12:58:10.866351Z","iopub.status.idle":"2025-02-10T13:03:35.118934Z","shell.execute_reply.started":"2025-02-10T12:58:10.8663Z","shell.execute_reply":"2025-02-10T13:03:35.117673Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = XGBRegressor(random_state=123)\nmodel.fit(X_train_with_outliers, y_train)\ny_pred = model.predict(X_test_with_outliers)\nprint(f'MAE: {mean_absolute_error(y_test, y_pred).round(2)}')\nprint(f'R2_score: {round(r2_score(y_test, y_pred),5)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T13:03:35.120312Z","iopub.execute_input":"2025-02-10T13:03:35.120657Z","iopub.status.idle":"2025-02-10T13:03:40.144223Z","shell.execute_reply.started":"2025-02-10T13:03:35.120626Z","shell.execute_reply":"2025-02-10T13:03:40.143337Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = DecisionTreeRegressor(random_state=123)\nmodel.fit(X_train_with_outliers, y_train)\ny_pred = model.predict(X_test_with_outliers)\nprint(f'MAE: {mean_absolute_error(y_test, y_pred).round(2)}')\nprint(f'R2_score: {round(r2_score(y_test, y_pred),5)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T13:03:40.145121Z","iopub.execute_input":"2025-02-10T13:03:40.145447Z","iopub.status.idle":"2025-02-10T13:04:06.481106Z","shell.execute_reply.started":"2025-02-10T13:03:40.145397Z","shell.execute_reply":"2025-02-10T13:04:06.480026Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = RandomForestRegressor(random_state=123)\nmodel.fit(X_train_with_outliers, y_train)\ny_pred = model.predict(X_test_with_outliers)\nprint(f'MAE: {mean_absolute_error(y_test, y_pred).round(2)}')\nprint(f'R2_score: {round(r2_score(y_test, y_pred),5)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-10T13:04:06.482418Z","iopub.execute_input":"2025-02-10T13:04:06.482768Z"}},"outputs":[],"execution_count":null}]}