{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **EDA & Transformation**\nWe’ve created the [Starter Pack](https://www.kaggle.com/code/isaranja/s4e12-starter-pack-by-ama) to provide a solid pipeline for this competition. This is the second step of the solution approach.\n\nNow, let’s dive into an in-depth analysis, starting with comprehensive Exploratory Data Analysis (EDA). Since this is a tabular dataset, we can apply transformations that will significantly enhance model-building efforts.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"# Importing libraries\n\nfrom IPython.core.interactiveshell import InteractiveShell #\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom scipy.stats import f_oneway, pointbiserialr, pearsonr, spearmanr # statistical analysis\n\nfrom sklearn.preprocessing import LabelEncoder,OneHotEncoder # transformation\n\nimport seaborn as sns # plots\nimport matplotlib.pyplot as plt # plots\n\nimport warnings #\n\nfrom tabulate import tabulate # tabulate printing","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:57.904605Z","iopub.execute_input":"2024-12-29T09:33:57.905086Z","iopub.status.idle":"2024-12-29T09:33:59.400189Z","shell.execute_reply.started":"2024-12-29T09:33:57.905031Z","shell.execute_reply":"2024-12-29T09:33:59.398483Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# settings for jupyter envioronment\n\n# This ensures that plots are rendered inline\n%matplotlib inline\n\n# This ensures that all output, including text and plots, is shown automatically\nInteractiveShell.ast_node_interactivity = \"all\"\n\n# Switching off the future warrnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.402217Z","iopub.execute_input":"2024-12-29T09:33:59.403049Z","iopub.status.idle":"2024-12-29T09:33:59.412753Z","shell.execute_reply.started":"2024-12-29T09:33:59.402987Z","shell.execute_reply":"2024-12-29T09:33:59.410911Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#anova Test\ndef anova_test(feature):\n\n    df_lcl = df.loc[(df.src=='trn') & (df[feature].notna())].copy()\n    # Group the continuous values based on the categorical column\n    groups = [group['Premium Amount'].values for name, group in df_lcl.groupby(feature)]\n    \n    # Perform ANOVA\n    f_stat, p_value = f_oneway(*groups)\n    \n    print(f\"F-statistic: {f_stat:.4f}\")\n    print(f\"P-value: {p_value:.4f}\")\n    \n    if p_value < 0.05:\n        print(f\"There is a \\033[1msignificant correlation\\033[0m between the {feature} and Premium Amount variables.\")\n    else:\n        print(f\"There is \\033[1mno significant\\033[0m correlation between the {feature} and Premium Amount variables.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.414776Z","iopub.execute_input":"2024-12-29T09:33:59.415351Z","iopub.status.idle":"2024-12-29T09:33:59.429428Z","shell.execute_reply.started":"2024-12-29T09:33:59.415289Z","shell.execute_reply":"2024-12-29T09:33:59.427615Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def pearson_correlation(feature):\n    df_lcl = df.loc[(df.src=='trn') & (df[feature].notna())].copy()\n    # Calculate Pearson correlation\n    correlation, p_value = pearsonr(df_lcl[feature].values, df_lcl['Premium Amount'])\n    \n    # Categorize the correlation strength\n    if abs(correlation) >= 0.8:\n        strength = \"high\"\n    elif abs(correlation) >= 0.5:\n        strength = \"moderate\"\n    else:\n        strength = \"weak\"\n    \n    # Print results\n    print(f\"Pearson Correlation Coefficient: {correlation:.4f}\")\n    print(f\"P-value: {p_value:.4f}\")\n    print(f\"The correlation is \\033[1m{strength}\\033[0m.\")\n    \n    # Check statistical significance\n    if p_value < 0.05:\n        print(\"The correlation is statistically \\033[1msignificant\\033[0m.\")\n    else:\n        print(\"The correlation is \\033[1mNOT\\033[0m statistically significant.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.431296Z","iopub.execute_input":"2024-12-29T09:33:59.431775Z","iopub.status.idle":"2024-12-29T09:33:59.451753Z","shell.execute_reply.started":"2024-12-29T09:33:59.431718Z","shell.execute_reply":"2024-12-29T09:33:59.450277Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def spearman_correlation(feature):\n\n    df_lcl = df.loc[(df.src=='trn') & (df[feature].notna())].copy()\n\n    # Calculate Spearman's rank correlation\n    correlation, p_value = spearmanr(df_lcl[feature].values, df_lcl['Premium Amount'])\n    \n    # Categorize the correlation strength\n    if abs(correlation) >= 0.8:\n        strength = \"strong\"\n    elif abs(correlation) >= 0.5:\n        strength = \"moderate\"\n    else:\n        strength = \"weak\"\n    \n    # Print results\n    print(f\"Spearman Correlation Coefficient: {correlation:.4f}\")\n    print(f\"P-value: {p_value:.4f}\")\n    print(f\"The correlation is \\033[1m{strength}\\033[0m.\")\n    \n    # Check if correlation is impactful\n    if p_value < 0.05:\n        print(\"The correlation is statistically significant (\\033[1mimpactful\\033[0m).\")\n    else:\n        print(\"The correlation is \\033[1mNOT\\033[0m statistically significant (not impactful).\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.455764Z","iopub.execute_input":"2024-12-29T09:33:59.456326Z","iopub.status.idle":"2024-12-29T09:33:59.467098Z","shell.execute_reply.started":"2024-12-29T09:33:59.45628Z","shell.execute_reply":"2024-12-29T09:33:59.465606Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def point_biserial_correlation(feature):\n\n    df_lcl = df.loc[(df.src=='trn') & (df[feature].notna()),:].copy()\n        # Encode the binary string column to 0s and 1s\n    le = LabelEncoder()\n    df_lcl.loc[:,'Category Encoded'] = le.fit_transform(df_lcl[feature])\n    # Calculate point-biserial correlation\n    correlation, p_value = pointbiserialr(df_lcl['Category Encoded'], df_lcl['Premium Amount'])\n    \n    print(f\"\\nPoint-Biserial Correlation: {correlation:.4f}\")\n    print(f\"P-value: {p_value:.4f}\")\n    \n    if p_value < 0.05:\n        print(f\"There is a \\033[1msignificant correlation\\033[0m between the binary {feature} and 'Premium Amount' variables.\\n\")\n    else:\n        print(\"There is \\033[1mNO\\033[0m significant correlation.\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.469451Z","iopub.execute_input":"2024-12-29T09:33:59.470155Z","iopub.status.idle":"2024-12-29T09:33:59.493871Z","shell.execute_reply.started":"2024-12-29T09:33:59.470095Z","shell.execute_reply":"2024-12-29T09:33:59.491978Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Function to generate EDA for Ordinal features\ndef analyseOrdinal(feature):\n\n    # Null value analysis\n    data = [\n        [\"Null count\", df.loc[df.src=='trn',feature].isnull().sum(), df.loc[df.src=='tst',feature].isnull().sum()],\n        [\"Null percentage\", round(df.loc[df.src=='trn',feature].isnull().mean()*100,2), round(df.loc[df.src=='tst',feature].isnull().mean()*100,2)],]\n\n\n    print(tabulate(data, headers=[\"Attribute\", \"train\", \"test\"], tablefmt=\"simple_grid\",numalign=\"right\"))\n\n    df[feature] = df[feature].fillna(-1)\n    \n    # Value distribution \n    print('\\n')\n    pivot = df.pivot_table(index='src', columns=feature, aggfunc='size', fill_value=0)\n    pivot_percentage = round(pivot.div(pivot.sum(axis=1), axis=0) * 100)\n    pivot_percentage_with_symbol = pivot_percentage.applymap(lambda x: f\"{x:.2f}%\")\n    \n    print(tabulate(pivot_percentage_with_symbol, headers='keys', tablefmt='simple_grid',numalign=\"right\"))\n    \n    # Correlation \n    print('\\n')\n    # Print the correlation\n    spearman_correlation(feature)\n    print('\\n')\n    \n    \n    # Create a violing plot for column 'Gender'\n    _ = plt.figure(figsize=(20, 5))\n    _ = sns.violinplot(x=feature, y='Premium Amount', data=df.loc[df.src=='trn',:])\n    _ = plt.title(f'Premium amount vs {feature}')\n    \n    # Adjust layout to avoid overlap\n    plt.tight_layout()\n    \n    # Show the plot\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.496023Z","iopub.execute_input":"2024-12-29T09:33:59.497429Z","iopub.status.idle":"2024-12-29T09:33:59.518748Z","shell.execute_reply.started":"2024-12-29T09:33:59.497374Z","shell.execute_reply":"2024-12-29T09:33:59.51695Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Function to generate EDA for categorical features\ndef analyseCategorical(feature):\n\n    # Null value analysis\n    data = [\n        [\"Null count\", df.loc[df.src=='trn',feature].isnull().sum(), df.loc[df.src=='tst',feature].isnull().sum()],\n        [\"Null percentage\", round(df.loc[df.src=='trn',feature].isnull().mean()*100,2), round(df.loc[df.src=='tst',feature].isnull().mean()*100,2)],]\n\n\n    print(tabulate(data, headers=[\"Attribute\", \"train\", \"test\"], tablefmt=\"simple_grid\",numalign=\"right\"))\n\n    df[feature] = df[feature].fillna('unknown')\n\n    # Value distribution \n    print('\\n')\n    pivot = df.pivot_table(index='src', columns=feature, aggfunc='size', fill_value=0)\n    pivot_percentage = round(pivot.div(pivot.sum(axis=1), axis=0) * 100)\n    pivot_percentage_with_symbol = pivot_percentage.applymap(lambda x: f\"{x:.2f}%\")\n    \n    print(tabulate(pivot_percentage_with_symbol, headers='keys', tablefmt='simple_grid',numalign=\"right\"))\n\n    #correlation\n    print('\\n')\n    if df[feature].nunique() == 2 :\n        point_biserial_correlation(feature)\n    else:\n        anova_test(feature)\n    print('\\n')\n    \n    # Create a violing plot for column 'Gender'\n    _ = plt.figure(figsize=(20, 5))\n    _ = sns.violinplot(x=feature, y='Premium Amount', data=df.loc[df.src=='trn',:])\n    _ = plt.title(f'Premium amount vs {feature}')\n    \n    # Adjust layout to avoid overlap\n    plt.tight_layout()\n    \n    # Show the plot\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.520872Z","iopub.execute_input":"2024-12-29T09:33:59.521372Z","iopub.status.idle":"2024-12-29T09:33:59.546174Z","shell.execute_reply.started":"2024-12-29T09:33:59.521327Z","shell.execute_reply":"2024-12-29T09:33:59.544481Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# function to generate EDA for continues feature\ndef analyseContinues(feature):\n    data = [\n        [\"Null count\", df.loc[df.src=='trn',feature].isnull().sum(), df.loc[df.src=='tst',feature].isnull().sum()],\n        [\"Null percentage\", round(df.loc[df.src=='trn',feature].isnull().mean()*100,2), round(df.loc[df.src=='tst',feature].isnull().mean()*100,2)],\n        [\"Min\", df.loc[df.src=='trn',feature].min(), df.loc[df.src=='tst',feature].min()],\n        [\"Max\", df.loc[df.src=='trn',feature].max(), df.loc[df.src=='tst',feature].max()],]\n    \n    print(tabulate(data, headers=[\"Attribute\", \"train\", \"test\"], tablefmt=\"simple_grid\", numalign=\"right\"))\n    \n    # Print the correlation\n    print('\\n')\n    pearson_correlation(feature)\n    print('\\n')\n    \n    # plotting \n    fig = plt.figure(figsize=(20, 8))\n    \n    # Define grid for 2 rows, 2 columns\n    gs = fig.add_gridspec(2, 2)\n    \n    # First subplot in the first row, first column\n    ax1 = fig.add_subplot(gs[0, 0])\n    _ = sns.histplot(df.loc[df.src=='trn',feature], kde=True, ax=ax1)\n    _ = ax1.set_title(f'Histogram of {feature} in Train')\n    \n    # Second subplot in the first row, second column\n    ax2 = fig.add_subplot(gs[0, 1])\n    _ = sns.histplot(df.loc[df.src=='tst',feature], kde=True, ax=ax2)\n    _ = ax2.set_title(f'Histogram of {feature} in Test')\n    \n    # Third subplot in the second row, spanning both columns\n    \n    ax3 = fig.add_subplot(gs[1, :])  # Span both columns\n    \n    bins = 100\n    bins = np.linspace(df[feature].min(), df[feature].max(), bins)\n    \n    # Create a new column for bin labels (which bin each value of x falls into)\n    feature_binned = feature + '_binned'\n    \n    dfl = df.assign(**{feature_binned:pd.cut(df[feature], bins)})\n    \n    # Calculate the mean of 'y' for each bin\n    bin_mean = dfl[dfl['src']=='trn'].groupby(feature_binned)['Premium Amount'].mean().reset_index()\n\n    _ = sns.histplot(dfl.loc[dfl['src']=='trn',feature], bins=bins, kde=False, color='lightgray', edgecolor='black', ax=ax3)\n    _ = ax3.set_xlabel(feature_binned)\n    _ = ax3.set_ylabel('Frequency')\n    _ = ax3.grid(True)\n    _ = ax3.set_title(f'Histogram of {feature_binned} with Mean of Premium Amount per Bin')\n\n    _ = ax4 = ax3.twinx() # Create the second y-axis (ax2) that shares the x-axis with ax1\n\n    # Plot the mean of 'y' in each bin as a line plot on the second y-axis\n    _ = ax4.plot(bin_mean[feature_binned].apply(lambda x: x.mid), bin_mean['Premium Amount'], color='blue', marker='o', linestyle='-', linewidth=2)\n    _ = ax4.set_ylabel(f'Mean of {feature_binned}')\n    \n    # Adjust layout to avoid overlap\n    plt.tight_layout()\n    \n    # Show the plot\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.548608Z","iopub.execute_input":"2024-12-29T09:33:59.549298Z","iopub.status.idle":"2024-12-29T09:33:59.57232Z","shell.execute_reply.started":"2024-12-29T09:33:59.549216Z","shell.execute_reply":"2024-12-29T09:33:59.57059Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Function to generate EDA for date column\ndef analyseDate(feature):\n    print('Attribute \\t\\t| train \\t\\t\\t| test ')\n    print(\"-\" * 60)\n    print('Null value count \\t| ', df.loc[df.src=='trn',feature].isnull().sum(),'\\t\\t\\t\\t| ', df.loc[df.src=='tst',feature].isnull().sum())\n    print('Null value percentage \\t| ', round(df.loc[df.src=='trn',feature].isnull().mean()*100,2),'%\\t\\t\\t| ', round(df.loc[df.src=='tst',feature].isnull().mean()*100,2),'%')\n    print(f'Max {feature} \\t| ', df.loc[df.src=='trn',feature].max(),'\\t| ', df.loc[df.src=='tst',feature].max())\n    print(f'Min {feature} \\t| ', df.loc[df.src=='trn',feature].min(),'\\t| ', df.loc[df.src=='tst',feature].min())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.574598Z","iopub.execute_input":"2024-12-29T09:33:59.575282Z","iopub.status.idle":"2024-12-29T09:33:59.597389Z","shell.execute_reply.started":"2024-12-29T09:33:59.575211Z","shell.execute_reply":"2024-12-29T09:33:59.59564Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Loading dataset\ntrain_df = pd.read_csv(\"/kaggle/input/playground-series-s4e12/train.csv\",parse_dates=['Policy Start Date'])\ntest_df = pd.read_csv(\"/kaggle/input/playground-series-s4e12/test.csv\",parse_dates=['Policy Start Date'])\n\n# Merging two dataframes after adding trn and tst tag\ntrain_df['src']='trn'\ntest_df['src']='tst'\n\ndf = pd.concat([train_df, test_df], ignore_index=True)\n\ndf_bkp = df.copy()\n\n## Replace 'inf' and '-inf' with NaN\ndf.replace([np.inf, -np.inf], np.nan, inplace=True)\n\nwith pd.option_context('display.max_columns', None): # setting the max rows\n    display(df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:33:59.599361Z","iopub.execute_input":"2024-12-29T09:33:59.599988Z","iopub.status.idle":"2024-12-29T09:34:16.93558Z","shell.execute_reply.started":"2024-12-29T09:33:59.599929Z","shell.execute_reply":"2024-12-29T09:34:16.93428Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Premium Amount**\nLet's start the analysis with the target column.\n* No null values\n* Max value is 4999\n* Min value is 20.0\n* Values are right skewed","metadata":{}},{"cell_type":"code","source":"# checking values\ndata = [\n    [\"Null count\", df.loc[df.src=='trn','Premium Amount'].isnull().sum()],\n    [\"Null percentage\", round(df.loc[df.src=='trn','Premium Amount'].isnull().mean()*100,2)],\n    [\"Min\", df.loc[df.src=='trn','Premium Amount'].min()],\n    [\"Max\", df.loc[df.src=='trn','Premium Amount'].max()]]\n\nprint(tabulate(data, headers=[\"Attribute\", \"train\"], tablefmt=\"simple_grid\"))\n\ndf['premium_amount_log'] = np.log1p(df['Premium Amount'])\n\n# Create a histogram for column 'Premium Amount'\nfig, axes = plt.subplots(2, 1, figsize=(20, 5)) \n\n_ = sns.histplot(data=df.loc[df.src=='trn',:], x='Premium Amount', bins=100, kde=True, color=\"blue\", ax=axes[0])\n_ = sns.histplot(data=df.loc[df.src=='trn',:], x='premium_amount_log', bins=100, kde=True, color=\"green\", ax=axes[1])\n\n# Add labels and title\n#_ = plt.xlabel('Values')\n#_ = plt.ylabel('Frequency')\n#_ = plt.title('Histogram of Premium Amount')\n\n# Show the plot\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:34:16.937571Z","iopub.execute_input":"2024-12-29T09:34:16.938074Z","iopub.status.idle":"2024-12-29T09:34:29.008531Z","shell.execute_reply.started":"2024-12-29T09:34:16.938024Z","shell.execute_reply":"2024-12-29T09:34:29.006637Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Age**\nWe can notice that\n* Null values are present in the both dataset and % also the same\n* Min Max values are same for both data soruces\n* Distribution is uniform\n* Unfortunatly we can't see a clear correlation between Premium Amount with Age\n\n**Possible Transformation**\n* Imputing null values","metadata":{}},{"cell_type":"code","source":"analyseOrdinal('Age')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:34:29.010492Z","iopub.execute_input":"2024-12-29T09:34:29.010991Z","iopub.status.idle":"2024-12-29T09:34:38.844538Z","shell.execute_reply.started":"2024-12-29T09:34:29.010948Z","shell.execute_reply":"2024-12-29T09:34:38.842903Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Gender**\nWe can notice that\n* Null values are **NOT** present in the both dataset\n* Gender distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation between Premium Amount with Gender\n\n**Possible Transformations**\n* onehot encoding.","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Gender')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:34:38.846751Z","iopub.execute_input":"2024-12-29T09:34:38.84742Z","iopub.status.idle":"2024-12-29T09:34:45.903571Z","shell.execute_reply.started":"2024-12-29T09:34:38.84736Z","shell.execute_reply":"2024-12-29T09:34:45.902027Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Anual Income**\nWe can notice that\n* Null values are present in the both dataset and % is same.\n* Min Max values are same for both data soruces\n* Unfortunatly we can't see a clear correlation between Premium Amount with Annual Income\n\n**Possible Transformation**\n* log transformation\n* min max transformation\n* Impute null values with median or mean","metadata":{"execution":{"iopub.status.busy":"2024-12-16T08:08:13.877132Z","iopub.execute_input":"2024-12-16T08:08:13.878087Z","iopub.status.idle":"2024-12-16T08:08:14.291965Z","shell.execute_reply.started":"2024-12-16T08:08:13.878044Z","shell.execute_reply":"2024-12-16T08:08:14.290697Z"}}},{"cell_type":"code","source":"\nanalyseContinues('Annual Income')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:34:45.905546Z","iopub.execute_input":"2024-12-29T09:34:45.90612Z","iopub.status.idle":"2024-12-29T09:35:03.228214Z","shell.execute_reply.started":"2024-12-29T09:34:45.906061Z","shell.execute_reply":"2024-12-29T09:35:03.226533Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"</details>","metadata":{}},{"cell_type":"markdown","source":"---\n## **Marital Status**\nWe can notice that\n* Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.\n* Introduce a null category","metadata":{"execution":{"iopub.status.busy":"2024-12-16T08:28:52.003611Z","iopub.execute_input":"2024-12-16T08:28:52.004387Z","iopub.status.idle":"2024-12-16T08:28:52.046957Z","shell.execute_reply.started":"2024-12-16T08:28:52.004347Z","shell.execute_reply":"2024-12-16T08:28:52.045878Z"}}},{"cell_type":"code","source":"\nanalyseCategorical('Marital Status')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:35:03.230013Z","iopub.execute_input":"2024-12-29T09:35:03.230461Z","iopub.status.idle":"2024-12-29T09:35:10.624169Z","shell.execute_reply.started":"2024-12-29T09:35:03.230419Z","shell.execute_reply":"2024-12-29T09:35:10.622331Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Number of Dependents**\nWe can notice that\n* Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.\n* Introduce a null category\n* Convert to an Int column from decimal","metadata":{}},{"cell_type":"code","source":"\nanalyseOrdinal('Number of Dependents')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:35:10.626681Z","iopub.execute_input":"2024-12-29T09:35:10.627281Z","iopub.status.idle":"2024-12-29T09:35:17.092481Z","shell.execute_reply.started":"2024-12-29T09:35:10.627221Z","shell.execute_reply":"2024-12-29T09:35:17.091111Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Education Level**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.","metadata":{"execution":{"iopub.status.busy":"2024-12-16T09:02:58.71645Z","iopub.execute_input":"2024-12-16T09:02:58.716974Z","iopub.status.idle":"2024-12-16T09:02:58.724057Z","shell.execute_reply.started":"2024-12-16T09:02:58.716941Z","shell.execute_reply":"2024-12-16T09:02:58.722616Z"}}},{"cell_type":"code","source":"\nanalyseCategorical('Education Level')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:35:17.094193Z","iopub.execute_input":"2024-12-29T09:35:17.094603Z","iopub.status.idle":"2024-12-29T09:35:24.29736Z","shell.execute_reply.started":"2024-12-29T09:35:17.094564Z","shell.execute_reply":"2024-12-29T09:35:24.295634Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Occupation**\n\nWe can notice that\n* Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.\n* Null value impute with new category","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Occupation')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:35:24.299411Z","iopub.execute_input":"2024-12-29T09:35:24.299925Z","iopub.status.idle":"2024-12-29T09:35:31.696408Z","shell.execute_reply.started":"2024-12-29T09:35:24.299875Z","shell.execute_reply":"2024-12-29T09:35:31.695059Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Health Score**\nWe can notice that\n* Null values are present in the both dataset and Null % also the same.\n* Min Max values are same for both data soruces\n* Unfortunatly we can't see a clear correlation between Premium Amount with Annual Income\n\n**Possible Transformation**\n* Impute null values with median or mean","metadata":{}},{"cell_type":"code","source":"\nanalyseContinues('Health Score')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:35:31.701466Z","iopub.execute_input":"2024-12-29T09:35:31.701978Z","iopub.status.idle":"2024-12-29T09:35:49.067301Z","shell.execute_reply.started":"2024-12-29T09:35:31.70193Z","shell.execute_reply":"2024-12-29T09:35:49.065615Z"},"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Location**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Location')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:35:49.069086Z","iopub.execute_input":"2024-12-29T09:35:49.069545Z","iopub.status.idle":"2024-12-29T09:35:55.98202Z","shell.execute_reply.started":"2024-12-29T09:35:49.069483Z","shell.execute_reply":"2024-12-29T09:35:55.980426Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Policy Type**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Policy Type')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:35:55.984185Z","iopub.execute_input":"2024-12-29T09:35:55.984749Z","iopub.status.idle":"2024-12-29T09:36:03.084295Z","shell.execute_reply.started":"2024-12-29T09:35:55.984693Z","shell.execute_reply":"2024-12-29T09:36:03.082617Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Previous Claims**\n\nWe can notice that\n* Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.\n* Convert to int column\n","metadata":{"execution":{"iopub.status.busy":"2024-12-16T11:44:51.265903Z","iopub.execute_input":"2024-12-16T11:44:51.266747Z","iopub.status.idle":"2024-12-16T11:44:51.273184Z","shell.execute_reply.started":"2024-12-16T11:44:51.26671Z","shell.execute_reply":"2024-12-16T11:44:51.271916Z"}}},{"cell_type":"code","source":"\nanalyseOrdinal('Previous Claims')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:03.086435Z","iopub.execute_input":"2024-12-29T09:36:03.087116Z","iopub.status.idle":"2024-12-29T09:36:09.520002Z","shell.execute_reply.started":"2024-12-29T09:36:03.08705Z","shell.execute_reply":"2024-12-29T09:36:09.518589Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Vehicle Age**\n\nWe can notice that\n* Very few Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* onehot encoding.","metadata":{"execution":{"iopub.status.busy":"2024-12-16T12:42:29.076706Z","iopub.execute_input":"2024-12-16T12:42:29.077123Z","iopub.status.idle":"2024-12-16T12:42:29.085558Z","shell.execute_reply.started":"2024-12-16T12:42:29.077089Z","shell.execute_reply":"2024-12-16T12:42:29.083683Z"}}},{"cell_type":"code","source":"\nanalyseOrdinal('Vehicle Age')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:09.522134Z","iopub.execute_input":"2024-12-29T09:36:09.522683Z","iopub.status.idle":"2024-12-29T09:36:17.259735Z","shell.execute_reply.started":"2024-12-29T09:36:09.522627Z","shell.execute_reply":"2024-12-29T09:36:17.258287Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Credit Score**\nWe can notice that\n* Null values are present in the both dataset and Null % also the same.\n* Min Max values are same for both data soruces\n* Unfortunatly we can't see a clear correlation between Premium Amount with Annual Income\n* Distribution is uniform\n\n**Possible Transformation**\n* Impute null values with median or mean","metadata":{}},{"cell_type":"code","source":"\nanalyseContinues('Credit Score')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:17.261688Z","iopub.execute_input":"2024-12-29T09:36:17.262845Z","iopub.status.idle":"2024-12-29T09:36:32.807491Z","shell.execute_reply.started":"2024-12-29T09:36:17.262778Z","shell.execute_reply":"2024-12-29T09:36:32.805344Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Insurance Duration**\n\nWe can notice that\n* Very few Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* Impute Null Values","metadata":{}},{"cell_type":"code","source":"\nanalyseOrdinal('Insurance Duration')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:32.810145Z","iopub.execute_input":"2024-12-29T09:36:32.810854Z","iopub.status.idle":"2024-12-29T09:36:39.912884Z","shell.execute_reply.started":"2024-12-29T09:36:32.810674Z","shell.execute_reply":"2024-12-29T09:36:39.911228Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Policy Start Date**\n\nThis is a date colum hence we will add the date type during the data loading.\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* Generation of a new features like how old the policy","metadata":{}},{"cell_type":"code","source":"\nanalyseDate('Policy Start Date')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:39.916066Z","iopub.execute_input":"2024-12-29T09:36:39.916586Z","iopub.status.idle":"2024-12-29T09:36:41.29837Z","shell.execute_reply.started":"2024-12-29T09:36:39.916544Z","shell.execute_reply":"2024-12-29T09:36:41.296927Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n## **Customer Feedback**\n\nWe can notice that\n* Null values are present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* Missing value imputation","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Customer Feedback')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:41.300167Z","iopub.execute_input":"2024-12-29T09:36:41.300638Z","iopub.status.idle":"2024-12-29T09:36:48.723543Z","shell.execute_reply.started":"2024-12-29T09:36:41.300596Z","shell.execute_reply":"2024-12-29T09:36:48.722098Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n## **Smoking Status**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* .","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Smoking Status')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:48.725332Z","iopub.execute_input":"2024-12-29T09:36:48.725774Z","iopub.status.idle":"2024-12-29T09:36:55.712085Z","shell.execute_reply.started":"2024-12-29T09:36:48.725727Z","shell.execute_reply":"2024-12-29T09:36:55.709941Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Exercise Frequency**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* .","metadata":{}},{"cell_type":"code","source":"\nanalyseCategorical('Exercise Frequency')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:36:55.71435Z","iopub.execute_input":"2024-12-29T09:36:55.714974Z","iopub.status.idle":"2024-12-29T09:37:03.170732Z","shell.execute_reply.started":"2024-12-29T09:36:55.714913Z","shell.execute_reply":"2024-12-29T09:37:03.168212Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## **Property Type**\n\nWe can notice that\n* Null values are **NOT** present\n* Distribution is same for both test and train\n* Unfortunatly we can't see a clear correlation with Premium Amount\n\n**Possible Transformations**\n* .","metadata":{"execution":{"iopub.status.busy":"2024-12-16T16:09:21.312959Z","iopub.execute_input":"2024-12-16T16:09:21.313384Z","iopub.status.idle":"2024-12-16T16:09:21.322238Z","shell.execute_reply.started":"2024-12-16T16:09:21.313346Z","shell.execute_reply":"2024-12-16T16:09:21.320403Z"}}},{"cell_type":"code","source":"\nanalyseCategorical('Property Type')","metadata":{"execution":{"iopub.status.busy":"2024-12-29T09:37:12.091415Z","iopub.status.idle":"2024-12-29T09:37:12.091917Z","shell.execute_reply.started":"2024-12-29T09:37:12.091714Z","shell.execute_reply":"2024-12-29T09:37:12.091736Z"},"trusted":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Transformation**\nLets apply following transformations\n\n* missing value imputation\n* Log transformation\n* Date column removal","metadata":{}},{"cell_type":"code","source":"# Transformations\ndf= df_bkp.copy()\n# Missing value imputation ------------------------------------------------------------\nfill_values = {'Age': 0, \n               'Annual Income': df['Annual Income'].max(), # since very low count there\n               'Marital Status': 'unknown',\n               'Number of Dependents':5.0,\n               'Occupation':'unknown',\n               'Health Score':0.0,\n               'Previous Claims':10.0,\n               'Vehicle Age':0.0,\n               'Credit Score':900,\n               'Insurance Duration':0.0,\n               'Customer Feedback':'unknown'}\n\n\ndf = df.fillna(value=fill_values)\n\n# Log transformation --------------------------------------------------------------------\ndf['annual_income_log'] = np.log1p(df['Annual Income'])\ndf['premium_amount_log'] = np.log1p(df['Premium Amount'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:39:45.700482Z","iopub.execute_input":"2024-12-29T09:39:45.701121Z","iopub.status.idle":"2024-12-29T09:39:47.229607Z","shell.execute_reply.started":"2024-12-29T09:39:45.701074Z","shell.execute_reply":"2024-12-29T09:39:47.227895Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# OneHot encoding\n\n# Select categorical columns\n#cat_cols = ['Gender','Marital Status','Education Level',\n#            'Occupation','Location','Policy Type','Customer Feedback','Smoking Status',\n#            'Exercise Frequency','Property Type']\n\n# Apply OneHotEncoder  --------------------------------------------------------------------\n#ohe = OneHotEncoder(sparse=False, handle_unknown='ignore')\n#ohe_data = ohe.fit_transform(nn_df[cat_cols])\n\n# Get encoded column names\n#ohe_columns = ohe.get_feature_names_out(cat_cols)\n\n# Create a DataFrame for encoded columns\n#ohe_df = pd.DataFrame(ohe_data, columns=ohe_columns)\n\n# Combine with the original numerical columns\n#numerical_df = nn_df.drop(columns=cat_cols)  # Keep non-categorical columns\n#transformed_df = pd.concat([ohe_df, numerical_df], axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:37:12.077337Z","iopub.status.idle":"2024-12-29T09:37:12.077907Z","shell.execute_reply.started":"2024-12-29T09:37:12.077658Z","shell.execute_reply":"2024-12-29T09:37:12.077684Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Features list\nfeature_list = [\n    #'id',\n     'Age',\n     'Gender',\n    # 'Annual Income',\n     'Marital Status',\n     'Number of Dependents',\n     'Education Level',\n     'Occupation',\n     'Health Score',\n     'Location',\n     'Policy Type',\n     'Previous Claims',\n     'Vehicle Age',\n     'Credit Score',\n     'Insurance Duration',\n     'Policy Start Date',\n     'Customer Feedback',\n     'Smoking Status',\n     'Exercise Frequency',\n     'Property Type',\n    # 'Premium Amount',\n    # 'src',\n     'annual_income_log',\n     'premium_amount_log'\n]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:39:56.521726Z","iopub.execute_input":"2024-12-29T09:39:56.522267Z","iopub.status.idle":"2024-12-29T09:39:56.529415Z","shell.execute_reply.started":"2024-12-29T09:39:56.522225Z","shell.execute_reply":"2024-12-29T09:39:56.527951Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Model Training with transformed data**","metadata":{}},{"cell_type":"code","source":"# ML training\n# data set\n#train_df = df.loc[df['src']=='trn',feature_list].sample(frac=0.2, random_state=42) # downsampling to test out many algorithms\ntrain_df = df.loc[df['src']=='trn',feature_list]\n# running autoML\n\nimport h2o\nimport pandas as pd\nfrom h2o.automl import H2OAutoML\nfrom sklearn.metrics import mean_squared_log_error, mean_squared_error\n\n# Initialize H2O cluster\nh2o.init()\n\n# Convert pandas DataFrame to H2OFrame\ntrain_hf = h2o.H2OFrame(train_df)\n\ncat_cols = ['Gender','Marital Status','Education Level','Occupation','Location','Policy Type','Customer Feedback','Smoking Status','Exercise Frequency','Property Type',\n            'Age','Insurance Duration','Vehicle Age','Previous Claims','Number of Dependents']\n\ntrain_hf[cat_cols] = train_hf[cat_cols].asfactor()\n\n\n# Define the target and feature columns\n\n#y = 'Premium Amount'\ny = 'premium_amount_log'\nX = feature_list\n\n# Split the data into training and testing sets\ntrn_hf, val_hf = train_hf.split_frame(ratios=[.8], seed=42)\n\n#h2o.display.toggle_user_tips(False)\n\n# Initialize H2O AutoML model\naml = H2OAutoML(max_models=20, \n                seed=1, \n                max_runtime_secs=1800,  \n                #keep_cross_validation_predictions=False,\n                #verbosity=\"info\",\n                #include_algos=[\"GBM\", \"DRF\", \"XGBoost\", \"DeepLearning\"]\n               )\n\n# Train the model\n_ = aml.train(x=X, y=y, training_frame=trn_hf, validation_frame=val_hf)\n\n\nleaderboard = aml.leaderboard.as_data_frame(use_multi_thread=True)\ndisplay(leaderboard)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T09:40:07.50215Z","iopub.execute_input":"2024-12-29T09:40:07.502684Z","iopub.status.idle":"2024-12-29T09:46:02.461245Z","shell.execute_reply.started":"2024-12-29T09:40:07.50264Z","shell.execute_reply":"2024-12-29T09:46:02.459641Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Submission file generation\nLets generate predictions and a submission file using the transformation identified during the EDA.","metadata":{}},{"cell_type":"code","source":"# Get the leader model (the best model)\nleader_model = aml.leader\n\ntest_df = df[df['src']=='tst']\n# loading test file\ntest_df = test_df.drop(columns=['premium_amount_log'])\ntest_hf = h2o.H2OFrame(test_df)\n\ncat_cols = ['Gender','Marital Status','Education Level','Occupation','Location','Policy Type','Customer Feedback','Smoking Status','Exercise Frequency','Property Type',\n            'Age','Insurance Duration','Vehicle Age','Previous Claims','Number of Dependents']\n\ntest_hf[cat_cols] = test_hf[cat_cols].asfactor()\n\ntest_hf['premium_amount_log'] = leader_model.predict(test_hf)\nsubmission_df = test_hf[['id','premium_amount_log']].as_data_frame(use_multi_thread=True)\nsubmission_df['Premium Amount'] = np.expm1(submission_df['premium_amount_log'])\nsubmission_df['Premium Amount'] = submission_df['Premium Amount'].round(decimals=3)\n\nsubmission_df.to_csv('submission.csv', index=False)\n\nprint(\"submission.csv file generation completed\")\n\n# Shutdown H2O cluster after use\nh2o.cluster().shutdown()","metadata":{"_kg_hide-input":true,"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Conclusion**\n**important features**\nNon of the features we analysed has the correlation with target variables.\n\n**Transformations**\n* All the identified missing values are imputed.\n* Some ordinal features are encoded\n* Annual Income feature is log transformed\n\nStill no improvements for the model outcome. Its time to explore new feature Engineering.\nWe will explore new features in this [feature engineering](https://www.kaggle.com/code/isaranja/s4e12-feature-engineering-by-ama) notebook\n  ","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}