{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:05.62681Z","iopub.execute_input":"2024-12-12T02:29:05.627186Z","iopub.status.idle":"2024-12-12T02:29:06.037523Z","shell.execute_reply.started":"2024-12-12T02:29:05.627149Z","shell.execute_reply":"2024-12-12T02:29:06.036162Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Competition","metadata":{}},{"cell_type":"markdown","source":"**Competition Introduction**\r\n\r\n**Objective**: Predict the medical expenses of insurance customers. This typically involves using given features (such as age, gender, BMI, smoking status, etc.) to predict a continuous variable (insurance charges).\r\n\r\n**Primary Tasks**:\r\n\r\n**Exploratory Data Analysis (EDA)**:\r\n\r\n- **Data Understanding**: Get familiar with the provided dataset, including the types of feature variables, their distribution, and the presence of missing values.\r\n- **Data Cleaning**: Handle missing values, outliers, and convert data types as needed.\r\n- **Feature Engineering**: Create new features or transform existing ones to help the model learn better.\r\n- **Visualization**: Use your imported visualization libraries (like Matplotlib, Seaborn, Plotly) to create charts and understand the distribution of data and relationships between features.\r\n\r\n**Model Selection and Training**:\r\n\r\n- **Baseline Model**: Establish a simple baseline model (like Linear Regression) to compare with subsequent models.\r\n- **Model Selection**: Choose appropriate regression models based on the nature of the data (such as Decision Tree Regression, Random Forest Regression, Gradient Boosting Regression, XGBoost, etc.).\r\n- **Model Training**: Train your model using the training data.\r\n\r\n**Model Evaluation**:\r\n\r\n- **Cross-Validation**: Use cross-validation to assess model performance and prevent overfitting.\r\n- **Metric Selection**: Choose appropriate evaluation metrics like Mean Squared Error (MSE), Root Mean Squared Error (RMSE), R² score, etc.\r\n\r\n**Model Optimization**:\r\n\r\n- **Hyperparameter Tuning**: Use grid search, random search, or Bayesian optimization to fine-tune model hyperparameters.\r\n- **Feature Selection**: Re-evaluate feature selection and importance to determine which features contribute the most to the model.\r\n\r\n**Submitting Predictions**:\r\n\r\n- **Generate Predictions**: Use the test set data to generate predictions.\r\n- **Submission**: Submit your predictions in the format required by Kaggle.\r\n\r\n**Rough Framework**:\r\n\r\n- **Data Preparation**: Includes data loading, cleaning, and feature engineering.\r\n- **Model Building**: Select models, train models, and perform cross-validation.\r\n- **Model Evaluation and Optimization**: Evaluate model performance and optimize hyperparameters.\r\n- **Prediction and Submission**: Use the optimized model to predict on the test set and submit results.\r\n\r\n**Key Considerations**:\r\n\r\n- **Scoring Standard**: Understand the competition's scoring standard, which is typically based on the error between predicted and actual values.\r\n- **Data Leakage**: Avoid using test set data during model training.\r\n- **Overfitting**: Be mindful of model complexity to prevent overfitting.\r\n","metadata":{}},{"cell_type":"markdown","source":"#  Import Libraries","metadata":{}},{"cell_type":"code","source":"# Here are some of the packages that will be used\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.graph_objs as go\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:06.039765Z","iopub.execute_input":"2024-12-12T02:29:06.040389Z","iopub.status.idle":"2024-12-12T02:29:06.904662Z","shell.execute_reply.started":"2024-12-12T02:29:06.04032Z","shell.execute_reply":"2024-12-12T02:29:06.903582Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data viewing and overview","metadata":{}},{"cell_type":"code","source":"#load data\ntrain = pd.read_csv(\"/kaggle/input/playground-series-s4e12/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/playground-series-s4e12/test.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:06.906265Z","iopub.execute_input":"2024-12-12T02:29:06.906998Z","iopub.status.idle":"2024-12-12T02:29:17.45972Z","shell.execute_reply.started":"2024-12-12T02:29:06.906961Z","shell.execute_reply":"2024-12-12T02:29:17.458645Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#View the first 5 rows of data\ntrain.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:17.461009Z","iopub.execute_input":"2024-12-12T02:29:17.46134Z","iopub.status.idle":"2024-12-12T02:29:17.498204Z","shell.execute_reply.started":"2024-12-12T02:29:17.461306Z","shell.execute_reply":"2024-12-12T02:29:17.497192Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Look at the last 5 lines of data\ntrain.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:17.500474Z","iopub.execute_input":"2024-12-12T02:29:17.50079Z","iopub.status.idle":"2024-12-12T02:29:17.523692Z","shell.execute_reply.started":"2024-12-12T02:29:17.50076Z","shell.execute_reply":"2024-12-12T02:29:17.522578Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Why do you need to process the test set?**\r\n\r\n- **Consistency**: The training set and test set should be consistent in terms of feature engineering, data cleaning, etc., to ensure that the model's performance on the test set accurately reflects its performance on real data.\r\n\r\n- **Handling Missing Values**: If there are missing values in the training set, you will likely need to handle these missing values in the test set as well.\r\n\r\n- **Outlier Handling**: The treatment of outliers should be consistent between the training and test sets to avoid bias in the model's performance on the test set.\r\n\r\n- **Feature Engineering**: Any new features you create or transformations you apply to existing features should be applied to the test set in the same way.\r\n\r\n- **Data Type Conversion**: Ensure that the data types in the test set are consistent with those in the training set.\r\n\r\n","metadata":{}},{"cell_type":"code","source":"test.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:17.524844Z","iopub.execute_input":"2024-12-12T02:29:17.525252Z","iopub.status.idle":"2024-12-12T02:29:17.552932Z","shell.execute_reply.started":"2024-12-12T02:29:17.52522Z","shell.execute_reply":"2024-12-12T02:29:17.551795Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:17.554347Z","iopub.execute_input":"2024-12-12T02:29:17.554754Z","iopub.status.idle":"2024-12-12T02:29:17.578672Z","shell.execute_reply.started":"2024-12-12T02:29:17.554719Z","shell.execute_reply":"2024-12-12T02:29:17.577448Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:17.579897Z","iopub.execute_input":"2024-12-12T02:29:17.580239Z","iopub.status.idle":"2024-12-12T02:29:18.242106Z","shell.execute_reply.started":"2024-12-12T02:29:17.580196Z","shell.execute_reply":"2024-12-12T02:29:18.24083Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:18.243465Z","iopub.execute_input":"2024-12-12T02:29:18.243776Z","iopub.status.idle":"2024-12-12T02:29:18.671442Z","shell.execute_reply.started":"2024-12-12T02:29:18.243746Z","shell.execute_reply":"2024-12-12T02:29:18.670304Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Returns the number of rows and columns for the data set\ntrain.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:18.673511Z","iopub.execute_input":"2024-12-12T02:29:18.673927Z","iopub.status.idle":"2024-12-12T02:29:18.680387Z","shell.execute_reply.started":"2024-12-12T02:29:18.673881Z","shell.execute_reply":"2024-12-12T02:29:18.679404Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:18.681844Z","iopub.execute_input":"2024-12-12T02:29:18.682281Z","iopub.status.idle":"2024-12-12T02:29:18.695166Z","shell.execute_reply.started":"2024-12-12T02:29:18.682236Z","shell.execute_reply":"2024-12-12T02:29:18.693979Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#View the number of missing values per column\ntrain.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:18.696602Z","iopub.execute_input":"2024-12-12T02:29:18.697592Z","iopub.status.idle":"2024-12-12T02:29:19.336599Z","shell.execute_reply.started":"2024-12-12T02:29:18.697556Z","shell.execute_reply":"2024-12-12T02:29:19.335501Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:19.337721Z","iopub.execute_input":"2024-12-12T02:29:19.33803Z","iopub.status.idle":"2024-12-12T02:29:19.767502Z","shell.execute_reply.started":"2024-12-12T02:29:19.337992Z","shell.execute_reply":"2024-12-12T02:29:19.766394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data visualization","metadata":{}},{"cell_type":"markdown","source":"Here are some suggestions on how to visualize features, which is a crucial step in data analysis:\r\n\r\n1. **Selective Visualization**\r\n   Not every feature needs to be visualized. You can choose which features to analyze in depth based on their importance, correlation with the target variable, or the complexity of the data. For instance, if a feature has a very low correlation with the target variable, it might not require extensive visualization3\r\n\r\n2. **Key Focus Areas**\r\n   - **Target Variable (Premium Amount)**: Start by thoroughly analyzing the distribution of the target variable, its statistical characteristics (like mean, standard deviation, minimum, maximum, etc.), and its relationships with other features.\r\n   \r\n   - **Numerical Features**: For numerical features:\r\n     - Use histograms or density plots to look at distribution.\r\n     - Box plots or violin plots to compare distributions under different categories.\r\n     - Scatter plots to explore relationships with the target variable or other features.\r\n   \r\n   - **Categorical Features**: For categorical features:\r\n     - Bar charts or pie charts to view category distributions.\r\n     - Box plots or violin plots to compare the distribution of the target variable across different categories.\r\n   \r\n   - **Correlation Matrix**: Draw a heatmap to showcase the correlation betwee4 all features.\r\n\r\n3. **Specific Visualization Recommendations**\r\n   - **Univariate Analysis**:\r\n     - **Histograms**: Show the distribution of numerical features.\r\n     - **Box Plots**: Display central tendency, dispersion, and outliers of the data.\r\n     - **Bar Plots**: For frequency distribution of categorical variables.\r\n   \r\n   - **Bivariate Analysis**:\r\n     - **Scatter Plots**: To examine the relationship between two numerical features.\r\n     - **Box/Violin Plots**: To compare numerical variables across different categories.\r\n     - **Heatmaps**: To display correlation matrices.\r\n   \r\n   - **Multivariate Analysis**:\r\n     - **Scatter Plot Matrix (Pair Plot)**: To simultaneously view relationships among multiple features.\r\n     - **Interactive Charts**: Tools like Plotly can provide more interactivity, such as hovering for data po5nt information, zooming, etc.\r\n\r\n4. **Interactive Visualization**\r\n   Utilize interactive visualization libraries like Plotly, which allow for deeper data exploration. For example, you can hover over data points to see specific information, zoom i6 on charts to view details, etc.\r\n\r\n5. **Focus on Outliers and Missing Values**\r\n   - Check for outliers and visualize how these outliers might impact your model.\r\n   - For missing values, visua7ize their distribution and patterns.\r\n\r\n6. **Transformation of the Target Variable**\r\n   If the target variable (e.g., Premium Amount) is not uniformly distributed (e.g., right-skewed), consider applying transformations like logarith8ic transformation before visualization.\r\n\r\n7. **Visualization After Feature Engineering**\r\n   After feature engineering (like creating new features or transforming existing ones), you should also visualize these new features t9 ensure they are beneficial for the model.\r\n\r\n8. **Summarize and Gain Insights**\r\n   Through visualization, you can gain insights into data distribution, outliers, missing values, feature importance, etc. These insights can guide subsequent data preprocessing, feature selection, and model selection decisions.","metadata":{}},{"cell_type":"markdown","source":"**For the features in the data set, I divide them into numerical features, classification features, and datatime features**","metadata":{}},{"cell_type":"code","source":"# Get numerical and categorical columns\nnum_cols = list(train.select_dtypes(include=['number']).columns)\ncat_cols = list(train.select_dtypes(exclude=['number', 'datetime64[ns]']).columns)\n\n# Define datetime columns\ndatetime_cols = ['Policy Start Date']\n\n#Remove the 'id' column from num_cols\nif 'id' in num_cols:\n    num_cols.remove('id')\n# Remove the 'Policy Start Date' column from cat_cols\nif 'Policy Start Date' in cat_cols:\n    cat_cols.remove('Policy Start Date')\n\n# Print results\nprint(\"Numerical columns:\")\nprint(num_cols)\nprint(\"\\nCategorical columns excluding datetime columns:\")\nprint(cat_cols)\nprint(\"\\nDatetime column:\")\nprint(datetime_cols)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:19.772196Z","iopub.execute_input":"2024-12-12T02:29:19.772649Z","iopub.status.idle":"2024-12-12T02:29:19.895603Z","shell.execute_reply.started":"2024-12-12T02:29:19.772616Z","shell.execute_reply":"2024-12-12T02:29:19.894533Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#View statistical descriptions of numerical features, including the mean, standard deviation, minimum, maximum, and so on\ntrain[num_cols].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:19.896811Z","iopub.execute_input":"2024-12-12T02:29:19.897218Z","iopub.status.idle":"2024-12-12T02:29:20.571577Z","shell.execute_reply.started":"2024-12-12T02:29:19.897174Z","shell.execute_reply":"2024-12-12T02:29:20.570571Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Numerical feature visualization","metadata":{}},{"cell_type":"code","source":"# Set graphic style\nsns.set_style(\"whitegrid\")\n\n# Plot the frequency of each numerical feature\nfor col in num_cols:\n    plt.figure(figsize=(8, 6))\n    sns.histplot(train[col], kde=True)\n    plt.title(f'Frequency Distribution of {col}')\n    plt.xlabel(col)\n    plt.ylabel('Frequency')\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:29:20.572995Z","iopub.execute_input":"2024-12-12T02:29:20.573419Z","iopub.status.idle":"2024-12-12T02:30:09.311151Z","shell.execute_reply.started":"2024-12-12T02:29:20.573375Z","shell.execute_reply":"2024-12-12T02:30:09.310089Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"With this heat map, you can intuitively understand the correlation between numerical features, which is useful for feature selection, identifying multicollinearity, and understanding the effect of features on target variables","metadata":{}},{"cell_type":"code","source":"# Calculate the correlation coefficient matrix\ncorrelation_matrix = train[num_cols].corr()\n\nplt.figure(figsize=(12, 8))\n\nsns.heatmap(correlation_matrix, annot=True, cmap='RdYlBu', linewidths=0.5) \n\nplt.title('Correlation Heatmap of Numerical Features', fontsize=16, fontweight='bold')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:09.312429Z","iopub.execute_input":"2024-12-12T02:30:09.312749Z","iopub.status.idle":"2024-12-12T02:30:10.305842Z","shell.execute_reply.started":"2024-12-12T02:30:09.312717Z","shell.execute_reply":"2024-12-12T02:30:10.304627Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Sorting the correlation between numerical features and Premium Amount can help us more intuitively see which numerical features have the strongest correlation with premium amount, which helps to understand the relationship between data features and target variables","metadata":{}},{"cell_type":"code","source":"\n# Make sure that 'Premium Amount' is listed in num_cols\nif 'Premium Amount' not in num_cols:\n    num_cols.append('Premium Amount')\n\n# Calculate the correlation coefficient matrix\ncorrelation_matrix = train[num_cols].corr()\n\n# Gets the correlation with the Premium Amount\ncorrelation_with_premium = correlation_matrix.loc[:, 'Premium Amount'].drop('Premium Amount', errors='ignore')\n\n# Order the correlation coefficients\nsorted_correlation = correlation_with_premium.sort_values(ascending=False)\n\n# Draw the sorted bar chart\nplt.figure(figsize=(12, 8))\nsns.barplot(x=sorted_correlation.values, y=sorted_correlation.index, palette=\"viridis\")\n\nplt.title('Correlation of Numerical Features with Premium Amount ', fontsize=16, fontweight='bold')\nplt.xlabel('Correlation Coefficient', fontsize=12)\nplt.ylabel('Features', fontsize=12)\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:10.307248Z","iopub.execute_input":"2024-12-12T02:30:10.307605Z","iopub.status.idle":"2024-12-12T02:30:11.268806Z","shell.execute_reply.started":"2024-12-12T02:30:10.307572Z","shell.execute_reply":"2024-12-12T02:30:11.26771Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Classification feature visualization","metadata":{}},{"cell_type":"markdown","source":"The pie chart is used here because it intuitively helps us understand the distribution of different categories within each classification feature in the data set","metadata":{}},{"cell_type":"code","source":"\nsns.set_style(\"whitegrid\")\n\nfor col in cat_cols:\n    counts = train[col].value_counts()\n    total = counts.sum()\n    \n    plt.figure(figsize=(8, 6))\n\n    ax = sns.barplot(x=counts.index, y=counts.values)\n    \n    # Add a percentage label for each bar\n    for p in ax.patches:\n        percentage = f'{100 * p.get_height() / total:.1f}%'\n        x = p.get_x() + p.get_width() / 2\n        y = p.get_height()\n        ax.annotate(percentage, (x, y), ha='center', va='bottom')\n\n    plt.xlabel(col)\n    plt.ylabel('Count')\n    plt.title(f'Distribution of {col}', fontsize=16, fontweight='bold')\n    plt.tight_layout()\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:11.27018Z","iopub.execute_input":"2024-12-12T02:30:11.27055Z","iopub.status.idle":"2024-12-12T02:30:14.780495Z","shell.execute_reply.started":"2024-12-12T02:30:11.270516Z","shell.execute_reply":"2024-12-12T02:30:14.779308Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## datatime features visualization","metadata":{}},{"cell_type":"code","source":"#Converts the Policy Start Date type to datetime\ntrain['Policy Start Date'] = pd.to_datetime(train['Policy Start Date'], errors='coerce')\n\n# View the number of policies per month\nmonthly_policies = train.groupby(train['Policy Start Date'].dt.to_period('M')).size()\n\n# Plot a time series\nimport matplotlib.pyplot as plt\nplt.figure(figsize=(12, 6))\nmonthly_policies.plot()\nplt.title('Number of Policies per Month')\nplt.xlabel('Date')\nplt.ylabel('Number of Policies')\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:14.782084Z","iopub.execute_input":"2024-12-12T02:30:14.782546Z","iopub.status.idle":"2024-12-12T02:30:15.684007Z","shell.execute_reply.started":"2024-12-12T02:30:14.782511Z","shell.execute_reply":"2024-12-12T02:30:15.682778Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"## Missing value processing","metadata":{}},{"cell_type":"code","source":"# Check the number of missing values in each column\nmissing_values_count = train.isnull().sum()\n\n# Output only columns with missing values\nmissing_cols = missing_values_count[missing_values_count > 0]\nif not missing_cols.empty:\n    print(\"Columns with missing values:\")\n    print(missing_cols)\nelse:\n    print(\"No missing values found in any columns.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:15.685177Z","iopub.execute_input":"2024-12-12T02:30:15.685539Z","iopub.status.idle":"2024-12-12T02:30:16.261616Z","shell.execute_reply.started":"2024-12-12T02:30:15.685493Z","shell.execute_reply":"2024-12-12T02:30:16.260427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n# Check for missing values\nmissing_data = train.isnull().sum()\nmissing_percentage = missing_data / len(train) * 100\nprint(missing_percentage)\n\n# Set a threshold for deleting missing values (for example, missing more than 50%)\nthreshold = 50\ncolumns_to_drop = missing_percentage[missing_percentage > threshold].index\ntrain = train.drop(columns=columns_to_drop)\nprint(f\"Deleted column: {columns_to_drop.tolist()}\")\n\n# Fill in the remaining missing values\nfor col in train.columns:\n    if train[col].dtype in ['float64', 'int64']:  \n        if col in ['Age', 'Annual Income', 'Health Score', 'Credit Score']:  \n            # Fill important columns with medians\n            train[col] = train[col].fillna(train[col].median())\n        else:\n            # \n            train[col] = train[col].fillna(train[col].mean())\n    elif train[col].dtype == 'object':  \n        # Fill missing values with the mode\n        train[col] = train[col].fillna(train[col].mode()[0])\n\nprint(\"Missing value after processing:\")\nprint(train.isnull().sum())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:16.262779Z","iopub.execute_input":"2024-12-12T02:30:16.263111Z","iopub.status.idle":"2024-12-12T02:30:19.361172Z","shell.execute_reply.started":"2024-12-12T02:30:16.263081Z","shell.execute_reply":"2024-12-12T02:30:19.360201Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Categorical Transformation","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\n# Encode all class features\ncategorical_columns = train.select_dtypes(include='object').columns\nlabel_encoders = {}\nfor col in categorical_columns:\n    le = LabelEncoder()\n    train[col] = le.fit_transform(train[col].astype(str))  \n    label_encoders[col] = le  ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:19.362576Z","iopub.execute_input":"2024-12-12T02:30:19.362962Z","iopub.status.idle":"2024-12-12T02:30:22.341246Z","shell.execute_reply.started":"2024-12-12T02:30:19.362919Z","shell.execute_reply":"2024-12-12T02:30:22.340096Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Extract datatime","metadata":{}},{"cell_type":"code","source":"#datetime_cols = ['Policy Start Date']\nfor col in datetime_cols:\n    if col in train.columns:\n        \n        train[col] = pd.to_datetime(train[col], errors='coerce')\n        \n        # Extract the year, month, day, day, and so on from the date as features\n        train[f'{col}_Year'] = train[col].dt.year\n        train[f'{col}_Month'] = train[col].dt.month\n        train[f'{col}_Day'] = train[col].dt.day\n        train[f'{col}_Weekday'] = train[col].dt.weekday  \n\n        reference_date = pd.to_datetime('2023-01-01') \n        train[f'{col}_Days_Since'] = (train[col] - reference_date).dt.days\n        \n        train = train.drop(col, axis=1)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:22.342535Z","iopub.execute_input":"2024-12-12T02:30:22.342822Z","iopub.status.idle":"2024-12-12T02:30:22.783105Z","shell.execute_reply.started":"2024-12-12T02:30:22.342794Z","shell.execute_reply":"2024-12-12T02:30:22.782029Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# model","metadata":{}},{"cell_type":"markdown","source":"Here I will try three different models (XGBoost, CatBoost, and LightGBM) to compare their fit effects","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom xgboost import XGBRegressor\nfrom catboost import CatBoostRegressor\nfrom lightgbm import LGBMRegressor\n\n\nX = train.drop('Premium Amount', axis=1)  \ny = train['Premium Amount']  \n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Define evaluation function\ndef evaluate_model(model, X_train, X_test, y_train, y_test):\n    model.fit(X_train, y_train)\n    y_pred = model.predict(X_test)\n    mse = mean_squared_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n    return mse, r2\n\n# XGB\nxgb = XGBRegressor(random_state=42, n_jobs=-1)\nxgb_params = {\n    'n_estimators': [100, 200],\n    'learning_rate': [0.05, 0.1],\n    'max_depth': [3, 5]\n}\nxgb_grid = GridSearchCV(xgb, xgb_params, cv=3, scoring='neg_mean_squared_error', verbose=1)\nxgb_mse, xgb_r2 = evaluate_model(xgb_grid, X_train, X_test, y_train, y_test)\n\n# Cat\ncat = CatBoostRegressor(random_state=42, silent=True)\ncat_params = {\n    'iterations': [100, 200],\n    'learning_rate': [0.05, 0.1],\n    'depth': [3, 5]\n}\ncat_grid = GridSearchCV(cat, cat_params, cv=3, scoring='neg_mean_squared_error', verbose=1)\ncat_mse, cat_r2 = evaluate_model(cat_grid, X_train, X_test, y_train, y_test)\n\n# LGB\nlgb = LGBMRegressor(random_state=42)\nlgb_params = {\n    'n_estimators': [100, 200],\n    'learning_rate': [0.05, 0.1],\n    'max_depth': [-1, 5]\n}\nlgb_grid = GridSearchCV(lgb, lgb_params, cv=3, scoring='neg_mean_squared_error', verbose=1)\nlgb_mse, lgb_r2 = evaluate_model(lgb_grid, X_train, X_test, y_train, y_test)\n\nresults = pd.DataFrame({\n    'Model': ['XGBoost', 'CatBoost', 'LightGBM'],\n    'MSE': [xgb_mse, cat_mse, lgb_mse],\n    'R2': [xgb_r2, cat_r2, lgb_r2]\n}).sort_values(by='MSE')\n\nprint(\"Model performance comparison:\")\nprint(results)\n\n# Select the optimal model\nbest_model_name = results.iloc[0]['Model']\nprint(f\"Optimum model: {best_model_name}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-12T02:30:22.784425Z","iopub.execute_input":"2024-12-12T02:30:22.784734Z"}},"outputs":[],"execution_count":null}]}