{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook aims to illustrate straightforward approaches to implementing FastAI, XGB, and LGBM. Let me know if you find it helpful!\n\nCompleted:\n1. FastAI, XGB, and LGBM Prototype models\n2. Initial Data Cleaning (i.e. missing value management)\n3. Initial Feature Engineering (i.e. deconstructing the policy start date to ML friendly fields)\n\nInitial Results:\n1. XGB - 25% Training Data - 1.064\n2. XGB - Full Training Data - 1.05978\n3. LGBM - Full Training Data - 1.06715\n4. FastAI - Full Training Data - 1.076\n\nNext Steps:\n1. Review and confirm FastAI LossFunc/Metric Configuration\n2. Conduct further EDA and to explore further feature engineering. Explore the confusion matrices for our categorical variables to see if we can with confidence assign any of the individual categories. We could also test alternate models in our imputation but for this playground series I will not be pursuing that.\n3. Implement Optuna for hyperparameter tuning / Test Feature Dropping.","metadata":{}},{"cell_type":"markdown","source":"#Environment Setup\n\nIn this code section, we load and install the appropriate libraries to enable our different models. We also silence warnings which are currently common when using FastAI with Pandas due to upcoming changes to how Pandas manages assignment.","metadata":{}},{"cell_type":"code","source":"#Basic Analysis\nimport numpy as np\nimport pandas as pd\n\n\n#system navigation\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\nfrom pathlib import Path\n\n#ML\nfrom fastai.tabular.all import *\nfrom fastai.callback.tracker import EarlyStoppingCallback\nfrom fastai.metrics import rmse\n\nfrom fastai.metrics import AccumMetric\n\n#Backbone for ML\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.tree import DecisionTreeClassifier, export_graphviz\n\n\nfrom xgboost import XGBRegressor, XGBClassifier\nimport xgboost as xgb\nimport lightgbm as lgb\n\n\nfrom sklearn.metrics import mean_squared_error, accuracy_score, classification_report\nfrom sklearn.model_selection import train_test_split\n\n\n#Silence Future Version Warnings - Pandas is changing and FastAI in particular has not updated their code\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\n\n#EDA\n!pip install -qq sweetviz\nimport sweetviz as sv\n\n#visualization\nimport seaborn as sns\nfrom tabulate import tabulate","metadata":{"_uuid":"cd2a4020-3be7-4308-98ac-8e70a4a12c83","_cell_guid":"6934297b-027f-43a1-bc4f-0116913290d5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:23:48.988362Z","iopub.execute_input":"2024-12-20T16:23:48.988677Z","iopub.status.idle":"2024-12-20T16:24:05.091518Z","shell.execute_reply.started":"2024-12-20T16:23:48.988651Z","shell.execute_reply":"2024-12-20T16:24:05.090328Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We then load our data from the CSVs into Pandas data frames. Pandas is going to enable us to easily handle the data, detect issues, clean it, etc. As a lifelong Excel user, I've come to adore the capabilities of pandas vs limiting yourself to core excel + vba. In this notebook, we will be actively leveraging Pandas to build features, handle data integrity, and of course store and transport values for eventual write to a csv file. ","metadata":{}},{"cell_type":"code","source":"#import data\npath = Path(\"/kaggle/input/playground-series-s4e12\")\ndf = pd.read_csv(path/'train.csv')\ntest_df = pd.read_csv(path/'test.csv')\ndep_var = 'Premium Amount' #Define the dependent variable/target variable\n#Because I will be using the log(dep_var) to simplify my loss metric, we will sequester the original variable\n#name so that as we are saving predictions, we can avoid hard coding the dep_var name into our code\norig_dep_var = dep_var","metadata":{"_uuid":"0112ac2f-baa8-42a3-8cdd-ffaf22a57b36","_cell_guid":"df3f9378-c907-4bb7-b613-db49e32a8807","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:24:05.093281Z","iopub.execute_input":"2024-12-20T16:24:05.094088Z","iopub.status.idle":"2024-12-20T16:24:15.878204Z","shell.execute_reply.started":"2024-12-20T16:24:05.094034Z","shell.execute_reply":"2024-12-20T16:24:15.877086Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This particular data set is a bit heavy. To help accelerate initial prototyping, I suggest taking a sample of training data to reduce runtimes while bug hunting. Be sure to ensure that your sample properly reflects your larger data as appropriate for testing needs (for example, a naive narrow sample, may not reveal that there are missing value in fields with high data integrity)","metadata":{}},{"cell_type":"code","source":"#used for prototyping and automation, remove before final execution\n#df = df.sample(frac=0.25, random_state=73) #fraction is our sample % and random state can be set to enable repeatability\n#test_df = test_df.sample(frac=0.2, random_state=73)","metadata":{"_uuid":"a8391130-377c-4758-a34e-82c2ae87cd61","_cell_guid":"62606e74-c1d7-4feb-8911-fd6cfb4e1c21","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:24:15.880525Z","iopub.execute_input":"2024-12-20T16:24:15.88095Z","iopub.status.idle":"2024-12-20T16:24:15.884636Z","shell.execute_reply.started":"2024-12-20T16:24:15.880913Z","shell.execute_reply":"2024-12-20T16:24:15.88371Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Clean-up and Engineering","metadata":{"_uuid":"3cd9c517-7543-4dee-a80d-4b4d65ba998c","_cell_guid":"c007f5df-54fc-4da8-8f1c-7a25c69e3420","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"#In place of RSMLE we will log the target variable and then reverse out the predictions at the end.\ndef log_target(data_frame,dep_var):\n    data_frame['Premium_Amount_log'] = np.log1p(data_frame[dep_var])\n    data_frame.drop(columns=[dep_var],axis=1, inplace=True)\n    return data_frame\n\ndf = log_target(df,dep_var)\ndep_var = 'Premium_Amount_log'","metadata":{"_uuid":"78fb715a-8c9c-43c0-bfd4-7241c0eb5309","_cell_guid":"32833d32-3de7-404b-bf3d-150d808bd555","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:24:15.885684Z","iopub.execute_input":"2024-12-20T16:24:15.886051Z","iopub.status.idle":"2024-12-20T16:24:16.138545Z","shell.execute_reply.started":"2024-12-20T16:24:15.88602Z","shell.execute_reply":"2024-12-20T16:24:16.137517Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"I love Sweetviz for an initial view of the data. For my personal workflow, I use it specifically for a quick view to understand variables (e.g., skew, cont vs cat assignment, typical values) as well as validate data integrity (though this is easily handled elsewhere in my notebook using pandas for a lighter view).","metadata":{}},{"cell_type":"code","source":"def gen_report(data_frame, target=None):\n    if target:  \n        my_report = sv.analyze([data_frame, 'Data'], target_feat=target)\n    else:\n        my_report = sv.analyze([data_frame, 'Data'])\n    my_report.show_notebook()\n    return my_report","metadata":{"_uuid":"fcbfa4ca-8a49-4eac-8b3c-c38bdb3a30d5","_cell_guid":"abcc12ea-3d41-4f17-92e4-15c90de088d0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:24:27.080052Z","iopub.execute_input":"2024-12-20T16:24:27.080516Z","iopub.status.idle":"2024-12-20T16:24:27.087337Z","shell.execute_reply.started":"2024-12-20T16:24:27.080471Z","shell.execute_reply":"2024-12-20T16:24:27.085839Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Comment out if you are not needing to conduct EDA, it is left in this notebook as a demo\nmy_report = gen_report(df,dep_var)","metadata":{"_uuid":"3c89c458-806f-4f89-a41f-0ba57a577e1a","_cell_guid":"2d43df49-4ab4-4b5c-b006-f3781540c8ee","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:24:28.208466Z","iopub.execute_input":"2024-12-20T16:24:28.208862Z","iopub.status.idle":"2024-12-20T16:29:20.29204Z","shell.execute_reply.started":"2024-12-20T16:24:28.20877Z","shell.execute_reply":"2024-12-20T16:29:20.290837Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#my_report = gen_report(test_df)","metadata":{"_uuid":"1b8c5ab1-ea62-485e-a1f7-43f48f0396c8","_cell_guid":"0c1569cf-adc5-4874-9ef1-286fbf03212d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:17:10.59745Z","iopub.execute_input":"2024-12-19T17:17:10.597742Z","iopub.status.idle":"2024-12-19T17:17:10.609382Z","shell.execute_reply.started":"2024-12-19T17:17:10.597707Z","shell.execute_reply":"2024-12-19T17:17:10.608712Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The timestamp variable included in our data must be broken down to enable ML 'readability'.\n\nWith time variables in particular, you should take a step-back and assess the potential features for inclusion. In our case, we have ~4-5 years of data. With that in mind, and given the industry, it is highly unlikely that we have dynamic pricing that is minute or even hour correlated; perhaps all of our luxury car owners do their insurance shopping between 10pm-11pm or we could split the day into morning or evening, but based on initial considerations, I chose to drop that level of resolution. ","metadata":{"_uuid":"2a573f4d-91d8-4a43-aca4-325799ee55c5","_cell_guid":"74e51a82-ef1c-489c-a9b6-d613ac8c1035","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"#manage Policy Start Date features\ndef extract_date_features(df):\n    # Convert 'Policy Start Date' to datetime if it's not already in datetime format\n    df['Policy Start Date'] = pd.to_datetime(df['Policy Start Date'])\n    # Extract year, month, day, day of the week, hour, etc.\n    \n    #Due to the limited number of years, we could potentially either treat this as a categorical or continuous variable\n    \n    # Normalize 'Year' to make it continuous (relative to the first year in the dataset)\n    df['Year'] = df['Policy Start Date'].dt.year - df['Policy Start Date'].dt.year.min()\n    #Alternate categorical code\n    #df['Year'] = df['Policy Start Date'].dt.year\n    \n    \n    df['Policy Start Month'] = df['Policy Start Date'].dt.month\n    df['Policy Start Day'] = df['Policy Start Date'].dt.day\n    df['Policy Start DayOfWeek'] = df['Policy Start Date'].dt.dayofweek  # 0 = Monday, 6 = Sunday\n    #We cut hour and minute as likely noise\n    #df['Policy Start Hour'] = df['Policy Start Date'].dt.hour\n    #df['Policy Start Minute'] = df['Policy Start Date'].dt.minute\n    \n\n    # Calculate the number of days since the reference date\n    \n    # Extract the first date in the training set as the reference date\n    reference_date = df['Policy Start Date'].min()  # Use the earliest date in the dataset\n    \n    df['Days Since Reference'] = (df['Policy Start Date'] - reference_date).dt.days\n\n    df.drop('Policy Start Date', axis=1, inplace=True)\n    return df\n\n#Convert Previous claims to a categorical variable with the long tail capped\n#Similar code could be applied to manage tails or categorize distinct groups in existing cont variables\ndef bucketize_previous_claims(df):\n    # Create a new category for 5 or more claims\n    df['Previous Claims'] = df['Previous Claims'].apply(lambda x: '5+' if x >= 5 else str(x))\n    return df\n\n#Building functions for data cleansing/engineering is preferred to ensure consistent prep across our training and test sets\n\n#Get our Time Features added to our DFs\ndf = extract_date_features(df)\ntest_df = extract_date_features(test_df)\n\n#Categorize our Previous Claims for our DFs\ndf = bucketize_previous_claims(df)\ntest_df = bucketize_previous_claims(test_df)\n\n","metadata":{"_uuid":"2a51f664-4ce9-4a58-beca-149cc896e729","_cell_guid":"704f4c1f-0ce1-42a3-910d-2a0172cb84eb","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:05.628228Z","iopub.execute_input":"2024-12-20T16:47:05.628712Z","iopub.status.idle":"2024-12-20T16:47:08.156207Z","shell.execute_reply.started":"2024-12-20T16:47:05.628668Z","shell.execute_reply":"2024-12-20T16:47:08.155028Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We will use FastAI's cont_cat_split funcction to take an initial shot at variable assignment. As you will see below, it does a reasonable job but always requires manual review to confirm assignment and usually requires some tweaks to be correct.","metadata":{}},{"cell_type":"code","source":"#Review Automated Assignment\ncont,cat = cont_cat_split(df, 1, dep_var=dep_var)\ncont.remove('id')\nprint(f'Continuous variables: \\n{cont}\\n\\nCategorical Variables: \\n{cat}')","metadata":{"_uuid":"e9fcc195-33bb-4269-ac1f-62af064b1fa9","_cell_guid":"cf8140bc-a116-452a-9ae0-d90d3f7ec06f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:08.799136Z","iopub.execute_input":"2024-12-20T16:47:08.799576Z","iopub.status.idle":"2024-12-20T16:47:08.885161Z","shell.execute_reply.started":"2024-12-20T16:47:08.799539Z","shell.execute_reply":"2024-12-20T16:47:08.883975Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As mentioned, we can see a few misplaced fields. We correct those and then preserve our cont vs cat lists to ensure they remain available for our final notebook section training our full models.","metadata":{}},{"cell_type":"code","source":"#Correct Assignments. FastAI tends to most often mistake cat variables as cont variables; so, be on the lookout.\ncat.append(cont.pop(cont.index('Policy Start DayOfWeek')))\ncat.append(cont.pop(cont.index('Number of Dependents')))\ncat.append(cont.pop(cont.index('Insurance Duration')))\ncat.append(cont.pop(cont.index('Policy Start Month')))\n\nprint(f'Continuous variables: \\n{cont}\\n\\nCategorical Variables: \\n{cat}')\n\n#We will need to retain the complete lists to ensure \ncat_full = cat\ncont_full = cont","metadata":{"_uuid":"00f7bfdb-62c8-490d-9600-fa09de8d0ac1","_cell_guid":"c3aff28b-05a4-4769-a6a3-ff22dad059cc","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:11.139494Z","iopub.execute_input":"2024-12-20T16:47:11.140159Z","iopub.status.idle":"2024-12-20T16:47:11.146722Z","shell.execute_reply.started":"2024-12-20T16:47:11.140085Z","shell.execute_reply":"2024-12-20T16:47:11.145456Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We then review the integrity of our datasets and uncover multiple features missing values.","metadata":{}},{"cell_type":"code","source":"feature_info_df = pd.DataFrame({\n    'Feature': df.columns,\n    'dtype': df.dtypes,\n    '% integrity': 100 * (1 - df.isnull().sum() / len(df))\n}).reset_index(drop=True)\n\n# Convert DataFrame to a tabulated format\nprint(tabulate(feature_info_df, headers='keys', tablefmt='grid', showindex=False))","metadata":{"_uuid":"1839144a-3a31-4676-97d3-3a6cbb5506fc","_cell_guid":"268da854-08dd-47e3-8e71-91085c70c7a3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:13.540448Z","iopub.execute_input":"2024-12-20T16:47:13.540976Z","iopub.status.idle":"2024-12-20T16:47:14.174011Z","shell.execute_reply.started":"2024-12-20T16:47:13.540926Z","shell.execute_reply":"2024-12-20T16:47:14.172858Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The next few cells pertain to early EDA to understand the value our different features provide in predictions. The view was created to make decisions on how to manage missing values in our training and testing data sets.","metadata":{}},{"cell_type":"code","source":"#prepare our data for training our imputation models. This\n# Filter rows with complete data\ncomplete_data = df.dropna()\nprocs = [Categorify, FillMissing, Normalize]\nsplits = RandomSplitter(valid_pct=0.2, seed=42)(range_of(complete_data))\n\nto = TabularPandas(complete_data, \n                    procs = procs, \n                    cat_names = cat, \n                    cont_names = cont,\n                    y_names = dep_var, \n                    y_block = RegressionBlock(),\n                    splits = splits)\n\ndls = to.dataloaders()#bs=128\n\n#create our individual data sets\nxs,y = dls.train.xs,dls.train.y\nvalid_xs,valid_y = dls.valid.xs,dls.valid.y","metadata":{"_uuid":"5e5557a6-1535-4c94-a79e-6143e47b3a8a","_cell_guid":"c5b84108-31e3-424b-9e7b-b2e900d4dda5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:16.523138Z","iopub.execute_input":"2024-12-20T16:47:16.523579Z","iopub.status.idle":"2024-12-20T16:47:18.793892Z","shell.execute_reply.started":"2024-12-20T16:47:16.523537Z","shell.execute_reply":"2024-12-20T16:47:18.792864Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train XGBoost\nmodel = XGBRegressor()\nmodel.fit(xs, y)\n\n# Feature importance\nfeature_importance = model.feature_importances_\nimportance_df = pd.DataFrame({\n    'Feature': xs.columns,\n    'Importance': feature_importance\n}).sort_values(by='Importance', ascending=False)","metadata":{"_uuid":"3486dbb4-23e6-455e-af34-1d5dc0fc861a","_cell_guid":"227fa401-7db9-4db8-8382-5cb5dbb747e3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:20.991773Z","iopub.execute_input":"2024-12-20T16:47:20.992283Z","iopub.status.idle":"2024-12-20T16:47:24.054608Z","shell.execute_reply.started":"2024-12-20T16:47:20.992247Z","shell.execute_reply":"2024-12-20T16:47:24.053403Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds = model.predict(valid_xs)\nrmse = mean_squared_error(valid_y, preds, squared=False)\nprint(f\"Training RMSE: {rmse}\")","metadata":{"_uuid":"8967520c-24aa-40c0-b4d0-db35c0db77ca","_cell_guid":"5f325f4b-7e8e-498a-be02-69c49621e0ee","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-20T16:47:32.890149Z","iopub.execute_input":"2024-12-20T16:47:32.890674Z","iopub.status.idle":"2024-12-20T16:47:33.099224Z","shell.execute_reply.started":"2024-12-20T16:47:32.890633Z","shell.execute_reply":"2024-12-20T16:47:33.097769Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"result = pd.merge(importance_df, feature_info_df, on='Feature', how='left')\nprint(result)","metadata":{"_uuid":"656c79f2-d2dc-4d41-96b3-c9db3115d18b","_cell_guid":"d764d55e-3a0e-4ad3-b3e0-6165ee19018c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:17:17.983178Z","iopub.execute_input":"2024-12-19T17:17:17.983547Z","iopub.status.idle":"2024-12-19T17:17:17.99964Z","shell.execute_reply.started":"2024-12-19T17:17:17.98351Z","shell.execute_reply":"2024-12-19T17:17:17.99895Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Integrity Management","metadata":{}},{"cell_type":"markdown","source":"Based on this initial view. I prioritized my data cleansing in a few ways:\n1. Occupation, Marrital Status, and Feedback will all receive a categorical value representing 'Missing'. I may later impute values for these, especially for marital status since it likely has a correlation to age and customer feedback since it likely correlates to insurance duration. However, due to relatively low feature importance, I am fast tracking for now.\n2. For Vehicle Age, Age, and number of dependents I will impute these values due to their high integrity providing lots of training data. here there are likely correlations that will help provide better judgments on these (such as Age & Marital status -> dependants).\n3. For Annual income and Credit score, of 34k records, only 662 are missing both. so I will impute Annual Income and Credit score. for those missing both, i could create a separate model or simply assign average values. for now, to reduce compute, I will just use the Median value for those missing both.\n4. That leaves health score and prior claims. here again, i think healthscore will be heavily linked with other factors such as age, marital status, dependants, gender. It is essentially and actuarial model. Prior claims also shows correlation with income and credit score.\n\nNote: I left this in to show my thinking at this point in EDA. Code that follows executes on cleaning up gaps based on further exploration.","metadata":{"_uuid":"caab42a1-175d-4937-82f2-d14966b0f313","_cell_guid":"1027fa5f-e654-4626-b04f-498439e2c6ee","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def build_predictor(data_frame,dep_var,model_type,return_dls=False):\n    procs = [Categorify, FillMissing, Normalize]\n    splits = RandomSplitter(valid_pct=0.2, seed=42)(range_of(data_frame))\n    cont, cat = cont_full.copy(), cat_full.copy()\n    if dep_var in cont:\n        cont.remove(dep_var)\n    elif dep_var in cat:\n        cat.remove(dep_var)\n    #cont.remove('Premium_Amount_log') #remove our CORE dep_var\n    #cont.remove('id') #remove the ID column\n    \n    if model_type == 'XGBRegressor':\n        block_type = RegressionBlock()\n    elif model_type == 'XGBClassifier':\n        block_type = CategoryBlock()\n        \n    to = TabularPandas(data_frame, \n                    procs = procs, \n                    cat_names = cat, \n                    cont_names = cont,\n                    y_names = dep_var, \n                    y_block = block_type,#we must select RegressionBlock() or CategoryBlock() depending on needs\n                    splits = splits)\n\n    dls = to.dataloaders()#bs=128\n    #create our individual data sets\n    xs,y = dls.train.xs,dls.train.y\n    valid_xs,valid_y = dls.valid.xs,dls.valid.y\n    #train our model for the selected dep_var\n    if model_type == 'XGBRegressor':\n        bst = XGBRegressor(objective='reg:squarederror', eval_metric='rmse')\n    elif model_type == 'XGBClassifier':\n        bst = XGBClassifier(objective='binary:logistic', eval_metric='error')\n    else:\n        raise ValueError(\"Invalid model_type. Must be 'XGBRegressor' or 'XGBClassifier'.\")\n    # fit model\n    bst.fit(xs, y)\n    # make predictions\n    y_pred = bst.predict(valid_xs)\n\n    if model_type == 'XGBRegressor':\n        rmse = mean_squared_error(valid_y, y_pred, squared=False)\n        average_value = np.mean(valid_y)\n\n        # Print RMSE with average value for context\n        print(f'Variable: {dep_var} | RMSE: {rmse} | Average Target Value: {average_value} | RMSE as a percentage of the average: {rmse / average_value * 100:.2f}%')\n\n    elif model_type == 'XGBClassifier':\n        accuracy = accuracy_score(valid_y, y_pred)\n        print(f'Variable: {dep_var} | Accuracy: {accuracy}')\n    \n    if return_dls:\n        return bst, dls\n    else:\n        return bst  # Return only the model","metadata":{"_uuid":"90997663-01e7-4657-9527-33a16c6710a3","_cell_guid":"9126a237-69fc-4d70-b2c7-c41f768f4f43","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:17:18.000507Z","iopub.execute_input":"2024-12-19T17:17:18.000722Z","iopub.status.idle":"2024-12-19T17:17:18.015865Z","shell.execute_reply.started":"2024-12-19T17:17:18.000692Z","shell.execute_reply":"2024-12-19T17:17:18.015068Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cat = ['Occupation', 'Marital Status', 'Customer Feedback', 'Number of Dependents','Previous Claims']\nbst_models = {}\n\nfor each in cat:\n    bst_models[each] = build_predictor(complete_data, each, 'XGBClassifier')\n\ncont = ['Age', 'Annual Income', 'Credit Score', 'Health Score', 'Vehicle Age']\nbst_models_regressor = {}\n\nfor each in cont:\n    bst_models_regressor[each] = build_predictor(complete_data, each, 'XGBRegressor')","metadata":{"_uuid":"dace21ba-7c3d-498c-bd80-0694f886189a","_cell_guid":"8044ab21-675a-4c00-bedb-0e751793dbcd","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:17:18.018956Z","iopub.execute_input":"2024-12-19T17:17:18.019183Z","iopub.status.idle":"2024-12-19T17:18:37.262069Z","shell.execute_reply.started":"2024-12-19T17:17:18.019163Z","shell.execute_reply":"2024-12-19T17:18:37.261189Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Based on initial outputs our first model will manage missing data as follows:\n1. Categoricals will all be assigned 'Missing' for NA values due to low Accuracy scores.\n2. We will use our models to predict: Age, Credit Score\n3. Median: Annual Income,  Health Score, Vehicle Age, Previous Claims","metadata":{"_uuid":"5c6ab5f8-9702-4fff-b0c1-2c4e7742b8a9","_cell_guid":"f7dbb77f-fec5-4b4a-9d9c-26d4655624ac","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Function to impute categorical variables by filling missing values with 'Missing'\ndef impute_categorical(df, cat_missing):\n    \"\"\" Helper function to fill missing values in categorical columns with 'Missing' \"\"\"\n    for each in cat_missing:\n        df[each].fillna('Missing', inplace=True)  # Fill missing values in the column\n    return df\n\n# Function to impute continuous variables using the median value from df\ndef impute_continuous(df, cont_median):\n    \"\"\" Helper function to fill missing values in continuous columns with the median \"\"\"\n    for each in cont_median:\n        median = df[each].median()  # Calculate the median from df\n        df[each].fillna(median, inplace=True)  # Impute the missing values with the median\n    return df\n\n# Function to apply the trained model to fill missing values for Age and Credit Score\ndef apply_model_to_missing_values(df: pd.DataFrame, models: dict, AgeDLS, ScoreDLS):\n    \"\"\" Helper function to predict and fill missing values using trained models \"\"\"\n    for each, model in models.items():\n        missing_values = df[each].isnull()\n        if missing_values.any(): \n            data_with_missing = df[missing_values].drop(columns=[each])  # Drop the column we are predicting\n            # Apply the corresponding DataLoader for 'Age' or 'Credit Score'\n            if each == 'Age':\n                data_with_missing_dl = AgeDLS.test_dl(data_with_missing)\n            elif each == 'Credit Score':\n                data_with_missing_dl = ScoreDLS.test_dl(data_with_missing)\n            # Predict and fill the missing values\n            predictions = model.predict(data_with_missing_dl.xs)\n            df.loc[missing_values, each] = predictions\n    return df\n\n# Function to train models for Age and Credit Score, and impute missing values\ndef data_integrity(df1: pd.DataFrame, test_df1: pd.DataFrame) -> (pd.DataFrame, pd.DataFrame):\n    # Categorical variables with missing data are handled by filling 'Missing'\n    cat_missing = ['Occupation', 'Marital Status', 'Customer Feedback', 'Number of Dependents', 'Previous Claims']\n    df1 = impute_categorical(df1, cat_missing)  # Fill 'Missing' for categorical variables\n    test_df1 = impute_categorical(test_df1, cat_missing)\n\n    # Continuous variables with median imputation\n    cont_median = ['Annual Income', 'Health Score', 'Vehicle Age', 'Insurance Duration']\n    df1 = impute_continuous(df1, cont_median)  # Impute using median for these continuous variables\n    test_df1 = impute_continuous(test_df1, cont_median)\n\n    # Train models for Age and Credit Score to impute missing values\n    complete_dfAge = df1.dropna(subset=['Age'])  # Drop rows where Age is missing\n    bst_Age, AgeDLS = build_predictor(complete_dfAge, 'Age', 'XGBRegressor', True)\n    \n    complete_dfScore = df1.dropna(subset=['Credit Score'])  # Drop rows where Credit Score is missing\n    bst_Credit_Score, ScoreDLS = build_predictor(complete_dfScore, 'Credit Score', 'XGBRegressor', True)\n    \n    # List of models to use for imputation\n    models = {\n        'Age': bst_Age,\n        'Credit Score': bst_Credit_Score\n    }\n    \n    # Apply the models to fill missing values in df1\n    df1 = apply_model_to_missing_values(df1, models, AgeDLS, ScoreDLS)\n    \n    # Now apply the same model to test_df1\n    test_df1 = apply_model_to_missing_values(test_df1, models, AgeDLS, ScoreDLS)\n\n    return df1, test_df1  # Return both dataframes with missing values filled","metadata":{"_uuid":"ab089ba1-7da2-405d-b742-7455f85d3971","_cell_guid":"6cf62411-2f3b-4427-89bf-74aa21f2fe98","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:37.269585Z","iopub.execute_input":"2024-12-19T17:18:37.269791Z","iopub.status.idle":"2024-12-19T17:18:37.294315Z","shell.execute_reply.started":"2024-12-19T17:18:37.269772Z","shell.execute_reply":"2024-12-19T17:18:37.293418Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df, test_df = data_integrity(df,test_df)","metadata":{"_uuid":"ec93f025-84dc-4a25-a1d2-892de679b899","_cell_guid":"47a40dd4-3d17-4cac-bc6e-77194541399f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:37.295131Z","iopub.execute_input":"2024-12-19T17:18:37.295353Z","iopub.status.idle":"2024-12-19T17:18:53.795458Z","shell.execute_reply.started":"2024-12-19T17:18:37.295334Z","shell.execute_reply":"2024-12-19T17:18:53.794734Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Ensure there are no NaN values left\nprint(f\"Missing values in df: \\n{df.isnull().sum()}\")\nprint(f\"Missing values in test_df: \\n{test_df.isnull().sum()}\")\n\n# Verify that no NaN values remain\nprint(f\"Any NaN in df: {df.isnull().any().any()}\")\nprint(f\"Any NaN in test_df: {test_df.isnull().any().any()}\")","metadata":{"_uuid":"e15fa1f7-141f-492b-8d88-7d3e823682df","_cell_guid":"c7859a9e-b86f-490b-a1d3-baf0bf56c143","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:53.796383Z","iopub.execute_input":"2024-12-19T17:18:53.796689Z","iopub.status.idle":"2024-12-19T17:18:55.605308Z","shell.execute_reply.started":"2024-12-19T17:18:53.796657Z","shell.execute_reply":"2024-12-19T17:18:55.604305Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# FastAI","metadata":{}},{"cell_type":"code","source":"#prepare our data loader\nprocs = [Categorify, FillMissing, Normalize]\nsplits = RandomSplitter(valid_pct=0.2, seed=42)(range_of(df))\n\nto = TabularPandas(df, \n                    procs = procs, \n                    cat_names = cat_full, \n                    cont_names = cont_full,\n                    y_names = dep_var, \n                    y_block = RegressionBlock(),\n                    splits = splits)\n\n\ndls = to.dataloaders()#bs=128\n\n#create our individual data sets\nxs,y = dls.train.xs,dls.train.y\nvalid_xs,valid_y = dls.valid.xs,dls.valid.y","metadata":{"_uuid":"b44a4d4c-2d3d-41ce-8345-cd1024fab88c","_cell_guid":"72a4ebef-06a6-49c9-86c3-b0fb1e802c00","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:55.611687Z","iopub.execute_input":"2024-12-19T17:18:55.611877Z","iopub.status.idle":"2024-12-19T17:18:58.285841Z","shell.execute_reply.started":"2024-12-19T17:18:55.61186Z","shell.execute_reply":"2024-12-19T17:18:58.285119Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"layers = [200, 200]  # Define your architecture layers","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T17:18:58.286583Z","iopub.execute_input":"2024-12-19T17:18:58.28682Z","iopub.status.idle":"2024-12-19T17:18:58.290236Z","shell.execute_reply.started":"2024-12-19T17:18:58.2868Z","shell.execute_reply":"2024-12-19T17:18:58.289447Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn = tabular_learner(dls, \n                        layers=layers, \n                        loss_func=MSELossFlat(), \n                        #metrics=rmse\n                        )","metadata":{"_uuid":"a3a8acd6-fdce-4bb0-a4e5-71c78191acfd","_cell_guid":"a4ef33c2-d75c-44d1-ac5a-c0eeb38bf8e7","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:58.291199Z","iopub.execute_input":"2024-12-19T17:18:58.291537Z","iopub.status.idle":"2024-12-19T17:18:58.349805Z","shell.execute_reply.started":"2024-12-19T17:18:58.291502Z","shell.execute_reply":"2024-12-19T17:18:58.34915Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"var1, var2 = learn.lr_find(suggest_funcs=(slide, valley))\nprint(f'slide = {var1},  valley = {var2}')\nlr= (var1+var2)/2","metadata":{"_uuid":"1dde2a7f-456f-48dd-b02a-0915333902bd","_cell_guid":"0c53b8c9-8a93-4151-be60-0aae3b59c4b5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:18:58.350387Z","iopub.execute_input":"2024-12-19T17:18:58.350605Z","iopub.status.idle":"2024-12-19T17:19:00.974602Z","shell.execute_reply.started":"2024-12-19T17:18:58.350586Z","shell.execute_reply":"2024-12-19T17:19:00.973718Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(learn.metrics)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T17:19:00.975529Z","iopub.execute_input":"2024-12-19T17:19:00.975743Z","iopub.status.idle":"2024-12-19T17:19:00.980163Z","shell.execute_reply.started":"2024-12-19T17:19:00.975723Z","shell.execute_reply":"2024-12-19T17:19:00.9794Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit_one_cycle(\n    500, \n    lr_max=.03,\n    cbs=[EarlyStoppingCallback(monitor='valid_loss', min_delta=0.01, patience=10)]\n)","metadata":{"_uuid":"5f0f5035-e431-4b24-aca4-fadc7de256b2","_cell_guid":"3d02ca72-3e6c-4967-a55e-f5227d48f5c0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T18:03:10.710164Z","iopub.execute_input":"2024-12-19T18:03:10.71061Z","iopub.status.idle":"2024-12-19T18:33:50.981416Z","shell.execute_reply.started":"2024-12-19T18:03:10.710566Z","shell.execute_reply":"2024-12-19T18:33:50.98048Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Recreate the test DataLoader\ntest_dl = dls.test_dl(test_df)\n# After predicting for the test set\ntest_preds, _ = learn.get_preds(dl=test_dl)\n\n# Reverse the log transformation for test predictions\ntest_preds = np.expm1(test_preds)  # Undo the log transformation\n\n# Assign the predictions back to the test dataframe\ntest_df[orig_dep_var] = test_preds\n\n# Create the final output dataframe with the 'id' and predicted 'Premium_Amount'\noutput_df = test_df[['id', orig_dep_var]]\noutput_df.to_csv('FastAI_fullData.csv', index=False)","metadata":{"_uuid":"2ec2b6f3-ae3b-45c1-b2ef-d57cb8668d96","_cell_guid":"399c6d7d-d377-4e9a-8f3f-6ee43befb090","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T17:40:16.652493Z","iopub.execute_input":"2024-12-19T17:40:16.652732Z","iopub.status.idle":"2024-12-19T17:41:07.045756Z","shell.execute_reply.started":"2024-12-19T17:40:16.65271Z","shell.execute_reply":"2024-12-19T17:41:07.044819Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# XGB","metadata":{}},{"cell_type":"code","source":"m = xgb.XGBRegressor(base_score=0.5, booster='gbtree',    \n                       n_estimators=1000,\n                       early_stopping_rounds=100,\n                       objective='reg:squarederror',\n                       max_depth=10,\n                       learning_rate=0.015)\nm.fit(xs, y,\n        eval_set=[(xs, y), (valid_xs, valid_y)],\n        verbose=100)","metadata":{"_uuid":"71454bf3-3c44-4e44-b841-d4f1ba2acce8","_cell_guid":"fe4fc42b-525e-4eb5-8a85-6c528b01b195","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T16:53:40.373404Z","iopub.execute_input":"2024-12-19T16:53:40.373902Z","iopub.status.idle":"2024-12-19T16:56:12.363157Z","shell.execute_reply.started":"2024-12-19T16:53:40.373849Z","shell.execute_reply":"2024-12-19T16:56:12.361867Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_dl = dls.test_dl(test_df)\ntest_preds = m.predict(test_dl.xs)\n\n# Step 2: Reverse the log transformation (if log1p was used)\ntest_preds = np.expm1(test_preds)\n\n# Step 3: Attach the predictions to your test DataFrame\ntest_df[orig_dep_var] = test_preds\n\n# Step 4: Create the final results DataFrame and save to CSV\nrf_results_df = test_df[['id', orig_dep_var]]\nrf_results_df.to_csv('XGB_tree_fullData.csv', index=False)","metadata":{"_uuid":"3eb5cd56-bf55-496b-92a5-13fd57b1c9c0","_cell_guid":"5b17b01c-5350-470c-b2ec-2c04d44818bf","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-12-19T16:56:12.364503Z","iopub.execute_input":"2024-12-19T16:56:12.364869Z","iopub.status.idle":"2024-12-19T16:56:39.841588Z","shell.execute_reply.started":"2024-12-19T16:56:12.364828Z","shell.execute_reply":"2024-12-19T16:56:39.840086Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# LGBM","metadata":{}},{"cell_type":"code","source":"model = lgb.LGBMRegressor(boosting_type = 'gbdt', \n                            n_estimators=1000, \n                            learning_rate =  0.012, \n                            #device='gpu',#uncomment this line when running with accelerator\n                            num_leaves = 250, \n                            subsample_for_bin= 165700, \n                            min_child_samples= 114, \n                            reg_alpha= 2.075e-06, \n                            reg_lambda= 3.839e-07, \n                            colsample_bytree= 0.9634,\n                            subsample= 0.9592, \n                            max_depth= 10,\n                            random_state=0,\n                            verbosity=-1\n                            )\nmodel.fit(xs, y,\n            eval_set=[(valid_xs, valid_y)],\n            callbacks=[lgb.early_stopping(stopping_rounds=100), lgb.log_evaluation(100)]) ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T17:01:10.660613Z","iopub.execute_input":"2024-12-19T17:01:10.661567Z","iopub.status.idle":"2024-12-19T17:03:10.962041Z","shell.execute_reply.started":"2024-12-19T17:01:10.661521Z","shell.execute_reply":"2024-12-19T17:03:10.960757Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_preds = model.predict(test_dl.xs)\n# Step 2: Reverse the log transformation (if log1p was used)\ntest_preds = np.expm1(test_preds)\n\n# Step 3: Attach the predictions to your test DataFrame\ntest_df[orig_dep_var] = test_preds\n\n# Step 4: Create the final results DataFrame and save to CSV\nrf_results_df = test_df[['id', orig_dep_var]]\nrf_results_df.to_csv('LGBM_fullData.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-19T17:03:10.963445Z","iopub.execute_input":"2024-12-19T17:03:10.963864Z","iopub.status.idle":"2024-12-19T17:04:22.947794Z","shell.execute_reply.started":"2024-12-19T17:03:10.963824Z","shell.execute_reply":"2024-12-19T17:04:22.944972Z"}},"outputs":[],"execution_count":null}]}