{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:34.371956Z","iopub.execute_input":"2024-12-23T17:22:34.372524Z","iopub.status.idle":"2024-12-23T17:22:34.384175Z","shell.execute_reply.started":"2024-12-23T17:22:34.372476Z","shell.execute_reply":"2024-12-23T17:22:34.382479Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory Data Analysis\n\nFollowing the guidelines from: \n- https://www.kdnuggets.com/7-steps-to-mastering-exploratory-data-analysis\n- https://www.kaggle.com/discussions/getting-started/194728\n- https://www.kaggle.com/code/bbusath5/ensemble-modeling-with-sklearn-pipelines","metadata":{}},{"cell_type":"markdown","source":"## 1. Data Collection\n\nThe initial step in any data project is having the data itself. This first step is where data is gathered from various sources for subsequent analysis.","metadata":{}},{"cell_type":"code","source":"trainDs = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv', index_col='id')\nX_test = pd.read_csv('/kaggle/input/playground-series-s4e12/test.csv', index_col='id')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:36.966588Z","iopub.execute_input":"2024-12-23T17:22:36.966979Z","iopub.status.idle":"2024-12-23T17:22:49.242708Z","shell.execute_reply.started":"2024-12-23T17:22:36.966946Z","shell.execute_reply":"2024-12-23T17:22:49.241319Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainDs['Policy Start Date'] = pd.to_datetime(trainDs['Policy Start Date'])\nX_test['Policy Start Date'] = pd.to_datetime(X_test['Policy Start Date'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:49.244726Z","iopub.execute_input":"2024-12-23T17:22:49.245089Z","iopub.status.idle":"2024-12-23T17:22:49.933329Z","shell.execute_reply.started":"2024-12-23T17:22:49.245055Z","shell.execute_reply":"2024-12-23T17:22:49.932317Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainDs","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:49.934495Z","iopub.execute_input":"2024-12-23T17:22:49.934781Z","iopub.status.idle":"2024-12-23T17:22:49.9744Z","shell.execute_reply.started":"2024-12-23T17:22:49.934752Z","shell.execute_reply":"2024-12-23T17:22:49.973218Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Summary Statistics\n\nIn data analysis, handling tabular data is quite common. During the analysis of such data, it's often necessary to gain rapid insights into the data's patterns and distribution. \n\nThese initial insights serve as a base for further exploration and in-depth analysis and are known as summary statistics. \n\nThey offer a concise overview of the dataset's distribution and patterns, encapsulated through metrics such as mean, median, mode, variance, standard deviation, range, percentiles, and quartiles.","metadata":{}},{"cell_type":"code","source":"def printStatistics(ds): \n    print('Column classification')\n    numeric_columns = [ c for c in ds.columns if ds[c].dtype in ['float64'] ]\n    date_columns = [ c for c in ds.columns if ds[c].dtype == 'datetime64[ns]' ]\n    categoric_columns = [ c for c in ds.columns if ds[c].dtype == 'object' ]\n    \n    print('numeric:', numeric_columns)\n    print('date:', date_columns)\n    print('categoric: ', categoric_columns)\n\n    print('Numeric statistics')\n    for c in numeric_columns: \n        print(c)\n        print(f\"\\t > Max: {ds[c].max()}\")\n        print(f\"\\t > Min: {ds[c].min()}\")\n        print(f\"\\t > Mean: {ds[c].mean()}\")\n        print(f\"\\t > StdDev: {ds[c].std()}\")\n        print(f\"\\t > Num missing: {sum(ds[c].isna())}\")\n        print(f\"\\t > Num Values:\", ds[c].nunique())\n\n    print('Categoric statistics')\n    for c in categoric_columns: \n        print(c)\n        print(f\"\\t > Num values: {ds[c].nunique()}\")\n        print(f\"\\t > Num missing: {sum(ds[c].isna())}\")\n        print(f\"\\t > Values:\", ds[c].unique())\n\n    print('Datetime statistics')\n    for c in date_columns: \n        print(c)\n        print(f\"\\t > Most recent: {trainDs[c].max()}\")\n        print(f\"\\t > Oldest:\", trainDs[c].min())\n        print(f\"\\t > Num missing: {sum(trainDs[c].isna())}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:49.976769Z","iopub.execute_input":"2024-12-23T17:22:49.977221Z","iopub.status.idle":"2024-12-23T17:22:49.9873Z","shell.execute_reply.started":"2024-12-23T17:22:49.977182Z","shell.execute_reply":"2024-12-23T17:22:49.985904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printStatistics(trainDs)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:22:51.107229Z","iopub.execute_input":"2024-12-23T17:22:51.107643Z","iopub.status.idle":"2024-12-23T17:22:55.532259Z","shell.execute_reply.started":"2024-12-23T17:22:51.107609Z","shell.execute_reply":"2024-12-23T17:22:55.530626Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Data Preparation \n\nBefore starting our exploration, data usually needs to be prepared for further analysis. Data preparation involves transforming, aggregating, or cleaning data using Python's pandas library to suit the needs of your analysis. \n\nThis step is tailored to the data’s structure and can include grouping, appending, merging, sorting, categorizing, and dealing with duplicates.\n\nIn Python, accomplishing this task is facilitated by the pandas library through its various modules. \n\nThe preparation process for tabular data doesn't adhere to a universal method; instead, it's shaped by the specific characteristics of our data, including its rows, columns, data types, and the values it contains.\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Data Visualization\n\nVisualization is a core component of EDA, making complex relationships and trends within the dataset easily comprehensible. \n\nUsing the right charts can help us identify trends within a big dataset and find hidden patterns or outliers. Python offers different libraries for data visualization, including Matplotlib or Seaborn among others. ","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T17:23:03.086419Z","iopub.execute_input":"2024-12-23T17:23:03.087833Z","iopub.status.idle":"2024-12-23T17:23:03.095173Z","shell.execute_reply.started":"2024-12-23T17:23:03.087779Z","shell.execute_reply":"2024-12-23T17:23:03.093612Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def visualizeData(data):     \n    for i, col in enumerate(data.columns):\n        print(\"Visualizing data from column: \", col)\n        plt.figure(i)\n        if data[col].dtype == 'object':\n            plot=sns.histplot(data[col], label=col)\n            for index, item in enumerate(plot.get_xticklabels()):\n                item.set_rotation(45)\n        elif data[col].dtype == 'float64':\n            plot=sns.displot(data[col], label=col)\n        elif data[col].dtype == 'datetime64[ns]': \n            pass\n        plt.show()\n\nvisualizeData(trainDs)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:59:32.812964Z","iopub.execute_input":"2024-12-23T13:59:32.813357Z","iopub.status.idle":"2024-12-23T14:00:04.249993Z","shell.execute_reply.started":"2024-12-23T13:59:32.813324Z","shell.execute_reply":"2024-12-23T14:00:04.248909Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.1 NaN's visualization\n\nLet's see the relation of different NaNs with the target value, this should help with imputation values.","metadata":{}},{"cell_type":"code","source":"%%time \n\n# See the statistics of the targets such that `Annual Income` is Nan\ndef study_nan_distribution(ds, x_col, y_col, delta=1): \n    nan_ds = ds[ds[x_col].isnull()]\n    non_nan_ds = ds[~ds[x_col].isnull()]\n    fig, axes = plt.subplots(1, 3, figsize=(27, 10))\n    sns.histplot(ax=axes[0], data=nan_ds[y_col], label=x_col, color='red').set_title(f\"{y_col} distribution when {x_col} is NaN\")\n    y_nan = nan_ds[y_col]\n    y_nan_set = set([ y // delta * delta for y in y_nan ])\n    selector = [ (t // delta * delta) in y_nan_set for t in non_nan_ds[y_col] ]\n    print('Number of y values considered:', len(y_nan_set))\n    print('Fraction of selected non nan values:', sum(selector) / len(selector))\n    sns.histplot(ax=axes[1], data=non_nan_ds[selector][x_col], label=x_col, color='blue').set_title(f\"{x_col} distribution for values similar to those in the previous plot\")\n    sns.histplot(ax=axes[2], data=ds[x_col], label=x_col, color='green').set_title(f\"{x_col} distribution\")\n    plt.show()\n    \ny_col = 'Premium Amount'\nfor c in trainDs.columns: \n    if c != y_col and trainDs[c].isnull().any(): \n        study_nan_distribution(trainDs, c, y_col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:03:55.536427Z","iopub.execute_input":"2024-12-23T19:03:55.536817Z","iopub.status.idle":"2024-12-23T19:04:39.817419Z","shell.execute_reply.started":"2024-12-23T19:03:55.536781Z","shell.execute_reply":"2024-12-23T19:04:39.815966Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Variable Analysis\n\nVariable analysis can be either univariate, bivariate, or multivariate. Each of them provides insights into the distribution and correlations between the dataset's variables. Techniques vary depending on the number of variables analyzed:","metadata":{}},{"cell_type":"code","source":"target = trainDs['Premium Amount']","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:02:40.463275Z","iopub.execute_input":"2024-12-23T14:02:40.463665Z","iopub.status.idle":"2024-12-23T14:02:40.469152Z","shell.execute_reply.started":"2024-12-23T14:02:40.463631Z","shell.execute_reply":"2024-12-23T14:02:40.467935Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Univariate\n\nThe main focus in univariate analysis is on examining each variable within our dataset on its own. During this analysis, we can uncover insights such as the median, mode, maximum, range, and outliers. \n\nThis type of analysis is applicable to both categorical and numerical variables.","metadata":{}},{"cell_type":"code","source":"trainDs.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:02:42.28514Z","iopub.execute_input":"2024-12-23T14:02:42.28606Z","iopub.status.idle":"2024-12-23T14:02:43.019159Z","shell.execute_reply.started":"2024-12-23T14:02:42.286017Z","shell.execute_reply":"2024-12-23T14:02:43.018143Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Bivariate\n\nBivariate analysis aims to reveal insights between two chosen variables and focuses on understanding the distribution and relationship between these two variables. \n\nAs we analyze two variables at the same time, this type of analysis can be trickier. It can encompass three different pairs of variables: numerical-numerical, numerical-categorical, and categorical-categorical.","metadata":{}},{"cell_type":"code","source":"import imp \ntry: \n    imp.find_module('dython')\nexcept ImportError: \n    !pip install dython","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:02:17.597053Z","iopub.execute_input":"2024-12-23T14:02:17.597497Z","iopub.status.idle":"2024-12-23T14:02:29.745887Z","shell.execute_reply.started":"2024-12-23T14:02:17.597452Z","shell.execute_reply":"2024-12-23T14:02:29.744467Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from dython.nominal import associations\n\ndef bivariateAnalysis(data, target): \n    # Integer values\n    plt.figure(1)\n    integerData = data.loc[:, ['Age', 'Vehicle Age', 'Number of Dependents', 'Previous Claims', 'Insurance Duration']]\n    sns.heatmap(integerData.assign(target=target).corr(), annot=True)\n    plt.figure(2)\n    floatData = data.loc[:, ['Annual Income', 'Health Score', 'Credit Score']]\n    sns.heatmap(floatData.assign(target=target).corr(), annot=True)\n\n    associations(dataset=data.select_dtypes(include='object').assign(target=target), nominal_columns='all',plot=True)\n\nbivariateAnalysis(trainDs, target)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:02:46.141873Z","iopub.execute_input":"2024-12-23T14:02:46.142948Z","iopub.status.idle":"2024-12-23T14:03:05.749023Z","shell.execute_reply.started":"2024-12-23T14:02:46.142892Z","shell.execute_reply":"2024-12-23T14:03:05.747744Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Transformation Pipeline","metadata":{}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin\n\nclass NanColumnTransformer(BaseEstimator, TransformerMixin):        \n    def fit(self, X, y=None):\n        return self\n    \n    def transform(self, X, y=None):\n        X2 = X.copy()\n        for c in X.columns: \n            X2[f\"{c} is NaN\"] = X[c].isna().astype(int)\n        return X2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:12:18.663895Z","iopub.execute_input":"2024-12-23T19:12:18.665248Z","iopub.status.idle":"2024-12-23T19:12:18.673508Z","shell.execute_reply.started":"2024-12-23T19:12:18.66519Z","shell.execute_reply":"2024-12-23T19:12:18.672001Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainDs.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:12:21.189221Z","iopub.execute_input":"2024-12-23T19:12:21.189607Z","iopub.status.idle":"2024-12-23T19:12:21.216524Z","shell.execute_reply.started":"2024-12-23T19:12:21.189578Z","shell.execute_reply":"2024-12-23T19:12:21.215092Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.pipeline import Pipeline \nfrom sklearn.impute import SimpleImputer \nfrom sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, FunctionTransformer, MinMaxScaler, StandardScaler\nfrom sklearn.compose import ColumnTransformer\n\ndef makePreprocessingPipeline(ds):\n    boolean_transformer = Pipeline(steps=[\n        # Discarded due to https://www.kaggle.com/competitions/playground-series-s4e12/discussion/553000#3078972\n        ('add_nan_column', NanColumnTransformer()),\n        ('missing', SimpleImputer(strategy='constant', fill_value=-1)), \n        ('bool2int', FunctionTransformer(lambda x: (x == 'Yes').astype(int))),\n    ])\n\n    age_transformer = Pipeline(steps=[\n        # Discarded due to https://www.kaggle.com/competitions/playground-series-s4e12/discussion/553000#3078972\n        ('add_nan_column', NanColumnTransformer()),\n        ('imputer', SimpleImputer(strategy='constant', fill_value=-1)),\n        ('scaler', StandardScaler()),\n    ])\n\n    count_transformer = Pipeline(steps=[\n        # Discarded due to https://www.kaggle.com/competitions/playground-series-s4e12/discussion/553000#3078972\n        ('add_nan_column', NanColumnTransformer()),\n        ('imputer', SimpleImputer(strategy='constant', fill_value=-1)),\n        ('scaler', StandardScaler()),\n    ])\n\n    float_transformer = Pipeline(steps=[\n        # ('imputer', SimpleImputer(strategy='mean')),\n        # Recommended in: https://www.kaggle.com/competitions/playground-series-s4e12/discussion/552165\n        ('add_nan_column', NanColumnTransformer()),\n        ('imputer', SimpleImputer(strategy='constant', fill_value=-1)),\n        ('scaler', StandardScaler()),\n    ])\n\n    categorical_transformer = Pipeline(steps=[\n        ('add_nan_column', NanColumnTransformer()),\n        ('imputer', SimpleImputer(strategy='constant', fill_value='Not Specified')), \n        # ('ordinal', OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)), \n        # ('scaler', StandardScaler()),\n        # ('scaler', MinMaxScaler()),\n        ('onehot', OneHotEncoder(handle_unknown='ignore',sparse_output=False)), \n    ])\n    \n    columns_to_drop = ['Policy Start Date']\n    boolean_cols = ['Smoking Status']\n    age_cols = ['Age', 'Vehicle Age']\n    count_cols = ['Number of Dependents', 'Previous Claims', 'Insurance Duration']\n    float_cols =  ['Annual Income', 'Health Score', 'Credit Score']\n    categorical_cols = [ c for c in ds.columns if ds[c].dtype == 'object' and c not in boolean_cols and c not in columns_to_drop ]\n    # Discarded due to https://www.kaggle.com/competitions/playground-series-s4e12/discussion/553000#3078972\n    nan_cols = [ f\"{c} is NaN\" for c in ds.columns if c not in columns_to_drop ]\n    # nan_cols = []\n    \n    column_transformer = ColumnTransformer(transformers=[\n        ('column_dropper', 'drop', columns_to_drop),\n        ('age_columns', age_transformer, age_cols),\n        ('count_imputer', count_transformer, count_cols),\n        ('float_imputer', float_transformer, float_cols),\n        ('booleanize', boolean_transformer, boolean_cols),\n        ('categorical', categorical_transformer, categorical_cols),\n    ])\n\n    preprocessor = Pipeline([\n        (\"preprocessor\", column_transformer), \n        # (\"pandarizer\", FunctionTransformer(lambda x: pd.DataFrame(x, columns = age_cols + count_cols + float_cols + boolean_cols + categorical_cols + nan_cols))),\n    ])\n    \n    return preprocessor\n\ntest_ds = trainDs.loc[:10000, ~trainDs.columns.isin(['Premium Amount'])]\nprint('Original shape', test_ds.shape)\npreprocessor = makePreprocessingPipeline(test_ds)\npreprocessor.fit(test_ds)\npreprocessor.transform(test_ds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:12:41.021789Z","iopub.execute_input":"2024-12-23T19:12:41.022226Z","iopub.status.idle":"2024-12-23T19:12:41.406728Z","shell.execute_reply.started":"2024-12-23T19:12:41.022191Z","shell.execute_reply":"2024-12-23T19:12:41.405264Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Modelling\n\nSubmissions are evaluated using the Root Mean Squared Logarithmic Error (RMSLE).\n","metadata":{}},{"cell_type":"markdown","source":"## 1. Split test-train data","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntargetColumn = 'Premium Amount'\ny_all = trainDs[targetColumn]\nX_all = trainDs.drop(targetColumn, axis=1)\nX_train, X_eval, y_train, y_eval = train_test_split(X_all, y_all, test_size=.20)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:21.054774Z","iopub.execute_input":"2024-12-23T19:13:21.055246Z","iopub.status.idle":"2024-12-23T19:13:22.549757Z","shell.execute_reply.started":"2024-12-23T19:13:21.055208Z","shell.execute_reply":"2024-12-23T19:13:22.547973Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Create preprocessor","metadata":{}},{"cell_type":"code","source":"preprocessor = makePreprocessingPipeline(X_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:23.289315Z","iopub.execute_input":"2024-12-23T19:13:23.290489Z","iopub.status.idle":"2024-12-23T19:13:23.297683Z","shell.execute_reply.started":"2024-12-23T19:13:23.290439Z","shell.execute_reply":"2024-12-23T19:13:23.295933Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Create estimators ","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression, Ridge, ElasticNet, SGDRegressor\nfrom sklearn.svm import LinearSVR\nfrom sklearn.neighbors import RadiusNeighborsRegressor\nfrom sklearn.gaussian_process import GaussianProcessRegressor\nfrom sklearn.cross_decomposition import PLSRegression\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.neural_network import MLPRegressor\nfrom xgboost import XGBRegressor\n\nfrom optuna.trial import Trial, FixedTrial\n\ndef makeLinearSimple(params:Trial): \n    return LinearRegression(\n        copy_X = False,\n        n_jobs = -1,\n        fit_intercept = bool(params.suggest_int('fit_intercept', 0, 1)),\n        positive = bool(params.suggest_int('positive', 0, 1)),\n    )\n\nestimators = [\n    # Linear Models: https://scikit-learn.org/1.5/modules/linear_model.html\n    \n    # Attempt to use optune for optimization\n    # ('linear_simple', makeLinearSimple(FixedTrial({'fit_intercept': 1, 'positive': 1}))), \n    \n    ('linear_simple', LinearRegression(n_jobs=-1)), \n    ('linear_ridge', Ridge()), \n    ('linear_elasticnet', ElasticNet()), \n    # Support Vector Machines: https://scikit-learn.org/1.5/modules/svm.html\n    ('svm_linear', LinearSVR()),\n    # Stochastic Gradient Descent: https://scikit-learn.org/1.5/modules/sgd.html\n    ('sgd_simple', SGDRegressor(loss='squared_error', max_iter=100, early_stopping=True)),\n    # Nearest Neighbors: https://scikit-learn.org/1.5/modules/neighbors.html\n    #     Discarded due to a high length in plotting CV estimations\n    # ('neighbors_radius', RadiusNeighborsRegressor(n_jobs=-1, radius=0.5, weights='distance', leaf_size=10)), \n    # Gaussian Processes: https://scikit-learn.org/1.5/modules/gaussian_process.html\n    #     Discarded due to a high memory usage, even with copy_X_train=False\n    #     TODO: investigate why.\n    # ('gauss_gpr', GaussianProcessRegressor()), \n    # Partial Least Squares: https://scikit-learn.org/1.5/modules/cross_decomposition.html\n    ('cross_pls', PLSRegression()),\n    # Decision Trees: https://scikit-learn.org/1.5/modules/tree.html\n    ('decision_tree', DecisionTreeRegressor(criterion='squared_error', max_depth=5)),\n    # Random Forest: https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestRegressor.html\n    ('random_forest', RandomForestRegressor(criterion='squared_error', max_depth=5, n_estimators=10)),\n    # Neural Networks: https://scikit-learn.org/1.5/modules/neural_networks_supervised.html\n    ('neural-net_mlp', MLPRegressor(hidden_layer_sizes=(10,), max_iter=3, early_stopping=True)), \n    # XGBoost: https://xgboost.readthedocs.io/en/stable/parameter.html\n    ('xgb', XGBRegressor(objective='reg:squaredlogerror', eval_metric='rmsle', n_jobs=None)),\n    \n]\n\nestimators = [\n    # Random Forest: https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestRegressor.html\n    ('random_forest', RandomForestRegressor(criterion='squared_error', max_depth=5, n_estimators=10)),\n    # Neural Networks: https://scikit-learn.org/1.5/modules/neural_networks_supervised.html\n    # ('neural-net_mlp', MLPRegressor(hidden_layer_sizes=(10,), max_iter=3, early_stopping=True)), \n    # XGBoost: https://xgboost.readthedocs.io/en/stable/parameter.html\n    ('xgb', XGBRegressor(objective='reg:squaredlogerror', eval_metric='rmsle', n_jobs=None)),\n]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Train first/lightweight version of estimators","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom datetime import datetime\n\ndef makePipelines(preprocessor, estimators, X, y):\n    pipelines = {}\n    for model in estimators:\n        pipeName = model[0]\n        print(f\"Starting training {pipeName}...\")\n        pipe = Pipeline(steps=[('preprocessor', preprocessor), model])\n        t0 = datetime.now()    \n        pipe.fit(X, y)\n        pipelines[pipeName] = pipe\n        t1 = datetime.now()\n        duration = t1 - t0\n        print(f\"\\tFinished {pipeName}, took: {duration.total_seconds()}s\")\n    return pipelines","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:23.999058Z","iopub.execute_input":"2024-12-23T19:13:23.999482Z","iopub.status.idle":"2024-12-23T19:13:24.007892Z","shell.execute_reply.started":"2024-12-23T19:13:23.999447Z","shell.execute_reply":"2024-12-23T19:13:24.00644Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipelines = makePipelines(preprocessor, estimators, X_train, y_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:26.49754Z","iopub.execute_input":"2024-12-23T19:13:26.497987Z","iopub.status.idle":"2024-12-23T19:13:43.39313Z","shell.execute_reply.started":"2024-12-23T19:13:26.497953Z","shell.execute_reply":"2024-12-23T19:13:43.392172Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Initial Scoring","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import KFold,cross_validate\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\ndef plot_estimators(pipelines, X, y, n_splits=5): \n    scorers = []\n    labels = []\n    for name, model in pipelines.items(): \n        print(f\"Cross-validating model {name}...\")\n        labels.append(name)\n        kf = KFold(n_splits)\n        t0 = datetime.now()\n        model_score = cross_validate(model, X, y, scoring={'rmsle': 'neg_root_mean_squared_error'}, cv=kf)\n        scorers.append(model_score)\n        t1 = datetime.now()\n        duration = t1 - t0\n        print(f\"\\tFinished {name} in {duration.total_seconds()}s\")\n\n    score_lists = {'rmsle': [ s['test_rmsle'] for s in scorers] }\n    for i, (title, _list) in enumerate(score_lists.items()): \n        plt.figure(i)\n        positive_scores = [ -1 * l for l in _list ]\n        plot = sns.boxplot(data=positive_scores).set_xticklabels(labels, rotation=45)\n        plt.title(title)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:43.395088Z","iopub.execute_input":"2024-12-23T19:13:43.395691Z","iopub.status.idle":"2024-12-23T19:13:43.407153Z","shell.execute_reply.started":"2024-12-23T19:13:43.395652Z","shell.execute_reply":"2024-12-23T19:13:43.405951Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\n# TODO: select subset of estimators based on lightweight training time\nto_plot = list(pipelines.keys())\n# to_plot = ['linear_simple', 'linear_ridge']\npipelines_to_plot = { name: model for name, model in pipelines.items() if name in to_plot }\nplot_estimators(pipelines_to_plot, X_train, y_train, n_splits=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:13:43.408754Z","iopub.execute_input":"2024-12-23T19:13:43.409105Z","iopub.status.idle":"2024-12-23T19:14:55.352876Z","shell.execute_reply.started":"2024-12-23T19:13:43.409074Z","shell.execute_reply":"2024-12-23T19:14:55.350997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Hyper-parameter Tuning","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n\nclass HyperParameterTuning: \n\n    def __init__(self, pipelines, X, y): \n        self.pipes = pipelines\n        self.data = X \n        self.target = y \n        self._forceRun = False\n\n    def forceRun(self): \n        self._forceRun = True \n\n    def getParameters(self, name): \n        if name not in self.pipes: \n            print(f\"Could not find a pipeline named: '{name}'\")\n            return []\n        return self.pipes[name].named_steps[name].get_params()\n\n    def tuneParamGrid(self, name, param_grid, refit='rmsle', cv=5):\n        # Comment out the next line to allow skipping tuning\n        # self._forceRun = True \n        if not self._forceRun:\n            print('Skipping for performance issues. Call `tuner.forceRun` to activate it.')\n            return \n        self._forceRun = False\n        model = self.pipes[name]\n        param_grid = { f\"{name}__{key}\" : param_grid[key] for key in param_grid.keys() }\n        metrics = {'rmsle': 'neg_root_mean_squared_error'}\n        print(f\"Fine-tuning model {name}...\")\n        t0 = datetime.now()\n        xgbcv = GridSearchCV(model, param_grid, scoring=metrics, refit=refit, cv=cv, return_train_score=True, verbose=2)\n        xgbcv.fit(self.data, self.target)\n        t1 = datetime.now()\n        duration = t1 - t0\n        print(f\"\\tFinished {name} in {duration.total_seconds()}s\")\n    \n        print('best score: ' + str(xgbcv.best_score_))\n        print('best params: ' + str(xgbcv.best_params_))\n        results = pd.DataFrame(xgbcv.cv_results_)\n\n        if len(param_grid) == 1: \n            for i,param in enumerate(param_grid.keys()):\n                param_col = 'param_'+param\n                graph_data = results[[param_col, 'mean_test_'+refit, 'mean_train_'+refit]]\n                graph_data[param_col] = [ self._tupleKey(v) if type(v) is tuple else v for v in graph_data[param_col] ]\n                graph_data = graph_data.rename(columns={'mean_test_'+refit:'test', 'mean_train_'+refit:'train'})\n                graph_data = graph_data.melt('param_'+param, var_name='type', value_name=refit)\n                plt.figure(i)\n                plot = sns.lineplot(x='param_'+param, y=refit, hue='type', data=graph_data)\n        elif len(param_grid) == 2: \n            param1 = list(param_grid.keys())[0]\n            param2 = list(param_grid.keys())[1]\n            graph_data = results[['param_'+param1,'param_'+param2,'mean_test_'+refit]]\n            graph_data = graph_data.pivot(index='param_'+param1, columns='param_'+param2, values='mean_test_'+refit)\n            sns.heatmap(graph_data, annot=True, xticklabels=True, yticklabels=True).set(xlabel=param2, ylabel=param1)\n\n    def _tupleKey(self, t): \n        return '-'.join([str(ti) for ti in t])\n\ntuner = HyperParameterTuning(pipelines, X_train, y_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:14:55.356594Z","iopub.execute_input":"2024-12-23T19:14:55.357213Z","iopub.status.idle":"2024-12-23T19:14:55.376572Z","shell.execute_reply.started":"2024-12-23T19:14:55.35716Z","shell.execute_reply":"2024-12-23T19:14:55.374952Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params = {}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:14:55.37853Z","iopub.execute_input":"2024-12-23T19:14:55.379293Z","iopub.status.idle":"2024-12-23T19:14:55.397563Z","shell.execute_reply.started":"2024-12-23T19:14:55.37923Z","shell.execute_reply":"2024-12-23T19:14:55.395993Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: linear_simple\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('linear_simple')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:06:58.739836Z","iopub.execute_input":"2024-12-23T14:06:58.740242Z","iopub.status.idle":"2024-12-23T14:06:58.747633Z","shell.execute_reply.started":"2024-12-23T14:06:58.74021Z","shell.execute_reply":"2024-12-23T14:06:58.746602Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'positive': [True, False]}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_simple', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T10:27:40.210999Z","iopub.execute_input":"2024-12-23T10:27:40.211378Z","iopub.status.idle":"2024-12-23T10:27:40.217254Z","shell.execute_reply.started":"2024-12-23T10:27:40.211346Z","shell.execute_reply":"2024-12-23T10:27:40.216068Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['linear_simple'] = {\n    # Fine-tuned\n    'positive': False,\n    # Default\n    'copy_X': True, \n    'fit_intercept': True, \n    'n_jobs': -1, \n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T10:27:40.37788Z","iopub.execute_input":"2024-12-23T10:27:40.378263Z","iopub.status.idle":"2024-12-23T10:27:40.384023Z","shell.execute_reply.started":"2024-12-23T10:27:40.378233Z","shell.execute_reply":"2024-12-23T10:27:40.382702Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: linear_ridge\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('linear_ridge')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:38.932096Z","iopub.execute_input":"2024-12-21T10:24:38.93276Z","iopub.status.idle":"2024-12-21T10:24:38.943403Z","shell.execute_reply.started":"2024-12-21T10:24:38.93271Z","shell.execute_reply":"2024-12-21T10:24:38.942457Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'positive': [True, False]}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_ridge', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:38.944614Z","iopub.execute_input":"2024-12-21T10:24:38.944871Z","iopub.status.idle":"2024-12-21T10:24:38.953473Z","shell.execute_reply.started":"2024-12-21T10:24:38.944845Z","shell.execute_reply":"2024-12-21T10:24:38.952504Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'solver': ['auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga', 'lbfgs']}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_ridge', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:38.954741Z","iopub.execute_input":"2024-12-21T10:24:38.955074Z","iopub.status.idle":"2024-12-21T10:24:38.966352Z","shell.execute_reply.started":"2024-12-21T10:24:38.955044Z","shell.execute_reply":"2024-12-21T10:24:38.965223Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'alpha': [ x / 10 for x in range(1, 16)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_ridge', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:38.968129Z","iopub.execute_input":"2024-12-21T10:24:38.968503Z","iopub.status.idle":"2024-12-21T10:24:38.976338Z","shell.execute_reply.started":"2024-12-21T10:24:38.968431Z","shell.execute_reply":"2024-12-21T10:24:38.975403Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_iter': list(range(100, 1001, 100))}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_ridge', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:38.977587Z","iopub.execute_input":"2024-12-21T10:24:38.977862Z","iopub.status.idle":"2024-12-21T10:24:38.992005Z","shell.execute_reply.started":"2024-12-21T10:24:38.977834Z","shell.execute_reply":"2024-12-21T10:24:38.990853Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['linear_ridge'] = {\n    # Fine-tuned\n    'positive': False,\n    'solver': 'saga',\n    'alpha': 1.5,\n    'max_iter': 100,\n    # Default\n    'copy_X': True,\n    'fit_intercept': True,\n    'random_state': None,\n    'tol': 0.0001\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T12:15:30.932512Z","iopub.execute_input":"2024-12-21T12:15:30.932966Z","iopub.status.idle":"2024-12-21T12:15:30.939004Z","shell.execute_reply.started":"2024-12-21T12:15:30.932928Z","shell.execute_reply":"2024-12-21T12:15:30.937668Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: linear_elasticnet\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('linear_elasticnet')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.012694Z","iopub.execute_input":"2024-12-21T10:24:39.013044Z","iopub.status.idle":"2024-12-21T10:24:39.019804Z","shell.execute_reply.started":"2024-12-21T10:24:39.013012Z","shell.execute_reply":"2024-12-21T10:24:39.018829Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'positive': [True, False], 'selection': ['cyclic', 'random']}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_elasticnet', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.021086Z","iopub.execute_input":"2024-12-21T10:24:39.021427Z","iopub.status.idle":"2024-12-21T10:24:39.03529Z","shell.execute_reply.started":"2024-12-21T10:24:39.021384Z","shell.execute_reply":"2024-12-21T10:24:39.03422Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'alpha': [ x / 10 for x in range(1, 16)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_elasticnet', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.036588Z","iopub.execute_input":"2024-12-21T10:24:39.036908Z","iopub.status.idle":"2024-12-21T10:24:39.051479Z","shell.execute_reply.started":"2024-12-21T10:24:39.036877Z","shell.execute_reply":"2024-12-21T10:24:39.050258Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'l1_ratio': [ x / 10 for x in range(0, 11)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_elasticnet', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.052838Z","iopub.execute_input":"2024-12-21T10:24:39.053131Z","iopub.status.idle":"2024-12-21T10:24:39.069628Z","shell.execute_reply.started":"2024-12-21T10:24:39.053099Z","shell.execute_reply":"2024-12-21T10:24:39.068535Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_iter': list(range(100, 1001, 100))}\n# tuner.forceRun()\ntuner.tuneParamGrid('linear_elasticnet', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.071034Z","iopub.execute_input":"2024-12-21T10:24:39.071356Z","iopub.status.idle":"2024-12-21T10:24:39.083529Z","shell.execute_reply.started":"2024-12-21T10:24:39.071324Z","shell.execute_reply":"2024-12-21T10:24:39.08251Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['linear_elasticnet'] = {\n    # Fine-tuned\n    'positive': False, \n    'selection': 'cyclic',\n    'alpha': 1.0,\n    'l1_ratio': 1.0,\n    'max_iter': 100,\n    # Default\n    'copy_X': True,\n    'fit_intercept': True,\n    'positive': False,\n    'precompute': False,\n    'random_state': None,\n    'tol': 0.0001,\n    'warm_start': False,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T12:15:34.478832Z","iopub.execute_input":"2024-12-21T12:15:34.479242Z","iopub.status.idle":"2024-12-21T12:15:34.485738Z","shell.execute_reply.started":"2024-12-21T12:15:34.479206Z","shell.execute_reply":"2024-12-21T12:15:34.484371Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: svm_linear\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('svm_linear')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.097266Z","iopub.execute_input":"2024-12-21T10:24:39.097767Z","iopub.status.idle":"2024-12-21T10:24:39.112968Z","shell.execute_reply.started":"2024-12-21T10:24:39.097723Z","shell.execute_reply":"2024-12-21T10:24:39.111945Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'epsilon': [ 0.1, 0.5, 1.0 ], 'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive']}\n# tuner.forceRun()\ntuner.tuneParamGrid('svm_linear', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.114418Z","iopub.execute_input":"2024-12-21T10:24:39.114843Z","iopub.status.idle":"2024-12-21T10:24:39.125573Z","shell.execute_reply.started":"2024-12-21T10:24:39.114798Z","shell.execute_reply":"2024-12-21T10:24:39.124631Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'tol': [ 1 / 10**x for x in range(0, 7)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('svm_linear', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.126724Z","iopub.execute_input":"2024-12-21T10:24:39.12705Z","iopub.status.idle":"2024-12-21T10:24:39.137796Z","shell.execute_reply.started":"2024-12-21T10:24:39.127021Z","shell.execute_reply":"2024-12-21T10:24:39.136799Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'C': [ x for x in range(1, 5)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('svm_linear', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.139048Z","iopub.execute_input":"2024-12-21T10:24:39.139351Z","iopub.status.idle":"2024-12-21T10:24:39.154938Z","shell.execute_reply.started":"2024-12-21T10:24:39.139322Z","shell.execute_reply":"2024-12-21T10:24:39.153592Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# param_grid={'dual': [ 'auto', True, False ]}\n# tuner.tuneParamGrid('svm_linear', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.156247Z","iopub.execute_input":"2024-12-21T10:24:39.156629Z","iopub.status.idle":"2024-12-21T10:24:39.167692Z","shell.execute_reply.started":"2024-12-21T10:24:39.156588Z","shell.execute_reply":"2024-12-21T10:24:39.16683Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_iter': list(range(100, 1001, 100))}\n# tuner.forceRun()\ntuner.tuneParamGrid('svm_linear', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.169115Z","iopub.execute_input":"2024-12-21T10:24:39.169806Z","iopub.status.idle":"2024-12-21T10:24:39.180427Z","shell.execute_reply.started":"2024-12-21T10:24:39.169757Z","shell.execute_reply":"2024-12-21T10:24:39.179364Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['svm_linear'] = {\n    # Fine-tuned\n    'epsilon': 0.1,\n    'loss': 'squared_epsilon_insensitive',\n    'tol': 0.1,\n    'C': 2.0,\n    'max_iter': 900,\n    # Default\n    'dual': True,\n    'fit_intercept': True,\n    'intercept_scaling': 1.0,\n    'random_state': None,\n    'verbose': 0,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T12:15:37.583112Z","iopub.execute_input":"2024-12-21T12:15:37.583508Z","iopub.status.idle":"2024-12-21T12:15:37.589655Z","shell.execute_reply.started":"2024-12-21T12:15:37.583472Z","shell.execute_reply":"2024-12-21T12:15:37.588432Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: sgd_simple\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.SGDRegressor.html#sklearn.linear_model.SGDRegressor","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('sgd_simple')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.193861Z","iopub.execute_input":"2024-12-21T10:24:39.19425Z","iopub.status.idle":"2024-12-21T10:24:39.205292Z","shell.execute_reply.started":"2024-12-21T10:24:39.194217Z","shell.execute_reply":"2024-12-21T10:24:39.204213Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'loss': ['squared_error', 'epsilon_insensitive', 'squared_epsilon_insensitive']}\n# tuner.forceRun()\ntuner.tuneParamGrid('sgd_simple', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.206701Z","iopub.execute_input":"2024-12-21T10:24:39.207035Z","iopub.status.idle":"2024-12-21T10:24:39.215964Z","shell.execute_reply.started":"2024-12-21T10:24:39.206989Z","shell.execute_reply":"2024-12-21T10:24:39.214905Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'alpha': [ 1 / 10**x for x in range(1, 7)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('sgd_simple', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.217489Z","iopub.execute_input":"2024-12-21T10:24:39.217917Z","iopub.status.idle":"2024-12-21T10:24:39.227719Z","shell.execute_reply.started":"2024-12-21T10:24:39.217872Z","shell.execute_reply":"2024-12-21T10:24:39.226643Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_iter': list(range(1000, 5001, 1000))}\n# tuner.forceRun()\ntuner.tuneParamGrid('sgd_simple', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.228948Z","iopub.execute_input":"2024-12-21T10:24:39.229372Z","iopub.status.idle":"2024-12-21T10:24:39.239959Z","shell.execute_reply.started":"2024-12-21T10:24:39.229333Z","shell.execute_reply":"2024-12-21T10:24:39.238929Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['sgd_simple'] = {\n    # Fine-tuned\n    'loss': 'squared_error',\n    'alpha': 0.0001,\n    'max_iter': 1000,\n    # Default\n    'average': False,\n    'early_stopping': True,\n    'epsilon': 0.1,\n    'eta0': 0.01,\n    'fit_intercept': True,\n    'l1_ratio': 0.15,\n    'learning_rate': 'invscaling',\n    'n_iter_no_change': 5,\n    'penalty': 'l2',\n    'power_t': 0.25,\n    'random_state': None,\n    'shuffle': True,\n    'tol': 0.001,\n    'validation_fraction': 0.1,\n    'verbose': 0,\n    'warm_start': False,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T12:15:40.614548Z","iopub.execute_input":"2024-12-21T12:15:40.614992Z","iopub.status.idle":"2024-12-21T12:15:40.621661Z","shell.execute_reply.started":"2024-12-21T12:15:40.614955Z","shell.execute_reply":"2024-12-21T12:15:40.620379Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: decision_tree\nSource: https://scikit-learn.org/1.2/modules/generated/sklearn.tree.DecisionTreeRegressor.html","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('decision_tree')\nn_samples, n_features = trainDs.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.254387Z","iopub.execute_input":"2024-12-21T10:24:39.254704Z","iopub.status.idle":"2024-12-21T10:24:39.268926Z","shell.execute_reply.started":"2024-12-21T10:24:39.254673Z","shell.execute_reply":"2024-12-21T10:24:39.267834Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_depth': list(range(5, n_features + 1, 5))}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.270284Z","iopub.execute_input":"2024-12-21T10:24:39.270705Z","iopub.status.idle":"2024-12-21T10:24:39.281652Z","shell.execute_reply.started":"2024-12-21T10:24:39.270674Z","shell.execute_reply":"2024-12-21T10:24:39.280582Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_features': list(range(2, n_features + 1))}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.282894Z","iopub.execute_input":"2024-12-21T10:24:39.283231Z","iopub.status.idle":"2024-12-21T10:24:39.294842Z","shell.execute_reply.started":"2024-12-21T10:24:39.283201Z","shell.execute_reply":"2024-12-21T10:24:39.293736Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import math\nparam_grid={'min_samples_split': list(range(1, int(math.log(n_samples) + 1)))}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.296143Z","iopub.execute_input":"2024-12-21T10:24:39.29651Z","iopub.status.idle":"2024-12-21T10:24:39.310434Z","shell.execute_reply.started":"2024-12-21T10:24:39.296468Z","shell.execute_reply":"2024-12-21T10:24:39.309307Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'min_samples_leaf': list(range(1, int(math.log(n_samples) + 1)))}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.311931Z","iopub.execute_input":"2024-12-21T10:24:39.312273Z","iopub.status.idle":"2024-12-21T10:24:39.324011Z","shell.execute_reply.started":"2024-12-21T10:24:39.312243Z","shell.execute_reply":"2024-12-21T10:24:39.322785Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_leaf_nodes': [ 10 ** e for e in range(5) ]}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.325213Z","iopub.execute_input":"2024-12-21T10:24:39.325548Z","iopub.status.idle":"2024-12-21T10:24:39.339083Z","shell.execute_reply.started":"2024-12-21T10:24:39.325497Z","shell.execute_reply":"2024-12-21T10:24:39.337703Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'ccp_alpha': [ x / 10 for x in range(0, 50, 5)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('decision_tree', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.340704Z","iopub.execute_input":"2024-12-21T10:24:39.341016Z","iopub.status.idle":"2024-12-21T10:24:39.351705Z","shell.execute_reply.started":"2024-12-21T10:24:39.340986Z","shell.execute_reply":"2024-12-21T10:24:39.350707Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['decision_tree'] = {\n    # Fine-tuned\n    'max_depth': 10,\n    'max_features': 18,\n    'min_samples_split': 2,\n    'min_samples_leaf': 1,\n    'max_leaf_nodes': 100,\n    'ccp_alpha': 0.0,\n    # Default\n    'criterion': 'squared_error',\n    'min_impurity_decrease': 0.0,\n    'min_weight_fraction_leaf': 0.0,\n    'random_state': None,\n    'splitter': 'best',\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T12:15:44.023166Z","iopub.execute_input":"2024-12-21T12:15:44.023601Z","iopub.status.idle":"2024-12-21T12:15:44.031133Z","shell.execute_reply.started":"2024-12-21T12:15:44.023563Z","shell.execute_reply":"2024-12-21T12:15:44.029863Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: random_forest\nSource: https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestRegressor.html","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('random_forest')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.368229Z","iopub.execute_input":"2024-12-21T10:24:39.368685Z","iopub.status.idle":"2024-12-21T10:24:39.380967Z","shell.execute_reply.started":"2024-12-21T10:24:39.368635Z","shell.execute_reply":"2024-12-21T10:24:39.379857Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"n_samples, n_features = trainDs.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.382193Z","iopub.execute_input":"2024-12-21T10:24:39.38252Z","iopub.status.idle":"2024-12-21T10:24:39.390064Z","shell.execute_reply.started":"2024-12-21T10:24:39.382489Z","shell.execute_reply":"2024-12-21T10:24:39.389099Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'n_estimators': [10, 50, 100, 200]}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T10:24:39.391144Z","iopub.execute_input":"2024-12-21T10:24:39.391405Z","execution_failed":"2024-12-21T11:58:25.397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_depth': list(range(5, n_features + 1, 5))}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.399Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import math\nparam_grid={'min_samples_split': list(range(1, int(math.log(n_samples) + 1)))}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.398Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'min_samples_leaf': list(range(1, int(math.log(n_samples) + 1)))}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.398Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_leaf_nodes': [ 10 ** e for e in range(5) ]}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.399Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'ccp_alpha': [ x / 10 for x in range(0, 50, 5)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('random_forest', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.398Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['random_forest'] = {\n    # Fine-tuned\n    'n_estimators': 100,\n    'max_depth': 10,\n    'min_samples_split': 8,\n    'min_samples_leaf': 7,\n    'max_leaf_nodes': 1000,\n    'ccp_alpha': 0.0,\n    # Default\n    'bootstrap': True,\n    'criterion': 'squared_error',\n    'max_features': 1.0,\n    'max_samples': None,\n    'min_impurity_decrease': 0.0,\n    'min_weight_fraction_leaf': 0.0,\n    'n_jobs': None,\n    'oob_score': False,\n    'random_state': None,\n    'verbose': 0,\n    'warm_start': False,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-21T14:26:30.946373Z","iopub.execute_input":"2024-12-21T14:26:30.94675Z","iopub.status.idle":"2024-12-21T14:26:30.952885Z","shell.execute_reply.started":"2024-12-21T14:26:30.946718Z","shell.execute_reply":"2024-12-21T14:26:30.95161Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: neural-net_mlp\nSource: https://scikit-learn.org/1.2/modules/generated/sklearn.neural_network.MLPRegressor.html","metadata":{}},{"cell_type":"code","source":"tuner.getParameters('neural-net_mlp')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:16.816223Z","iopub.execute_input":"2024-12-20T16:29:16.816659Z","iopub.status.idle":"2024-12-20T16:29:16.823821Z","shell.execute_reply.started":"2024-12-20T16:29:16.816626Z","shell.execute_reply":"2024-12-20T16:29:16.822637Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'alpha': [ 10**-e for e in range(1, 10, 2)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:17.020439Z","iopub.execute_input":"2024-12-20T16:29:17.020816Z","iopub.status.idle":"2024-12-20T16:29:17.026711Z","shell.execute_reply.started":"2024-12-20T16:29:17.020787Z","shell.execute_reply":"2024-12-20T16:29:17.025668Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'max_iter': [10, 50, 100, 500, 1000] }\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:17.257176Z","iopub.execute_input":"2024-12-20T16:29:17.25792Z","iopub.status.idle":"2024-12-20T16:29:17.263291Z","shell.execute_reply.started":"2024-12-20T16:29:17.257881Z","shell.execute_reply":"2024-12-20T16:29:17.262237Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'early_stopping': [ True, False ]}\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:17.461086Z","iopub.execute_input":"2024-12-20T16:29:17.462213Z","iopub.status.idle":"2024-12-20T16:29:17.467458Z","shell.execute_reply.started":"2024-12-20T16:29:17.462171Z","shell.execute_reply":"2024-12-20T16:29:17.46644Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'beta_1': [ 0, 0.3, 0.6, 0.9 ], 'beta_2': [ 0, 0.333, 0.666, 0.999 ]}\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:17.665468Z","iopub.execute_input":"2024-12-20T16:29:17.666238Z","iopub.status.idle":"2024-12-20T16:29:17.671893Z","shell.execute_reply.started":"2024-12-20T16:29:17.666195Z","shell.execute_reply":"2024-12-20T16:29:17.670576Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'epsilon': [ 10**-e for e in range(1, 10, 2)]}\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:19.212674Z","iopub.execute_input":"2024-12-20T16:29:19.213504Z","iopub.status.idle":"2024-12-20T16:29:19.218824Z","shell.execute_reply.started":"2024-12-20T16:29:19.213463Z","shell.execute_reply":"2024-12-20T16:29:19.21775Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'n_iter_no_change': list(range(10, 50, 10))}\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:19.404105Z","iopub.execute_input":"2024-12-20T16:29:19.404916Z","iopub.status.idle":"2024-12-20T16:29:19.410391Z","shell.execute_reply.started":"2024-12-20T16:29:19.404877Z","shell.execute_reply":"2024-12-20T16:29:19.409268Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'hidden_layer_sizes': [ (s, ) for s in [100, 200, 500, 1000, 2000, 3000] ] }\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:19.639496Z","iopub.execute_input":"2024-12-20T16:29:19.639891Z","iopub.status.idle":"2024-12-20T16:29:19.645569Z","shell.execute_reply.started":"2024-12-20T16:29:19.639857Z","shell.execute_reply":"2024-12-20T16:29:19.644592Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'hidden_layer_sizes': [ (20, s, 100, ) for s in [100, 500, 1000, 3000 ] ] }\n# tuner.forceRun()\ntuner.tuneParamGrid('neural-net_mlp', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T16:29:20.917262Z","iopub.execute_input":"2024-12-20T16:29:20.918275Z","iopub.status.idle":"2024-12-20T16:29:20.923807Z","shell.execute_reply.started":"2024-12-20T16:29:20.918235Z","shell.execute_reply":"2024-12-20T16:29:20.922786Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['neural-net_mlp'] = {\n    # Fine-tuned\n    'alpha': 1e-07,\n    'max_iter': 1000,\n    'early_stopping': False,\n    'beta_1': 0.0,\n    'beta_2': 0.999,\n    'epsilon': 1e-07,\n    'n_iter_no_change': 30,\n    # 'hidden_layer_sizes': (3000,), # -> -863.3391011879818\n    'hidden_layer_sizes': (20, 3000, 100), # -> -861.6904875191443\n    # Default\n    'activation': 'relu',\n    'batch_size': 'auto',\n    'learning_rate': 'constant',\n    'learning_rate_init': 0.001,\n    'max_fun': 15000,\n    'momentum': 0.9,\n    'nesterovs_momentum': True,\n    'power_t': 0.5,\n    'random_state': None,\n    'shuffle': True,\n    'solver': 'adam',\n    'tol': 0.0001,\n    'validation_fraction': 0.1,\n    'verbose': False,\n    'warm_start': False,\n}","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T19:28:50.964Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Parameter Tuning: xgb\nSource: https://xgboost.readthedocs.io/en/stable/parameter.html","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"tuner.getParameters('xgb')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:38:45.824811Z","iopub.execute_input":"2024-12-23T14:38:45.825719Z","iopub.status.idle":"2024-12-23T14:38:45.833754Z","shell.execute_reply.started":"2024-12-23T14:38:45.825663Z","shell.execute_reply":"2024-12-23T14:38:45.832584Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'n_estimators': [1, 10, 100, 1000, 10000]}\n# tuner.forceRun()\ntuner.tuneParamGrid('xgb', param_grid, cv=2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T14:39:50.95319Z","iopub.execute_input":"2024-12-23T14:39:50.953581Z","iopub.status.idle":"2024-12-23T14:51:06.083143Z","shell.execute_reply.started":"2024-12-23T14:39:50.953546Z","shell.execute_reply":"2024-12-23T14:51:06.081927Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'n_estimators': [10, 50, 100, 500, 1000]}\n# tuner.forceRun()\ntuner.tuneParamGrid('xgb', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.4Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'learning_rate': [0.001, 0.01, 0.1, 0.3, 0.5]}\n# tuner.forceRun()\ntuner.tuneParamGrid('xgb', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.4Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'reg_alpha': [0.001, 0.5, 1, 5, 10, 50, 100, 500]}\n# tuner.forceRun()\ntuner.tuneParamGrid('xgb', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.4Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"param_grid={'reg_lambda': [0.001, 0.5, 1, 5, 10, 50, 100, 500]}\n# tuner.forceRun()\ntuner.tuneParamGrid('xgb', param_grid, cv=5)","metadata":{"trusted":true,"execution":{"execution_failed":"2024-12-21T11:58:25.4Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"best_params['xgb'] = {\n    # Defined by the competition\n    'objective': 'reg:squaredlogerror',\n    'eval_metric': 'rmsle',\n    # Fine-tuned\n    'max_depth': 8,\n    'n_estimators': 1000,\n    'learning_rate': 0.5,\n    'reg_alpha': 0.001,\n    'reg_lambda': 1,\n    'missing': -1, # Decided as of: https://www.kaggle.com/competitions/playground-series-s4e12/discussion/552165\n    # Default\n    'base_score': None,\n    'booster': None,\n    'callbacks': None,\n    'colsample_bylevel': None,\n    'colsample_bynode': None,\n    'colsample_bytree': None,\n    'device': None,\n    'early_stopping_rounds': None,\n    'enable_categorical': False,\n    'feature_types': None,\n    'gamma': None,\n    'grow_policy': None,\n    'importance_type': None,\n    'interaction_constraints': None,\n    'max_bin': None,\n    'max_cat_threshold': None,\n    'max_cat_to_onehot': None,\n    'max_delta_step': None,\n    'max_leaves': None,\n    'min_child_weight': None,\n    'monotone_constraints': None,\n    'multi_strategy': None,\n    'n_jobs': None,\n    'num_parallel_tree': None,\n    'random_state': None,\n    'sampling_method': None,\n    'scale_pos_weight': None,\n    'subsample': None,\n    'tree_method': None,\n    'validate_parameters': None,\n    'verbosity': None,\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:15:15.352152Z","iopub.execute_input":"2024-12-23T19:15:15.352715Z","iopub.status.idle":"2024-12-23T19:15:15.361818Z","shell.execute_reply.started":"2024-12-23T19:15:15.352676Z","shell.execute_reply":"2024-12-23T19:15:15.360281Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Scores after fine-tuning","metadata":{}},{"cell_type":"code","source":"%%time\n\nfineTuned_preprocessor = makePreprocessingPipeline(X_train)\n\nfineTuned_estimators = [\n    # ('linear_simple', LinearRegression(**best_params['linear_simple'])), \n    # ('linear_ridge', Ridge(**best_params['linear_ridge'])), \n    # ('linear_elasticnet', ElasticNet(**best_params['linear_elasticnet'])), \n    # ('svm_linear', LinearSVR(**best_params['svm_linear'])),\n    # ('sgd_simple', SGDRegressor(**best_params['sgd_simple'])),\n    # ('cross_pls', PLSRegression()), # No fine-tuning available\n    # ('decision_tree', DecisionTreeRegressor(**best_params['decision_tree'])),\n    ('random_forest', RandomForestRegressor(**best_params['random_forest'])),\n    # ('neural-net_mlp', MLPRegressor(**best_params['neural-net_mlp'])), # Too long to train, bad results anyway\n    ('xgb', XGBRegressor(**best_params['xgb'])),\n]\n\nfineTuned_pipelines = makePipelines(\n    fineTuned_preprocessor, \n    fineTuned_estimators, \n    X_train, \n    y_train\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:15:16.177177Z","iopub.execute_input":"2024-12-23T19:15:16.177585Z","iopub.status.idle":"2024-12-23T19:15:59.771385Z","shell.execute_reply.started":"2024-12-23T19:15:16.177553Z","shell.execute_reply":"2024-12-23T19:15:59.77019Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\nto_plot = ['random_forest', 'xgb']\nplot_estimators(\n    { name: model for name, model in fineTuned_pipelines.items() if name in to_plot },\n    X_train, \n    y_train, \n    n_splits=5\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T19:15:59.773255Z","iopub.execute_input":"2024-12-23T19:15:59.7736Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Ensemble","metadata":{}},{"cell_type":"markdown","source":"### 5.1 Learning Curves","metadata":{}},{"cell_type":"code","source":"%%time\nfrom sklearn.model_selection import learning_curve\nfrom sklearn.model_selection import ShuffleSplit\n\ndef plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,\n                        n_jobs=-1, train_sizes=np.linspace(.1, 1.0, 5)):\n    '''Generate a simple plot of the test and training learning curve'''\n    plt.figure()\n    plt.title(title)\n    if ylim is not None:\n        plt.ylim(*ylim)\n    plt.xlabel(\"Training examples\")\n    plt.ylabel(\"Score\")\n    train_sizes, train_scores, test_scores = learning_curve(\n        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)\n    train_scores_mean = np.mean(train_scores, axis=1)\n    train_scores_std = np.std(train_scores, axis=1)\n    test_scores_mean = np.mean(test_scores, axis=1)\n    test_scores_std = np.std(test_scores, axis=1)\n    plt.grid()\n\n    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n                     train_scores_mean + train_scores_std, alpha=0.1,\n                     color=\"r\")\n    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n                     test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n    plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n             label=\"Training score\")\n    plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n             label=\"Cross-validation score\")\n\n    plt.legend(loc=\"best\")\n    return plt\n\nto_plot = [ 'random_forest', 'xgb']\nfor name, model in fineTuned_pipelines.items():\n    if name in to_plot:\n        g = plot_learning_curve(\n            model,\n            name + ' learning curves',\n            X_train,\n            y_train,\n            cv=KFold(5),\n            n_jobs=4\n        )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T10:43:20.859289Z","iopub.execute_input":"2024-12-23T10:43:20.859697Z","iopub.status.idle":"2024-12-23T10:47:27.557046Z","shell.execute_reply.started":"2024-12-23T10:43:20.859663Z","shell.execute_reply":"2024-12-23T10:47:27.555465Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 5.2 Estimator Prediction Correlation","metadata":{}},{"cell_type":"code","source":"ensemble_results = pd.DataFrame()\nfor name, pipe in fineTuned_pipelines.items():\n    ensemble_results[name] = pipe.predict(X_test)\nsns.heatmap(ensemble_results.corr(),annot=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:56:59.174083Z","iopub.execute_input":"2024-12-23T13:56:59.174372Z","iopub.status.idle":"2024-12-23T13:57:02.552648Z","shell.execute_reply.started":"2024-12-23T13:56:59.174345Z","shell.execute_reply":"2024-12-23T13:57:02.551619Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error, mean_squared_log_error\nimport math\n\ndef print_predictions(target,predictions):\n    print(f\"mse: {mean_squared_error(target,predictions)}\")\n    if min(target) >= 0 and min(predictions) >= 0: \n        print(f\"msle: {mean_squared_log_error(target,predictions)}\")\n        print(f\"rmsle: {math.sqrt(mean_squared_log_error(target,predictions))}\")\n    else: \n        print(\"Min target: \", min(target), \", Min predictions: \", min(predictions))\n    # See Appendix\n    # print(f\"rmsle: {root_mean_squared_log_error(target,predictions)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:02.554105Z","iopub.execute_input":"2024-12-23T13:57:02.554522Z","iopub.status.idle":"2024-12-23T13:57:02.560971Z","shell.execute_reply.started":"2024-12-23T13:57:02.554476Z","shell.execute_reply":"2024-12-23T13:57:02.559863Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\nfrom sklearn.ensemble import VotingRegressor\nfrom sklearn.base import clone\n\n# Avoid repeating the same preprocessor for all models\nvoters = ['random_forest', 'xgb']\nestimators = [ (pipe.steps[1][0], clone(pipe.steps[1][1])) for pipe in fineTuned_pipelines.values() \n              if pipe.steps[1][0] in voters ] \nensembleModel = Pipeline(steps=[\n    ('preprocessor', preprocessor),\n    ('voter', VotingRegressor(estimators))\n])\nensembleModel.fit(X_train, y_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:02.562484Z","iopub.execute_input":"2024-12-23T13:57:02.562975Z","iopub.status.idle":"2024-12-23T13:57:37.20931Z","shell.execute_reply.started":"2024-12-23T13:57:02.562932Z","shell.execute_reply":"2024-12-23T13:57:37.208107Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Training scores\npred_train = ensembleModel.predict(X_train)\nprint_predictions(y_train, pred_train)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:37.211087Z","iopub.execute_input":"2024-12-23T13:57:37.211406Z","iopub.status.idle":"2024-12-23T13:57:41.107762Z","shell.execute_reply.started":"2024-12-23T13:57:37.211374Z","shell.execute_reply":"2024-12-23T13:57:41.106761Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Evaluation scores\npred_eval = ensembleModel.predict(X_eval)\nprint_predictions(y_eval, pred_eval)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:41.111599Z","iopub.execute_input":"2024-12-23T13:57:41.112044Z","iopub.status.idle":"2024-12-23T13:57:42.064576Z","shell.execute_reply.started":"2024-12-23T13:57:41.112007Z","shell.execute_reply":"2024-12-23T13:57:42.063482Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for name, model in fineTuned_pipelines.items():\n    print(name)\n    ith_model_pred_eval = model.predict(X_eval)\n    print_predictions(y_eval,ith_model_pred_eval)\n    print()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:42.065835Z","iopub.execute_input":"2024-12-23T13:57:42.066143Z","iopub.status.idle":"2024-12-23T13:57:43.028854Z","shell.execute_reply.started":"2024-12-23T13:57:42.066113Z","shell.execute_reply":"2024-12-23T13:57:43.027773Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"preds_test = ensembleModel.predict(X_test)\noutput = pd.DataFrame({'id': X_test.index,\n                       'Premium Amount': preds_test})\noutput.to_csv('submission.csv', index=False)\npd.read_csv('submission.csv')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:43.040171Z","iopub.execute_input":"2024-12-23T13:57:43.040509Z","iopub.status.idle":"2024-12-23T13:57:47.578413Z","shell.execute_reply.started":"2024-12-23T13:57:43.040478Z","shell.execute_reply":"2024-12-23T13:57:47.577246Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for name, model in fineTuned_pipelines.items():\n    csv_name = f\"submission_{name}.csv\"\n    print('Submission file:', csv_name)\n    ith_preds_test = model.predict(X_test)\n    output = pd.DataFrame({'id': X_test.index,\n                           'Premium Amount': ith_preds_test})\n    output.to_csv(csv_name, index=False)\n    print(pd.read_csv(csv_name).head())\n    print()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-23T13:57:47.579921Z","iopub.execute_input":"2024-12-23T13:57:47.580334Z","iopub.status.idle":"2024-12-23T13:57:51.971762Z","shell.execute_reply.started":"2024-12-23T13:57:47.580288Z","shell.execute_reply":"2024-12-23T13:57:51.970585Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Appendix","metadata":{}},{"cell_type":"markdown","source":"I tried this to update the sklearn package but it didn't work: \n\n```python\n# sklearn.metrics.root_mean_squared_log_error is only available for sklearn >= 1.4\n# See: https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.root_mean_squared_log_error.html\nimport sklearn\nprint(f\"sklearn version: {sklearn.__version__}\")\nif sklearn.__version__ < '1.4': \n    !pip install -U scikit-learn\n    import importlib \n    importlib.reload(sklearn)\n    print(f\"sklearn version after upgrade: {sklearn.__version__}\")\n```","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}