{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"},{"sourceId":9178166,"sourceType":"datasetVersion","datasetId":5547076},{"sourceId":10165290,"sourceType":"datasetVersion","datasetId":6277364}],"dockerImageVersionId":30554,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Insurance Premium Prediction with Keras\n\n## Overview\nThis notebook demonstrates a machine learning workflow for predicting the \"Premium Amount\" using a deep learning model built with Keras and its hyperparameter tunning tool KerasTuner. The workflow includes data loading, feature engineering, normalization, hyperparameter tuning, model training, and submission generation. The dataset is split into training and test sets, and the model is evaluated using K-Fold cross-validation. The final predictions are saved in a submission file.","metadata":{}},{"cell_type":"markdown","source":"## Imports\n\nThis section imports the necessary libraries for the notebook. These libraries include:\n\n* **Pandas**: For data manipulation and analysis.\n* **Numpy**: For numerical operations.\n* **Matplotlib**: For data visualization.\n* **TensorFlow and Keras**: For building and training deep learning models.\n* **Keras Tuner**: For hyperparameter tuning.\n* **Scikit-learn**: For preprocessing, model evaluation, and cross-validation.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport gc\nimport tensorflow as tf\nimport keras_tuner as kt\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import KFold","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:26:04.049662Z","iopub.execute_input":"2024-12-11T15:26:04.05045Z","iopub.status.idle":"2024-12-11T15:26:12.539967Z","shell.execute_reply.started":"2024-12-11T15:26:04.050417Z","shell.execute_reply":"2024-12-11T15:26:12.539134Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Configuration\n\nThis section defines a configuration class (CFG) to store global settings and paths. These settings include:\n\n* **data_path**: The path to the dataset.\n* **is_tuning**: A flag to enable or disable hyperparameter tuning.\n* **is_training**: A flag to enable or disable model training.\n* **target_column**: The name of the target column (\"Premium Amount\").","metadata":{}},{"cell_type":"code","source":"class CFG:\n    data_path = \"/kaggle/input/playground-series-s4e12/\"\n    is_tuning = False\n    is_training = True\n    target_column = \"Premium Amount\"","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:29:22.405669Z","iopub.execute_input":"2024-12-11T15:29:22.405977Z","iopub.status.idle":"2024-12-11T15:29:22.410643Z","shell.execute_reply.started":"2024-12-11T15:29:22.40595Z","shell.execute_reply":"2024-12-11T15:29:22.409605Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load data\n\nThis section loads the training and test datasets from CSV files.","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(f\"{CFG.data_path}train.csv\")\ntrain.pop(\"id\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:27:23.740653Z","iopub.execute_input":"2024-12-11T15:27:23.741422Z","iopub.status.idle":"2024-12-11T15:27:27.124308Z","shell.execute_reply.started":"2024-12-11T15:27:23.741393Z","shell.execute_reply":"2024-12-11T15:27:27.123351Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"origin_data = pd.read_csv(\"/kaggle/input/insurance-premium-prediction/Insurance Premium Prediction Dataset.csv\")\norigin_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T15:27:29.295114Z","iopub.execute_input":"2024-12-11T15:27:29.295928Z","iopub.status.idle":"2024-12-11T15:27:30.011642Z","shell.execute_reply.started":"2024-12-11T15:27:29.2959Z","shell.execute_reply":"2024-12-11T15:27:30.010646Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test = pd.read_csv(f\"{CFG.data_path}test.csv\")\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:28:26.069595Z","iopub.execute_input":"2024-12-11T15:28:26.069911Z","iopub.status.idle":"2024-12-11T15:28:28.32095Z","shell.execute_reply.started":"2024-12-11T15:28:26.069886Z","shell.execute_reply":"2024-12-11T15:28:28.320015Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Feature Engineering\n\nThis section performs feature engineering on the dataset. The feature_engineering function:\n\n* Identifies categorical and numerical columns.\n* Converts the \"Policy Start Date\" column to datetime and extracts features like year, month, day, and weekday.\n* Encodes categorical columns using **LabelEncoder**.\n* Handles missing values by filling them with the mode (for categorical) or mean (for numerical) values.","metadata":{}},{"cell_type":"code","source":"def feature_engineering(train, origin_data, test):\n    categorical_columns = []\n    numerical_columns = []\n    for column in train.columns:\n        if column == CFG.target_column:\n            origin_data['Premium Amount'] = origin_data['Premium Amount'].fillna(train['Premium Amount'].mean())\n            continue\n        if column == 'Policy Start Date':\n            for df in [train, origin_data, test]:\n                df[column] = pd.to_datetime(df[column])\n                df[\"year\"] = df[column].dt.year\n                df[\"month\"] = df[column].dt.month\n                df[\"day\"] = df[column].dt.day\n                df[\"weekday\"] = df[column].dt.weekday\n                df['day_sin']      = np.sin(2 * np.pi * df['day'] / 30)  \n                df['day_cos']      = np.cos(2 * np.pi * df['day'] / 30)\n                df['weekday_sin']  = np.sin(2 * np.pi * df['weekday'] / 7)\n                df['weekday_cos']  = np.cos(2 * np.pi * df['weekday'] / 7)\n            numerical_columns += [\"year\", \"day_sin\", \"day_cos\", \"weekday_sin\", \"weekday_cos\"]\n            categorical_columns += [\"month\", \"day\",\"weekday\"]\n        elif train[column].dtype == object:\n            if train[column].nunique() <= 1:\n                continue\n            categorical_columns.append(column)\n            if train[column].isnull().sum() > 0:\n                train[column] = train[column].fillna(train[column].mode()[0])\n                origin_data[column] = origin_data[column].fillna(train[column].mode()[0])\n                test[column] = test[column].fillna(train[column].mode()[0])\n            encoder = LabelEncoder()\n            train[column] = encoder.fit_transform(train[column])\n            origin_data[column] = encoder.transform(origin_data[column]) \n            test[column] = encoder.transform(test[column]) \n        else:\n            numerical_columns.append(column)\n            if train[column].isnull().sum() > 0:\n                train[column] = train[column].fillna(train[column].mean())\n                origin_data[column] = origin_data[column].fillna(train[column].mean())\n                test[column] = test[column].fillna(train[column].mean())\n    for df in [train, origin_data, test]:\n        df[\"income_age_ratio\"]   = np.clip(df[\"Annual Income\"] / df[\"Age\"], a_min = 1e-6, a_max = 1e9)\n        df[\"score\"]             = df[\"Credit Score\"] + df[\"Health Score\"]\n    numerical_columns += [\"income_age_ratio\", \"score\"]\n    return train, origin_data, test, categorical_columns, numerical_columns","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:33:04.335823Z","iopub.execute_input":"2024-12-11T15:33:04.336439Z","iopub.status.idle":"2024-12-11T15:33:04.345406Z","shell.execute_reply.started":"2024-12-11T15:33:04.336408Z","shell.execute_reply":"2024-12-11T15:33:04.344476Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train, origin_data, test, categorical_columns, numerical_columns = feature_engineering(train, origin_data, test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T15:28:56.319104Z","iopub.execute_input":"2024-12-11T15:28:56.320022Z","iopub.status.idle":"2024-12-11T15:29:04.373528Z","shell.execute_reply.started":"2024-12-11T15:28:56.319992Z","shell.execute_reply":"2024-12-11T15:29:04.372762Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T15:31:23.317945Z","iopub.execute_input":"2024-12-11T15:31:23.318293Z","iopub.status.idle":"2024-12-11T15:31:23.362893Z","shell.execute_reply.started":"2024-12-11T15:31:23.318266Z","shell.execute_reply":"2024-12-11T15:31:23.361864Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Normalization\n\nThis section normalizes the numerical features using TensorFlow's `Normalization` layer. The `adapt()` method is used to fit the normalization layer to the training data.","metadata":{}},{"cell_type":"code","source":"normalization = tf.keras.layers.Normalization()\nnormalization.adapt(train[numerical_columns], batch_size=512)","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:33:10.352591Z","iopub.execute_input":"2024-12-11T15:33:10.353457Z","iopub.status.idle":"2024-12-11T15:33:14.15735Z","shell.execute_reply.started":"2024-12-11T15:33:10.353425Z","shell.execute_reply":"2024-12-11T15:33:14.156585Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Additionally, the maximum values for categorical columns are calculated for use in embedding layers.","metadata":{}},{"cell_type":"code","source":"categorical_column_max_values = [train[column].max() for column in categorical_columns]\ncategorical_column_max_values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T15:29:14.5894Z","iopub.execute_input":"2024-12-11T15:29:14.589717Z","iopub.status.idle":"2024-12-11T15:29:14.611463Z","shell.execute_reply.started":"2024-12-11T15:29:14.589692Z","shell.execute_reply":"2024-12-11T15:29:14.610401Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Hyper Parameter Tuning\n\nThis section defines functions for creating datasets and building models. The build_model function uses Keras Tuner to define a hyperparameter search space. The search space includes:\n\n* Dropout rate\n* Learning rate\n* Number of units in each layer\n* Activation function\n* Regularization strength\n* Loss function\n\nThe `get_model` function constructs the actual Keras model using the hyperparameters.","metadata":{}},{"cell_type":"code","source":"def create_dataset(df, shuffle=False, batch_size=512):\n    ds = tf.data.Dataset.from_tensor_slices(((df[numerical_columns], df[categorical_columns]), df[CFG.target_column]))\n    if shuffle:\n        ds = ds.shuffle(batch_size * 10)\n    ds = ds.batch(batch_size).cache().prefetch(tf.data.AUTOTUNE)\n    return ds\n\ndef build_model(hp):\n    use_dropout = hp.Choice(\"use_dropout\", [True, False])\n    dropout_value = hp.Float(\"dropout\", min_value=0.1, max_value=0.3)\n    learning_rate = hp.Float(\"learning_rate\", min_value=1e-5, max_value=1e-3, sampling=\"log\")\n    depth = 6\n    params = {\n        \"depth\": depth,\n        \"use_dropout\": use_dropout,\n        \"dropout\": dropout_value,\n        \"learning_rate\": learning_rate,\n    }\n    for i in range(depth):\n        params[f\"unit_{i}\"] = hp.Int(f\"unit_{i}\", min_value=16, max_value=256, step=16)\n    params[\"activation\"] = hp.Choice(\"activation\", [\"relu\", \"swish\", \"tanh\"])\n    params[\"l2\"] = hp.Choice(\"l2\", [1e-5, 3e-5, 5e-5, 1e-6, 5e-6])\n    #params[\"loss\"] = hp.Choice(\"loss\", [\"mae\", \"huber_loss\", \"mse\", \"mean_squared_logarithmic_error\"])\n    params[\"loss\"] = \"mean_squared_logarithmic_error\"\n    return get_model(params)\nmsle =tf.keras.metrics.MeanSquaredLogarithmicError(name=\"msle\")\ndef rmsle(y_true, y_pred):\n    return tf.sqrt(msle(y_true, y_pred))\ndef get_model(params):\n    use_dropout = params[\"use_dropout\"]\n    dropout_value = params[\"dropout\"]\n    learning_rate = params[\"learning_rate\"]\n    depth = params[\"depth\"]\n    units = [params[f\"unit_{i}\"] for i in range(depth)]\n    activation = params[\"activation\"]\n    l2_factor = params[\"l2\"]\n    numerical_inputs = tf.keras.Input(shape=(len(numerical_columns)), dtype=tf.float32)\n    categorical_inputs = tf.keras.Input(shape=(len(categorical_columns)), dtype=tf.int32)\n    numerical_x = normalization(numerical_inputs)\n    vectors = [numerical_x]\n    embeddings = []\n    for i in range(len(categorical_columns)):\n        output_dim = int(np.ceil(categorical_column_max_values[i]))\n        embeddings.append(tf.keras.layers.Embedding(categorical_column_max_values[i] + 1, output_dim=output_dim, input_length=1))\n    \n    for i in range(len(categorical_columns)):\n        cat_x = categorical_inputs[:, i:i+1]\n        cat_x = tf.keras.layers.Reshape((-1, 1))(cat_x)\n        cat_x = embeddings[i](cat_x)\n        output_dim = int(np.ceil(categorical_column_max_values[i]))\n        cat_x = tf.keras.layers.Reshape((output_dim,))(cat_x)\n        vectors.append(cat_x)\n    x = tf.keras.layers.Concatenate()(vectors) \n    for i in range(depth):\n        kernel_regularizer = tf.keras.regularizers.l2(l2_factor)\n        x = tf.keras.layers.Dense(units[i], activation=activation, kernel_regularizer=kernel_regularizer)(x)\n        if use_dropout:\n            x = tf.keras.layers.Dropout(dropout_value)(x)\n    output = tf.keras.layers.Dense(1)(x)\n    model = tf.keras.Model(inputs=[numerical_inputs, categorical_inputs], outputs=output)\n    loss = params[\"loss\"]\n    model.compile(loss=loss, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[rmsle])\n    return model","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:33:21.68642Z","iopub.execute_input":"2024-12-11T15:33:21.687273Z","iopub.status.idle":"2024-12-11T15:33:21.704884Z","shell.execute_reply.started":"2024-12-11T15:33:21.687241Z","shell.execute_reply":"2024-12-11T15:33:21.704109Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"If hyperparameter tuning is enabled (CFG.is_tuning), the notebook performs Bayesian optimization using Keras Tuner to find the best hyperparameters.","metadata":{}},{"cell_type":"code","source":"if CFG.is_tuning:\n    train_df, val_df = train_test_split(train, random_state=42, shuffle=True)\n    train_ds = create_dataset(train_df.sample(frac=0.2, random_state=42))\n    val_ds = create_dataset(val_df)\n    tuner = kt.BayesianOptimization(\n        build_model,\n        objective=kt.Objective(\"val_rmsle\", direction=\"min\"),\n        max_trials=30,\n        overwrite=True\n    )\n    tuner.search(\n        train_ds,\n        epochs=5,\n        validation_data=val_ds,\n        verbose=2\n    )\n    tuner.results_summary()","metadata":{"execution":{"iopub.status.busy":"2024-12-11T15:33:24.434246Z","iopub.execute_input":"2024-12-11T15:33:24.434582Z","iopub.status.idle":"2024-12-11T15:33:24.440052Z","shell.execute_reply.started":"2024-12-11T15:33:24.434556Z","shell.execute_reply":"2024-12-11T15:33:24.439214Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Keep track of previous hyperparameter tuning log.\n```\nTrial 30 Complete [00h 00m 30s]\nval_rmsle: 2.5852959156036377\n\nBest val_rmsle So Far: 1.6358801126480103\nTotal elapsed time: 00h 13m 56s\nResults summary\nResults in ./untitled_project\nShowing 10 best trials\nObjective(name=\"val_rmsle\", direction=\"min\")\n\nTrial 01 summary\nHyperparameters:\nuse_dropout: 0\ndropout: 0.13774630230636115\nlearning_rate: 0.0003223115413405983\nunit_0: 208\nunit_1: 256\nunit_2: 48\nunit_3: 208\nunit_4: 144\nunit_5: 32\nactivation: relu\nl2: 1e-06\nScore: 1.6358801126480103\n\nTrial 00 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.23212606292135063\nlearning_rate: 4.216932977754679e-05\nunit_0: 256\nunit_1: 64\nunit_2: 128\nunit_3: 256\nunit_4: 208\nunit_5: 96\nactivation: relu\nl2: 3e-05\nScore: 1.976195216178894\n\nTrial 02 summary\nHyperparameters:\nuse_dropout: 0\ndropout: 0.14274226089606473\nlearning_rate: 2.536178844274438e-05\nunit_0: 192\nunit_1: 224\nunit_2: 16\nunit_3: 96\nunit_4: 48\nunit_5: 64\nactivation: swish\nl2: 3e-05\nScore: 2.2184629440307617\n\nTrial 03 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.22392819014676008\nlearning_rate: 0.00015012820193408534\nunit_0: 224\nunit_1: 144\nunit_2: 144\nunit_3: 32\nunit_4: 128\nunit_5: 128\nactivation: relu\nl2: 1e-06\nScore: 2.271772861480713\n\nTrial 04 summary\nHyperparameters:\nuse_dropout: 0\ndropout: 0.1326177079751804\nlearning_rate: 0.00041557572152236774\nunit_0: 192\nunit_1: 48\nunit_2: 208\nunit_3: 192\nunit_4: 16\nunit_5: 96\nactivation: tanh\nl2: 5e-06\nScore: 2.3390138149261475\n\nTrial 05 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.1959805440408351\nlearning_rate: 3.120623562291715e-05\nunit_0: 96\nunit_1: 32\nunit_2: 176\nunit_3: 144\nunit_4: 176\nunit_5: 224\nactivation: relu\nl2: 5e-05\nScore: 2.493769884109497\n\nTrial 29 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.2764803682750224\nlearning_rate: 0.00046916701471866573\nunit_0: 192\nunit_1: 176\nunit_2: 192\nunit_3: 32\nunit_4: 208\nunit_5: 96\nactivation: swish\nl2: 1e-05\nScore: 2.5852959156036377\n\nTrial 28 summary\nHyperparameters:\nuse_dropout: 0\ndropout: 0.1412247276780964\nlearning_rate: 0.0007836255783976533\nunit_0: 80\nunit_1: 64\nunit_2: 48\nunit_3: 240\nunit_4: 256\nunit_5: 48\nactivation: swish\nl2: 5e-05\nScore: 2.6195244789123535\n\nTrial 26 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.24401837511932967\nlearning_rate: 3.728121606190353e-05\nunit_0: 176\nunit_1: 224\nunit_2: 112\nunit_3: 32\nunit_4: 224\nunit_5: 240\nactivation: relu\nl2: 1e-05\nScore: 2.622511863708496\n\nTrial 25 summary\nHyperparameters:\nuse_dropout: 1\ndropout: 0.20663158614549187\nlearning_rate: 0.00046196045482696854\nunit_0: 160\nunit_1: 224\nunit_2: 144\nunit_3: 112\nunit_4: 128\nunit_5: 144\nactivation: relu\nl2: 1e-05\nScore: 2.6412360668182373\n```","metadata":{}},{"cell_type":"markdown","source":"## Training models\n\nTrain the model using K-Fold cross-validation. For each fold:\n* The dataset is split into training and validation sets.\n* The model is either built with the best hyperparameters (if tuning is disabled) or trained with the best hyperparameters found during tuning.\n* The model is saved and evaluated on the validation set.","metadata":{}},{"cell_type":"code","source":"kfold = KFold(5, random_state=42, shuffle=True)\nmodels = []\nrmsles = []\nfor fold, (train_index, val_index) in enumerate(kfold.split(train)):\n    print(\"=\" * 30)\n    print(f\"Fold {fold}\")\n    print(\"=\" * 30)\n    df = pd.concat([train.iloc[train_index], origin_data])\n    train_ds = create_dataset(df)\n    val_ds = create_dataset(train.iloc[val_index])\n    if CFG.is_training:\n        model_path = f\"model_{fold}.keras\"\n    else:\n        model_path = f\"/kaggle/input/s4e12-keras-models/model_{fold}.keras\"\n    if CFG.is_tuning:\n        model = build_model(hp)\n    else:\n        model = get_model({\n            \"use_dropout\": 0,\n            \"dropout\": 0.13774630230636115,\n            \"learning_rate\": 0.0003223115413405983,\n            \"unit_0\": 208,\n            \"unit_1\": 256,\n            \"unit_2\": 48,\n            \"unit_3\": 208,\n            \"unit_4\": 144,\n            \"unit_5\": 32,\n            \"activation\": \"relu\",\n            \"l2\": 1e-06,\n            \"loss\": \"mean_squared_logarithmic_error\",\n            \"depth\": 6\n        })\n    if CFG.is_training:\n        model.fit(\n            train_ds, \n            epochs=30, \n            validation_data=val_ds,\n            callbacks=[\n                tf.keras.callbacks.ModelCheckpoint(\n                    model_path, \n                    monitor=\"val_rmsle\", \n                    mode=\"min\", \n                    restore_best_weights=True, \n                    save_best_only=True\n                ),\n                tf.keras.callbacks.EarlyStopping(patience=5)\n            ]\n        )\n    model = tf.keras.models.load_model(model_path, custom_objects={\n        \"rmsle\": rmsle\n    })\n    val_loss, val_rmsle = model.evaluate(val_ds)\n    print(f\"Validation Loss: {val_loss:.2f} Validation RMSLE: {val_rmsle:.2f}\")\n    rmsles.append(val_rmsle)\n    models.append(model)\nprint(f\"OOF: {np.mean(rmsles): .2f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T15:33:30.437325Z","iopub.execute_input":"2024-12-11T15:33:30.437893Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create Submission\n\nGenerates predictions for the test dataset using the trained models. The predictions are averaged across all folds and saved to a CSV file for submission.","metadata":{}},{"cell_type":"code","source":"test[CFG.target_column] = 0\ntest_ds = create_dataset(test, shuffle=False)\ny_pred = np.mean([model.predict(test_ds, verbose=0) for model in models], axis=0)\ntest[CFG.target_column] = y_pred\ntest[[\"id\", CFG.target_column]].to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}