{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Regression with an Insurance Dataset EDA\n## Season 4 Episode 12 of the Playground Series\n\nAs usual, we have not much information about the dataset we've been given for this playground series. We'll need to do an exploratory data analysis to uncover what we're dealing with here...\n\nWork to do:\n1. Import Libraries\n2. Read in the Data\n3. Characterize the Data\n4. Nulls Analysis\n5. Compare Distributions (train vs test)\n6. Ordinal Encode the Categories\n7. View Correlations\n8. Next Steps\n\n## Import Libraries","metadata":{}},{"cell_type":"code","source":"#import libraries\nimport matplotlib.pyplot as plt\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport os\nfrom sklearn.preprocessing import OrdinalEncoder as OE\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:57:40.075419Z","iopub.execute_input":"2024-12-09T05:57:40.07584Z","iopub.status.idle":"2024-12-09T05:57:47.869132Z","shell.execute_reply.started":"2024-12-09T05:57:40.075804Z","shell.execute_reply":"2024-12-09T05:57:47.867946Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Read in the Data","metadata":{}},{"cell_type":"code","source":"#read in the data\ndf = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv')\ndt = pd.read_csv('/kaggle/input/playground-series-s4e12/test.csv')\n\n#How big are these?\ndf.shape,dt.shape","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Characterize the Data","metadata":{}},{"cell_type":"code","source":"df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:35:16.205149Z","iopub.execute_input":"2024-12-09T05:35:16.205558Z","iopub.status.idle":"2024-12-09T05:35:16.248953Z","shell.execute_reply.started":"2024-12-09T05:35:16.205522Z","shell.execute_reply":"2024-12-09T05:35:16.247928Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.options.display.float_format = '{:.0f}'.format\ndf.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:42:51.069379Z","iopub.execute_input":"2024-12-09T06:42:51.069799Z","iopub.status.idle":"2024-12-09T06:42:52.445289Z","shell.execute_reply.started":"2024-12-09T06:42:51.069764Z","shell.execute_reply":"2024-12-09T06:42:52.444167Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dt.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:43:03.395498Z","iopub.execute_input":"2024-12-09T06:43:03.395873Z","iopub.status.idle":"2024-12-09T06:43:03.413583Z","shell.execute_reply.started":"2024-12-09T06:43:03.395844Z","shell.execute_reply":"2024-12-09T06:43:03.412459Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dt.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:43:13.834617Z","iopub.execute_input":"2024-12-09T06:43:13.835101Z","iopub.status.idle":"2024-12-09T06:43:14.742107Z","shell.execute_reply.started":"2024-12-09T06:43:13.835061Z","shell.execute_reply":"2024-12-09T06:43:14.740967Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# make columns lists\ndfc = df.columns.tolist()\ndtc = dt.columns.tolist()\n#dfc, dtc\n\n#set target\nTARGET = 'Premium Amount'\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:43:58.104536Z","iopub.execute_input":"2024-12-09T06:43:58.10553Z","iopub.status.idle":"2024-12-09T06:43:58.110256Z","shell.execute_reply.started":"2024-12-09T06:43:58.105492Z","shell.execute_reply":"2024-12-09T06:43:58.109129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Nulls Analysis\n\nLet's compare our two dataframes. Do the columns with nulls match across dataframes? Do the number of nulls look similar? Let's find out...","metadata":{}},{"cell_type":"code","source":"nullset = pd.DataFrame({'train':df.isnull().sum(),'trainperc': range(df.shape[1]),'test':dt.isnull().sum(), 'testperc': list(range(df.shape[1]))})\nnullset['trainperc'] = nullset['trainperc'].astype(float)\nnullset['trainperc'] = (nullset['train'] / df.shape[0]) * 100\nnullset['testperc'] = (nullset['test'] /  dt.shape[0]) * 100\nnullset","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:57:58.664455Z","iopub.execute_input":"2024-12-09T06:57:58.664838Z","iopub.status.idle":"2024-12-09T06:57:58.743796Z","shell.execute_reply.started":"2024-12-09T06:57:58.664805Z","shell.execute_reply":"2024-12-09T06:57:58.74263Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"With that table, we can see that the percentages of nulls are identical across the train and test sets. The same columns have nulls across the train and test set too.\n\n## Uniques Analysis\n\nDo the number of unique values seem to be consistent across the train and test sets?","metadata":{}},{"cell_type":"code","source":"for i in dtc:\n    print(f\"{i}: Uniques {df[i].nunique()} , {dt[i].nunique()} \")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:43:22.765107Z","iopub.execute_input":"2024-12-09T05:43:22.765524Z","iopub.status.idle":"2024-12-09T05:43:24.691474Z","shell.execute_reply.started":"2024-12-09T05:43:22.765487Z","shell.execute_reply":"2024-12-09T05:43:24.690444Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Yes, the Uniques align across the train and test set. Several of these columns will be easily one-hot encoded or numerically encoded to appease a littany of algorithms.\n\n## Compare Distributions\n\nThe columnar data from the train and test set \"Should\" have similar distributions of data. We are looking to see if any of the distributions are significantly off. This would clue us in to a non-representative sampling that took place upstream somewhere...","metadata":{}},{"cell_type":"code","source":"for i in dtc[1:]:\n    if dt[i].dtype in [int,float]: \n        print(f'{i} Plots:')\n        fig, (ax1,ax2) = plt.subplots(nrows=1,ncols=2, sharex=False, sharey=True)\n        ax1.hist(df[i].astype(float).values, density=True, alpha=0.7)\n     \n        ax1.tick_params(left=False, bottom=False)\n        ax1.set_xlabel(i)\n        ax1.set_title(f\"Train Set {i}\")\n        for ax, spine in ax1.spines.items():\n            spine.set_visible(False)\n        \n        ax2.hist(dt[i].astype(float).values, density=True, color='red', alpha=0.7)\n       \n        ax2.tick_params(left=False, bottom=False)\n        ax2.set_xlabel(i)\n        ax2.set_title(f\"Test Set {i}\")\n        for ax, spine in ax2.spines.items():\n            spine.set_visible(False)\n        \n        plt.show()\n        print('')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:45:53.814693Z","iopub.execute_input":"2024-12-09T05:45:53.815079Z","iopub.status.idle":"2024-12-09T05:45:58.135793Z","shell.execute_reply.started":"2024-12-09T05:45:53.815046Z","shell.execute_reply":"2024-12-09T05:45:58.134676Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Great. The distributions look similar across each of the columns. Let's move on to the correlations.\n\n## Correlation Plots\n\nSince we are asked to run regressions on this data, we should take a look at the correlation matrices. Higher correlations (positive or negative) would translate to more significant variables in the ensuing models. Let's see how they look.","metadata":{}},{"cell_type":"code","source":"CAT_COLS , NUM_COLS = [],[]\nfor i in dtc:\n    if dt[i].dtype in [int,float]:\n        NUM_COLS.append(i)\n    else:\n        CAT_COLS.append(i)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:59:53.102954Z","iopub.execute_input":"2024-12-09T05:59:53.10348Z","iopub.status.idle":"2024-12-09T05:59:53.111054Z","shell.execute_reply.started":"2024-12-09T05:59:53.103429Z","shell.execute_reply":"2024-12-09T05:59:53.109964Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"oe = OE( handle_unknown = 'use_encoded_value', unknown_value= -1, encoded_missing_value= -2)\noe.fit(df[CAT_COLS])\ndf[CAT_COLS] = oe.transform(df[CAT_COLS])\ndt[CAT_COLS] = oe.transform(dt[CAT_COLS])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T05:59:55.255011Z","iopub.execute_input":"2024-12-09T05:59:55.255519Z","iopub.status.idle":"2024-12-09T06:00:04.478322Z","shell.execute_reply.started":"2024-12-09T05:59:55.255447Z","shell.execute_reply":"2024-12-09T06:00:04.477036Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.heatmap(df[NUM_COLS].corr())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:00:44.634945Z","iopub.execute_input":"2024-12-09T06:00:44.635328Z","iopub.status.idle":"2024-12-09T06:00:45.509508Z","shell.execute_reply.started":"2024-12-09T06:00:44.635294Z","shell.execute_reply":"2024-12-09T06:00:45.508022Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.heatmap(df.corr())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:01:11.284905Z","iopub.execute_input":"2024-12-09T06:01:11.285433Z","iopub.status.idle":"2024-12-09T06:01:13.528544Z","shell.execute_reply.started":"2024-12-09T06:01:11.285364Z","shell.execute_reply":"2024-12-09T06:01:13.527199Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.heatmap(dt[NUM_COLS].corr())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:01:47.495058Z","iopub.execute_input":"2024-12-09T06:01:47.495489Z","iopub.status.idle":"2024-12-09T06:01:48.151647Z","shell.execute_reply.started":"2024-12-09T06:01:47.495448Z","shell.execute_reply":"2024-12-09T06:01:48.150205Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.heatmap(dt.corr())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-09T06:00:14.531607Z","iopub.execute_input":"2024-12-09T06:00:14.532114Z","iopub.status.idle":"2024-12-09T06:00:16.122555Z","shell.execute_reply.started":"2024-12-09T06:00:14.532065Z","shell.execute_reply":"2024-12-09T06:00:16.121477Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"WOAH! That is an interesting series of correlation plots. \n- The Bad: There seems to be very little signal in this data. There are no correlations greater than +/- .2. So we are going to need to run very sensitive models to pull the signal out and make accurate predictions.\n- The Good: At least the train and test set have similar numbers. This is encouraging because the data was probably the result of a good random sampling.\n\n## Next Steps:\n\nSo, what we found was both encouraging and challenging:\n- Lots of data, 2M observations split 60% train, 40% test.\n- The nulls in both train and test are consistent. If able to mitigate train set nulls, test set nulls should act similarly.\n- The unique values in the categoricals look to match across train and test.\n- Sampling looks good, distributions seem consistent across train and test.\n- The correlation plot show very little signal in this data. We'll have to run very sensitive models to accurately predict values in the test set.\n\nThanks for taking a look.\nIf this was helpful, you know what to do.\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}