{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70367,"databundleVersionId":9188054,"sourceType":"competition"},{"sourceId":193061185,"sourceType":"kernelVersion"}],"dockerImageVersionId":30746,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":1087.424982,"end_time":"2024-08-03T13:00:18.164826","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-08-03T12:42:10.739844","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Ariel Data Challenge 2024: Introductory model: inference\n\nIn this notebook, we compute test predictions using the model saved in [Baseline + more feature(train)](https://www.kaggle.com/code/bingyuniu/baseline-more-feature-train?kernelSessionId=193061185).\n\nThe Baseline is from [ADC24 Intro inference ⭐️⭐️⭐️⭐️⭐️](https://www.kaggle.com/code/ambrosm/adc24-intro-inference?kernelSessionId=192682085)\n\n<img width=\"700\" src=\"https://www.ariel-datachallenge.space/static/images/transit_situation.png\" />\n\nThis image has been taken from [last year's competition](https://www.ariel-datachallenge.space/ML/documentation/about). It shows how a planet transits in front of its star and how this transit maps to the lightcurve (a dip in the brightness of the star). This dip is directly proportional to the ratio of the areas of the planet and star. It's this ratio (the \"transit depth\") that we are modeling in the present notebook.\n\nThe present notebook is simple:\n- It reads the pre- and postprocessing code, which is the same as the code used for training.\n- It reads the test data.\n- It reads the saved model.\n- It executes the prediction pipeline and saves the submission file.\n\nThe real work was done in the [Baseline + more feature(train)](https://www.kaggle.com/code/bingyuniu/baseline-more-feature-train?kernelSessionId=193061185)!","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.009657,"end_time":"2024-08-03T12:42:14.074227","exception":false,"start_time":"2024-08-03T12:42:14.06457","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport scipy.stats\nfrom tqdm import tqdm\nimport pickle\n\nfrom sklearn.linear_model import Ridge\n","metadata":{"_kg_hide-input":true,"papermill":{"duration":3.023083,"end_time":"2024-08-03T12:42:17.107326","exception":false,"start_time":"2024-08-03T12:42:14.084243","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-08-19T06:22:15.964215Z","iopub.execute_input":"2024-08-19T06:22:15.964667Z","iopub.status.idle":"2024-08-19T06:22:19.631011Z","shell.execute_reply.started":"2024-08-19T06:22:15.964631Z","shell.execute_reply":"2024-08-19T06:22:19.629833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Feature engineering**\n\nIn our feature engineering process, we aim to capture the key characteristics of how the brightness of stars changes when obscured by planets. Specifically, we want to quantify how much darker the images become during planetary transits. The time series data shows that planets reduce the brightness of stars by approximately 0.2% on average (e.g., from 228.2 to 227.6 or from 1371 to 1368).\n\nTo achieve this, we have extracted a variety of features:\n\nBasic Statistical Features: These features capture the overall distribution of the data, including:\n\nRelative Reduction: The difference in brightness between the obscured and unobscured regions, normalized by the unobscured brightness. Signal-to-Noise Ratio (SNR): The ratio of unobscured brightness to the overall standard deviation, indicating the quality of the signal. Variance, Skewness, Kurtosis: These statistical metrics describe the spread, symmetry, and peak characteristics of the brightness data. Subset Features: These features focus on specific time intervals, particularly the regions where the planet partially obscures the star:\n\nHalf Reduction: We calculate the relative reduction in brightness for the two half-obscured regions of the time series, providing a more detailed view of the transit effect. Sliding Window Features: By applying a sliding window approach, we capture the local variations in the time series data:\n\nFor each window, we compute the mean, standard deviation, minimum, and maximum brightness values, which help us understand how brightness fluctuates over time.\n\nThe feature engineering was Done in [Baseline + more feature(train)](https://www.kaggle.com/code/bingyuniu/baseline-more-feature-train?kernelSessionId=193061185).","metadata":{}},{"cell_type":"code","source":"directory = \"/kaggle/input/baseline-more-feature-train/\"\n\nexec(open(directory + 'f_read_and_preprocess.py', 'r').read())\nexec(open(directory + 'a_read_and_preprocess.py', 'r').read())\nexec(open(directory + 'feature_engineering.py', 'r').read())\nexec(open(directory + 'postprocessing.py', 'r').read())\n","metadata":{"papermill":{"duration":1067.258124,"end_time":"2024-08-03T13:00:08.197814","exception":false,"start_time":"2024-08-03T12:42:20.93969","status":"completed"},"tags":[],"_kg_hide-input":false,"execution":{"iopub.status.busy":"2024-08-19T06:22:19.63336Z","iopub.execute_input":"2024-08-19T06:22:19.633992Z","iopub.status.idle":"2024-08-19T06:22:19.665876Z","shell.execute_reply.started":"2024-08-19T06:22:19.633953Z","shell.execute_reply":"2024-08-19T06:22:19.664721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"People have been asking how to choose a good value for sigma_pred. As explained in [Understanding the competition metric](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/528114), with sigma_pred we indicate what root mean squared error (rmse) we expect for our test predictions.\n\nThe training data cover planets of only two stars (stars 0 and 1), but the test data include planets of other stars.\n\nThis leads to the following recipe:\n- For known stars (stars 0 and 1), we expect the test rmse to be equal to our cross-validation rmse, i.e. we predict the out-of-fold rmse of our model (0.000293 as shown in the training notebook).\n- For unknown stars, the prediction error can only be higher. We thus predict a higher value (0.001 in this notebook).","metadata":{}},{"cell_type":"code","source":"# Load the data\nwavelengths = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/wavelengths.csv')\ntest_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n                           index_col='planet_id')\nf_raw_test = f_read_and_preprocess('test', test_adc_info, test_adc_info.index)\na_raw_test = a_read_and_preprocess('test', test_adc_info, test_adc_info.index)\ntest = feature_engineering(f_raw_test, a_raw_test)\nprint(test)\nprint(f\"Number of features:{test.shape[1]}\")\n# Load the model\nwith open(directory + 'model.pickle', 'rb') as f:\n    model = pickle.load(f)\nwith open(directory + 'sigma_pred.pickle', 'rb') as f:\n    sigma_pred = pickle.load(f)\n    \n# Predict\ntest_pred = model.predict(test)\n\n# Package into submission file\nsub_df = postprocessing(test_pred,\n                        test_adc_info.index,\n                        sigma_pred=np.tile(np.where(test_adc_info[['star']] <= 1, 0.000155, 0.001), (1, 283)))\ndisplay(sub_df)\nsub_df.to_csv('submission.csv')\n#!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2024-08-19T06:25:11.904006Z","iopub.execute_input":"2024-08-19T06:25:11.904426Z","iopub.status.idle":"2024-08-19T06:25:14.812165Z","shell.execute_reply.started":"2024-08-19T06:25:11.904393Z","shell.execute_reply":"2024-08-19T06:25:14.810516Z"},"trusted":true},"execution_count":null,"outputs":[]}]}