{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70367,"databundleVersionId":9188054,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":1087.424982,"end_time":"2024-08-03T13:00:18.164826","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-08-03T12:42:10.739844","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nimport scipy.stats\nfrom tqdm import tqdm\nimport pickle\n\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.metrics import r2_score, mean_squared_error","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:40.337579Z","iopub.execute_input":"2024-09-09T06:59:40.338599Z","iopub.status.idle":"2024-09-09T06:59:42.161657Z","shell.execute_reply.started":"2024-09-09T06:59:40.338545Z","shell.execute_reply":"2024-09-09T06:59:42.160409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_ariel_train = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_adc_info.csv',index_col='planet_id')\ntrain_labels = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_labels.csv',index_col='planet_id')\nwavelengths = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/wavelengths.csv')","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:42.164152Z","iopub.execute_input":"2024-09-09T06:59:42.164707Z","iopub.status.idle":"2024-09-09T06:59:42.350134Z","shell.execute_reply.started":"2024-09-09T06:59:42.164673Z","shell.execute_reply":"2024-09-09T06:59:42.348864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check data","metadata":{}},{"cell_type":"code","source":"fgs1_data = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/train/{100468857}/FGS1_signal.parquet')\nfgs1_data","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:42.3518Z","iopub.execute_input":"2024-09-09T06:59:42.352291Z","iopub.status.idle":"2024-09-09T06:59:44.524741Z","shell.execute_reply.started":"2024-09-09T06:59:42.352246Z","shell.execute_reply":"2024-09-09T06:59:44.523235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The FGS1 measurements consist of one file per planet (673 files for 673 planets for training). For now, we ignore the calibration files.\n\nAIRS is the other sensor of the satellite","metadata":{}},{"cell_type":"code","source":"airis_data = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/train/{4249337798}/AIRS-CH0_signal.parquet')\nairis_data","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:44.527623Z","iopub.execute_input":"2024-09-09T06:59:44.528051Z","iopub.status.idle":"2024-09-09T06:59:47.139326Z","shell.execute_reply.started":"2024-09-09T06:59:44.52799Z","shell.execute_reply":"2024-09-09T06:59:47.137766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data analysis:\n\n\n- We have 673 planets for training. These planets belong to two different stars.\n- There will be roughly 800 planets for testing (but the test data is hidden).\n- The competition is a multi-output regression task with 283 targets to predict.","metadata":{}},{"cell_type":"markdown","source":"TimeSeries","metadata":{}},{"cell_type":"code","source":"_, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, sharex=True, figsize=(12, 4))\n\nmean_signal_1 = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/train/{14485303}/FGS1_signal.parquet').values.mean(axis=1)\nsignal_arr = [20500, 23500, 44000, 47000]\n\ncum_signal = (mean_signal_1[1::2] - mean_signal_1[0::2]).cumsum()\n\n\nax1.plot(mean_signal_1[1::2] - mean_signal_1[0::2], label='raw signal')\nax1.legend()\nax3.plot((cum_signal[800:] - cum_signal[:-800]) / 800)\n\nfor t in signal_arr:\n    ax3.axvline(t, color='gray')\n\n\nmean_signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/train/{4249337798}/FGS1_signal.parquet').values.mean(axis=1)\n\nnet_signal = mean_signal[1::2] - mean_signal[0::2]\ncum_signal = net_signal.cumsum()\nwindow=800\nsmooth_signal = (cum_signal[window:] - cum_signal[:-window]) / window\n\nax2.plot(net_signal, label='raw signal')\nax2.legend()\nax4.plot(smooth_signal)\nfor t in signal_arr:\n    ax4.axvline(t, color='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:47.141249Z","iopub.execute_input":"2024-09-09T06:59:47.141615Z","iopub.status.idle":"2024-09-09T06:59:52.034359Z","shell.execute_reply.started":"2024-09-09T06:59:47.141585Z","shell.execute_reply":"2024-09-09T06:59:52.03307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"signal = airis_data.values.reshape(11250, 32, 356)[1]\n\nplt.figure(figsize=(10, 3))\nsns.heatmap(signal)\nplt.ylabel('spatial dimension')\nplt.xlabel('wavelength dimension')\nplt.show()","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.807236,"end_time":"2024-08-03T13:00:15.817499","exception":false,"start_time":"2024-08-03T13:00:15.010263","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-09-09T06:59:52.036187Z","iopub.execute_input":"2024-09-09T06:59:52.036665Z","iopub.status.idle":"2024-09-09T06:59:52.707671Z","shell.execute_reply.started":"2024-09-09T06:59:52.036621Z","shell.execute_reply":"2024-09-09T06:59:52.706394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile f_read_and_preprocess.py\n\ndef f_read_and_preprocess(dataset, adc_info, planet_ids):\n    f_raw_train = np.full((len(planet_ids), 67500), np.nan, dtype=np.float32)\n    for i, planet_id in tqdm(list(enumerate(planet_ids))):\n        f_signal = pl.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/FGS1_signal.parquet')\n        mean_signal = f_signal.cast(pl.Int32).sum_horizontal().cast(pl.Float32).to_numpy() / 1024 # mean over the 32*32 pixels\n        net_signal = mean_signal[1::2] - mean_signal[0::2]\n        f_raw_train[i] = net_signal\n    return f_raw_train","metadata":{"papermill":{"duration":1067.258124,"end_time":"2024-08-03T13:00:08.197814","exception":false,"start_time":"2024-08-03T12:42:20.93969","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-09-09T06:59:52.709369Z","iopub.execute_input":"2024-09-09T06:59:52.709819Z","iopub.status.idle":"2024-09-09T06:59:52.718905Z","shell.execute_reply.started":"2024-09-09T06:59:52.709761Z","shell.execute_reply":"2024-09-09T06:59:52.717465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exec(open('f_read_and_preprocess.py', 'r').read())\nf_raw_train = f_read_and_preprocess('train', data_ariel_train, train_labels.index)\nwith open('f_raw_train.pickle', 'wb') as f:\n    pickle.dump(f_raw_train, f)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T06:59:52.720753Z","iopub.execute_input":"2024-09-09T06:59:52.721188Z","iopub.status.idle":"2024-09-09T07:11:48.521935Z","shell.execute_reply.started":"2024-09-09T06:59:52.721148Z","shell.execute_reply":"2024-09-09T07:11:48.518479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile a_read_and_preprocess.py\ndef a_read_and_preprocess(dataset, adc_info, planet_ids):\n    a_raw_train = np.full((len(planet_ids), 5625), np.nan, dtype=np.float32)\n    for i, planet_id in tqdm(list(enumerate(planet_ids))):\n        signal = pl.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/AIRS-CH0_signal.parquet')\n        mean_signal = signal.cast(pl.Int32).sum_horizontal().cast(pl.Float32).to_numpy() / (32*356) # mean over the 32*356 pixels\n        net_signal = mean_signal[1::2] - mean_signal[0::2]\n        a_raw_train[i] = net_signal\n    return a_raw_train","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:11:48.52698Z","iopub.execute_input":"2024-09-09T07:11:48.52763Z","iopub.status.idle":"2024-09-09T07:11:48.544257Z","shell.execute_reply.started":"2024-09-09T07:11:48.527557Z","shell.execute_reply":"2024-09-09T07:11:48.542958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exec(open('a_read_and_preprocess.py', 'r').read())\na_raw_train = a_read_and_preprocess('train', data_ariel_train, train_labels.index)\nwith open('a_raw_train.pickle', 'wb') as f:\n    pickle.dump(a_raw_train, f)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:11:48.553024Z","iopub.execute_input":"2024-09-09T07:11:48.553934Z","iopub.status.idle":"2024-09-09T07:31:40.253587Z","shell.execute_reply.started":"2024-09-09T07:11:48.553893Z","shell.execute_reply":"2024-09-09T07:31:40.252112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile feature_engineering.py\ndef feature_engineering(f_raw, a_raw):\n    obscured = f_raw[:, 23500:44000].mean(axis=1)\n    unobscured = (f_raw[:, :20500].mean(axis=1) + f_raw[:, 47000:].mean(axis=1)) / 2\n    f_relative_reduction = (unobscured - obscured) / unobscured\n    obscured = a_raw[:, 1958:3666].mean(axis=1)\n    unobscured = (a_raw[:, :1708].mean(axis=1) + a_raw[:, 3916:].mean(axis=1)) / 2\n    a_relative_reduction = (unobscured - obscured) / unobscured    \n    return pd.DataFrame({'a_relative_reduction': a_relative_reduction,\n                       'f_relative_reduction': f_relative_reduction})","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:31:40.255961Z","iopub.execute_input":"2024-09-09T07:31:40.256435Z","iopub.status.idle":"2024-09-09T07:31:40.265751Z","shell.execute_reply.started":"2024-09-09T07:31:40.256394Z","shell.execute_reply":"2024-09-09T07:31:40.264468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exec(open('feature_engineering.py', 'r').read())\ntrain = feature_engineering(f_raw_train, a_raw_train)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:31:40.267524Z","iopub.execute_input":"2024-09-09T07:31:40.268206Z","iopub.status.idle":"2024-09-09T07:31:40.329539Z","shell.execute_reply.started":"2024-09-09T07:31:40.268158Z","shell.execute_reply":"2024-09-09T07:31:40.328079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:31:40.331549Z","iopub.execute_input":"2024-09-09T07:31:40.332149Z","iopub.status.idle":"2024-09-09T07:31:40.737186Z","shell.execute_reply.started":"2024-09-09T07:31:40.332111Z","shell.execute_reply":"2024-09-09T07:31:40.73557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = RandomForestRegressor()\npred = cross_val_predict(model, train, train_labels)\nsigma_pred = mean_squared_error(train_labels, pred, squared=False)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:45:44.556115Z","iopub.execute_input":"2024-09-09T07:45:44.556534Z","iopub.status.idle":"2024-09-09T07:45:51.61255Z","shell.execute_reply.started":"2024-09-09T07:45:44.556503Z","shell.execute_reply":"2024-09-09T07:45:51.611256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check model","metadata":{}},{"cell_type":"code","source":"col = 1\nplt.scatter(pred[:,col], train_labels.iloc[:,col], s=15, c='lightgreen')\nplt.gca().set_aspect('equal')\nplt.xlabel('y_pred')\nplt.ylabel('y_true')\nplt.title('Comparing y_true and y_pred')\nplt.show()","metadata":{"papermill":{"duration":0.700113,"end_time":"2024-08-03T13:00:09.837053","exception":false,"start_time":"2024-08-03T13:00:09.13694","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-09T07:45:51.614997Z","iopub.execute_input":"2024-09-09T07:45:51.615535Z","iopub.status.idle":"2024-09-09T07:45:51.980083Z","shell.execute_reply.started":"2024-09-09T07:45:51.61549Z","shell.execute_reply":"2024-09-09T07:45:51.978827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile competition_score.py\n\nclass ParticipantVisibleError(Exception):\n    pass\n\ndef competition_score(solution: pd.DataFrame,submission: pd.DataFrame,naive_mean: float,naive_sigma: float,\n                      sigma_true: float,row_id_column_name='planet_id',) -> float:\n    \n    del solution[row_id_column_name]\n    del submission[row_id_column_name]\n            \n    n_wavelengths = len(solution.columns)\n    y_pred = submission.iloc[:, :n_wavelengths].values\n    sigma_pred = np.clip(submission.iloc[:, n_wavelengths:].values, a_min=10**-15, a_max=None)\n    y_true = solution.values\n\n    GLL_pred = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_pred, scale=sigma_pred))\n    GLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))\n    GLL_mean = np.sum(scipy.stats.norm.logpdf(y_true, loc=naive_mean * np.ones_like(y_true), scale=naive_sigma * np.ones_like(y_true)))\n    submit_score = (GLL_pred - GLL_mean)/(GLL_true - GLL_mean)\n    \n    return float(np.clip(submit_score, 0.0, 1.0))","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.029243,"end_time":"2024-08-03T12:42:17.162838","exception":false,"start_time":"2024-08-03T12:42:17.133595","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-09-09T07:45:51.981723Z","iopub.execute_input":"2024-09-09T07:45:51.982227Z","iopub.status.idle":"2024-09-09T07:45:51.99054Z","shell.execute_reply.started":"2024-09-09T07:45:51.982183Z","shell.execute_reply":"2024-09-09T07:45:51.989196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile postprocessing.py\n\ndef postprocessing(pred_array, index, sigma_pred):\n    new_df = pd.concat([pd.DataFrame(pred_array.clip(0, None), index=index, columns=wavelengths.columns),pd.DataFrame(sigma_pred, index=index, columns=[f\"sigma_{i}\" for i in range(1, 284)])],axis=1)\n    return new_df","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:45:54.495918Z","iopub.execute_input":"2024-09-09T07:45:54.496336Z","iopub.status.idle":"2024-09-09T07:45:54.503955Z","shell.execute_reply.started":"2024-09-09T07:45:54.496305Z","shell.execute_reply":"2024-09-09T07:45:54.50244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exec(open('competition_score.py', 'r').read())\nexec(open('postprocessing.py', 'r').read())\noof_df = postprocessing(pred, data_ariel_train.index, sigma_pred)\ndisplay(oof_df)","metadata":{"papermill":{"duration":0.231627,"end_time":"2024-08-03T13:00:10.149224","exception":false,"start_time":"2024-08-03T13:00:09.917597","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-09-09T07:45:55.576341Z","iopub.execute_input":"2024-09-09T07:45:55.576924Z","iopub.status.idle":"2024-09-09T07:45:55.621865Z","shell.execute_reply.started":"2024-09-09T07:45:55.576886Z","shell.execute_reply":"2024-09-09T07:45:55.62061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gll_score = competition_score(train_labels.copy().reset_index(),\n                              oof_df.copy().reset_index(),naive_mean=train_labels.values.mean(),\n                              naive_sigma=train_labels.values.std(),sigma_true=0.000003)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:45:56.620093Z","iopub.execute_input":"2024-09-09T07:45:56.621329Z","iopub.status.idle":"2024-09-09T07:45:56.666503Z","shell.execute_reply.started":"2024-09-09T07:45:56.621272Z","shell.execute_reply":"2024-09-09T07:45:56.665044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"# score: {gll_score}\")","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:45:57.752192Z","iopub.execute_input":"2024-09-09T07:45:57.753196Z","iopub.status.idle":"2024-09-09T07:45:57.759403Z","shell.execute_reply.started":"2024-09-09T07:45:57.753155Z","shell.execute_reply":"2024-09-09T07:45:57.757761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(train, train_labels)\nwith open('model.pickle', 'wb') as f:\n    pickle.dump(model, f)\nwith open('sigma_pred.pickle', 'wb') as f:\n    pickle.dump(sigma_pred, f)","metadata":{"execution":{"iopub.status.busy":"2024-09-09T07:45:58.972111Z","iopub.execute_input":"2024-09-09T07:45:58.973243Z","iopub.status.idle":"2024-09-09T07:46:01.693707Z","shell.execute_reply.started":"2024-09-09T07:45:58.973202Z","shell.execute_reply":"2024-09-09T07:46:01.692124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{"papermill":{"duration":0.077479,"end_time":"2024-08-03T13:00:10.305108","exception":false,"start_time":"2024-08-03T13:00:10.227629","status":"completed"},"tags":[]}},{"cell_type":"code","source":"test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',index_col='planet_id')\nsample_submission = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv',index_col='planet_id')\n\nf_raw_test = f_read_and_preprocess('test', test_adc_info, sample_submission.index)\na_raw_test = a_read_and_preprocess('test', test_adc_info, sample_submission.index)\ntest = feature_engineering(f_raw_test, a_raw_test)\n\nwith open('model.pickle', 'rb') as f:\n    model = pickle.load(f)\nwith open('sigma_pred.pickle', 'rb') as f:\n    sigma_pred = pickle.load(f)\n\ntest_pred = model.predict(test)\n\nsub_df = postprocessing(test_pred, sample_submission.index, sigma_pred)\ndisplay(sub_df)\nsub_df.to_csv('submission.csv')","metadata":{"papermill":{"duration":1.822768,"end_time":"2024-08-03T13:00:12.204995","exception":false,"start_time":"2024-08-03T13:00:10.382227","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-09-09T07:46:01.696654Z","iopub.execute_input":"2024-09-09T07:46:01.697235Z","iopub.status.idle":"2024-09-09T07:46:05.465872Z","shell.execute_reply.started":"2024-09-09T07:46:01.697187Z","shell.execute_reply":"2024-09-09T07:46:05.464777Z"},"trusted":true},"execution_count":null,"outputs":[]}]}