{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70367,"databundleVersionId":9188054,"sourceType":"competition"},{"sourceId":191155259,"sourceType":"kernelVersion"},{"sourceId":196522559,"sourceType":"kernelVersion"}],"dockerImageVersionId":30761,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"One of the [outstanding members](https://www.kaggle.com/cdeotte) of the Kaggle community, grandmaster, gives a link to his [work](https://www.kaggle.com/code/cdeotte/top-solutions-ensemble-0-947), where he shows the refinement of the prediction using an experimental example. In order to try to refine our predictions at the [NeurIPS - Ariel Data Challenge 2024](https://www.kaggle.com/competitions/ariel-data-challenge-2024/leaderboard) competition\n\n**[Ariel | Ensemble of solutions](https://www.kaggle.com/code/vyacheslavbolotin/ensemble-of-solutions/edit)**: Leaderboard=0.5??\n\n1. (0.514) [ariel_only_correlation](https://www.kaggle.com/code/sergeifironov/ariel-only-correlation) by grandmaster [Sergei Fironov](https://www.kaggle.com/sergeifironov)\n2. (0.515) [ariel_only_correlation LB: 0.515](https://www.kaggle.com/code/jaejohn/ariel-only-correlation-lb-0-515) by contributor [G John Rao](https://www.kaggle.com/jaejohn)\n3. (0.446) [Fork of NeurIPS Ariel 2024 - Starter 5be123](https://www.kaggle.com/code/regisvargas/fork-of-neurips-ariel-2024-starter-5be123) by expert [Regis Vargas](https://www.kaggle.com/regisvargas)\n4. (0.441) [AD24[TRAIN-INF]Ridge_AddFE[LB.441]](https://www.kaggle.com/code/hideyukizushi/ad24-train-inf-ridge-addfe-lb-441) by expert [yukiZ](https://www.kaggle.com/hideyukizushi)\n5. (0.434) [NeurIPS Ariel 2024 - Starter withdifferentparametr](https://www.kaggle.com/code/bingyuniu/neurips-ariel-2024-starter-withdifferentparametr) by contributor [yu🐂](https://www.kaggle.com/bingyuniu)\n6. ()\n7. (0.517) [ariel_only_correlation | param upd[LB.517]](https://www.kaggle.com/code/hideyukizushi/ariel-only-correlation-param-upd-lb-517) by expert [yukiZ](https://www.kaggle.com/hideyukizushi)\n8. (0.544) [Ariel Data Challenge 2024](https://www.kaggle.com/code/xiaocao123/ariel-data-challenge-2024) by expert [qianc](https://www.kaggle.com/xiaocao123)\n\n\n#### options\n\n- option 1 -> LB=0.515 work (1,2)\n- option 2 -> LB= ?.??? work (3,4,5) \n- option 3 -> LB=0.000 work (1,2,3)         \n- option 4 -> LB= ?.??? work (4,5)\n- option 5 -> LB=0.514 work (1,2) + weight (0.15+0.85)\n- option 7 -> LB=0.515 work (1,2) + weight (0.85+0.15)\n- option 8 -> LB=0.515 work (1,2) + weight (0.07+0.93)\n- option 9 -> LB=0.517 work (2,7)\n- option 10 -> LB=0.516 work (1,2,7)\n- option 11 -> LB=0.517 work (2,7) + weight (0.10+0.90)\n- option 12 -> LB=0.517 work (2,7) + weight (0.05+0.95)\n- option 14 -> LB=0.544 work (7,8) + weight (0.005+0.995)\n- option 15 -> LB=0.544 work (7,8) + weight (0.008+0.992)\n- option 16 -> LB=0.544 work (7,8) + weight (0.011+0.989)\n- option 17 -> LB=0.544 work (7,8) + weight (0.015+0.985)\n- option 18 -> LB=**0.544** work (7,8) + weight (0.02+0.98)\n- option 19 -> LB=0.544 work (7,8) + weight (0.05+0.95)\n- option 20 -> LB=0.544 work (7,8) + weight (0.07+0.93)\n\noption.14 < option.15 < option.16 < option.17 < **option.18**\n\noption.19 < option.20 < **option.18**\n\nbest option:\n- option 18 -> LB=0.544 work (7,8) + weight (0.02+0.98), **Version 37** \n\ncurrent option: \n- option 21 -> work (7,8) + weight (0.03+0.97)\n- option 22 -> work (7,8) + weight (0.04+0.96)\n\nnext option: \n- option 23 -> ?","metadata":{}},{"cell_type":"markdown","source":"### Notebook from https://www.kaggle.com/code/vyacheslavbolotin/ariel-ensemble-of-solutions\n### If it is useful to you, please give your like.\n\n### Below is the link to my other notebook, which is also the first sharer.\n\n### https://www.kaggle.com/code/xiaocao123/ariel-data-challenge-2024","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LAUNCH_VARIANT = 'option 21'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [ariel_only_correlation](https://www.kaggle.com/code/vyacheslavbolotin/isic-2024-2-image-lines)\n\n### [Sergei Fironov](https://www.kaggle.com/sergeifironov)","metadata":{}},{"cell_type":"markdown","source":"### preface \nI was inspired by @AmbrosM idea https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/530152#2969648\n\nThe idea of ​​this approach is to predict the average answer for each test sample. Instead of building models, we will search for the correct answer experimentally. We will select for each spectrum such a multiplier that the transit part multiplied by it will \"line up\" with the other points. The line can be either a straight line or a polynomial up to the 5th degree. For selection, we will use the Nelder-Mead method. The found multiplication factor minus one is our answer.\n\nThere are some changes in data preparation here.\n* dt for dark calibration changed in favor https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data/comments#2964759\n* signal clipped to zero due this https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/530247#2970709\n* I masked hot and dead pixels with NaN in flat and averaging through spatial dimension.","metadata":{}},{"cell_type":"markdown","source":"### **upd for version 2**: it seems I forgot to remove **5x sigma** copied from somewhere. I wonder how this will affect the score.","metadata":{}},{"cell_type":"markdown","source":"### preprocess the data","metadata":{}},{"cell_type":"code","source":"# import pandas as pd\n# import matplotlib.pyplot as plt\n# import numpy as np\n# import seaborn as sns\n# import scipy.stats\n# from tqdm import tqdm\n\n# from sklearn.model_selection import cross_val_predict\n# from sklearn.linear_model import Ridge\n# from sklearn.metrics import r2_score, mean_squared_error\n# import itertools\n# from scipy.optimize import minimize\n# from functools import partial\n# import random, os\n# from astropy.stats import sigma_clip","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n#                            index_col='planet_id')\n# axis_info = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/axis_info.parquet')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def apply_linear_corr(linear_corr,clean_signal):\n#     linear_corr = np.flip(linear_corr, axis=0)\n#     for x, y in itertools.product(\n#                 range(clean_signal.shape[1]), range(clean_signal.shape[2])\n#             ):\n#         poli = np.poly1d(linear_corr[:, x, y])\n#         clean_signal[:, x, y] = poli(clean_signal[:, x, y])\n#     return clean_signal\n\n# def clean_dark(signal, dark, dt):\n#     dark = np.tile(dark, (signal.shape[0], 1, 1))\n#     signal -= dark* dt[:, np.newaxis, np.newaxis]\n#     return signal\n\n# def preproc(dataset, adc_info, sensor, binning = 15):\n#     cut_inf, cut_sup = 39, 321\n#     sensor_sizes_dict = {\"AIRS-CH0\":[[11250, 32, 356], [1, 32, cut_sup-cut_inf]], \"FGS1\":[[135000, 32, 32], [1, 32, 32]]}\n#     binned_dict = {\"AIRS-CH0\":[11250 // binning // 2, 282], \"FGS1\":[135000 // binning // 2]}\n#     linear_corr_dict = {\"AIRS-CH0\":(6, 32, 356), \"FGS1\":(6, 32, 32)}\n#     planet_ids = adc_info.index\n    \n#     feats = []\n#     for i, planet_id in tqdm(list(enumerate(planet_ids))):\n#         signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/{sensor}_signal.parquet').to_numpy()\n#         dark_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dark.parquet', engine='pyarrow').to_numpy()\n#         dead_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dead.parquet', engine='pyarrow').to_numpy()\n#         flat_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/flat.parquet', engine='pyarrow').to_numpy()\n#         linear_corr = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/linear_corr.parquet').values.astype(np.float64).reshape(linear_corr_dict[sensor])\n\n#         signal = signal.reshape(sensor_sizes_dict[sensor][0]) \n#         gain = adc_info[f'{sensor}_adc_gain'].values[i]\n#         offset = adc_info[f'{sensor}_adc_offset'].values[i]\n#         signal = signal / gain + offset\n        \n#         hot = sigma_clip(\n#             dark_frame, sigma=5, maxiters=5\n#         ).mask\n        \n#         if sensor != \"FGS1\":\n#             signal = signal[:, :, cut_inf:cut_sup] #11250 * 32 * 282\n#             #dt = axis_info['AIRS-CH0-integration_time'].dropna().values\n#             dt = np.ones(len(signal))*0.1 \n#             dt[1::2] += 4.5 #@bilzard idea\n#             linear_corr = linear_corr[:, :, cut_inf:cut_sup]\n#             dark_frame = dark_frame[:, cut_inf:cut_sup]\n#             dead_frame = dead_frame[:, cut_inf:cut_sup]\n#             flat_frame = flat_frame[:, cut_inf:cut_sup]\n#             hot = hot[:, cut_inf:cut_sup]\n#         else:\n#             dt = np.ones(len(signal))*0.1\n#             dt[1::2] += 0.1\n            \n#         signal = signal.clip(0) #@graySnow idea\n#         linear_corr_signal = apply_linear_corr(linear_corr, signal)\n#         signal = clean_dark(linear_corr_signal, dark_frame, dt)\n        \n#         flat = flat_frame.reshape(sensor_sizes_dict[sensor][1])\n#         flat[dead_frame.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n#         flat[hot.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n#         signal = signal / flat\n        \n#         if sensor == \"FGS1\":\n#             signal = signal.reshape((sensor_sizes_dict[sensor][0][0], sensor_sizes_dict[sensor][0][1]*sensor_sizes_dict[sensor][0][2]))\n        \n#         mean_signal = np.nanmean(signal, axis=1) # mean over the 32*32(FGS1) or 32(CH0) pixels\n#         cds_signal = (mean_signal[1::2] - mean_signal[0::2])\n        \n#         binned = np.zeros((binned_dict[sensor]))\n#         for j in range(cds_signal.shape[0] // binning):\n#             binned[j] = cds_signal[j*binning:j*binning+binning].mean(axis=0)\n                   \n#         if sensor == \"FGS1\":\n#             binned = binned.reshape((binned.shape[0],1))\n            \n#         feats.append(binned)\n        \n#     return np.stack(feats)\n    \n# pre_train = np.concatenate([preproc('test', test_adc_info, \"FGS1\", 30*12), preproc('test', test_adc_info, \"AIRS-CH0\", 30)], axis=2)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### fit polynoms for each sample","metadata":{}},{"cell_type":"code","source":"# def phase_detector(signal):\n#     phase1, phase2 = None, None\n#     best_drop = 0\n#     for i in range(50//2,150//2):        \n#         t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n#         if t1 > best_drop:\n#             phase1 = i+(20+5)//2\n#             best_drop = t1\n    \n#     best_drop = 0\n#     for i in range(200//2,250//2):\n#         t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n#         if t1 > best_drop:\n#             phase2 = i-5//2\n#             best_drop = t1\n    \n#     return phase1, phase2\n\n# def try_s(signal, p1, p2, deg, s):\n#     out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#     x, y = out, signal[out].tolist()\n#     x = x + list(range(p1,p2))\n\n#     y = y + (signal[p1:p2] * (1 + s[0])).tolist()\n#     z = np.polyfit(x, y, deg)\n#     p = np.poly1d(z)\n#     q = np.abs(p(x) - y).mean()\n\n#     if s < 1e-4:\n#         return q + 1e3\n\n#     return q\n    \n# def calibrate_signal(signal):\n#     p1,p2 = phase_detector(signal)\n\n#     best_deg, best_score = 1, 1e12\n#     for deg in range(1, 6):\n#         f = partial(try_s, signal, p1, p2, deg)\n#         r = minimize(f, [0.001], method = 'Nelder-Mead')\n#         s = r.x[0]\n\n#         out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#         x, y = out, signal[out].tolist()\n#         x = x + list(range(p1,p2))\n#         y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n#         z = np.polyfit(x, y, deg)\n#         p = np.poly1d(z)\n#         q = np.abs(p(x) - y).mean()\n        \n#         if q < best_score:\n#             best_score = q\n#             best_deg = deg\n        \n#         print(deg, q)\n            \n#     z = np.polyfit(x, y, best_deg)\n#     p = np.poly1d(z)\n\n#     return s, x, y, p(x)\n\n# def calibrate_train(signal):\n#     p1,p2 = phase_detector(signal)\n    \n#     best_deg, best_score = 1, 1e12\n#     for deg in range(1, 6):\n#         f = partial(try_s, signal, p1, p2, deg)\n#         r = minimize(f, [0.001], method = 'Nelder-Mead')\n#         s = r.x[0]\n\n#         out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#         x, y = out, signal[out].tolist()\n#         x = x + list(range(p1,p2))\n#         y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n#         z = np.polyfit(x, y, deg)\n#         p = np.poly1d(z)\n#         q = np.abs(p(x) - y).mean()\n        \n#         if q < best_score:\n#             best_score = q\n#             best_deg = deg\n            \n#     z = np.polyfit(x, y, best_deg)\n#     p = np.poly1d(z)\n    \n#     return s, p(np.arange(signal.shape[0])), p1, p2\n\n\n# train = pre_train.copy()\n# all_s = []\n# for i in range(len(test_adc_info)):\n#     signal = train[i,:,1:].mean(axis=1)\n#     s, p, p1, p2 = calibrate_train(pre_train[i,:,1:].mean(axis=1))\n#     all_s.append(s)\n        \n# #copy answer 283 times because we predict mean value\n# train_s = np.repeat(np.array(all_s), 283).reshape((len(all_s), 283))        \n# train_sigma = np.ones_like(train_s) * 0.00016","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probably we can accurately estimate sigma from train","metadata":{}},{"cell_type":"code","source":"# n = 0\n# s, x, y, y_new = calibrate_signal(pre_train[n,:,1:].mean(axis=1))\n# plt.scatter(x,y)\n# plt.scatter(x,y_new)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I call the orange line \"starline\". This is probably what we would see if the planet weren't in the way.","metadata":{}},{"cell_type":"markdown","source":"### Making submission","metadata":{}},{"cell_type":"code","source":"# ss = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv')\n\n# preds = train_s.clip(0)\n# sigmas = train_sigma\n# submission = pd.DataFrame(np.concatenate([preds,sigmas], axis=1), columns=ss.columns[1:])\n# submission.index = test_adc_info.index\n# submission.to_csv('submission_1.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [ariel_local_purple_hat_inference](https://www.kaggle.com/code/sergeifironov/ariel-local-purple-hat-inference)\n\n### [Sergei Fironov](https://www.kaggle.com/sergeifironov)","metadata":{}},{"cell_type":"code","source":"","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [ariel_only_correlation LB: 0.515](https://www.kaggle.com/code/jaejohn/ariel-only-correlation-lb-0-515)\n\n### [G John Rao](https://www.kaggle.com/jaejohn)","metadata":{}},{"cell_type":"markdown","source":"### preface \nI was inspired by @AmbrosM idea https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/530152#2969648\n\nThe idea of ​​this approach is to predict the average answer for each test sample. Instead of building models, we will search for the correct answer experimentally. We will select for each spectrum such a multiplier that the transit part multiplied by it will \"line up\" with the other points. The line can be either a straight line or a polynomial up to the 5th degree. For selection, we will use the Nelder-Mead method. The found multiplication factor minus one is our answer.\n\nThere are some changes in data preparation here.\n* dt for dark calibration changed in favor https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data/comments#2964759\n* signal clipped to zero due this https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/530247#2970709\n* I masked hot and dead pixels with NaN in flat and averaging through spatial dimension.","metadata":{}},{"cell_type":"markdown","source":"### **upd for version 2**: it seems I forgot to remove **5x sigma** copied from somewhere. I wonder how this will affect the score.\n\n#### preprocess the data","metadata":{}},{"cell_type":"code","source":"# import pandas as pd\n# import matplotlib.pyplot as plt\n# import numpy as np\n# import seaborn as sns\n# import scipy.stats\n# from tqdm import tqdm\n\n# from sklearn.model_selection import cross_val_predict\n# from sklearn.linear_model import Ridge\n# from sklearn.metrics import r2_score, mean_squared_error\n# import itertools\n# from scipy.optimize import minimize\n# from functools import partial\n# import random, os\n# from astropy.stats import sigma_clip","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n#                            index_col='planet_id')\n# axis_info = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/axis_info.parquet')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def apply_linear_corr(linear_corr,clean_signal):\n#     linear_corr = np.flip(linear_corr, axis=0)\n#     for x, y in itertools.product(\n#                 range(clean_signal.shape[1]), range(clean_signal.shape[2])\n#             ):\n#         poli = np.poly1d(linear_corr[:, x, y])\n#         clean_signal[:, x, y] = poli(clean_signal[:, x, y])\n#     return clean_signal\n\n# def clean_dark(signal, dark, dt):\n#     dark = np.tile(dark, (signal.shape[0], 1, 1))\n#     signal -= dark* dt[:, np.newaxis, np.newaxis]\n#     return signal\n\n# def preproc(dataset, adc_info, sensor, binning = 15):\n#     cut_inf, cut_sup = 39, 321\n#     sensor_sizes_dict = {\"AIRS-CH0\":[[11250, 32, 356], [1, 32, cut_sup-cut_inf]], \"FGS1\":[[135000, 32, 32], [1, 32, 32]]}\n#     binned_dict = {\"AIRS-CH0\":[11250 // binning // 2, 282], \"FGS1\":[135000 // binning // 2]}\n#     linear_corr_dict = {\"AIRS-CH0\":(6, 32, 356), \"FGS1\":(6, 32, 32)}\n#     planet_ids = adc_info.index\n    \n#     feats = []\n#     for i, planet_id in tqdm(list(enumerate(planet_ids))):\n#         signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/{sensor}_signal.parquet').to_numpy()\n#         dark_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dark.parquet', engine='pyarrow').to_numpy()\n#         dead_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dead.parquet', engine='pyarrow').to_numpy()\n#         flat_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/flat.parquet', engine='pyarrow').to_numpy()\n#         linear_corr = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/linear_corr.parquet').values.astype(np.float64).reshape(linear_corr_dict[sensor])\n\n#         signal = signal.reshape(sensor_sizes_dict[sensor][0]) \n#         gain = adc_info[f'{sensor}_adc_gain'].values[i]\n#         offset = adc_info[f'{sensor}_adc_offset'].values[i]\n#         signal = signal / gain + offset\n        \n#         hot = sigma_clip(\n#             dark_frame, sigma=5, maxiters=5\n#         ).mask\n        \n#         if sensor != \"FGS1\":\n#             signal = signal[:, :, cut_inf:cut_sup] #11250 * 32 * 282\n#             #dt = axis_info['AIRS-CH0-integration_time'].dropna().values\n#             dt = np.ones(len(signal))*0.1 \n#             dt[1::2] += 4.5 #@bilzard idea\n#             linear_corr = linear_corr[:, :, cut_inf:cut_sup]\n#             dark_frame = dark_frame[:, cut_inf:cut_sup]\n#             dead_frame = dead_frame[:, cut_inf:cut_sup]\n#             flat_frame = flat_frame[:, cut_inf:cut_sup]\n#             hot = hot[:, cut_inf:cut_sup]\n#         else:\n#             dt = np.ones(len(signal))*0.1\n#             dt[1::2] += 0.1\n            \n#         signal = signal.clip(0) #@graySnow idea\n#         linear_corr_signal = apply_linear_corr(linear_corr, signal)\n#         signal = clean_dark(linear_corr_signal, dark_frame, dt)\n        \n#         flat = flat_frame.reshape(sensor_sizes_dict[sensor][1])\n#         flat[dead_frame.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n#         flat[hot.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n#         signal = signal / flat\n        \n#         if sensor == \"FGS1\":\n#             signal = signal.reshape((sensor_sizes_dict[sensor][0][0], sensor_sizes_dict[sensor][0][1]*sensor_sizes_dict[sensor][0][2]))\n        \n#         mean_signal = np.nanmean(signal, axis=1) # mean over the 32*32(FGS1) or 32(CH0) pixels\n#         cds_signal = (mean_signal[1::2] - mean_signal[0::2])\n        \n#         binned = np.zeros((binned_dict[sensor]))\n#         for j in range(cds_signal.shape[0] // binning):\n#             binned[j] = cds_signal[j*binning:j*binning+binning].mean(axis=0)\n                   \n#         if sensor == \"FGS1\":\n#             binned = binned.reshape((binned.shape[0],1))\n            \n#         feats.append(binned)\n        \n#     return np.stack(feats)\n    \n# pre_train = np.concatenate([preproc('test', test_adc_info, \"FGS1\", 30*12), preproc('test', test_adc_info, \"AIRS-CH0\", 30)], axis=2)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### fit polynoms for each sample","metadata":{}},{"cell_type":"code","source":"# def phase_detector(signal):\n#     phase1, phase2 = None, None\n#     best_drop = 0\n#     for i in range(50//2,150//2):        \n#         t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n#         if t1 > best_drop:\n#             phase1 = i+(20+5)//2\n#             best_drop = t1\n    \n#     best_drop = 0\n#     for i in range(200//2,250//2):\n#         t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n#         if t1 > best_drop:\n#             phase2 = i-5//2\n#             best_drop = t1\n    \n#     return phase1, phase2\n\n# def try_s(signal, p1, p2, deg, s):\n#     out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#     x, y = out, signal[out].tolist()\n#     x = x + list(range(p1,p2))\n\n#     y = y + (signal[p1:p2] * (1 + s[0])).tolist()\n#     z = np.polyfit(x, y, deg)\n#     p = np.poly1d(z)\n#     q = np.abs(p(x) - y).mean()\n\n#     if s < 1e-4:\n#         return q + 1e3\n\n#     return q\n    \n# def calibrate_signal(signal):\n#     p1,p2 = phase_detector(signal)\n\n#     best_deg, best_score = 1, 1e12\n#     for deg in range(1, 6):\n#         f = partial(try_s, signal, p1, p2, deg)\n#         r = minimize(f, [0.001], method = 'Nelder-Mead')\n#         s = r.x[0]\n\n#         out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#         x, y = out, signal[out].tolist()\n#         x = x + list(range(p1,p2))\n#         y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n#         z = np.polyfit(x, y, deg)\n#         p = np.poly1d(z)\n#         q = np.abs(p(x) - y).mean()\n        \n#         if q < best_score:\n#             best_score = q\n#             best_deg = deg\n        \n#         print(deg, q)\n            \n#     z = np.polyfit(x, y, best_deg)\n#     p = np.poly1d(z)\n\n#     return s, x, y, p(x)\n\n# def calibrate_train(signal):\n#     p1,p2 = phase_detector(signal)\n    \n#     best_deg, best_score = 1, 1e12\n#     for deg in range(1, 6):\n#         f = partial(try_s, signal, p1, p2, deg)\n#         r = minimize(f, [0.001], method = 'Nelder-Mead')\n#         s = r.x[0]\n\n#         out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n#         x, y = out, signal[out].tolist()\n#         x = x + list(range(p1,p2))\n#         y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n#         z = np.polyfit(x, y, deg)\n#         p = np.poly1d(z)\n#         q = np.abs(p(x) - y).mean()\n        \n#         if q < best_score:\n#             best_score = q\n#             best_deg = deg\n            \n#     z = np.polyfit(x, y, best_deg)\n#     p = np.poly1d(z)\n    \n#     return s, p(np.arange(signal.shape[0])), p1, p2\n\n\n# train = pre_train.copy()\n# all_s = []\n# for i in range(len(test_adc_info)):\n#     signal = train[i,:,1:].mean(axis=1)\n#     s, p, p1, p2 = calibrate_train(pre_train[i,:,1:].mean(axis=1))\n#     all_s.append(s)\n        \n# #copy answer 283 times because we predict mean value\n# train_s = np.repeat(np.array(all_s), 283).reshape((len(all_s), 283))        \n# train_sigma = np.ones_like(train_s) * 0.000176","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probably we can accurately estimate sigma from train","metadata":{}},{"cell_type":"code","source":"# n = 0\n# s, x, y, y_new = calibrate_signal(pre_train[n,:,1:].mean(axis=1))\n# plt.scatter(x,y)\n# plt.scatter(x,y_new)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I call the orange line \"starline\". This is probably what we would see if the planet weren't in the way.","metadata":{}},{"cell_type":"markdown","source":"### Making submission","metadata":{}},{"cell_type":"code","source":"# ss = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv')\n\n# preds = train_s.clip(0)\n# sigmas = train_sigma\n# submission = pd.DataFrame(np.concatenate([preds,sigmas], axis=1), columns=ss.columns[1:])\n# submission.index = test_adc_info.index\n# submission.to_csv('submission_2.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [Fork of NeurIPS Ariel 2024 - Starter 5be123](https://www.kaggle.com/code/regisvargas/fork-of-neurips-ariel-2024-starter-5be123)\n\n### [Regis Vargas](https://www.kaggle.com/regisvargas)","metadata":{}},{"cell_type":"markdown","source":"References\n\n1. [Why Did Calibration Lead to a Lower Public Score When Combining Two Kaggle Notebooks?](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/530472)\n\n2. [Fork of NeurIPS Ariel 2024 - Starter 5be123](https://www.kaggle.com/code/regisvargas/fork-of-neurips-ariel-2024-starter-5be123)\n\n3. [NeurIPS Ariel 2024 - Starter withdifferentparametr](https://www.kaggle.com/code/bingyuniu/neurips-ariel-2024-starter-withdifferentparametr)\n\n4. [[UPDATE]Calibrating and Binning Astronomical Data](https://www.kaggle.com/code/gordonyip/update-calibrating-and-binning-astronomical-data)\n\n5. [[UPDATE]Calibrating and Binning Astronomical Data (copy)](https://www.kaggle.com/code/aaronjday/update-calibrating-and-binning-astronomical-data)","metadata":{}},{"cell_type":"markdown","source":"### Initialization\n\nThis competition seems requires strong scientific background and I had lot of confusion during EDA process. Therefore, I just build a simple starter for future coding.\n\n#### Load Library","metadata":{}},{"cell_type":"code","source":"# import matplotlib.pyplot as plt\n# import seaborn as sns\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# import torch\n# # import torch.nn as nn\n# # import torch.optim as optim\n# # from torch.utils.data import DataLoader, Dataset\n# from tqdm import tqdm\n# import pickle\n# import time\n# import os\n# import pickle\n# import seaborn as sns\n# import scipy.stats\n# from sklearn.model_selection import cross_val_predict\n# from sklearn.linear_model import Ridge\n# from sklearn.metrics import r2_score, mean_squared_error","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Load Meta-Data\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# train_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_adc_info.csv', \n#                              index_col='planet_id')\n# train_labels = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_labels.csv',\n#                            index_col='planet_id')\n# wavelengths = pd.read_csv(f'{PATH}/wavelengths.csv')\n# axis_info = pd.read_parquet(os.path.join(PATH,'axis_info.parquet'))","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_adc_info['AIRS-CH0_adc_gain'].loc[785834]","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pre-Processing\n#### Load Functions","metadata":{}},{"cell_type":"code","source":"# import pywt\n# def wavelet_denoising(data, wavelet = 'db10', sigma=None):\n#     \"\"\"Denoises the signal using SURE wavelet shrinkage with the specified wavelet.\"\"\"\n#     # Criar uma cópia do array para evitar o erro de \"read-only\"\n#     data = np.array(data, copy=True)\n    \n#     # Decompose the signal using discrete wavelet transform\n#     coeffs = pywt.wavedec(data, wavelet)\n    \n#     # Estimate noise level if sigma is not provided\n#     if sigma is None:\n#         # Using the Median Absolute Deviation (MAD) estimator for noise level\n#         sigma = np.median(np.abs(coeffs[-1])) / 0.6745\n#     # Apply thresholding (SURE or hard/soft)\n#     threshold = sigma * np.sqrt(2 * np.log(len(data)))\n#     new_coeffs = [pywt.threshold(c, threshold, mode='soft') for c in coeffs]\n#     # Reconstruct the signal using the modified coefficients\n#     return pywt.waverec(new_coeffs, wavelet)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%writefile utils.py\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# from tqdm import tqdm\n# import pickle\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# cut_inf, cut_sup = 39, 321\n# def load_signal_data(planet_id, dataset, instrument, img_size):\n#     file_path = f'{PATH}/{dataset}/{planet_id}/{instrument}_signal.parquet'\n#     signal = pd.read_parquet(file_path)\n#     if instrument == \"AIRS-CH0\":\n#         signal = signal.values.astype(np.float64).reshape((signal.shape[0], 32, 356))\n#     else:\n#         signal = signal.values.astype(np.float64).reshape((signal.shape[0], 32, 32))\n#     if dataset == 'train':\n#         gain = train_adc_info[instrument+'_adc_gain'].loc[planet_id]\n#         offset = train_adc_info[instrument+'_adc_offset'].loc[planet_id]\n#         signal = ADC_convert(signal, gain, offset)\n#     else:\n#         gain = test_adc_info[instrument+'_adc_gain'].loc[planet_id]\n#         offset = test_adc_info[instrument+'_adc_offset'].loc[planet_id]\n#         signal = ADC_convert(signal, gain, offset)\n#     if instrument == \"AIRS-CH0\":\n#         dt_airs = axis_info['AIRS-CH0-integration_time'].dropna().values\n#         dt_airs[1::2] += 0.1\n#         signal = signal[:, :, cut_inf:cut_sup]\n#     signal = signal.reshape(signal.shape[0], signal.shape[1] * signal.shape[2])\n#     mean_signal = signal.mean(axis=1)\n#     mean_signal = mean_signal / np.linalg.norm(mean_signal)\n#     net_signal = mean_signal[1::2] - mean_signal[0::2]\n#     #return wavelet_denoising(net_signal)\n#     return net_signal\n# def ADC_convert(signal, gain, offset):\n#     signal = signal.astype(np.float64)\n#     signal /= gain\n#     signal += offset\n#     return signal\n# def read_and_preprocess(dataset, planet_ids, instrument = \"AIRS-CH0\"):\n#     \"\"\"Read the files for all planet_ids and extract the time series.\n#     Parameters\n#     dataset: 'train' or 'test'\n#     planet_ids: list of planet ids\n#     instrument: the instrument of observation, 'AIRS-CH0' or 'FGS1', default to 'AIRS-CH0'\n#     Returns\n#     dataframe with one row per planet_id and 67500 values per row for FGS1 and 5624 for AIRS-CH0\n#     \"\"\"\n#     img_size = 1024 if instrument == \"FGS1\" else 32*356\n#     column_num = 67500 if instrument == 'FGS1' else 5625\n#     raw_train = np.full((len(planet_ids), column_num), np.nan, dtype=np.float32)\n#     for i, planet_id in tqdm(list(enumerate(planet_ids))):\n#         raw_train[i] = load_signal_data(planet_id, dataset, instrument, img_size)\n#     return raw_train\n# def feature_engineering(f_raw, a_raw, adc_info, window_size=50, step_size=15):\n#     \"\"\"Create a dataframe with combined features from the raw data, including sliding window and time-series statistics.\n    \n#     Parameters:\n#     f_raw: ndarray of shape (n_planets, 67500)\n#     a_raw: ndarray of shape (n_planets, 5625)\n#     window_size: int, size of the sliding window for time-series statistics\n#     step_size: int, step size for the sliding window\n    \n#     Return value:\n#     df: DataFrame of shape (n_planets, several features)\n#     \"\"\"\n#     f_obscured = f_raw[:, 23500:44000].mean(axis=1)\n#     f_unobscured = (f_raw[:, :20500].mean(axis=1) + f_raw[:, 47000:].mean(axis=1)) / 2\n#     f_relative_reduction = (f_unobscured - f_obscured) / f_unobscured\n#     f_std_dev = f_raw.std(axis=1)\n#     f_signal_to_noise = f_unobscured / f_std_dev\n#     a_obscured = a_raw[:, 1958:3666].mean(axis=1)\n#     a_unobscured = (a_raw[:, :1708].mean(axis=1) + a_raw[:, 3916:].mean(axis=1)) / 2\n#     a_relative_reduction = (a_unobscured - a_obscured) / a_unobscured\n#     a_std_dev = a_raw.std(axis=1)\n#     a_signal_to_noise = a_unobscured / a_std_dev\n#     f_variance = f_raw.var(axis=1)\n#     a_variance = a_raw.var(axis=1)\n    \n#     f_skewness = pd.DataFrame(f_raw).skew(axis=1).values\n#     a_skewness = pd.DataFrame(a_raw).skew(axis=1).values\n#     f_kurtosis = pd.DataFrame(f_raw).kurtosis(axis=1).values\n#     a_kurtosis = pd.DataFrame(a_raw).kurtosis(axis=1).values\n    \n#     f_half_obscured1 = f_raw[:, 20500:23500].mean(axis=1)\n#     f_half_obscured2 = f_raw[:, 44000:47000].mean(axis=1)\n#     f_half_reduction1 = (f_unobscured - f_half_obscured1) / f_unobscured\n#     f_half_reduction2 = (f_unobscured - f_half_obscured2) / f_unobscured\n#     a_half_obscured1 = a_raw[:, 1708:1958].mean(axis=1)\n#     a_half_obscured2 = a_raw[:, 3666:3916].mean(axis=1)\n#     a_half_reduction1 = (a_unobscured - a_half_obscured1) / a_unobscured\n#     a_half_reduction2 = (a_unobscured - a_half_obscured2) / a_unobscured\n#     # Sliding window features\n#     def sliding_window_features(data, window_size, step_size):\n#         features = []\n#         max_index = data.shape[1]\n#         for start in range(0, max_index - window_size + 1, step_size):\n#             end = start + window_size\n#             window = data[:, start:end]\n#             features.append([\n#                 np.mean(window, axis=1),\n#                 np.std(window, axis=1),\n#                 np.min(window, axis=1),\n#                 np.max(window, axis=1)\n#             ])\n#         if features:\n#             return np.vstack(features).T  # Stack vertically and transpose to get the correct shape\n#         else:\n#             return np.empty((data.shape[0], 0))  # Return empty array with correct shape\n    \n#     f_sliding_features = sliding_window_features(f_raw, window_size, step_size)\n#     a_sliding_features = sliding_window_features(a_raw, window_size, step_size)\n#     print(f'f_sliding_features.shape: {f_sliding_features.shape}')\n#     print(f'a_sliding_features.shape: {a_sliding_features.shape}')\n#     df = pd.DataFrame({\n#         'f_relative_reduction': f_relative_reduction,\n#         'f_signal_to_noise': f_signal_to_noise,\n#         'f_variance': f_variance,\n#         'f_skewness': f_skewness,\n#         'f_kurtosis': f_kurtosis,\n#         'a_relative_reduction': a_relative_reduction,\n#         'a_signal_to_noise': a_signal_to_noise,\n#         'a_variance': a_variance,\n#         'a_skewness': a_skewness,\n#         'a_kurtosis': a_kurtosis,\n#         'f_half_reduction1': f_half_reduction1,\n#         'f_half_reduction2': f_half_reduction2,\n#         'a_half_reduction1': a_half_reduction1,\n#         'a_half_reduction2': a_half_reduction2\n#     })\n#     if f_sliding_features.size > 0:\n#         f_sliding_df = pd.DataFrame(f_sliding_features, columns=[f'f_slide_{i}' for i in range(f_sliding_features.shape[1])])\n#         df = pd.concat([df, f_sliding_df], axis=1)\n#     if a_sliding_features.size > 0:\n#         a_sliding_df = pd.DataFrame(a_sliding_features, columns=[f'a_slide_{i}' for i in range(a_sliding_features.shape[1])])\n#         df = pd.concat([df, a_sliding_df], axis=1)\n    \n#     df = pd.concat([df, adc_info.reset_index().iloc[:, 1:6]], axis=1)\n    \n#     return df","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%writefile -a utils.py\n\n# def postprocessing(pred_array, index, sigma_pred):\n#     \"\"\"Create a submission dataframe from its components\n    \n#     Parameters:\n#     pred_array: ndarray of shape (n_samples, 283)\n#     index: pandas.Index of length n_samples with name 'planet_id'\n#     sigma_pred: float\n    \n#     Return value:\n#     df: DataFrame of shape (n_samples, 566) with planet_id as index\n#     \"\"\"\n#     return pd.concat([pd.DataFrame(pred_array.clip(0, None), index=index, columns=wavelengths.columns),\n#                       pd.DataFrame(sigma_pred, index=index, columns=[f\"sigma_{i}\" for i in range(1, 284)])],\n#                      axis=1)\n\n# class ParticipantVisibleError(Exception):\n#     pass\n\n# def competition_score(\n#         solution: pd.DataFrame,\n#         submission: pd.DataFrame,\n#         naive_mean: float,\n#         naive_sigma: float,\n#         sigma_true: float,\n#         row_id_column_name='planet_id',\n#     ) -> float:\n#     '''\n#     This is a Gaussian Log Likelihood based metric. For a submission, which contains the predicted mean (x_hat) and variance (x_hat_std),\n#     we calculate the Gaussian Log-likelihood (GLL) value to the provided ground truth (x). We treat each pair of x_hat,\n#     x_hat_std as a 1D gaussian, meaning there will be 283 1D gaussian distributions, hence 283 values for each test spectrum,\n#     the GLL value for one spectrum is the sum of all of them.\n\n#     Inputs:\n#         - solution: Ground Truth spectra (from test set)\n#             - shape: (nsamples, n_wavelengths)\n#         - submission: Predicted spectra and errors (from participants)\n#             - shape: (nsamples, n_wavelengths*2)\n#         naive_mean: (float) mean from the train set.\n#         naive_sigma: (float) standard deviation from the train set.\n#         sigma_true: (float) essentially sets the scale of the outputs.\n#     '''\n\n#     del solution[row_id_column_name]\n#     del submission[row_id_column_name]\n\n#     if submission.min().min() < 0:\n#         raise ParticipantVisibleError('Negative values in the submission')\n#     for col in submission.columns:\n#         if not pd.api.types.is_numeric_dtype(submission[col]):\n#             raise ParticipantVisibleError(f'Submission column {col} must be a number')\n\n#     n_wavelengths = len(solution.columns)\n#     if len(submission.columns) != n_wavelengths*2:\n#         raise ParticipantVisibleError('Wrong number of columns in the submission')\n\n#     y_pred = submission.iloc[:, :n_wavelengths].values\n#     # Set a non-zero minimum sigma pred to prevent division by zero errors.\n#     sigma_pred = np.clip(submission.iloc[:, n_wavelengths:].values, a_min=10**-15, a_max=None)\n#     y_true = solution.values\n\n#     GLL_pred = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_pred, scale=sigma_pred))\n#     GLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))\n#     GLL_mean = np.sum(scipy.stats.norm.logpdf(y_true, loc=naive_mean * np.ones_like(y_true), scale=naive_sigma * np.ones_like(y_true)))\n\n#     submit_score = (GLL_pred - GLL_mean)/(GLL_true - GLL_mean)\n#     return float(np.clip(submit_score, 0.0, 1.0))","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# exec(open('utils.py', 'r').read())","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Data","metadata":{}},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/f_raw_train.pickle\"):\n#     f_raw_train = np.load('/kaggle/input/adc24-intro-training/f_raw_train.pickle', allow_pickle=True)\n# else:\n#     f_raw_train = read_and_preprocess('train', train_labels.index, 'FGS1')\n#     with open('f_raw_train.pickle', 'wb') as f:\n#         pickle.dump(f_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/a_raw_train.pickle\"):\n#     a_raw_train = np.load('/kaggle/input/adc24-intro-training/a_raw_train.pickle', allow_pickle=True)\n# else:\n#     a_raw_train = read_and_preprocess('train', train_labels.index)\n#     with open('a_raw_train.pickle', 'wb') as f:\n#         pickle.dump(a_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering","metadata":{}},{"cell_type":"code","source":"# %%time\n# train = feature_engineering(f_raw_train, a_raw_train, train_adc_info)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train.head()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = train.iloc[:,:-1]","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Plot","metadata":{}},{"cell_type":"code","source":"# plt.figure(figsize=(6, 2))\n# plt.plot(f_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step, color='gray')\n# plt.xlabel('time step')\n# plt.title('FGS1: Overall mean')\n# plt.show()\n\n# plt.figure(figsize=(6, 2))\n# plt.plot(a_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step * 11250 // 135000, color='gray')\n# plt.xlabel('time step')\n# plt.title('AIRS-CH0: Overall mean')\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# color_array = np.array(plt.rcParams['axes.prop_cycle'].by_key()['color'])\n# plt.scatter(train.a_relative_reduction, train_labels.wl_1, s=15, alpha=0.5,\n#             c=color_array[train_adc_info.star])\n# plt.xlabel('relative signal reduction when planet is in front')\n# plt.ylabel('target')\n# plt.title('Correlation between relative signal reduction and target')\n# # plt.gca().set_aspect('equal')\n# points = [plt.Line2D([0], [0], label=f'star {i}', marker='o', markersize=3,\n#          markeredgecolor=color_array[i], markerfacecolor=color_array[i], linestyle='') for i in range(2)]\n\n# plt.legend(handles=points)\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model\n#### Rigde Model","metadata":{}},{"cell_type":"code","source":"# model = Ridge(alpha=1e-12)\n\n# oof_pred = cross_val_predict(model, train, train_labels)\n\n# print(f\"# R2 score: {r2_score(train_labels, oof_pred):.4f}\")\n# sigma_pred = mean_squared_error(train_labels, oof_pred, squared=False)\n# print(f\"# Root mean squared error: {sigma_pred:.7f}\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# oof_df = postprocessing(oof_pred, train_adc_info.index, sigma_pred)\n# display(oof_df)\n\n# gll_score = competition_score(train_labels.copy().reset_index(),\n#                               oof_df.copy().reset_index(),\n#                               naive_mean=train_labels.values.mean(),\n#                               naive_sigma=train_labels.values.std(),\n#                               sigma_true=0.000003)\n# print(f\"# Estimated competition score: {gll_score:.4f}\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.fit(train, train_labels)\n# with open('model.pickle', 'wb') as f:\n#     pickle.dump(model, f)\n# with open('sigma_pred.pickle', 'wb') as f:\n#     pickle.dump(sigma_pred, f)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference","metadata":{}},{"cell_type":"code","source":"# # Load the data\n# test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n#                            index_col='planet_id')\n# sample_submission = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv',\n#                                 index_col='planet_id')\n# f_raw_test = read_and_preprocess('test', sample_submission.index, 'FGS1')\n# a_raw_test = read_and_preprocess('test', sample_submission.index)\n# test = feature_engineering(f_raw_test, a_raw_test, test_adc_info)\n# test = test.iloc[: , :-1]\n# # Load the model\n# with open('model.pickle', 'rb') as f:\n#     model = pickle.load(f)\n# with open('sigma_pred.pickle', 'rb') as f:\n#     sigma_pred = pickle.load(f)\n\n# # Predict\n# test_pred = model.predict(test)\n\n# # Package into submission file\n# sub_df = sub_df = postprocessing(test_pred,\n#                         test_adc_info.index,\n#                         sigma_pred=np.tile(np.where(test_adc_info[['star']] <= 1, 0.0001555, 0.00085), (1, 283)))\n# display(sub_df)\n# sub_df.to_csv('submission_3.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [AD24[TRAIN-INF]Ridge_AddFE[LB.441]](https://www.kaggle.com/code/hideyukizushi/ad24-train-inf-ridge-addfe-lb-441)\n\n### [yukiZ](https://www.kaggle.com/hideyukizushi)","metadata":{}},{"cell_type":"markdown","source":"### ℹ️ **Info**\n* **forked original great work kernels**\n\n    * https://www.kaggle.com/code/ambrosm/adc24-intro-inference\n    * https://www.kaggle.com/code/hugowjd/neurips-ariel-2024-starter\n    * https://www.kaggle.com/code/xiaocao123/neurips-ariel-2024-starter?scriptVersionId=193156800\n    * https://www.kaggle.com/code/bingyuniu/neurips-ariel-2024-starter-withdifferentparametr\n\n* **2024/08/26 My Additional**\n    * percentile FE add.\n```\nnp.percentile(window, 5, axis=1),\nnp.percentile(window, 10, axis=1),\nnp.percentile(window, 15, axis=1),\nnp.percentile(window, 20, axis=1),\nnp.percentile(window, 25, axis=1),\nnp.percentile(window, 30, axis=1),\nnp.percentile(window, 35, axis=1),\nnp.percentile(window, 40, axis=1),\nnp.percentile(window, 60, axis=1),\nnp.percentile(window, 65, axis=1),\nnp.percentile(window, 70, axis=1),\nnp.percentile(window, 75, axis=1),\nnp.percentile(window, 80, axis=1),\nnp.percentile(window, 85, axis=1),\nnp.percentile(window, 90, axis=1),\nnp.percentile(window, 95, axis=1),\nnp.median(window, axis=1),\nnp.var(window, axis=1),\n```\n\n* **2024/08/29 My Additional**\n    * add FE\n```\nf_sliding_features2 = sliding_window_features(f_raw, 100, 10)\na_sliding_features2 = sliding_window_features(a_raw, 100, 10)\n```","metadata":{}},{"cell_type":"code","source":"# import matplotlib.pyplot as plt\n# import seaborn as sns\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# import torch\n# # import torch.nn as nn\n# # import torch.optim as optim\n# # from torch.utils.data import DataLoader, Dataset\n# from tqdm import tqdm\n# import pickle\n# import time\n# import os\n# import pickle\n# import seaborn as sns\n# import scipy.stats\n# from sklearn.model_selection import cross_val_predict\n# from sklearn.linear_model import Ridge\n# from sklearn.metrics import r2_score, mean_squared_error\n# from scipy.stats import kurtosis\n# from scipy.stats import skew","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Load Meta-Data\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# train_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_adc_info.csv', \n#                              index_col='planet_id')\n# train_labels = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_labels.csv',\n#                            index_col='planet_id')\n# wavelengths = pd.read_csv(f'{PATH}/wavelengths.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pre-Processing\n#### Load Functions","metadata":{}},{"cell_type":"code","source":"# %%writefile utils.py\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# from tqdm import tqdm\n# import pickle\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# def load_signal_data(planet_id, dataset, instrument, img_size):\n#     file_path = f'{PATH}/{dataset}/{planet_id}/{instrument}_signal.parquet'\n#     signal = pl.read_parquet(file_path)\n#     mean_signal = signal.cast(pl.Int32).sum_horizontal().cast(pl.Float32).to_numpy() / img_size # mean over the 32*32 pixels\n#     net_signal = mean_signal[1::2] - mean_signal[0::2]\n#     return net_signal\n\n# def read_and_preprocess(dataset, planet_ids, instrument = \"AIRS-CH0\"):\n#     \"\"\"Read the files for all planet_ids and extract the time series.\n#     Parameters\n#     dataset: 'train' or 'test'\n#     planet_ids: list of planet ids\n#     instrument: the instrument of observation, 'AIRS-CH0' or 'FGS1', default to 'AIRS-CH0'\n#     Returns\n#     dataframe with one row per planet_id and 67500 values per row for FGS1 and 5624 for AIRS-CH0\n#     \"\"\"\n#     img_size = 1024 if instrument == \"FGS1\" else 32*356\n#     column_num = 67500 if instrument == 'FGS1' else 5625\n#     raw_train = np.full((len(planet_ids), column_num), np.nan, dtype=np.float32)\n#     for i, planet_id in tqdm(list(enumerate(planet_ids))):\n#         raw_train[i] = load_signal_data(planet_id, dataset, instrument, img_size)\n#     return raw_train\n\n# def feature_engineering(f_raw, a_raw, adc_info, window_size=60, step_size=15):\n#     \"\"\"Create a dataframe with combined features from the raw data, including sliding window and time-series statistics.\n    \n#     Parameters:\n#     f_raw: ndarray of shape (n_planets, 67500)\n#     a_raw: ndarray of shape (n_planets, 5625)\n#     window_size: int, size of the sliding window for time-series statistics\n#     step_size: int, step size for the sliding window\n    \n#     Return value:\n#     df: DataFrame of shape (n_planets, several features)\n#     \"\"\"\n#     f_obscured = f_raw[:, 23500:44000].mean(axis=1)\n#     f_unobscured = (f_raw[:, :20500].mean(axis=1) + f_raw[:, 47000:].mean(axis=1)) / 2\n#     f_relative_reduction = (f_unobscured - f_obscured) / f_unobscured\n#     f_std_dev = f_raw.std(axis=1)\n#     f_signal_to_noise = f_unobscured / f_std_dev\n\n#     a_obscured = a_raw[:, 1958:3666].mean(axis=1)\n#     a_unobscured = (a_raw[:, :1708].mean(axis=1) + a_raw[:, 3916:].mean(axis=1)) / 2\n#     a_relative_reduction = (a_unobscured - a_obscured) / a_unobscured\n#     a_std_dev = a_raw.std(axis=1)\n#     a_signal_to_noise = a_unobscured / a_std_dev\n\n#     f_variance = f_raw.var(axis=1)\n#     a_variance = a_raw.var(axis=1)\n    \n#     f_skewness = pd.DataFrame(f_raw).skew(axis=1).values\n#     a_skewness = pd.DataFrame(a_raw).skew(axis=1).values\n\n#     f_kurtosis = pd.DataFrame(f_raw).kurtosis(axis=1).values\n#     a_kurtosis = pd.DataFrame(a_raw).kurtosis(axis=1).values\n    \n#     f_half_obscured1 = f_raw[:, 20500:23500].mean(axis=1)\n#     f_half_obscured2 = f_raw[:, 44000:47000].mean(axis=1)\n#     f_half_reduction1 = (f_unobscured - f_half_obscured1) / f_unobscured\n#     f_half_reduction2 = (f_unobscured - f_half_obscured2) / f_unobscured\n\n#     a_half_obscured1 = a_raw[:, 1708:1958].mean(axis=1)\n#     a_half_obscured2 = a_raw[:, 3666:3916].mean(axis=1)\n#     a_half_reduction1 = (a_unobscured - a_half_obscured1) / a_unobscured\n#     a_half_reduction2 = (a_unobscured - a_half_obscured2) / a_unobscured\n\n#     # Sliding window features\n#     def sliding_window_features(data, window_size, step_size):\n#         features = []\n#         max_index = data.shape[1]\n#         for start in range(0, max_index - window_size + 1, step_size):\n#             end = start + window_size\n#             window = data[:, start:end]\n#             features.append([\n#                 np.mean(window, axis=1),\n#                 np.std(window, axis=1),\n#                 np.min(window, axis=1),\n#                 np.max(window, axis=1),\n#                 np.percentile(window, 5, axis=1),\n#                 np.percentile(window, 10, axis=1),\n#                 np.percentile(window, 15, axis=1),\n#                 np.percentile(window, 20, axis=1),\n#                 np.percentile(window, 25, axis=1),\n#                 np.percentile(window, 30, axis=1),\n#                 np.percentile(window, 35, axis=1),\n#                 np.percentile(window, 40, axis=1),\n#                 np.percentile(window, 60, axis=1),\n#                 np.percentile(window, 65, axis=1),\n#                 np.percentile(window, 70, axis=1),\n#                 np.percentile(window, 75, axis=1),\n#                 np.percentile(window, 80, axis=1),\n#                 np.percentile(window, 85, axis=1),\n#                 np.percentile(window, 90, axis=1),\n#                 np.percentile(window, 95, axis=1),\n#                 np.median(window, axis=1),\n#                 np.var(window, axis=1),\n# #                 kurtosis(window, axis=1),\n# #                 skew(window, axis=1),\n#             ])\n#         if features:\n#             return np.vstack(features).T  # Stack vertically and transpose to get the correct shape\n#         else:\n#             return np.empty((data.shape[0], 0))  # Return empty array with correct shape\n    \n#     f_sliding_features = sliding_window_features(f_raw, window_size, step_size)\n#     a_sliding_features = sliding_window_features(a_raw, window_size, step_size)\n    \n#     f_sliding_features2 = sliding_window_features(f_raw, 100, 10)\n#     a_sliding_features2 = sliding_window_features(a_raw, 100, 10)\n\n\n#     print(f'f_sliding_features.shape: {f_sliding_features.shape}')\n#     print(f'a_sliding_features.shape: {a_sliding_features.shape}')\n\n\n#     df = pd.DataFrame({\n#         'f_relative_reduction': f_relative_reduction,\n#         'f_signal_to_noise': f_signal_to_noise,\n#         'f_variance': f_variance,\n#         'f_skewness': f_skewness,\n#         'f_kurtosis': f_kurtosis,\n#         'a_relative_reduction': a_relative_reduction,\n#         'a_signal_to_noise': a_signal_to_noise,\n#         'a_variance': a_variance,\n#         'a_skewness': a_skewness,\n#         'a_kurtosis': a_kurtosis,\n#         'f_half_reduction1': f_half_reduction1,\n#         'f_half_reduction2': f_half_reduction2,\n#         'a_half_reduction1': a_half_reduction1,\n#         'a_half_reduction2': a_half_reduction2\n#     })\n\n\n#     if f_sliding_features.size > 0:\n#         f_sliding_df = pd.DataFrame(f_sliding_features, columns=[f'f_slide_{i}' for i in range(f_sliding_features.shape[1])])\n#         df = pd.concat([df, f_sliding_df], axis=1)\n#     if a_sliding_features.size > 0:\n#         a_sliding_df = pd.DataFrame(a_sliding_features, columns=[f'a_slide_{i}' for i in range(a_sliding_features.shape[1])])\n#         df = pd.concat([df, a_sliding_df], axis=1)\n    \n#     if f_sliding_features2.size > 0:\n#         f_sliding_df = pd.DataFrame(f_sliding_features2, columns=[f'f_slide2_{i}' for i in range(f_sliding_features2.shape[1])])\n#         df = pd.concat([df, f_sliding_df], axis=1)\n#     if a_sliding_features2.size > 0:\n#         a_sliding_df = pd.DataFrame(a_sliding_features2, columns=[f'a_slide2_{i}' for i in range(a_sliding_features2.shape[1])])\n#         df = pd.concat([df, a_sliding_df], axis=1)\n    \n    \n#     df = pd.concat([df, adc_info.reset_index().iloc[:, 1:6]], axis=1)\n    \n#     return df","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%writefile -a utils.py\n\n# def postprocessing(pred_array, index, sigma_pred):\n#     \"\"\"Create a submission dataframe from its components\n    \n#     Parameters:\n#     pred_array: ndarray of shape (n_samples, 283)\n#     index: pandas.Index of length n_samples with name 'planet_id'\n#     sigma_pred: float\n    \n#     Return value:\n#     df: DataFrame of shape (n_samples, 566) with planet_id as index\n#     \"\"\"\n#     return pd.concat([pd.DataFrame(pred_array.clip(0, None), index=index, columns=wavelengths.columns),\n#                       pd.DataFrame(sigma_pred, index=index, columns=[f\"sigma_{i}\" for i in range(1, 284)])],\n#                      axis=1)\n\n# class ParticipantVisibleError(Exception):\n#     pass\n\n# def competition_score(\n#         solution: pd.DataFrame,\n#         submission: pd.DataFrame,\n#         naive_mean: float,\n#         naive_sigma: float,\n#         sigma_true: float,\n#         row_id_column_name='planet_id',\n#     ) -> float:\n#     '''\n#     This is a Gaussian Log Likelihood based metric. For a submission, which contains the predicted mean (x_hat) and variance (x_hat_std),\n#     we calculate the Gaussian Log-likelihood (GLL) value to the provided ground truth (x). We treat each pair of x_hat,\n#     x_hat_std as a 1D gaussian, meaning there will be 283 1D gaussian distributions, hence 283 values for each test spectrum,\n#     the GLL value for one spectrum is the sum of all of them.\n\n#     Inputs:\n#         - solution: Ground Truth spectra (from test set)\n#             - shape: (nsamples, n_wavelengths)\n#         - submission: Predicted spectra and errors (from participants)\n#             - shape: (nsamples, n_wavelengths*2)\n#         naive_mean: (float) mean from the train set.\n#         naive_sigma: (float) standard deviation from the train set.\n#         sigma_true: (float) essentially sets the scale of the outputs.\n#     '''\n\n#     del solution[row_id_column_name]\n#     del submission[row_id_column_name]\n\n#     if submission.min().min() < 0:\n#         raise ParticipantVisibleError('Negative values in the submission')\n#     for col in submission.columns:\n#         if not pd.api.types.is_numeric_dtype(submission[col]):\n#             raise ParticipantVisibleError(f'Submission column {col} must be a number')\n\n#     n_wavelengths = len(solution.columns)\n#     if len(submission.columns) != n_wavelengths*2:\n#         raise ParticipantVisibleError('Wrong number of columns in the submission')\n\n#     y_pred = submission.iloc[:, :n_wavelengths].values\n#     # Set a non-zero minimum sigma pred to prevent division by zero errors.\n#     sigma_pred = np.clip(submission.iloc[:, n_wavelengths:].values, a_min=10**-15, a_max=None)\n#     y_true = solution.values\n\n#     GLL_pred = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_pred, scale=sigma_pred))\n#     GLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))\n#     GLL_mean = np.sum(scipy.stats.norm.logpdf(y_true, loc=naive_mean * np.ones_like(y_true), scale=naive_sigma * np.ones_like(y_true)))\n\n#     submit_score = (GLL_pred - GLL_mean)/(GLL_true - GLL_mean)\n#     return float(np.clip(submit_score, 0.0, 1.0))","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# exec(open('utils.py', 'r').read())","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Data","metadata":{}},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/f_raw_train.pickle\"):\n#     f_raw_train = np.load('/kaggle/input/adc24-intro-training/f_raw_train.pickle', allow_pickle=True)\n# else:\n#     f_raw_train = read_and_preprocess('train', train_labels.index, 'FGS1')\n#     with open('f_raw_train.pickle', 'wb') as f:\n#         pickle.dump(f_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/a_raw_train.pickle\"):\n#     a_raw_train = np.load('/kaggle/input/adc24-intro-training/a_raw_train.pickle', allow_pickle=True)\n# else:\n#     a_raw_train = read_and_preprocess('train', train_labels.index)\n#     with open('a_raw_train.pickle', 'wb') as f:\n#         pickle.dump(a_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering","metadata":{}},{"cell_type":"code","source":"# %%time\n# train = feature_engineering(f_raw_train, a_raw_train, train_adc_info)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train.head()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = train.iloc[:,:-1]","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Plot","metadata":{}},{"cell_type":"code","source":"# plt.figure(figsize=(6, 2))\n# plt.plot(f_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step, color='gray')\n# plt.xlabel('time step')\n# plt.title('FGS1: Overall mean')\n# plt.show()\n\n# plt.figure(figsize=(6, 2))\n# plt.plot(a_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step * 11250 // 135000, color='gray')\n# plt.xlabel('time step')\n# plt.title('AIRS-CH0: Overall mean')\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# color_array = np.array(plt.rcParams['axes.prop_cycle'].by_key()['color'])\n# plt.scatter(train.a_relative_reduction, train_labels.wl_1, s=15, alpha=0.5,\n#             c=color_array[train_adc_info.star])\n# plt.xlabel('relative signal reduction when planet is in front')\n# plt.ylabel('target')\n# plt.title('Correlation between relative signal reduction and target')\n# # plt.gca().set_aspect('equal')\n# points = [plt.Line2D([0], [0], label=f'star {i}', marker='o', markersize=3,\n#          markeredgecolor=color_array[i], markerfacecolor=color_array[i], linestyle='') for i in range(2)]\n\n# plt.legend(handles=points)\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model\n#### Rigde Model","metadata":{}},{"cell_type":"code","source":"# model = Ridge(alpha=1e-12)\n\n# oof_pred = cross_val_predict(model, train, train_labels)\n\n# print(f\"# R2 score: {r2_score(train_labels, oof_pred):.4f}\")\n# sigma_pred = mean_squared_error(train_labels, oof_pred, squared=False)\n# print(f\"# Root mean squared error: {sigma_pred:.7f}\")","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# oof_df = postprocessing(oof_pred, train_adc_info.index, sigma_pred)\n# display(oof_df)\n\n# gll_score = competition_score(train_labels.copy().reset_index(),\n#                               oof_df.copy().reset_index(),\n#                               naive_mean=train_labels.values.mean(),\n#                               naive_sigma=train_labels.values.std(),\n#                               sigma_true=0.000003)\n# print(f\"# Estimated competition score: {gll_score:.4f}\")","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.fit(train, train_labels)\n# with open('model.pickle', 'wb') as f:\n#     pickle.dump(model, f)\n# with open('sigma_pred.pickle', 'wb') as f:\n#     pickle.dump(sigma_pred, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference","metadata":{}},{"cell_type":"code","source":"# # Load the data\n# test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n#                            index_col='planet_id')\n# sample_submission = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv',\n#                                 index_col='planet_id')\n# f_raw_test = read_and_preprocess('test', sample_submission.index, 'FGS1')\n# a_raw_test = read_and_preprocess('test', sample_submission.index)\n# test = feature_engineering(f_raw_test, a_raw_test, test_adc_info)\n# test = test.iloc[: , :-1]\n# # Load the model\n# with open('model.pickle', 'rb') as f:\n#     model = pickle.load(f)\n# with open('sigma_pred.pickle', 'rb') as f:\n#     sigma_pred = pickle.load(f)\n\n# # Predict\n# test_pred = model.predict(test)\n\n# # Package into submission file\n# sub_df = sub_df = postprocessing(test_pred,\n#                         test_adc_info.index,\n#                         sigma_pred=np.tile(np.where(test_adc_info[['star']] <= 1, 0.0001555, 0.00085), (1, 283)))\n# display(sub_df)\n# sub_df.to_csv('submission_4.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [NeurIPS Ariel 2024 - Starter withdifferentparametr](https://www.kaggle.com/code/hideyukizushi/ad24-train-inf-ridge-addfe-lb-441)\n\n### [yu🐂](https://www.kaggle.com/bingyuniu)","metadata":{}},{"cell_type":"markdown","source":"### Initialization\n\nThis competition seems requires strong scientific background and I had lot of confusion during EDA process. Therefore, I just build a simple starter for future coding.\n\n#### Load Library","metadata":{}},{"cell_type":"code","source":"# import matplotlib.pyplot as plt\n# import seaborn as sns\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# import torch\n# # import torch.nn as nn\n# # import torch.optim as optim\n# # from torch.utils.data import DataLoader, Dataset\n# from tqdm import tqdm\n# import pickle\n# import time\n# import os\n# import pickle\n# import seaborn as sns\n# import scipy.stats\n# from sklearn.model_selection import cross_val_predict\n# from sklearn.linear_model import Ridge\n# from sklearn.metrics import r2_score, mean_squared_error","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Load Meta-Data\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# train_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_adc_info.csv', \n#                              index_col='planet_id')\n# train_labels = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/train_labels.csv',\n#                            index_col='planet_id')\n# wavelengths = pd.read_csv(f'{PATH}/wavelengths.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pre-Processing\n#### Load Functions","metadata":{}},{"cell_type":"code","source":"# %%writefile utils.py\n# import pandas as pd\n# import polars as pl\n# import numpy as np\n# from tqdm import tqdm\n# import pickle\n# PATH = \"/kaggle/input/ariel-data-challenge-2024\"\n# def load_signal_data(planet_id, dataset, instrument, img_size):\n#     file_path = f'{PATH}/{dataset}/{planet_id}/{instrument}_signal.parquet'\n#     signal = pl.read_parquet(file_path)\n#     mean_signal = signal.cast(pl.Int32).sum_horizontal().cast(pl.Float32).to_numpy() / img_size # mean over the 32*32 pixels\n#     net_signal = mean_signal[1::2] - mean_signal[0::2]\n#     return net_signal\n\n# def read_and_preprocess(dataset, planet_ids, instrument = \"AIRS-CH0\"):\n#     \"\"\"Read the files for all planet_ids and extract the time series.\n#     Parameters\n#     dataset: 'train' or 'test'\n#     planet_ids: list of planet ids\n#     instrument: the instrument of observation, 'AIRS-CH0' or 'FGS1', default to 'AIRS-CH0'\n#     Returns\n#     dataframe with one row per planet_id and 67500 values per row for FGS1 and 5624 for AIRS-CH0\n#     \"\"\"\n#     img_size = 1024 if instrument == \"FGS1\" else 32*356\n#     column_num = 67500 if instrument == 'FGS1' else 5625\n#     raw_train = np.full((len(planet_ids), column_num), np.nan, dtype=np.float32)\n#     for i, planet_id in tqdm(list(enumerate(planet_ids))):\n#         raw_train[i] = load_signal_data(planet_id, dataset, instrument, img_size)\n#     return raw_train\n\n# def feature_engineering(f_raw, a_raw, adc_info, window_size=50, step_size=15):\n#     \"\"\"Create a dataframe with combined features from the raw data, including sliding window and time-series statistics.\n    \n#     Parameters:\n#     f_raw: ndarray of shape (n_planets, 67500)\n#     a_raw: ndarray of shape (n_planets, 5625)\n#     window_size: int, size of the sliding window for time-series statistics\n#     step_size: int, step size for the sliding window\n    \n#     Return value:\n#     df: DataFrame of shape (n_planets, several features)\n#     \"\"\"\n#     f_obscured = f_raw[:, 23500:44000].mean(axis=1)\n#     f_unobscured = (f_raw[:, :20500].mean(axis=1) + f_raw[:, 47000:].mean(axis=1)) / 2\n#     f_relative_reduction = (f_unobscured - f_obscured) / f_unobscured\n#     f_std_dev = f_raw.std(axis=1)\n#     f_signal_to_noise = f_unobscured / f_std_dev\n\n#     a_obscured = a_raw[:, 1958:3666].mean(axis=1)\n#     a_unobscured = (a_raw[:, :1708].mean(axis=1) + a_raw[:, 3916:].mean(axis=1)) / 2\n#     a_relative_reduction = (a_unobscured - a_obscured) / a_unobscured\n#     a_std_dev = a_raw.std(axis=1)\n#     a_signal_to_noise = a_unobscured / a_std_dev\n\n#     f_variance = f_raw.var(axis=1)\n#     a_variance = a_raw.var(axis=1)\n    \n#     f_skewness = pd.DataFrame(f_raw).skew(axis=1).values\n#     a_skewness = pd.DataFrame(a_raw).skew(axis=1).values\n\n#     f_kurtosis = pd.DataFrame(f_raw).kurtosis(axis=1).values\n#     a_kurtosis = pd.DataFrame(a_raw).kurtosis(axis=1).values\n    \n#     f_half_obscured1 = f_raw[:, 20500:23500].mean(axis=1)\n#     f_half_obscured2 = f_raw[:, 44000:47000].mean(axis=1)\n#     f_half_reduction1 = (f_unobscured - f_half_obscured1) / f_unobscured\n#     f_half_reduction2 = (f_unobscured - f_half_obscured2) / f_unobscured\n\n#     a_half_obscured1 = a_raw[:, 1708:1958].mean(axis=1)\n#     a_half_obscured2 = a_raw[:, 3666:3916].mean(axis=1)\n#     a_half_reduction1 = (a_unobscured - a_half_obscured1) / a_unobscured\n#     a_half_reduction2 = (a_unobscured - a_half_obscured2) / a_unobscured\n\n#     # Sliding window features\n#     def sliding_window_features(data, window_size, step_size):\n#         features = []\n#         max_index = data.shape[1]\n#         for start in range(0, max_index - window_size + 1, step_size):\n#             end = start + window_size\n#             window = data[:, start:end]\n#             features.append([\n#                 np.mean(window, axis=1),\n#                 np.std(window, axis=1),\n#                 np.min(window, axis=1),\n#                 np.max(window, axis=1)\n#             ])\n#         if features:\n#             return np.vstack(features).T  # Stack vertically and transpose to get the correct shape\n#         else:\n#             return np.empty((data.shape[0], 0))  # Return empty array with correct shape\n    \n#     f_sliding_features = sliding_window_features(f_raw, window_size, step_size)\n#     a_sliding_features = sliding_window_features(a_raw, window_size, step_size)\n\n\n#     print(f'f_sliding_features.shape: {f_sliding_features.shape}')\n#     print(f'a_sliding_features.shape: {a_sliding_features.shape}')\n\n\n#     df = pd.DataFrame({\n#         'f_relative_reduction': f_relative_reduction,\n#         'f_signal_to_noise': f_signal_to_noise,\n#         'f_variance': f_variance,\n#         'f_skewness': f_skewness,\n#         'f_kurtosis': f_kurtosis,\n#         'a_relative_reduction': a_relative_reduction,\n#         'a_signal_to_noise': a_signal_to_noise,\n#         'a_variance': a_variance,\n#         'a_skewness': a_skewness,\n#         'a_kurtosis': a_kurtosis,\n#         'f_half_reduction1': f_half_reduction1,\n#         'f_half_reduction2': f_half_reduction2,\n#         'a_half_reduction1': a_half_reduction1,\n#         'a_half_reduction2': a_half_reduction2\n#     })\n\n\n#     if f_sliding_features.size > 0:\n#         f_sliding_df = pd.DataFrame(f_sliding_features, columns=[f'f_slide_{i}' for i in range(f_sliding_features.shape[1])])\n#         df = pd.concat([df, f_sliding_df], axis=1)\n\n#     if a_sliding_features.size > 0:\n#         a_sliding_df = pd.DataFrame(a_sliding_features, columns=[f'a_slide_{i}' for i in range(a_sliding_features.shape[1])])\n#         df = pd.concat([df, a_sliding_df], axis=1)\n    \n#     df = pd.concat([df, adc_info.reset_index().iloc[:, 1:6]], axis=1)\n    \n#     return df","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%writefile -a utils.py\n\n# def postprocessing(pred_array, index, sigma_pred):\n#     \"\"\"Create a submission dataframe from its components\n    \n#     Parameters:\n#     pred_array: ndarray of shape (n_samples, 283)\n#     index: pandas.Index of length n_samples with name 'planet_id'\n#     sigma_pred: float\n    \n#     Return value:\n#     df: DataFrame of shape (n_samples, 566) with planet_id as index\n#     \"\"\"\n#     return pd.concat([pd.DataFrame(pred_array.clip(0, None), index=index, columns=wavelengths.columns),\n#                       pd.DataFrame(sigma_pred, index=index, columns=[f\"sigma_{i}\" for i in range(1, 284)])],\n#                      axis=1)\n\n# class ParticipantVisibleError(Exception):\n#     pass\n\n# def competition_score(\n#         solution: pd.DataFrame,\n#         submission: pd.DataFrame,\n#         naive_mean: float,\n#         naive_sigma: float,\n#         sigma_true: float,\n#         row_id_column_name='planet_id',\n#     ) -> float:\n#     '''\n#     This is a Gaussian Log Likelihood based metric. For a submission, which contains the predicted mean (x_hat) and variance (x_hat_std),\n#     we calculate the Gaussian Log-likelihood (GLL) value to the provided ground truth (x). We treat each pair of x_hat,\n#     x_hat_std as a 1D gaussian, meaning there will be 283 1D gaussian distributions, hence 283 values for each test spectrum,\n#     the GLL value for one spectrum is the sum of all of them.\n\n#     Inputs:\n#         - solution: Ground Truth spectra (from test set)\n#             - shape: (nsamples, n_wavelengths)\n#         - submission: Predicted spectra and errors (from participants)\n#             - shape: (nsamples, n_wavelengths*2)\n#         naive_mean: (float) mean from the train set.\n#         naive_sigma: (float) standard deviation from the train set.\n#         sigma_true: (float) essentially sets the scale of the outputs.\n#     '''\n\n#     del solution[row_id_column_name]\n#     del submission[row_id_column_name]\n\n#     if submission.min().min() < 0:\n#         raise ParticipantVisibleError('Negative values in the submission')\n#     for col in submission.columns:\n#         if not pd.api.types.is_numeric_dtype(submission[col]):\n#             raise ParticipantVisibleError(f'Submission column {col} must be a number')\n\n#     n_wavelengths = len(solution.columns)\n#     if len(submission.columns) != n_wavelengths*2:\n#         raise ParticipantVisibleError('Wrong number of columns in the submission')\n\n#     y_pred = submission.iloc[:, :n_wavelengths].values\n#     # Set a non-zero minimum sigma pred to prevent division by zero errors.\n#     sigma_pred = np.clip(submission.iloc[:, n_wavelengths:].values, a_min=10**-15, a_max=None)\n#     y_true = solution.values\n\n#     GLL_pred = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_pred, scale=sigma_pred))\n#     GLL_true = np.sum(scipy.stats.norm.logpdf(y_true, loc=y_true, scale=sigma_true * np.ones_like(y_true)))\n#     GLL_mean = np.sum(scipy.stats.norm.logpdf(y_true, loc=naive_mean * np.ones_like(y_true), scale=naive_sigma * np.ones_like(y_true)))\n\n#     submit_score = (GLL_pred - GLL_mean)/(GLL_true - GLL_mean)\n#     return float(np.clip(submit_score, 0.0, 1.0))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-10T00:49:04.388773Z","iopub.execute_input":"2024-09-10T00:49:04.389548Z","iopub.status.idle":"2024-09-10T00:49:04.423061Z","shell.execute_reply.started":"2024-09-10T00:49:04.389498Z","shell.execute_reply":"2024-09-10T00:49:04.42168Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# exec(open('utils.py', 'r').read())","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Data","metadata":{}},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/f_raw_train.pickle\"):\n#     f_raw_train = np.load('/kaggle/input/adc24-intro-training/f_raw_train.pickle', allow_pickle=True)\n# else:\n#     f_raw_train = read_and_preprocess('train', train_labels.index, 'FGS1')\n#     with open('f_raw_train.pickle', 'wb') as f:\n#         pickle.dump(f_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# if os.path.exists(\"/kaggle/input/adc24-intro-training/a_raw_train.pickle\"):\n#     a_raw_train = np.load('/kaggle/input/adc24-intro-training/a_raw_train.pickle', allow_pickle=True)\n# else:\n#     a_raw_train = read_and_preprocess('train', train_labels.index)\n#     with open('a_raw_train.pickle', 'wb') as f:\n#         pickle.dump(a_raw_train, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering","metadata":{}},{"cell_type":"code","source":"# %%time\n# train = feature_engineering(f_raw_train, a_raw_train, train_adc_info)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train.head()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = train.iloc[:,:-1]","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Plot","metadata":{}},{"cell_type":"code","source":"# plt.figure(figsize=(6, 2))\n# plt.plot(f_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step, color='gray')\n# plt.xlabel('time step')\n# plt.title('FGS1: Overall mean')\n# plt.show()\n\n# plt.figure(figsize=(6, 2))\n# plt.plot(a_raw_train.mean(axis=0))\n# for time_step in [20500, 23500, 44000, 47000]:\n#     plt.axvline(time_step * 11250 // 135000, color='gray')\n# plt.xlabel('time step')\n# plt.title('AIRS-CH0: Overall mean')\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# color_array = np.array(plt.rcParams['axes.prop_cycle'].by_key()['color'])\n# plt.scatter(train.a_relative_reduction, train_labels.wl_1, s=15, alpha=0.5,\n#             c=color_array[train_adc_info.star])\n# plt.xlabel('relative signal reduction when planet is in front')\n# plt.ylabel('target')\n# plt.title('Correlation between relative signal reduction and target')\n# # plt.gca().set_aspect('equal')\n# points = [plt.Line2D([0], [0], label=f'star {i}', marker='o', markersize=3,\n#          markeredgecolor=color_array[i], markerfacecolor=color_array[i], linestyle='') for i in range(2)]\n\n# plt.legend(handles=points)\n# plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model\n#### Rigde Model","metadata":{}},{"cell_type":"code","source":"# model = Ridge(alpha=1e-12)\n\n# oof_pred = cross_val_predict(model, train, train_labels)\n\n# print(f\"# R2 score: {r2_score(train_labels, oof_pred):.4f}\")\n# sigma_pred = mean_squared_error(train_labels, oof_pred, squared=False)\n# print(f\"# Root mean squared error: {sigma_pred:.7f}\")","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# oof_df = postprocessing(oof_pred, train_adc_info.index, sigma_pred)\n# display(oof_df)\n\n# gll_score = competition_score(train_labels.copy().reset_index(),\n#                               oof_df.copy().reset_index(),\n#                               naive_mean=train_labels.values.mean(),\n#                               naive_sigma=train_labels.values.std(),\n#                               sigma_true=0.000003)\n# print(f\"# Estimated competition score: {gll_score:.4f}\")","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.fit(train, train_labels)\n# with open('model.pickle', 'wb') as f:\n#     pickle.dump(model, f)\n# with open('sigma_pred.pickle', 'wb') as f:\n#     pickle.dump(sigma_pred, f)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference","metadata":{}},{"cell_type":"code","source":"# # Load the data\n# test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n#                            index_col='planet_id')\n# sample_submission = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv',\n#                                 index_col='planet_id')\n# f_raw_test = read_and_preprocess('test', sample_submission.index, 'FGS1')\n# a_raw_test = read_and_preprocess('test', sample_submission.index)\n# test = feature_engineering(f_raw_test, a_raw_test, test_adc_info)\n# test = test.iloc[: , :-1]\n# # Load the model\n# with open('model.pickle', 'rb') as f:\n#     model = pickle.load(f)\n# with open('sigma_pred.pickle', 'rb') as f:\n#     sigma_pred = pickle.load(f)\n\n# # Predict\n# test_pred = model.predict(test)\n\n# # Package into submission file\n# sub_df = sub_df = postprocessing(test_pred,\n#                         test_adc_info.index,\n#                         sigma_pred=np.tile(np.where(test_adc_info[['star']] <= 1, 0.0001555, 0.00085), (1, 283)))\n# display(sub_df)\n# sub_df.to_csv('submission_5.csv')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [ariel_only_correlation | param upd[LB.517]](https://www.kaggle.com/code/hideyukizushi/ariel-only-correlation-param-upd-lb-517)\n### [yukiZ](https://www.kaggle.com/hideyukizushi)","metadata":{}},{"cell_type":"markdown","source":"### ℹ️ **Info**\n* **forked original great work kernels**\n    * https://www.kaggle.com/code/sergeifironov/ariel-only-correlation\n\n* **2024/09/08 My Changed**\n    * scipy minimize() param & other params update","metadata":{}},{"cell_type":"markdown","source":"---\n---","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nimport scipy.stats\nfrom tqdm import tqdm\n\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.linear_model import Ridge\nfrom sklearn.metrics import r2_score, mean_squared_error\nimport itertools\nfrom scipy.optimize import minimize\nfrom functools import partial\nimport random, os\nfrom astropy.stats import sigma_clip","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_adc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/test_adc_info.csv',\n                           index_col='planet_id')\naxis_info = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/axis_info.parquet')","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def apply_linear_corr(linear_corr,clean_signal):\n    linear_corr = np.flip(linear_corr, axis=0)\n    for x, y in itertools.product(\n                range(clean_signal.shape[1]), range(clean_signal.shape[2])\n            ):\n        poli = np.poly1d(linear_corr[:, x, y])\n        clean_signal[:, x, y] = poli(clean_signal[:, x, y])\n    return clean_signal\n\ndef clean_dark(signal, dark, dt):\n    dark = np.tile(dark, (signal.shape[0], 1, 1))\n    signal -= dark* dt[:, np.newaxis, np.newaxis]\n    return signal\n\ndef preproc(dataset, adc_info, sensor, binning = 15):\n    cut_inf, cut_sup = 39, 321\n    sensor_sizes_dict = {\"AIRS-CH0\":[[11250, 32, 356], [1, 32, cut_sup-cut_inf]], \"FGS1\":[[135000, 32, 32], [1, 32, 32]]}\n    binned_dict = {\"AIRS-CH0\":[11250 // binning // 2, 282], \"FGS1\":[135000 // binning // 2]}\n    linear_corr_dict = {\"AIRS-CH0\":(6, 32, 356), \"FGS1\":(6, 32, 32)}\n    planet_ids = adc_info.index\n    \n    feats = []\n    for i, planet_id in tqdm(list(enumerate(planet_ids))):\n        signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/{sensor}_signal.parquet').to_numpy()\n        dark_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dark.parquet', engine='pyarrow').to_numpy()\n        dead_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dead.parquet', engine='pyarrow').to_numpy()\n        flat_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/flat.parquet', engine='pyarrow').to_numpy()\n        linear_corr = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/linear_corr.parquet').values.astype(np.float64).reshape(linear_corr_dict[sensor])\n\n        signal = signal.reshape(sensor_sizes_dict[sensor][0]) \n        gain = adc_info[f'{sensor}_adc_gain'].values[i]\n        offset = adc_info[f'{sensor}_adc_offset'].values[i]\n        signal = signal / gain + offset\n        \n        hot = sigma_clip(\n            dark_frame, sigma=8, maxiters=5\n        ).mask\n        \n        if sensor != \"FGS1\":\n            signal = signal[:, :, cut_inf:cut_sup] #11250 * 32 * 282\n            #dt = axis_info['AIRS-CH0-integration_time'].dropna().values\n            dt = np.ones(len(signal))*0.1 \n            dt[1::2] += 4.5 #@bilzard idea\n            linear_corr = linear_corr[:, :, cut_inf:cut_sup]\n            dark_frame = dark_frame[:, cut_inf:cut_sup]\n            dead_frame = dead_frame[:, cut_inf:cut_sup]\n            flat_frame = flat_frame[:, cut_inf:cut_sup]\n            hot = hot[:, cut_inf:cut_sup]\n        else:\n            dt = np.ones(len(signal))*0.1\n            dt[1::2] += 0.1\n            \n        signal = signal.clip(0) #@graySnow idea\n        linear_corr_signal = apply_linear_corr(linear_corr, signal)\n        signal = clean_dark(linear_corr_signal, dark_frame, dt)\n        \n        flat = flat_frame.reshape(sensor_sizes_dict[sensor][1])\n        flat[dead_frame.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n        flat[hot.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n        signal = signal / flat\n        \n        if sensor == \"FGS1\":\n            signal = signal.reshape((sensor_sizes_dict[sensor][0][0], sensor_sizes_dict[sensor][0][1]*sensor_sizes_dict[sensor][0][2]))\n        \n        mean_signal = np.nanmean(signal, axis=1) # mean over the 32*32(FGS1) or 32(CH0) pixels\n        cds_signal = (mean_signal[1::2] - mean_signal[0::2])\n        \n        binned = np.zeros((binned_dict[sensor]))\n        for j in range(cds_signal.shape[0] // binning):\n            binned[j] = cds_signal[j*binning:j*binning+binning].mean(axis=0)\n                   \n        if sensor == \"FGS1\":\n            binned = binned.reshape((binned.shape[0],1))\n            \n        feats.append(binned)\n        \n    return np.stack(feats)\n    \npre_train = np.concatenate([preproc('test', test_adc_info, \"FGS1\", 30*12), preproc('test', test_adc_info, \"AIRS-CH0\", 30)], axis=2)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### fit polynoms for each sample","metadata":{}},{"cell_type":"code","source":"def phase_detector(signal):\n    phase1, phase2 = None, None\n    best_drop = 0\n    for i in range(50//2,150//2):        \n        t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n        if t1 > best_drop:\n            phase1 = i+(20+5)//2\n            best_drop = t1\n    \n    best_drop = 0\n    for i in range(200//2,250//2):\n        t1 = signal[i:i+20//2].max() - signal[i:i+20//2].min()\n        if t1 > best_drop:\n            phase2 = i-5//2\n            best_drop = t1\n    \n    return phase1, phase2\n\ndef try_s(signal, p1, p2, deg, s):\n    out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n    x, y = out, signal[out].tolist()\n    x = x + list(range(p1,p2))\n\n    y = y + (signal[p1:p2] * (1 + s[0])).tolist()\n    z = np.polyfit(x, y, deg)\n    p = np.poly1d(z)\n    q = np.abs(p(x) - y).mean()\n\n    if s < 1e-4:\n        return q + 1e3\n\n    return q\n    \ndef calibrate_signal(signal):\n    p1,p2 = phase_detector(signal)\n\n    best_deg, best_score = 1, 1e12\n    for deg in range(1, 6):\n        f = partial(try_s, signal, p1, p2, deg)\n        r = minimize(f, [0.001], method = 'Nelder-Mead')\n        s = r.x[0]\n\n        out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n        x, y = out, signal[out].tolist()\n        x = x + list(range(p1,p2))\n        y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n        z = np.polyfit(x, y, deg)\n        p = np.poly1d(z)\n        q = np.abs(p(x) - y).mean()\n        \n        if q < best_score:\n            best_score = q\n            best_deg = deg\n        \n        print(deg, q)\n            \n    z = np.polyfit(x, y, best_deg)\n    p = np.poly1d(z)\n\n    return s, x, y, p(x)\n\ndef calibrate_train(signal):\n    p1,p2 = phase_detector(signal)\n    \n    best_deg, best_score = 1, 1e12\n    for deg in range(1, 6):\n        f = partial(try_s, signal, p1, p2, deg)\n        r = minimize(f, [0.0001], method = 'Nelder-Mead')\n        s = r.x[0]\n\n        out = list(range(p1-30)) + list(range(p2+30,signal.shape[0]))\n        x, y = out, signal[out].tolist()\n        x = x + list(range(p1,p2))\n        y = y + (signal[p1:p2] * (1 + s)).tolist()\n    \n        z = np.polyfit(x, y, deg)\n        p = np.poly1d(z)\n        q = np.abs(p(x) - y).mean()\n        \n        if q < best_score:\n            best_score = q\n            best_deg = deg\n            \n    z = np.polyfit(x, y, best_deg)\n    p = np.poly1d(z)\n    \n    return s, p(np.arange(signal.shape[0])), p1, p2\n\n\ntrain = pre_train.copy()\nall_s = []\nfor i in range(len(test_adc_info)):\n    signal = train[i,:,1:].mean(axis=1)\n    s, p, p1, p2 = calibrate_train(pre_train[i,:,1:].mean(axis=1))\n    all_s.append(s)\n        \n#copy answer 283 times because we predict mean value\ntrain_s = np.repeat(np.array(all_s), 283).reshape((len(all_s), 283))        \ntrain_sigma = np.ones_like(train_s) * 0.000176","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probably we can accurately estimate sigma from train","metadata":{}},{"cell_type":"code","source":"n = 0\ns, x, y, y_new = calibrate_signal(pre_train[n,:,1:].mean(axis=1))\nplt.scatter(x,y)\nplt.scatter(x,y_new)","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I call the orange line \"starline\". This is probably what we would see if the planet weren't in the way.","metadata":{}},{"cell_type":"markdown","source":"### Making submission","metadata":{}},{"cell_type":"code","source":"ss = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv')\n\npreds = train_s.clip(0)\nsigmas = train_sigma\nsubmission = pd.DataFrame(np.concatenate([preds,sigmas], axis=1), columns=ss.columns[1:])\nsubmission.index = test_adc_info.index\nsubmission.to_csv('submission_7.csv')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [Ariel Data Challenge 2024](https://www.kaggle.com/code/xiaocao123/ariel-data-challenge-2024)\n### [qianc](https://www.kaggle.com/xiaocao123)","metadata":{}},{"cell_type":"markdown","source":"This notebook is an update of https://www.kaggle.com/code/sergeifironov/ariel-only-correlation\nfrom Sergei Fironov\n\nUpdates :\n- keep 10:22 pixels from the 32 (the image are well centred)\n- Use the derivative for the determination of the beginning and end of the signal during eclipse (idea from Reza R. Choubeh)\n- 'Simplification' of the code for minimize\n- Degree of polyfit <= 4\n- Predictions of test after training Ridge regression with the modelization results (targets predictions with modelization) and the True targets. ","metadata":{}},{"cell_type":"markdown","source":"### Librairies","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom tqdm import tqdm\nimport joblib\n\nfrom sklearn.linear_model import Ridge\nfrom sklearn.metrics import r2_score, mean_squared_error\nimport itertools\n\nfrom scipy.optimize import minimize\nfrom scipy import optimize\n\nfrom astropy.stats import sigma_clip","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = 'test'\nadc_info = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/'+f'{dataset}_adc_info.csv',index_col='planet_id')\naxis_info = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/axis_info.parquet')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Calibration","metadata":{}},{"cell_type":"code","source":"def apply_linear_corr(linear_corr,clean_signal):\n    linear_corr = np.flip(linear_corr, axis=0)\n    for x, y in itertools.product(\n                range(clean_signal.shape[1]), range(clean_signal.shape[2])\n            ):\n        poli = np.poly1d(linear_corr[:, x, y])\n        clean_signal[:, x, y] = poli(clean_signal[:, x, y])\n    return clean_signal\n\ndef clean_dark(signal, dark, dt):\n    dark = np.tile(dark, (signal.shape[0], 1, 1))\n    signal -= dark* dt[:, np.newaxis, np.newaxis]\n    return signal\n\ndef preproc(dataset, adc_info, sensor, binning = 15):\n    cut_inf, cut_sup = 39, 321\n    sensor_sizes_dict = {\"AIRS-CH0\":[[11250, 32, 356], [1, 32, cut_sup-cut_inf]], \"FGS1\":[[135000, 32, 32], [1, 32, 32]]}\n    binned_dict = {\"AIRS-CH0\":[11250 // binning // 2, 282], \"FGS1\":[135000 // binning // 2]}\n    linear_corr_dict = {\"AIRS-CH0\":(6, 32, 356), \"FGS1\":(6, 32, 32)}\n    planet_ids = adc_info.index\n    \n    feats = []\n    for i, planet_id in tqdm(list(enumerate(planet_ids))):\n        signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/{planet_id}/{sensor}_signal.parquet').to_numpy()\n        dark_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dark.parquet', engine='pyarrow').to_numpy()\n        dead_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/dead.parquet', engine='pyarrow').to_numpy()\n        flat_frame = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/flat.parquet', engine='pyarrow').to_numpy()\n        linear_corr = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2024/{dataset}/' + str(planet_id) + '/' + sensor + '_calibration/linear_corr.parquet').values.astype(np.float64).reshape(linear_corr_dict[sensor])\n\n        signal = signal.reshape(sensor_sizes_dict[sensor][0]) \n        gain = adc_info[f'{sensor}_adc_gain'].values[i]\n        offset = adc_info[f'{sensor}_adc_offset'].values[i]\n        signal = signal / gain + offset\n        \n        hot = sigma_clip(\n            dark_frame, sigma=5, maxiters=5\n        ).mask\n        \n        if sensor != \"FGS1\":\n            signal = signal[:, :, cut_inf:cut_sup] \n            dt = np.ones(len(signal))*0.1 \n            dt[1::2] += 4.5 #@bilzard idea\n            linear_corr = linear_corr[:, :, cut_inf:cut_sup]\n            dark_frame = dark_frame[:, cut_inf:cut_sup]\n            dead_frame = dead_frame[:, cut_inf:cut_sup]\n            flat_frame = flat_frame[:, cut_inf:cut_sup]\n            hot = hot[:, cut_inf:cut_sup]\n        else:\n            dt = np.ones(len(signal))*0.1\n            dt[1::2] += 0.1\n            \n        signal = signal.clip(0) #@graySnow idea\n        linear_corr_signal = apply_linear_corr(linear_corr, signal)\n        signal = clean_dark(linear_corr_signal, dark_frame, dt)\n        \n        flat = flat_frame.reshape(sensor_sizes_dict[sensor][1])\n        flat[dead_frame.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n        flat[hot.reshape(sensor_sizes_dict[sensor][1])] = np.nan\n        signal = signal / flat\n        \n        \n        if sensor == \"FGS1\":\n            signal = signal[:,10:22,10:22] # **** updates ****\n            signal = signal.reshape(sensor_sizes_dict[sensor][0][0],144) # # **** updates ****\n\n        if sensor != \"FGS1\":\n            signal = signal[:,10:22,:] # **** updates ****\n\n        mean_signal = np.nanmean(signal, axis=1) \n        cds_signal = (mean_signal[1::2] - mean_signal[0::2])\n        \n        binned = np.zeros((binned_dict[sensor]))\n        for j in range(cds_signal.shape[0] // binning):\n            binned[j] = cds_signal[j*binning:j*binning+binning].mean(axis=0) \n                   \n        if sensor == \"FGS1\":\n            binned = binned.reshape((binned.shape[0],1))\n        \n        feats.append(binned)\n        \n    return np.stack(feats)\n    \npre_train = np.concatenate([preproc(f'{dataset}', adc_info, \"FGS1\", 30*12), preproc(f'{dataset}', adc_info, \"AIRS-CH0\", 30)], axis=2)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Modelization","metadata":{}},{"cell_type":"code","source":"def phase_detector(signal):\n    \n    MIN = np.argmin(signal[30:140])+30\n    signal1 = signal[:MIN ]\n    signal2 = signal[MIN :]\n\n    first_derivative1 = np.gradient(signal1)\n    first_derivative1 /= first_derivative1.max()\n    first_derivative2 = np.gradient(signal2)\n    first_derivative2 /= first_derivative2.max()\n\n    phase1 = np.argmin(first_derivative1)\n    phase2 = np.argmax(first_derivative2) + MIN\n\n    return phase1, phase2\n    \ndef objective(s):\n    \n    best_q = 1e10\n    for i in range(4) :\n        delta = 2\n        x = list(range(signal.shape[0]-delta*4))\n        y = signal[:p1-delta].tolist() + (signal[p1+delta:p2 - delta] * (1 + s)).tolist() + signal[p2+delta:].tolist()\n        \n        z = np.polyfit(x, y, deg=i)\n        p = np.poly1d(z)\n        q = np.abs(p(x) - y).mean()\n    \n    if q < best_q :\n        best_q = q\n    \n    return q\n\n\nall_s = []\nfor i in tqdm(range(len(adc_info))):\n    \n    signal = pre_train[i,:,1:].mean(axis=1)\n    p1,p2 = phase_detector(signal)\n \n    r = minimize(\n                objective,\n                [0.0001],\n                method= 'Nelder-Mead'\n                  )\n    s = r.x[0]\n    all_s.append(s)\n    \nall_s = np.repeat(np.array(all_s), 283).reshape((len(all_s), 283))        ","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predictions with Ridge model","metadata":{}},{"cell_type":"code","source":"# pd.DataFrame(all_s)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model = joblib.load(\"/kaggle/input/adc24-meta-model-ridge/model_ridge_10_22_delta2.joblib\")\n# pred = model.predict(all_s)\n# pd.DataFrame(pred)\n# import pickle\n# with open('/kaggle/input/ad24-train-inf-ridge-addfe-lb-441/model.pickle', 'rb') as f:\n#     model = pickle.load(f)\n# pred = model.predict(all_s)\n# pd.DataFrame(pred)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission","metadata":{}},{"cell_type":"code","source":"ss = pd.read_csv('/kaggle/input/ariel-data-challenge-2024/sample_submission.csv')\nsigma = np.ones_like(all_s) * 0.0001422 \npred = all_s.clip(0) \nsubmission = pd.DataFrame(np.concatenate([pred,sigma], axis=1), columns=ss.columns[1:])\nsubmission.index = adc_info.index\nsubmission.to_csv('submission_8.csv')\nsubmission\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble of solutions","metadata":{}},{"cell_type":"code","source":"# #### options\n\n# - option 1 -> LB=0.515 work (1,2)\n# - option 2 -> LB= ?.??? work (3,4,5) \n# - option 3 -> LB=0.000 work (1,2,3)         \n# - option 4 -> LB= ?.??? work (4,5)\n# - option 5 -> LB=0.514 work (1,2) + weight (0.15+0.85)\n# - option 7 -> LB=0.515 work (1,2) + weight (0.85+0.15)\n# - option 8 -> LB=0.515 work (1,2) + weight (0.07+0.93)\n# - option 9 -> LB=0.517 work (2,7)\n# - option 10 -> LB=0.516 work (1,2,7)\n# - option 11 -> LB=0.517 work (2,7) + weight (0.10+0.90)\n# - option 12 -> LB=0.517 work (2,7) + weight (0.05+0.95)\n# - option 14 -> LB=0.544 work (7,8) + weight (0.005+0.995)\n# - option 15 -> LB=0.544 work (7,8) + weight (0.008+0.992)\n# - option 16 -> LB=0.544 work (7,8) + weight (0.011+0.989)\n# - option 17 -> LB=0.544 work (7,8) + weight (0.015+0.985)\n# - option 18 -> LB=**0.544** work (7,8) + weight (0.02+0.98)\n# - option 19 -> LB=0.544 work (7,8) + weight (0.05+0.95)\n# - option 20 -> LB=0.544 work (7,8) + weight (0.07+0.93)\n\n# option.14 < option.15 < option.16 < option.17 < **option.18**\n# option.19 < option.20 < **option.18**\n\n# best option:\n# - option 18 -> LB=0.544 work (7,8) + weight (0.02+0.98), **Version 37** \n\n# current option: \n# - option 21 -> work (7,8) + weight (0.03+0.97)\n# - option 22 -> work (7,8) + weight (0.04+0.96)\n\n# next option: \n# - option 23 -> ?","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if LAUNCH_VARIANT == 'option 22':\n    df_solution_x = pd.read_csv('submission_7.csv')\n    df_solution_7 = pd.read_csv('submission_7.csv')\n    df_solution_8 = pd.read_csv('submission_8.csv')\n    df_solution_7 = df_solution_7.map(lambda x:x*0.04)\n    df_solution_8 = df_solution_8.map(lambda x:x*0.96)\n    df_temp = df_solution_7.add(df_solution_8)\n    df_submission = df_temp.map(lambda x:x)\n    df_submission['planet_id'] = df_solution_x['planet_id']\n    \nelif LAUNCH_VARIANT == 'option 21':\n    df_solution_x = pd.read_csv('submission_7.csv')\n    df_solution_7 = pd.read_csv('submission_7.csv')\n    df_solution_8 = pd.read_csv('submission_8.csv')\n    df_solution_7 = df_solution_7.map(lambda x:x*0.03)\n    df_solution_8 = df_solution_8.map(lambda x:x*0.97)\n    df_temp = df_solution_7.add(df_solution_8)\n    df_submission = df_temp.map(lambda x:x)\n    df_submission['planet_id'] = df_solution_x['planet_id']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submission.to_csv('submission.csv', index=False, float_format='%.7f')\ndf_submission","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#                                          arhiv options\n# if LAUNCH_VARIANT == 'option 20':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.07)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.93)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n    \n# elif LAUNCH_VARIANT == 'option 19':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.05)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.95)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 18':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.020)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.980)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n    \n# elif LAUNCH_VARIANT == 'option 17':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.015)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.985)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 16':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.011)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.989)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 15':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.008)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.992)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 14':\n#     df_solution_x = pd.read_csv('submission_7.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_8 = pd.read_csv('submission_8.csv')\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.005)\n#     df_solution_8 = df_solution_8.map(lambda x:x*0.995)\n#     df_temp = df_solution_7.add(df_solution_8)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 12':\n#     df_solution_x = pd.read_csv('submission_2.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_2 = df_solution_2.map(lambda x:x*0.05)\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.95)\n#     display(df_solution_2, df_solution_7)\n#     df_temp = df_solution_2.add(df_solution_7)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 11':\n#     df_solution_x = pd.read_csv('submission_2.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_7 = pd.read_csv('submission_7.csv')\n#     df_solution_2 = df_solution_2.map(lambda x:x*0.10)\n#     df_solution_7 = df_solution_7.map(lambda x:x*0.90)\n#     display(df_solution_2, df_solution_7)\n#     df_temp = df_solution_2.add(df_solution_7)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 10':\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_6 = pd.read_csv('submission_6.csv')\n#     display(df_solution_1, df_solution_2, df_solution_6)\n#     df_temp = df_solution_1.add(df_solution_2).add(df_solution_6)\n#     df_submission = df_temp.map(lambda x: x/3)\n#     df_submission['planet_id'] = df_solution_1['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 9':\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_6 = pd.read_csv('submission_6.csv')\n#     display(df_solution_2, df_solution_6)\n#     df_temp = df_solution_2.add(df_solution_6)\n#     df_submission = df_temp.map(lambda x: x/2)\n#     df_submission['planet_id'] = df_solution_2['planet_id']\n# \n# elif LAUNCH_VARIANT == 'option 8':\n#     df_solution_x = pd.read_csv('submission_1.csv')\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_1 = df_solution_1.map(lambda x: x*0.07)\n#     df_solution_2 = df_solution_2.map(lambda x: x*0.93)\n#     display(df_solution_1, df_solution_2)\n#     df_temp = df_solution_1.add(df_solution_2)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 7':\n#     df_solution_x = pd.read_csv('submission_1.csv')\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_1 = df_solution_1.map(lambda x: x*0.85)\n#     df_solution_2 = df_solution_2.map(lambda x: x*0.15)\n#     display(df_solution_1, df_solution_2)\n#     df_temp = df_solution_1.add(df_solution_2)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 5':\n#     df_solution_x = pd.read_csv('submission_1.csv')\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_1 = df_solution_1.map(lambda x: x*0.15)\n#     df_solution_2 = df_solution_2.map(lambda x: x*0.85)\n#     display(df_solution_1, df_solution_2)\n#     df_temp = df_solution_1.add(df_solution_2)\n#     df_submission = df_temp.map(lambda x:x)\n#     df_submission['planet_id'] = df_solution_x['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 4':\n#     df_solution_4 = pd.read_csv('submission_4.csv')\n#     df_solution_5 = pd.read_csv('submission_5.csv')\n#     display(df_solution_4, df_solution_5)\n#     df_temp = df_solution_4.add(df_solution_5)\n#     df_submission = df_temp.map(lambda x: x/2)\n#     df_submission['planet_id'] = df_solution_1['planet_id']\n# \n# elif LAUNCH_VARIANT == 'option 3':\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     df_solution_3 = pd.read_csv('submission_3.csv')\n#     display(df_solution_1, df_solution_2, df_solution_3)\n#     df_temp = df_solution_1.add(df_solution_2).add(df_solution_3)\n#     df_submission = df_temp.map(lambda x: x/3)\n#     df_submission['planet_id'] = df_solution_1['planet_id']\n#\n# elif LAUNCH_VARIANT == 'option 2':\n#     df_solution_3 = pd.read_csv('submission_3.csv')\n#     df_solution_4 = pd.read_csv('submission_4.csv')\n#     df_solution_5 = pd.read_csv('submission_5.csv')\n#     display(df_solution_3, df_solution_4, df_solution_5)\n#     df_temp = df_solution_3.add(df_solution_4).add(df_solution_5)\n#     df_submission = df_temp.map(lambda x: x/3)\n#     df_submission['planet_id'] = df_solution_3['planet_id']\n#      \n# elif LAUNCH_VARIANT == 'option 1':\n#     df_solution_1 = pd.read_csv('submission_1.csv')\n#     df_solution_2 = pd.read_csv('submission_2.csv')\n#     display(df_solution_1, df_solution_2)\n#     df_temp = df_solution_1.add(df_solution_2)\n#     df_submission = df_temp.map(lambda x: x/2)\n#     df_submission['planet_id'] = df_solution_1['planet_id']","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]}]}