{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70367,"databundleVersionId":9188054,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Noise correction and sensor fusion\n\nIn this notebook, I will correct for the infrared and visual wavelength sensors' errors. I will also explore the wavelength.csv files, what labels look like, and what the axis information entails\n\n## Noise Correction for AIRS-CH0 Infrared Sensor and FGS1 Visual wavelength sensor on one planet's signal\n\nStarting off, I am just going to use gain and offset as well as dark, dead, linear correlation, flat, and read frames to correct for measurement noise, as laid out in the Kaggle data section for this competition. In the following code block, I am going to load in the noise correction frames, and get the infrared and visual signal for a random planet. I will go through the process of noise correction for this one planet and visualize the transition to ensure that the correction works properly.","metadata":{}},{"cell_type":"code","source":"# import data, libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom matplotlib import pyplot as plt\n\n# load gain, offset\ngain_offset_csv = pd.read_csv(\"/kaggle/input/ariel-data-challenge-2024/train_adc_info.csv\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-08-01T20:36:26.452016Z","iopub.execute_input":"2024-08-01T20:36:26.452744Z","iopub.status.idle":"2024-08-01T20:36:27.574811Z","shell.execute_reply.started":"2024-08-01T20:36:26.452712Z","shell.execute_reply":"2024-08-01T20:36:27.573646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since correction information is stored in the same way, we can make a function for both sensors and all planets. For an explaination of why I am dividing some correction frames, subtracting others and treating others as polynomial coefficients, read the linked article in the kaggle data section for this competition.","metadata":{}},{"cell_type":"code","source":"sensor_sizes_dict = {\"AIRS-CH0\":[[11250, 32, 356], [1, 32, 356]], \"FGS1\":[[135000, 32, 32], [1, 32, 32]]} # switch data sizes depending on the sensor\n\n# define data loading function\ndef load_data(sensor, planet_id):\n    \n    # load offset, gain for this planet\n    planet_gain_offset = gain_offset_csv[gain_offset_csv[\"planet_id\"] == planet_id]\n    \n    # get all noise correction frames and signal\n    signal = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_signal.parquet', engine='pyarrow')\n    dark_frame = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_calibration/dark.parquet', engine='pyarrow')\n    dead_frame = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_calibration/dead.parquet', engine='pyarrow')\n    linear_corr_frame = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_calibration/linear_corr.parquet', engine='pyarrow')\n    flat_frame = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_calibration/flat.parquet', engine='pyarrow')\n    read_frame = pd.read_parquet('/kaggle/input/ariel-data-challenge-2024/train/' + str(planet_id) + '/' + sensor + '_calibration/read.parquet', engine='pyarrow')\n\n    # to numpy\n    signal = signal.to_numpy().reshape(sensor_sizes_dict[sensor][0])\n\n    # read and dark frame correction, just subtraction\n    signal = signal - dark_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n    signal = signal - read_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n\n    # flat frame correction + ensure dead pixels do not disrupt. We divide by the flat frame\n    flat = flat_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n    flat[dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])] = 1\n    signal = signal/flat\n\n    # linear correction - treat as sixth degree polynomial, first number is C, not coefficient?\n    coefficients = linear_corr_frame.to_numpy().reshape([6] + sensor_sizes_dict[sensor][1])\n    coefficients = np.repeat(coefficients, sensor_sizes_dict[sensor][0][0], axis=1)\n    corrected = coefficients[0]\n    for i in range(5):\n        corrected += np.multiply(np.power(signal, i+1),coefficients[i+1])\n        \n    # gain, offset\n    corrected = corrected*planet_gain_offset[sensor + \"_adc_gain\"].values + planet_gain_offset[sensor + \"_adc_offset\"].values\n        \n    # zero out malfunctioning pixels\n    corrected[np.repeat(dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1]), sensor_sizes_dict[sensor][0][0], axis=0)] = 0\n        \n    return corrected\n","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:40:38.236526Z","iopub.execute_input":"2024-08-01T20:40:38.237234Z","iopub.status.idle":"2024-08-01T20:40:38.24877Z","shell.execute_reply.started":"2024-08-01T20:40:38.237199Z","shell.execute_reply":"2024-08-01T20:40:38.247755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# planet\nplanet_id = 100468857\n\ninfrared_data = load_data(\"AIRS-CH0\", planet_id)\nvisual_data = load_data(\"FGS1\", planet_id)","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:41:57.962791Z","iopub.execute_input":"2024-08-01T20:41:57.963168Z","iopub.status.idle":"2024-08-01T20:42:25.26737Z","shell.execute_reply.started":"2024-08-01T20:41:57.963137Z","shell.execute_reply":"2024-08-01T20:42:25.266248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# graph\n%matplotlib inline \nim_data = infrared_data[2, :, :]\n\nprint(np.min(im_data))\nprint(np.max(im_data))\n\nim_data = (im_data - np.min(im_data))/np.max(im_data)\n\nplt.imshow(im_data, cmap='inferno')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:44:58.048914Z","iopub.execute_input":"2024-08-01T20:44:58.049316Z","iopub.status.idle":"2024-08-01T20:44:58.340436Z","shell.execute_reply.started":"2024-08-01T20:44:58.049285Z","shell.execute_reply":"2024-08-01T20:44:58.339368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"im_data = visual_data[0, :, :]\n\nprint(np.min(im_data))\nprint(np.max(im_data))\n\nim_data = (im_data - np.min(im_data))/np.max(im_data)\n\nplt.figure(figsize=(10, 10))\nplt.imshow(im_data, cmap='inferno')\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:45:00.151913Z","iopub.execute_input":"2024-08-01T20:45:00.15228Z","iopub.status.idle":"2024-08-01T20:45:00.512438Z","shell.execute_reply.started":"2024-08-01T20:45:00.152251Z","shell.execute_reply":"2024-08-01T20:45:00.51124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you are wondering how much the cleaning process changes the data, look at version one of the data for a comparison w/ raw data. Short answer is that the data only changes very slightly. I think these data look pretty sensible. If we are going to make informed predictions from these data, however, we should know what we are predicting. Lets look at the wavelength.csv and labels for this challenge:","metadata":{}},{"cell_type":"markdown","source":"## Wavelength and Label examples","metadata":{}},{"cell_type":"code","source":"wavelengths = pd.read_csv(\"/kaggle/input/ariel-data-challenge-2024/wavelengths.csv\")\nlabels = pd.read_csv(\"/kaggle/input/ariel-data-challenge-2024/train_labels.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:53:40.007195Z","iopub.execute_input":"2024-08-01T20:53:40.007915Z","iopub.status.idle":"2024-08-01T20:53:40.106434Z","shell.execute_reply.started":"2024-08-01T20:53:40.007881Z","shell.execute_reply":"2024-08-01T20:53:40.10536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(wavelengths)","metadata":{"execution":{"iopub.status.busy":"2024-08-01T20:53:43.705302Z","iopub.execute_input":"2024-08-01T20:53:43.706186Z","iopub.status.idle":"2024-08-01T20:53:43.719614Z","shell.execute_reply.started":"2024-08-01T20:53:43.706151Z","shell.execute_reply":"2024-08-01T20:53:43.718448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like wavelengths are measured in micrometers! That means that most of the things they care about are actually in the infrared part of the absorbtion spectrum. It may be important later to only predict for certain wavelengths using the sensors that have those wavelengths within thier observation capability.","metadata":{}},{"cell_type":"code","source":"print(labels)\nprint(max(labels.iloc[1].values))","metadata":{"execution":{"iopub.status.busy":"2024-08-01T21:04:27.151958Z","iopub.execute_input":"2024-08-01T21:04:27.15304Z","iopub.status.idle":"2024-08-01T21:04:27.169735Z","shell.execute_reply.started":"2024-08-01T21:04:27.153Z","shell.execute_reply":"2024-08-01T21:04:27.168656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like the absorbtion spectra are extremely sparse, so softmax output functions might be our friend with this.","metadata":{}},{"cell_type":"markdown","source":"## Axis Info Exploration","metadata":{}},{"cell_type":"code","source":"axis_data = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2024/axis_info.parquet\", engine='pyarrow')\naxis_data","metadata":{"execution":{"iopub.status.busy":"2024-08-01T21:06:59.635331Z","iopub.execute_input":"2024-08-01T21:06:59.63609Z","iopub.status.idle":"2024-08-01T21:06:59.680712Z","shell.execute_reply.started":"2024-08-01T21:06:59.636058Z","shell.execute_reply":"2024-08-01T21:06:59.679648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After taking a quick look at the data, I realized that this is not the first go-round of this competition. Be sure to read previous research papers: https://arxiv.org/pdf/2309.09337","metadata":{}}]}