{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":70367,"databundleVersionId":9188054,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n
\n
\n \"Book\n
\n
\n

CREATED BY: SUBHANJAN DAS

\n
","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n TABLE OF CONTENTS\n

\n\n
\n\n

1    MISSION BRIEFING

\n\n
\n\n

2    COSMIC BACKGROUND INFO

\n\n
\n\n

3    IMPORTS

\n\n
\n\n

4    CALIBRATION FUNCTION

\n\n
\n\n

5    3D EXPLORATORY DATA ANALYSIS

\n\n
\n\n

6    SUBMISSION

\n\n
\n\n

7    CHANGE LOG

\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

1  MISSION BRIEFING    ⤒

","metadata":{}},{"cell_type":"markdown","source":"

1.1 MISSION OVERVIEW

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"

1.2 OBJECTIVES OF THE MISSION

\n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"

1.3 APPROACH AND METHODOLOGY

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"

1.4 CHALLENGES AND ASSUMPTIONS

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2  COSMIC BACKGROUND INFO    ⤒

\n\n
\n\nThis is where we'll talk about the scientific foundations behind the data, the instruments used to collect it, and the key concepts we'll need to understand to make sense of our findings. Prepare for an interstellar briefing on all things cosmic!\n\n
","metadata":{}},{"cell_type":"markdown","source":"

2.1 THE ARIEL SPACE MISSION

\n
\n\n

The Ariel (Atmospheric Remote-sensing Infrared Exoplanet Large-survey) mission is a pioneering space telescope set to launch in 2029, aimed at unlocking the secrets of the atmospheres of distant exoplanets. This mission, orchestrated by the European Space Agency (ESA), represents a significant leap forward in our quest to understand the diversity of planets beyond our solar system.

\n\n

While most space telescopes, like Kepler and TESS, have focused primarily on discovering exoplanets, Ariel's mission is to perform detailed studies of their atmospheres. By observing a diverse sample of exoplanets—from gas giants to rocky planets—Ariel will help scientists unravel the composition, formation, and evolution of these distant worlds.

\n\n
\n
\n \"Book\n
\n
\n\n

The Science Behind Ariel

\n

Ariel is specifically designed to study the chemical fingerprints in the atmospheres of exoplanets. These fingerprints, observed in the form of absorption and emission spectra, provide critical information about the molecules present in the atmospheres, such as water vapor, carbon dioxide, methane, and more. By analyzing these spectra, Ariel will be able to determine the composition of exoplanetary atmospheres, temperature structures, and even cloud cover.

\n\n

Unlike previous missions that might observe a single planet in great detail, Ariel is a large-scale survey mission. It will observe around 1,000 exoplanets, providing a comprehensive view of the diversity of planetary atmospheres in our galaxy. This will allow scientists to draw broader conclusions about planet formation and evolution across different environments.

\n\n

Mission Objectives

\n\n\n
\n
\n \"Book\n
\n
\n\n

Why Ariel is Important

\n

While the discovery of exoplanets has been one of the most exciting developments in modern astronomy, understanding what these planets are made of is the next frontier. Ariel is poised to answer key questions about the formation and evolution of planets in our galaxy. Are planets around other stars made of the same stuff as Earth? How do planetary systems form and evolve? Ariel's observations will help us answer these questions by providing the first large-scale survey of the chemical composition of exoplanetary atmospheres.

\n\n

The data collected by Ariel will not only help us understand exoplanets better but also offer clues about the potential for life elsewhere in the universe. By comparing the atmospheres of different planets, Ariel will help scientists identify worlds that might be habitable—or at least, worlds that share characteristics with Earth.

\n\n

The Instruments Onboard

\n

Ariel will be equipped with a suite of sophisticated instruments designed to capture the faint light from distant exoplanets. The primary instrument, a meter-class telescope, will observe in the visible to infrared spectrum (0.5 to 7.8 micrometers), which is ideal for detecting the signatures of molecules in exoplanetary atmospheres. Additionally, the mission includes an infrared spectrometer and a photometer to capture detailed spectra and images, providing a wealth of data for analysis.

\n\n

The mission will also rely on the use of two main instruments: the AIRS (Ariel Infrared Spectrometer) and the FGS (Fine Guidance Sensor). The AIRS will be crucial for capturing high-resolution spectra, while the FGS will ensure that the telescope remains precisely pointed at its targets during observations, ensuring the accuracy of the data collected.

\n\n

With its powerful instruments and ambitious objectives, the Ariel mission is set to revolutionize our understanding of exoplanets and their atmospheres, opening up a new chapter in the search for life beyond Earth.

\n\n

Stay tuned as we dive deeper into the specific instruments and techniques Ariel will use to achieve its ambitious goals!

\n","metadata":{}},{"cell_type":"markdown","source":"

2.2 THE CHALLENGE

\n
\n\n

As we venture further into the cosmos with the Ariel mission, we find ourselves at the forefront of one of the most complex and fascinating challenges in modern astronomy: the characterization of exoplanetary atmospheres. While Ariel's advanced instruments are designed to capture the faint signals emitted or absorbed by these distant worlds, the real challenge lies in deciphering this data to reveal the secrets hidden in the atmospheres of exoplanets.

\n\n

The Heart of the Challenge

\n

The discovery of over 5,600 exoplanets has revolutionized our understanding of the universe, proving that planets are not just a rare phenomenon but a common feature of stars. However, merely detecting these planets is just the beginning. To truly understand their nature, potential habitability, and how they compare to our own Earth, we must delve into their atmospheres.

\n\n

This is where the Ariel mission comes into play, with its goal of conducting the first comprehensive study of 1,000 exoplanet atmospheres. However, the task is anything but simple. When an exoplanet transits its host star, a tiny fraction of the starlight—about 50 to 200 photons per million—passes through the planet's atmospheric annulus. This light interacts with the planet's atmospheric molecules, clouds, and winds, imprinting a faint spectral signature that can tell us what the atmosphere is made of.

\n\n

The magnitude of these signals, ranging from 50 parts per million (ppm) for smaller, rocky Super-Earths to around 200 ppm for larger, gaseous Jupiter-like planets, makes this one of the most difficult data-analysis challenges in astronomy today. Adding to the complexity, these signals are often contaminated by instrumental noise, making it even harder to extract meaningful data.

\n\n
\n
\n \"Book\n
\n
\n\n

What This Competition Entails

\n

The objective of this competition is to extract the atmospheric spectra from the observational data captured during these transits. Competitors are tasked with developing methods to detrend a large number of sequential 2D images of the spectral focal plane. These images, taken over several hours as the exoplanet transits its star, contain the faint atmospheric signatures we seek to extract.

\n\n

The challenge lies in cleaning and processing this raw data to remove noise and other confounding factors, allowing the true atmospheric signal to emerge. This process, known as detrending, is a critical step in modern astronomical data analysis. It must be done with great precision to ensure that the extracted spectra are accurate and that the uncertainty levels associated with these spectra are well quantified.

\n\n

The Importance of Detrending

\n

Detrending is not just a technical step; it is the gateway to unlocking the information contained in exoplanetary atmospheres. Without effective detrending, the faint signals that reveal the chemical composition, temperature, and even the weather patterns of these distant worlds would remain obscured by noise and distortions.

\n\n

In this competition, we will be required to handle a dataset that simulates the type of observations Ariel will collect. The challenge is to develop algorithms that can effectively separate the true planetary signal from the noise, delivering clean spectra that can then be used for scientific analysis. This is not just a computational problem; it is a test of our ability to push the boundaries of data analysis in astronomy, paving the way for the next generation of exoplanet discoveries.

\n","metadata":{}},{"cell_type":"markdown","source":"

2.3 DATA OVERVIEW

\n
\n\n

Understanding the data and its organization is crucial for successfully navigating the Ariel Data Challenge. This section will provide a comprehensive overview of the data types, file structures, and key elements you'll encounter throughout the competition. So, grab your cosmic map—we're about to chart our course through the dataset!

\n\n

The Dataset

\n

The dataset provided for this competition is extensive and designed to simulate real-world conditions that the Ariel mission will encounter when observing exoplanets. It includes various types of data captured by Ariel’s instruments during the simulated observations of exoplanets. The goal is to use this data to develop models that can accurately predict certain planetary characteristics, such as atmospheric composition, temperature profiles, and cloud structures.

\n\n

Data Types

\n

The data is divided into several types, each corresponding to different aspects of the observations:

\n\n\n

Directory Structure

\n

The dataset is organized into a clear directory structure to help you easily navigate and access the different types of data. Here's a breakdown of the key directories and their contents:

\n\n```\n/kaggle/input/ariel-data-challenge-2024/\n├── axis_info.parquet\n├── wavelengths.csv\n├── train_adc_info.csv\n├── train_labels.csv\n├── sample_submission.csv\n├── test_adc_info.csv\n├── train/\n│ ├── 100468857/\n│ │ ├── AIRS-CH0_calibration/\n│ │ │ ├── dark.parquet\n│ │ │ ├── dead.parquet\n│ │ │ ├── flat.parquet\n│ │ │ ├── linear_corr.parquet\n│ │ ├── FGS1_calibration/\n│ │ │ ├── dark.parquet\n│ │ │ ├── dead.parquet\n│ │ │ ├── flat.parquet\n│ │ │ ├── linear_corr.parquet\n│ │ ├── AIRS-CH0_signal.parquet\n│ │ ├── FGS1_signal.parquet\n│ ├── ...\n├── test/\n│ ├── 499191466/\n│ │ ├── AIRS-CH0_calibration/\n│ │ │ ├── dark.parquet\n│ │ │ ├── dead.parquet\n│ │ │ ├── flat.parquet\n│ │ │ ├── linear_corr.parquet\n│ │ ├── FGS1_calibration/\n│ │ │ ├── dark.parquet\n│ │ │ ├── dead.parquet\n│ │ │ ├── flat.parquet\n│ │ │ ├── linear_corr.parquet\n│ │ ├── AIRS-CH0_signal.parquet\n│ │ ├── FGS1_signal.parquet\n│ ├── ...\n```\n\n
\n\n\n

Data Structure Overview

\n

Each planet in the training set has its data stored in a hierarchical structure, making it easier to manage and process the data. The primary data files are the signal data files, which are large arrays representing the signal intensity across different spatial locations, wavelengths, and time points. These are supplemented by calibration files, which are necessary to correct and clean the raw data.

\n\n

The data is provided in parquet format, which is highly efficient for storing and processing large datasets, particularly in the context of big data analysis. The structure of the data files is consistent across all planets, allowing for standardized processing methods to be applied to each dataset.

\n\n

Column Overview

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
planet_idUnique identifier for each planet.
wl_1 to wl_283Spectral intensities at various wavelengths, representing the target variables for prediction.
FGS1_adc_offsetOffset value used during the ADC process for the FGS1 instrument.
FGS1_adc_gainGain value used during the ADC process for the FGS1 instrument.
AIRS-CH0_adc_offsetOffset value used during the ADC process for the AIRS-CH0 instrument.
AIRS-CH0_adc_gainGain value used during the ADC process for the AIRS-CH0 instrument.
starBinary indicator for whether the planet's system contains a star (1: contains star, 0: no star).
time_axisTimestamps corresponding to the signal data measurements.
spatial_axisInformation related to the spatial dimension of the instrument readings.
wavelength_axisWavelength axis information that maps to the wl_1 to wl_283 columns.
wavelength_idIdentifier for each wavelength in the wavelengths.csv file.
wavelength_valueActual wavelength value (in nanometers) corresponding to each wavelength_id.
dark.parquetDark frames used for subtracting dark current from the signal data.
dead.parquetDead pixel maps indicating non-functional pixels that should be ignored.
flat.parquetFlat field images used to correct for pixel sensitivity variations across the detector.
linear_corr.parquetPolynomial coefficients for correcting the non-linearity of the pixel response.
read.parquetAdditional calibration data necessary for processing the signal.
AIRS-CH0_signal.parquetRaw signal data collected by the AIRS-CH0 instrument.
FGS1_signal.parquetRaw signal data collected by the FGS1 instrument.
\n\n

How This Data Will Be Used

\n

The data provided in this competition will be used to train machine learning models that can accurately predict the atmospheric and planetary characteristics of exoplanets based on the observed signals. The goal is to develop robust models that can generalize well to new, unseen planets in the test set.

\n\n

The raw signal data, once calibrated and cleaned, will serve as the primary input to the models. The labels provided in the training set will be used to evaluate the models' performance during development. The calibration data ensures that the models are working with the best possible version of the signal data, free from noise and artifacts that could otherwise hinder model performance.

\n\n

With a clear understanding of the data and its structure, we're now ready to dive into the exploratory data analysis (EDA) phase, where we'll uncover patterns and insights that will guide our modeling efforts.

\n","metadata":{}},{"cell_type":"markdown","source":"

2.4 MISSION EVALUATION

\n
\n\n

In the vast expanse of the cosmos, the success of our mission hinges on one crucial metric: the Gaussian Log-Likelihood (GLL) function. As we traverse the celestial data, our goal is to predict the spectra (\\(\\mu_{user}\\)) and their corresponding uncertainties (\\(\\sigma_{user}\\)) across different wavelengths. The ultimate aim? To align these predictions as closely as possible with the ground truth spectra (\\(y\\)) provided by our Earth-based observers.

\n\n

The Gaussian Log-Likelihood (GLL) Function

\n\nThe GLL function is our guiding star, mathematically represented as:\n\n$$\n\\text{GLL} = -\\frac{1}{2} \\left( \\log(2\\pi) + \\log(\\sigma_{user}^2) + \\frac{(y - \\mu_{user})^2}{\\sigma_{user}^2} \\right)\n$$\n\nThis equation ensures that every prediction we make is evaluated against the ground truth, taking into account both the difference in values and the uncertainty associated with our prediction. The closer our \\(\\mu_{user}\\) is to \\(y\\), and the smaller our uncertainty \\(\\sigma_{user}\\), the better our GLL score.\n\n

Summing Across the Wavelengths

\n\nOnce the GLL values for all wavelength predictions are calculated, they are summed across the entire spectrum to produce a final GLL value, denoted as \\(L\\). This final \\(L\\) value is then transformed into a score that helps us evaluate our model's performance.\n\n

Score Calculation

\n\nThe transformation from \\(L\\) to the final score is achieved using the following conversion function:\n\n$$\n\\text{score} = \\frac{L - L_{\\text{ref}}}{L_{\\text{ideal}} - L_{\\text{ref}}}\n$$\n\nHere’s what each term represents:\n\n- $(L_{\\text{ideal}}$): The GLL value in a hypothetical perfect scenario where our predictions perfectly match the ground truth with an uncertainty of 10 parts per million (ppm). This ideal case is defined based on Ariel's Stability Requirement.\n- $(L_{\\text{ref}}$): A reference GLL value calculated using the mean and variance of the training dataset for all instances.\n\nThe score we obtain will be a float in the interval \\([0, 1]\\), with a higher score corresponding to better-performing models. Any score below 0 will be treated as 0.\n\nIn essence, this scoring mechanism provides a stringent evaluation of our model's ability to predict spectra with high accuracy and low uncertainty. Just like a spacecraft navigating through the stars, precision and reliability are paramount.\n\n

Final Note:

\n\nIn this mission, we aim to push the boundaries of what’s possible in spectral prediction. Every step we take in improving our GLL score brings us closer to unraveling the mysteries of the universe, one wavelength at a time.\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

3  IMPORTS    ⤒

\n\n
","metadata":{}},{"cell_type":"code","source":"# Basic Data Handling and Analysis\nimport numpy as np # Numerical operations\nimport pandas as pd # Data manipulation with DataFrames\n\n# Basic 2D and 3D Plotting\nimport matplotlib.pyplot as plt # Basic plotting\nfrom mpl_toolkits.mplot3d import Axes3D # Static 3D plots with Matplotlib\n\n# Seaborn for Enhanced 2D Visualizations\nimport seaborn as sns # Advanced and aesthetically pleasing 2D plots\n\n# Plotly for Interactive 3D Visualization\nimport plotly.graph_objects as go # Flexible and powerful 3D visualizations\nimport plotly.express as px # Quick and simple 3D visualizations\nfrom plotly.subplots import make_subplots # For creating subplots\nfrom sklearn.decomposition import PCA\nfrom IPython.core.display import HTML, Markdown\n\n# Path Handling\nfrom pathlib import Path # For efficient file path handling\n\n# File I/O\nimport os # Interacting with the operating system\n\n# Time Series Analysis\nfrom statsmodels.tsa.stattools import adfuller, kpss # Stationarity tests\nfrom statsmodels.graphics.tsaplots import plot_acf, plot_pacf # Autocorrelation plots\n\n# Utility Libraries\nfrom tqdm import tqdm # Progress bars for loops\nimport gc # Garbage collection for memory management\nimport warnings # Suppress warnings\n\nfrom astropy.stats import sigma_clip\nfrom tqdm import tqdm # For progress tracking\n\n# Ignore warnings to keep the notebook clean\nwarnings.filterwarnings('ignore')\nprint(\"All imports successful!\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-08-19T15:38:28.280966Z","iopub.execute_input":"2024-08-19T15:38:28.282474Z","iopub.status.idle":"2024-08-19T15:38:28.293633Z","shell.execute_reply.started":"2024-08-19T15:38:28.282426Z","shell.execute_reply":"2024-08-19T15:38:28.292209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n

4  CALIBRATION FUNCTION    ⤒

\n\nEmbarking on our celestial journey to unravel the mysteries of exoplanets, we must first ensure that our instruments—though marvels of modern technology—are perfectly tuned. The function we have devised is like a space engineer fine-tuning the sensors on a starship, ensuring every photon captured from a distant world is recorded with the highest fidelity. Below is a breakdown of the calibration steps within our `load_and_calibrate_signal` function, each designed to bring us closer to an accurate understanding of these distant celestial bodies.\n
","metadata":{}},{"cell_type":"markdown","source":"

4.1 Analog-to-Digital Conversion (ADC) Correction

\n
\nWhen our space telescope collects light from a distant exoplanet, this light is converted into an electrical signal, which is then digitized. However, this digitization process is not perfect—it can introduce errors. To counteract these errors, we apply a `gain` and `offset` correction to the raw signal data.\n\n

Why?

\n Restoring the signal's full dynamic range is crucial for accurate analysis. Without this correction, the signal may be distorted, leading to inaccuracies in further data processing steps.\n \n

The Math

\n The corrected signal $( S_{\\text{corrected}} $) is computed as: \n $$\n S_{\\text{corrected}} = \\frac{S_{\\text{raw}}}{\\text{gain}} + \\text{offset}\n $$\n where:\n - $( S_{\\text{raw}} $) is the raw signal.\n - `gain` and `offset` are calibration parameters specific to each planet's instrument, retrieved from the `train_adc_info` DataFrame.\n \n

Expectations

\n After applying this correction, we expect the signal to accurately reflect the intensity of the incoming light, with the full range of detector response restored.\n\n\n

4.2 Wavelength Range Cropping

\n
\nExoplanet observations often span a wide range of wavelengths. However, not all wavelengths are equally important or useful for our analysis. In this step, we crop the signal to focus on a specific range of wavelengths that hold the most valuable information.\n\n

Why?

\n Different parts of the spectrum may contain noise or irrelevant data. By focusing on a specific wavelength range (from index 39 to 321), we can enhance the quality of our analysis and avoid potential confounding factors.\n \n

The Math

\n This is a straightforward slice of the signal matrix:\n $$\n S_{\\text{chopped}} = S_{\\text{corrected}}[:, :, \\text{cut\\_inf} : \\text{cut\\_sup}]\n $$\n \n

Expectations

\n The result is a more focused dataset, where only the most informative wavelengths are retained, reducing computational load and noise in subsequent steps.\n\n

4.3 Dark Frame Correction

\n
\nEven in the absence of light, the sensors on our space telescope may record a small signal—this is known as **dark current**. This unwanted signal can obscure the true light we want to measure. To correct for this, we subtract the dark frame from the signal.\n\n

Why?

\n Dark current is an unavoidable artifact in detector systems. Failing to subtract this would leave our signal contaminated with non-signal-related noise, leading to erroneous conclusions.\n \n

The Math

\n The dark-corrected signal $( S_{\\text{dark\\_corrected}} $) is computed as:\n $$\n S_{\\text{dark\\_corrected}} = S_{\\text{chopped}} - \\text{dark}\n $$\n where `dark` is the dark frame calibration file.\n \n

Expectations

\n By removing the dark current, we expect the remaining signal to be a true representation of the incoming light, devoid of detector-induced noise.\n\n

4.4 Dead and Hot Pixel Masking

\n
\nNot all pixels on our sensor array are created equal. Some pixels might be \"dead\" (non-responsive to light) or \"hot\" (overly responsive to light). These faulty pixels can introduce artifacts into our data. In this step, we identify and mask these problematic pixels to prevent them from skewing our analysis.\n\n

Why?

\n Both hot and dead pixels are imperfections in the detector. If not accounted for, they can introduce significant errors into our analysis, leading to incorrect interpretations of the data.\n \n

The Math

\n A mask is applied to the signal:\n $$\n S_{\\text{cleaned}} = \\text{masked\\_where}( \\text{dead\\_pixels} \\lor \\text{hot\\_pixels}, S_{\\text{dark\\_corrected}})\n $$\n Here, the `sigma_clip` function helps identify hot pixels, and the dead pixel map is applied. The logical OR (`\\lor`) operation ensures that any pixel identified as either hot or dead is masked out.\n \n

Expectations

\n After masking, the signal should be free from the distortions caused by faulty pixels, allowing for a more accurate analysis of the celestial data.","metadata":{}},{"cell_type":"code","source":"def load_and_calibrate_signal(planet_id, train_adc_info, axis_info, path_folder, cut_inf=39, cut_sup=321):\n \"\"\"\n Load and calibrate the signal data for a given planet.\n \n Parameters:\n - planet_id: The ID of the planet.\n - train_adc_info: DataFrame containing ADC information for all planets.\n - axis_info: DataFrame containing axis information for all planets.\n - path_folder: The root path to the data files.\n - cut_inf: Lower bound for wavelength cut (default=39).\n - cut_sup: Upper bound for wavelength cut (default=321).\n \n Returns:\n - Tuple of cleaned AIRS-CH0 and FGS1 signal data.\n \"\"\"\n # Load AIRS-CH0 signal and apply ADC conversion\n airs_signal_file = os.path.join(path_folder, f'train/{planet_id}/AIRS-CH0_signal.parquet')\n df = pd.read_parquet(airs_signal_file)\n signal = df.values.astype(np.float64).reshape((df.shape[0], 32, 356))\n \n gain = train_adc_info['AIRS-CH0_adc_gain'].loc[planet_id]\n offset = train_adc_info['AIRS-CH0_adc_offset'].loc[planet_id]\n signal = (signal / gain) + offset\n \n # Cut the wavelength range\n chopped_signal = signal[:, :, cut_inf:cut_sup]\n \n # Load calibration files\n dark = pd.read_parquet(os.path.join(path_folder, f'train/{planet_id}/AIRS-CH0_calibration/dark.parquet')).values.astype(np.float64)\n dead_airs = pd.read_parquet(os.path.join(path_folder, f'train/{planet_id}/AIRS-CH0_calibration/dead.parquet')).values.astype(bool)\n \n # Apply the same cut to dark and dead arrays to match chopped_signal shape\n dark = dark[:, cut_inf:cut_sup]\n dead_airs = dead_airs[:, cut_inf:cut_sup]\n \n # Mask hot and dead pixels\n hot = sigma_clip(dark, sigma=5, maxiters=5).mask.astype(bool)\n hot = np.tile(hot, (chopped_signal.shape[0], 1, 1))\n dead_airs = np.tile(dead_airs, (chopped_signal.shape[0], 1, 1))\n AIRS_CH0_clean = np.ma.masked_where(dead_airs | hot, chopped_signal)\n \n # Repeat similar steps for FGS1\n fgs_signal_file = os.path.join(path_folder, f'train/{planet_id}/FGS1_signal.parquet')\n df = pd.read_parquet(fgs_signal_file)\n signal = df.values.astype(np.float64).reshape((df.shape[0], 32, 32))\n \n gain = train_adc_info['FGS1_adc_gain'].loc[planet_id]\n offset = train_adc_info['FGS1_adc_offset'].loc[planet_id]\n signal = (signal / gain) + offset\n \n # Load FGS1 calibration files\n dark_fgs1 = pd.read_parquet(os.path.join(path_folder, f'train/{planet_id}/FGS1_calibration/dark.parquet')).values.astype(np.float64)\n dead_fgs1 = pd.read_parquet(os.path.join(path_folder, f'train/{planet_id}/FGS1_calibration/dead.parquet')).values.astype(bool)\n \n # Mask hot and dead pixels for FGS1\n hot_fgs1 = sigma_clip(dark_fgs1, sigma=5, maxiters=5).mask.astype(bool)\n hot_fgs1 = np.tile(hot_fgs1, (signal.shape[0], 1, 1))\n dead_fgs1 = np.tile(dead_fgs1, (signal.shape[0], 1, 1))\n FGS1_clean = np.ma.masked_where(dead_fgs1 | hot_fgs1, signal)\n \n return AIRS_CH0_clean, FGS1_clean","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:30.179695Z","iopub.execute_input":"2024-08-19T15:38:30.180075Z","iopub.status.idle":"2024-08-19T15:38:30.198218Z","shell.execute_reply.started":"2024-08-19T15:38:30.180047Z","shell.execute_reply":"2024-08-19T15:38:30.196799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set the base data path\npath_folder = Path('/kaggle/input/ariel-data-challenge-2024')\n\n# Define specific sub-directory paths\ntrain_path = path_folder / 'train'\ntrain_adc_info = pd.read_csv(path_folder / 'train_adc_info.csv')\ntrain_labels = pd.read_csv(path_folder / 'train_labels.csv')\naxis_info = pd.read_parquet(path_folder / 'axis_info.parquet')\nwavelength = pd.read_csv(path_folder / 'wavelengths.csv')","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:32.013222Z","iopub.execute_input":"2024-08-19T15:38:32.014202Z","iopub.status.idle":"2024-08-19T15:38:32.12194Z","shell.execute_reply.started":"2024-08-19T15:38:32.014133Z","shell.execute_reply":"2024-08-19T15:38:32.120589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_adc_info = pd.read_csv(path_folder / 'train_adc_info.csv')\ntrain_adc_info.set_index('planet_id', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:49.575473Z","iopub.execute_input":"2024-08-19T15:38:49.575896Z","iopub.status.idle":"2024-08-19T15:38:49.586339Z","shell.execute_reply.started":"2024-08-19T15:38:49.575864Z","shell.execute_reply":"2024-08-19T15:38:49.585178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_planet_id = train_adc_info.index[0] # Get a valid planet_id\nairs_ch0_cleaned, fgs1_cleaned = load_and_calibrate_signal(\n planet_id=sample_planet_id,\n train_adc_info=train_adc_info,\n axis_info=axis_info,\n path_folder=path_folder\n)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:51.227711Z","iopub.execute_input":"2024-08-19T15:38:51.228135Z","iopub.status.idle":"2024-08-19T15:38:58.548338Z","shell.execute_reply.started":"2024-08-19T15:38:51.228099Z","shell.execute_reply":"2024-08-19T15:38:58.547279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Output the shapes of the cleaned data\nprint(\"AIRS-CH0 Cleaned Data Shape:\", airs_ch0_cleaned.shape)\nprint(\"FGS1 Cleaned Data Shape:\", fgs1_cleaned.shape)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:58.550551Z","iopub.execute_input":"2024-08-19T15:38:58.551135Z","iopub.status.idle":"2024-08-19T15:38:58.558114Z","shell.execute_reply.started":"2024-08-19T15:38:58.551076Z","shell.execute_reply":"2024-08-19T15:38:58.556812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

5  3D EXPLORATORY DATA ANALYSIS     ⤒

\n\n
","metadata":{}},{"cell_type":"markdown","source":"

5.1 FGS1 Signal Data: The Detector Readouts

\n\n

What is FGS1 Signal Data?

\n\nThe FGS1 (Fine Guidance Sensor) signal data is a critical component of our analysis. It represents the signals captured by the FGS1 instrument onboard the satellite or space probe. This instrument plays a pivotal role in ensuring accurate pointing of the telescope by providing feedback on its alignment, but the signal data it generates can also be rich in scientific content.\n\n

Structure and Content of FGS1 Signal Data

\n\nThe FGS1 signal data is organized into a structured matrix, typically consisting of 135,000 time frames across 32 spatial dimensions. Each entry in this matrix represents the signal intensity captured by a specific sensor in the FGS1 array at a given time point. These signal intensities are essentially the digitized voltages that the sensors record as they detect incoming photons.\n\n

Data Characteristics

\n\n1. **Time Dimension (135,000 frames):** Represents the duration over which the signals were recorded. The high number of frames provides a detailed temporal resolution, which is crucial for understanding the dynamics of the observed object and the instrumental behavior over time.\n\n2. **Spatial Dimensions (32x32 array):** The FGS1 instrument’s sensor array is arranged in a 32x32 grid. Each point in this grid corresponds to a specific location on the detector, capturing photons that strike that particular sensor. This spatial dimension allows us to map how the signal varies across the sensor array, which is essential for understanding the uniformity of the detector’s response and identifying any anomalies.\n\n3. **Signal Intensity:** Each value in the data matrix represents the signal intensity at a specific time and spatial location. This intensity is influenced by the brightness of the observed object, the sensitivity of the detector, and any noise present in the system.\n\n

Why is FGS1 Signal Data Important?

\n\nUnderstanding the FGS1 signal data is crucial for several reasons:\n- **Alignment and Stability:** The FGS1 data is instrumental in ensuring the telescope’s alignment remains stable, which is vital for the accuracy of the observations. Any drift or misalignment can lead to blurred images or incorrect data interpretation.\n- **Detector Characterization:** By analyzing the signal data, we can characterize the detector’s performance. This includes identifying dead pixels, hot pixels, and areas of varying sensitivity, which are critical for data calibration and correction.\n- **Scientific Insights:** Beyond technical calibration, the FGS1 data can provide scientific insights, especially in high-precision astrometric measurements or when studying faint objects that require meticulous alignment.\n","metadata":{}},{"cell_type":"code","source":"# Reshape AIRS-CH0 cleaned data to a 2D grid (first frame)\nch0_image = airs_ch0_cleaned[0, :, :]\n\n# Reshape FGS1 cleaned data to a 2D grid (first frame)\nfgs1_image = fgs1_cleaned[0, :, :]","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:38:58.559838Z","iopub.execute_input":"2024-08-19T15:38:58.560325Z","iopub.status.idle":"2024-08-19T15:38:58.62903Z","shell.execute_reply.started":"2024-08-19T15:38:58.56029Z","shell.execute_reply":"2024-08-19T15:38:58.627875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting FGS1 Signal\nplt.figure(figsize=(10, 10))\nplt.imshow(fgs1_image, cmap='viridis')\nplt.colorbar()\nplt.title('FGS1 Cleaned Signal - First Frame')\nplt.xlabel('Spatial Dimension 1')\nplt.ylabel('Spatial Dimension 2')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:39:00.885614Z","iopub.execute_input":"2024-08-19T15:39:00.88605Z","iopub.status.idle":"2024-08-19T15:39:01.322292Z","shell.execute_reply.started":"2024-08-19T15:39:00.886019Z","shell.execute_reply":"2024-08-19T15:39:01.321088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.1.1 FGS1 Cleaned Signal - First Frame Visualization

\n\n

What is this graph?

\nThis 2D heatmap represents the intensity of the cleaned signal captured by the FGS1 instrument for the very first frame in the data. \n\n

Overview

\n\n- **Axes:**\n \n - **Spatial Dimension 1 (X-axis)** and **Spatial Dimension 2 (Y-axis)** represent the horizontal and vertical coordinates on the FGS1 detector.\n \n - **Color Map:** Ranges from dark purple (low signal) to bright yellow (high signal), showing where the signal is strongest on the detector.\n\n- **Signal Hotspot:** The bright yellow spot indicates a region of very high signal intensity, likely where the instrument is focused.\n\n- **Dead Pixels:** The white squares show dead pixels where no signal is recorded, masked out during calibration.\n\n

What does it mean?

\n\n- **Signal Focus:** The concentrated signal in the center indicates the FGS1 is functioning as expected.\n- **Detector Health:** Dead pixels are noted, which is typical and shows the calibration steps were effective.\n\nThis visualization confirms that the FGS1 instrument is capturing reliable data, crucial for accurate scientific analysis.\n","metadata":{}},{"cell_type":"code","source":"# 3D Visualization of FGS1 Cleaned Signal\nfig_fgs1 = go.Figure(data=[go.Surface(z=fgs1_image, colorscale='Viridis')])\nfig_fgs1.update_layout(\n title=\"FGS1 Cleaned Signal - 3D Visualization\",\n scene=dict(\n xaxis_title='Spatial Dimension 1 (Rows)',\n yaxis_title='Spatial Dimension 2 (Columns)',\n zaxis_title='Intensity'\n ),\n autosize=False,\n width=800,\n height=800\n)\nfig_fgs1.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:39:02.431485Z","iopub.execute_input":"2024-08-19T15:39:02.431927Z","iopub.status.idle":"2024-08-19T15:39:02.56854Z","shell.execute_reply.started":"2024-08-19T15:39:02.431893Z","shell.execute_reply":"2024-08-19T15:39:02.56697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.1.2 FGS1 Cleaned Signal - 3D Visualization

\n\nThis 3D visualization represents the cleaned FGS1 signal data, providing an insightful look into how the intensity varies across the detector's spatial dimensions. The two axes on the base of the plot correspond to the two spatial dimensions, labeled as \"Spatial Dimension 1 (Rows)\" and \"Spatial Dimension 2 (Columns).\" The height of the plot represents the signal intensity, with the color gradient giving an additional layer of detail.\n\n

What Does It Show?

\n\n- **Peak Intensity:** The sharp peak in the center of the plot indicates the region with the highest signal intensity. This suggests that a significant portion of the signal is concentrated in a small area of the detector, which could be an indication of the instrument focusing on a specific point in space.\n \n- **Spatial Distribution:** The spread of the signal intensity across the spatial dimensions shows that most of the energy is concentrated around the peak, with a rapid drop-off in intensity as you move away from this central point.\n\n

Why Is It Important?

\n\nUnderstanding the spatial distribution of the signal intensity is crucial for assessing the performance of the detector. If the signal is too concentrated, it might indicate a need for recalibration, as certain areas of the detector may be underutilized. Conversely, if the signal is too spread out, it could suggest issues with focusing or signal degradation.\n\nThis plot helps us visually inspect how the signal is distributed across the detector, allowing us to identify potential areas of improvement in the calibration process or the detector's design. By analyzing this distribution, we can ensure that the detector is functioning optimally, capturing as much relevant data as possible.","metadata":{}},{"cell_type":"code","source":"# FGS1 3D Surface Plot\nx_fgs = np.arange(fgs1_cleaned.shape[0]) # Time dimension\ny_fgs = np.arange(fgs1_cleaned.shape[1]) # Spatial dimension\nz_fgs = fgs1_cleaned[:, :, 0] # Intensity for the first spatial slice\n\n# Create the surface plot\nfig_fgs = go.Figure(data=[go.Surface(z=z_fgs, x=x_fgs, y=y_fgs, colorscale='Cividis')])\n\n# Update the layout\nfig_fgs.update_layout(\n title=\"FGS1 3D Signal Visualization\",\n scene=dict(\n xaxis_title='Time',\n yaxis_title='Spatial Dimension',\n zaxis_title='Intensity'\n ),\n autosize=False,\n width=900,\n height=700\n)\n\nfig_fgs.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:39:07.231246Z","iopub.execute_input":"2024-08-19T15:39:07.231676Z","iopub.status.idle":"2024-08-19T15:39:08.853845Z","shell.execute_reply.started":"2024-08-19T15:39:07.231643Z","shell.execute_reply":"2024-08-19T15:39:08.852616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.1.3 FGS1 3D Signal Visualization

\n\n

Overview

\n\nThis visualization captures the **3D signal intensity** as recorded by the **FGS1 instrument** over time and across spatial dimensions. The x-axis represents time, the y-axis represents the spatial dimension (specifically, the rows of the detector array), and the z-axis reflects the intensity of the signal.\n\n

What the Graph Shows

\n\n- **Spatial Dimension**: Represents one dimension of the FGS1 detector array. Each unit on this axis corresponds to different spatial rows in the detector where signals are recorded.\n \n- **Time**: Tracks the progression of signal measurements across different frames during the data capture process.\n\n- **Intensity**: Shows the magnitude of the signal captured by the FGS1 detector. Peaks indicate areas of high intensity, while valleys may indicate noise or low signal regions.\n\n

Analysis of This Graph

\n\nThis graph shows a highly variable signal intensity across both time and spatial dimensions:\n\n- **Peaks and Valleys**: The sharp peaks suggest strong signal detections, while the valleys could indicate either lower signal intensity or the presence of noise.\n \n- **Noise Detection**: The irregularities and potential negative values in the graph might suggest regions where noise is dominant, possibly requiring further investigation or recalibration.\n\n- **Spatial and Temporal Variability**: The distribution of peaks across both dimensions suggests that certain rows of the detector might be more sensitive to signal changes over time, while others may consistently show lower intensity.\n\n

Comparison with the Previous FGS1 3D Signal Visualization

\n\nThe previous FGS1 3D signal visualization focused on a **single frame**, examining the spatial distribution of signal intensity at a specific point in time. That visualization highlighted how intensity varied across the spatial grid of the detector, showing where the signal was strongest within a particular frame.\n\nIn contrast, this current visualization extends that analysis across **multiple time frames**:\n\n- **Temporal Dynamics**: By incorporating the time dimension, this graph offers a broader view of how the signal intensity evolves over time, rather than providing a snapshot at one point in time.\n\n- **Pattern Identification**: While the earlier graph could identify static spatial patterns within a single frame, this graph allows us to detect dynamic patterns, observing how intensity peaks and troughs move across the spatial grid as time progresses.\n\n- **Complexity**: This visualization is more complex as it integrates both spatial and temporal data, making it a powerful tool for understanding how signal intensity behaves not just in space, but also as time elapses.\n\n

Implications for Analysis

\n\nThe insights provided by this 3D visualization are critical for multiple reasons:\n\n1. **Temporal Noise Trends**: By observing intensity changes over time, we can better understand and possibly filter out noise that might be temporal rather than spatial.\n \n2. **Dynamic Signal Patterns**: Identifying how signal intensity evolves over time can help in understanding whether certain spatial regions consistently behave in specific ways, potentially revealing underlying trends or issues.\n\n3. **Detector Performance**: This analysis can also give clues about the performance of the FGS1 detector over time, highlighting any inconsistencies that might be tied to temporal factors rather than just spatial ones.\n\nIn summary, while the previous FGS1 graph was more focused on a single-time analysis, this one broadens the scope, making it a valuable tool for a comprehensive understanding of the FGS1 data in both space and time.","metadata":{}},{"cell_type":"markdown","source":"

5.2 AIRS-CH0 Signal Data: Exploring the Core of Spectral Readings

\n\n

What is AIRS-CH0 Signal Data?

\n\nThe AIRS-CH0 signal data is a fundamental element of our analysis, capturing the spectral information from the AIRS-CH0 (Ariel Infrared Spectrometer - Channel 0) instrument. This instrument is designed to measure infrared spectra, providing crucial data for studying the atmospheres of exoplanets. The signals recorded by AIRS-CH0 are essential for understanding the composition and characteristics of these distant worlds.\n\n

Structure and Content of AIRS-CH0 Signal Data

\n\nThe AIRS-CH0 signal data is organized into a highly structured matrix, with each matrix representing a specific wavelength range across various spatial dimensions. Each entry in this matrix corresponds to the signal intensity captured by the AIRS-CH0 detector at a particular time and wavelength.\n\n

Data Characteristics

\n\n1. **Time Dimension:** This dimension captures the duration over which the signals were recorded, providing temporal resolution that is vital for detecting changes in the observed spectra over time.\n\n2. **Spatial Dimensions (32x356 array):** The AIRS-CH0 detector is arranged in a 32x356 grid, with each point in this grid corresponding to a specific location on the detector. This setup allows for a detailed mapping of how the signal intensity varies across the detector’s surface, which is crucial for understanding the uniformity of the detector’s response and identifying any potential anomalies.\n\n3. **Wavelength Coverage:** The 356 wavelength channels covered by AIRS-CH0 span a significant portion of the infrared spectrum. Each channel provides information about the light intensity at a specific wavelength, enabling the detection of molecular absorption features in the spectra of exoplanet atmospheres.\n\n4. **Signal Intensity:** The signal intensity recorded in each matrix entry reflects the amount of infrared light detected at a specific time, spatial position, and wavelength. This intensity is influenced by the properties of the observed exoplanet’s atmosphere, the sensitivity of the detector, and any background noise.\n\n

Why is AIRS-CH0 Signal Data Important?

\n\nUnderstanding the AIRS-CH0 signal data is crucial for several reasons:\n- **Spectral Analysis:** The data allows scientists to analyze the infrared spectra of exoplanets, helping to identify molecular signatures such as water vapor, methane, and carbon dioxide.\n- **Detector Characterization:** By analyzing the signal data, we can characterize the performance of the AIRS-CH0 detector, including its sensitivity, noise levels, and any dead or hot pixels. This is critical for ensuring the accuracy and reliability of the data.\n- **Atmospheric Studies:** The AIRS-CH0 data is essential for studying the atmospheric composition and dynamics of exoplanets, contributing to our understanding of their potential habitability and the processes governing their climates.","metadata":{}},{"cell_type":"code","source":"# Plotting AIRS-CH0 Signal\nplt.figure(figsize=(20, 5))\nplt.imshow(ch0_image, cmap='viridis')\nplt.colorbar()\nplt.title('AIRS-CH0 Cleaned Signal - First Frame')\nplt.xlabel('Wavelength')\nplt.ylabel('Spatial Dimension')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.1 AIRS-CH0 Cleaned Signal - First Frame Visualization

\n\nThis 2D plot visualizes the **AIRS-CH0 Cleaned Signal** for the first frame, with **Spatial Dimension** on the vertical axis and **Wavelength** on the horizontal axis. The color scale represents **Signal Intensity**—from low (purple/blue) to high (yellow/green).\n\n

Key Insights:

\n\n- **Central Signal Concentration:** A prominent horizontal band suggests the main signal is concentrated around certain wavelengths and spatial positions.\n- **Noise and Artifacts:** Scattered white pixels indicate noise or possible dead pixels, highlighting areas that might need further calibration.\n- **Intensity Range:** The wide range of colors shows significant variability in signal strength across the spectrum.\n\n

Importance:

\n\nThis visualization is crucial for identifying detector performance, verifying data preprocessing, and understanding where signals are most and least intense. It helps ensure that the detector data is reliable for subsequent analysis.\n","metadata":{}},{"cell_type":"code","source":"# AIRS-CH0 3D Surface Plot\nx = np.arange(airs_ch0_cleaned.shape[0]) # Time dimension\ny = np.arange(airs_ch0_cleaned.shape[1]) # Spatial dimension\nz = airs_ch0_cleaned[:, :, 0] # Intensity for the first wavelength slice\n\n# Create the surface plot\nfig = go.Figure(data=[go.Surface(z=z, x=x, y=y, colorscale='Viridis')])\n\n# Update the layout\nfig.update_layout(\n title=\"AIRS-CH0 3D Signal Visualization\",\n scene=dict(\n xaxis_title='Time',\n yaxis_title='Spatial Dimension',\n zaxis_title='Intensity'\n ),\n autosize=False,\n width=900,\n height=700\n)\n\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.2 AIRS-CH0 3D Signal Visualization

\n\nThe AIRS-CH0 instrument is a critical component of our dataset, capturing spectral data across multiple wavelengths and spatial dimensions over time. In the context of this visualization, we're looking at a 3D surface plot that effectively summarizes the intensity of signals captured by the AIRS-CH0 instrument.\n\n

What Does the Graph Show?

\n\nThis 3D plot visualizes the relationship between **Time**, **Spatial Dimension**, and **Signal Intensity**. The spatial dimension on the x-axis represents the array of pixels along one of the detector's dimensions. The y-axis denotes time, which corresponds to different observational frames captured over the observation period. The z-axis (color-coded) reflects the intensity of the signal captured by the instrument.\n\n

Interpretation of the Graph

\n\nThe towering peaks represent moments when the AIRS-CH0 detector recorded higher signal intensities at specific times and spatial locations. These peaks can indicate areas of interest, such as the detection of a celestial body or some other high-energy event. The uniform pattern suggests consistent data collection across the spatial dimension, with no glaring anomalies or significant noise. \n\nThe graph is instrumental in identifying how the signal fluctuates over time and across different spatial regions, which is critical for understanding the stability of the detector and the consistency of the recorded data. Such insights are invaluable for subsequent data processing steps, where we might need to apply corrections or filters based on the observed patterns.\n\nThis visualization is an essential tool for diagnosing the behavior of the detector and understanding how the instrument's data might be influenced by various factors over the course of the observation.","metadata":{}},{"cell_type":"code","source":"# 3D Scatter Plot for AIRS-CH0\nfig_scatter = go.Figure()\n\n# Sample down the data to avoid overplotting\nx_sampled = np.arange(0, airs_ch0_cleaned.shape[0], 100) # Time samples\ny_sampled = np.arange(0, airs_ch0_cleaned.shape[1], 1) # Spatial dimension samples\n\n# Add scatter points\nfor i in y_sampled:\n fig_scatter.add_trace(\n go.Scatter3d(\n x=x_sampled,\n y=[i] * len(x_sampled),\n z=airs_ch0_cleaned[x_sampled, i, 0], # Intensity at the first wavelength\n mode='markers',\n marker=dict(size=2, color=airs_ch0_cleaned[x_sampled, i, 0], colorscale='Viridis')\n )\n )\n\n# Update the layout\nfig_scatter.update_layout(\n title=\"3D Scatter Plot of AIRS-CH0 Data\",\n scene=dict(\n xaxis_title='Time',\n yaxis_title='Spatial Dimension',\n zaxis_title='Intensity'\n ),\n autosize=False,\n width=900,\n height=700\n)\n\nfig_scatter.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.3 AIRS-CH0 Signal Data: 3D Scatter Plot Analysis

\n\n

Overview of the Plot

\n\nThe 3D scatter plot depicted above represents the AIRS-CH0 signal data, showcasing the intricate relationships between spatial dimension, time, and intensity. Each dot in the scatter plot corresponds to a specific measurement of signal intensity at a given spatial dimension and time point.\n\n

Key Observations:

\n\n1. **Spatial Dimension (X-axis):** The spatial dimension on the x-axis indicates the various rows or columns of the AIRS-CH0 detector. It helps us understand how the signal intensity is distributed across different parts of the detector.\n\n2. **Time (Z-axis):** The time axis reveals the progression of signal capture over time. Observing this axis allows us to identify temporal patterns or anomalies that might affect the signal intensity.\n\n3. **Intensity (Y-axis):** The intensity axis displays the magnitude of the signal captured by the detector. Higher points indicate higher signal intensity, which could correlate with brighter objects or specific events captured by the detector.\n\n

Insights and Importance:

\n\n- **Uniformity in Signal Capture:** The scatter plot allows us to observe if the signal intensity is uniformly distributed across spatial dimensions and time. Any deviations or patterns could indicate potential calibration issues or specific phenomena of interest.\n \n- **Anomalies and Noise:** By examining outliers or clusters in the plot, we can identify anomalies in the data, such as unexpected spikes or dips in intensity. These could be due to noise, cosmic rays, or other instrumental effects that require further investigation.\n\n- **Comparative Analysis:** When analyzed alongside other visualizations, this scatter plot provides a comprehensive view of how the AIRS-CH0 data behaves over time and across different spatial dimensions, offering valuable insights for both calibration and scientific analysis.\n","metadata":{}},{"cell_type":"code","source":"# Define variables\nsignal_data = airs_ch0_cleaned # Cleaned AIRS-CH0 data\ntime_steps = np.arange(signal_data.shape[0]) # Time/Frame axis\nwavelengths = np.linspace(0, 282, signal_data.shape[2]) # Assuming 282 wavelengths\n\n# Create 3D Surface Plot\nfig = go.Figure(data=[go.Surface(z=signal_data[:, 0, :], x=time_steps, y=wavelengths, colorscale='Viridis')])\n\n# Update layout for better aesthetics\nfig.update_layout(\n title=\"Signal Intensity Across Wavelengths\",\n scene=dict(\n xaxis_title='Time/Frames',\n yaxis_title='Wavelengths',\n zaxis_title='Signal Intensity'\n ),\n autosize=False,\n width=1000,\n height=800\n)\n\n# Show the figure\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.4 Signal Intensity Across Wavelengths

\n\n

Graph Explanation

\n\nThe graph represents a 3D surface plot visualizing the **Signal Intensity** across different **Wavelengths** and **Time Frames**. Here’s a breakdown of the axes:\n- **X-Axis (Wavelengths):** This axis represents the range of wavelengths being observed, offering a spectrum of data.\n- **Y-Axis (Time Frames):** Time frames indicate the temporal evolution of the observation, with each frame representing a specific point in time during the data acquisition process.\n- **Z-Axis (Signal Intensity):** The vertical axis (z-axis) illustrates the intensity of the signal detected at each combination of wavelength and time frame.\n\n

Key Insights

\n\n1. **Signal Distribution**: The graph clearly shows how signal intensity fluctuates across different wavelengths and time frames. Peaks in the graph signify areas of higher signal intensity, indicating regions of interest where the observed object might be emitting strongly or where the instrument is particularly sensitive.\n\n2. **Temporal Dynamics**: Observing the y-axis (time frames) provides insight into how the signal changes over time. Consistent patterns along this axis might indicate stable observations, while any abrupt changes or trends could suggest temporal variations in the observed object or potential instrumental artifacts.\n\n3. **Wavelength Sensitivity**: The variation in signal intensity across different wavelengths can help in understanding the spectral characteristics of the observed object. Certain wavelengths may show higher intensities, pointing towards specific emission or absorption features in the object's spectrum.\n\n4. **Potential Anomalies**: The graph might also reveal anomalies such as unexpected drops or spikes in signal intensity. Identifying these can be crucial for further data analysis and could indicate either astrophysical phenomena or instrumental issues that need to be addressed.\n","metadata":{}},{"cell_type":"code","source":"# Prepare data: Aggregate the signal intensity over time by taking the mean\nmean_signal_intensity = np.mean(airs_ch0_cleaned, axis=0)\n\n# Create 3D surface plot\nfig = go.Figure(data=[go.Surface(z=mean_signal_intensity, colorscale='Viridis')])\n\n# Update layout\nfig.update_layout(\n title=\"Signal Intensity Across Wavelengths\",\n scene=dict(\n xaxis_title='Wavelength Index',\n yaxis_title='Spatial Dimension',\n zaxis_title='Mean Signal Intensity'\n ),\n autosize=False,\n width=1200,\n height=800\n)\n\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.5 Signal Intensity Across Wavelengths - 3D Surface Plot Analysis

\n\n

Overview

\nThis 3D surface plot provides a comprehensive visualization of how signal intensity varies across different wavelengths and spatial dimensions. By viewing the graph from multiple angles, we can gain deeper insights into the distribution and behavior of the signal across the entire dataset.\n\n

Insights

\n\n- **Intensity Distribution**: \n \n - The plot reveals that the majority of the signal intensity is concentrated at lower wavelength indices, with a sharp peak observed at specific spatial dimensions. This indicates that certain wavelengths are particularly responsive, capturing more intense signals compared to others.\n - The rapid decline in intensity as the wavelength index increases suggests that beyond a certain wavelength, the instrument becomes less sensitive, or the observed phenomena are less prominent.\n\n- **Spatial Dimension Patterns**: The signal intensity also shows a significant dependency on the spatial dimension. The intensity is not uniformly distributed; rather, it shows a strong peak at certain spatial coordinates. This may imply that the instrument has areas with higher sensitivity or that certain regions of the detector are more exposed to the signal source.\n\n- **Anomalies and Gradients**: The plot also highlights some subtle gradients and potential anomalies. For example, the gradual slope in the intensity after the initial peak might suggest a fading signal, while the occasional spikes could indicate noise or transient events captured by the instrument.\n\n- **3D Perspective Advantage**: By viewing the plot from different angles, we can better understand the interplay between wavelength, spatial dimension, and intensity. This multidimensional view is crucial for identifying patterns that might be overlooked in 2D representations, such as how certain wavelengths are consistently more intense across different spatial dimensions.\n","metadata":{}},{"cell_type":"code","source":"# Select specific spatial locations for analysis (e.g., 10, 15, 20)\nselected_spatial_dims = [10, 15, 20]\n\n# Extract the signal data for the selected spatial dimensions\nsignal_over_time = airs_ch0_cleaned[:, selected_spatial_dims, :]\n\n# Create the 3D surface plot\nfig = go.Figure()\n\nfor i, spatial_dim in enumerate(selected_spatial_dims):\n fig.add_trace(go.Surface(\n z=signal_over_time[:, i, :], # Signal intensity over time for this spatial dimension\n x=np.arange(signal_over_time.shape[0]), # Time axis\n y=np.arange(signal_over_time.shape[2]), # Wavelength index axis\n colorscale='Viridis',\n name=f'Spatial Dim {spatial_dim}',\n showscale=False\n ))\n\n# Update layout for better visualization\nfig.update_layout(\n title=\"Signal Intensity Over Time for Selected Spatial Dimensions\",\n scene=dict(\n xaxis_title='Time Frames',\n yaxis_title='Wavelength Index',\n zaxis_title='Signal Intensity'\n ),\n width=900,\n height=700\n)\n\n# Show the 3D surface plot\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.6 Signal Intensity Over Time for Selected Spatial Dimensions

\n\nThe 3D plots displayed provide a comprehensive visualization of how signal intensity varies across different wavelengths and time frames for selected spatial dimensions. These visualizations are essential for understanding the temporal and spectral behavior of the observed signals.\n\n

Insights from the Visualization:

\n\n- **Signal Intensity Peaks**: The plots reveal significant intensity peaks that appear consistent across various wavelengths and time frames. These peaks may indicate specific events or phenomena captured by the detector, highlighting areas of interest for further analysis.\n \n- **Temporal Evolution**: The signal intensity tends to decrease over time, as shown by the slope in the 3D plots. This decline could be due to the natural fading of the observed object, changes in the detector's sensitivity, or other factors affecting the signal's strength.\n\n- **Spectral Behavior**: The variation in intensity across different wavelengths provides insights into the spectral characteristics of the observed signals. The consistency of these variations across time frames suggests that certain wavelengths may be more sensitive or relevant to the observed phenomena.\n\n- **Spatial Dimension Distribution**: The plots also show how signal intensity is distributed across the spatial dimensions, with some regions consistently recording higher intensities. This could indicate areas of the detector with higher sensitivity or specific spatial regions of interest in the observed object.","metadata":{}},{"cell_type":"code","source":"# Flatten the 3D data into 2D arrays for the scatter plot\nx_data = np.repeat(np.arange(airs_ch0_cleaned.shape[1]), airs_ch0_cleaned.shape[2]) # Spatial dimension\ny_data = np.tile(np.arange(airs_ch0_cleaned.shape[2]), airs_ch0_cleaned.shape[1]) # Wavelength index\nz_data = airs_ch0_cleaned.mean(axis=0).flatten() # Mean signal intensity across time\n\n# Create the 3D scatter plot\nfig = go.Figure(data=[go.Scatter3d(\n x=x_data,\n y=y_data,\n z=z_data,\n mode='markers',\n marker=dict(\n size=3,\n color=z_data, # Set color to the signal intensity\n colorscale='Viridis',\n opacity=0.8\n )\n)])\n\n# Update the layout for better visualization\nfig.update_layout(\n title=\"Signal Intensity Distribution by Spatial Dimension\",\n scene=dict(\n xaxis_title='Spatial Dimension',\n yaxis_title='Wavelength Index',\n zaxis_title='Mean Signal Intensity'\n ),\n width=900,\n height=700\n)\n\n# Show the 3D scatter plot\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.7 Signal Intensity Distribution by Spatial Dimension

","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\n\n# Flatten data for PCA\nflattened_data = airs_ch0_cleaned.reshape(airs_ch0_cleaned.shape[0], -1)\n\n# Apply PCA\npca = PCA(n_components=3)\npca_result = pca.fit_transform(flattened_data)\n\n# Plot the PCA results\nfig = px.scatter_3d(\n x=pca_result[:, 0], \n y=pca_result[:, 1], \n z=pca_result[:, 2], \n color=pca_result[:, 2], \n title='3D PCA of Signal Intensity'\n)\nfig.update_layout(scene=dict(\n xaxis_title='PCA 1',\n yaxis_title='PCA 2',\n zaxis_title='PCA 3'\n))\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.8 3D PCA of Signal Intensity: Dimensionality Reduction

\n\nThe Principal Component Analysis (PCA) is a powerful technique used to reduce the dimensionality of large datasets while preserving as much variance as possible. This 3D visualization represents the PCA applied to the signal intensity data, providing a condensed view of the most significant patterns within the data.\n\n

Breakdown of the Graph

\n\n- **Axes Representation:**\n - **PCA 1, PCA 2, PCA 3:** These axes represent the first three principal components, which are linear combinations of the original variables (signal intensities across different dimensions). They capture the directions of maximum variance in the data.\n - **Color Scale:** The color of the points varies from purple (low signal intensity) to yellow (high signal intensity), helping to visualize the distribution of signal intensities within the PCA space.\n\n- **Clustering and Distribution:**\n - The points in the scatter plot are grouped in certain regions, indicating clusters or patterns in the data that are captured by the PCA. These clusters might correspond to different signal behaviors or characteristics across the various dimensions.\n - The small, isolated group of points (bottom left in some views) could represent outliers or distinct signal events that differ significantly from the rest of the data.\n\n

Insights and Interpretations

\n\n1. **Dimensionality Reduction:** By reducing the dimensionality from the original signal data to just three principal components, we can more easily identify patterns, trends, and anomalies. This reduction helps in simplifying the complexity of the dataset while retaining the most important information.\n\n2. **Signal Variance Capture:**\n - The spread of the points along the PCA axes indicates how much variance each principal component captures. The wider the spread, the more variance is explained by that component.\n - The color gradient provides an additional layer of information, showing how signal intensity varies across different principal components.\n\n3. **Potential Anomalies:** The distinct group of points separated from the main cluster suggests there might be anomalous or unique signal behaviors worth investigating further. These could be due to specific events, instrumental noise, or other factors that cause a deviation from the norm.\n\n4. **Visualization for Model Input:** This PCA visualization is crucial for understanding how to feed the most relevant features into predictive models. By focusing on the principal components that capture the most variance, we can build more efficient and accurate models.\n","metadata":{}},{"cell_type":"markdown","source":"

5.3 Light Curve Analysis

\n\nA light curve is a graph that shows the brightness of a celestial object over time. When an exoplanet passes in front of its host star (a phenomenon known as a **transit**), it causes a temporary dip in the star's brightness. This dip is recorded as a light curve and is crucial in the study of exoplanets.\n\nThe depth and duration of the dip can provide critical information about the exoplanet, such as its size, orbital period, and even atmospheric properties. In a typical light curve, the **flux** (brightness) of the star decreases as the planet transits and increases back to its original level once the transit is complete.\n\n

Measuring Exoplanet Properties

\nFrom the image below, we see that the transit of an exoplanet allows astronomers to measure the radius of the planet by analyzing the dip in the light curve. The light that passes through the planet's atmosphere during the transit can be studied to understand the composition and characteristics of the atmosphere.\n\nDifferent molecules in the atmosphere absorb different wavelengths of light. This absorption results in a characteristic fingerprint on the light curve, known as the **transit depth**. The deeper the transit depth, the more significant the absorption by atmospheric molecules, which provides insights into the planet's atmospheric composition, including the presence of gases like water vapor, methane, and carbon dioxide.\n
\n
\n \"Book\n
\n

The Importance of Light Curves

\n\nThe light curves from FGS1 and AIRS-CH0 are not just mere graphs; they are windows into distant worlds. They allow us to:\n\n- **Detect Exoplanets**: The dips in the light curves are the first indicators of a planet orbiting a star.\n\n- **Measure Planetary Sizes**: The depth of the dip is related to the size of the planet. A larger dip corresponds to a larger planet.\n\n- **Study Atmospheres**: Variations in the light curve at different wavelengths can reveal the composition of a planet's atmosphere, providing clues about its climate and potential for supporting life.\n
","metadata":{}},{"cell_type":"code","source":"# Assuming `fgs1_cleaned` is the 3D array (time, spatial_dim1, spatial_dim2)\nfgs1_signal = np.mean(fgs1_cleaned, axis=(1, 2)) # Mean across spatial dimensions to flatten to 1D\n\n# No need to calculate mean_signal again since fgs1_signal is already the mean\nnet_signal = fgs1_signal[1::2] - fgs1_signal[::2]\ncum_signal = np.cumsum(net_signal)\nwindow = 800\nsmooth_signal = (cum_signal[window:] - cum_signal[:-window]) / window\n\n# Create subplots\nfig = make_subplots(rows=2, cols=1, shared_xaxes=True, vertical_spacing=0.1)\n\n# Raw net signal plot\nfig.add_trace(go.Scatter(x=np.arange(len(net_signal)), y=net_signal,\n mode='lines', name='Raw Net Signal',\n line=dict(color='firebrick')), row=1, col=1)\n\n# Smoothed net signal plot\nfig.add_trace(go.Scatter(x=np.arange(len(smooth_signal)), y=smooth_signal,\n mode='lines', name='Smoothed Net Signal',\n line=dict(color='firebrick')), row=2, col=1)\n\n# Update layout\nfig.update_layout(height=700, width=1000, \n title_text=\"FGS1 Light Curve\",\n title_x=0.5)\n\n# Update axis labels\nfig.update_xaxes(title_text=\"Time\", row=2, col=1)\nfig.update_yaxes(title_text=\"Net Signal\", row=1, col=1)\nfig.update_yaxes(title_text=\"Smoothed Signal\", row=2, col=1)\n\n# Show the figure\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:44:38.578385Z","iopub.execute_input":"2024-08-19T15:44:38.578928Z","iopub.status.idle":"2024-08-19T15:44:40.69073Z","shell.execute_reply.started":"2024-08-19T15:44:38.578872Z","shell.execute_reply":"2024-08-19T15:44:40.687268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.3.1 FGS1 Light Curve

\n\n- **Net Signal**: The FGS1 light curve displays variations in the net signal, with numerous small fluctuations that could be attributed to both noise and genuine astrophysical phenomena. The spikes in the net signal might correspond to transient events, possibly caused by instrumental effects or cosmic rays.\n\n- **Smoothed Signal**: The smoothed signal curve offers a clearer view by averaging out the noise. In this graph, we observe a consistent pattern with peaks and troughs, which may suggest periodic events, such as the regular transit of an exoplanet. The light curve's analysis can reveal not only the presence of an exoplanet but also details about its orbit and atmospheric properties.\n","metadata":{}},{"cell_type":"code","source":"# Assuming `airs_ch0_cleaned` is the 3D array (time, spatial_dim1, spatial_dim2)\nairs_ch0_signal = np.mean(airs_ch0_cleaned, axis=(1, 2)) # Mean across spatial dimensions to flatten to 2D\n\n# Calculate the signals\nmean_signal = np.mean(airs_ch0_signal)\nnet_signal = airs_ch0_signal[1::2] - airs_ch0_signal[::2]\ncum_signal = np.cumsum(net_signal)\nwindow = 100\nsmooth_signal = (cum_signal[window:] - cum_signal[:-window]) / window\n\n# Create subplots\nfig = make_subplots(rows=2, cols=1, shared_xaxes=True, vertical_spacing=0.1)\n\n# Raw net signal plot\nfig.add_trace(go.Scatter(x=np.arange(len(net_signal)), y=net_signal,\n mode='lines', name='Raw Net Signal',\n line=dict(color='firebrick')), row=1, col=1)\n\n# Smoothed net signal plot\nfig.add_trace(go.Scatter(x=np.arange(len(smooth_signal)), y=smooth_signal,\n mode='lines', name='Smoothed Net Signal',\n line=dict(color='firebrick')), row=2, col=1)\n\n# Update layout\nfig.update_layout(height=700, width=1000, \n title_text=\"AIRS-CH0 Light Curve\",\n title_x=0.5)\n\n# Update axis labels\nfig.update_xaxes(title_text=\"Time\", row=2, col=1)\nfig.update_yaxes(title_text=\"Net Signal\", row=1, col=1)\nfig.update_yaxes(title_text=\"Smoothed Signal\", row=2, col=1)\n\n# Show the figure\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T15:40:50.354444Z","iopub.execute_input":"2024-08-19T15:40:50.354838Z","iopub.status.idle":"2024-08-19T15:40:51.920309Z","shell.execute_reply.started":"2024-08-19T15:40:50.354806Z","shell.execute_reply":"2024-08-19T15:40:51.919108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.3.2 AIRS-CH0 Light Curve

\n\n- **Net Signal**: The AIRS-CH0 light curve appears to have a more uniform net signal compared to FGS1, but with a distinct downward trend, possibly indicating a more prolonged and significant change in brightness. This could be indicative of a longer-duration transit or a more substantial celestial event.\n\n- **Smoothed Signal**: The smoothed signal for AIRS-CH0 reveals a pronounced dip, which is a strong indicator of a transit. This dip, followed by a gradual increase, suggests that the planet passed in front of its star, leading to a decrease in observed light. Such observations are essential for confirming the existence of exoplanets and studying their atmospheres.\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

6  MACHINE LEARNING    ⤒

","metadata":{}},{"cell_type":"markdown","source":"
\n ⚠️ WARNING ⚠️

THIS IS A WORK IN PROGRESS
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

7  CHANGE LOG    ⤒

","metadata":{}},{"cell_type":"markdown","source":"\n\n\n\n
","metadata":{}}]}