{
  "id": 543676,
  "title": "34th place solution",
  "url": "/competitions/ariel-data-challenge-2024/discussion/543676",
  "author_name": "2g",
  "post_date": "2024-11-01T00:35:23.316000",
  "votes": 15,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>Preparing the Transit Signal</h1>\n<p>(Based on <a href=\"https://www.kaggle.com/code/pourchot/ariel-data-challenge-2024\" target=\"_blank\">this notebook</a>)</p>\n<ul>\n<li>Applied a lowpass filter instead of binning</li>\n<li>Downsampled to 232 points</li>\n<li>Computed a moving average across wavelengths with a window size of 31</li>\n<li>Extracted and flipped the transit signal before optimization<br>\n<a href=\"https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/538139#3027343\" target=\"_blank\">ref. 1</a>, <a href=\"https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/529412#2965289\" target=\"_blank\">ref. 2</a></li>\n<li>Use joblib Parallel, delayed</li>\n</ul>\n<p>Applied a lowpass filter instead of binning, Downsampled to 232 points</p>\n<pre><code> scipy.signal  sci_signal\n\n ():\n    nyquist =  * fs\n    normal_cutoff = cutoff / nyquist\n    b, a = sci_signal.butter(order, normal_cutoff, btype=, analog=)\n     b, a\n\n ():\n    b, a = butter_lowpass(cutoff, fs, order=order)\n    y = sci_signal.filtfilt(b, a, data)\n     y\n\n ():\n\n    cut = ( / interval)\n    signal = signal[::interval, :]\n    signal = signal[cut:-cut, :]\n     signal\n</code></pre>\n<pre><code>     i  (binned.shape[]):\n        binned[:, i] = apply_lowpass_filter(binned[:, i],  , )\n\n    \n    binned = downsampling(binned, config.interval)\n</code></pre>\n<p>Computed a moving average across wavelengths with a window size of 31</p>\n<pre><code> ():\n     np.convolve(arr, np.ones(window_size)/window_size, mode=)\n\nwindow_size = \n\n i  tqdm((pre_train.shape[])):\n    pre_train[i, :, :] = np.apply_along_axis(moving_average, axis=, arr=pre_train[i, :, :], window_size=window_size)\n</code></pre>\n<p>Extracted and flipped the transit signal before optimization</p>\n<pre><code>pre_train = np.concatenate([pre_train[:, :, []], np.flip(pre_train[:, :, :], axis=)], axis=)\n</code></pre>\n<h1>Optimization</h1>\n<p>The optimization process estimates (R/R)^2, the baseline (representing planet radius), and the concentration of each gas molecule as follows:</p>\n<ul>\n<li>Set initial values for the baseline and concentrations of each molecule</li>\n<li>Construct the complete signal by summing the baseline and the individual signals for each gas molecule (each molecule’s signal was collected from ExoMol data)</li>\n<li>Calculate RMSE by comparing the depth of the generated signal to the observed transit signal across all wavelengths</li>\n<li>Use joblib Parallel, delayed</li>\n</ul>\n<pre><code> ():\n    \n    result = np.repeat(np.array([s_list[]], dtype=defalut_dtype), spectrum_array.shape[])\n     i, p  (s_list[:]):\n        result += (spectrum_array[i, :]**) * p\n     result\n\n\n ():\n\n    best_q = \n\n     i  () :\n        delta = ( / config.interval)\n\n        x = np.arange(signal.shape[], dtype=defalut_dtype)\n        y = signal.copy()\n\n        y = np.concatenate([y[:p1-delta], y[p1+delta:p2-delta]* ( + s), y[p2+delta:]])\n        x = np.concatenate([x[:p1-delta], x[p1+delta:p2-delta], x[p2+delta:]])\n\n        z = np.polyfit(x, y, deg=i)\n        p = np.poly1d(z)\n        q = np.mean((p(x) - y)**)\n\n         q &lt; best_q :\n            best_q = q\n\n     best_q\n\n ():\n\n    spectrum = make_signal(s_list)\n    mae = np.mean([objective_each_signal(spectrum[wl], signals[:, wl], p1, p2)  wl  (, signals.shape[], )])\n     mae\n\n ():\n\n    p1, p2 = phase_dict[i]\n\n      \n    \n    initial_guess = [, , , ,\n                    , , , ,\n                    , ]\n\n    result = minimize(\n        objective, \n        initial_guess, \n        args=(pre_train[i, :, :], p1, p2), \n        method=, \n        bounds=[(, )] * ((initial_guess))\n    )\n\n    spectrum = make_signal(result.x)\n\n     spectrum\n</code></pre>\n<h1>Post-Processing</h1>\n<p>For <code>wl_1</code>, compute a weighted mean of the predicted <code>wl_1</code> value and the mean of the other wavelengths.</p>\n<pre><code>pred_mean[:, ] = np.ones_like(pred_mean[:, ]) * (pred_mean[:, ] *  + pred_mean[:, :].mean(axis=) * )\n</code></pre>\n<h1>Determining Sigma</h1>\n<ul>\n<li>Calculate sigma based on the RMSE for each transit signal and the mean transit signal across each wavelength.</li>\n<li>Clipping to 1 * sigma</li>\n</ul>\n<pre><code> ():\n    rmse_list = []\n     wl  (pred.shape[]):\n        rmse_list.append(mean_squared_error(gt[:, wl], pred[:, wl], squared=))\n     rmse_list\n\nsigma_values = np.array(calc_rmse_per_wl(train_labels.values, pred_mean))\nconst = \nsigma_const = \nsigma_values = sigma_values * const\n\nsigma = np.std(sigma_values)\nmean = np.mean(sigma_values)\n\nsigma_values_clipped = sigma_values.copy()\nsigma_values_clipped = sigma_values_clipped.clip(-sigma_const*sigma+mean,sigma_const*sigma+mean)\n\npred_sigma_wl = np.repeat(sigma_values_clipped.reshape(, -), (adc_info), axis=)\n</code></pre>",
  "messages": [
    {
      "id": 3033261,
      "postDate": "2024-11-01T00:35:23.317Z",
      "content": "<h1>Preparing the Transit Signal</h1>\n<p>(Based on <a href=\"https://www.kaggle.com/code/pourchot/ariel-data-challenge-2024\" target=\"_blank\">this notebook</a>)</p>\n<ul>\n<li>Applied a lowpass filter instead of binning</li>\n<li>Downsampled to 232 points</li>\n<li>Computed a moving average across wavelengths with a window size of 31</li>\n<li>Extracted and flipped the transit signal before optimization<br>\n<a href=\"https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/538139#3027343\" target=\"_blank\">ref. 1</a>, <a href=\"https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/529412#2965289\" target=\"_blank\">ref. 2</a></li>\n<li>Use joblib Parallel, delayed</li>\n</ul>\n<p>Applied a lowpass filter instead of binning, Downsampled to 232 points</p>\n<pre><code> scipy.signal  sci_signal\n\n ():\n    nyquist =  * fs\n    normal_cutoff = cutoff / nyquist\n    b, a = sci_signal.butter(order, normal_cutoff, btype=, analog=)\n     b, a\n\n ():\n    b, a = butter_lowpass(cutoff, fs, order=order)\n    y = sci_signal.filtfilt(b, a, data)\n     y\n\n ():\n\n    cut = ( / interval)\n    signal = signal[::interval, :]\n    signal = signal[cut:-cut, :]\n     signal\n</code></pre>\n<pre><code>     i  (binned.shape[]):\n        binned[:, i] = apply_lowpass_filter(binned[:, i],  , )\n\n    \n    binned = downsampling(binned, config.interval)\n</code></pre>\n<p>Computed a moving average across wavelengths with a window size of 31</p>\n<pre><code> ():\n     np.convolve(arr, np.ones(window_size)/window_size, mode=)\n\nwindow_size = \n\n i  tqdm((pre_train.shape[])):\n    pre_train[i, :, :] = np.apply_along_axis(moving_average, axis=, arr=pre_train[i, :, :], window_size=window_size)\n</code></pre>\n<p>Extracted and flipped the transit signal before optimization</p>\n<pre><code>pre_train = np.concatenate([pre_train[:, :, []], np.flip(pre_train[:, :, :], axis=)], axis=)\n</code></pre>\n<h1>Optimization</h1>\n<p>The optimization process estimates (R/R)^2, the baseline (representing planet radius), and the concentration of each gas molecule as follows:</p>\n<ul>\n<li>Set initial values for the baseline and concentrations of each molecule</li>\n<li>Construct the complete signal by summing the baseline and the individual signals for each gas molecule (each molecule’s signal was collected from ExoMol data)</li>\n<li>Calculate RMSE by comparing the depth of the generated signal to the observed transit signal across all wavelengths</li>\n<li>Use joblib Parallel, delayed</li>\n</ul>\n<pre><code> ():\n    \n    result = np.repeat(np.array([s_list[]], dtype=defalut_dtype), spectrum_array.shape[])\n     i, p  (s_list[:]):\n        result += (spectrum_array[i, :]**) * p\n     result\n\n\n ():\n\n    best_q = \n\n     i  () :\n        delta = ( / config.interval)\n\n        x = np.arange(signal.shape[], dtype=defalut_dtype)\n        y = signal.copy()\n\n        y = np.concatenate([y[:p1-delta], y[p1+delta:p2-delta]* ( + s), y[p2+delta:]])\n        x = np.concatenate([x[:p1-delta], x[p1+delta:p2-delta], x[p2+delta:]])\n\n        z = np.polyfit(x, y, deg=i)\n        p = np.poly1d(z)\n        q = np.mean((p(x) - y)**)\n\n         q &lt; best_q :\n            best_q = q\n\n     best_q\n\n ():\n\n    spectrum = make_signal(s_list)\n    mae = np.mean([objective_each_signal(spectrum[wl], signals[:, wl], p1, p2)  wl  (, signals.shape[], )])\n     mae\n\n ():\n\n    p1, p2 = phase_dict[i]\n\n      \n    \n    initial_guess = [, , , ,\n                    , , , ,\n                    , ]\n\n    result = minimize(\n        objective, \n        initial_guess, \n        args=(pre_train[i, :, :], p1, p2), \n        method=, \n        bounds=[(, )] * ((initial_guess))\n    )\n\n    spectrum = make_signal(result.x)\n\n     spectrum\n</code></pre>\n<h1>Post-Processing</h1>\n<p>For <code>wl_1</code>, compute a weighted mean of the predicted <code>wl_1</code> value and the mean of the other wavelengths.</p>\n<pre><code>pred_mean[:, ] = np.ones_like(pred_mean[:, ]) * (pred_mean[:, ] *  + pred_mean[:, :].mean(axis=) * )\n</code></pre>\n<h1>Determining Sigma</h1>\n<ul>\n<li>Calculate sigma based on the RMSE for each transit signal and the mean transit signal across each wavelength.</li>\n<li>Clipping to 1 * sigma</li>\n</ul>\n<pre><code> ():\n    rmse_list = []\n     wl  (pred.shape[]):\n        rmse_list.append(mean_squared_error(gt[:, wl], pred[:, wl], squared=))\n     rmse_list\n\nsigma_values = np.array(calc_rmse_per_wl(train_labels.values, pred_mean))\nconst = \nsigma_const = \nsigma_values = sigma_values * const\n\nsigma = np.std(sigma_values)\nmean = np.mean(sigma_values)\n\nsigma_values_clipped = sigma_values.copy()\nsigma_values_clipped = sigma_values_clipped.clip(-sigma_const*sigma+mean,sigma_const*sigma+mean)\n\npred_sigma_wl = np.repeat(sigma_values_clipped.reshape(, -), (adc_info), axis=)\n</code></pre>",
      "rawMarkdown": "# Preparing the Transit Signal\n\n(Based on [this notebook](https://www.kaggle.com/code/pourchot/ariel-data-challenge-2024))\n\n- Applied a lowpass filter instead of binning\n- Downsampled to 232 points\n- Computed a moving average across wavelengths with a window size of 31\n- Extracted and flipped the transit signal before optimization\n[ref. 1](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/538139#3027343), [ref. 2](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/529412#2965289)\n- Use joblib Parallel, delayed\n\nApplied a lowpass filter instead of binning, Downsampled to 232 points\n\n```python\nimport scipy.signal as sci_signal\n\ndef butter_lowpass(cutoff, fs, order=5):\n    nyquist = 0.5 * fs\n    normal_cutoff = cutoff / nyquist\n    b, a = sci_signal.butter(order, normal_cutoff, btype='low', analog=False)\n    return b, a\n\ndef apply_lowpass_filter(data, cutoff, fs, order=5):\n    b, a = butter_lowpass(cutoff, fs, order=order)\n    y = sci_signal.filtfilt(b, a, data)\n    return y\n\ndef downsampling(signal, interval):\n\n    cut = int(500 / interval)\n    signal = signal[::interval, :]\n    signal = signal[cut:-cut, :]\n    return signal\n```\n\n```python\n    for i in range(binned.shape[1]):\n        binned[:, i] = apply_lowpass_filter(binned[:, i],  0.005, 1)\n\n    # downsampling\n    binned = downsampling(binned, config.interval)\n```\n\nComputed a moving average across wavelengths with a window size of 31\n\n```python\ndef moving_average(arr, window_size):\n    return np.convolve(arr, np.ones(window_size)/window_size, mode='same')\n\nwindow_size = 31\n\nfor i in tqdm(range(pre_train.shape[0])):\n    pre_train[i, :, 1:] = np.apply_along_axis(moving_average, axis=1, arr=pre_train[i, :, 1:], window_size=window_size)\n```\n\nExtracted and flipped the transit signal before optimization\n\n```python\npre_train = np.concatenate([pre_train[:, :, [0]], np.flip(pre_train[:, :, 39:321], axis=2)], axis=2)\n```\n\n# Optimization\n\nThe optimization process estimates (R/R)^2, the baseline (representing planet radius), and the concentration of each gas molecule as follows:\n\n- Set initial values for the baseline and concentrations of each molecule\n- Construct the complete signal by summing the baseline and the individual signals for each gas molecule (each molecule’s signal was collected from ExoMol data)\n- Calculate RMSE by comparing the depth of the generated signal to the observed transit signal across all wavelengths\n- Use joblib Parallel, delayed\n\n```python\ndef make_signal(s_list):\n    \"\"\"make signal\n    s_list: baseline and concentrations of each molecule\n    \"\"\"\n    result = np.repeat(np.array([s_list[0]], dtype=defalut_dtype), spectrum_array.shape[1])\n    for i, p in enumerate(s_list[1:]):\n        result += (spectrum_array[i, :]**2) * p\n    return result\n    \n    \ndef objective_each_signal(s, signal, p1, p2):\n\n    best_q = 1e10\n\n    for i in range(4) :\n        delta = int(150 / config.interval)\n\n        x = np.arange(signal.shape[0], dtype=defalut_dtype)\n        y = signal.copy()\n\n        y = np.concatenate([y[:p1-delta], y[p1+delta:p2-delta]* (1 + s), y[p2+delta:]])\n        x = np.concatenate([x[:p1-delta], x[p1+delta:p2-delta], x[p2+delta:]])\n\n        z = np.polyfit(x, y, deg=i)\n        p = np.poly1d(z)\n        q = np.mean((p(x) - y)**2)\n        \n        if q < best_q :\n            best_q = q\n        \n    return best_q\n\ndef objective(s_list, signals, p1, p2):\n\n    spectrum = make_signal(s_list)\n    mae = np.mean([objective_each_signal(spectrum[wl], signals[:, wl], p1, p2) for wl in range(1, signals.shape[1], 2)])\n    return mae\n\ndef optimize_signal(i, phase_dict, pre_train):\n    \n    p1, p2 = phase_dict[i]\n\n\t  # baselline and amount of each gas molecules\n    # ('1H2-16O','12C-1H4','12C-16O2','12C-16O','14N-1H3','1H-12C-14N','1H2-32S','48Ti-16O','51V-16O')\n    initial_guess = [2.42760318e-03, 1.23843539e-05, 1.62354048e-04, 8.55036534e-05,\n                    5.03158911e-06, 2.44615243e-05, 7.56269106e-06, 3.30325790e-05,\n                    2.41132168e-05, 2.12473388e-05]\n    \n    result = minimize(\n        objective, \n        initial_guess, \n        args=(pre_train[i, :, :], p1, p2), \n        method='L-BFGS-B', \n        bounds=[(0, None)] * (len(initial_guess))\n    )\n\n    spectrum = make_signal(result.x)\n\n    return spectrum\n```\n\n# Post-Processing\n\nFor `wl_1`, compute a weighted mean of the predicted `wl_1` value and the mean of the other wavelengths.\n\n```python\npred_mean[:, 0] = np.ones_like(pred_mean[:, 1]) * (pred_mean[:, 0] * 0.2 + pred_mean[:, 1:].mean(axis=1) * 0.8)\n```\n\n# Determining Sigma\n\n- Calculate sigma based on the RMSE for each transit signal and the mean transit signal across each wavelength.\n- Clipping to 1 * sigma\n\n```python\ndef calc_rmse_per_wl(gt, pred):\n    rmse_list = []\n    for wl in range(pred.shape[1]):\n        rmse_list.append(mean_squared_error(gt[:, wl], pred[:, wl], squared=False))\n    return rmse_list\n\nsigma_values = np.array(calc_rmse_per_wl(train_labels.values, pred_mean))\nconst = 1.4\nsigma_const = 1\nsigma_values = sigma_values * const\n\nsigma = np.std(sigma_values)\nmean = np.mean(sigma_values)\n\nsigma_values_clipped = sigma_values.copy()\nsigma_values_clipped = sigma_values_clipped.clip(-sigma_const*sigma+mean,sigma_const*sigma+mean)\n\npred_sigma_wl = np.repeat(sigma_values_clipped.reshape(1, -1), len(adc_info), axis=0)\n```",
      "votes": 15
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3033261": "# Preparing the Transit Signal\n\n(Based on [this notebook](https://www.kaggle.com/code/pourchot/ariel-data-challenge-2024))\n\n- Applied a lowpass filter instead of binning\n- Downsampled to 232 points\n- Computed a moving average across wavelengths with a window size of 31\n- Extracted and flipped the transit signal before optimization\n[ref. 1](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/538139#3027343), [ref. 2](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/529412#2965289)\n- Use joblib Parallel, delayed\n\nApplied a lowpass filter instead of binning, Downsampled to 232 points\n\n```python\nimport scipy.signal as sci_signal\n\ndef butter_lowpass(cutoff, fs, order=5):\n    nyquist = 0.5 * fs\n    normal_cutoff = cutoff / nyquist\n    b, a = sci_signal.butter(order, normal_cutoff, btype='low', analog=False)\n    return b, a\n\ndef apply_lowpass_filter(data, cutoff, fs, order=5):\n    b, a = butter_lowpass(cutoff, fs, order=order)\n    y = sci_signal.filtfilt(b, a, data)\n    return y\n\ndef downsampling(signal, interval):\n\n    cut = int(500 / interval)\n    signal = signal[::interval, :]\n    signal = signal[cut:-cut, :]\n    return signal\n```\n\n```python\n    for i in range(binned.shape[1]):\n        binned[:, i] = apply_lowpass_filter(binned[:, i],  0.005, 1)\n\n    # downsampling\n    binned = downsampling(binned, config.interval)\n```\n\nComputed a moving average across wavelengths with a window size of 31\n\n```python\ndef moving_average(arr, window_size):\n    return np.convolve(arr, np.ones(window_size)/window_size, mode='same')\n\nwindow_size = 31\n\nfor i in tqdm(range(pre_train.shape[0])):\n    pre_train[i, :, 1:] = np.apply_along_axis(moving_average, axis=1, arr=pre_train[i, :, 1:], window_size=window_size)\n```\n\nExtracted and flipped the transit signal before optimization\n\n```python\npre_train = np.concatenate([pre_train[:, :, [0]], np.flip(pre_train[:, :, 39:321], axis=2)], axis=2)\n```\n\n# Optimization\n\nThe optimization process estimates (R/R)^2, the baseline (representing planet radius), and the concentration of each gas molecule as follows:\n\n- Set initial values for the baseline and concentrations of each molecule\n- Construct the complete signal by summing the baseline and the individual signals for each gas molecule (each molecule’s signal was collected from ExoMol data)\n- Calculate RMSE by comparing the depth of the generated signal to the observed transit signal across all wavelengths\n- Use joblib Parallel, delayed\n\n```python\ndef make_signal(s_list):\n    \"\"\"make signal\n    s_list: baseline and concentrations of each molecule\n    \"\"\"\n    result = np.repeat(np.array([s_list[0]], dtype=defalut_dtype), spectrum_array.shape[1])\n    for i, p in enumerate(s_list[1:]):\n        result += (spectrum_array[i, :]**2) * p\n    return result\n    \n    \ndef objective_each_signal(s, signal, p1, p2):\n\n    best_q = 1e10\n\n    for i in range(4) :\n        delta = int(150 / config.interval)\n\n        x = np.arange(signal.shape[0], dtype=defalut_dtype)\n        y = signal.copy()\n\n        y = np.concatenate([y[:p1-delta], y[p1+delta:p2-delta]* (1 + s), y[p2+delta:]])\n        x = np.concatenate([x[:p1-delta], x[p1+delta:p2-delta], x[p2+delta:]])\n\n        z = np.polyfit(x, y, deg=i)\n        p = np.poly1d(z)\n        q = np.mean((p(x) - y)**2)\n        \n        if q < best_q :\n            best_q = q\n        \n    return best_q\n\ndef objective(s_list, signals, p1, p2):\n\n    spectrum = make_signal(s_list)\n    mae = np.mean([objective_each_signal(spectrum[wl], signals[:, wl], p1, p2) for wl in range(1, signals.shape[1], 2)])\n    return mae\n\ndef optimize_signal(i, phase_dict, pre_train):\n    \n    p1, p2 = phase_dict[i]\n\n\t  # baselline and amount of each gas molecules\n    # ('1H2-16O','12C-1H4','12C-16O2','12C-16O','14N-1H3','1H-12C-14N','1H2-32S','48Ti-16O','51V-16O')\n    initial_guess = [2.42760318e-03, 1.23843539e-05, 1.62354048e-04, 8.55036534e-05,\n                    5.03158911e-06, 2.44615243e-05, 7.56269106e-06, 3.30325790e-05,\n                    2.41132168e-05, 2.12473388e-05]\n    \n    result = minimize(\n        objective, \n        initial_guess, \n        args=(pre_train[i, :, :], p1, p2), \n        method='L-BFGS-B', \n        bounds=[(0, None)] * (len(initial_guess))\n    )\n\n    spectrum = make_signal(result.x)\n\n    return spectrum\n```\n\n# Post-Processing\n\nFor `wl_1`, compute a weighted mean of the predicted `wl_1` value and the mean of the other wavelengths.\n\n```python\npred_mean[:, 0] = np.ones_like(pred_mean[:, 1]) * (pred_mean[:, 0] * 0.2 + pred_mean[:, 1:].mean(axis=1) * 0.8)\n```\n\n# Determining Sigma\n\n- Calculate sigma based on the RMSE for each transit signal and the mean transit signal across each wavelength.\n- Clipping to 1 * sigma\n\n```python\ndef calc_rmse_per_wl(gt, pred):\n    rmse_list = []\n    for wl in range(pred.shape[1]):\n        rmse_list.append(mean_squared_error(gt[:, wl], pred[:, wl], squared=False))\n    return rmse_list\n\nsigma_values = np.array(calc_rmse_per_wl(train_labels.values, pred_mean))\nconst = 1.4\nsigma_const = 1\nsigma_values = sigma_values * const\n\nsigma = np.std(sigma_values)\nmean = np.mean(sigma_values)\n\nsigma_values_clipped = sigma_values.copy()\nsigma_values_clipped = sigma_values_clipped.clip(-sigma_const*sigma+mean,sigma_const*sigma+mean)\n\npred_sigma_wl = np.repeat(sigma_values_clipped.reshape(1, -1), len(adc_info), axis=0)\n```"
  }
}