![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2F01f1f97ba14205a2a30c26e8a1e0ae4b%2FAriel1.png?generation=1722808009657800&alt=media\")
Hi everyone,
\nI'm trying to find out the correct procedure to produce calibrated images based on ARIEL red book (https://sci.esa.int/documents/34022/36216/Ariel_Definition_Study_Report_2020.pdf). Here are my thoughts. What do you think?
Does this pipeline looks reasonable to you?
\n
Hi everyone,
\nI'm trying to find out the correct procedure to produce calibrated images based on ARIEL red book (https://sci.esa.int/documents/34022/36216/Ariel_Definition_Study_Report_2020.pdf). Here are my thoughts. What do you think?
Does this pipeline looks reasonable to you?
\n
Thank you @sakvaua for the heroic effort! We will be publishing the pipeline soon so that you can cross check your pipeline with ours, again, the calibration procedures are not perfect and they are prone to non-linear effects (jitter etc) in the data. so please do not take our pipeline as the ONLY pipeline.
", "rawMarkdown": "Thank you @sakvaua for the heroic effort! We will be publishing the pipeline soon so that you can cross check your pipeline with ours, again, the calibration procedures are not perfect and they are prone to non-linear effects (jitter etc) in the data. so please do not take our pipeline as the ONLY pipeline. ", "votes": 3 }, { "id": 2947155, "postDate": "2024-08-05T02:39:50.760Z", "content": "Notebook - Sensor Noise Correction @jnesbit6 shared public notebook
\n@sakvaua dead cells remove logic missing
\n signal = signal.to_numpy().reshape(sensor_sizes_dict[sensor][])\n\n \n signal = signal - dark_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n signal = signal - read_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n\n \n flat = flat_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n flat[dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])] = \n signal = signal/flat\n\n \n coefficients = linear_corr_frame.to_numpy().reshape([] + sensor_sizes_dict[sensor][])\n coefficients = np.repeat(coefficients, sensor_sizes_dict[sensor][][], axis=)\n corrected = coefficients[]\n i ():\n corrected += np.multiply(np.power(signal, i+),coefficients[i+])\n\n \n corrected = corrected*planet_gain_offset[sensor + ].values + planet_gain_offset[sensor + ].values\n\n \n corrected[np.repeat(dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][]), sensor_sizes_dict[sensor][][], axis=)] = \nYeah, I forgot to mention dead pixels. I'm unsure if the order of calibration steps in this code snippet is correct.
", "rawMarkdown": "Yeah, I forgot to mention dead pixels. I'm unsure if the order of calibration steps in this code snippet is correct.", "votes": 1, "replies": [ { "id": 2947395, "postDate": "2024-08-05T06:51:03.340Z", "content": "Thanks, now i understood what you mean by order of calibrations.
", "rawMarkdown": "Thanks, now i understood what you mean by order of calibrations." } ] } ] }, { "id": 2947042, "postDate": "2024-08-04T22:54:47.257Z", "content": "Do hosts truly expect us to reverse-engineering SOC@ESA pre-processing pipeline?
\nI think more specification should be needed on how to handle calibration data (i.e. Level1 ~ Level1.5 product).
E.g.
\nAlso, even and odd images from the AIRS-CH0 instrument have different exposures yet only one dark frame for calibration.
", "rawMarkdown": "Also, even and odd images from the AIRS-CH0 instrument have different exposures yet only one dark frame for calibration." }, { "id": 2947623, "postDate": "2024-08-05T10:19:49.893Z", "content": "No no no not at all! We are doing some final checks before we push out the calibration notebook to everyone :)
\nWe understand not everyone knows calibration, and it will be a BIG BIG ask for people to know
", "rawMarkdown": "No no no not at all! We are doing some final checks before we push out the calibration notebook to everyone :) \n\nWe understand not everyone knows calibration, and it will be a BIG BIG ask for people to know", "votes": 6 } ] } ], "comments": [ { "id": 2947626, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-05T10:22:18.643000", "content": "Thank you @sakvaua for the heroic effort! We will be publishing the pipeline soon so that you can cross check your pipeline with ours, again, the calibration procedures are not perfect and they are prone to non-linear effects (jitter etc) in the data. so please do not take our pipeline as the ONLY pipeline.
", "votes": 3, "replies": [] }, { "id": 2947155, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-05T02:39:50.760000", "content": "Notebook - Sensor Noise Correction @jnesbit6 shared public notebook
\n@sakvaua dead cells remove logic missing
\n signal = signal.to_numpy().reshape(sensor_sizes_dict[sensor][])\n\n \n signal = signal - dark_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n signal = signal - read_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n\n \n flat = flat_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])\n flat[dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][])] = \n signal = signal/flat\n\n \n coefficients = linear_corr_frame.to_numpy().reshape([] + sensor_sizes_dict[sensor][])\n coefficients = np.repeat(coefficients, sensor_sizes_dict[sensor][][], axis=)\n corrected = coefficients[]\n i ():\n corrected += np.multiply(np.power(signal, i+),coefficients[i+])\n\n \n corrected = corrected*planet_gain_offset[sensor + ].values + planet_gain_offset[sensor + ].values\n\n \n corrected[np.repeat(dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][]), sensor_sizes_dict[sensor][][], axis=)] = \nYeah, I forgot to mention dead pixels. I'm unsure if the order of calibration steps in this code snippet is correct.
", "votes": 1, "replies": [ { "id": 2947395, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-05T06:51:03.340000", "content": "Thanks, now i understood what you mean by order of calibrations.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2947042, "author_name": "Bilzard", "author_url": "", "post_date": "2024-08-04T22:54:47.257000", "content": "Do hosts truly expect us to reverse-engineering SOC@ESA pre-processing pipeline?
\nI think more specification should be needed on how to handle calibration data (i.e. Level1 ~ Level1.5 product).
E.g.
\nAlso, even and odd images from the AIRS-CH0 instrument have different exposures yet only one dark frame for calibration.
", "votes": 0, "replies": [] }, { "id": 2947623, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-05T10:19:49.893000", "content": "No no no not at all! We are doing some final checks before we push out the calibration notebook to everyone :)
\nWe understand not everyone knows calibration, and it will be a BIG BIG ask for people to know
", "votes": 6, "replies": [] } ] } ], "raw_markdown_by_id": { "2947006": "Hi everyone,\nI'm trying to find out the correct procedure to produce calibrated images based on ARIEL red book (https://sci.esa.int/documents/34022/36216/Ariel_Definition_Study_Report_2020.pdf). Here are my thoughts. What do you think?\n1. We are given a signal in ADU. We need to convert it to electrons by multiplying by gain and adding offset\n2. Subtract readout noise (Subtract zeroth read?)\n3. Non-linearity correction. There are no details in the red book. But HST calibration (https://hst-docs.stsci.edu/wfc3dhb/chapter-3-wfc3-data-calibration/3-3-ir-data-calibration-steps#id-3.3IRDataCalibrationSteps-3.3.2IRZero-ReadSignalCorrection) is Fc=F*(1+C1+C2*F+C3*F^2+c4*F^3) but in our case there are 6 coefficients\n4. Subtract dark current\n5. Divide by flat field\n\nDoes this pipeline looks reasonable to you?\n\n\n", "2947626": "Thank you @sakvaua for the heroic effort! We will be publishing the pipeline soon so that you can cross check your pipeline with ours, again, the calibration procedures are not perfect and they are prone to non-linear effects (jitter etc) in the data. so please do not take our pipeline as the ONLY pipeline. ", "2947155": "[Notebook - Sensor Noise Correction](https://www.kaggle.com/code/jnesbit6/sensor-noise-correction) @jnesbit6 shared public notebook\n\n@sakvaua dead cells remove logic missing\n\n```python\n signal = signal.to_numpy().reshape(sensor_sizes_dict[sensor][0])\n\n # read and dark frame correction, just subtraction\n signal = signal - dark_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n signal = signal - read_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n\n # flat frame correction + ensure dead pixels do not disrupt. We divide by the flat frame\n flat = flat_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])\n flat[dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1])] = 1\n signal = signal/flat\n\n # linear correction - treat as sixth degree polynomial, first number is C, not coefficient?\n coefficients = linear_corr_frame.to_numpy().reshape([6] + sensor_sizes_dict[sensor][1])\n coefficients = np.repeat(coefficients, sensor_sizes_dict[sensor][0][0], axis=1)\n corrected = coefficients[0]\n for i in range(5):\n corrected += np.multiply(np.power(signal, i+1),coefficients[i+1])\n \n # gain, offset\n corrected = corrected*planet_gain_offset[sensor + \"_adc_gain\"].values + planet_gain_offset[sensor + \"_adc_offset\"].values\n \n # zero out malfunctioning pixels\n corrected[np.repeat(dead_frame.to_numpy().reshape(sensor_sizes_dict[sensor][1]), sensor_sizes_dict[sensor][0][0], axis=0)] = 0\n```", "2947042": "Do hosts truly expect us to reverse-engineering SOC@ESA pre-processing pipeline?\nI think more specification should be needed on how to handle calibration data (i.e. Level1 ~ Level1.5 product).\n\nE.g.\n\n* what linear-correction coefficient mean?" } }