{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!wget http://bit.ly/3ZLyF82 -O CSS.css -q\n    \nfrom IPython.core.display import HTML\nwith open('./CSS.css', 'r') as file:\n    custom_css = file.read()\n\nHTML(custom_css)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-11T23:20:47.528Z","iopub.execute_input":"2023-05-11T23:20:47.528533Z","iopub.status.idle":"2023-05-11T23:20:48.99687Z","shell.execute_reply.started":"2023-05-11T23:20:47.528491Z","shell.execute_reply":"2023-05-11T23:20:48.995812Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Libraries</p>","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation\nimport seaborn as sns\nfrom tqdm.auto import tqdm\nfrom pandas.io.formats.style import Styler\nimport json\nimport os\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\ntqdm.pandas()\n\nrc = {\n    \"axes.facecolor\": \"#F8F8F8\",\n    \"figure.facecolor\": \"#F8F8F8\",\n    \"axes.edgecolor\": \"#000000\",\n    \"grid.color\": \"#EBEBE7\" + \"30\",\n    \"font.family\": \"serif\",\n    \"axes.labelcolor\": \"#000000\",\n    \"xtick.color\": \"#000000\",\n    \"ytick.color\": \"#000000\",\n    \"grid.alpha\": 0.4\n}\n\nsns.set(rc=rc)\npalette = ['#302c36', '#037d97', '#91013E', '#C09741',\n           '#EC5B6D', '#90A6B1', '#6ca957', '#D8E3E2']\n\nfrom colorama import Style, Fore\nblk = Style.BRIGHT + Fore.BLACK\nmgt = Style.BRIGHT + Fore.MAGENTA\nred = Style.BRIGHT + Fore.RED\nblu = Style.BRIGHT + Fore.BLUE\nres = Style.RESET_ALL\n\nfrom IPython import display as disp","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-11T23:20:49.00771Z","iopub.execute_input":"2023-05-11T23:20:49.008123Z","iopub.status.idle":"2023-05-11T23:20:49.025889Z","shell.execute_reply.started":"2023-05-11T23:20:49.008078Z","shell.execute_reply":"2023-05-11T23:20:49.024589Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=18x38CsqqP051tnspB-r2Kol4Ik0ACAbe\"/>\n</p>","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Introduction</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ The aim of this competition is to <b>help researchers improve the accuracy of their contrail models by validating them with satellite imagery</b>.</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ <a href=\"https://en.wikipedia.org/wiki/Contrail\"><font size=\"3.8\" color='#1976D2'><b>Contrails</b></font></a> (short for \"condensation trails\") or vapor trails are line-shaped clouds produced by aircraft engine exhaust or changes in air pressure, typically at aircraft cruising altitudes several miles above the Earth's surface. Contrails are composed primarily of water, in the form of ice crystals.</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Contrails <b>can contribute to global warming</b> by trapping heat in the atmosphere.</p>\n    \n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Contrail avoidance is potentially one of the most scalable, cost-effective sustainability solutions <b>for reducing the impact on climate change</b> available to airlines today.</p>    \n    \n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Submissions are evaluated on the global <a href=\"https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient\"><font size=\"4\" color='#1976D2'><b>Dice coefficient</b></font></a>. The formula is given by:</p>    \n\n$$\\frac{2 * \\left| X \\bigcap_{}^{} Y \\right|}{\\left| X \\right| + \\left| Y \\right|}$$\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">where <b>X</b> is the entire set of predicted contrail pixels for <b>all</b> observations in the test data and <b>Y</b> is the ground truth set of <b>all</b> contrail pixels in the test data. The maximum score for the Dice coefficient is <b>1</b>, which indicates a perfect overlap between the predicted and ground truth binary masks.</p>    \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ The competition format requires a space delimited list of pairs. For example, '1 3 10 5' implies pixels 1,2,3,10,11,12,13,14 are to be included in the mask. The metric checks that the pairs are sorted, positive, and the decoded pixel values are not duplicated. The pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc..</p>    ","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Data</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ <b>The dataset</b> for this competition was obtained from the <a href=\"https://www.goes-r.gov/spacesegment/abi.html\"><font size=\"3.8\" color='#1976D2'><b>GOES-16 Advanced Baseline Imager (ABI)</b></font></a>, which is publicly available on <a href=\"https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16\"><font size=\"3.8\" color='#1976D2'><b>Google Cloud Storage</b></font></a>. Our task is to use  geostationary satellite images to identify aviation contrails.</p> \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ The original full-disk images were reprojected using bilinear resampling to generate a local scene image. Because contrails are easier to identify with temporal context, a sequence of images at 10-minute intervals are provided. Each example (<code>record_id</code>) contains exactly one labeled frame.</p> \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Learn more about the dataset from the preprint: <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://arxiv.org/abs/2304.02122\"><font size=\"3.8\" color='#1976D2'><b>OpenContrails: Benchmarking Contrail Detection on GOES-16 ABI.</b></font></a></p> \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Labeling instructions can be found at in this <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://storage.googleapis.com/goes_contrails_dataset/20230419/Contrail_Detection_Dataset_Instruction.pdf\"><font size=\"3.8\" color='#1976D2'><b>supplementary material</b></font></a> Some key labeling guidance:</p>\n\n<ul style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">\n    <li>Contrails must contain at least <b>10</b> pixels</li>\n    <li>At some time in their life, Contrails must be at least <b>3x longer</b> than they are wide</li>\n<li>Contrails must either appear suddenly or enter from the sides of the image</li>\n<li>Contrails should be visible in at least two image</li>\n</ul>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Ground truth was determined by (generally) <b>4+</b> different labelers annotating each image. Pixels were considered a contrail when &gt;<b>50%</b> of the labelers annotated it as such. Individual annotations (<code>human_individual_masks.npy</code>) as well as the aggregated ground truth annotations (<code>human_pixel_masks.npy</code>) are included in the training data. The validation data only includes the aggregated ground truth annotations.</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">To get a sense of the data, here are a few illustrations from the preprint above: </p> \n\n<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=1CcujW7EUxQsHPjMsdx6YQikEqxQQ_MnX\"/>\n</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ The dataset provided with this challenge consists of the following <b>folders and files</b>:</p> \n\n<ul style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">\n<li><strong>train/</strong> - the training set; each folder represents a <code>record_id</code> and contains the following data:<ul>\n<li><strong>band_{08-16}.npy</strong>: array with size of <code>H x W x T</code>, where <code>T = n_times_before + n_times_after + 1</code>, representing the number of images in the sequence. There are <code>n_times_before</code> and <code>n_times_after</code> images before and after the labeled frame respectively. In our dataset all examples have  <code>n_times_before=4</code> and <code>n_times_after=3</code>. Each band represents an infrared channel at different wavelengths and is converted to brightness temperatures based on the calibration parameters. The number in the filename corresponds to the GOES-16 ABI band number. Details of the ABI bands can be found <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://www.goes-r.gov/mission/ABI-bands-quick-info.html\">here</a>.</li>\n<li><strong>human_individual_masks.npy</strong>: array with size of <code>H x W x 1 x R</code>. Each example is labeled by <code>R</code> individual human labelers. <code>R</code> is not the same for all samples. The labeled masks have value either 0 or 1 and correspond to the <code>(n_times_before+1)</code>-th image in <code>band_{08-16}.npy</code>. They are available only in the training set.</li>\n<li><strong>human_pixel_masks.npy</strong>: array with size of <code>H x W x 1</code> containing the  binary ground truth. A pixel is regarded as contrail pixel in evaluation if it is labeled as contrail by more than half of the labelers. </li></ul></li>\n<li><strong>validation/</strong> - the same as the training set, without the individual label annotations; it is permitted to use this as training data if desired</li>\n<li><strong>test/</strong> - the test set; your objective is to identify contrails found in these records. <strong>Note:</strong> Since this is a Code competition, you do not have access to the actual test set that your notebook is rerun against. The records shown here are copies of the first two records of the validation data (without the labels). The hidden test set is approximately the same size (± 5%) as the validation set. <strong>IMPORTANT:</strong> Submissions should use run-length encoding with empty predictions (e.g., no contrails) should be marked by <code>'-'</code> in the submission. (See this <a target=\"_blank\" href=\"https://www.kaggle.com/code/inversion/contrails-rle-submission\">notebook</a> for details.)</li>\n<li><strong>{train|validation}_metadata.json</strong> - metadata information for each record; contains the timestamps and the projection parameters to reproduce the satellite images.</li>\n<li><strong>sample_submission.csv</strong> - a sample submission file in the correct format</li>\n</ul>","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Preprint Notes</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ <b>Dataset</b> contains <b>20,544</b> examples in the train set and <b>1,866</b> examples in the validation set. The examples are randomly partitioned except for the satellites scenes that were identified as likely to have contrails by Google Street View, which are only included in the training set. <b>9,283</b> of the training examples contain at least one annotated contrail. About <b>1.2%</b> of the pixels in the training set are labeled as contrails.</p> \n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣  The dataset contains a wide variety of times and locations, as shown in Figure <b>5</b> and <b>6</b>. The examples are not uniformly distributed in space and time as the images are sampled to include more contrail examples as described above.</p> \n\n<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=1cmu8dPA8hu0Xo_ZeB1zROI_VTfHk6-Rp\"/>\n</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Contrails occur more often in cloudy scenes. We estimate the cloud cover fraction using the GOES-16 ABI L2 Cloud Top Phase product and regard all non-clear-sky pixels as clouds. We show the example counts in our dataset by the cloud coverage fraction in Figure 7. Our dataset contains more cloudy scenes than clear sky images, and the positives examples are roughly proportionally distributed at different cloud coverage fractions. It is possible that contrails are harder to be identified in cloudy scenes. </p> \n\n<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=1NtlxEZA0sm2wLb0p-rY9-NwWzbDG7lcr\"/>\n</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ The authors used <b>a convolutional neural network model</b> for contrail detection. The model outputs a score (between 0 to 1) for each pixel indicating the confidence that pixel is part of a contrail. First, the authors trained a binary image segmentation model for classifying each pixel as contrail or background, then the semantic segmentation architecture <b>DeeplabV3+</b> was employed experimenting with <b>different image backbones</b> Below is an example of network achitercture for multi-frame detection model:</p> \n\n<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=1SvS3_cOOsk3iXCrrRIxPJU6u079xykrt\"/>\n</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ <b>The training results</b> of the single frame models and multi-frame\n    model with different backbones were summarized in <b>Table I</b>. All models achieve reasonable detection performance, showing that our dataset is sufficiently large for training contrail detection models. The multi-frame model slightly outperforms the single-frame based models, showing the model is able to use temporal context to improve detection</p> \n \n<p align=\"right\">\n  <img src=\"https://drive.google.com/uc?export=view&id=1lvRdVaEdWRTN34bG6KDi0gda-sSjgwAm\"/>\n</p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ To access additional details, please, refer to the preprint <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://arxiv.org/abs/2304.02122\"><font size=\"3.8\" color='#1976D2'><b>OpenContrails: Benchmarking Contrail Detection on GOES-16 ABI.</b></font></a></p>","metadata":{}},{"cell_type":"code","source":"BASE_DIR = '/kaggle/input/google-research-identify-contrails-reduce-global-warming/train'\nTRAIN_META_PATH = '/kaggle/input/google-research-identify-contrails-reduce-global-warming/train_metadata.json'\nTEST_META_PATH = '/kaggle/input/google-research-identify-contrails-reduce-global-warming/validation_metadata.json'\nTRAIN_SIZES_PATH = '/kaggle/input/train-file-sizes-google-identify-contrails/train_file_sizes.csv'\n\ndef process_json_meta_data(file_path: str, verbose: bool=False):\n    '''Loads the json meta file and preprocesses it.\n    \n        Args:\n            file_path: path to the json file with metadata.\n            verbose: displays count of distinct values of split attributes\n                     for projection_wkt feature. The difference is only appears in \"central_meridian\".\n                     Those are the negative integers from 2 to 3 digits.\n    '''\n    # Loads JSON file.\n    with open(file_path, 'r') as f:\n        dict_meta = json.load(f)\n    \n    df = pd.DataFrame.from_dict(dict_meta)\n    \n    if verbose:\n        display(df.projection_wkt.str.split(',', expand=True).nunique())\n        display(df.projection_wkt.str.split(',', expand=True)[22].value_counts())\n    \n    expr = r'(?<=\"central_meridian\",)([-]*\\d{2,3})(?=])'\n    df['central_meridian'] = df.projection_wkt.str.extract(expr).astype(int) \n    df = df.drop(columns='projection_wkt')\n    df['record_id'] = df['record_id'].astype(int)\n    return df\n    \ntrain_meta = process_json_meta_data(TRAIN_META_PATH)\ntest_meta = process_json_meta_data(TEST_META_PATH)\ntrain_sizes = pd.read_csv(TRAIN_SIZES_PATH)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T23:22:10.21567Z","iopub.execute_input":"2023-05-11T23:22:10.216068Z","iopub.status.idle":"2023-05-11T23:22:11.555457Z","shell.execute_reply.started":"2023-05-11T23:22:10.216039Z","shell.execute_reply":"2023-05-11T23:22:11.554318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Metadata Overview</p>","metadata":{}},{"cell_type":"markdown","source":"This notebook provides a code snippet of how effectively display dataframes in a tidy format using Pandas Styler class.\n\nWe are going to leverage CSS styling language to manipulate many parameters including colors, fonts, borders, background, format and make our tables interactive.\n\n\n**Reference**:\n* [Pandas Table Visualization](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html).","metadata":{}},{"cell_type":"markdown","source":"At this point, you will require a certain level of understanding in web development.\n\nPrimarily, you will have to modify the CSS of the `td`, `tr`, and `th` tags.\n\n**You can refer** to the following materials to learn **HTML/CSS**:\n* [w3schools HTML Tutorial](https://www.w3schools.com/html/default.asp)\n* [w3schools CSS Reference](https://www.w3schools.com/cssref/index.php)","metadata":{}},{"cell_type":"code","source":" def magnify(is_test: bool = False):\n        base_color = '#91013E'\n        if is_test:\n            highlight_target_row = []\n        else:\n            highlight_target_row = [dict(selector='tr:last-child',\n                                         props=[('background-color', f'{base_color}'+'20')])]\n            \n        return [dict(selector=\"th\",\n                     props=[(\"font-size\", \"11pt\"),\n                            ('background-color', f'{base_color}'),\n                            ('color', 'white'),\n                            ('font-weight', 'bold'),\n                            ('border-bottom', '0.1px solid white'),\n                            ('border-left', '0.1px solid white'),\n                            ('text-align', 'right')]),\n\n                dict(selector='th.blank.level0', \n                    props=[('font-weight', 'bold'),\n                           ('border-left', '1.7px solid white'),\n                           ('background-color', 'white')]),\n\n                dict(selector=\"td\",\n                     props=[('padding', \"0.5em 1em\"),\n                            ('text-align', 'right')]),\n\n                dict(selector=\"th:hover\",\n                     props=[(\"font-size\", \"14pt\")]),\n\n                dict(selector=\"tr:hover td:hover\",\n                     props=[('max-width', '250px'),\n                            ('font-size', '14pt'),\n                            ('color', f'{base_color}'),\n                            ('font-weight', 'bold'),\n                            ('background-color', 'white'),\n                            ('border', f'1px dashed {base_color}')]),\n                \n                 dict(selector=\"caption\",\n                      props=[(('caption-side', 'bottom'))])] + highlight_target_row\n\ndef stylize_min_max_count(pivot_table):\n    \"\"\"Waps the min_max_count pivot_table into the Styler.\n\n        Args:\n            df: |min_train| max_train |min_test |max_test |top10_counts_train |top_10_counts_train|\n\n        Returns:\n            s: the dataframe wrapped into Styler.\n    \"\"\"\n    s = pivot_table\n    # A formatting dictionary for controlling each column precision (.000 <-). \n    di_frmt = {(i if i.startswith('m') else i):\n              ('{:.3f}' if i.startswith('m') else '{:}') for i in s.columns}\n\n    s = s.style.set_table_styles(magnify(True))\\\n        .format(di_frmt)\\\n        .set_caption(f\"The train and test datasets min, max, top10 values side by side (hover to magnify).\")\n    return s\n  \n    \ndef stylize_describe(df: pd.DataFrame, dataset_name: str = 'train', is_test: bool = False) -> Styler:\n    \"\"\"Applies .descibe() method to the df and wraps it into the Styler.\n    \n        Args:\n            df: any dataframe (train/test/origin)\n            dataset_name: default 'train'\n            is_test: the bool parameter passed into magnify() function\n                     in order to control the highlighting of the last row.\n                     \n        Returns:\n            s: the dataframe wrapped into Styler.\n    \"\"\"\n    s = df.describe().T\n    # A formatting dictionary for controlling each column precision (.000 <-). \n    di_frmt = {(i if i == 'count' else i):\n              ('{:.0f}' if i == 'count' else '{:.3f}') for i in s.columns}\n    \n    s = s.style.set_table_styles(magnify(is_test))\\\n        .format(di_frmt)\\\n        .set_caption(f\"The {dataset_name} dataset descriptive statistics (hover to magnify).\")\n    return s\n\ndef stylize_simple(df: pd.DataFrame, caption: str) -> Styler:\n    \"\"\"Waps the min_max_count pivot_table into the Styler.\n\n        Args:\n            df: any dataframe (train/test/origin)\n\n        Returns:\n            s: the dataframe wrapped into Styler.\n    \"\"\"\n    s = df\n    s = s.style.set_table_styles(magnify(True)).set_caption(f\"{caption}\")\n    return s\n\nft = ['row_min','row_size','col_min', 'col_size', 'timestamp', 'central_meridian']\ndisplay(stylize_simple(train_meta[ft].head(4), 'The train meta file 4 top rows (hover to magnify).'))\ndisplay(stylize_simple(test_meta[ft].head(4), 'The test meta file 4 top rows (hover to magnify).'))\nrename_di = {'index': 'file_size', 'file_size': 'counts'}\nagg =  train_sizes.groupby(by='file_size').agg({'count', 'last'})['file_path']\nagg = agg.reset_index().sort_values(by='count', ascending=False)\n\ndisplay(stylize_simple(agg, 'The train file sizes (KB) (hover to magnify).'))\ndisplay(stylize_describe(train_sizes.drop(columns=['record_id', 'file_path'])/1e6, 'sizes (MB) of train'))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-11T23:22:38.628467Z","iopub.execute_input":"2023-05-11T23:22:38.62944Z","iopub.status.idle":"2023-05-11T23:22:38.772943Z","shell.execute_reply.started":"2023-05-11T23:22:38.629401Z","shell.execute_reply":"2023-05-11T23:22:38.771792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**:\n* The train files `band_{}` are relatively bigger than `human_individual_masks` but we need to check if there any masks with the same size in the data.\n* It will be nice to sanity check a symmetric difference between **train/** `record_id` and **train_metadata.json** `record_id`.\n\nLet's check this out:","metadata":{}},{"cell_type":"code","source":"record_id_diff = set(train_sizes.record_id).symmetric_difference(train_meta.record_id)\nprint(f'{blk}[INFO] Any difference in record_id for train_metadata.json and train dir:'\n      f'{blk}\\n[+] -> {red}{len(record_id_diff) > 0}')\n\nmask_sizes = train_sizes[~train_sizes.file_path.str.contains('band') & train_sizes.file_size.eq(2097280)]\n\nprint(f'\\n{blk}[INFO] There are following number of non band_ files with size 2097280 KB:'\n      f'{blk}\\n[+] -> {red}{mask_sizes.shape[0]}\\n')\n\nstylize_simple(mask_sizes.head(4), 'Exampes of non- band_{} files with size 2097280 KB')","metadata":{"execution":{"iopub.status.busy":"2023-05-11T23:22:54.564622Z","iopub.execute_input":"2023-05-11T23:22:54.56501Z","iopub.status.idle":"2023-05-11T23:22:54.790981Z","shell.execute_reply.started":"2023-05-11T23:22:54.564971Z","shell.execute_reply":"2023-05-11T23:22:54.789829Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**:\n\n* There are **57** `human_individual_masks.npy` with the same size as `band_{}`. It makes sense to visualize such samples later on.","metadata":{}},{"cell_type":"code","source":"# kudos to @docxian\ns = sns.pairplot(train_meta[ft],\n                 plot_kws={'alpha' : 0.2, 'color' : palette[2]},\n                 diag_kws={'color' : palette[2]})\ns.fig.suptitle(f'Pair Plot for the train metadata file',\n               fontweight='bold', fontsize=20, y=1.03);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-11T23:23:22.095741Z","iopub.execute_input":"2023-05-11T23:23:22.096129Z","iopub.status.idle":"2023-05-11T23:23:35.515331Z","shell.execute_reply.started":"2023-05-11T23:23:22.0961Z","shell.execute_reply":"2023-05-11T23:23:35.514209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**:\n* `col_size` and `row_size` are not always symmetrical. It might make sense to check such examples.","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Train records</p>","metadata":{}},{"cell_type":"code","source":"# original source code https://www.kaggle.com/code/inversion/visualizing-contrails\n# I have prepared the custom functions below based on the original code.\n\nN_TIMES_BEFORE = 4\n\ndef normalize_range(data, bounds):\n    \"\"\"Maps data to the range [0, 1].\"\"\"\n    return (data - bounds[0]) / (bounds[1] - bounds[0])\n\ndef read_record(record_id: str):\n\n    dir_files = os.listdir(os.path.join(BASE_DIR, record_id))\n    # band_files = sorted([f for f in dir_files if f.startswith('band_')])[::2][:3]\n    band_files = ['band_11.npy', 'band_14.npy', 'band_15.npy']\n    files_to_viz = band_files + ['human_pixel_masks.npy', 'human_individual_masks.npy']\n\n    loaded_files = []\n    for ftz in files_to_viz:\n        try:\n            with open(os.path.join(BASE_DIR, record_id, ftz), 'rb') as f:\n                loaded_files.append(np.load(f))\n        except Exception as e:\n            print(f'[ERROR] One of the files is missing in the train dir. {e}.')\n\n    _T11_BOUNDS = (243, 303)\n    _CLOUD_TOP_TDIFF_BOUNDS = (-4, 5)\n    _TDIFF_BOUNDS = (-4, 2)    \n\n    r = normalize_range(loaded_files[2] - loaded_files[1], _TDIFF_BOUNDS)\n    g = normalize_range(loaded_files[1] - loaded_files[0], _CLOUD_TOP_TDIFF_BOUNDS)\n    b = normalize_range(loaded_files[1], _T11_BOUNDS)\n    false_color = np.clip(np.stack([r, g, b], axis=2), 0, 1)\n    return loaded_files, false_color\n\ndef viz_record(img: np.ndarray, hpm: np.ndarray) -> None:\n    fig, ax = plt.subplots(1, 3, figsize=(18, 6))\n    ax = ax.flatten()\n\n    ax[0].imshow(img)\n    ax[0].set_title('False color image')\n    ax[0].grid(False)\n\n    ax[1].imshow(hpm, interpolation='none')\n    ax[1].set_title('Ground truth contrail mask')\n    ax[1].grid(False)\n\n    ax[2].imshow(img)\n    ax[2].imshow(hpm, cmap='Reds', alpha=.4, interpolation='none')\n    ax[2].set_title('Contrail mask on false color image')\n    ax[2].grid(False)\n    fig.suptitle(f'Record #: {record_id}')\n    plt.show()\n    \ndef viz_human_individual_masks(him: np.ndarray) -> None:\n    \"\"\"Draws Individual human masks\n        \n        Args:\n            him: human_individual_mask\n        \n        Returns:\n            None\n    \"\"\"\n    n = human_individual_mask.shape[-1]\n    fig = plt.figure(figsize=(16, 4))\n    for i in range(n):\n        plt.subplot(1, n, i+1)\n        plt.imshow(human_individual_mask[..., i], interpolation='none')\n        plt.grid(False)\n        fig.suptitle(f'Record #: {record_id} \\n human_individual_masks')\n        \ndef run_animation(fc: np.array):    \n    \"\"\"Runs matplotlib animation.\n    \n        Args: \n            fc: false_color \n            \n        Return:\n            animation.FuncAnimation\n    \"\"\"\n    fig = plt.figure(figsize=(12, 8))\n    img = plt.imshow(fc[..., 0]) \n    plt.axis('off')\n    plt.close()\n    \n    def frame(i):\n        img.set_array(fc[..., i])\n        return [img]\n\n    return animation.FuncAnimation(fig, frame, frames=false_color.shape[-1],\n                                   interval=500, blit=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-11T23:23:35.516857Z","iopub.execute_input":"2023-05-11T23:23:35.517146Z","iopub.status.idle":"2023-05-11T23:23:35.542656Z","shell.execute_reply.started":"2023-05-11T23:23:35.51712Z","shell.execute_reply":"2023-05-11T23:23:35.541085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\"> <b>Combine bands into a false color image</b>.</p>\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ In order to view contrails in GOES, we use the \"ash\" color scheme. This color scheme was originally developed for viewing volcanic ash in the atmosphere but is also useful for viewing thin cirrus, including contrails. In this color scheme, contrails appear in the image as dark blue.</p> \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\"> <b>Note</b> that the Kaggle Team used a modified version of the ash color scheme here, developed by Kulik et al., which uses slightly different bands and bounds tuned for contrails.</p> \n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\"> <b>References</b></p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">- <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://rammb.cira.colostate.edu/training/visit/quick_guides/GOES_Ash_RGB.pdf\"><font size=\"3.8\" color='#1976D2'><b>Original Ash RGB description</b></font></a></p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">- <a rel=\"noreferrer nofollow\" target=\"_blank\" href=\"https://dspace.mit.edu/handle/1721.1/124179?show=full\"><font size=\"3.8\" color='#1976D2'><b>Modified Ash Color Scheme (Kulik et al., page 22)</b></font></a></p>\n\n<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Let's look at our first randomly chosen record <b>18467979031221222</b>. We can easily load it and plot it with the hidden custom functions above.</p>","metadata":{}},{"cell_type":"code","source":"record_id = '18467979031221222'\n\n# Loads the data, creates img and various masks.\nloaded_files, false_color = read_record(record_id)\nimg = false_color[..., N_TIMES_BEFORE]\nhuman_pixel_masks = loaded_files[3]\nhuman_individual_mask = loaded_files[4]\nviz_record(img, human_pixel_masks)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T23:23:38.635266Z","iopub.execute_input":"2023-05-11T23:23:38.635677Z","iopub.status.idle":"2023-05-11T23:23:40.104379Z","shell.execute_reply.started":"2023-05-11T23:23:38.635646Z","shell.execute_reply":"2023-05-11T23:23:40.103432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**:\n* This example shows really faint traces of contrails (at least to the naked eye).\n* The ground truth contrail mask is fair.\n\nLet's look at the `human_individual_masks:","metadata":{}},{"cell_type":"code","source":"viz_human_individual_masks(human_individual_mask)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-05-11T23:23:42.803658Z","iopub.execute_input":"2023-05-11T23:23:42.804972Z","iopub.status.idle":"2023-05-11T23:23:43.611364Z","shell.execute_reply.started":"2023-05-11T23:23:42.804903Z","shell.execute_reply":"2023-05-11T23:23:43.610278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**:\n* It can be clearly seen that some human labelers struggled with the task.\n\nLet's animate:","metadata":{}},{"cell_type":"code","source":"anim = run_animation(false_color)\ndisp.HTML(anim.to_jshtml())","metadata":{"execution":{"iopub.status.busy":"2023-05-11T23:23:45.058667Z","iopub.execute_input":"2023-05-11T23:23:45.059277Z","iopub.status.idle":"2023-05-11T23:23:49.006131Z","shell.execute_reply.started":"2023-05-11T23:23:45.059244Z","shell.execute_reply":"2023-05-11T23:23:49.004627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">‣ Now Let's examine the records which draw our attention during the metadata exploration.</p>\n\n<ul style=\"font-family:JetBrains Mono; font-weight:normal; font-size: 16px;\">\n    <li>The examples are characterized by file size 2097280 KB</li>\n    <li>There are <b>57</b> train examples where <code>human_individual_masks</code> files got the equals size as <code>bank_{}</code> files: </li>\n\n</ul>","metadata":{}},{"cell_type":"code","source":"records_to_check = mask_sizes.record_id.sample(3, random_state=322).tolist()\nprint(f'{blk}[INFO] There are folliwing records_id to be checked: {red}{records_to_check}{res}')","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-05-11T23:23:49.008878Z","iopub.execute_input":"2023-05-11T23:23:49.009785Z","iopub.status.idle":"2023-05-11T23:23:49.020812Z","shell.execute_reply.started":"2023-05-11T23:23:49.009732Z","shell.execute_reply":"2023-05-11T23:23:49.019116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for r in records_to_check:\n    record_id = str(r)\n    # Loads the data, creates img and various masks.\n    loaded_files, false_color = read_record(record_id)\n    img = false_color[..., N_TIMES_BEFORE]\n    human_pixel_masks = loaded_files[3]\n    human_individual_mask = loaded_files[4]\n    viz_record(img, human_pixel_masks)\n    viz_human_individual_masks(human_individual_mask)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T23:23:49.02288Z","iopub.execute_input":"2023-05-11T23:23:49.02328Z","iopub.status.idle":"2023-05-11T23:23:56.522248Z","shell.execute_reply.started":"2023-05-11T23:23:49.023245Z","shell.execute_reply":"2023-05-11T23:23:56.521203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**:\n* Well, it seems some of them just got the blank masks (That's an interesting fact).","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Acknowledgement</p>","metadata":{}},{"cell_type":"markdown","source":"@jcaliz for .css and plotting ideas.","metadata":{}},{"cell_type":"markdown","source":"## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 2px; color:#91013E; font-size:140%; text-align:left;padding: 0px; border-bottom: 3px solid #91013E\">Outro and future work</p>\n\nThe work is still in progress. I hope to continue working on it and add more to EDA, training and inference part. Good luck in the competition!","metadata":{}}]}