![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2Fe736d4a036fe8ac76485380c7bd71f3e%2Frec_3528549774485167627.png?generation=1683848300543927&alt=media\")
Dear all,
\nI ran a simple bash script to analyze the train dataset folder. The script walked through the files in the dataset and collected file sizes. After collecting this metadata, I found an interesting pattern in the data (mb not) that I wanted to share with you all.
\nIn total, I identified 57 records of human_individual_masks.npy files that had the exact same size as band_{}.npy files. Most of those masks are blank:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F330e944cd197fb6a4a920b065dcdec08%2Fhm_3528549774485167627.png?generation=1683848319676525&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F16ddf1ddcb9d419de6d0e091ca219261%2Frec_8656707129988451421.png?generation=1683848352746919&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F957f0f7ecaa1f7b8fb344011ad9bc3dc%2Fhm_8656707129988451421.png?generation=1683848373031425&alt=media\")
The output file can be found here along with the bash script and the python preprocessing code.
\nAlso I tried to explore these newly obtained meta records with the train_metadata.json:
\n", "messages": [ { "id": 2255694, "postDate": "2023-05-11T23:39:54.570Z", "content": "Dear all,
\nI ran a simple bash script to analyze the train dataset folder. The script walked through the files in the dataset and collected file sizes. After collecting this metadata, I found an interesting pattern in the data (mb not) that I wanted to share with you all.
\nIn total, I identified 57 records of human_individual_masks.npy files that had the exact same size as band_{}.npy files. Most of those masks are blank:
The output file can be found here along with the bash script and the python preprocessing code.
\nAlso I tried to explore these newly obtained meta records with the train_metadata.json:
\n", "rawMarkdown": "Dear all,\n\nI ran a simple bash script to analyze the train dataset folder. The script walked through the files in the dataset and collected file sizes. After collecting this metadata, I found an interesting pattern in the data (mb not) that I wanted to share with you all.\n\nIn total, I identified **57** records of `human_individual_masks.npy` files that had the exact same size as `band_{}.npy` files. Most of those masks are blank:\n\n\n\n\n\n\n\n\n\nThe **output file** can be found [here](https://www.kaggle.com/datasets/sergiosaharovskiy/train-file-sizes-google-identify-contrails) along with the bash script and the python preprocessing code.\n\nAlso I tried to explore these newly obtained meta records with the **train_metadata.json**:\n\n[Check this out](https://www.kaggle.com/code/sergiosaharovskiy/google-contrails-2023-eda-and-submission)", "votes": 12 }, { "id": 2257063, "postDate": "2023-05-13T01:27:01.623Z", "content": "Cool stuff!
", "rawMarkdown": "Cool stuff!", "votes": 1 } ], "comments": [ { "id": 2257063, "author_name": "Aryan Garg", "author_url": "", "post_date": "2023-05-13T01:27:01.623000", "content": "Cool stuff!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2255694": "Dear all,\n\nI ran a simple bash script to analyze the train dataset folder. The script walked through the files in the dataset and collected file sizes. After collecting this metadata, I found an interesting pattern in the data (mb not) that I wanted to share with you all.\n\nIn total, I identified **57** records of `human_individual_masks.npy` files that had the exact same size as `band_{}.npy` files. Most of those masks are blank:\n\n\n\n\n\n\n\n\n\nThe **output file** can be found [here](https://www.kaggle.com/datasets/sergiosaharovskiy/train-file-sizes-google-identify-contrails) along with the bash script and the python preprocessing code.\n\nAlso I tried to explore these newly obtained meta records with the **train_metadata.json**:\n\n[Check this out](https://www.kaggle.com/code/sergiosaharovskiy/google-contrails-2023-eda-and-submission)", "2257063": "Cool stuff!" } }