{ "id": 410727, "title": "Missing masks", "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/410727", "author_name": "Tord Malmgren", "post_date": "2023-05-16T11:35:22.827000", "votes": 2, "comment_count": 3, "views": 0, "content": "

For 1000216489776414077, all masks are blank.
\nFor 1008786564566646931, the \"pixel\" is empty, but \"individual\" #2 has some.
\nOut of the first 120 entries (ordered by recordId), I have 49 with empty masks.
\nIs this in line with what others are seeing?

", "messages": [ { "id": 2262200, "postDate": "2023-05-16T19:21:35.930Z", "content": "

Hi Tord,

\n

What you're seeing is expected. Most scenes do not contain any contrails, so most masks are all 0's. The distribution of empty masks should roughly reflect what is seen in the real world for the space-time region that the dataset covers. The goal is to train a model that can eventually be run on an arbitrary scene from a satellite image and identify contrails if they are there, even though most of the time there won't be any. You're welcome to try out different training approaches that might involve upweighting the samples that do have contrails or subsampling the ones that don't.

\n

It's also expected that the combined mask can be all 0's in cases where individual masks are not all 0's. This is because the combined mask is a majority vote per-pixel of all the individual masks.

", "rawMarkdown": "Hi Tord,\n\nWhat you're seeing is expected. Most scenes do not contain any contrails, so most masks are all 0's. The distribution of empty masks should roughly reflect what is seen in the real world for the space-time region that the dataset covers. The goal is to train a model that can eventually be run on an arbitrary scene from a satellite image and identify contrails if they are there, even though most of the time there won't be any. You're welcome to try out different training approaches that might involve upweighting the samples that do have contrails or subsampling the ones that don't.\n\nIt's also expected that the combined mask can be all 0's in cases where individual masks are not all 0's. This is because the combined mask is a majority vote per-pixel of all the individual masks.", "votes": 4 }, { "id": 2265003, "postDate": "2023-05-18T23:16:29.247Z", "content": "

I confirm that more than half the pixel masks are blanks (around 11000), and around 8000 of them are blank because every individual mask is blank.

", "rawMarkdown": "I confirm that more than half the pixel masks are blanks (around 11000), and around 8000 of them are blank because every individual mask is blank.", "votes": 1 }, { "id": 2261794, "postDate": "2023-05-16T14:51:28.287Z", "content": "

Yes, i also checked the whole train dataset and found that a lot of record ids dont have any mask ( so no contrails)

", "rawMarkdown": "Yes, i also checked the whole train dataset and found that a lot of record ids dont have any mask ( so no contrails)", "votes": 1 }, { "id": 2261515, "postDate": "2023-05-16T11:35:22.827Z", "content": "

For 1000216489776414077, all masks are blank.
\nFor 1008786564566646931, the \"pixel\" is empty, but \"individual\" #2 has some.
\nOut of the first 120 entries (ordered by recordId), I have 49 with empty masks.
\nIs this in line with what others are seeing?

", "rawMarkdown": "For 1000216489776414077, all masks are blank.\nFor 1008786564566646931, the \"pixel\" is empty, but \"individual\" #2 has some.\nOut of the first 120 entries (ordered by recordId), I have 49 with empty masks.\nIs this in line with what others are seeing?", "votes": 1 } ], "comments": [ { "id": 2262200, "author_name": "Aaron Sarna", "author_url": "", "post_date": "2023-05-16T19:21:35.930000", "content": "

Hi Tord,

\n

What you're seeing is expected. Most scenes do not contain any contrails, so most masks are all 0's. The distribution of empty masks should roughly reflect what is seen in the real world for the space-time region that the dataset covers. The goal is to train a model that can eventually be run on an arbitrary scene from a satellite image and identify contrails if they are there, even though most of the time there won't be any. You're welcome to try out different training approaches that might involve upweighting the samples that do have contrails or subsampling the ones that don't.

\n

It's also expected that the combined mask can be all 0's in cases where individual masks are not all 0's. This is because the combined mask is a majority vote per-pixel of all the individual masks.

", "votes": 4, "replies": [] }, { "id": 2265003, "author_name": "iceman273k", "author_url": "", "post_date": "2023-05-18T23:16:29.247000", "content": "

I confirm that more than half the pixel masks are blanks (around 11000), and around 8000 of them are blank because every individual mask is blank.

", "votes": 1, "replies": [] }, { "id": 2261794, "author_name": "Jan H", "author_url": "", "post_date": "2023-05-16T14:51:28.287000", "content": "

Yes, i also checked the whole train dataset and found that a lot of record ids dont have any mask ( so no contrails)

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2262200": "Hi Tord,\n\nWhat you're seeing is expected. Most scenes do not contain any contrails, so most masks are all 0's. The distribution of empty masks should roughly reflect what is seen in the real world for the space-time region that the dataset covers. The goal is to train a model that can eventually be run on an arbitrary scene from a satellite image and identify contrails if they are there, even though most of the time there won't be any. You're welcome to try out different training approaches that might involve upweighting the samples that do have contrails or subsampling the ones that don't.\n\nIt's also expected that the combined mask can be all 0's in cases where individual masks are not all 0's. This is because the combined mask is a majority vote per-pixel of all the individual masks.", "2265003": "I confirm that more than half the pixel masks are blanks (around 11000), and around 8000 of them are blank because every individual mask is blank.", "2261794": "Yes, i also checked the whole train dataset and found that a lot of record ids dont have any mask ( so no contrails)", "2261515": "For 1000216489776414077, all masks are blank.\nFor 1008786564566646931, the \"pixel\" is empty, but \"individual\" #2 has some.\nOut of the first 120 entries (ordered by recordId), I have 49 with empty masks.\nIs this in line with what others are seeing?" } }