![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2Fe8cfa227e41ced0a643622e02ffdfbfa%2F4b03dfa0-73fc-4609-9c07-37ff50c91f2f.png?generation=1687973048856614&alt=media\")
Hi,
\nI would like to open the discussion on 3D models. According to the Contrails paper, their best result is for a 3D backbone based on I3D. They keep time dimension in 3D convolutions and pick the frame with the annotated masks to push in the decoder and make a prediction.
\nTop winners of the just completed vesuvius competition also use some models with 3D backbones. The problem is slightly different but the idea is the same.
\nI've spent a few days on testing a lot of 3D backbones coming from the great MMAction2 implementations such as:
\nIt works but never better than a basic 2D backbone as we can see in the NoteBooks/Code section.
\nWhatever pooling or extracting the correct slice to feed the decoder, the results look not matching the paper.
Anyone with similar disapointing results on 3D modeling?
", "messages": [ { "id": 2320356, "postDate": "2023-06-27T18:06:55.530Z", "content": "Hi,
\nI would like to open the discussion on 3D models. According to the Contrails paper, their best result is for a 3D backbone based on I3D. They keep time dimension in 3D convolutions and pick the frame with the annotated masks to push in the decoder and make a prediction.
\nTop winners of the just completed vesuvius competition also use some models with 3D backbones. The problem is slightly different but the idea is the same.
\nI've spent a few days on testing a lot of 3D backbones coming from the great MMAction2 implementations such as:
\nIt works but never better than a basic 2D backbone as we can see in the NoteBooks/Code section.
\nWhatever pooling or extracting the correct slice to feed the decoder, the results look not matching the paper.
Anyone with similar disapointing results on 3D modeling?
", "rawMarkdown": "Hi,\n\nI would like to open the discussion on 3D models. According to the Contrails paper, their best result is for a 3D backbone based on I3D. They keep time dimension in 3D convolutions and pick the frame with the annotated masks to push in the decoder and make a prediction.\n\nTop winners of the just completed [vesuvius](https://www.kaggle.com/competitions/vesuvius-challenge-ink-detection) competition also use some models with 3D backbones. The problem is slightly different but the idea is the same.\n\nI've spent a few days on testing a lot of 3D backbones coming from the great [MMAction2](https://github.com/open-mmlab/mmaction2) implementations such as:\n- Vanilla Resnet3D \n- Resnet3D - irCSN 50/152\n- Resnet(2+1)D\n- MViTv2\nand more ...\n\nIt works but never better than a basic 2D backbone as we can see in the NoteBooks/Code section.\nWhatever pooling or extracting the correct slice to feed the decoder, the results look not matching the paper. \n\nAnyone with similar disapointing results on 3D modeling?", "votes": 27 }, { "id": 2320611, "postDate": "2023-06-28T00:41:53.817Z", "content": "@mpware your AUCPR results already bit the paper. I don't think that there is any practical sense to \"match\" something like \"we trained on 16 Tensor Processing Unit V3 (TPUv3) chips\" with batch size 256. I have checked on 3D as well and it slowed me down for both training and LB progression. I found many examples where contrails were present on other frames but they never appeared on the labeled frame. My rationale is that it can add more noise to 3d model rather than benefit… (it is my humble opinion though)
", "rawMarkdown": "@mpware your AUCPR results already bit the paper. I don't think that there is any practical sense to \"match\" something like \"we trained on 16 Tensor Processing Unit V3 (TPUv3) chips\" with batch size 256. I have checked on 3D as well and it slowed me down for both training and LB progression. I found many examples where contrails were present on other frames but they never appeared on the labeled frame. My rationale is that it can add more noise to 3d model rather than benefit... (it is my humble opinion though)", "votes": 6, "replies": [ { "id": 2320888, "postDate": "2023-06-28T06:20:41.013Z", "content": "\n\nyour AUCPR results already bit the paper
\n
That's true and my 3D model also beats their ResNet3D-101 AUCPR. I was hoping some predictive power for frames close to the one annotated.
", "rawMarkdown": "> your AUCPR results already bit the paper\n\nThat's true and my 3D model also beats their ResNet3D-101 AUCPR. I was hoping some predictive power for frames close to the one annotated.", "replies": [ { "id": 2321588, "postDate": "2023-06-28T17:24:17.580Z", "content": "just to add to the discussion:
I see this one does not help the model. Thanks for showing it.
", "rawMarkdown": "I see this one does not help the model. Thanks for showing it.", "votes": 1 }, { "id": 2367995, "postDate": "2023-08-01T01:06:21.503Z", "content": "well in fact, the absense of a contrail in the frame 3 should help in theory. I remember that in the paper or in the labelling rules they were saying that one factor to determine if a line was a contrail or not was to see it moving or appearing. So in that case that should help to better distinguish contrails from other static shadows/clouds/things that shouldn't be labeled as contrails.
", "rawMarkdown": "well in fact, the absense of a contrail in the frame 3 should help in theory. I remember that in the paper or in the labelling rules they were saying that one factor to determine if a line was a contrail or not was to see it moving or appearing. So in that case that should help to better distinguish contrails from other static shadows/clouds/things that shouldn't be labeled as contrails." } ] } ] }, { "id": 2321461, "postDate": "2023-06-28T15:37:01.570Z", "content": "I also try to 2.5D and 3D models, accding to the ink competition, using 4-8 frames….no models beat my 2D models
", "rawMarkdown": "I also try to 2.5D and 3D models, accding to the ink competition, using 4-8 frames....no models beat my 2D models", "votes": 1 }, { "id": 2321319, "postDate": "2023-06-28T13:06:24.437Z", "content": "Did you use MMACTION framework or only the backbones ?
", "rawMarkdown": "Did you use MMACTION framework or only the backbones ?", "votes": 1, "replies": [ { "id": 2321618, "postDate": "2023-06-28T17:42:11.067Z", "content": "Only the backbones by inheritance to get a custom output.
\nFor example for R(2+1)d:
\n ():\n …
\n ():\n \n\n x = self.conv1(x)\n x = self.maxpool(x)\n \n out = [x]\n layer_name self.res_layers:\n res_layer = (self, layer_name)\n \n x = res_layer(x)\n out.append(x)\n\n out\nwhen I use mmaction's r2+1D ,an error is reported: KeyError: 'Cannot find None in registry under scope name mmengine'. I found the place where the error occurred: /mmaction/lib/python3.8/site-packages/mmcv/cnn/bricks/conv.py, I think that conv2+1d was not added to MODELS.register_module. But I don't know how to modify it, can you tell me what should I do, thanks for your help. If this is about contest privacy, please skip this comment and thank you as well
", "rawMarkdown": "when I use mmaction's r2+1D ,an error is reported: KeyError: 'Cannot find None in registry under scope name mmengine'. I found the place where the error occurred: /mmaction/lib/python3.8/site-packages/mmcv/cnn/bricks/conv.py, I think that conv2+1d was not added to MODELS.register_module. But I don't know how to modify it, can you tell me what should I do, thanks for your help. If this is about contest privacy, please skip this comment and thank you as well" }, { "id": 2349006, "postDate": "2023-07-18T06:08:41.703Z", "content": "When you instantiate do you have this?
\ncfg = (\n =,\n depth=model_depth,\n pretrained=,\n pretrained2d=,\n norm_eval=,\n conv_cfg=(=),\n \n conv1_kernel=(, , ),\n conv1_stride_t=,\n ...\nI had to comment SyncBN because I've a single GPU.
Thanks for your reply, I will check my code again based on your reply
", "rawMarkdown": "Thanks for your reply, I will check my code again based on your reply" } ] } ] }, { "id": 2320637, "postDate": "2023-06-28T01:46:19.810Z", "content": "I tried x3d on the backbone, which also significantly lowered LB.
\nAnd the tendency for the 3D models to be unfavorable was evident from the early stages of the study. Did you see the same trend?
I'm just monitoring my CV and it's lower, it's around 0.66 with 3D backbone (frames used = 4,3,2,1) instead of 0.69 with 2D.
", "rawMarkdown": "I'm just monitoring my CV and it's lower, it's around 0.66 with 3D backbone (frames used = 4,3,2,1) instead of 0.69 with 2D.", "votes": 1, "replies": [ { "id": 2321383, "postDate": "2023-06-28T14:14:56.420Z", "content": "Thank you for the reply.
\nAre you using the same decoder as in the paper for these 3D backbones?
No but I've tried several and results depends more on encoder.
", "rawMarkdown": "No but I've tried several and results depends more on encoder.", "votes": 1 }, { "id": 2321957, "postDate": "2023-06-29T02:40:45.547Z", "content": "\n\nNo but I've tried several and results depends more on encoder.
\n
Then I guess my decoder is wrong.
\nI trained with x3d backbone using frames1-4 as you did, but I can't get to 0.6 at all.
\nAny hints you can give me about the decoder would be appreciated.
I've tried a few experiments with 2.5D (just using a 2D model and simply adding temporal context as extra channels so images will have 6 or 9 channels instead of the regular 3) but I haven't found anything promising. I was thinking of dabbling with 3D if I rent out a GPU but that's about it right now.
", "rawMarkdown": "I've tried a few experiments with 2.5D (just using a 2D model and simply adding temporal context as extra channels so images will have 6 or 9 channels instead of the regular 3) but I haven't found anything promising. I was thinking of dabbling with 3D if I rent out a GPU but that's about it right now.", "votes": 2 }, { "id": 2367465, "postDate": "2023-07-31T15:07:58.307Z", "content": "hi :),
\n i am unable to understand on picking the frame with the annotated masks . if the input is of shape 3,256,256,8 after the encoder layer the shape would be something like this right 64,32,32,2 given 4th frame is annotated how to pick it .
\ni am a begginer in computer vision do correct me if i am wrong somewhere .
\ndo suggest me if there is good source to learn about 3d models.
\nthank you :)
@mpware your AUCPR results already bit the paper. I don't think that there is any practical sense to \"match\" something like \"we trained on 16 Tensor Processing Unit V3 (TPUv3) chips\" with batch size 256. I have checked on 3D as well and it slowed me down for both training and LB progression. I found many examples where contrails were present on other frames but they never appeared on the labeled frame. My rationale is that it can add more noise to 3d model rather than benefit… (it is my humble opinion though)
", "votes": 6, "replies": [ { "id": 2320888, "author_name": "MPWARE", "author_url": "", "post_date": "2023-06-28T06:20:41.013000", "content": "\n\nyour AUCPR results already bit the paper
\n
That's true and my 3D model also beats their ResNet3D-101 AUCPR. I was hoping some predictive power for frames close to the one annotated.
", "votes": 0, "replies": [ { "id": 2321588, "author_name": "SSS", "author_url": "", "post_date": "2023-06-28T17:24:17.580000", "content": "just to add to the discussion:
I see this one does not help the model. Thanks for showing it.
", "votes": 1, "replies": [] }, { "id": 2367995, "author_name": "Enric Domingo", "author_url": "", "post_date": "2023-08-01T01:06:21.503000", "content": "well in fact, the absense of a contrail in the frame 3 should help in theory. I remember that in the paper or in the labelling rules they were saying that one factor to determine if a line was a contrail or not was to see it moving or appearing. So in that case that should help to better distinguish contrails from other static shadows/clouds/things that shouldn't be labeled as contrails.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2321461, "author_name": "lyu", "author_url": "", "post_date": "2023-06-28T15:37:01.570000", "content": "I also try to 2.5D and 3D models, accding to the ink competition, using 4-8 frames….no models beat my 2D models
", "votes": 1, "replies": [] }, { "id": 2321319, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2023-06-28T13:06:24.437000", "content": "Did you use MMACTION framework or only the backbones ?
", "votes": 1, "replies": [ { "id": 2321618, "author_name": "MPWARE", "author_url": "", "post_date": "2023-06-28T17:42:11.067000", "content": "Only the backbones by inheritance to get a custom output.
\nFor example for R(2+1)d:
\n ():\n …
\n ():\n \n\n x = self.conv1(x)\n x = self.maxpool(x)\n \n out = [x]\n layer_name self.res_layers:\n res_layer = (self, layer_name)\n \n x = res_layer(x)\n out.append(x)\n\n out\nwhen I use mmaction's r2+1D ,an error is reported: KeyError: 'Cannot find None in registry under scope name mmengine'. I found the place where the error occurred: /mmaction/lib/python3.8/site-packages/mmcv/cnn/bricks/conv.py, I think that conv2+1d was not added to MODELS.register_module. But I don't know how to modify it, can you tell me what should I do, thanks for your help. If this is about contest privacy, please skip this comment and thank you as well
", "votes": 0, "replies": [] }, { "id": 2349006, "author_name": "MPWARE", "author_url": "", "post_date": "2023-07-18T06:08:41.703000", "content": "When you instantiate do you have this?
\ncfg = (\n =,\n depth=model_depth,\n pretrained=,\n pretrained2d=,\n norm_eval=,\n conv_cfg=(=),\n \n conv1_kernel=(, , ),\n conv1_stride_t=,\n ...\nI had to comment SyncBN because I've a single GPU.
Thanks for your reply, I will check my code again based on your reply
", "votes": 0, "replies": [] } ] } ] }, { "id": 2320637, "author_name": "Shota Nishiyama", "author_url": "", "post_date": "2023-06-28T01:46:19.810000", "content": "I tried x3d on the backbone, which also significantly lowered LB.
\nAnd the tendency for the 3D models to be unfavorable was evident from the early stages of the study. Did you see the same trend?
I'm just monitoring my CV and it's lower, it's around 0.66 with 3D backbone (frames used = 4,3,2,1) instead of 0.69 with 2D.
", "votes": 1, "replies": [ { "id": 2321383, "author_name": "Shota Nishiyama", "author_url": "", "post_date": "2023-06-28T14:14:56.420000", "content": "Thank you for the reply.
\nAre you using the same decoder as in the paper for these 3D backbones?
No but I've tried several and results depends more on encoder.
", "votes": 1, "replies": [] }, { "id": 2321957, "author_name": "Shota Nishiyama", "author_url": "", "post_date": "2023-06-29T02:40:45.547000", "content": "\n\nNo but I've tried several and results depends more on encoder.
\n
Then I guess my decoder is wrong.
\nI trained with x3d backbone using frames1-4 as you did, but I can't get to 0.6 at all.
\nAny hints you can give me about the decoder would be appreciated.
I've tried a few experiments with 2.5D (just using a 2D model and simply adding temporal context as extra channels so images will have 6 or 9 channels instead of the regular 3) but I haven't found anything promising. I was thinking of dabbling with 3D if I rent out a GPU but that's about it right now.
", "votes": 2, "replies": [] }, { "id": 2367465, "author_name": "Monkey D Donut", "author_url": "", "post_date": "2023-07-31T15:07:58.307000", "content": "hi :),
\n i am unable to understand on picking the frame with the annotated masks . if the input is of shape 3,256,256,8 after the encoder layer the shape would be something like this right 64,32,32,2 given 4th frame is annotated how to pick it .
\ni am a begginer in computer vision do correct me if i am wrong somewhere .
\ndo suggest me if there is good source to learn about 3d models.
\nthank you :)