{
  "id": 218535,
  "title": "Average Precision per Class ?",
  "url": "/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/218535",
  "author_name": "JIN",
  "post_date": "2021-02-11T03:29:32.868000",
  "votes": 4,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi guys, Is there anyone who checked AP per Class ? It seems like that some classes show very low AP. </p>\n<p>I used </p>\n<ul>\n<li>512x512</li>\n<li>WBF before training</li>\n<li>DetectoRS(r50)</li>\n<li>MultiLabel Stratifiedkfold : 5fold</li>\n<li>mAP@0.5 : 0.3420(fold_0)<br>\n<img src=\"https://drive.google.com/uc?export=view&amp;id=1SX4haHfjNrAqcmtFaZrYBBOrA-4k6Gd_\" alt=\"\"></li>\n</ul>\n<p>I haven't made inference code, so I'll update LB score soon.</p>\n<p>-------- update 02.23 ------------<br>\nCV </p>\n<ul>\n<li>mAP@0.5 : 0.3470(fold_0)</li>\n<li>mAP@0.4 : 0.4060(fold_0)</li>\n</ul>\n<p><img src=\"https://drive.google.com/uc?export=view&amp;id=1RVGxu7SeE0oJdIfZ_xOTSswSCgSdGvqO\" alt=\"\"></p>\n<p>LB</p>\n<ul>\n<li>0.169<ul>\n<li>confidence score : 0.001</li>\n<li>nms_iou : 0.5</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 1195742,
      "postDate": "2021-02-11T03:29:32.870Z",
      "content": "<p>Hi guys, Is there anyone who checked AP per Class ? It seems like that some classes show very low AP. </p>\n<p>I used </p>\n<ul>\n<li>512x512</li>\n<li>WBF before training</li>\n<li>DetectoRS(r50)</li>\n<li>MultiLabel Stratifiedkfold : 5fold</li>\n<li>mAP@0.5 : 0.3420(fold_0)<br>\n<img src=\"https://drive.google.com/uc?export=view&amp;id=1SX4haHfjNrAqcmtFaZrYBBOrA-4k6Gd_\" alt=\"\"></li>\n</ul>\n<p>I haven't made inference code, so I'll update LB score soon.</p>\n<p>-------- update 02.23 ------------<br>\nCV </p>\n<ul>\n<li>mAP@0.5 : 0.3470(fold_0)</li>\n<li>mAP@0.4 : 0.4060(fold_0)</li>\n</ul>\n<p><img src=\"https://drive.google.com/uc?export=view&amp;id=1RVGxu7SeE0oJdIfZ_xOTSswSCgSdGvqO\" alt=\"\"></p>\n<p>LB</p>\n<ul>\n<li>0.169<ul>\n<li>confidence score : 0.001</li>\n<li>nms_iou : 0.5</li></ul></li>\n</ul>",
      "rawMarkdown": "Hi guys, Is there anyone who checked AP per Class ? It seems like that some classes show very low AP. \n\nI used \n- 512x512\n- WBF before training\n- DetectoRS(r50)\n- MultiLabel Stratifiedkfold : 5fold\n- mAP@0.5 : 0.3420(fold_0)\n![](https://drive.google.com/uc?export=view&id=1SX4haHfjNrAqcmtFaZrYBBOrA-4k6Gd_\")\n\nI haven't made inference code, so I'll update LB score soon.\n\n-------- update 02.23 ------------\nCV \n - mAP@0.5 : 0.3470(fold_0)\n - mAP@0.4 : 0.4060(fold_0)\n\n![](https://drive.google.com/uc?export=view&id=1RVGxu7SeE0oJdIfZ_xOTSswSCgSdGvqO\")\n\nLB\n - 0.169\n- confidence score : 0.001\n- nms_iou : 0.5\n",
      "votes": 4
    },
    {
      "id": 1197033,
      "postDate": "2021-02-11T21:30:37.227Z",
      "content": "<p>I see similar low AP for calcification, atelectasis, nodule mass, other lesion cases. </p>\n<ul>\n<li>Calcification: I think the reason is that I trained my model on 512*512 images. Because of downsizing images, in low resolution images information about comparatively smaller calcification boxes is lost and the AP for that becomes small per as well. What is the size of your images that you are using for training? </li>\n<li>Atelectasis :- by default has smaller detection <br>\nI am still unsure about how we get low AP for other cases. </li>\n</ul>",
      "rawMarkdown": "I see similar low AP for calcification, atelectasis, nodule mass, other lesion cases. \n- Calcification: I think the reason is that I trained my model on 512*512 images. Because of downsizing images, in low resolution images information about comparatively smaller calcification boxes is lost and the AP for that becomes small per as well. What is the size of your images that you are using for training? \n- Atelectasis :- by default has smaller detection \nI am still unsure about how we get low AP for other cases. ",
      "votes": 1,
      "replies": [
        {
          "id": 1197540,
          "postDate": "2021-02-12T08:23:29.963Z",
          "content": "<p>Thank you for the information.<br>\nas you mentioned, there are very small object in some case(512x512).<br>\n<img src=\"https://drive.google.com/uc?export=view&amp;id=1iP1R0NvrS-CMcZSoU95uDC2TfBsT9lEl\" alt=\"\"><br>\nI need to tune <code>anchor_size</code> if I still want to use 512x512 images but I'm not sure is it critical issue for AP</p>",
          "rawMarkdown": "Thank you for the information.\nas you mentioned, there are very small object in some case(512x512).\n![](https://drive.google.com/uc?export=view&id=1iP1R0NvrS-CMcZSoU95uDC2TfBsT9lEl\")\nI need to tune `anchor_size` if I still want to use 512x512 images but I'm not sure is it critical issue for AP\n\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 1196610,
      "postDate": "2021-02-11T14:03:00.497Z",
      "content": "<p>i used detectoRS too, but inference look not good.</p>",
      "rawMarkdown": "i used detectoRS too, but inference look not good.",
      "votes": 1
    },
    {
      "id": 1195826,
      "postDate": "2021-02-11T05:17:51.070Z",
      "content": "<p>In my case, Atelectasis, Calcification, Nodule, Other_lesion have low AP. As you know, bbox area of Calcification, Nodule are quite small. But I don't have any ideas about Atelectasis.</p>",
      "rawMarkdown": "In my case, Atelectasis, Calcification, Nodule, Other_lesion have low AP. As you know, bbox area of Calcification, Nodule are quite small. But I don't have any ideas about Atelectasis.",
      "votes": 2,
      "replies": [
        {
          "id": 1196444,
          "postDate": "2021-02-11T12:12:43.347Z",
          "content": "<p>For Atelectasis the reason is maybe that there are only few cases?</p>",
          "rawMarkdown": "For Atelectasis the reason is maybe that there are only few cases?",
          "votes": 2
        },
        {
          "id": 1196519,
          "postDate": "2021-02-11T13:03:29.767Z",
          "content": "<p>It might be right, there are very few images and labels for Atelectasis in train dataset<br>\nonly 279 annotations and 186 images for Atelectasis :)</p>",
          "rawMarkdown": "It might be right, there are very few images and labels for Atelectasis in train dataset\nonly 279 annotations and 186 images for Atelectasis :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 1658187,
      "postDate": "2022-01-20T20:12:40.057Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/jinssaa\" target=\"_blank\">@jinssaa</a> ! Can you share the code you used to compute per class Average Precision? <br>\nI tried to modify coco_eval.py like this ( <a href=\"https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes\" target=\"_blank\">https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes</a> ) but it didn't work for me. Thank you.</p>",
      "rawMarkdown": "Hi @jinssaa ! Can you share the code you used to compute per class Average Precision? \nI tried to modify coco_eval.py like this ( https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes ) but it didn't work for me. Thank you."
    }
  ],
  "comments": [
    {
      "id": 1197033,
      "author_name": "sourabhsc",
      "author_url": "",
      "post_date": "2021-02-11T21:30:37.227000",
      "content": "<p>I see similar low AP for calcification, atelectasis, nodule mass, other lesion cases. </p>\n<ul>\n<li>Calcification: I think the reason is that I trained my model on 512*512 images. Because of downsizing images, in low resolution images information about comparatively smaller calcification boxes is lost and the AP for that becomes small per as well. What is the size of your images that you are using for training? </li>\n<li>Atelectasis :- by default has smaller detection <br>\nI am still unsure about how we get low AP for other cases. </li>\n</ul>",
      "votes": 1,
      "replies": [
        {
          "id": 1197540,
          "author_name": "JIN",
          "author_url": "",
          "post_date": "2021-02-12T08:23:29.963000",
          "content": "<p>Thank you for the information.<br>\nas you mentioned, there are very small object in some case(512x512).<br>\n<img src=\"https://drive.google.com/uc?export=view&amp;id=1iP1R0NvrS-CMcZSoU95uDC2TfBsT9lEl\" alt=\"\"><br>\nI need to tune <code>anchor_size</code> if I still want to use 512x512 images but I'm not sure is it critical issue for AP</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1196610,
      "author_name": "Manh Lab",
      "author_url": "",
      "post_date": "2021-02-11T14:03:00.497000",
      "content": "<p>i used detectoRS too, but inference look not good.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1195826,
      "author_name": "Sunghyun Jun",
      "author_url": "",
      "post_date": "2021-02-11T05:17:51.070000",
      "content": "<p>In my case, Atelectasis, Calcification, Nodule, Other_lesion have low AP. As you know, bbox area of Calcification, Nodule are quite small. But I don't have any ideas about Atelectasis.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1196444,
          "author_name": "Hannes Öhler",
          "author_url": "",
          "post_date": "2021-02-11T12:12:43.347000",
          "content": "<p>For Atelectasis the reason is maybe that there are only few cases?</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1196519,
          "author_name": "JIN",
          "author_url": "",
          "post_date": "2021-02-11T13:03:29.767000",
          "content": "<p>It might be right, there are very few images and labels for Atelectasis in train dataset<br>\nonly 279 annotations and 186 images for Atelectasis :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1658187,
      "author_name": "Yasir Irfan",
      "author_url": "",
      "post_date": "2022-01-20T20:12:40.057000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/jinssaa\" target=\"_blank\">@jinssaa</a> ! Can you share the code you used to compute per class Average Precision? <br>\nI tried to modify coco_eval.py like this ( <a href=\"https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes\" target=\"_blank\">https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes</a> ) but it didn't work for me. Thank you.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1195742": "Hi guys, Is there anyone who checked AP per Class ? It seems like that some classes show very low AP. \n\nI used \n- 512x512\n- WBF before training\n- DetectoRS(r50)\n- MultiLabel Stratifiedkfold : 5fold\n- mAP@0.5 : 0.3420(fold_0)\n![](https://drive.google.com/uc?export=view&id=1SX4haHfjNrAqcmtFaZrYBBOrA-4k6Gd_\")\n\nI haven't made inference code, so I'll update LB score soon.\n\n-------- update 02.23 ------------\nCV \n - mAP@0.5 : 0.3470(fold_0)\n - mAP@0.4 : 0.4060(fold_0)\n\n![](https://drive.google.com/uc?export=view&id=1RVGxu7SeE0oJdIfZ_xOTSswSCgSdGvqO\")\n\nLB\n - 0.169\n- confidence score : 0.001\n- nms_iou : 0.5\n",
    "1197033": "I see similar low AP for calcification, atelectasis, nodule mass, other lesion cases. \n- Calcification: I think the reason is that I trained my model on 512*512 images. Because of downsizing images, in low resolution images information about comparatively smaller calcification boxes is lost and the AP for that becomes small per as well. What is the size of your images that you are using for training? \n- Atelectasis :- by default has smaller detection \nI am still unsure about how we get low AP for other cases. ",
    "1196610": "i used detectoRS too, but inference look not good.",
    "1195826": "In my case, Atelectasis, Calcification, Nodule, Other_lesion have low AP. As you know, bbox area of Calcification, Nodule are quite small. But I don't have any ideas about Atelectasis.",
    "1658187": "Hi @jinssaa ! Can you share the code you used to compute per class Average Precision? \nI tried to modify coco_eval.py like this ( https://stackoverflow.com/questions/56247323/coco-api-evaluation-for-subset-of-classes ) but it didn't work for me. Thank you."
  }
}