{
  "id": 386801,
  "title": "[new paper] Scaling Vision Transformers to 22 Billion Parameters",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/386801",
  "author_name": "hengck23",
  "post_date": "2023-02-14T16:13:34.475000",
  "votes": 10,
  "comment_count": 9,
  "views": 0,
  "content": "<p>[1] Scaling Vision Transformers to 22 Billion Parameters- google,2023<br>\n<a href=\"https://arxiv.org/abs/2302.05442\" target=\"_blank\">https://arxiv.org/abs/2302.05442</a></p>\n<p>i am interested in the zero-shot results.</p>\n<p>\"ViT-22B demonstrates the potential for \"LLM-like\" scaling in vision, …\"<br>\nmaybe one day, it is difficult to do kaggle as a hobby without poweful gpu.</p>\n<p>as a reference:<br>\nGPT-3 has a 175 billion parameters. <br>\nGPT-2 was 1.5 billion parameters.<br>\nResNet-50 model has over 23 million trainable parameters.</p>",
  "messages": [
    {
      "id": 2143918,
      "postDate": "2023-02-14T16:13:34.477Z",
      "content": "<p>[1] Scaling Vision Transformers to 22 Billion Parameters- google,2023<br>\n<a href=\"https://arxiv.org/abs/2302.05442\" target=\"_blank\">https://arxiv.org/abs/2302.05442</a></p>\n<p>i am interested in the zero-shot results.</p>\n<p>\"ViT-22B demonstrates the potential for \"LLM-like\" scaling in vision, …\"<br>\nmaybe one day, it is difficult to do kaggle as a hobby without poweful gpu.</p>\n<p>as a reference:<br>\nGPT-3 has a 175 billion parameters. <br>\nGPT-2 was 1.5 billion parameters.<br>\nResNet-50 model has over 23 million trainable parameters.</p>",
      "rawMarkdown": "[1] Scaling Vision Transformers to 22 Billion Parameters- google,2023\nhttps://arxiv.org/abs/2302.05442\n\ni am interested in the zero-shot results.\n\n\"ViT-22B demonstrates the potential for \"LLM-like\" scaling in vision, ...\"\nmaybe one day, it is difficult to do kaggle as a hobby without poweful gpu.\n\nas a reference:\nGPT-3 has a 175 billion parameters. \nGPT-2 was 1.5 billion parameters.\nResNet-50 model has over 23 million trainable parameters.\n",
      "votes": 10
    },
    {
      "id": 2144246,
      "postDate": "2023-02-14T21:01:47.380Z",
      "content": "<p>Altman made an interesting point - at first we thought it'd be the manual labor type jobs that would get replaced by AI.  Like checkout lines and truck drivers, but maybe it'll be the other way around.  </p>\n<p>Hmm, wonder how good my carpentry skills are…</p>",
      "rawMarkdown": "Altman made an interesting point - at first we thought it'd be the manual labor type jobs that would get replaced by AI.  Like checkout lines and truck drivers, but maybe it'll be the other way around.  \n\nHmm, wonder how good my carpentry skills are...",
      "votes": 1
    },
    {
      "id": 2144085,
      "postDate": "2023-02-14T18:18:33.360Z",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing. I have not read the paper yet but can one use this for zero-shot results with Colab or Kaggle kernel?</p>\n<p>Also, do you think it is too late to join this RSNA competition? I only use Colab and Kaggle kernel:-).</p>",
      "rawMarkdown": "Thanks @hengck23 for sharing. I have not read the paper yet but can one use this for zero-shot results with Colab or Kaggle kernel?\n\nAlso, do you think it is too late to join this RSNA competition? I only use Colab and Kaggle kernel:-).",
      "votes": 1,
      "replies": [
        {
          "id": 2144224,
          "postDate": "2023-02-14T20:33:15.907Z",
          "content": "<p>Given your background, I guess that 15 days is Ok to enter a competition, unless you only target the gold zone.</p>\n<p>If you are a tensorflow user, <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu?scriptVersionId=117309374\" target=\"_blank\">this kernel</a> is a good starter.</p>\n<p>If a pytorch user, you can try <a href=\"https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403\" target=\"_blank\">that one</a>.</p>\n<p>Reaching at least the bronze zone is feasible in that timeframe. Except that defining a reliable CV scheme may be somewhat tricky in this comp: there's a lot of variability reported from single fold submissions and LB scores. I guess that the leaderboard will be somewhat shaky in the end …</p>",
          "rawMarkdown": "Given your background, I guess that 15 days is Ok to enter a competition, unless you only target the gold zone.\n\nIf you are a tensorflow user, [this kernel](https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu?scriptVersionId=117309374) is a good starter.\n\nIf a pytorch user, you can try [that one](https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403).\n\nReaching at least the bronze zone is feasible in that timeframe. Except that defining a reliable CV scheme may be somewhat tricky in this comp: there's a lot of variability reported from single fold submissions and LB scores. I guess that the leaderboard will be somewhat shaky in the end ...",
          "votes": 1,
          "replies": [
            {
              "id": 2146641,
              "postDate": "2023-02-16T03:02:45.247Z",
              "content": "<p>Thanks <a href=\"https://www.kaggle.com/fabiendaniel\" target=\"_blank\">@fabiendaniel</a>. I joined and made my 1st submission yesterday.</p>",
              "rawMarkdown": "Thanks @fabiendaniel. I joined and made my 1st submission yesterday.",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2144034,
      "postDate": "2023-02-14T17:28:31.183Z",
      "content": "<p>time will pass, prices will fall, kaggle will remain, well known rabbit hole :)</p>",
      "rawMarkdown": "time will pass, prices will fall, kaggle will remain, well known rabbit hole :)",
      "votes": 2
    },
    {
      "id": 2145393,
      "postDate": "2023-02-15T02:41:28.153Z",
      "content": "<p>Kaggle competition produces huge amount of carbon dioxide.</p>",
      "rawMarkdown": "Kaggle competition produces huge amount of carbon dioxide.",
      "votes": 1,
      "replies": [
        {
          "id": 2150682,
          "postDate": "2023-02-19T13:58:52.940Z",
          "rawMarkdown": "",
          "votes": 1,
          "isDeleted": true
        },
        {
          "id": 2153374,
          "postDate": "2023-02-21T11:50:18.610Z",
          "content": "<p><a href=\"https://sustainability.google/carbon-free/#home\" target=\"_blank\">https://sustainability.google/carbon-free/#home</a></p>",
          "rawMarkdown": "https://sustainability.google/carbon-free/#home",
          "votes": 1
        }
      ]
    },
    {
      "id": 2144268,
      "postDate": "2023-02-14T21:50:17.333Z",
      "content": "<p>22B is couple bytes behind kaggle env memory :)</p>",
      "rawMarkdown": "22B is couple bytes behind kaggle env memory :)"
    }
  ],
  "comments": [
    {
      "id": 2144246,
      "author_name": "@kaggleqrdl",
      "author_url": "",
      "post_date": "2023-02-14T21:01:47.380000",
      "content": "<p>Altman made an interesting point - at first we thought it'd be the manual labor type jobs that would get replaced by AI.  Like checkout lines and truck drivers, but maybe it'll be the other way around.  </p>\n<p>Hmm, wonder how good my carpentry skills are…</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2144085,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2023-02-14T18:18:33.360000",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing. I have not read the paper yet but can one use this for zero-shot results with Colab or Kaggle kernel?</p>\n<p>Also, do you think it is too late to join this RSNA competition? I only use Colab and Kaggle kernel:-).</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2144224,
          "author_name": "FabienDaniel",
          "author_url": "",
          "post_date": "2023-02-14T20:33:15.907000",
          "content": "<p>Given your background, I guess that 15 days is Ok to enter a competition, unless you only target the gold zone.</p>\n<p>If you are a tensorflow user, <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu?scriptVersionId=117309374\" target=\"_blank\">this kernel</a> is a good starter.</p>\n<p>If a pytorch user, you can try <a href=\"https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403\" target=\"_blank\">that one</a>.</p>\n<p>Reaching at least the bronze zone is feasible in that timeframe. Except that defining a reliable CV scheme may be somewhat tricky in this comp: there's a lot of variability reported from single fold submissions and LB scores. I guess that the leaderboard will be somewhat shaky in the end …</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2146641,
              "author_name": "YaGana Sheriff-Hussaini",
              "author_url": "",
              "post_date": "2023-02-16T03:02:45.247000",
              "content": "<p>Thanks <a href=\"https://www.kaggle.com/fabiendaniel\" target=\"_blank\">@fabiendaniel</a>. I joined and made my 1st submission yesterday.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2144034,
      "author_name": "Eleftherios Fanioudakis",
      "author_url": "",
      "post_date": "2023-02-14T17:28:31.183000",
      "content": "<p>time will pass, prices will fall, kaggle will remain, well known rabbit hole :)</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2145393,
      "author_name": "ECO",
      "author_url": "",
      "post_date": "2023-02-15T02:41:28.153000",
      "content": "<p>Kaggle competition produces huge amount of carbon dioxide.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2150682,
          "author_name": "",
          "author_url": "",
          "post_date": "2023-02-19T13:58:52.940000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2153374,
          "author_name": "AmorfEvo",
          "author_url": "",
          "post_date": "2023-02-21T11:50:18.610000",
          "content": "<p><a href=\"https://sustainability.google/carbon-free/#home\" target=\"_blank\">https://sustainability.google/carbon-free/#home</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2144268,
      "author_name": "A.P.",
      "author_url": "",
      "post_date": "2023-02-14T21:50:17.333000",
      "content": "<p>22B is couple bytes behind kaggle env memory :)</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2143918": "[1] Scaling Vision Transformers to 22 Billion Parameters- google,2023\nhttps://arxiv.org/abs/2302.05442\n\ni am interested in the zero-shot results.\n\n\"ViT-22B demonstrates the potential for \"LLM-like\" scaling in vision, ...\"\nmaybe one day, it is difficult to do kaggle as a hobby without poweful gpu.\n\nas a reference:\nGPT-3 has a 175 billion parameters. \nGPT-2 was 1.5 billion parameters.\nResNet-50 model has over 23 million trainable parameters.\n",
    "2144246": "Altman made an interesting point - at first we thought it'd be the manual labor type jobs that would get replaced by AI.  Like checkout lines and truck drivers, but maybe it'll be the other way around.  \n\nHmm, wonder how good my carpentry skills are...",
    "2144085": "Thanks @hengck23 for sharing. I have not read the paper yet but can one use this for zero-shot results with Colab or Kaggle kernel?\n\nAlso, do you think it is too late to join this RSNA competition? I only use Colab and Kaggle kernel:-).",
    "2144034": "time will pass, prices will fall, kaggle will remain, well known rabbit hole :)",
    "2145393": "Kaggle competition produces huge amount of carbon dioxide.",
    "2144268": "22B is couple bytes behind kaggle env memory :)"
  }
}