{
  "id": 444857,
  "title": "CosineDecay error in Tensorflow",
  "url": "/competitions/rsna-2023-abdominal-trauma-detection/discussion/444857",
  "author_name": "Ayush Yajnik",
  "post_date": "2023-10-04T00:27:43.954000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am introducing CosineDecay in my model but I am getting the below error:</p>\n<p>AttributeError: module 'keras.optimizers.schedules' has no attribute 'CosineDecay'</p>\n<p>Any idea why I am getting this one?</p>",
  "messages": [
    {
      "id": 2466777,
      "postDate": "2023-10-04T04:57:43.157Z",
      "content": "<p>Looks like your missing tensorflow at the front… tf.keras.optimizers.schedules.CosineDecay is where I found the class. Posting your code might also help.</p>",
      "rawMarkdown": "Looks like your missing tensorflow at the front... tf.keras.optimizers.schedules.CosineDecay is where I found the class. Posting your code might also help.",
      "votes": 1,
      "replies": [
        {
          "id": 2467257,
          "postDate": "2023-10-04T11:47:10.183Z",
          "content": "<p>Hey Graham,</p>\n<p>this is what i am using:</p>\n<p>def build_model(warmup_steps, decay_steps):</p>\n<pre><code>\ninputs = keras.Input(=config.IMAGE_SIZE + [3,], =config.BATCH_SIZE)\n\n\nbackbone = keras_cv.models.ResNetBackbone.from_preset()\nbackbone.include_rescaling = \nx = backbone(inputs)\n\n\ngap = keras.layers.GlobalAveragePooling2D()\nx = gap(x)\n\n\nx_bowel = keras.layers.Dense(32, =)(x)\nx_extra = keras.layers.Dense(32, =)(x)\nx_liver = keras.layers.Dense(32, =)(x)\nx_kidney = keras.layers.Dense(32, =)(x)\nx_spleen = keras.layers.Dense(32, =)(x)\n\n\nout_bowel = keras.layers.Dense(1, =, =)(x_bowel) # use sigmoid  convert predictions  [0-1]\nout_extra = keras.layers.Dense(1, =, =)(x_extra) # use sigmoid  convert predictions  [0-1]\nout_liver = keras.layers.Dense(3, =, =)(x_liver) # use softmax  the liver head\nout_kidney = keras.layers.Dense(3, =, =)(x_kidney) # use softmax  the kidney head\nout_spleen = keras.layers.Dense(3, =, =)(x_spleen) # use softmax  the spleen head\n\n\noutputs = [out_bowel, out_extra, out_liver, out_kidney, out_spleen]\n\n\n()\nmodel = keras.Model(=inputs, =outputs)\n\n\ncosine_decay = keras.optimizers.schedules.CosineDecay(\n    =1e-4,\n    =decay_steps,\n    =0.0,\n    =1e-3,\n    =warmup_steps)\n\n\noptimizer = keras.optimizers.Adam(=cosine_decay)\nloss = {\n    :keras.losses.BinaryCrossentropy(),\n    :keras.losses.BinaryCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n}\nmetrics = {\n    :[],\n    :[],\n    :[],\n    :[],\n    :[],\n}\n()\nmodel.compile(\n    =optimizer,\n    =loss,\n    =metrics)\n\nreturn model\n</code></pre>",
          "rawMarkdown": "Hey Graham,\n\nthis is what i am using:\n\ndef build_model(warmup_steps, decay_steps):\n    \n    # define input\n    inputs = keras.Input(shape=config.IMAGE_SIZE + [3,], batch_size=config.BATCH_SIZE)\n    \n    # define backbone\n    backbone = keras_cv.models.ResNetBackbone.from_preset(\"resnet50_imagenet\")\n    backbone.include_rescaling = False\n    x = backbone(inputs)\n    \n    # GAP to get the activation maps\n    gap = keras.layers.GlobalAveragePooling2D()\n    x = gap(x)\n    \n    # Define 'necks' for each head\n    x_bowel = keras.layers.Dense(32, activation='silu')(x)\n    x_extra = keras.layers.Dense(32, activation='silu')(x)\n    x_liver = keras.layers.Dense(32, activation='silu')(x)\n    x_kidney = keras.layers.Dense(32, activation='silu')(x)\n    x_spleen = keras.layers.Dense(32, activation='silu')(x)\n    \n    # Define heads\n    out_bowel = keras.layers.Dense(1, name='bowel', activation='sigmoid')(x_bowel) # use sigmoid to convert predictions to [0-1]\n    out_extra = keras.layers.Dense(1, name='extra', activation='sigmoid')(x_extra) # use sigmoid to convert predictions to [0-1]\n    out_liver = keras.layers.Dense(3, name='liver', activation='softmax')(x_liver) # use softmax for the liver head\n    out_kidney = keras.layers.Dense(3, name='kidney', activation='softmax')(x_kidney) # use softmax for the kidney head\n    out_spleen = keras.layers.Dense(3, name='spleen', activation='softmax')(x_spleen) # use softmax for the spleen head\n    \n    # concatenate the outputs\n    outputs = [out_bowel, out_extra, out_liver, out_kidney, out_spleen]\n    \n    # create model\n    print(\"[INFO] Building the model...\")\n    model = keras.Model(inputs=inputs, outputs=outputs)\n    \n    # cosine decay\n    cosine_decay = keras.optimizers.schedules.CosineDecay(\n        initial_learning_rate=1e-4,\n        decay_steps=decay_steps,\n        alpha=0.0,\n        warmup_target=1e-3,\n        warmup_steps=warmup_steps)\n    \n    # compile the model, Adam is a SGD algorithm\n    optimizer = keras.optimizers.Adam(learning_rate=cosine_decay)\n    loss = {\n        \"bowel\":keras.losses.BinaryCrossentropy(),\n        \"extra\":keras.losses.BinaryCrossentropy(),\n        \"liver\":keras.losses.CategoricalCrossentropy(),\n        \"kidney\":keras.losses.CategoricalCrossentropy(),\n        \"spleen\":keras.losses.CategoricalCrossentropy(),\n    }\n    metrics = {\n        \"bowel\":[\"accuracy\"],\n        \"extra\":[\"accuracy\"],\n        \"liver\":[\"accuracy\"],\n        \"kidney\":[\"accuracy\"],\n        \"spleen\":[\"accuracy\"],\n    }\n    print(\"[INFO] Compiling the model...\")\n    model.compile(\n        optimizer=optimizer,\n        loss=loss,\n        metrics=metrics)\n    \n    return model"
        }
      ]
    },
    {
      "id": 2466574,
      "postDate": "2023-10-04T00:27:43.953Z",
      "content": "<p>I am introducing CosineDecay in my model but I am getting the below error:</p>\n<p>AttributeError: module 'keras.optimizers.schedules' has no attribute 'CosineDecay'</p>\n<p>Any idea why I am getting this one?</p>",
      "rawMarkdown": "I am introducing CosineDecay in my model but I am getting the below error:\n\n AttributeError: module 'keras.optimizers.schedules' has no attribute 'CosineDecay'\n\nAny idea why I am getting this one?\n",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 2466777,
      "author_name": "graham broughton",
      "author_url": "",
      "post_date": "2023-10-04T04:57:43.157000",
      "content": "<p>Looks like your missing tensorflow at the front… tf.keras.optimizers.schedules.CosineDecay is where I found the class. Posting your code might also help.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2467257,
          "author_name": "Ayush Yajnik",
          "author_url": "",
          "post_date": "2023-10-04T11:47:10.183000",
          "content": "<p>Hey Graham,</p>\n<p>this is what i am using:</p>\n<p>def build_model(warmup_steps, decay_steps):</p>\n<pre><code>\ninputs = keras.Input(=config.IMAGE_SIZE + [3,], =config.BATCH_SIZE)\n\n\nbackbone = keras_cv.models.ResNetBackbone.from_preset()\nbackbone.include_rescaling = \nx = backbone(inputs)\n\n\ngap = keras.layers.GlobalAveragePooling2D()\nx = gap(x)\n\n\nx_bowel = keras.layers.Dense(32, =)(x)\nx_extra = keras.layers.Dense(32, =)(x)\nx_liver = keras.layers.Dense(32, =)(x)\nx_kidney = keras.layers.Dense(32, =)(x)\nx_spleen = keras.layers.Dense(32, =)(x)\n\n\nout_bowel = keras.layers.Dense(1, =, =)(x_bowel) # use sigmoid  convert predictions  [0-1]\nout_extra = keras.layers.Dense(1, =, =)(x_extra) # use sigmoid  convert predictions  [0-1]\nout_liver = keras.layers.Dense(3, =, =)(x_liver) # use softmax  the liver head\nout_kidney = keras.layers.Dense(3, =, =)(x_kidney) # use softmax  the kidney head\nout_spleen = keras.layers.Dense(3, =, =)(x_spleen) # use softmax  the spleen head\n\n\noutputs = [out_bowel, out_extra, out_liver, out_kidney, out_spleen]\n\n\n()\nmodel = keras.Model(=inputs, =outputs)\n\n\ncosine_decay = keras.optimizers.schedules.CosineDecay(\n    =1e-4,\n    =decay_steps,\n    =0.0,\n    =1e-3,\n    =warmup_steps)\n\n\noptimizer = keras.optimizers.Adam(=cosine_decay)\nloss = {\n    :keras.losses.BinaryCrossentropy(),\n    :keras.losses.BinaryCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n    :keras.losses.CategoricalCrossentropy(),\n}\nmetrics = {\n    :[],\n    :[],\n    :[],\n    :[],\n    :[],\n}\n()\nmodel.compile(\n    =optimizer,\n    =loss,\n    =metrics)\n\nreturn model\n</code></pre>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2466777": "Looks like your missing tensorflow at the front... tf.keras.optimizers.schedules.CosineDecay is where I found the class. Posting your code might also help.",
    "2466574": "I am introducing CosineDecay in my model but I am getting the below error:\n\n AttributeError: module 'keras.optimizers.schedules' has no attribute 'CosineDecay'\n\nAny idea why I am getting this one?\n"
  }
}