Hi to all!
\nI haven't been able to really compete in the last while, but I'll share the changes to the public notebook: https://www.kaggle.com/code/saidineshpola/aslfr-ctc-based-on-prev-comp-1st- place (special thanks to the author). Only two changes, I reduced the drop_rate and added another Dropout. Here's what the function looks like
def get_model(dim = 384,num_blocks = 6,drop_rate = 0.35):
\n inp = tf.keras.Input(INPUT_SHAPE)
\n x = tf.keras.layers.Masking(mask_value=0.0)(inp)
\n x = tf.keras.layers.Dense(dim, use_bias=False, name='stem_conv')(x)
\n pe = tf.cast(positional_encoding(INPUT_SHAPE[0], dim), dtype=x.dtype)
\n x = x + pe
\n x = tf.keras.layers.BatchNormalization(momentum=0.95,name='stem_bn')(x)
i range(num_blocks):\n x = (x)\n x = (x)\n x = (x)\n x = (x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.,name='classifier')(x)\n\n model = tf.keras.\n\n loss = CTCLoss\n\n # Adam Optimizer\n optimizer = tfa.optimizers.\n optimizer = tfa.optimizers.\n\n model.compile(loss=loss, optimizer=optimizer)\n\n return model\nThe changes are really minor, but worked because the base rate of 0.4 was too big, it slowed learning too much, and I knew from the previous competition that increasing the number of epochs could improve the result. Here, my logic was to simplify learning instead of increasing the number of epochs.
\nGood luck to all of you and a peaceful sky above your head!
Very interesting and clever approach. Can you share the number of epochs.
", "rawMarkdown": "Very interesting and clever approach. Can you share the number of epochs.", "votes": 1, "replies": [ { "id": 2410278, "postDate": "2023-08-26T19:22:06.483Z", "content": "Of course, training lasted 50 epochs
", "rawMarkdown": "Of course, training lasted 50 epochs" } ] }, { "id": 2409242, "postDate": "2023-08-26T05:53:21.907Z", "content": "Hi to all!
\nI haven't been able to really compete in the last while, but I'll share the changes to the public notebook: https://www.kaggle.com/code/saidineshpola/aslfr-ctc-based-on-prev-comp-1st- place (special thanks to the author). Only two changes, I reduced the drop_rate and added another Dropout. Here's what the function looks like
def get_model(dim = 384,num_blocks = 6,drop_rate = 0.35):
\n inp = tf.keras.Input(INPUT_SHAPE)
\n x = tf.keras.layers.Masking(mask_value=0.0)(inp)
\n x = tf.keras.layers.Dense(dim, use_bias=False, name='stem_conv')(x)
\n pe = tf.cast(positional_encoding(INPUT_SHAPE[0], dim), dtype=x.dtype)
\n x = x + pe
\n x = tf.keras.layers.BatchNormalization(momentum=0.95,name='stem_bn')(x)
i range(num_blocks):\n x = (x)\n x = (x)\n x = (x)\n x = (x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.(x)\n x = tf.keras.layers.,name='classifier')(x)\n\n model = tf.keras.\n\n loss = CTCLoss\n\n # Adam Optimizer\n optimizer = tfa.optimizers.\n optimizer = tfa.optimizers.\n\n model.compile(loss=loss, optimizer=optimizer)\n\n return model\nThe changes are really minor, but worked because the base rate of 0.4 was too big, it slowed learning too much, and I knew from the previous competition that increasing the number of epochs could improve the result. Here, my logic was to simplify learning instead of increasing the number of epochs.
\nGood luck to all of you and a peaceful sky above your head!
Very interesting and clever approach. Can you share the number of epochs.
", "votes": 1, "replies": [ { "id": 2410278, "author_name": "Andrij", "author_url": "", "post_date": "2023-08-26T19:22:06.483000", "content": "Of course, training lasted 50 epochs
", "votes": 0, "replies": [] } ] }, { "id": 2426092, "author_name": "", "author_url": "", "post_date": "2023-09-06T11:57:26.797000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2410119": "Very interesting and clever approach. Can you share the number of epochs.", "2409242": "Hi to all!\nI haven't been able to really compete in the last while, but I'll share the changes to the public notebook: https://www.kaggle.com/code/saidineshpola/aslfr-ctc-based-on-prev-comp-1st- place (special thanks to the author). Only two changes, I reduced the drop_rate and added another Dropout. Here's what the function looks like\n\n\ndef get_model(dim = 384,num_blocks = 6,drop_rate = 0.35):\n inp = tf.keras.Input(INPUT_SHAPE)\n x = tf.keras.layers.Masking(mask_value=0.0)(inp)\n x = tf.keras.layers.Dense(dim, use_bias=False, name='stem_conv')(x)\n pe = tf.cast(positional_encoding(INPUT_SHAPE[0], dim), dtype=x.dtype)\n x = x + pe\n x = tf.keras.layers.BatchNormalization(momentum=0.95,name='stem_bn')(x)\n \n for i in range(num_blocks):\n x = Conv1DBlock(dim, 11, drop_rate=drop_rate)(x)\n x = Conv1DBlock(dim, 5, drop_rate=drop_rate)(x)\n x = Conv1DBlock(dim, 3, drop_rate=drop_rate)(x)\n x = TransformerBlock(dim, expand=2)(x)\n x = tf.keras.layers.Dropout(drop_rate)(x)\n x = tf.keras.layers.Dense(dim*2,activation='relu',name='top_conv')(x)\n x = tf.keras.layers.Dropout(drop_rate)(x)\n x = tf.keras.layers.Dense(len(char_to_num),name='classifier')(x)\n\n model = tf.keras.Model(inp, x)\n\n loss = CTCLoss\n \n # Adam Optimizer\n optimizer = tfa.optimizers.RectifiedAdam(sma_threshold=4)\n optimizer = tfa.optimizers.Lookahead(optimizer, sync_period=5)\n\n model.compile(loss=loss, optimizer=optimizer)\n\n return model\n\nThe changes are really minor, but worked because the base rate of 0.4 was too big, it slowed learning too much, and I knew from the previous competition that increasing the number of epochs could improve the result. Here, my logic was to simplify learning instead of increasing the number of epochs.\nGood luck to all of you and a peaceful sky above your head!\n\n", "2426092": "" } }