Past 2 days i was trying to add CTC loss to markwijkhuizen public notebook.
\nIt gets best metric in CV(.93) within just 2 epochs, which i strongly suspect shows an issue of target leakage, but i didn't change much except loss function.
\nJust curious,has anyone experienced the same issues.
Update:- Notebook to support the claims.
", "messages": [ { "id": 2414455, "postDate": "2023-08-29T16:35:56.020Z", "content": "Hi @level14taken,
\nI have checked your code, and it just seems that your calculate_val_lev function is miscalculating the levenshtein distance, as the ctc loss seems to be right,
\nFor extra checking, you can always print some predictions as a sanity check.
\nHope I have helped, feel free to ask any further questions.
Hi @fanaticlizard,
\nThanks for the reply.
\nCan you exactly point out what looks like a bug to you, i can't quite figure it out.
\nAnd also if you look at the last cell in the notebook you can see than jumbled targets has loss similar to that of normal data, which is definitely not normal and it has nothing to do with lev callback.
Hi again @level14taken,
\nThe jumbled targets have loss similar to that of normal data, because the model is pretty bad anyway, meaning if you try to print outputs of the model it will probably be random characters, so it doesn't matter if the targets are jumbled or not.
\nI can't be sure about the bug in the calculate_val_lev function, if you have the weights saved you can try to rerun the model with the weights loaded and print some (preds, target) pairs in the calculate_val_lev function.
", "rawMarkdown": "Hi again @level14taken,\n\nThe jumbled targets have loss similar to that of normal data, because the model is pretty bad anyway, meaning if you try to print outputs of the model it will probably be random characters, so it doesn't matter if the targets are jumbled or not.\n\nI can't be sure about the bug in the calculate_val_lev function, if you have the weights saved you can try to rerun the model with the weights loaded and print some (preds, target) pairs in the calculate_val_lev function." }, { "id": 2416402, "postDate": "2023-08-31T02:49:13.343Z", "content": "But,CTC loss of 24 is very good for a model on random data, it should be around 100 in my opinion.
\nAnyways i loaded a trained model weights and printed outputs this time, check here.
Past 2 days i was trying to add CTC loss to markwijkhuizen public notebook.
\nIt gets best metric in CV(.93) within just 2 epochs, which i strongly suspect shows an issue of target leakage, but i didn't change much except loss function.
\nJust curious,has anyone experienced the same issues.
Update:- Notebook to support the claims.
", "rawMarkdown": "Past 2 days i was trying to add CTC loss to [markwijkhuizen public notebook](https://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference).\nIt gets best metric in CV(.93) within just 2 epochs, which i strongly suspect shows an issue of target leakage, but i didn't change much except loss function. \nJust curious,has anyone experienced the same issues. \n\nUpdate:- [Notebook](https://www.kaggle.com/code/level14taken/notebook0f2da16929/notebook) to support the claims." } ], "comments": [ { "id": 2414455, "author_name": "Fanatic Lizard", "author_url": "", "post_date": "2023-08-29T16:35:56.020000", "content": "Hi @level14taken,
\nI have checked your code, and it just seems that your calculate_val_lev function is miscalculating the levenshtein distance, as the ctc loss seems to be right,
\nFor extra checking, you can always print some predictions as a sanity check.
\nHope I have helped, feel free to ask any further questions.
Hi @fanaticlizard,
\nThanks for the reply.
\nCan you exactly point out what looks like a bug to you, i can't quite figure it out.
\nAnd also if you look at the last cell in the notebook you can see than jumbled targets has loss similar to that of normal data, which is definitely not normal and it has nothing to do with lev callback.
Hi again @level14taken,
\nThe jumbled targets have loss similar to that of normal data, because the model is pretty bad anyway, meaning if you try to print outputs of the model it will probably be random characters, so it doesn't matter if the targets are jumbled or not.
\nI can't be sure about the bug in the calculate_val_lev function, if you have the weights saved you can try to rerun the model with the weights loaded and print some (preds, target) pairs in the calculate_val_lev function.
", "votes": 0, "replies": [] }, { "id": 2416402, "author_name": "losingself", "author_url": "", "post_date": "2023-08-31T02:49:13.343000", "content": "But,CTC loss of 24 is very good for a model on random data, it should be around 100 in my opinion.
\nAnyways i loaded a trained model weights and printed outputs this time, check here.