So for some reason my yolov5 model only gets 0.065 while most of the public yolov5 kernels are getting 0.11+. So while I was investigating why, I came across some of these questions:
\n1) I am aware that yolo only saves your best weights (or checkpoints if you will), this is to avoid overfitting (I am assuming). So a question about this, if it only saves the weights when the validation loss decreases (right?) then the model can never overfit? Or can it? Cuz I trained mine for around 60 epochs while most notebooks are trained for about 30 (both with pre-trained weights) so am sensing it might have overfitted
\n2) Some notebooks only train the models on 1 fold (1 variation of cross-validation). This doesn't make much sense to me… Why would you have cross validation in your notebook if you are only training the model 1 time? I thought the point was to rotate on different folds and then average..? Is there something I am missing here? (for instance, this one https://www.kaggle.com/awsaf49/vinbigdata-cxr-ad-yolov5-14-class-train )
", "messages": [ { "id": 1228782, "postDate": "2021-03-06T18:57:40.773Z", "content": "There's the possibility that you're overfitting the validation data (I don't think that's necessarily the case, but it's possible). If you take a model and change things around somewhat randomly for a very long time and save the model whenever the validation performance improved, you're essentially fitting to the validation data (in a very inefficient way).
\nThe main reason notebooks would only fit one fold is usually that it would either not finish within the time a notebook can run on Kaggle, or the author was short on GPU / TPU time. Doing a multiple fold CV is, of course, usually the sensible way to go (unless the dataset is huge) - doing one out of, say, five folds is really the same as a 80:20 train-test-split.
\nA proper CV has multiple benefits:
\nSo for some reason my yolov5 model only gets 0.065 while most of the public yolov5 kernels are getting 0.11+. So while I was investigating why, I came across some of these questions:
\n1) I am aware that yolo only saves your best weights (or checkpoints if you will), this is to avoid overfitting (I am assuming). So a question about this, if it only saves the weights when the validation loss decreases (right?) then the model can never overfit? Or can it? Cuz I trained mine for around 60 epochs while most notebooks are trained for about 30 (both with pre-trained weights) so am sensing it might have overfitted
\n2) Some notebooks only train the models on 1 fold (1 variation of cross-validation). This doesn't make much sense to me… Why would you have cross validation in your notebook if you are only training the model 1 time? I thought the point was to rotate on different folds and then average..? Is there something I am missing here? (for instance, this one https://www.kaggle.com/awsaf49/vinbigdata-cxr-ad-yolov5-14-class-train )
", "rawMarkdown": "So for some reason my yolov5 model only gets 0.065 while most of the public yolov5 kernels are getting 0.11+. So while I was investigating why, I came across some of these questions:\n\n1) I am aware that yolo only saves your best weights (or checkpoints if you will), this is to avoid overfitting (I am assuming). So a question about this, if it only saves the weights when the validation loss decreases (right?) then the model can never overfit? Or can it? Cuz I trained mine for around 60 epochs while most notebooks are trained for about 30 (both with pre-trained weights) so am sensing it might have overfitted\n\n2) Some notebooks only train the models on 1 fold (1 variation of cross-validation). This doesn't make much sense to me... Why would you have cross validation in your notebook if you are only training the model 1 time? I thought the point was to rotate on different folds and then average..? Is there something I am missing here? (for instance, this one https://www.kaggle.com/awsaf49/vinbigdata-cxr-ad-yolov5-14-class-train )", "votes": 1 }, { "id": 1227607, "postDate": "2021-03-05T17:13:16.540Z", "content": "Probably you need to check again your data. I train yolov5x with default config can get ~ 0.12x
", "rawMarkdown": "Probably you need to check again your data. I train yolov5x with default config can get ~ 0.12x", "votes": 2 }, { "id": 1239362, "postDate": "2021-03-15T16:46:45.437Z", "content": "Saving intermediate checkpoints gives you a few benefits:
\nResilience: If you are training for a very long time, or doing distributed training on many machines, the likelihood of machine failure increases. If a machine fails, TensorFlow can resume from the last saved checkpoint instead of having to start from scratch. This behavior is automatic — TensorFlow looks for checkpoints and resumes from the last checkpoint.\nGeneralization: In general, the longer you train, the lower the loss on the training dataset. However, at some point, the error on the held-out, evaluation dataset might stop decreasing. If you have a very large model, and you are not doing sufficient regularization, the error on the evaluation dataset might even start to increase. If this happens, it can be helpful to go back and export the model that had the best validation error. This is also called early stopping because you could stop if you see the validation error start to increase. (A better idea is, of course, to decrease model complexity or increase the regularization so that this scenario doesn’t happen). The only way you can go back to the best validation error or do early stopping is if you have been periodically evaluating and checkpointing the model.\nTuneability: In a well-behaved training loop, gradient descent behaves such that you get to the neighborhood of the optimal error quickly on the basis of the majority of your data and then slowly converge towards the lowest error by optimizing on the corner cases. Now, imagine that you need to periodically retrain the model on fresh data \nThere's the possibility that you're overfitting the validation data (I don't think that's necessarily the case, but it's possible). If you take a model and change things around somewhat randomly for a very long time and save the model whenever the validation performance improved, you're essentially fitting to the validation data (in a very inefficient way).
\nThe main reason notebooks would only fit one fold is usually that it would either not finish within the time a notebook can run on Kaggle, or the author was short on GPU / TPU time. Doing a multiple fold CV is, of course, usually the sensible way to go (unless the dataset is huge) - doing one out of, say, five folds is really the same as a 80:20 train-test-split.
\nA proper CV has multiple benefits:
\nProbably you need to check again your data. I train yolov5x with default config can get ~ 0.12x
", "votes": 2, "replies": [] }, { "id": 1239362, "author_name": "Naim Mhedhbi", "author_url": "", "post_date": "2021-03-15T16:46:45.437000", "content": "Saving intermediate checkpoints gives you a few benefits:
\nResilience: If you are training for a very long time, or doing distributed training on many machines, the likelihood of machine failure increases. If a machine fails, TensorFlow can resume from the last saved checkpoint instead of having to start from scratch. This behavior is automatic — TensorFlow looks for checkpoints and resumes from the last checkpoint.\nGeneralization: In general, the longer you train, the lower the loss on the training dataset. However, at some point, the error on the held-out, evaluation dataset might stop decreasing. If you have a very large model, and you are not doing sufficient regularization, the error on the evaluation dataset might even start to increase. If this happens, it can be helpful to go back and export the model that had the best validation error. This is also called early stopping because you could stop if you see the validation error start to increase. (A better idea is, of course, to decrease model complexity or increase the regularization so that this scenario doesn’t happen). The only way you can go back to the best validation error or do early stopping is if you have been periodically evaluating and checkpointing the model.\nTuneability: In a well-behaved training loop, gradient descent behaves such that you get to the neighborhood of the optimal error quickly on the basis of the majority of your data and then slowly converge towards the lowest error by optimizing on the corner cases. Now, imagine that you need to periodically retrain the model on fresh data \n