Hello everybody!
\nFound an interesting thing.
\nI am exploring versions of https://www.kaggle.com/masterray notebook:
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3/notebook?scriptVersionId=109514186
And found out that Version 8 and Version 11 has the same code, but were run on CPU (Version 8 ) and GPU (Version 11) and had differente submissions results. And of course differente Public Scores (Version 8 - 0.640 vs Version 11 - 0.608).
\nI'm a newbie in Data Science. And got some questions:
\nIs it because of floating point operations are managed slightly differently on the cpu and gpu?
Is it always better to use CPU? :-)
Version 8
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3/notebook?scriptVersionId=109464914
\nCPU
\nPublic Score
\n0.640
Version 11
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3?scriptVersionId=109514186
\nGPU T4 x2
\nPublic Score
\n0.608
No, its not cause of CPU. It is just the random shuffle which leads to different solutions per run. I checked the code, the seed in only on torch stuff, but there is also this line:* kfold = StratifiedKFold(n_splits=nfold, random_state=42,** shuffle=True*) which leads to different models per run. The 64% was a lucky shot.
", "rawMarkdown": "No, its not cause of CPU. It is just the random shuffle which leads to different solutions per run. I checked the code, the seed in only on torch stuff, but there is also this line:* kfold = StratifiedKFold(n_splits=nfold, random_state=42,** shuffle=True**)* which leads to different models per run. The 64% was a lucky shot.", "votes": 6, "replies": [ { "id": 2011201, "postDate": "2022-10-31T12:21:27.273Z", "content": "Tnx for clear clarification!
", "rawMarkdown": "Tnx for clear clarification!" }, { "id": 2013026, "postDate": "2022-11-01T15:31:36.927Z", "content": "Can you explain further? If I want to conduct a controlled variable experiment and don't want any randomness to affect the CV, which random seeds should I fix?
\nI currently see two places in this program where I can fix random seeds, one is here.
\nkfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True)
\n I think it has been set to 42, which assumes that the samples for each fold in the Kfold are already fixed. I'm not sure if I'm right?
There is also one for torch, which I see the author has written as such.
\ntorch.manual_seed(42 + ifold + 1)
\nThis means that this seed is not fixed right?
So as long as it is changed to 42 here it will produce the exact same CV every time it is trained?
\nI'm new to Kaggle and I'm having to face such experiments for the first time, so if you could give some advice on controlling the variables, it would be greatly appreciated!
", "rawMarkdown": "Can you explain further? If I want to conduct a controlled variable experiment and don't want any randomness to affect the CV, which random seeds should I fix?\n\nI currently see two places in this program where I can fix random seeds, one is here.\n`kfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True)`\n I think it has been set to 42, which assumes that the samples for each fold in the Kfold are already fixed. I'm not sure if I'm right?\n\nThere is also one for torch, which I see the author has written as such.\n`torch.manual_seed(42 + ifold + 1)`\nThis means that this seed is not fixed right?\n\nSo as long as it is changed to 42 here it will produce the exact same CV every time it is trained?\n\nI'm new to Kaggle and I'm having to face such experiments for the first time, so if you could give some advice on controlling the variables, it would be greatly appreciated!" }, { "id": 2014119, "postDate": "2022-11-02T10:39:23.040Z", "content": "If you want a solid seed for reproducable results in order to perform fair comparisons you can seed everything with the following code:
\ndef seed_everything(seed_value):
\n random.seed(seed_value)
\n np.random.seed(seed_value)
\n torch.manual_seed(seed_value)
\n os.environ['PYTHONHASHSEED'] = str(seed_value)
\n if torch.cuda.is_available():
\n torch.cuda.manual_seed(seed_value)
\n torch.cuda.manual_seed_all(seed_value)
\n torch.backends.cudnn.deterministic = True
\n torch.backends.cudnn.benchmark = True
\nseed = 42
\nseed_everything(seed)
However, keep in mind that when you are training a NN, every small change can result in different models. In the kfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True), the shuffle term can lead to different models even if the folds are seeded/same. This is because, a different order which you feed the NN will result also in different NN even if the total tr data is the same per fold. This also happens when you change the batch size.
\nHope that helped :)
", "rawMarkdown": "If you want a solid seed for reproducable results in order to perform fair comparisons you can seed everything with the following code:\n\n# ------- Seeding __________ DO NOT CHANGE THESE!!!!!\ndef seed_everything(seed_value):\n random.seed(seed_value)\n np.random.seed(seed_value)\n torch.manual_seed(seed_value)\n os.environ['PYTHONHASHSEED'] = str(seed_value) \n if torch.cuda.is_available(): \n torch.cuda.manual_seed(seed_value)\n torch.cuda.manual_seed_all(seed_value)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\nseed = 42\nseed_everything(seed)\n# ------- Seeding __________ DO NOT CHANGE THESE!!!!!\n\nHowever, keep in mind that when you are training a NN, every small change can result in different models. In the kfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True), the shuffle term can lead to different models even if the folds are seeded/same. This is because, a different order which you feed the NN will result also in different NN even if the total tr data is the same per fold. This also happens when you change the batch size.\n\nHope that helped :)", "votes": 5 }, { "id": 2014269, "postDate": "2022-11-02T13:22:19.713Z", "content": "Thank you very much for the answer and the code, it's very helpful for newbies.
\nI would like to confirm that fixing these seeds is common practice when people are picking hyperparameters for their models other than batch size?
\nAt least I think for this competition, the training set is only 600 samples and any randomness introduced will affect the CV to a large extent and we cannot pick the best model and hyperparameters out of these 600 training samples without eliminating randomness.
", "rawMarkdown": "Thank you very much for the answer and the code, it's very helpful for newbies.\n\nI would like to confirm that fixing these seeds is common practice when people are picking hyperparameters for their models other than batch size? \n\nAt least I think for this competition, the training set is only 600 samples and any randomness introduced will affect the CV to a large extent and we cannot pick the best model and hyperparameters out of these 600 training samples without eliminating randomness.", "votes": 1 }, { "id": 2014343, "postDate": "2022-11-02T14:07:54.793Z", "content": "I just discovered something interesting, I copied the code \"def seed_everything(seed_value):\" into my ipynb and I can print out the exact same thing every time I train, which is great.
\nBut the interesting thing is that after I converted this ipynb file to a py file, each time I trained, the numbers printed out a little differently, but running it a few more times, I found that two of the results were exactly the same, thinking that the py file would introduce a new random factor compared to the ipynb file, and that this random factor could not take many values, and it would only take a few runs for the exact same result to appear once. What do you think about this? Do you know what the reason is?
This may be cause of different order in loading files. If you run it locally for example, the load order will be different comparing to loading from kaggle api.
", "rawMarkdown": "This may be cause of different order in loading files. If you run it locally for example, the load order will be different comparing to loading from kaggle api.", "votes": 1 }, { "id": 2014451, "postDate": "2022-11-02T15:02:13.167Z", "content": "Haha, but I am running both files in the same folder, both locally, both with pycharm pro
", "rawMarkdown": "Haha, but I am running both files in the same folder, both locally, both with pycharm pro" }, { "id": 2014460, "postDate": "2022-11-02T15:10:42.090Z", "content": "Then this is strange 😆. NN chaotic black nature!
\nHowever keep in mind, that randomness is desirable, since finding a good shot is more about gambling. Also it is needed when building an ensemble solution( which is the only way for most of competitions). Randomness can create different models which can lead to a great ensemble 😁👌
Hello everybody!
\nFound an interesting thing.
\nI am exploring versions of https://www.kaggle.com/masterray notebook:
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3/notebook?scriptVersionId=109514186
And found out that Version 8 and Version 11 has the same code, but were run on CPU (Version 8 ) and GPU (Version 11) and had differente submissions results. And of course differente Public Scores (Version 8 - 0.640 vs Version 11 - 0.608).
\nI'm a newbie in Data Science. And got some questions:
\nIs it because of floating point operations are managed slightly differently on the cpu and gpu?
Is it always better to use CPU? :-)
Version 8
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3/notebook?scriptVersionId=109464914
\nCPU
\nPublic Score
\n0.640
Version 11
\nhttps://www.kaggle.com/code/masterray/basic-spectrogram-image-classification-v3?scriptVersionId=109514186
\nGPU T4 x2
\nPublic Score
\n0.608
No, its not cause of CPU. It is just the random shuffle which leads to different solutions per run. I checked the code, the seed in only on torch stuff, but there is also this line:* kfold = StratifiedKFold(n_splits=nfold, random_state=42,** shuffle=True*) which leads to different models per run. The 64% was a lucky shot.
", "votes": 6, "replies": [ { "id": 2011201, "author_name": "Petr Kushnir", "author_url": "", "post_date": "2022-10-31T12:21:27.273000", "content": "Tnx for clear clarification!
", "votes": 0, "replies": [] }, { "id": 2013026, "author_name": "Chen Lin", "author_url": "", "post_date": "2022-11-01T15:31:36.927000", "content": "Can you explain further? If I want to conduct a controlled variable experiment and don't want any randomness to affect the CV, which random seeds should I fix?
\nI currently see two places in this program where I can fix random seeds, one is here.
\nkfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True)
\n I think it has been set to 42, which assumes that the samples for each fold in the Kfold are already fixed. I'm not sure if I'm right?
There is also one for torch, which I see the author has written as such.
\ntorch.manual_seed(42 + ifold + 1)
\nThis means that this seed is not fixed right?
So as long as it is changed to 42 here it will produce the exact same CV every time it is trained?
\nI'm new to Kaggle and I'm having to face such experiments for the first time, so if you could give some advice on controlling the variables, it would be greatly appreciated!
", "votes": 0, "replies": [] }, { "id": 2014119, "author_name": "", "author_url": "", "post_date": "2022-11-02T10:39:23.040000", "content": "If you want a solid seed for reproducable results in order to perform fair comparisons you can seed everything with the following code:
\ndef seed_everything(seed_value):
\n random.seed(seed_value)
\n np.random.seed(seed_value)
\n torch.manual_seed(seed_value)
\n os.environ['PYTHONHASHSEED'] = str(seed_value)
\n if torch.cuda.is_available():
\n torch.cuda.manual_seed(seed_value)
\n torch.cuda.manual_seed_all(seed_value)
\n torch.backends.cudnn.deterministic = True
\n torch.backends.cudnn.benchmark = True
\nseed = 42
\nseed_everything(seed)
However, keep in mind that when you are training a NN, every small change can result in different models. In the kfold = StratifiedKFold(n_splits=nfold, random_state=42, shuffle=True), the shuffle term can lead to different models even if the folds are seeded/same. This is because, a different order which you feed the NN will result also in different NN even if the total tr data is the same per fold. This also happens when you change the batch size.
\nHope that helped :)
", "votes": 5, "replies": [] }, { "id": 2014269, "author_name": "Chen Lin", "author_url": "", "post_date": "2022-11-02T13:22:19.713000", "content": "Thank you very much for the answer and the code, it's very helpful for newbies.
\nI would like to confirm that fixing these seeds is common practice when people are picking hyperparameters for their models other than batch size?
\nAt least I think for this competition, the training set is only 600 samples and any randomness introduced will affect the CV to a large extent and we cannot pick the best model and hyperparameters out of these 600 training samples without eliminating randomness.
", "votes": 1, "replies": [] }, { "id": 2014343, "author_name": "Chen Lin", "author_url": "", "post_date": "2022-11-02T14:07:54.793000", "content": "I just discovered something interesting, I copied the code \"def seed_everything(seed_value):\" into my ipynb and I can print out the exact same thing every time I train, which is great.
\nBut the interesting thing is that after I converted this ipynb file to a py file, each time I trained, the numbers printed out a little differently, but running it a few more times, I found that two of the results were exactly the same, thinking that the py file would introduce a new random factor compared to the ipynb file, and that this random factor could not take many values, and it would only take a few runs for the exact same result to appear once. What do you think about this? Do you know what the reason is?
This may be cause of different order in loading files. If you run it locally for example, the load order will be different comparing to loading from kaggle api.
", "votes": 1, "replies": [] }, { "id": 2014451, "author_name": "Chen Lin", "author_url": "", "post_date": "2022-11-02T15:02:13.167000", "content": "Haha, but I am running both files in the same folder, both locally, both with pycharm pro
", "votes": 0, "replies": [] }, { "id": 2014460, "author_name": "", "author_url": "", "post_date": "2022-11-02T15:10:42.090000", "content": "Then this is strange 😆. NN chaotic black nature!
\nHowever keep in mind, that randomness is desirable, since finding a good shot is more about gambling. Also it is needed when building an ensemble solution( which is the only way for most of competitions). Randomness can create different models which can lead to a great ensemble 😁👌