I have tried all kinds of data agumentations and models and generated different examples using different hyperparameters. But it seems too difficult to reach 0.75, my best lb is 0.723. Anyone can tell the key to reach higher lb?
", "messages": [ { "id": 2069623, "postDate": "2022-12-19T07:51:24.827Z", "content": "I have tried all kinds of data agumentations and models and generated different examples using different hyperparameters. But it seems too difficult to reach 0.75, my best lb is 0.723. Anyone can tell the key to reach higher lb?
", "rawMarkdown": "I have tried all kinds of data agumentations and models and generated different examples using different hyperparameters. But it seems too difficult to reach 0.75, my best lb is 0.723. Anyone can tell the key to reach higher lb?", "votes": 15 }, { "id": 2070329, "postDate": "2022-12-19T21:06:08.617Z", "content": "There is a way. But I'll say it after this Challenge ends.
\nP.S.: A hint: It's not a classificator, It's a regressor. But the key isn't the model itself, is the data you generate for. I got 0.787 with that single model.
\nCheers :)
", "rawMarkdown": "There is a way. But I'll say it after this Challenge ends.\n\nP.S.: A hint: It's not a classificator, It's a regressor. But the key isn't the model itself, is the data you generate for. I got 0.787 with that single model.\n\nCheers :)", "votes": 10 }, { "id": 2070061, "postDate": "2022-12-19T15:08:13.580Z", "content": "Only ensembles.
\nAlmost impossible for a single model to reach 75. The only one is based on the 74.8 from large kernel approach in the public codes of this comp. But we don’t know how it was trained. It is also an ensemble of 64 models in TTA approach, so it cannot be considered “Single model”.
@manolispintelas Do you mean you reach such a high score by ensembling multiple models with AUC lower than 0.75? Really impressive!
", "rawMarkdown": "@manolispintelas Do you mean you reach such a high score by ensembling multiple models with AUC lower than 0.75? Really impressive!", "votes": 1, "replies": [ { "id": 2070759, "postDate": "2022-12-20T09:46:38.160Z", "content": "It depends on what someone consider as ensemble. For example a 5-fold averaging or a TTA is also ensemble. Also it depends on what a single model is considered, for example a multi-view-cnn is an end-end single model but heavy as multi ensembles Cnns.
\nMy phd research start, was about: Global large model vs Local small models ensembling in conquer and divide approach. Thus I knew some stuff in ensemble practices.
", "rawMarkdown": "It depends on what someone consider as ensemble. For example a 5-fold averaging or a TTA is also ensemble. Also it depends on what a single model is considered, for example a multi-view-cnn is an end-end single model but heavy as multi ensembles Cnns.\n\nMy phd research start, was about: Global large model vs Local small models ensembling in conquer and divide approach. Thus I knew some stuff in ensemble practices.", "votes": -1 }, { "id": 2085131, "postDate": "2023-01-04T00:29:01.937Z", "content": "I noticed that your private lb dropped a lot, how did you overfit so much? By ensemble? Would you be willing to make your method public for others to notice afterwards?
", "rawMarkdown": "I noticed that your private lb dropped a lot, how did you overfit so much? By ensemble? Would you be willing to make your method public for others to notice afterwards?", "votes": 1 }, { "id": 2085181, "postDate": "2023-01-04T01:03:56.597Z", "content": "Zollkron also dropped much more (-446). Overfit for sure, but I guess also ROC is a bit unstable metric.
\nI mainly used hand-crafted features with CNN ensemble followed by some manual labeling in an active learning way. Maybe my manual labeling destroyed my solution, since it created spikes and anomalies in probs which destroyed the roc evaluation.
\nIm dissapointed a bit since i really invested too much time (8h/day) for this comp.
Anyway, i learned a lot for sure.
\nKnowledge is the gift from kaggling anyway :).
\nGz for your 19 place btw.
Thank you :)
\nI've never heard of active learning before and I've tried asking ChatGPT but I'm still not sure how you do it. Do you pick out test samples where the model prediction is close to 0.5 for manual labelling? How do you label them? Because there are many samples, the human eyes can't tell if there is a signal or not.
\nWhat are hand-crafted features? Is it a traditional feature extraction method? I thought all these methods could be completely replaced by CNNs, because the multi-channel convolutional kernel can learn them.
", "rawMarkdown": "Thank you :) \n\nI've never heard of active learning before and I've tried asking ChatGPT but I'm still not sure how you do it. Do you pick out test samples where the model prediction is close to 0.5 for manual labelling? How do you label them? Because there are many samples, the human eyes can't tell if there is a signal or not.\n\nWhat are hand-crafted features? Is it a traditional feature extraction method? I thought all these methods could be completely replaced by CNNs, because the multi-channel convolutional kernel can learn them." }, { "id": 2085359, "postDate": "2023-01-04T05:19:44.047Z", "content": "I did a risky assumption. I believed that the uncertainty could be quantified using Isaac Horowitz's Quatitative Feedback Theory (https://en.wikipedia.org/wiki/Quantitative_feedback_theory). I tried to bring this theory in practice here with the detectors like if they will be an \"industrial plant\". I failed in my premise and I fell in a great Bias. That's all.
", "rawMarkdown": "I did a risky assumption. I believed that the uncertainty could be quantified using Isaac Horowitz's Quatitative Feedback Theory ([https://en.wikipedia.org/wiki/Quantitative_feedback_theory](url)). I tried to bring this theory in practice here with the detectors like if they will be an \"industrial plant\". I failed in my premise and I fell in a great Bias. That's all.", "votes": 1 } ] } ] }, { "id": 2070584, "postDate": "2022-12-20T07:16:21.753Z", "content": "I can reach LB 0.762 with a single model. as other comments & public notebooks, generating the data is the one of the key points to boost the score I guess. good luck!
", "rawMarkdown": "I can reach LB 0.762 with a single model. as other comments & public notebooks, generating the data is the one of the key points to boost the score I guess. good luck!", "votes": 1, "replies": [ { "id": 2071710, "postDate": "2022-12-21T09:41:05.230Z", "content": "How much data do you generate? @kozistr @zollkron
", "rawMarkdown": "How much data do you generate? @kozistr @zollkron ", "votes": 1, "replies": [ { "id": 2071845, "postDate": "2022-12-21T12:56:00.390Z", "content": "Honestly, I'm not using PyFstat. I'm using computed metadata generated by my own. My data has 180 features right now and growing.
", "rawMarkdown": "Honestly, I'm not using PyFstat. I'm using computed metadata generated by my own. My data has 180 features right now and growing.", "votes": 2 }, { "id": 2071895, "postDate": "2022-12-21T13:38:33.057Z", "content": "in my case, I'm using about 100k generated samples for now (but i guess there's no need to use lots of samples).
", "rawMarkdown": "in my case, I'm using about 100k generated samples for now (but i guess there's no need to use lots of samples).", "votes": 1 }, { "id": 2071898, "postDate": "2022-12-21T13:40:25.317Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2071899, "postDate": "2022-12-21T13:40:54.660Z", "content": "epic! looking forward to hear your solution after the competition ends :)
", "rawMarkdown": "epic! looking forward to hear your solution after the competition ends :)", "votes": 1 }, { "id": 2071910, "postDate": "2022-12-21T13:58:27.967Z", "content": "OMG! That's a true lot! I guess you can reduce your samples looking for similarity with the test data. Someone, I don't remember who, commented he is using SIC for select samples close similar to the test samples. May be It can help.
", "rawMarkdown": "OMG! That's a true lot! I guess you can reduce your samples looking for similarity with the test data. Someone, I don't remember who, commented he is using SIC for select samples close similar to the test samples. May be It can help.", "votes": 1 }, { "id": 2071912, "postDate": "2022-12-21T14:03:22.393Z", "content": "I can say that my solution pretends to lead a Mathematical Theory (not mine), applied in Industry, in practice in this domain. So I must generate my own data and algorythms for that, after that I can apply a regressor and… et voilà!
", "rawMarkdown": "I can say that my solution pretends to lead a Mathematical Theory (not mine), applied in Industry, in practice in this domain. So I must generate my own data and algorythms for that, after that I can apply a regressor and... et voilà!", "votes": 1 }, { "id": 2072061, "postDate": "2022-12-21T16:37:42.903Z", "content": "Really looking forward to hear more about it after the competition ends. I suspected there are some tricks that are not classic deep learning that work really well. In my case, I usually do 100k-200k samples to train a model. 400k wasn't helping :D I am lazy to do augmentations 😂
", "rawMarkdown": "Really looking forward to hear more about it after the competition ends. I suspected there are some tricks that are not classic deep learning that work really well. In my case, I usually do 100k-200k samples to train a model. 400k wasn't helping :D I am lazy to do augmentations 😂", "votes": 1 }, { "id": 2072191, "postDate": "2022-12-21T20:29:19.030Z", "content": "Thank you very much. I think that you will find interesting because It's a good trick that can be used in other problems like that. In other words, problems with a big Homogeinity in the Variances. A coleague of mine call them \"White Noise Problems\", and I like this term 😅. I would like to do a Notebook explaining all at the end, I hope get enough time to do it.
", "rawMarkdown": "Thank you very much. I think that you will find interesting because It's a good trick that can be used in other problems like that. In other words, problems with a big Homogeinity in the Variances. A coleague of mine call them \"White Noise Problems\", and I like this term 😅. I would like to do a Notebook explaining all at the end, I hope get enough time to do it.", "votes": 3 }, { "id": 2072715, "postDate": "2022-12-22T11:16:30.937Z", "content": "Can you tell me what \"SIC\" stands for?
", "rawMarkdown": "Can you tell me what \"SIC\" stands for?" }, { "id": 2072735, "postDate": "2022-12-22T11:37:48.397Z", "content": "Spectrogram Image Classification.
", "rawMarkdown": "Spectrogram Image Classification." } ] } ] }, { "id": 2072632, "postDate": "2022-12-22T09:43:33.297Z", "content": "generating more data, I guess.
", "rawMarkdown": "generating more data, I guess." }, { "id": 2069909, "postDate": "2022-12-19T12:36:12.853Z", "content": "IMO.
\nIn this competition, the public training data is very small(600) I think it is not enough to score in LB.
\nFor reference, the number of public train data for similar competitions in the past is SETi (60,000) and G2Net-Old(560,000).
I have tried generating enough train data (8000), but it improved a little (0.7->0.72)
", "rawMarkdown": "I have tried generating enough train data (8000), but it improved a little (0.7->0.72)", "replies": [ { "id": 2073208, "postDate": "2022-12-22T18:58:30.213Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2073367, "postDate": "2022-12-23T01:31:46.603Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 2070329, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-19T21:06:08.617000", "content": "There is a way. But I'll say it after this Challenge ends.
\nP.S.: A hint: It's not a classificator, It's a regressor. But the key isn't the model itself, is the data you generate for. I got 0.787 with that single model.
\nCheers :)
", "votes": 10, "replies": [] }, { "id": 2070061, "author_name": "", "author_url": "", "post_date": "2022-12-19T15:08:13.580000", "content": "Only ensembles.
\nAlmost impossible for a single model to reach 75. The only one is based on the 74.8 from large kernel approach in the public codes of this comp. But we don’t know how it was trained. It is also an ensemble of 64 models in TTA approach, so it cannot be considered “Single model”.
@manolispintelas Do you mean you reach such a high score by ensembling multiple models with AUC lower than 0.75? Really impressive!
", "votes": 1, "replies": [ { "id": 2070759, "author_name": "", "author_url": "", "post_date": "2022-12-20T09:46:38.160000", "content": "It depends on what someone consider as ensemble. For example a 5-fold averaging or a TTA is also ensemble. Also it depends on what a single model is considered, for example a multi-view-cnn is an end-end single model but heavy as multi ensembles Cnns.
\nMy phd research start, was about: Global large model vs Local small models ensembling in conquer and divide approach. Thus I knew some stuff in ensemble practices.
", "votes": -1, "replies": [] }, { "id": 2085131, "author_name": "Chen Lin", "author_url": "", "post_date": "2023-01-04T00:29:01.937000", "content": "I noticed that your private lb dropped a lot, how did you overfit so much? By ensemble? Would you be willing to make your method public for others to notice afterwards?
", "votes": 1, "replies": [] }, { "id": 2085181, "author_name": "", "author_url": "", "post_date": "2023-01-04T01:03:56.597000", "content": "Zollkron also dropped much more (-446). Overfit for sure, but I guess also ROC is a bit unstable metric.
\nI mainly used hand-crafted features with CNN ensemble followed by some manual labeling in an active learning way. Maybe my manual labeling destroyed my solution, since it created spikes and anomalies in probs which destroyed the roc evaluation.
\nIm dissapointed a bit since i really invested too much time (8h/day) for this comp.
Anyway, i learned a lot for sure.
\nKnowledge is the gift from kaggling anyway :).
\nGz for your 19 place btw.
Thank you :)
\nI've never heard of active learning before and I've tried asking ChatGPT but I'm still not sure how you do it. Do you pick out test samples where the model prediction is close to 0.5 for manual labelling? How do you label them? Because there are many samples, the human eyes can't tell if there is a signal or not.
\nWhat are hand-crafted features? Is it a traditional feature extraction method? I thought all these methods could be completely replaced by CNNs, because the multi-channel convolutional kernel can learn them.
", "votes": 0, "replies": [] }, { "id": 2085359, "author_name": "Zollkron", "author_url": "", "post_date": "2023-01-04T05:19:44.047000", "content": "I did a risky assumption. I believed that the uncertainty could be quantified using Isaac Horowitz's Quatitative Feedback Theory (https://en.wikipedia.org/wiki/Quantitative_feedback_theory). I tried to bring this theory in practice here with the detectors like if they will be an \"industrial plant\". I failed in my premise and I fell in a great Bias. That's all.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2070584, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2022-12-20T07:16:21.753000", "content": "I can reach LB 0.762 with a single model. as other comments & public notebooks, generating the data is the one of the key points to boost the score I guess. good luck!
", "votes": 1, "replies": [ { "id": 2071710, "author_name": "CroDoc", "author_url": "", "post_date": "2022-12-21T09:41:05.230000", "content": "How much data do you generate? @kozistr @zollkron
", "votes": 1, "replies": [ { "id": 2071845, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-21T12:56:00.390000", "content": "Honestly, I'm not using PyFstat. I'm using computed metadata generated by my own. My data has 180 features right now and growing.
", "votes": 2, "replies": [] }, { "id": 2071895, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2022-12-21T13:38:33.057000", "content": "in my case, I'm using about 100k generated samples for now (but i guess there's no need to use lots of samples).
", "votes": 1, "replies": [] }, { "id": 2071898, "author_name": "", "author_url": "", "post_date": "2022-12-21T13:40:25.317000", "content": "", "votes": 0, "replies": [] }, { "id": 2071899, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2022-12-21T13:40:54.660000", "content": "epic! looking forward to hear your solution after the competition ends :)
", "votes": 1, "replies": [] }, { "id": 2071910, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-21T13:58:27.967000", "content": "OMG! That's a true lot! I guess you can reduce your samples looking for similarity with the test data. Someone, I don't remember who, commented he is using SIC for select samples close similar to the test samples. May be It can help.
", "votes": 1, "replies": [] }, { "id": 2071912, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-21T14:03:22.393000", "content": "I can say that my solution pretends to lead a Mathematical Theory (not mine), applied in Industry, in practice in this domain. So I must generate my own data and algorythms for that, after that I can apply a regressor and… et voilà!
", "votes": 1, "replies": [] }, { "id": 2072061, "author_name": "CroDoc", "author_url": "", "post_date": "2022-12-21T16:37:42.903000", "content": "Really looking forward to hear more about it after the competition ends. I suspected there are some tricks that are not classic deep learning that work really well. In my case, I usually do 100k-200k samples to train a model. 400k wasn't helping :D I am lazy to do augmentations 😂
", "votes": 1, "replies": [] }, { "id": 2072191, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-21T20:29:19.030000", "content": "Thank you very much. I think that you will find interesting because It's a good trick that can be used in other problems like that. In other words, problems with a big Homogeinity in the Variances. A coleague of mine call them \"White Noise Problems\", and I like this term 😅. I would like to do a Notebook explaining all at the end, I hope get enough time to do it.
", "votes": 3, "replies": [] }, { "id": 2072715, "author_name": "GabeTheHuman", "author_url": "", "post_date": "2022-12-22T11:16:30.937000", "content": "Can you tell me what \"SIC\" stands for?
", "votes": 0, "replies": [] }, { "id": 2072735, "author_name": "Zollkron", "author_url": "", "post_date": "2022-12-22T11:37:48.397000", "content": "Spectrogram Image Classification.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2072632, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-12-22T09:43:33.297000", "content": "generating more data, I guess.
", "votes": 0, "replies": [] }, { "id": 2069909, "author_name": "yukiZ", "author_url": "", "post_date": "2022-12-19T12:36:12.853000", "content": "IMO.
\nIn this competition, the public training data is very small(600) I think it is not enough to score in LB.
\nFor reference, the number of public train data for similar competitions in the past is SETi (60,000) and G2Net-Old(560,000).
I have tried generating enough train data (8000), but it improved a little (0.7->0.72)
", "votes": 0, "replies": [ { "id": 2073208, "author_name": "", "author_url": "", "post_date": "2022-12-22T18:58:30.213000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2073367, "author_name": "", "author_url": "", "post_date": "2022-12-23T01:31:46.603000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "2069623": "I have tried all kinds of data agumentations and models and generated different examples using different hyperparameters. But it seems too difficult to reach 0.75, my best lb is 0.723. Anyone can tell the key to reach higher lb?", "2070329": "There is a way. But I'll say it after this Challenge ends.\n\nP.S.: A hint: It's not a classificator, It's a regressor. But the key isn't the model itself, is the data you generate for. I got 0.787 with that single model.\n\nCheers :)", "2070061": "Only ensembles.\nAlmost impossible for a single model to reach 75. The only one is based on the 74.8 from large kernel approach in the public codes of this comp. But we don’t know how it was trained. It is also an ensemble of 64 models in TTA approach, so it cannot be considered “Single model”.", "2070584": "I can reach LB 0.762 with a single model. as other comments & public notebooks, generating the data is the one of the key points to boost the score I guess. good luck!", "2072632": "generating more data, I guess.", "2069909": "IMO.\nIn this competition, the public training data is very small(600) I think it is not enough to score in LB.\nFor reference, the number of public train data for similar competitions in the past is SETi (60,000) and G2Net-Old(560,000).", "2073367": "" } }