This is to get people guessing what type of models might perform better on this type of data, standard models like efficient net, alexnet and so on or built-from-scratch models. This is an exercise to test our theoretical knowledge. Let's try and give reasons as to which model might be best for this task.
\nI personally vote for basic models we make on our own. While I get that standard CNN architectures are robust and built for powerful classification problems, I can;t help but wonder if they give too much importance to the rest of the noise in the image as well. After all, we are only looking for a line in a noisy picture.
\nWhat are your thoughts? Theoretical explanations and any experimental data you might already have to support that.
", "messages": [ { "id": 2061412, "postDate": "2022-12-11T05:41:20.440Z", "content": "This is to get people guessing what type of models might perform better on this type of data, standard models like efficient net, alexnet and so on or built-from-scratch models. This is an exercise to test our theoretical knowledge. Let's try and give reasons as to which model might be best for this task.
\nI personally vote for basic models we make on our own. While I get that standard CNN architectures are robust and built for powerful classification problems, I can;t help but wonder if they give too much importance to the rest of the noise in the image as well. After all, we are only looking for a line in a noisy picture.
\nWhat are your thoughts? Theoretical explanations and any experimental data you might already have to support that.
", "rawMarkdown": "This is to get people guessing what type of models might perform better on this type of data, standard models like efficient net, alexnet and so on or built-from-scratch models. This is an exercise to test our theoretical knowledge. Let's try and give reasons as to which model might be best for this task.\n\nI personally vote for basic models we make on our own. While I get that standard CNN architectures are robust and built for powerful classification problems, I can;t help but wonder if they give too much importance to the rest of the noise in the image as well. After all, we are only looking for a line in a noisy picture.\n\nWhat are your thoughts? Theoretical explanations and any experimental data you might already have to support that.", "votes": 7 }, { "id": 2061674, "postDate": "2022-12-11T11:39:51.487Z", "content": "Pre-Trained models always work better, if you avoid overfitting during the new task. The problem may sound easy as “just detect a line in noise” but it actually needs a lot of intelligence for a model to do that. If you try to classify by your own some hard instances, you will notice that it requires a lot of imagination and intelligence to do that, something that non-pre-trained models will fail. For example, some parts of a line may be faded will other parts may slightly appear revealing some non-random gravity line patterns. Also it requires intelligence for a model to understand and discretise some random line patterns that the noise may created just by luck, comparing to the actual line patterns of a hidden gravity wave.
\nIf we look at the sky, we may see some shapes that the clouds created, but we will understand that this is noise 😉.
", "rawMarkdown": "Pre-Trained models always work better, if you avoid overfitting during the new task. The problem may sound easy as “just detect a line in noise” but it actually needs a lot of intelligence for a model to do that. If you try to classify by your own some hard instances, you will notice that it requires a lot of imagination and intelligence to do that, something that non-pre-trained models will fail. For example, some parts of a line may be faded will other parts may slightly appear revealing some non-random gravity line patterns. Also it requires intelligence for a model to understand and discretise some random line patterns that the noise may created just by luck, comparing to the actual line patterns of a hidden gravity wave. \n\nIf we look at the sky, we may see some shapes that the clouds created, but we will understand that this is noise 😉.\n", "votes": 5, "replies": [ { "id": 2062082, "postDate": "2022-12-11T17:58:14.133Z", "content": "Thanks a lot for the great response.
\nIf by above you mean model should be able to take into account angle of line, along with presence of one, I believe, given right data, I.e some negative line containing samples, models can learn that. In this case I don’t think borrowed intelligence is needed. If my understanding is not all you meant, can you show some examples to prove the claim, I.e examples of humans being able to distinguish between random line patterns and actual line patterns created by gravity wave?
\nMy understanding is that if we see a near horizontal line, we classify it contains gravity wave. Now neither nor any model ever can learn to predict randomness. I.e. no model or human ever can know if a given almost horizontal line is of gravity wave or result of just randomness. This is my understanding again, please correct me if I am wrong, I would like to learn more always.
\nIs there some line pattern, as you speak of, that we can easily identify as belonging to randomness or some different gravitational event as opposed to gravitational wave we are looking for here?
\nIt would be great if you could show us some hard samples to the points you include in response. Thanks.
Hard examples can be many type of instances.
\n[1] The most common one, are those positives where the gravity signal has corrupted by too much noise and thus is very hard to identify.
\n[2] Another more tricky one, could be be some negatives where the noise has created just by luck, signs of lines which can tricky a model and even a human to false classify them as positives.
\n[3] There are also some other positives, where the line has just faded and transformed in such a way, where it may appear as a cloud/blur type of signal line looking like noise.
\nSuch instances, in order to classify as much good as possible, it requires a lot of intelligence by a model, and thus there is the point that a non-pre-trained model will fail. (A 2D-pre-trained model, carries prior-general knowledge for shapes and texture from 2D-patterns. Therefore, during training on new (any) task, its weights start from a much greater starting point comparing to a random initialized non-pre-trained model.)
\nThe easy ones, where the signal is clearly identified, then ok, a naive and not-intelligent model can also do the job. For example a Canny algorithm where identifies edges and lines or other traditional line detection non-intelligent approaches could easily do the job, for the clear signals (easy instances).
", "rawMarkdown": "Hard examples can be many type of instances. \n\n[1] The most common one, are those positives where the gravity signal has corrupted by too much noise and thus is very hard to identify. \n\n[2] Another more tricky one, could be be some negatives where the noise has created just by luck, signs of lines which can tricky a model and even a human to false classify them as positives.\n\n[3] There are also some other positives, where the line has just faded and transformed in such a way, where it may appear as a cloud/blur type of signal line looking like noise.\n\nSuch instances, in order to classify as much good as possible, it requires a lot of intelligence by a model, and thus there is the point that a non-pre-trained model will fail. (A 2D-pre-trained model, carries prior-general knowledge for shapes and texture from 2D-patterns. Therefore, during training on new (any) task, its weights start from a much greater starting point comparing to a random initialized non-pre-trained model.)\n\nThe easy ones, where the signal is clearly identified, then ok, a naive and not-intelligent model can also do the job. For example a Canny algorithm where identifies edges and lines or other traditional line detection non-intelligent approaches could easily do the job, for the clear signals (easy instances).", "votes": 4 } ] }, { "id": 2063271, "postDate": "2022-12-12T19:24:59.730Z", "content": "I agree that own models (simpler and with less parameters) should perform quite well and maybe even better than state of the art models. I have written in a comment before:
\n\n", "rawMarkdown": "I agree that own models (simpler and with less parameters) should perform quite well and maybe even better than state of the art models. I have written in a comment before:\n\n\n> Furthermore I think that the models used for training (in your case EfficientNet-B4) are to complex and powerful to detect a in comparision rather \"easy\" signal (e.g. in https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202 the author used a self developed tiny model with few parameters and operations and it performed very well) Powerful models have a huge problem with very noisy data, because they tend to overfit fast by learning noisy patterns (also regarding the tiny provided train set) I would suggest you to try \"weaker\" models (e.g. B0, B1) or use more regularization (weight-decay, dropout, label-smoothing, …)", "votes": 3, "replies": [ { "id": 2063298, "postDate": "2022-12-12T19:49:59.153Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2063327, "postDate": "2022-12-12T20:05:24.280Z", "content": "Furthermore I think that the models used for training (in your case EfficientNet-B4) are to complex and powerful to detect a in comparision rather \"easy\" signal (e.g. in https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202 the author used a self developed tiny model with few parameters and operations and it performed very well) Powerful models have a huge problem with very noisy data, because they tend to overfit fast by learning noisy patterns (also regarding the tiny provided train set) I would suggest you to try \"weaker\" models (e.g. B0, B1) or use more regularization (weight-decay, dropout, label-smoothing, …)
\n
The question was more about pre-training or not. Small or Large model, pre-training almost always helps both large and small.
\nNow, i totally agree that large models overfit on noisy data, but on small datasets. If we had to use only the 600 instances given, then a small MobileNetV1 can do the job very well. Large models need large datasets in order to avoid learning noise and overfit. If a large model get trained with augmentations, data enlargements and so on, then most of the times do better. A small model trained on large dataset, has its limits.
\nAlso, this notebook uses B0,B1 but it did not state the score in the actual test set, but only on simulations.
\nMost of the highest score public notebooks, ended-up in b4-7-ns models, which are definitely not small. But, they also used augmentations, and that was the key of their success.
If you do a Levene's Test*, you'll notice that p-value is very bigger than alpha. A basic model won't suite the problem unless you do a denoising preprocessing first. This is my personal opinion based in my own experimentations.
\n\n\n\n
\n- In statistics, Levene's test is an inferential statistic used to assess the equality of variances for a variable calculated for two or more groups.[1] Some common statistical procedures assume that variances of the populations from which different samples are drawn are equal. Levene's test assesses this assumption. It tests the null hypothesis that the population variances are equal (called homogeneity of variance or homoscedasticity). If the resulting p-value of Levene's test is less than some significance level (typically 0.05), the obtained differences in sample variances are unlikely to have occurred based on random sampling from a population with equal variances. Thus, the null hypothesis of equal variances is rejected and it is concluded that there is a difference between the variances in the population.
\n
Source: https://en.wikipedia.org/wiki/Levene%27s_test
", "rawMarkdown": "If you do a Levene's Test*, you'll notice that p-value is very bigger than alpha. A basic model won't suite the problem unless you do a denoising preprocessing first. This is my personal opinion based in my own experimentations.\n\n> * In statistics, Levene's test is an inferential statistic used to assess the equality of variances for a variable calculated for two or more groups.[1] Some common statistical procedures assume that variances of the populations from which different samples are drawn are equal. Levene's test assesses this assumption. It tests the null hypothesis that the population variances are equal (called homogeneity of variance or homoscedasticity). If the resulting p-value of Levene's test is less than some significance level (typically 0.05), the obtained differences in sample variances are unlikely to have occurred based on random sampling from a population with equal variances. Thus, the null hypothesis of equal variances is rejected and it is concluded that there is a difference between the variances in the population.\n\nSource: [https://en.wikipedia.org/wiki/Levene%27s_test](url)", "votes": 3 } ], "comments": [ { "id": 2061674, "author_name": "", "author_url": "", "post_date": "2022-12-11T11:39:51.487000", "content": "Pre-Trained models always work better, if you avoid overfitting during the new task. The problem may sound easy as “just detect a line in noise” but it actually needs a lot of intelligence for a model to do that. If you try to classify by your own some hard instances, you will notice that it requires a lot of imagination and intelligence to do that, something that non-pre-trained models will fail. For example, some parts of a line may be faded will other parts may slightly appear revealing some non-random gravity line patterns. Also it requires intelligence for a model to understand and discretise some random line patterns that the noise may created just by luck, comparing to the actual line patterns of a hidden gravity wave.
\nIf we look at the sky, we may see some shapes that the clouds created, but we will understand that this is noise 😉.
", "votes": 5, "replies": [ { "id": 2062082, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-11T17:58:14.133000", "content": "Thanks a lot for the great response.
\nIf by above you mean model should be able to take into account angle of line, along with presence of one, I believe, given right data, I.e some negative line containing samples, models can learn that. In this case I don’t think borrowed intelligence is needed. If my understanding is not all you meant, can you show some examples to prove the claim, I.e examples of humans being able to distinguish between random line patterns and actual line patterns created by gravity wave?
\nMy understanding is that if we see a near horizontal line, we classify it contains gravity wave. Now neither nor any model ever can learn to predict randomness. I.e. no model or human ever can know if a given almost horizontal line is of gravity wave or result of just randomness. This is my understanding again, please correct me if I am wrong, I would like to learn more always.
\nIs there some line pattern, as you speak of, that we can easily identify as belonging to randomness or some different gravitational event as opposed to gravitational wave we are looking for here?
\nIt would be great if you could show us some hard samples to the points you include in response. Thanks.
Hard examples can be many type of instances.
\n[1] The most common one, are those positives where the gravity signal has corrupted by too much noise and thus is very hard to identify.
\n[2] Another more tricky one, could be be some negatives where the noise has created just by luck, signs of lines which can tricky a model and even a human to false classify them as positives.
\n[3] There are also some other positives, where the line has just faded and transformed in such a way, where it may appear as a cloud/blur type of signal line looking like noise.
\nSuch instances, in order to classify as much good as possible, it requires a lot of intelligence by a model, and thus there is the point that a non-pre-trained model will fail. (A 2D-pre-trained model, carries prior-general knowledge for shapes and texture from 2D-patterns. Therefore, during training on new (any) task, its weights start from a much greater starting point comparing to a random initialized non-pre-trained model.)
\nThe easy ones, where the signal is clearly identified, then ok, a naive and not-intelligent model can also do the job. For example a Canny algorithm where identifies edges and lines or other traditional line detection non-intelligent approaches could easily do the job, for the clear signals (easy instances).
", "votes": 4, "replies": [] } ] }, { "id": 2063271, "author_name": "Ali Abdin", "author_url": "", "post_date": "2022-12-12T19:24:59.730000", "content": "I agree that own models (simpler and with less parameters) should perform quite well and maybe even better than state of the art models. I have written in a comment before:
\n\n", "votes": 3, "replies": [ { "id": 2063298, "author_name": "", "author_url": "", "post_date": "2022-12-12T19:49:59.153000", "content": "", "votes": 0, "replies": [] }, { "id": 2063327, "author_name": "", "author_url": "", "post_date": "2022-12-12T20:05:24.280000", "content": "Furthermore I think that the models used for training (in your case EfficientNet-B4) are to complex and powerful to detect a in comparision rather \"easy\" signal (e.g. in https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202 the author used a self developed tiny model with few parameters and operations and it performed very well) Powerful models have a huge problem with very noisy data, because they tend to overfit fast by learning noisy patterns (also regarding the tiny provided train set) I would suggest you to try \"weaker\" models (e.g. B0, B1) or use more regularization (weight-decay, dropout, label-smoothing, …)
\n
The question was more about pre-training or not. Small or Large model, pre-training almost always helps both large and small.
\nNow, i totally agree that large models overfit on noisy data, but on small datasets. If we had to use only the 600 instances given, then a small MobileNetV1 can do the job very well. Large models need large datasets in order to avoid learning noise and overfit. If a large model get trained with augmentations, data enlargements and so on, then most of the times do better. A small model trained on large dataset, has its limits.
\nAlso, this notebook uses B0,B1 but it did not state the score in the actual test set, but only on simulations.
\nMost of the highest score public notebooks, ended-up in b4-7-ns models, which are definitely not small. But, they also used augmentations, and that was the key of their success.
If you do a Levene's Test*, you'll notice that p-value is very bigger than alpha. A basic model won't suite the problem unless you do a denoising preprocessing first. This is my personal opinion based in my own experimentations.
\n\n\n", "votes": 3, "replies": [] } ], "raw_markdown_by_id": { "2061412": "This is to get people guessing what type of models might perform better on this type of data, standard models like efficient net, alexnet and so on or built-from-scratch models. This is an exercise to test our theoretical knowledge. Let's try and give reasons as to which model might be best for this task.\n\nI personally vote for basic models we make on our own. While I get that standard CNN architectures are robust and built for powerful classification problems, I can;t help but wonder if they give too much importance to the rest of the noise in the image as well. After all, we are only looking for a line in a noisy picture.\n\nWhat are your thoughts? Theoretical explanations and any experimental data you might already have to support that.", "2061674": "Pre-Trained models always work better, if you avoid overfitting during the new task. The problem may sound easy as “just detect a line in noise” but it actually needs a lot of intelligence for a model to do that. If you try to classify by your own some hard instances, you will notice that it requires a lot of imagination and intelligence to do that, something that non-pre-trained models will fail. For example, some parts of a line may be faded will other parts may slightly appear revealing some non-random gravity line patterns. Also it requires intelligence for a model to understand and discretise some random line patterns that the noise may created just by luck, comparing to the actual line patterns of a hidden gravity wave. \n\nIf we look at the sky, we may see some shapes that the clouds created, but we will understand that this is noise 😉.\n", "2063271": "I agree that own models (simpler and with less parameters) should perform quite well and maybe even better than state of the art models. I have written in a comment before:\n\n\n> Furthermore I think that the models used for training (in your case EfficientNet-B4) are to complex and powerful to detect a in comparision rather \"easy\" signal (e.g. in https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202 the author used a self developed tiny model with few parameters and operations and it performed very well) Powerful models have a huge problem with very noisy data, because they tend to overfit fast by learning noisy patterns (also regarding the tiny provided train set) I would suggest you to try \"weaker\" models (e.g. B0, B1) or use more regularization (weight-decay, dropout, label-smoothing, …)", "2061419": "If you do a Levene's Test*, you'll notice that p-value is very bigger than alpha. A basic model won't suite the problem unless you do a denoising preprocessing first. This is my personal opinion based in my own experimentations.\n\n> * In statistics, Levene's test is an inferential statistic used to assess the equality of variances for a variable calculated for two or more groups.[1] Some common statistical procedures assume that variances of the populations from which different samples are drawn are equal. Levene's test assesses this assumption. It tests the null hypothesis that the population variances are equal (called homogeneity of variance or homoscedasticity). If the resulting p-value of Levene's test is less than some significance level (typically 0.05), the obtained differences in sample variances are unlikely to have occurred based on random sampling from a population with equal variances. Thus, the null hypothesis of equal variances is rejected and it is concluded that there is a difference between the variances in the population.\n\nSource: [https://en.wikipedia.org/wiki/Levene%27s_test](url)" } }\n
\n- In statistics, Levene's test is an inferential statistic used to assess the equality of variances for a variable calculated for two or more groups.[1] Some common statistical procedures assume that variances of the populations from which different samples are drawn are equal. Levene's test assesses this assumption. It tests the null hypothesis that the population variances are equal (called homogeneity of variance or homoscedasticity). If the resulting p-value of Levene's test is less than some significance level (typically 0.05), the obtained differences in sample variances are unlikely to have occurred based on random sampling from a population with equal variances. Thus, the null hypothesis of equal variances is rejected and it is concluded that there is a difference between the variances in the population.
\n