{
  "id": 363069,
  "title": "Accuracy on train dataset(as validation) is always about 2/3?",
  "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363069",
  "author_name": "zzzc18",
  "post_date": "2022-10-30T20:30:04.256000",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Has anyone tried to use generated data as train input and original train dataset as validation set?</p>\n<p>I did so and find several methods(mainly based on Jun Koda's code and method) all achieve accuracy 2/3 or 66.7% on the original train dataset, when the AUROC vary, which means we can easily meet the upperbounds of our algorithms. Lots of the error cases are labeled with 1 and I can't recognize them by myself.</p>\n<p>So I wonder if someone can get a better accuracy on the original train dataset? Or what we can do is just to improve AUROC?</p>",
  "messages": [
    {
      "id": 2010531,
      "postDate": "2022-10-30T20:30:04.257Z",
      "content": "<p>Has anyone tried to use generated data as train input and original train dataset as validation set?</p>\n<p>I did so and find several methods(mainly based on Jun Koda's code and method) all achieve accuracy 2/3 or 66.7% on the original train dataset, when the AUROC vary, which means we can easily meet the upperbounds of our algorithms. Lots of the error cases are labeled with 1 and I can't recognize them by myself.</p>\n<p>So I wonder if someone can get a better accuracy on the original train dataset? Or what we can do is just to improve AUROC?</p>",
      "rawMarkdown": "Has anyone tried to use generated data as train input and original train dataset as validation set?\n\nI did so and find several methods(mainly based on Jun Koda's code and method) all achieve accuracy 2/3 or 66.7% on the original train dataset, when the AUROC vary, which means we can easily meet the upperbounds of our algorithms. Lots of the error cases are labeled with 1 and I can't recognize them by myself.\n\nSo I wonder if someone can get a better accuracy on the original train dataset? Or what we can do is just to improve AUROC?",
      "votes": 3
    },
    {
      "id": 2012289,
      "postDate": "2022-11-01T06:52:03.173Z",
      "content": "<p>based on my experiments (similar with <a href=\"https://www.kaggle.com/code/werus23/g2net-pytorch-with-generated-data\" target=\"_blank\">werus23 generated data</a>)</p>\n<p>auroc and accuracy are about 0.79 ~ 0.81, 0.69 ~ 0.71, respectively (trained on the generated data, validated on the original train dataset (600 samples). Also, the <code>pretraining</code> gives a boost about 0.04 ~ 0.05 CV &amp; LB.</p>",
      "rawMarkdown": "based on my experiments (similar with [werus23 generated data](https://www.kaggle.com/code/werus23/g2net-pytorch-with-generated-data))\n\nauroc and accuracy are about 0.79 ~ 0.81, 0.69 ~ 0.71, respectively (trained on the generated data, validated on the original train dataset (600 samples). Also, the `pretraining` gives a boost about 0.04 ~ 0.05 CV & LB.",
      "votes": 1
    },
    {
      "id": 2011481,
      "postDate": "2022-10-31T15:43:40.730Z",
      "content": "<p>I did so and got approximately the same results. In fact, if we put target=1 for all train data, we'll get accuracy of 2/3, that is class inbalance in train data.<br>\nFor some models I was able to get about 68% accuracy on original train dataset (which was used as validation) with AUCROC about 0.77. However, these models are not good on original test dataset. I think, the reason is these datasets are very different. Test dataset has higher depths and a lot of artifacts, which can be easily seen on spectrograms.</p>",
      "rawMarkdown": "I did so and got approximately the same results. In fact, if we put target=1 for all train data, we'll get accuracy of 2/3, that is class inbalance in train data.\nFor some models I was able to get about 68% accuracy on original train dataset (which was used as validation) with AUCROC about 0.77. However, these models are not good on original test dataset. I think, the reason is these datasets are very different. Test dataset has higher depths and a lot of artifacts, which can be easily seen on spectrograms.",
      "votes": 1
    },
    {
      "id": 2011074,
      "postDate": "2022-10-31T11:09:18.283Z",
      "content": "<p>That's exactly what I mean. So I wonder if there may exist some handcraft filters so that we can visulize the difference between 1s and 0s, I think classifying unvisible features using CNN is not a wise choice.</p>",
      "rawMarkdown": "That's exactly what I mean. So I wonder if there may exist some handcraft filters so that we can visulize the difference between 1s and 0s, I think classifying unvisible features using CNN is not a wise choice.",
      "votes": 1,
      "replies": [
        {
          "id": 2015719,
          "postDate": "2022-11-03T13:50:13.850Z",
          "rawMarkdown": "",
          "votes": -1,
          "isDeleted": true
        }
      ]
    },
    {
      "id": 2010793,
      "postDate": "2022-10-31T06:16:57.793Z",
      "content": "<p>Yes, I have generated some data and trained with it (train + validation + test), and then evaluated the kaggle train dataset, mostly using the common public method (Jun Coda's). <br>\nThis is from one of my runs, without too much tweaking (learning rate, hyperparams, etc).</p>\n<p>As you can see I also get ~2/3 accuracy. The Accuracy on my \"test\" set (generated data) is somewhat meaningless, as it depends on the distribution of depths chosen (sqrtSX / h0). Higher depths are of course harder to predict, and my generated data has hitherto been limited to stationary noise without glitches nor other artifacts.</p>\n<pre><code>Size of test set: 330\nAUC of ROC: 0.8968823944539002\nAccuracy: 0.8151515151515152\n\n-- KAGGLE TRAIN SET Evaluation --\nSize of Kaggle train set: 600\nAUC of ROC: 0.735025\nAccuracy: 0.6466666666666666\n</code></pre>",
      "rawMarkdown": "Yes, I have generated some data and trained with it (train + validation + test), and then evaluated the kaggle train dataset, mostly using the common public method (Jun Coda's). \nThis is from one of my runs, without too much tweaking (learning rate, hyperparams, etc).\n\nAs you can see I also get ~2/3 accuracy. The Accuracy on my \"test\" set (generated data) is somewhat meaningless, as it depends on the distribution of depths chosen (sqrtSX / h0). Higher depths are of course harder to predict, and my generated data has hitherto been limited to stationary noise without glitches nor other artifacts.\n\n```\nSize of test set: 330\nAUC of ROC: 0.8968823944539002\nAccuracy: 0.8151515151515152\n\n-- KAGGLE TRAIN SET Evaluation --\nSize of Kaggle train set: 600\nAUC of ROC: 0.735025\nAccuracy: 0.6466666666666666\n```"
    }
  ],
  "comments": [
    {
      "id": 2012289,
      "author_name": "HyeongChan Kim",
      "author_url": "",
      "post_date": "2022-11-01T06:52:03.173000",
      "content": "<p>based on my experiments (similar with <a href=\"https://www.kaggle.com/code/werus23/g2net-pytorch-with-generated-data\" target=\"_blank\">werus23 generated data</a>)</p>\n<p>auroc and accuracy are about 0.79 ~ 0.81, 0.69 ~ 0.71, respectively (trained on the generated data, validated on the original train dataset (600 samples). Also, the <code>pretraining</code> gives a boost about 0.04 ~ 0.05 CV &amp; LB.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2011481,
      "author_name": "Konstantin Dmitriev",
      "author_url": "",
      "post_date": "2022-10-31T15:43:40.730000",
      "content": "<p>I did so and got approximately the same results. In fact, if we put target=1 for all train data, we'll get accuracy of 2/3, that is class inbalance in train data.<br>\nFor some models I was able to get about 68% accuracy on original train dataset (which was used as validation) with AUCROC about 0.77. However, these models are not good on original test dataset. I think, the reason is these datasets are very different. Test dataset has higher depths and a lot of artifacts, which can be easily seen on spectrograms.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2011074,
      "author_name": "zzzc18",
      "author_url": "",
      "post_date": "2022-10-31T11:09:18.283000",
      "content": "<p>That's exactly what I mean. So I wonder if there may exist some handcraft filters so that we can visulize the difference between 1s and 0s, I think classifying unvisible features using CNN is not a wise choice.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2015719,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-11-03T13:50:13.850000",
          "content": "",
          "votes": -1,
          "replies": []
        }
      ]
    },
    {
      "id": 2010793,
      "author_name": "Victor Gonzalez",
      "author_url": "",
      "post_date": "2022-10-31T06:16:57.793000",
      "content": "<p>Yes, I have generated some data and trained with it (train + validation + test), and then evaluated the kaggle train dataset, mostly using the common public method (Jun Coda's). <br>\nThis is from one of my runs, without too much tweaking (learning rate, hyperparams, etc).</p>\n<p>As you can see I also get ~2/3 accuracy. The Accuracy on my \"test\" set (generated data) is somewhat meaningless, as it depends on the distribution of depths chosen (sqrtSX / h0). Higher depths are of course harder to predict, and my generated data has hitherto been limited to stationary noise without glitches nor other artifacts.</p>\n<pre><code>Size of test set: 330\nAUC of ROC: 0.8968823944539002\nAccuracy: 0.8151515151515152\n\n-- KAGGLE TRAIN SET Evaluation --\nSize of Kaggle train set: 600\nAUC of ROC: 0.735025\nAccuracy: 0.6466666666666666\n</code></pre>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2010531": "Has anyone tried to use generated data as train input and original train dataset as validation set?\n\nI did so and find several methods(mainly based on Jun Koda's code and method) all achieve accuracy 2/3 or 66.7% on the original train dataset, when the AUROC vary, which means we can easily meet the upperbounds of our algorithms. Lots of the error cases are labeled with 1 and I can't recognize them by myself.\n\nSo I wonder if someone can get a better accuracy on the original train dataset? Or what we can do is just to improve AUROC?",
    "2012289": "based on my experiments (similar with [werus23 generated data](https://www.kaggle.com/code/werus23/g2net-pytorch-with-generated-data))\n\nauroc and accuracy are about 0.79 ~ 0.81, 0.69 ~ 0.71, respectively (trained on the generated data, validated on the original train dataset (600 samples). Also, the `pretraining` gives a boost about 0.04 ~ 0.05 CV & LB.",
    "2011481": "I did so and got approximately the same results. In fact, if we put target=1 for all train data, we'll get accuracy of 2/3, that is class inbalance in train data.\nFor some models I was able to get about 68% accuracy on original train dataset (which was used as validation) with AUCROC about 0.77. However, these models are not good on original test dataset. I think, the reason is these datasets are very different. Test dataset has higher depths and a lot of artifacts, which can be easily seen on spectrograms.",
    "2011074": "That's exactly what I mean. So I wonder if there may exist some handcraft filters so that we can visulize the difference between 1s and 0s, I think classifying unvisible features using CNN is not a wise choice.",
    "2010793": "Yes, I have generated some data and trained with it (train + validation + test), and then evaluated the kaggle train dataset, mostly using the common public method (Jun Coda's). \nThis is from one of my runs, without too much tweaking (learning rate, hyperparams, etc).\n\nAs you can see I also get ~2/3 accuracy. The Accuracy on my \"test\" set (generated data) is somewhat meaningless, as it depends on the distribution of depths chosen (sqrtSX / h0). Higher depths are of course harder to predict, and my generated data has hitherto been limited to stationary noise without glitches nor other artifacts.\n\n```\nSize of test set: 330\nAUC of ROC: 0.8968823944539002\nAccuracy: 0.8151515151515152\n\n-- KAGGLE TRAIN SET Evaluation --\nSize of Kaggle train set: 600\nAUC of ROC: 0.735025\nAccuracy: 0.6466666666666666\n```"
  }
}