I have generated around 500k samples of signal and noise, using the parameters I will describe below. I have added these 100k to original 600 and trained effecient_net_b7_ns, using time, frequency masking, flips (horizontal and vertical) and vertical shift augmentations, for 10 epochs. I got validation accuracy above 85%. When I infer the above model on test data, the prediction range is 0.7 to 1, basically my model thinks ALL of test data contains signal.
\nCan anyone identify the mistake I am making? Thanks.
\nSignal generation parameters
\n writer_kwargs = {
\n \"tstart\": 1238166018,
\n \"duration\": 365 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n }
signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(\n priors={\n \"tref\": writer_kwargs[\"tstart\"],\n \"F0\": np.random.randint(50,500),\n \"F1\": -1e-9*(1+np.random.randn()*0.01),\n \"F2\":0,\n \"h0\": 1e-22*0.01*0.01*np.random.uniform(1,100),\n 'cosi': np.random.uniform(0, 1), \n 'psi': 0.7853981633974483*np.random.uniform(-1,1), \n 'phi': np.random.uniform(0,1)*6.283185307179586,\n **pyfstat.injection_parameters.isotropic_amplitude_priors,\n },\n)\nNoise generation parameters,
\n writer_kwargs = {
\n \"tstart\": 1238166018,
\n \"duration\": 365 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n }
\n signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(
\n priors={
\n \"tref\": writer_kwargs[\"tstart\"],
\n \"F0\": np.random.randint(50,500),
\n \"F1\": -1e-9(1+np.random.randn()0.01),
\n \"h0\": 0,
\n **pyfstat.injection_parameters.isotropic_amplitude_priors,
\n },
\n )
The part of the samples, which do contain a signal can be estimated in the following way.
\n1) Suppose we have a model, which returns \"1\" for the samples, which definetely contain a signal and \"0\" otherwise. This model can be constructed by eye-inspection of spectrograms, for example.
\n2) Now we submit this result and get the rocauc score. Let X be the fraction of \"1\" in the submitted result and P the probability of \"1\" in the target distribution. Then the simple formula can be derived: P=X/(2*rocauc-1). It can be proved by considering the rocauc graph, which simply consists of two lines for the case under consideration.
\n3) To apply this method we don't need to select many samples, but we must be sure not to label the absence of signal with \"1\". More samples we select, more accurate results we get.
My estimates give the probability very close to 1/2 for test set. And this differs for the probability of 2/3 of train set. And this is definetely not 1. Of course, if I have selected correct samples.
", "rawMarkdown": "The part of the samples, which do contain a signal can be estimated in the following way.\n1) Suppose we have a model, which returns \"1\" for the samples, which definetely contain a signal and \"0\" otherwise. This model can be constructed by eye-inspection of spectrograms, for example.\n2) Now we submit this result and get the rocauc score. Let X be the fraction of \"1\" in the submitted result and P the probability of \"1\" in the target distribution. Then the simple formula can be derived: P=X/(2*rocauc-1). It can be proved by considering the rocauc graph, which simply consists of two lines for the case under consideration.\n3) To apply this method we don't need to select many samples, but we must be sure not to label the absence of signal with \"1\". More samples we select, more accurate results we get.\n\nMy estimates give the probability very close to 1/2 for test set. And this differs for the probability of 2/3 of train set. And this is definetely not 1. Of course, if I have selected correct samples.", "votes": 1, "replies": [ { "id": 2081364, "postDate": "2022-12-31T05:38:25.333Z", "content": "Thanks for the insight, Konstantin.
", "rawMarkdown": "Thanks for the insight, Konstantin.", "replies": [ { "id": 2081463, "postDate": "2022-12-31T07:55:26.103Z", "content": "\n", "rawMarkdown": "> Thanks for the insight, Konstantin.\n\n" } ] } ] }, { "id": 2080749, "postDate": "2022-12-30T12:56:30.347Z", "content": "Thanks for the insight, Konstantin.
\n
You can check the data in the form of pictures, or check whether the distribution of the val dataset and the test dataset are consistent.^W^
", "rawMarkdown": "You can check the data in the form of pictures, or check whether the distribution of the val dataset and the test dataset are consistent.^W^", "votes": 1, "replies": [ { "id": 2080756, "postDate": "2022-12-30T13:00:53.207Z", "content": "I checked in form of pictures and all seems fine, also, I can't check distribution of snr, F0 etc, because, unfortunately, when generating, I threw away all of them only stored 2,360,350 npy files, due to HUGE data occupation of hdf5 files.
", "rawMarkdown": "I checked in form of pictures and all seems fine, also, I can't check distribution of snr, F0 etc, because, unfortunately, when generating, I threw away all of them only stored 2,360,350 npy files, due to HUGE data occupation of hdf5 files.", "votes": 1, "replies": [ { "id": 2080760, "postDate": "2022-12-30T13:11:02.963Z", "content": "What about mean and variance? If they were not the same, it would be very different.
", "rawMarkdown": "What about mean and variance? If they were not the same, it would be very different." }, { "id": 2080766, "postDate": "2022-12-30T13:17:19.193Z", "content": "Do you mean, mean and variance of frequency? If so, the above parameters provide that info right? If not, can you tell me what parameter's mean and variance to check? Thanks
", "rawMarkdown": "Do you mean, mean and variance of frequency? If so, the above parameters provide that info right? If not, can you tell me what parameter's mean and variance to check? Thanks" }, { "id": 2080772, "postDate": "2022-12-30T13:27:53.510Z", "content": "Well, that is to normalize the data before feeding it into the model, but in your previous discussion, I found that you may have normalized the data, if there is no problem, then I don't know how to solve it,sorry.
", "rawMarkdown": "Well, that is to normalize the data before feeding it into the model, but in your previous discussion, I found that you may have normalized the data, if there is no problem, then I don't know how to solve it,sorry.", "votes": 1 }, { "id": 2080785, "postDate": "2022-12-30T13:35:50.150Z", "content": "Thanks, I was actually hoping to see if there is any issue with my parameters I have used for generation primarily,as for the normalization, I have tried without normalization, and am yet to normalize the data, hopefully normalization will help. Meanwhile, I also wanted to know if there is something wrong with my parameters, hence the discussion. Thanks for the help, hopefully normalization is the only issue. BTW, any particular normalization you prefer over others?
", "rawMarkdown": "Thanks, I was actually hoping to see if there is any issue with my parameters I have used for generation primarily,as for the normalization, I have tried without normalization, and am yet to normalize the data, hopefully normalization will help. Meanwhile, I also wanted to know if there is something wrong with my parameters, hence the discussion. Thanks for the help, hopefully normalization is the only issue. BTW, any particular normalization you prefer over others?\n", "votes": 1 } ] } ] }, { "id": 2080725, "postDate": "2022-12-30T12:27:53.810Z", "content": "Did you generate samples without a signal?
", "rawMarkdown": "Did you generate samples without a signal?", "votes": 1, "replies": [ { "id": 2080726, "postDate": "2022-12-30T12:34:58.783Z", "content": "Yes. These are the noise generation parameters.
\nNoise generation parameters,
\nNoise generation parameters,
\nwriter_kwargs = {
\n\"tstart\": 1238166018,
\n\"duration\": 365 * 86400,
\n\"detectors\": \"H1,L1\",
\n\"sqrtSX\": 1e-23,
\n\"Tsft\": 1800,
\n\"SFTWindowType\": \"tukey\",
\n\"SFTWindowBeta\": 0.01,
\n}
\nsignal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(
\npriors={
\n\"tref\": writer_kwargs[\"tstart\"],
\n\"F0\": np.random.randint(50,500),
\n\"F1\": -1e-9(1+np.random.randn()0.01),
\n\"h0\": 0,
\n**pyfstat.injection_parameters.isotropic_amplitude_priors,
\n},
\n)
Anyway, the absolute value of your prediction doesn't matter for AUC. It's their relative position to each other. So if your prediction of no signal is 0.9 and for signal is 0.95 you still have a perfect AUCROC score.
", "rawMarkdown": "Anyway, the absolute value of your prediction doesn't matter for AUC. It's their relative position to each other. So if your prediction of no signal is 0.9 and for signal is 0.95 you still have a perfect AUCROC score.", "votes": 1 } ] }, { "id": 2080717, "postDate": "2022-12-30T12:15:45.753Z", "content": "I have generated around 500k samples of signal and noise, using the parameters I will describe below. I have added these 100k to original 600 and trained effecient_net_b7_ns, using time, frequency masking, flips (horizontal and vertical) and vertical shift augmentations, for 10 epochs. I got validation accuracy above 85%. When I infer the above model on test data, the prediction range is 0.7 to 1, basically my model thinks ALL of test data contains signal.
\nCan anyone identify the mistake I am making? Thanks.
\nSignal generation parameters
\n writer_kwargs = {
\n \"tstart\": 1238166018,
\n \"duration\": 365 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n }
signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(\n priors={\n \"tref\": writer_kwargs[\"tstart\"],\n \"F0\": np.random.randint(50,500),\n \"F1\": -1e-9*(1+np.random.randn()*0.01),\n \"F2\":0,\n \"h0\": 1e-22*0.01*0.01*np.random.uniform(1,100),\n 'cosi': np.random.uniform(0, 1), \n 'psi': 0.7853981633974483*np.random.uniform(-1,1), \n 'phi': np.random.uniform(0,1)*6.283185307179586,\n **pyfstat.injection_parameters.isotropic_amplitude_priors,\n },\n)\nNoise generation parameters,
\n writer_kwargs = {
\n \"tstart\": 1238166018,
\n \"duration\": 365 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n }
\n signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(
\n priors={
\n \"tref\": writer_kwargs[\"tstart\"],
\n \"F0\": np.random.randint(50,500),
\n \"F1\": -1e-9(1+np.random.randn()0.01),
\n \"h0\": 0,
\n **pyfstat.injection_parameters.isotropic_amplitude_priors,
\n },
\n )
The part of the samples, which do contain a signal can be estimated in the following way.
\n1) Suppose we have a model, which returns \"1\" for the samples, which definetely contain a signal and \"0\" otherwise. This model can be constructed by eye-inspection of spectrograms, for example.
\n2) Now we submit this result and get the rocauc score. Let X be the fraction of \"1\" in the submitted result and P the probability of \"1\" in the target distribution. Then the simple formula can be derived: P=X/(2*rocauc-1). It can be proved by considering the rocauc graph, which simply consists of two lines for the case under consideration.
\n3) To apply this method we don't need to select many samples, but we must be sure not to label the absence of signal with \"1\". More samples we select, more accurate results we get.
My estimates give the probability very close to 1/2 for test set. And this differs for the probability of 2/3 of train set. And this is definetely not 1. Of course, if I have selected correct samples.
", "votes": 1, "replies": [ { "id": 2081364, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-31T05:38:25.333000", "content": "Thanks for the insight, Konstantin.
", "votes": 0, "replies": [ { "id": 2081463, "author_name": "Wwwwhy", "author_url": "", "post_date": "2022-12-31T07:55:26.103000", "content": "\n", "votes": 0, "replies": [] } ] } ] }, { "id": 2080749, "author_name": "yoyobar", "author_url": "", "post_date": "2022-12-30T12:56:30.347000", "content": "Thanks for the insight, Konstantin.
\n
You can check the data in the form of pictures, or check whether the distribution of the val dataset and the test dataset are consistent.^W^
", "votes": 1, "replies": [ { "id": 2080756, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-30T13:00:53.207000", "content": "I checked in form of pictures and all seems fine, also, I can't check distribution of snr, F0 etc, because, unfortunately, when generating, I threw away all of them only stored 2,360,350 npy files, due to HUGE data occupation of hdf5 files.
", "votes": 1, "replies": [ { "id": 2080760, "author_name": "yoyobar", "author_url": "", "post_date": "2022-12-30T13:11:02.963000", "content": "What about mean and variance? If they were not the same, it would be very different.
", "votes": 0, "replies": [] }, { "id": 2080766, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-30T13:17:19.193000", "content": "Do you mean, mean and variance of frequency? If so, the above parameters provide that info right? If not, can you tell me what parameter's mean and variance to check? Thanks
", "votes": 0, "replies": [] }, { "id": 2080772, "author_name": "yoyobar", "author_url": "", "post_date": "2022-12-30T13:27:53.510000", "content": "Well, that is to normalize the data before feeding it into the model, but in your previous discussion, I found that you may have normalized the data, if there is no problem, then I don't know how to solve it,sorry.
", "votes": 1, "replies": [] }, { "id": 2080785, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-30T13:35:50.150000", "content": "Thanks, I was actually hoping to see if there is any issue with my parameters I have used for generation primarily,as for the normalization, I have tried without normalization, and am yet to normalize the data, hopefully normalization will help. Meanwhile, I also wanted to know if there is something wrong with my parameters, hence the discussion. Thanks for the help, hopefully normalization is the only issue. BTW, any particular normalization you prefer over others?
", "votes": 1, "replies": [] } ] } ] }, { "id": 2080725, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-12-30T12:27:53.810000", "content": "Did you generate samples without a signal?
", "votes": 1, "replies": [ { "id": 2080726, "author_name": "Aaftaab V", "author_url": "", "post_date": "2022-12-30T12:34:58.783000", "content": "Yes. These are the noise generation parameters.
\nNoise generation parameters,
\nNoise generation parameters,
\nwriter_kwargs = {
\n\"tstart\": 1238166018,
\n\"duration\": 365 * 86400,
\n\"detectors\": \"H1,L1\",
\n\"sqrtSX\": 1e-23,
\n\"Tsft\": 1800,
\n\"SFTWindowType\": \"tukey\",
\n\"SFTWindowBeta\": 0.01,
\n}
\nsignal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(
\npriors={
\n\"tref\": writer_kwargs[\"tstart\"],
\n\"F0\": np.random.randint(50,500),
\n\"F1\": -1e-9(1+np.random.randn()0.01),
\n\"h0\": 0,
\n**pyfstat.injection_parameters.isotropic_amplitude_priors,
\n},
\n)
Anyway, the absolute value of your prediction doesn't matter for AUC. It's their relative position to each other. So if your prediction of no signal is 0.9 and for signal is 0.95 you still have a perfect AUCROC score.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2081172": "The part of the samples, which do contain a signal can be estimated in the following way.\n1) Suppose we have a model, which returns \"1\" for the samples, which definetely contain a signal and \"0\" otherwise. This model can be constructed by eye-inspection of spectrograms, for example.\n2) Now we submit this result and get the rocauc score. Let X be the fraction of \"1\" in the submitted result and P the probability of \"1\" in the target distribution. Then the simple formula can be derived: P=X/(2*rocauc-1). It can be proved by considering the rocauc graph, which simply consists of two lines for the case under consideration.\n3) To apply this method we don't need to select many samples, but we must be sure not to label the absence of signal with \"1\". More samples we select, more accurate results we get.\n\nMy estimates give the probability very close to 1/2 for test set. And this differs for the probability of 2/3 of train set. And this is definetely not 1. Of course, if I have selected correct samples.", "2080749": "You can check the data in the form of pictures, or check whether the distribution of the val dataset and the test dataset are consistent.^W^", "2080725": "Did you generate samples without a signal?", "2080717": "I have generated around 500k samples of signal and noise, using the parameters I will describe below. I have added these 100k to original 600 and trained effecient_net_b7_ns, using time, frequency masking, flips (horizontal and vertical) and vertical shift augmentations, for 10 epochs. I got validation accuracy above 85%. When I infer the above model on test data, the prediction range is 0.7 to 1, basically my model thinks ALL of test data contains signal.\n\nCan anyone identify the mistake I am making? Thanks.\n\nSignal generation parameters\n writer_kwargs = {\n \"tstart\": 1238166018,\n \"duration\": 365 * 86400, \n \"detectors\": \"H1,L1\", \n \"sqrtSX\": 1e-23, \n \"Tsft\": 1800, \n \"SFTWindowType\": \"tukey\", \n \"SFTWindowBeta\": 0.01,\n }\n\n signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(\n priors={\n \"tref\": writer_kwargs[\"tstart\"],\n \"F0\": np.random.randint(50,500),\n \"F1\": -1e-9*(1+np.random.randn()*0.01),\n \"F2\":0,\n \"h0\": 1e-22*0.01*0.01*np.random.uniform(1,100),\n 'cosi': np.random.uniform(0, 1), \n 'psi': 0.7853981633974483*np.random.uniform(-1,1), \n 'phi': np.random.uniform(0,1)*6.283185307179586,\n **pyfstat.injection_parameters.isotropic_amplitude_priors,\n },\n )\n\nNoise generation parameters,\n writer_kwargs = {\n \"tstart\": 1238166018,\n \"duration\": 365 * 86400, \n \"detectors\": \"H1,L1\", \n \"sqrtSX\": 1e-23, \n \"Tsft\": 1800, \n \"SFTWindowType\": \"tukey\", \n \"SFTWindowBeta\": 0.01,\n }\n signal_parameters_generator = pyfstat.AllSkyInjectionParametersGenerator(\n priors={\n \"tref\": writer_kwargs[\"tstart\"],\n \"F0\": np.random.randint(50,500),\n \"F1\": -1e-9*(1+np.random.randn()*0.01),\n \"h0\": 0,\n **pyfstat.injection_parameters.isotropic_amplitude_priors,\n },\n )" } }