First of all, thanks @rodrigotenorio for such interesting competition! I have a problem with data generation - I understand that parameters is one of the mysteries that we need to reveal, but anyway I have no clue why it works like this. So:
\nIf we try to get some input from test data (for example), cut SFTs from approximately 4500 to 1024, than we will get an array with shape (1, 2, 360, 1024). Then do flatten for H1 detector - we will get this histplot:![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5452590%2Fd8f484636eafdba807156e3607cf2752%2Fdistrib.jpg?generation=1669717818846626&alt=media\")
\nWith this values:
\nFlatten shape: (368640,)
\nMax: 5.0543144e-22
\nMin: 2.4336356e-25
\nMean: 1.3294535e-22
\nStd: 6.4837456e-23
If I try to do it for noise generation with some default values (scrapped on forums, etc), I will get different result. Generation parameters:
\nwriter_kwargs = {
\n \"label\": \"single_detector_gaussian_noise\",
\n \"outdir\": \"PyFstat_example_data\",
\n \"tstart\": 1238166018,
\n \"duration\": 93 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"F0\": 100.0,
\n \"Band\": 0.2,
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n}
\nFinal shape after flatten for H1 and cutting to 1024 will be the same:
\nFlatten shape: (368640,)
\nMax: 1.204e-42
\nMin: 0.0
\nMean: 9e-44
\nStd: 0.0
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5452590%2Fb2437756d7217a2673fc5087511e40d6%2Fgendistrib.jpg?generation=1669718494046396&alt=media\")
\nWhat is the problem in my parameters and how to do generated data as mush as possible close to original? Thanks!
First of all, thanks @rodrigotenorio for such interesting competition! I have a problem with data generation - I understand that parameters is one of the mysteries that we need to reveal, but anyway I have no clue why it works like this. So:
\nIf we try to get some input from test data (for example), cut SFTs from approximately 4500 to 1024, than we will get an array with shape (1, 2, 360, 1024). Then do flatten for H1 detector - we will get this histplot:
\nWith this values:
\nFlatten shape: (368640,)
\nMax: 5.0543144e-22
\nMin: 2.4336356e-25
\nMean: 1.3294535e-22
\nStd: 6.4837456e-23
If I try to do it for noise generation with some default values (scrapped on forums, etc), I will get different result. Generation parameters:
\nwriter_kwargs = {
\n \"label\": \"single_detector_gaussian_noise\",
\n \"outdir\": \"PyFstat_example_data\",
\n \"tstart\": 1238166018,
\n \"duration\": 93 * 86400,
\n \"detectors\": \"H1,L1\",
\n \"F0\": 100.0,
\n \"Band\": 0.2,
\n \"sqrtSX\": 1e-23,
\n \"Tsft\": 1800,
\n \"SFTWindowType\": \"tukey\",
\n \"SFTWindowBeta\": 0.01,
\n}
\nFinal shape after flatten for H1 and cutting to 1024 will be the same:
\nFlatten shape: (368640,)
\nMax: 1.204e-42
\nMin: 0.0
\nMean: 9e-44
\nStd: 0.0
\n
\nWhat is the problem in my parameters and how to do generated data as mush as possible close to original? Thanks!
What's your 'cut' operation refers to, resize or the first 1024 SFTs?
\nAnd seems these figures have different order of magnitudes, so I guess the second figure is the histplot of the amplitude, but I'm not sure what's the first one.
\nAlso the second figure demonstrates a long tail distribution, which may influence binwidth parameter in the histplot function. Those are lower than 1E-22 in the first figure may be mergerd into the first bar in the second one.
\"Cutting\" operation is just for comfort, I am trying to get distribution of noise, so it is okay to take only part of this noise. \"Bins\" parameters for both histograms same - 50. I think this is problem of parameters for generation, but I dont understand what paramater is responsible for this. Anyway I have very strange Mean and STD and strange range of values on X-axis for
\nany generated noise.
So what's the exact value you are trying to histplot in both figures? Real part/image part/amplitude or squared amplitude?
", "rawMarkdown": "So what's the **exact value** you are trying to histplot in both figures? Real part/image part/amplitude or squared amplitude?", "votes": 1 }, { "id": 2048757, "postDate": "2022-11-29T16:49:46.343Z", "content": "Oh, I finally found an error. For test data i have try to take sqrt of amplitude, but for generated - just amplitude, without sqrt. So stupid, actually. Thank you SO MUCH for right questions <3
\nAnyway, if it is not a secret - where you takes all this parameters values for generation of noise and signal? Some of them looks very non-obvious, for example H0
thank you very much
", "rawMarkdown": "thank you very much" } ] }, { "id": 2048770, "postDate": "2022-11-29T17:00:09.507Z", "content": "You may browse the pinned post by the host and also this post for some insights. Also there are some notebooks for generating data in the Code area.
Hi @kerrit ,
\nSorry for the late reply, it looks like @zzzc18 already solved your problem, good job!
\nI am actually curious about your first plot: What is the continuous line? is it a KDE?
\nFor reference, if you are working with Gaussian noise, the power of your data should follow a chi2 distribution
\nwith 2 degrees of freedom (cause Real and Imaginary parts of the Fourier transform are Gaussian as well
\nand you are adding them as squared numbers).
What's your 'cut' operation refers to, resize or the first 1024 SFTs?
\nAnd seems these figures have different order of magnitudes, so I guess the second figure is the histplot of the amplitude, but I'm not sure what's the first one.
\nAlso the second figure demonstrates a long tail distribution, which may influence binwidth parameter in the histplot function. Those are lower than 1E-22 in the first figure may be mergerd into the first bar in the second one.
\"Cutting\" operation is just for comfort, I am trying to get distribution of noise, so it is okay to take only part of this noise. \"Bins\" parameters for both histograms same - 50. I think this is problem of parameters for generation, but I dont understand what paramater is responsible for this. Anyway I have very strange Mean and STD and strange range of values on X-axis for
\nany generated noise.
So what's the exact value you are trying to histplot in both figures? Real part/image part/amplitude or squared amplitude?
", "votes": 1, "replies": [] }, { "id": 2048757, "author_name": "Kerri", "author_url": "", "post_date": "2022-11-29T16:49:46.343000", "content": "Oh, I finally found an error. For test data i have try to take sqrt of amplitude, but for generated - just amplitude, without sqrt. So stupid, actually. Thank you SO MUCH for right questions <3
\nAnyway, if it is not a secret - where you takes all this parameters values for generation of noise and signal? Some of them looks very non-obvious, for example H0
thank you very much
", "votes": 0, "replies": [] } ] }, { "id": 2048770, "author_name": "zzzc18", "author_url": "", "post_date": "2022-11-29T17:00:09.507000", "content": "You may browse the pinned post by the host and also this post for some insights. Also there are some notebooks for generating data in the Code area.
Hi @kerrit ,
\nSorry for the late reply, it looks like @zzzc18 already solved your problem, good job!
\nI am actually curious about your first plot: What is the continuous line? is it a KDE?
\nFor reference, if you are working with Gaussian noise, the power of your data should follow a chi2 distribution
\nwith 2 degrees of freedom (cause Real and Imaginary parts of the Fourier transform are Gaussian as well
\nand you are adding them as squared numbers).