Hi Quentin
\nThanks for giving a try to PyFstat.
\n\n\nI understood that I need to create an instance of Writer and put the .sft file in the \"noiseSFTs\" argument
\n
Not quite. As explained in the tutorial, the Writer class is used to generate more data.
Once the data is generated as SFT files, you should use the get_sft_as_arrays function (from pyfstat.utils) to read the SFTs you just created into Numpy arrays (see last line of the first cell of Tutorial 1).
In your case, if your SFTs are in PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft, you could read them as
from pyfstat.utils import get_sft_as_array\n\nfrequency, timestamps, sfts = get_sft_as_array(\"PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft\")\nAs a side note, the path where the SFTs are created is stored in the sftfilepath attribute of Writer
\n(mind that it gets overwritten every time make_data is called!).
\n\nI also added \"SFTWindowType\": \"tukey\", which seems to be a mandatory argument according to lalpulsar_Makefakedata_v5 --help
\n
noiseSFTs is used to add noise or a signal into a specific file of SFTs. In that case, one needs to know which window
\nfunction was used, as failing to do so may significantly bias the SNR of a signal. You don't need this to read your data, but
\nit's worth to keep it in mind what window you are using whenever you generate more.
\n\nIf I run lalpuslar directly in a terminal it doesn't give any error, which makes me think my arguments are fine
\n
If you give noiseSFTs to MFD it will do nothing with them and terminate without producing any output.
\nGiven your input arguments, however, that should have thrown an error, as specifying tukey requires specifying
\na corresponding beta parameter, so… thanks for finding that!
Ok, that's much clearer now, thanks for the detailed reply!
", "rawMarkdown": "Ok, that's much clearer now, thanks for the detailed reply!", "votes": 1 } ] }, { "id": 1972036, "postDate": "2022-10-04T23:24:32.757Z", "content": "Hi R. Tenorio,
\nI was trying to work with PyFstat. However, I got that module error and couldn't go on.
\nModuleNotFoundError: No module named 'tutorial_utils'
\nI've installed PyFstat
\n!pip install pyfstat
\n!pip install git+https://github.com/PyFstat/PyFstat@python37
\nAnd even !pip install pyfstat jupyter
\nThough the: No module named 'tutorial_utils' persists. Any tip?
\nThanks in advance,
\nMarília Prata
Hi Marília,
\nIt sounds like you're running through the tutorials. Both tutorials make use of a local file called tutorial_utils.py which can be found in the same directory as the tutorial notebooks.
The error you're getting would imply that that file (tutorial_utils.py) isn't in the same directory as the notebooks. So you'll either need to copy that file to the directory you're running the notebooks in or edit your copies of tutorials so that they don't import it and instead define the functions it provides in the notebook.
Hope this helps,
\nMichael
I didn't get this e-mail. Thank you for answering it Michael.
", "rawMarkdown": "I didn't get this e-mail. Thank you for answering it Michael." }, { "id": 1990527, "postDate": "2022-10-16T15:42:25.997Z", "content": "here: https://www.kaggle.com/code/crischir/pyfstat-tutorial-adapted-to-kaggle tutorial adapted to kaggle: ai utils and tutorial_utils.py as functions and sanserif deleted from matlib plots :)
", "rawMarkdown": "here: https://www.kaggle.com/code/crischir/pyfstat-tutorial-adapted-to-kaggle tutorial adapted to kaggle: ai utils and tutorial_utils.py as functions and sanserif deleted from matlib plots :)" }, { "id": 1991637, "postDate": "2022-10-17T09:59:18.130Z", "content": "Hi George Chirita,
\nAwesome, I'm sure this will be very useful to newcomers!
\nSince this is largely a verbatim copy of the PyFstat tutorials, would you mind including that link at the top of your kernel? Something like the following would suffice
\nThe contents of this notebook were adapted from [the PyFstat tutorials](https://github.com/PyFstat/PyFstat/tree/master/examples/tutorials).\nThank you very much.
\n\n\nsanserif deleted from matlib plots
\n
Before the challenge I spent some time making sure there were no problems at plotting time (sometimes LaTeX was requested, pointlessly complicating the installation procedure). Did you have any specific issue with the current plotting routines or is this just a personal stylistic choice? If the former, may I ask you to fill up an issue so we can have a look at it?)
", "rawMarkdown": "Hi George Chirita,\n\nAwesome, I'm sure this will be very useful to newcomers!\n\nSince this is largely a verbatim copy of [the PyFstat tutorials](https://github.com/PyFstat/PyFstat/tree/master/examples/tutorials), would you mind including that link at the top of your kernel? Something like the following would suffice\n```\nThe contents of this notebook were adapted from [the PyFstat tutorials](https://github.com/PyFstat/PyFstat/tree/master/examples/tutorials).\n```\n\nThank you very much.\n\n\n> sanserif deleted from matlib plots\n\nBefore the challenge I spent some time making sure there were no problems at plotting time (sometimes LaTeX was requested, pointlessly complicating the installation procedure). Did you have any specific issue with the current plotting routines or is this just a personal stylistic choice? If the former, may I ask you to [fill up an issue](https://github.com/PyFstat/PyFstat/issues) so we can have a look at it?)", "votes": 2 }, { "id": 1994193, "postDate": "2022-10-18T20:39:49.280Z", "content": "I've put the acknowledgement in two places (also the title is obvious) . Please tell me if it is ok. I do not need credit for this notebook, it was adpted for comunity, and for my presonal gain :) learning.
\nsans serif trigger latex error in kaggle the (actual version of the container), It is documented here:
\nhttps://stackoverflow.com/questions/11354149/python-unable-to-render-tex-in-matplotlib
\nit is an issue of linux version i think :) there are some packages to be installed and this is complicated for a begginer user in kaggle.
There is a difference in the shape of data between the competition and tutorial?
\n\"Tsft\": 1800, # Fourier transform time duration in tutorial (shape in competition is 360. ) is data trimmed?
Cool, that's grand, thank you. I'm sure this is going to be useful to whoever comes next into this challenge.
\n\n\nsans serif trigger latex error in kaggle the (actual version of the container), It is documented here:
\n
\n https://stackoverflow.com/questions/11354149/python-unable-to-render-tex-in-matplotlib
That is a 10y/o thread and I don't seem to see where in your kernel you solve that problem, but fair if it works for you.
\nFor the record, I don't think I ran into that when I last checked the kernel.
\n\nThere is a difference in the shape of data between the competition and tutorial?
\n
\n \"Tsft\": 1800, # Fourier transform time duration in tutorial (shape in competition is 360. ) is data trimmed?
So, Tsft refers to the duration over which a Fourier transform was taken (i.e. each timestamp contains 1800s of data on which a Fourier transform was taken). The 360 is the number of bins in the frequency axis (i.e. how many frequencies does your Fourier transform contain). Since frequency resolution is 1/1800, what you get is that your data spans 360 * ( 1/1800) = 0.2 Hz, as you can see in the plots.
Thank you for your great work.
\nYou have right, in the verbatim copy of the PyFstat tutorials in work great. In my personal notebook it was not, probably a conflicted library, it is not an issue anyway.
\nthanks for clarify the 0.2Hz
\n\"Band\": 0.2, # Frequency band-width around F0 [Hz]
@sakvaua @ayuraj the generated data will surely contain the GW ? can we assign the taret for these synthetic data as 1
", "rawMarkdown": "@sakvaua @ayuraj the generated data will surely contain the GW ? can we assign the taret for these synthetic data as 1" }, { "id": 3103228, "postDate": "2025-01-23T07:17:41.833Z", "content": "Thank you for the competition and the effort you made to set it up!
", "rawMarkdown": "Thank you for the competition and the effort you made to set it up!" }, { "id": 2026697, "postDate": "2022-11-12T09:40:25.003Z", "content": "Hi, @rodrigotenorio
\nI'm trying to inject a simulated line-like detector artifact into a pre-generated SFT (passed with noiseSFTSs parameter) and apparently LineWriter cuts my frequency band to a minimum band that fits this line. For example:
\nSFTS minimum frequency BEFORE the injection 78.87 maximum 79.67
\nLineWriter kwargs:
\n{
\n'noiseSFTs': '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_H1_band_signal-1238170665-10367456.sft',
\n'F0': 79.22547569407091,
\n'h0': 9.241657937942324e-24,
\n'phi': 1.8502800869842844,
\n'SFTWindowType': 'tukey',
\n'SFTWindowBeta': 0.001,
\n'outdir': '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/'}
Log
\n22-11-12 11:21:45.435 pyfstat.core INFO : Creating LineWriter object…
\n22-11-12 11:21:45.436 pyfstat.utils.ephemeris INFO : No /home/sakvaua/.pyfstat.conf file found. Will fall back to lal's automatic path resolution for files [earth00-40-DE405.dat.gz,sun00-40-DE405.dat.gz]. Alternatively, set 'earth_ephem' and 'sun_ephem' class options.
\n22-11-12 11:21:45.453 pyfstat.make_sfts WARNING : noiseSFTs is not None: Inferring tstart, duration, Tsft. Input tstart and duration will be treated as SFT constraints using lalpulsar.SFTConstraints; Tsft will be checked for internal consistency accross input SFTs.
\n22-11-12 11:21:45.460 pyfstat.make_sfts INFO : SFT Constraints: [minStartTime:None, maxStartTime:None]
\n22-11-12 11:21:45.608 pyfstat.make_sfts WARNING : Injection of line artifacts only uses the following parameters:
\n['F0', 'phi', 'h0'].
\nAny other parameter will be purged from this class now
\n22-11-12 11:21:45.609 pyfstat.make_sfts INFO : Purging input parameters that are not meaningful for LineWriter: ['refTime', 'f1dot', 'psi', 'transientWindowType', 'f2dot']
\n22-11-12 11:21:45.610 pyfstat.make_sfts INFO : Estimating required SFT frequency range from properties of signal to inject plus 59 extra bins either side (corresponding to default F-statistic settings).
\n22-11-12 11:21:45.664 pyfstat.make_sfts INFO : Generating SFTs with fmin=79.18430415750552, Band=0.08234307313076569
\n22-11-12 11:21:45.665 pyfstat.make_sfts INFO : Checking if we can re-use injection config file…
\n22-11-12 11:21:45.666 pyfstat.make_sfts INFO : …OK: config file /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff already exists.
\n22-11-12 11:21:45.668 pyfstat.make_sfts INFO : …file contents unmatched, updating /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff.
\n22-11-12 11:21:45.668 pyfstat.make_sfts INFO : Writing config file: /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff
\n22-11-12 11:21:45.670 pyfstat.make_sfts INFO : Checking if we can re-use existing SFT data file(s)…
\n22-11-12 11:21:45.671 pyfstat.make_sfts INFO : …no SFT file matching '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft' found. Will create new SFT file(s).
\n22-11-12 11:21:45.672 pyfstat.utils.cli INFO : Now executing: lalpulsar_Makefakedata_v4 --lineFeature=TRUE --outSingleSFT=TRUE --outSFTbname=\"/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft\" --IFO=\"H1\" --noiseSFTs=\"/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_H1_band_signal-1238170665-10367456.sft\" --window=\"tukey\" --tukeyBeta=0.001 --startTime=1238170665 --duration=10367456 --fmin=79.18430415750552 --Band=0.08234307313076569 --Tsft=1800 --h0=9.241657937942324e-24 --Freq=79.22547569407091 --phi0=1.850280086984284 --cosi=0 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"
\n22-11-12 11:21:48.905 pyfstat.utils.cli INFO :
\n22-11-12 11:21:48.908 pyfstat.utils.cli INFO : WARNING: for SFT-creation we had to adjust (fmin,Band) to fmin_eff=79.1838888888889 and Band_eff=0.0827777777777778
\n22-11-12 11:21:48.909 pyfstat.utils.cli INFO :
\n22-11-12 11:21:48.911 pyfstat.make_sfts INFO : Successfully wrote SFTs to: /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft
\n22-11-12 11:21:48.912 pyfstat.make_sfts INFO : Now validating each SFT file…
And AFTER the LineWriter the minimum frequency is 79.18 and the maximum is 79.26 (was 78.87 - 79.67 before )
\nWhy does it have to adjust the fmin and Band?
\nThanks!
Hi @sakvaua ,
\nThis is another effect of not having direct access to fmin from Writer and derived classes, together with the fact that you did not specify a Band.
Since you didn't specify Band, MFDv4 assumed it so that you could run a fully-targeted F-statistic search.
\nIf you specify Band, on the other hand, this line applies and you'll get the same behavior as whenever using Writer.
Let me know if this helped.
\nAs a second option, you could as well generate several instances of \"noisy data\" with the same frequency range and then add them together at numpy-array level, as I said in this answer.
", "rawMarkdown": "Hi @sakvaua ,\n\nThis is another effect of not having direct access to `fmin` from `Writer` and derived classes, together with the fact that you did not specify a `Band`.\n\nSince you didn't specify `Band`, MFDv4 assumed it so that you could run a fully-targeted F-statistic search.\nIf you specify `Band`, on the other hand, [this line](https://github.com/PyFstat/PyFstat/blob/master/pyfstat/make_sfts.py#L528) applies and you'll get the same behavior as whenever using `Writer`.\n\nLet me know if this helped.\n\nAs a second option, you could as well generate several instances of \"noisy data\" with the same frequency range and then add them together at numpy-array level, as I said in [this answer](https://github.com/PyFstat/PyFstat/issues/499).\n", "votes": 1 }, { "id": 2030108, "postDate": "2022-11-15T07:46:33.407Z", "content": "Hi, Rodrigo. Thank you so very much for your time and answers.
\nWell, the noise stft I'm passing has all the information needed (like frequencies and timestamps) so I'm not sure why this writer needs to have the band specified. I currently use a workaround by providing Band, but the problem is that it is centered around F0 and if F0 in my STFT and F0 in my LineWriter are different - then the generated frequency bands for Line Stfts and my Noise STFTs will also be different. Thus making simple summation impossible. So I have to pick LineWriter F0 from the exact frequencies of the noise SFTS I generated.
Hi @sakvaua ,
\nApologies for the late replay.
\nI am afraid your solution is basically the way of doing this at the moment. IT is far from being perfect, but roughly it's equivalent to the way we sometimes do some of our analyses. (The main difference is that we tend to use LALSuite structures rather than numpy arrays, meaning codes get a bit messier, but that's roughly it).
\nI do agree with you in that we should really rethink what's going on with this class in PyFstat.
", "rawMarkdown": "Hi @sakvaua ,\n\nApologies for the late replay.\n\nI am afraid your solution is basically *the* way of doing this at the moment. IT is far from being perfect, but roughly it's equivalent to the way we sometimes do some of our analyses. (The main difference is that we tend to use LALSuite structures rather than numpy arrays, meaning codes get a bit messier, but that's roughly it).\n\nI do agree with you in that we should really rethink what's going on with this class in PyFstat. \n\n", "votes": 1 } ] }, { "id": 2025328, "postDate": "2022-11-11T06:44:49.913Z", "content": "Thanks for such a wonderful project! What books should I read to make sense of data？
", "rawMarkdown": "Thanks for such a wonderful project! What books should I read to make sense of data？" }, { "id": 2022761, "postDate": "2022-11-09T08:50:00.787Z", "content": "@rodrigotenorio
\nThank you for the interesting competition!
\nDoes test data is from the same gravitational-wave interferometers as training ones (LIGO Hanford & LIGO Livingston)?
Hi @hanejiyuto ,
\nYes, we only use the Advanced LIGO interferometers (Hanford and Livingstone) in this competition 👍.
", "rawMarkdown": "Hi @hanejiyuto ,\n\nYes, we only use the Advanced LIGO interferometers (Hanford and Livingstone) in this competition :+1:.", "votes": 1 }, { "id": 2037038, "postDate": "2022-11-20T11:49:06.043Z", "content": "Thanks for your reply!
", "rawMarkdown": "Thanks for your reply!" } ] }, { "id": 2020602, "postDate": "2022-11-07T16:20:55.667Z", "content": "Hello @rodrigotenorio ! Thanks for the informative tutorial and sample code.
\nI used pyfstat.yml from the link below to build my environment in my conda environment.
\n https://github.com/PyFstat/PyFstat/wiki/conda-environments
However, I am unable to import pyfstat by any means due to the following error.
\nSorry for the long post. Could you please give me some good advice?
\n
RuntimeError Traceback (most recent call last)
\nCell In [2], line 7
\n 4 import numpy as np
\n 5 import matplotlib.pyplot as plt
\n----> 7 import pyfstat
\n 9 from scipy import stats
\n 11 get_ipython().run_line_magic('matplotlib', 'inline')
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/init.py:37
\n 32 from .gridcorner import gridcorner
\n 33 from .injection_parameters import (
\n 34 AllSkyInjectionParametersGenerator,
\n 35 InjectionParametersGenerator,
\n 36 )
\n---> 37 from .make_sfts import (
\n 38 BinaryModulatedWriter,
\n 39 FrequencyAmplitudeModulatedArtifactWriter,
\n 40 FrequencyModulatedArtifactWriter,
\n 41 GlitchWriter,
\n 42 LineWriter,
\n 43 Writer,
\n 44 )
\n 45 from .mcmc_based_searches import (
\n 46 MCMCFollowUpSearch,
\n 47 MCMCGlitchSearch,
\n (…)
\n 50 MCMCTransientSearch,
\n 51 )
\n 52 from .snr import DetectorStates, SignalToNoiseRatio
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:21
\n 16 from pyfstat.core import BaseSearchClass, SearchForSignalWithJumps
\n 18 logger = logging.getLogger(name)
\n---> 21 class Writer(BaseSearchClass):
\n 22 \"\"\"The main class for generating data in the form of SFTs.
\n 23
\n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,
\n (…)
\n 35 for more detailed help with some of the parameters.
\n 36 \"\"\"
\n 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:38, in Writer()
\n 21 class Writer(BaseSearchClass):
\n 22 \"\"\"The main class for generating data in the form of SFTs.
\n 23
\n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,
\n (…)
\n 35 for more detailed help with some of the parameters.
\n 36 \"\"\"
\n---> 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")
\n 39 \"\"\"The executable; can be overridden by child classes.\"\"\"
\n 41 signal_parameter_labels = [
\n 42 \"tref\",
\n 43 \"F0\",
\n (…)
\n 54 \"transientTau\",
\n 55 ]
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/utils/runlalsuite.py:35, in get_lal_exec(cmd)
\n 33 full_cmd = shutil.which(\"lalpulsar_\" + cmd) or shutil.which(\"lalapps_\" + cmd)
\n 34 if full_cmd is None:
\n---> 35 raise RuntimeError(
\n 36 f\"Could not find either lalpulsar or lalapps version of command {cmd}.\"
\n 37 )
\n 38 return os.path.basename(full_cmd)
RuntimeError: Could not find either lalpulsar or lalapps version of command Makefakedata_v5.
", "rawMarkdown": "Hello @rodrigotenorio ! Thanks for the informative tutorial and sample code.\n\nI used pyfstat.yml from the link below to build my environment in my conda environment. \n https://github.com/PyFstat/PyFstat/wiki/conda-environments\n\nHowever, I am unable to import pyfstat by any means due to the following error.\nSorry for the long post. Could you please give me some good advice?\n\n---------------------------------------------------------------------------\nRuntimeError Traceback (most recent call last)\nCell In [2], line 7\n 4 import numpy as np\n 5 import matplotlib.pyplot as plt\n----> 7 import pyfstat\n 9 from scipy import stats\n 11 get_ipython().run_line_magic('matplotlib', 'inline')\n\nFile ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/__init__.py:37\n 32 from .gridcorner import gridcorner\n 33 from .injection_parameters import (\n 34 AllSkyInjectionParametersGenerator,\n 35 InjectionParametersGenerator,\n 36 )\n---> 37 from .make_sfts import (\n 38 BinaryModulatedWriter,\n 39 FrequencyAmplitudeModulatedArtifactWriter,\n 40 FrequencyModulatedArtifactWriter,\n 41 GlitchWriter,\n 42 LineWriter,\n 43 Writer,\n 44 )\n 45 from .mcmc_based_searches import (\n 46 MCMCFollowUpSearch,\n 47 MCMCGlitchSearch,\n (...)\n 50 MCMCTransientSearch,\n 51 )\n 52 from .snr import DetectorStates, SignalToNoiseRatio\n\nFile ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:21\n 16 from pyfstat.core import BaseSearchClass, SearchForSignalWithJumps\n 18 logger = logging.getLogger(__name__)\n---> 21 class Writer(BaseSearchClass):\n 22 \"\"\"The main class for generating data in the form of SFTs.\n 23 \n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,\n (...)\n 35 for more detailed help with some of the parameters.\n 36 \"\"\"\n 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")\n\nFile ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:38, in Writer()\n 21 class Writer(BaseSearchClass):\n 22 \"\"\"The main class for generating data in the form of SFTs.\n 23 \n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,\n (...)\n 35 for more detailed help with some of the parameters.\n 36 \"\"\"\n---> 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")\n 39 \"\"\"The executable; can be overridden by child classes.\"\"\"\n 41 signal_parameter_labels = [\n 42 \"tref\",\n 43 \"F0\",\n (...)\n 54 \"transientTau\",\n 55 ]\n\nFile ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/utils/runlalsuite.py:35, in get_lal_exec(cmd)\n 33 full_cmd = shutil.which(\"lalpulsar_\" + cmd) or shutil.which(\"lalapps_\" + cmd)\n 34 if full_cmd is None:\n---> 35 raise RuntimeError(\n 36 f\"Could not find either lalpulsar or lalapps version of command {cmd}.\"\n 37 )\n 38 return os.path.basename(full_cmd)\n\nRuntimeError: Could not find either lalpulsar or lalapps version of command Makefakedata_v5.\n", "replies": [ { "id": 2020626, "postDate": "2022-11-07T16:41:40.560Z", "content": "Hi @kami2suukyi ,
\nThanks for the post.
\nI see you are using Python 3.11…
\ncould you try to re-do your environment using Python 3.10 and see if the issue still persists?
@rodrigotenorio Thanks for the quick reply.
\nI followed your advice and created a clean environment with the following command:
\n$ conda create -n pyfstat python=3.10
Then I installed only pyfstat and jupyter.
\n$ conda activate pyfstat
\n$ conda install -c conda-forge pyfstat jupyter
But I got the exact same error as before. I had the same error with python=3.7 too.
", "rawMarkdown": "@rodrigotenorio Thanks for the quick reply.\n\nI followed your advice and created a clean environment with the following command: \n$ conda create -n pyfstat python=3.10\n\nThen I installed only pyfstat and jupyter.\n$ conda activate pyfstat\n$ conda install -c conda-forge pyfstat jupyter\n\nBut I got the exact same error as before. I had the same error with python=3.7 too." }, { "id": 2021862, "postDate": "2022-11-08T14:24:12.287Z", "content": "Hi @kami2suukyi ,
\nWhat operative system are you using? Could you open up a terminal with the conda environment activated
\nand run the following command:
$ which lalapps_version\n@rodrigotenorio My operating system is Ubuntu 20.04.5 LTS.
\nThe result of typing is as follows:
\n/home/USER/.local/bin/lalapps_version
Thanks again and again for your advice.
", "rawMarkdown": "@rodrigotenorio My operating system is Ubuntu 20.04.5 LTS.\nThe result of typing is as follows: \n/home/USER/.local/bin/lalapps_version\n\nThanks again and again for your advice." }, { "id": 2022158, "postDate": "2022-11-08T19:06:10.873Z", "content": "Hi @kami2suukyi , I'm surprised by that result (having that executable under $USER/.local/bin).
\nHave you, by any chance, installed lalsuite or pyfstat using your system's Python? (i.e. running pip install lalsuite or pip install --user lalsuite without any kind of virtual environment)?
Hello @rodrigotenorio, thanks to your advice I understood that there is a problem with the path to lalapps_Makefakedata_v5. I am sure I have pip install lalasuite because I have tried many things and did not work.
\nI pip uninstalled lalasuite and lscsoft-glue from all conda environments. Then I completely rebuilt the conda environment with a new python 3.10 and conda install -c conda-forge pyfstat jupyter.
\nThe return value of the result of \"$ which lalapps_version\" in its current state is nothing, but that of the result of \"$ which lalpulsar_Makefakedata_v5\" is \"~/.conda/envs/pyfstat/bin/lalpulsar_Makefakedata_v5\".
\nIn above situation I got the exact same error as before.
Hi @kami2suukyi ,
\nThanks for your answers.
\n\n\nIn above situation I got the exact same error as before.
\n
Could you paste which command you actually run?
\nCan I ask you to run the following commands?
\n- \n
- Outside and inside the conda environment: \n
echo $PATH \n- \n
- Inside the conda environment: \n
$ python\n>>> import os\n>>> os.environ(\"PATH\")\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nHello @rodrigotenorio, I'm sorry for late reply.
\nThanks very much for all your help and advice.
(1) I recreated clean python 3.10 conda environment, then I registered that environment with jupyter.
\n$ conda create -n pyfstat-py310 python=3.10\n$ conda activate pyfstat-py310\n$ conda install -c conda-forge pyfstat jupyter\n$ ipython kernel install --user --name pyfstat-py310 --display-name pyfstat-py310\n(2) outside conda environment:
\n$ echo $PATH\n/home/USER/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\n(3) inside conda environment:
\n$ python\nPython 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n>>> import os\n>>> os.environ[\"PATH\"]\n'/home/USER/.conda/envs/pyfstat-py310/bin:/home/USER/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Need one of --IFOs, --noiseSFTs or --inFrChannels to determine detectors\n\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Invalid argument\nXLAL Error - main (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:180): Check failed: XLALInitMakefakedata ( &GV, &uvar ) == XLAL_SUCCESS\nXLAL Error - main (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:180): Internal function call failed: Invalid argument\n65280\n>>>\n(4) I ran your notebook. The error occurred in the second cell.
\nimport os\nimport sys\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport pyfstat\n\nfrom scipy import stats\n\n%matplotlib inline\nAnd I got the exact same error as before.
\n22-11-12 22:59:34.406 pyfstat INFO : Running PyFstat version 1.18.1\n22-11-12 22:59:34.725 pyfstat.utils.importing INFO : No $DISPLAY environment variable found, so importing matplotlib.pyplot with non-interactive 'Agg' backend.\n---------------------------------------------------------------------------\nRuntimeError Traceback (most recent call last)\nCell In [2], line 7\n 4 import numpy as np\n 5 import matplotlib.pyplot as plt\n----> 7 import pyfstat\n 9 from scipy import stats\n 11 get_ipython().run_line_magic('matplotlib', 'inline')\n\nFile ~/.conda/envs/pyfstat-py310/lib/python3.10/site-packages/pyfstat/__init__.py:37\n\n...\n\nFile ~/.conda/envs/pyfstat-py310/lib/python3.10/site-packages/pyfstat/utils/runlalsuite.py:35, in get_lal_exec(cmd)\n 33 full_cmd = shutil.which(\"lalpulsar_\" + cmd) or shutil.which(\"lalapps_\" + cmd)\n 34 if full_cmd is None:\n---> 35 raise RuntimeError(\n 36 f\"Could not find either lalpulsar or lalapps version of command {cmd}.\"\n 37 )\n 38 return os.path.basename(full_cmd)\n\nRuntimeError: Could not find either lalpulsar or lalapps version of command Makefakedata_v5.\nHi @kami2suukyi ,
\nThank you very much for being so thorough. I think I know what's going on.
\nIf I'm not mistaken, the following line
\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Need one of --IFOs, --noiseSFTs or --inFrChannels to determine detectors\ntells us that you have Makefakedata installed in your system.
\nJust to be extra sure: Can you run which lalpulsar_Makefakedata_v5 outside your conda environment as well?
\n\nthen I registered that environment with jupyter.
\n
Ok, I think I've never done that myself.
\nCan you try to star the notebook by just running
\n$ jupyter notebook\nin your conda environment?
", "rawMarkdown": "Hi @kami2suukyi ,\n\nThank you very much for being so thorough. I *think* I know what's going on.\n\nIf I'm not mistaken, the following line\n```\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Need one of --IFOs, --noiseSFTs or --inFrChannels to determine detectors\n```\ntells us that you *have* Makefakedata installed in your system.\n\nJust to be extra sure: **Can you run `which lalpulsar_Makefakedata_v5` outside your conda environment** as well?\n\n> then I registered that environment with jupyter.\n\nOk, I think I've *never* done that myself.\n\nCan you try to star the notebook by just running\n```\n$ jupyter notebook\n```\n\nin your conda environment?", "votes": 1 }, { "id": 2027714, "postDate": "2022-11-13T06:09:02.367Z", "content": "Hi @rodrigotenorio, thanks to your advice I finally solved the problem.
\nSince I have started jupyter as a server in my environment, it was difficult to start jupyter locally as you requested, I added \"env\" to \"~/.local/share/jupyter/kernels/pyfstat-py310/kernel.json\" as follows
{\n \"argv\": [\n \"/home/USER/.conda/envs/pyfstat-py310/bin/python3.10\",\n \"-m\",\n \"ipykernel_launcher\",\n \"-f\",\n \"{connection_file}\"\n ],\n \"env\": {\n \"PATH\": \"${HOME}/.conda/envs/pyfstat-py310/bin:${HOME}/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\"\n },\n \"display_name\": \"pyfstat-py310\",\n \"language\": \"python\",\n \"metadata\": {\n \"debugger\": true\n }\n}\nNow I can call the registered environment \"pyfstat-py310\" from jupyter and run the code.
\nThanks again and again for your help.
", "rawMarkdown": "Hi @rodrigotenorio, thanks to your advice I finally solved the problem.\nSince I have started jupyter as a server in my environment, it was difficult to start jupyter locally as you requested, I added \"env\" to \"~/.local/share/jupyter/kernels/pyfstat-py310/kernel.json\" as follows\n```\n{\n \"argv\": [\n \"/home/USER/.conda/envs/pyfstat-py310/bin/python3.10\",\n \"-m\",\n \"ipykernel_launcher\",\n \"-f\",\n \"{connection_file}\"\n ],\n \"env\": {\n \"PATH\": \"${HOME}/.conda/envs/pyfstat-py310/bin:${HOME}/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\"\n },\n \"display_name\": \"pyfstat-py310\",\n \"language\": \"python\",\n \"metadata\": {\n \"debugger\": true\n }\n}\n```\nNow I can call the registered environment \"pyfstat-py310\" from jupyter and run the code.\n\nThanks again and again for your help.", "votes": 1 } ] }, { "id": 2020140, "postDate": "2022-11-07T07:56:08.007Z", "content": "I ran into a error while installing pyfstat in the tutorial notebooks and I am not able to understand what is causing it…
\nI cloned the entire pyfstat in github desktop and from there I accessed the tutorials.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8822382%2F7eb4146842cdb7c0517be136c06266b6%2Ferror1.jpg?generation=1667807727222615&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8822382%2F9d5c7c899fe74b446c8f8fb663455f08%2Ferroe.jpg?generation=1667807750457744&alt=media\")
Hi @bamunparticle ,
\nIt looks like you are trying to install PyFstat in a Windows machine, which is unfortunatelly unsupported at the moment.
\nCan you confirm that this is indeed the case?
\nSolutions may include using a linux vm, Kaggle kernels or Google Collab.
", "rawMarkdown": "Hi @bamunparticle ,\n\nIt looks like you are trying to install PyFstat in a Windows machine, which is unfortunatelly unsupported at the moment.\n\nCan you confirm that this is indeed the case?\n\nSolutions may include using a linux vm, Kaggle kernels or Google Collab.", "votes": 2 } ] }, { "id": 2016404, "postDate": "2022-11-04T01:23:58.957Z", "content": "Hi @rodrigotenorio Please how can I generate a data like this one:
\n\n\nID: 001121a05
\n\n
\n- \n
L1
\n -- SFTs: (360, 4653)
\n -- timestamps: (4653,)- \n
H1
\n -- SFTs: (360, 4612)
\n -- timestamps: (4612,)- \n
Frequency data: (360,)
I mean, which parameters do I have to set to achieve those shapes?
", "rawMarkdown": "Hi @rodrigotenorio Please how can I generate a data like this one:\n\n> ID: 001121a05 \n> \n- L1\n-- SFTs: (360, 4653)\n-- timestamps: (4653,) \n> \n- H1\n-- SFTs: (360, 4612)\n-- timestamps: (4612,) \n> \n- Frequency data: (360,)\n\nI mean, which parameters do I have to set to achieve those shapes?", "replies": [ { "id": 2016896, "postDate": "2022-11-04T10:38:55.233Z", "content": "Hi @felipefonte99 ,
\nWell, there's many ways in which you could achieve that, and basic tools for that are explained in tutorial 1.
\nEssentially, that 360 corresponds to 0.2Hz with Tsft = 1800, as discussed at different points in this competition.
\nThe best way you could get that is by generating a larger band and slicing out the relevant part of it.
As for timestamps, each timestamp will give you one of those 4653 (or 4612) points, so it's a matter of deciding where to start and how big the space between them.
\nNote that those numbers do not specify a starting frequency (the typical values of which you can get from some amazing EDA notebooks).
", "rawMarkdown": "Hi @felipefonte99 ,\n\nWell, there's many ways in which you could achieve that, and basic tools for that are explained in tutorial 1.\n\nEssentially, that 360 corresponds to 0.2Hz with Tsft = 1800, as discussed at different points in this competition.\nThe best way you could get that is by generating a larger band and slicing out the relevant part of it.\n\nAs for timestamps, each timestamp will give you one of those 4653 (or 4612) points, so it's a matter of deciding where to start and how big the space between them.\n\nNote that those numbers do *not* specify a starting frequency (the typical values of which you can get from some amazing EDA notebooks).\n\n", "votes": 3 }, { "id": 2058953, "postDate": "2022-12-08T10:53:29.447Z", "content": "Hi @rodrigotenorio,
\nFor timestamps to be similar like 4653, we have to select appropriate value of duration ?
Hi @neerajanandcoder ,
\nSo, each timestamp corresponds to an SFT lasting for 1800s, meaning you'd need duration = 8375400 (1800 * 4653) to get
\n4653 consecutive timestamps.
You could also create an array of timestamps (start at some GPS time after 2020 and add numbers bigger than 1800 until you fill up 4653) and pass that to Writer. This way, you'll be able to generate non-consecutive timestamps (such as the ones provided in the test set).
Let me know if this helped.
", "rawMarkdown": "Hi @neerajanandcoder ,\n\nSo, each timestamp corresponds to an SFT lasting for 1800s, meaning you'd need `duration = 8375400` (1800 * 4653) to get\n4653 *consecutive* timestamps.\n\nYou could also create an array of timestamps (start at some GPS time after 2020 and add numbers bigger than 1800 until you fill up 4653) and pass that to `Writer`. This way, you'll be able to generate *non-consecutive* timestamps (such as the ones provided in the test set).\n\nLet me know if this helped.", "votes": 1 }, { "id": 2059824, "postDate": "2022-12-09T09:00:44.833Z", "content": "Thanks for the reply!!
", "rawMarkdown": "Thanks for the reply!!\n", "votes": 1 } ] }, { "id": 2002494, "postDate": "2022-10-24T21:06:53.333Z", "content": "@rodrigotenorio
\nRodrigo, can you also elaborate a bit on the meaning of the SFTWindowBeta parameter? The documentation is kind of lacking \"Optional parameter for some windowing functions\" I see it set at 0.01 or 0.001.
\nWhich one do we need? :)
Hi @sakvaua , sure.
\nSome windows accept a \"tuning parameter\" which adjust their shape.
\nIn these examples, we use a Tukey window, which is one of the typical choices made in continuous wave searches. For this specific window, the SFTWindowBeta parameter then tunes how long is the fall-off of this window near the edges: A value of 0 sets a rectangular window, while a value of 1 gives you a Hann window.
We normally set this value to 0.01 or 0.001, and at that level I don't think it would matter that much which one you pick.
", "rawMarkdown": "Hi @sakvaua , sure.\n\nSome windows accept a \"tuning parameter\" which adjust their shape.\n\nIn these examples, we use a [Tukey window](https://en.wikipedia.org/wiki/Window_function#Tukey_window), which is one of the typical choices made in continuous wave searches. For this specific window, the `SFTWindowBeta` parameter then tunes how long is the fall-off of this window near the edges: A value of 0 sets a rectangular window, while a value of 1 gives you a Hann window.\n\nWe normally set this value to 0.01 or 0.001, and at that level I don't think it would matter that much which one you pick.\n", "votes": 5 }, { "id": 2003274, "postDate": "2022-10-25T11:48:35.003Z", "content": "Thanks, Rodrigo
\nNNs are very sensitive to such usually imperceivable differences. For example, switching from CV2 jpeg decoding to PIL can result in ca. 1% of accuracy on imagenet. Amazing!
1 Is the train / test dataset real collected dataset?
\n2 Are the sqrtSX for generating the (train/test) dataset almost fixed parameters?
\nIn the case of actual collected data, are the noise levels intentionally added after data collection equal or zero?
3 What range does h0 have to reproduce the example data (train dataset)?
\n\n\nThe typical amplitudes of the resulting signals are one or two orders of magnitude lower than the amplitude of the detector noise.
\n
Can we assume h0 = sqrtSX * (1 to 0.5) in this case?
\nIf not, does it mean log(h0) = log(sqrtSX) * (1~2) ?
4 I can't understand the following parameters. Is there any related data?
\nF1, Alpha, Delta, cosi, psi, phi
\nI presume these are variables related to astrophysics, but I'd like to understand how they can be adjusted to produce the shape of the signal I want.
For example I would like to know how to create an sft that looks like a cosine function with a shorter period.
", "rawMarkdown": "1 Is the train / test dataset real collected dataset?\n\n2 Are the sqrtSX for generating the (train/test) dataset almost fixed parameters?\nIn the case of actual collected data, are the noise levels intentionally added after data collection equal or zero?\n\n3 What range does h0 have to reproduce the example data (train dataset)?\n> The typical amplitudes of the resulting signals are one or two orders of magnitude lower than the amplitude of the detector noise.\n\nCan we assume `h0 = sqrtSX * (1 to 0.5)` in this case?\nIf not, does it mean `log(h0) = log(sqrtSX) * (1~2)` ?\n\n4 I can't understand the following parameters. Is there any related data?\n```F1, Alpha, Delta, cosi, psi, phi ```\nI presume these are variables related to astrophysics, but I'd like to understand how they can be adjusted to produce the shape of the signal I want.\n\nFor example I would like to know how to create an sft that looks like a cosine function with a shorter period. ", "replies": [ { "id": 1996595, "postDate": "2022-10-20T10:13:01.597Z", "content": "Hi @assign , thanks for taking part of this :)
\n- \n
Yes, part of the data we have provided is taken directly from the Advanced LIGO detectors. In some cases we have included a signal, in some cases we haven't, but noise properties should be essentially those of real data.
\nI'm not sure if I understand this question, but in some cases data is stationary (hence you could use
sqrtSXto describe it) and in some cases, such as real data, it's not, meaning a singlesqrtSXmay not be representative enough. You could actually try yourself using severalsqrtSXfor different periods of time whenever you generate your data. \nParameters like
F0, F1, Alpha, Deltaaffect the frequency evolution of your signal, whereas parameters such ascosi, phi, psiaffect the amplitude of the signal. For the latter, there's this nice visualization which may help in visualizing the angles. \n
Initially, I was baffled by the amount of data/documentation/symbols that seemed confusing to me as a non-expert, but after generating dozens of sample data in PyFstat, I understood to some extent how it affects the graph. I don't know exactly what units and numbers each affects though. It would be nice to be able to see the code it generates, but it's hard to trace because they use system commands.
\nThanks for the reply.
", "rawMarkdown": "Initially, I was baffled by the amount of data/documentation/symbols that seemed confusing to me as a non-expert, but after generating dozens of sample data in PyFstat, I understood to some extent how it affects the graph. I don't know exactly what units and numbers each affects though. It would be nice to be able to see the code it generates, but it's hard to trace because they use system commands.\n\nThanks for the reply." } ] }, { "id": 1994273, "postDate": "2022-10-18T22:57:44.977Z", "content": "Hi @rodrigotenorio !
\nThanks, for such brain-consuming competition.
\nShould we consider that there is signal in the test when the depth sensing is between 10 and 50 Hz**1/2 as stated in the previous notebooks?
\nOr are we being challenged to address deeper depth conditions?
Hi @cristojv ,
\nAll I can say is that there is a broad distribution of SNR / amplitudes / depth in the data.
\nHope that helps 😊.
\n\nIn total there are eight parameters which have all been randomised.
\n
Are they F0, F1, Alpha, Delta, Depth, cosi, psi, phi?
", "rawMarkdown": "> In total there are eight parameters which have all been randomised.\n\nAre they F0, F1, Alpha, Delta, Depth, cosi, psi, phi?", "replies": [ { "id": 1992244, "postDate": "2022-10-17T15:44:37.653Z", "content": "Hi zzy,
\nYou can also find a list of the randomized parameters in the data page.
\nThe relevant parameters are the ones you mentioned, except for Depth: Data-generation codes work in terms of h0, which is a more physical quantity, while Depth is reserved to interpret the results.
Cheers,
", "rawMarkdown": "Hi zzy,\n\nYou can also find a list of the randomized parameters in [the data page](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/data).\n\nThe relevant parameters are the ones you mentioned, except for `Depth`: Data-generation codes work in terms of `h0`, which is a more physical quantity, while `Depth` is reserved to interpret the results.\n\nCheers,\n", "votes": 1 }, { "id": 1992250, "postDate": "2022-10-17T15:47:12.297Z", "content": "Thanks for your reply!
", "rawMarkdown": "Thanks for your reply!" }, { "id": 1996241, "postDate": "2022-10-20T06:33:37.217Z", "content": "\n\n\nYou can also find a list of the randomized parameters in the data page.
\n
I can't understand this statement.
\nDoes that mean that 8 parameters are written on the data page?
\nI'd appreciate it if you could explain clearly
Hi @assign, just to clarify things. Rodrigo is referring to this sentence from the data page:
\n\n\nThe signals are characterised by the location and orientation of the hypothetical astrophysical source as well as two intrinsic parameters: frequency and spin-down. In total there are eight parameters which have all been randomised.
\n
We can break this down a bit, the eight parameters are:
\n- \n
- Two parameters for the location on the sky \n
- Three parameters for the orientation of the source \n
- One parameter that describes the \"strength\" of the signal (see Rodrigo's comment above about
h0) \n - Two intrinsic parameters: frequency and spin-down \n
Hope this helps :)
", "rawMarkdown": "Hi @assign, just to clarify things. Rodrigo is referring to this sentence from the data page: \n\n> The signals are characterised by the location and orientation of the hypothetical astrophysical source as well as two intrinsic parameters: frequency and spin-down. In total there are eight parameters which have all been randomised.\n\nWe can break this down a bit, the eight parameters are:\n* Two parameters for the location on the sky\n* Three parameters for the orientation of the source\n* One parameter that describes the \"strength\" of the signal (see Rodrigo's [comment above](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/347052#1992244) about `h0`)\n* Two intrinsic parameters: frequency and spin-down\n\nHope this helps :)", "votes": 4 }, { "id": 1996600, "postDate": "2022-10-20T10:15:55.453Z", "content": "@michaeljwill Thank you for answer. Now I understand what it means.
\nCould you please check if the parameters used in PyFstat are mapped the same as the following ones?
Two parameters for the location on the sky: Alpha, Delta
\nThree parameters for the orientation of the source: cosi, psi, phi
\nOne parameter that describes the \"strength\" of the signal: h0
\nTwo intrinsic parameters: frequency and spin-down: F0, F1
Thanks again
", "rawMarkdown": "@michaeljwill Thank you for answer. Now I understand what it means.\nCould you please check if the parameters used in PyFstat are mapped the same as the following ones?\n\nTwo parameters for the location on the sky: Alpha, Delta\nThree parameters for the orientation of the source: cosi, psi, phi\nOne parameter that describes the \"strength\" of the signal: h0\nTwo intrinsic parameters: frequency and spin-down: F0, F1 \n\nThanks again", "votes": 3 }, { "id": 1996624, "postDate": "2022-10-20T10:30:19.860Z", "content": "@assign Yes, that mapping is correct :)
", "rawMarkdown": "@assign Yes, that mapping is correct :)", "votes": 1 } ] }, { "id": 1987676, "postDate": "2022-10-14T19:29:46.260Z", "content": "Hey Rodrigo,
\nThanks for the information and tutorials.
I was trying to run the noise generation code.
\nThe first error its giving me is \"No /data/bverma/.pyfstat.conf file found\". This leads to the consiquent errors.
I have checked the pyfstat files and my computer files and I am unable to find the particular config file.
\nI have included the entire error in this thread.
\n\"
\n22-10-14 14:32:14.350 pyfstat.core INFO : Creating Writer object…
\n22-10-14 14:32:14.352 pyfstat.utils.ephemeris INFO : No /data/bverma/.pyfstat.conf file found. Will fall back to lal's automatic path resolution for files [earth00-40-DE405.dat.gz,sun00-40-DE405.dat.gz]. Alternatively, set 'earth_ephem' and 'sun_ephem' class options.
\n22-10-14 14:32:14.354 pyfstat.make_sfts INFO : Generating SFTs with fmin=99.5, Band=1.0
\n22-10-14 14:32:14.355 pyfstat.make_sfts INFO : Got h0=0, not writing an injection .cff file.
\n22-10-14 14:32:14.355 pyfstat.make_sfts INFO : Checking if we can re-use existing SFT data file(s)…
\n22-10-14 14:32:14.356 pyfstat.make_sfts INFO : …no SFT file matching 'PyFstat_example_data/H-240_H1_1800SFT_single_detector_gaussian_noise-1238166018-432000.sft' found. Will create new SFT file(s).
\n22-10-14 14:32:14.356 pyfstat.utils.cli INFO : Now executing: lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"
\n22-10-14 14:32:14.391 pyfstat.utils.cli ERROR : Execution failed: Command 'lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"' returned non-zero exit status 255.
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR : XLAL Error - XLALReadEphemerisFile (LALInitBarycenter.c:482): Failed to find ephemeris-file 'earth00-40-DE405.dat.gz[.gz]'
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR :
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR : XLAL Error - XLALReadEphemerisFile (LALInitBarycenter.c:482): Invalid argument
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR : XLAL Error - XLALInitBarycenter (LALInitBarycenter.c:252): XLALReadEphemerisFile('earth00-40-DE405.dat.gz') failed
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR :
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR : XLAL Error - XLALInitBarycenter (LALInitBarycenter.c:252): Internal function call failed: Invalid argument
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:369): Check failed: (cfg->edat = XLALInitBarycenter ( uvar->ephemEarth, uvar->ephemSun )) != ((void *)0)
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:369): Internal function call failed: Invalid argument
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - main (makefakedata_v5.c:181): Check failed: XLALInitMakefakedata ( &GV, &uvar ) == XLAL_SUCCESS
22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - main (makefakedata_v5.c:181): Internal function call failed: Invalid argument
\nCalledProcessError Traceback (most recent call last)
\nCell In [6], line 18
\n 15 writer = pyfstat.Writer(**writer_kwargs)
\n 17 # Create SFTs
\n---> 18 writer.make_data()
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/make_sfts.py:707, in Writer.make_data(self, verbose)
\n 705 else:
\n 706 logger.info(\"Got h0=0, not writing an injection .cff file.\")
\n--> 707 self.run_makefakedata()
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/make_sfts.py:721, in Writer.run_makefakedata(self)
\n 719 check_ok = self.check_cached_data_okay_to_use(cl_mfd)
\n 720 if check_ok is False:
\n--> 721 utils.run_commandline(cl_mfd)
\n 722 if not np.all([os.path.isfile(f) for f in self.sftfilenames]):
\n 723 raise IOError(
\n 724 f\"It seems we successfully ran {self.mfd},\"
\n 725 f\" but did not get the expected SFT file path(s): {self.sftfilepath}.\"
\n 726 f\" What we have in the output directory '{self.outdir}' is:\"
\n 727 f\" {os.listdir(self.outdir)}\"
\n 728 )
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/utils/cli.py:41, in run_commandline(cl, raise_error, return_output)
\n 37 logger.warning(
\n 38 \"Pipe ('|') found in commandline, errors may not be properly caught!\"
\n 39 )
\n 40 try:
\n---> 41 completed_process = subprocess.run(
\n 42 cl,
\n 43 check=True,
\n 44 shell=True,
\n 45 capture_output=True,
\n 46 text=True,
\n 47 )
\n 48 msg = completed_process.stdout
\n 49 if msg:
File ~/.conda/envs/ContGW/lib/python3.10/subprocess.py:524, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
\n 522 retcode = process.poll()
\n 523 if check and retcode:
\n--> 524 raise CalledProcessError(retcode, process.args,
\n 525 output=stdout, stderr=stderr)
\n 526 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command 'lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"' returned non-zero exit status 255.
\n\"
\nThanks for your help,
\nBhaskar Verma
Hi Bhaskar Verma,
\nThanks for reporting this.
\n\n\nThe first error its giving me is \"No /data/bverma/.pyfstat.conf file found\". This leads to the consiquent errors.
\n
That is sorta expected. The .pyfstat.conf file is something you could create in order to configure PyFstat in a specific way, but you are not required to do so. Rather, PyFstat should already include everything it needs to work.
Can I ask you to report this as an issue so we can follow it up?
\nIt looks like you don't have the required ephemeris files, which should have been pulled together if you were using the latest version of LALSuite. What pip version are you using? Can you paste pip --version and pip list in that issue as well?
Cheers,
", "rawMarkdown": "Hi Bhaskar Verma,\n\nThanks for reporting this.\n\n> The first error its giving me is \"No /data/bverma/.pyfstat.conf file found\". This leads to the consiquent errors.\n\nThat is sorta expected. The `.pyfstat.conf` file is something you *could* create in order to configure PyFstat in a specific way, but you are *not required* to do so. Rather, PyFstat should already include everything it needs to work.\n\nCan I ask you to report this as [an issue](https://github.com/PyFstat/PyFstat/issues/new/choose) so we can follow it up?\n\nIt looks like you don't have the required ephemeris files, which should have been pulled together if you were using the latest version of LALSuite. What pip version are you using? Can you paste `pip --version` and `pip list` in that issue as well?\n\nCheers,\n" } ] }, { "id": 1987054, "postDate": "2022-10-14T13:24:03.933Z", "content": "Hi Rodrigo,
\nThank you for this tutorial.
\nI have a question about the fact that competitors need to generate additional data for training: why do we have to generate our own samples, i.e. why did not you generate all samples by yourself and then upload them as the full training data? Is it because you except each competitor to have its own training dataset and also except that each competitor tries to generate data using personal approach (e.g. based on some parameter estimations using the available training instances)?
\nThanks in advance!
", "rawMarkdown": "Hi Rodrigo,\n\nThank you for this tutorial.\n\nI have a question about the fact that competitors need to generate additional data for training: why do we have to generate our own samples, i.e. why did not you generate all samples by yourself and then upload them as the full training data? Is it because you except each competitor to have its own training dataset and also except that each competitor tries to generate data using personal approach (e.g. based on some parameter estimations using the available training instances)?\n\nThanks in advance!", "replies": [ { "id": 1987365, "postDate": "2022-10-14T17:08:11.790Z", "content": "Hi @cverrier,
\nFirstly, thanks for taking part in the competition :)
\nThere were a few reasons for handling the data this way but the main factors were the following:
\n- \n
- There are limits to how much data we as hosts can upload for a competition, the ~200 GB is already pushing the limit, so we couldn't really upload significantly more. \n
- The data has to be split between training and testing. We felt it was more important to maximise the amount of test data since the scores on the leaderboard will be more reliable the more test data is used. This meant that ultimately we could only provide a limited amount of training data. Given the nature of the problem, we think it's likely that you'll need more training data, so we decided it would be fairer if we provide code for everyone to use. \n
- In the last competition we ran (G2Net Gravitational Wave Detection) one of the main points of feedback we received from the community was that having code to generate your own data would have been helpful. In fact, the first-place team won partly because they were one of the few teams that wrote their own code to generate data. \n
I hope this gives you some insight into our thought process when we were deciding how to provide the data.
", "rawMarkdown": "Hi @cverrier,\n\nFirstly, thanks for taking part in the competition :)\n\nThere were a few reasons for handling the data this way but the main factors were the following:\n\n* There are limits to how much data we as hosts can upload for a competition, the ~200 GB is already pushing the limit, so we couldn't really upload significantly more.\n* The data has to be split between training and testing. We felt it was more important to maximise the amount of test data since the scores on the leaderboard will be more reliable the more test data is used. This meant that ultimately we could only provide a limited amount of training data. Given the nature of the problem, we think it's likely that you'll need more training data, so we decided it would be fairer if we provide code for everyone to use.\n* In the last competition we ran ([G2Net Gravitational Wave Detection](https://www.kaggle.com/c/g2net-gravitational-wave-detection/overview)) one of the main points of feedback we received from the community was that having code to generate your own data would have been helpful. In fact, the first-place team won partly because they were one of the few teams that wrote their own code to generate data.\n\nI hope this gives you some insight into our thought process when we were deciding how to provide the data.", "votes": 4, "replies": [ { "id": 1989235, "postDate": "2022-10-15T19:41:25.683Z", "content": "Hi @michaeljwill,
\nThank you for your reply and detailed explanations.
\n\n\nI hope this gives you some insight into our thought process when we were deciding how to provide the data.
\n
Crystal clear! Thanks again 👍
", "rawMarkdown": "Hi @michaeljwill,\n\nThank you for your reply and detailed explanations.\n\n> I hope this gives you some insight into our thought process when we were deciding how to provide the data.\n\nCrystal clear! Thanks again 👍" } ] } ] }, { "id": 1979785, "postDate": "2022-10-09T17:28:46.987Z", "content": "Hi Rodrigo,
\nI am wondering about the SNR of the real data. What range of SNRs do physicist's actually expect?
\nIs it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?
", "rawMarkdown": "Hi Rodrigo, \n\nI am wondering about the SNR of the real data. What range of SNRs do physicist's actually expect? \n\nIs it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?", "replies": [ { "id": 1980712, "postDate": "2022-10-10T10:35:26.907Z", "content": "\n\nWhat range of SNRs do physicist's actually expect?
\n
0 to arbitrarily large, depending on how good your detector is ;)
\nSince we haven't detected any of these signals yet, we have no idea about how strong they actually are. You can check some of the LIGO papers (look for the CW tag) to get a sense of how sensitive our searches are (i.e. how strong could this signals be while remaining undetected); I wouldn't quote any specific numbers, however, since they are highly dependent on search setup and detector configuration.
\n\nIs it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?
\n
Minor comment on notation: when we talk about SNR, we refer to the SNR of a specific model. That is, given a set of parameters characterizing a signal, I can compute the SNR associated to that specific signal in the data. Data itself doesn't have an \"SNR\" in our language.
\nNow, going to your question: Absolutely. Part of the test set is based on actual data taken by the Advanced LIGO detectors, hence it may contain some instrumental artifacts in there with which you'll have to deal with.
", "rawMarkdown": "> What range of SNRs do physicist's actually expect? \n\n0 to arbitrarily large, depending on how good your detector is ;)\n\nSince we haven't detected any of these signals yet, we have no idea about how strong they actually are. You can check [some of the LIGO papers](https://pnp.ligo.org/ppcomm/Papers.html) (look for the `CW` tag) to get a sense of how sensitive our searches are (i.e. how strong could this signals be while remaining undetected); I wouldn't quote any specific numbers, however, since they are highly dependent on search setup and detector configuration.\n\n> Is it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?\n\nMinor comment on notation: when we talk about SNR, we refer to the SNR of a specific model. That is, given a set of parameters characterizing a signal, I can compute the SNR associated to *that specific signal* in the data. Data itself doesn't have an \"SNR\" in our language.\n\nNow, going to your question: Absolutely. Part of the test set is based on actual data taken by the Advanced LIGO detectors, hence it may contain some instrumental artifacts in there with which you'll have to deal with.\n", "votes": 1 } ] }, { "id": 1979255, "postDate": "2022-10-09T09:00:16.357Z", "content": "Hi.Is there a fixed window length for fourier transform？
", "rawMarkdown": "Hi.Is there a fixed window length for fourier transform？", "replies": [ { "id": 1979930, "postDate": "2022-10-09T20:23:03.160Z", "content": "For this kind of signal the recommended lenght is 1800seconds (30 mins). There might be gaps in the timestamp but the considered length is 1800 seconds. Rodrigo can confirm better. :)
", "rawMarkdown": "For this kind of signal the recommended lenght is 1800seconds (30 mins). There might be gaps in the timestamp but the considered length is 1800 seconds. Rodrigo can confirm better. :)", "votes": 2 }, { "id": 1980396, "postDate": "2022-10-10T06:25:42.070Z", "content": "So each timestamp coressponds to a period ?And why are the values for amplitude so small after the fourier transformation?
", "rawMarkdown": "So each timestamp coressponds to a period ?And why are the values for amplitude so small after the fourier transformation?" }, { "id": 1980689, "postDate": "2022-10-10T10:12:07.303Z", "content": "Hi GodGod3,
\nYes, each timestamp labels a period of time of 1800s (you can check that by noting the frequency resolution of 1/1800 Hz), but timestamps need not to be consecutive (i.e. there may be times at which no data is collected).
", "rawMarkdown": "Hi GodGod3,\n\nYes, each timestamp labels a period of time of 1800s (you can check that by noting the frequency resolution of 1/1800 Hz), but timestamps need not to be consecutive (i.e. there may be times at which no data is collected)." }, { "id": 1981127, "postDate": "2022-10-10T15:51:02.497Z", "content": "THANKS.To clarify,is the sampling frequency also the window for fourier transformation?How are the gaps handled during fourier transformation?
", "rawMarkdown": "THANKS.To clarify,is the sampling frequency also the window for fourier transformation?How are the gaps handled during fourier transformation?" }, { "id": 1984735, "postDate": "2022-10-12T20:36:44.127Z", "content": "Ups, sorry GodGod3, seems like I missed this comment.
\nFourier transforms are taken over continuous segments. For every timestamp, you can assume there is a continuous span of time on which we applied the Fourier transform. Whenever we hit a gap, we stop and wait until we have again enough data to compute another Fourier transform.
", "rawMarkdown": "Ups, sorry GodGod3, seems like I missed this comment.\n\nFourier transforms are taken over *continuous* segments. For every timestamp, you can assume there is a continuous span of time on which we applied the Fourier transform. Whenever we hit a gap, we stop and wait until we have again enough data to compute another Fourier transform.\n\n\n" } ] }, { "id": 1973698, "postDate": "2022-10-05T19:08:58.170Z", "content": "For those who are interested this is the main Writer class that generates the data: https://github.com/PyFstat/PyFstat/blob/b173b3d6e39088fa3f50110033e6c7ec2c1d6f54/pyfstat/make_sfts.py#L21
Hey Rodrigo Tenorio, thank you for providing resources for generating more data. I am unclear about the \"target\" part while generating the data. The kenrel provided to generate the data does not seem to the information about the label (target 1, 0, or -1). Or am I missing something? Thanks in advance.
", "rawMarkdown": "Hey Rodrigo Tenorio, thank you for providing resources for generating more data. I am unclear about the \"target\" part while generating the data. The kenrel provided to generate the data does not seem to the information about the label (target 1, 0, or -1). Or am I missing something? Thanks in advance.", "replies": [ { "id": 1973327, "postDate": "2022-10-05T15:11:19.807Z", "content": "Hi Ayush Thakur,
\nThanks for participating in the challenge.
\n\n\nThe kenrel provided to generate the data does not seem to the information about the label
\n
0/1 labels refer to whether we included a simulated signal to that sample (1) or if it consists only on noise.
\nIn terms of the quantities of the kernel your refer to, a label of 1 would correspond to a signal with an amplitude h0 greater than 0 (i. e. there's a signal), while a label of 0 would correspond to not having a signal (i.e. amplitude h0 = 0).
The same definition can be made in terms of SNR: Label of 1 corresponds to SNR > 0, label 0 corresponds to SNR = 0 (no signal added at all).
", "rawMarkdown": "Hi Ayush Thakur,\n\nThanks for participating in the challenge.\n\n> The kenrel provided to generate the data does not seem to the information about the label\n\n0/1 labels refer to whether we included a simulated signal to that sample (1) or if it consists only on noise.\n\nIn terms of the quantities of the kernel your refer to, a label of 1 would correspond to a signal with an amplitude `h0` greater than 0 (i. e. there's a signal), while a label of 0 would correspond to not having a signal (i.e. amplitude `h0` = 0).\n\nThe same definition can be made in terms of SNR: Label of 1 corresponds to SNR > 0, label 0 corresponds to SNR = 0 (no signal added at all).", "votes": 1 }, { "id": 1973367, "postDate": "2022-10-05T15:32:42.577Z", "content": "Thanks for the clarification. This helps. :)
", "rawMarkdown": "Thanks for the clarification. This helps. :)" } ] }, { "id": 2074247, "postDate": "2022-12-23T22:13:22.910Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2032972, "postDate": "2022-11-17T00:05:06.057Z", "content": "Hi, when I generate new data, I get different shapes of the SFTs. For example I get shape (100, 150) one time and then (123, 150) the other time (so the time is fixed but I get different shapes for the span of amplitudes). Could you please specify if there is a way to generate fixed shape SFTs.
", "rawMarkdown": "Hi, when I generate new data, I get different shapes of the SFTs. For example I get shape (100, 150) one time and then (123, 150) the other time (so the time is fixed but I get different shapes for the span of amplitudes). Could you please specify if there is a way to generate fixed shape SFTs.", "isDeleted": true, "replies": [ { "id": 2033017, "postDate": "2022-11-17T01:44:54.097Z", "content": "The \"height\" is controlled by the frequency band. To get 360 (like the train/test data), you want a band of 0.2 Hz, so you can set: \"Band\": 0.2, or you can make it larger and clip it
", "rawMarkdown": "The \"height\" is controlled by the frequency band. To get 360 (like the train/test data), you want a band of 0.2 Hz, so you can set: \"Band\": 0.2, or you can make it larger and clip it", "votes": 1 } ] }, { "id": 2028083, "postDate": "2022-11-13T14:12:51.630Z", "content": "[EDIT: I just understood that the Delta and Alpha coordinates are in equatorial coordinatees and not in ecliptic coordinates, which answers my question]
\nHello all,
\nIf I understood correctly, due to the relative speed between the detector and the source, there are two main frequency modulations of the signal: one due to the rotation of the earth, which is contained within a frequency band, and one due to the rotation of the earth.
\nThe second frequency modulation should disappear if the source direction is perpendicular to the ecliptic plan? Therefore I thought that by setting Delta = np.pi / 2, I would get a signal with a constant frequency but this is not the case. Such signals still have a yearly modulation of their frequency. Is it because there are other physical phenomenon that takes place? Or because Delta is measure from the equator and not the ecliptic?
\nHere are the parameters I'm using to generate such a signal:
\nparams = {\n \"tstart\": 1238166018,\n \"tref\": 1238166018,\n \"duration\": 2*365 * 24 * 60 * 60, # 2*365 days\n \"detectors\": \"H1,L1\",\n \"Band\": 0.2,\n \"sqrtSX\": 1e-23,\n \"Tsft\": 1800,\n \"SFTWindowType\": \"tukey\",\n \"SFTWindowBeta\": 0.01,\n \"h0\": 1e-23,\n \"F0\": 150.15,\n \"F1\": 0.0,\n \"F2\": 0.0,\n \"Alpha\": 0.0,\n \"Delta\": np.pi / 2.0,\n \"cosi\": 1,\n \"psi\": 0.0,\n \"phi\": 0.0,\n }\nMany thanks for your help!
\nTantto
Hello! Thank you for the competition and the effort you made to set it up! I have few questions on the data, for which I could not find the answer in that thread nor in the description:
\n(i) I'm not sure to understand the meaning of the label -1 ? If I understand it right it only concerns samples from the training dataset and not the test dataset?
\n(ii) Do we know the proportion of data sample in the train/test using simulated noise vs real-noise? As I could not see that information, I'm assuming it's not a public information of that challenge?
\n(iii) Is it at least the same proportion in the train and test dataset?
\n(iv) In the dataset using real-noise, how are we sure that there is no CW source within it? I mean, what differentiate a sample consisting of simulated noise + a very low simulated source VS a sample of real noise with no simulated source added but that could have potentially a real CW sourced that was not detected?
\n(v) A related question: did you set the minimum value of the amplitude of the simulated sources used bigger than what you could detect using your pipeline in order to avoid the situation described in the previous question?
Many thanks for your help! Let me know if my questions are unclear or already answered somewhere.
", "rawMarkdown": "Hello! Thank you for the competition and the effort you made to set it up! I have few questions on the data, for which I could not find the answer in that thread nor in the description:\n\n(i) I'm not sure to understand the meaning of the label -1 ? If I understand it right it only concerns samples from the training dataset and not the test dataset?\n(ii) Do we know the proportion of data sample in the train/test using simulated noise vs real-noise? As I could not see that information, I'm assuming it's not a public information of that challenge?\n(iii) Is it at least the same proportion in the train and test dataset?\n(iv) In the dataset using real-noise, how are we sure that there is no CW source within it? I mean, what differentiate a sample consisting of simulated noise + a very low simulated source VS a sample of real noise with no simulated source added but that could have potentially a real CW sourced that was not detected?\n(v) A related question: did you set the minimum value of the amplitude of the simulated sources used bigger than what you could detect using your pipeline in order to avoid the situation described in the previous question?\n\nMany thanks for your help! Let me know if my questions are unclear or already answered somewhere.\n\n\n", "votes": 1, "isDeleted": true, "replies": [ { "id": 2014526, "postDate": "2022-11-02T16:12:54.150Z", "content": "Hi @tantto ,
\n(i) Well, we are not sure either :) Due to their exceptional features, we wanted to get some input from the Kaggle community: maybe it's just noise and we failed to understand it, maybe it's a weird signal we don't know how to interpret. In any case, any shared knowledge on it will be very welcome.
\n(ii) The only public information is that there's both classes of noise in the test set.
\n(iii) The training set is a small sample we provided so that Kaggler (such as yourself :) ) would get a sense of how this competition would play out. I'm afraid we did not include any real data, since we wanted to put as much of it as possible in the test set.
\n(iv) We are not. It could well be the case that a real CW signal is buried deep down into the real noise. Of course, for that to be the case, such a signal would have escaped every single one of the LVK searches we have conducted so far.
\nI hope this clarifies your questions.
\nCheers,
", "rawMarkdown": "Hi @tantto ,\n\n(i) Well, we are not sure either :) Due to their exceptional features, we wanted to get some input from the Kaggle community: maybe it's just noise and we failed to understand it, maybe it's a weird signal we don't know how to interpret. In any case, any shared knowledge on it will be very welcome.\n\n(ii) The only public information is that there's both classes of noise in the test set.\n\n(iii) The training set is a small sample we provided so that Kaggler (such as yourself :) ) would get a sense of how this competition would play out. I'm afraid we did not include any real data, since we wanted to put as much of it as possible in the test set.\n\n(iv) We are not. It could well be the case that a *real* CW signal is buried deep down into the real noise. Of course, for that to be the case, such a signal would have escaped every single one of the LVK searches we have conducted so far.\n\nI hope this clarifies your questions.\n\n\nCheers,", "votes": 1 }, { "id": 2014710, "postDate": "2022-11-02T19:15:29.143Z", "content": "Thanks for the answers!
", "rawMarkdown": "Thanks for the answers!", "isDeleted": true }, { "id": 2061297, "postDate": "2022-12-10T23:38:43.060Z", "content": "What is this LVK search?
", "rawMarkdown": "What is this LVK search?" } ] } ], "comments": [ { "id": 2019429, "author_name": "Truth Seeker", "author_url": "", "post_date": "2022-11-06T15:30:30.460000", "content": "Can someone please explain why data generation is needed for this competition? I've read through the description and all effort is on \"how\" data can be generated, not \"why\".
\nTraditionally, in a supervised learning task, there're a labelled training set and a test set, and you train a model that hopefully predicts well on the test set. Where and why is there a need for data generation? If the reason is that the training set is too small and data collection is too costly/impossible, and if data generation makes sense for this problem, why can't the host just generate a whole lot more data for us, so that we can treat this problem as just a \"traditional supervised learning task\" described above?
", "votes": 10, "replies": [ { "id": 2020184, "author_name": "Michael J. Williams", "author_url": "", "post_date": "2022-11-07T08:48:25.077000", "content": "Hi @revealer, I've explained our motivation in this answer but, in short, whilst we could generate more data there are limits too how much data we as hosts can upload for a competition. The 200 GB we provided is already pushing that limit. So, to make things fairer for everyone we decided to provide the code to generate more data.
", "votes": 1, "replies": [] }, { "id": 2020423, "author_name": "Cristo JV", "author_url": "", "post_date": "2022-11-07T13:11:53.283000", "content": "That's is fine. But why you don't release at least the parameters used for data generation? Or at least some limits. I agree with you that there is real test data, but how can we extract, for example, h0 values considered for a signal to be labeled as 1?
", "votes": 6, "replies": [] }, { "id": 2055039, "author_name": "PiotrKlinke", "author_url": "", "post_date": "2022-12-04T17:22:47.227000", "content": "@michaeljwill pls correct me if I'm wrong but I would think even without the labels the generated data could be useful. For example, the generated data with noise and signal in separate files can be used in pretraining your model or training a denoising model that clears the signal from unwantwd noise.
", "votes": 0, "replies": [] }, { "id": 2055666, "author_name": "Michael J. Williams", "author_url": "", "post_date": "2022-12-05T09:54:34.277000", "content": "@piotrklinke, I'm not sure I entirely follow what you're asking, but I agree that extra data, even without labels, could be useful of the competition. But obviously it's up to you, the participants, to work out how it could be used.
", "votes": 0, "replies": [] } ] }, { "id": 1997129, "author_name": "Josef Slavicek", "author_url": "", "post_date": "2022-10-20T16:53:20.263000", "content": "Hello, thanks for exciting competition.
\nI would like to ask: what are meaningful values of F1 (spindown)? Are the data supposed to simulate isolated neutron stars (not binary systems)?
\nIn this notebook, positive values from range 1.0e-12 … 1.0e-8 are used. So, in fact, the frequency is growing with time.
\nIn this tutorial, negative value of -1.0e-9 is used.
\nIn both cases, the magnitude of spindown is surprisingly high for me. Maybe I'm wrong, but intuitively I would expect the rotation of neutron star to be much more stable.
Hi @josefslavicek ,
\n\n\npositive values from range 1.0e-12 … 1.0e-8 are used. So, in fact, the frequency is growing with time.
\n
Ups, that was an oversight on our side. They should have been negative values (since spin-down is related to a loss of energy), but some effects may produce positive spin-down values as well 😉.
\n\n\nIn both cases, the magnitude of spindown is surprisingly high for me. Maybe I'm wrong, but intuitively I would expect the rotation of neutron star to be much more stable.
\n
It depends on whether one is thinking about pulsars, actual neutron stars (NS) or continuous-wave searches.
\nAs you say, current electromagnetic observations of pulsars (neutron stars that we can actually see) give spin down values way lower than what we generated in these notebooks. When it comes to all-sky searches, however, we are looking for a (potentially) different population of NS, namely those that we cannot see.
Leaving aside any discussion on actual NS physics, an argument to go to higher (\"more negative\") spin-down values is related to the astrophysical reach of a search: roughly, the more a star spins down, the bigger the allowed ellipticity (which is proportional to the GW amplitude) hence the further away it can be while still being detectable by our instruments.
", "votes": 5, "replies": [] }, { "id": 1997233, "author_name": "Josef Slavicek", "author_url": "", "post_date": "2022-10-20T17:44:31.153000", "content": "Nice and clear explanation, thanks!
", "votes": 0, "replies": [] }, { "id": 1997599, "author_name": "hosuke", "author_url": "", "post_date": "2022-10-21T03:04:42.443000", "content": "Hi @rodrigotenorio, thanks for the interesting competition and for your kind responses on this page.
\n\n\nThey should have been negative values (since spin-down is related to a loss of energy)
\n
This makes perfect sense. In addition, as we are working on this competition, we would be grateful if you could clarify whether the artificial signals in the provided train/test data were generated using only positive F1, only negative, or both.
\nThanks in advance!
Hi, Rodrigo
\nThanks, for such a powerful brain-stimulant competition. Are the signals we are looking for located in the center of the frequency band?
Hi DennisSakva,
\nThanks for taking part of this competition.
\n\n\nAre the signals we are looking for located in the center of the frequency band?
\n
Not necessarily. You can assume that samples labeled with a 1 fully contain a signal within the 0.2 Hz band.
\nNow, that signal may be in the center, it may be in the upper half, it may even cross the from top left corner to the bottom right corner, you name it.
That's really helpful. Time to regenerate signals :)
", "votes": 0, "replies": [] }, { "id": 1987374, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-14T17:13:44.313000", "content": "So… How do I generate SFTs with signals not centered around [F0+Band/2,F0-Band/2]. The make_sfts doesn't seem to support the non-centered signals.
", "votes": 0, "replies": [] }, { "id": 1987496, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-14T18:05:15.017000", "content": "That's actually good question, and I don't think I covered it properly in the available tutorials. Let's see if the following works for you and then after discussing with my colleagues I'll update an example to the page.
\nThe Writer class is thought to generate CW signals with a broad-enough frequency band so that our analysis codes down the line are able to estimate noise properties from the data (for example, it makes sure there are enough extra bins to compute running medians around the band of interest). I though one could bypass those extra bins easily but there are no options to do so in PyFstat.
What I would recommend is to generate broad-band SFTs and the slice out the frequency band of interest once they are read as a NumPy array. This can be done by 1) running Writer to generate noise-only SFTs (so that Band and F0 can be used to specify a frequency band and 2) running Writer using the previously generated SFTs as noiseSFTs without specifying
For example, suppose I want a signal within the [150., 150.2]Hz band.
\nFirst, I'd create a set of noise SFTs covering that band
\nimport pyfstat\nimport numpy as np\n\n# Generate SFTs noise-only SFTs covering the band of interest\nnoise_kwargs = {\n \"tstart\": 1238166018,\n \"duration\": 4 * 30 * 86400,\n \"sqrtSX\": 1e-23,\n \"detectors\": \"H1,L1\",\n \"Tsft\": 1800,\n \"F0\": 150.1, # No signals: [F0 - Band/2, F0 + Band/2]\n \"Band\": 0.5, \n \"SFTWindowType\": \"tukey\",\n \"SFTWindowBeta\": 0.001,\n}\n\nnoise_writer = pyfstat.Writer(label=\"custom_band_noise\", **noise_kwargs)\nnoise_writer.make_data()\nThen, I'd inject the signal
\n# Now inject a signal in there. Make sure `noiseSFTs` are broad enough\n# for that signal to fit (including a few extra frequency bins!).\nsignal_kwargs = {\n \"noiseSFTs\": noise_writer.sftfilepath,\n \"F0\": 150.15,\n \"F1\": 1e-8,\n \"Alpha\": 0.3,\n \"Delta\": 0,\n \"h0\": 1e-23/10,\n \"cosi\": 1,\n \"psi\": 0.2,\n \"phi\": 0.\n }\nfor key in [\"SFTWindowType\", \"SFTWindowBeta\"]:\n signal_kwargs[key] = noise_kwargs[key]\n\nsignal_writer = pyfstat.Writer(label=\"custom_band_signal\", **signal_kwargs)\nsignal_writer.make_data()\nFinally, I read as a Numpy array and slice out the relevant frequency band:
\n# Slice out the band of interest\nfreqs, times, sft_data = pyfstat.utils.get_sft_as_arrays(signal_writer.sftfilepath)\n\nfirst_index = np.argmin(np.abs(freqs - 150.))\nlast_index = np.argmin(np.abs(freqs - 150.2))\n\nfreqs = freqs[first_index:last_index+1]\namplitudes = {key: val[first_index:last_index + 1, :]\n for key, val in sft_data.items()}\n\nprint(\"*\" * 20)\nprint(\"*\" * 20)\nprint(f\"These should be 150. (got {freqs[0]}) \"\n f\"and 150.2 (got {freqs[-1]}).\")\nLet me know if this is useful and I'll add a version of it into the provided notebook.
", "votes": 9, "replies": [] }, { "id": 1987710, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-14T19:55:53.060000", "content": "Thanks! Makes perfect sense. I was thinking along these lines but wasn't sure if it makes physical sense in terms of how the detectors work, data is collected, processed etc.
", "votes": 1, "replies": [] }, { "id": 1988269, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-15T07:40:26.913000", "content": "\n\nbut wasn't sure if it makes physical sense in terms of how the detectors work, data is collected, processed etc.
\n
Yes, it makes sense in the specific kind of search we are doing here. In fact, it's actually quite similar to what we (LIGO scientists) would do.
\nI've updated the tutorial kernel with this new piece of code, so it's in a more central place (should be v5 if I didn't mess up versioning too much).
\nhttps://www.kaggle.com/code/rodrigotenorio/generating-continuous-gravitational-wave-signals
\nCheers,
", "votes": 3, "replies": [] }, { "id": 1993568, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-18T13:37:03.933000", "content": "@rodrigotenorio
\nRodrigo, can you disclose what eight signal parameters are randomized? I'm thinking in terms of these parameters:
- \n
- Reference time (includes start time and some random gaps) \n
- Frequency of a signal F0 \n
- Average amplitude of a signal h0 (should be equal for both detectors, right? Even if sqrtSX for each detector is different) \n
- Linear spindown F1 \n
- cosi, psi, phi \n
- Noise level for each detector sqrtSX (independent, I assume) \n
Am I correct? Anything else to consider?
", "votes": 0, "replies": [] }, { "id": 1993858, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-18T16:21:03.593000", "content": "Hi Dennis,
\nSure. Signal parameters were clatified in this reply.
\nGoing to your questions:
\n\n\n\n 1. \n
\n
tref is just the reference time at which F0 is measured. We usually fix it at a convenient value (begining of the run, mid time of the run, reference time of electromagnetic observations), so you could pick whichever value you prefer and keep it like that.
\n\n\n\n 3. \n
\n
Yes, h0 is a property of the signal. As you point out, both detectors would see the same signal. The fact that they have a different sqrtSX simply means one detector will see it more clearly than the other.
\nThe rest of your assumptions are sound, so keep up with the good work !
", "votes": 2, "replies": [] }, { "id": 2035468, "author_name": "", "author_url": "", "post_date": "2022-11-18T22:22:18.007000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2064527, "author_name": "codingo", "author_url": "", "post_date": "2022-12-13T20:51:09.430000", "content": "\nI am trying to follow your code but there is something error.
\n- \n
- Environment: Jupyter Notebook -> Kaggle \n
- Run followed code: 'pip install pyfstat jupyter' \n
- But still error like: 'No module named 'pyfstat.utils' \n
How can I use your code in Kaggle Notebook?
\n(Additional Info: after i run 'pip install pyfstat jupyter')
\nSuccessfully installed bashplotlib-0.6.5 corner-2.2.1 lalsuite-7.5 ligo-segments-1.4.0 lscsoft-glue-3.0.1 peakutils-1.3.4 ptemcee-1.0.0 pyRXP-3.0.1 pyfstat-1.16.0 versioneer-0.28
\nWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">
\nNote: you may need to restart the kernel to use updated packages.
Solved!
\nI just read your notebook.
\nKaggle notebooks run on Python 3.7, which was dropped by PyFstat a few relases back.
\nPlease, use the following command to install PyFstat on a Kaggle notebook.
\nThis will install an up-to-date version of PyFstat with Python 3.7 support.
\nDo use the latest version of PyFstat if you use your own Python >= 3.8 installation.
\n!pip install git+https://github.com/PyFstat/PyFstat@python37
", "votes": 2, "replies": [] } ] }, { "id": 2027613, "author_name": "Jiawei Zhang", "author_url": "", "post_date": "2022-11-13T02:29:43.240000", "content": "Hello, thanks for exciting competition.
\nI was trying to use PyFstat to generate some data. However, I encountered an error that appeared erratically.
\nIt appears to occur in the signal injection.
Error: injection signal 0:'./band_signal.cff:TSO' needs frequency band [491.947874, 492.109913]Hz, injecting into [491.90000, 492.10000]Hz
\nI'm confused because I've been setting F0 from np.random.randint(50, 500). where did x.947874 and x.109913 come from?
\nI suspect that some underlying IO processes are conflicting when using loops to generate data. It's very frustrating.😣
", "votes": 1, "replies": [ { "id": 2027749, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-11-13T07:04:47.450000", "content": "It means that the signal doesn't fit into the 0.2 band you've provided. It drifts outside of it with this particular set of assumptions.
", "votes": 3, "replies": [] }, { "id": 2027807, "author_name": "Jiawei Zhang", "author_url": "", "post_date": "2022-11-13T07:58:35.030000", "content": "Thank you! And yes, after I change the value of band to 0.5, the error does not occur.
", "votes": 2, "replies": [] } ] }, { "id": 1976344, "author_name": "Quentin R", "author_url": "", "post_date": "2022-10-07T09:44:51.187000", "content": "Hi Rodrigo,
\nThanks for the information and the tutorials.
\nI am having troubles to read the .sft files written at the end of tutorial 1. I understood that I need to create an instance of Writer and put the .sft file in the \"noiseSFTs\" argument. I did this, I also added \"SFTWindowType\": \"tukey\", which seems to be a mandatory argument according to lalpulsar_Makefakedata_v5 --help
\nMy code looks like this
\nimport os\nimport h5py\nimport numpy as np\nimport pandas as pd\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\n\nimport pyfstat\nfrom pyfstat.utils import get_sft_as_arrays\n\nwriter_kwargs = {\n \"noiseSFTs\": \"PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft\",\n \"SFTWindowType\": \"tukey\"\n }\n\nwriter = pyfstat.Writer(**writer_kwargs)\nwhen I try to run it I get a I/O error
\nERROR: Failed to open matched file 'PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft'\n\nXLAL Error - XLALSFTdataFind (SFTfileIO.c:318): I/O error\nTraceback (most recent call last):\n File \"/home/quentin/Documents/MLCompetitions/Kaggle/G2Net/src/train_Conv2D_synthetic_data.py\", line 16, in <module>\n writer = pyfstat.Writer(**writer_kwargs)\n File \"/home/quentin/.local/lib/python3.10/site-packages/pyfstat/utils/importing.py\", line 22, in wrapper\n func(self, *args, **kargs)\n File \"/home/quentin/.local/lib/python3.10/site-packages/pyfstat/make_sfts.py\", line 195, in __init__\n self._basic_setup()\n File \"/home/quentin/.local/lib/python3.10/site-packages/pyfstat/make_sfts.py\", line 438, in _basic_setup\n self._get_setup_from_noiseSFTs()\n File \"/home/quentin/.local/lib/python3.10/site-packages/pyfstat/make_sfts.py\", line 264, in _get_setup_from_noiseSFTs\n lalpulsar.SFTdataFind(self.noiseSFTs, SFTConstraint)\nRuntimeError: I/O error\n[Finished in 0.734s]\nIf I run lalpuslar directly in a terminal it doesn't give any error, which makes me think my arguments are fine
\nlalpulsar_Makefakedata_v5 --noiseSFTs PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft --SFTWindowType=tukey
Thanks for your help,
\nQuentin
Hi Quentin
\nThanks for giving a try to PyFstat.
\n\n\nI understood that I need to create an instance of Writer and put the .sft file in the \"noiseSFTs\" argument
\n
Not quite. As explained in the tutorial, the Writer class is used to generate more data.
Once the data is generated as SFT files, you should use the get_sft_as_arrays function (from pyfstat.utils) to read the SFTs you just created into Numpy arrays (see last line of the first cell of Tutorial 1).
In your case, if your SFTs are in PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft, you could read them as
from pyfstat.utils import get_sft_as_array\n\nfrequency, timestamps, sfts = get_sft_as_array(\"PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft\")\nAs a side note, the path where the SFTs are created is stored in the sftfilepath attribute of Writer
\n(mind that it gets overwritten every time make_data is called!).
\n\nI also added \"SFTWindowType\": \"tukey\", which seems to be a mandatory argument according to lalpulsar_Makefakedata_v5 --help
\n
noiseSFTs is used to add noise or a signal into a specific file of SFTs. In that case, one needs to know which window
\nfunction was used, as failing to do so may significantly bias the SNR of a signal. You don't need this to read your data, but
\nit's worth to keep it in mind what window you are using whenever you generate more.
\n\nIf I run lalpuslar directly in a terminal it doesn't give any error, which makes me think my arguments are fine
\n
If you give noiseSFTs to MFD it will do nothing with them and terminate without producing any output.
\nGiven your input arguments, however, that should have thrown an error, as specifying tukey requires specifying
\na corresponding beta parameter, so… thanks for finding that!
Ok, that's much clearer now, thanks for the detailed reply!
", "votes": 1, "replies": [] } ] }, { "id": 1972036, "author_name": "Marília Prata", "author_url": "", "post_date": "2022-10-04T23:24:32.757000", "content": "Hi R. Tenorio,
\nI was trying to work with PyFstat. However, I got that module error and couldn't go on.
\nModuleNotFoundError: No module named 'tutorial_utils'
\nI've installed PyFstat
\n!pip install pyfstat
\n!pip install git+https://github.com/PyFstat/PyFstat@python37
\nAnd even !pip install pyfstat jupyter
\nThough the: No module named 'tutorial_utils' persists. Any tip?
\nThanks in advance,
\nMarília Prata
Hi Marília,
\nIt sounds like you're running through the tutorials. Both tutorials make use of a local file called tutorial_utils.py which can be found in the same directory as the tutorial notebooks.
The error you're getting would imply that that file (tutorial_utils.py) isn't in the same directory as the notebooks. So you'll either need to copy that file to the directory you're running the notebooks in or edit your copies of tutorials so that they don't import it and instead define the functions it provides in the notebook.
Hope this helps,
\nMichael
I didn't get this e-mail. Thank you for answering it Michael.
", "votes": 0, "replies": [] }, { "id": 1990527, "author_name": "George Chirita", "author_url": "", "post_date": "2022-10-16T15:42:25.997000", "content": "here: https://www.kaggle.com/code/crischir/pyfstat-tutorial-adapted-to-kaggle tutorial adapted to kaggle: ai utils and tutorial_utils.py as functions and sanserif deleted from matlib plots :)
", "votes": 0, "replies": [] }, { "id": 1991637, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-17T09:59:18.130000", "content": "Hi George Chirita,
\nAwesome, I'm sure this will be very useful to newcomers!
\nSince this is largely a verbatim copy of the PyFstat tutorials, would you mind including that link at the top of your kernel? Something like the following would suffice
\nThe contents of this notebook were adapted from [the PyFstat tutorials](https://github.com/PyFstat/PyFstat/tree/master/examples/tutorials).\nThank you very much.
\n\n\nsanserif deleted from matlib plots
\n
Before the challenge I spent some time making sure there were no problems at plotting time (sometimes LaTeX was requested, pointlessly complicating the installation procedure). Did you have any specific issue with the current plotting routines or is this just a personal stylistic choice? If the former, may I ask you to fill up an issue so we can have a look at it?)
", "votes": 2, "replies": [] }, { "id": 1994193, "author_name": "George Chirita", "author_url": "", "post_date": "2022-10-18T20:39:49.280000", "content": "I've put the acknowledgement in two places (also the title is obvious) . Please tell me if it is ok. I do not need credit for this notebook, it was adpted for comunity, and for my presonal gain :) learning.
\nsans serif trigger latex error in kaggle the (actual version of the container), It is documented here:
\nhttps://stackoverflow.com/questions/11354149/python-unable-to-render-tex-in-matplotlib
\nit is an issue of linux version i think :) there are some packages to be installed and this is complicated for a begginer user in kaggle.
There is a difference in the shape of data between the competition and tutorial?
\n\"Tsft\": 1800, # Fourier transform time duration in tutorial (shape in competition is 360. ) is data trimmed?
Cool, that's grand, thank you. I'm sure this is going to be useful to whoever comes next into this challenge.
\n\n\nsans serif trigger latex error in kaggle the (actual version of the container), It is documented here:
\n
\n https://stackoverflow.com/questions/11354149/python-unable-to-render-tex-in-matplotlib
That is a 10y/o thread and I don't seem to see where in your kernel you solve that problem, but fair if it works for you.
\nFor the record, I don't think I ran into that when I last checked the kernel.
\n\nThere is a difference in the shape of data between the competition and tutorial?
\n
\n \"Tsft\": 1800, # Fourier transform time duration in tutorial (shape in competition is 360. ) is data trimmed?
So, Tsft refers to the duration over which a Fourier transform was taken (i.e. each timestamp contains 1800s of data on which a Fourier transform was taken). The 360 is the number of bins in the frequency axis (i.e. how many frequencies does your Fourier transform contain). Since frequency resolution is 1/1800, what you get is that your data spans 360 * ( 1/1800) = 0.2 Hz, as you can see in the plots.
Thank you for your great work.
\nYou have right, in the verbatim copy of the PyFstat tutorials in work great. In my personal notebook it was not, probably a conflicted library, it is not an issue anyway.
\nthanks for clarify the 0.2Hz
\n\"Band\": 0.2, # Frequency band-width around F0 [Hz]
@sakvaua @ayuraj the generated data will surely contain the GW ? can we assign the taret for these synthetic data as 1
", "votes": 0, "replies": [] }, { "id": 3103228, "author_name": "Arn0dian", "author_url": "", "post_date": "2025-01-23T07:17:41.833000", "content": "Thank you for the competition and the effort you made to set it up!
", "votes": 0, "replies": [] }, { "id": 2026697, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-11-12T09:40:25.003000", "content": "Hi, @rodrigotenorio
\nI'm trying to inject a simulated line-like detector artifact into a pre-generated SFT (passed with noiseSFTSs parameter) and apparently LineWriter cuts my frequency band to a minimum band that fits this line. For example:
\nSFTS minimum frequency BEFORE the injection 78.87 maximum 79.67
\nLineWriter kwargs:
\n{
\n'noiseSFTs': '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_H1_band_signal-1238170665-10367456.sft',
\n'F0': 79.22547569407091,
\n'h0': 9.241657937942324e-24,
\n'phi': 1.8502800869842844,
\n'SFTWindowType': 'tukey',
\n'SFTWindowBeta': 0.001,
\n'outdir': '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/'}
Log
\n22-11-12 11:21:45.435 pyfstat.core INFO : Creating LineWriter object…
\n22-11-12 11:21:45.436 pyfstat.utils.ephemeris INFO : No /home/sakvaua/.pyfstat.conf file found. Will fall back to lal's automatic path resolution for files [earth00-40-DE405.dat.gz,sun00-40-DE405.dat.gz]. Alternatively, set 'earth_ephem' and 'sun_ephem' class options.
\n22-11-12 11:21:45.453 pyfstat.make_sfts WARNING : noiseSFTs is not None: Inferring tstart, duration, Tsft. Input tstart and duration will be treated as SFT constraints using lalpulsar.SFTConstraints; Tsft will be checked for internal consistency accross input SFTs.
\n22-11-12 11:21:45.460 pyfstat.make_sfts INFO : SFT Constraints: [minStartTime:None, maxStartTime:None]
\n22-11-12 11:21:45.608 pyfstat.make_sfts WARNING : Injection of line artifacts only uses the following parameters:
\n['F0', 'phi', 'h0'].
\nAny other parameter will be purged from this class now
\n22-11-12 11:21:45.609 pyfstat.make_sfts INFO : Purging input parameters that are not meaningful for LineWriter: ['refTime', 'f1dot', 'psi', 'transientWindowType', 'f2dot']
\n22-11-12 11:21:45.610 pyfstat.make_sfts INFO : Estimating required SFT frequency range from properties of signal to inject plus 59 extra bins either side (corresponding to default F-statistic settings).
\n22-11-12 11:21:45.664 pyfstat.make_sfts INFO : Generating SFTs with fmin=79.18430415750552, Band=0.08234307313076569
\n22-11-12 11:21:45.665 pyfstat.make_sfts INFO : Checking if we can re-use injection config file…
\n22-11-12 11:21:45.666 pyfstat.make_sfts INFO : …OK: config file /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff already exists.
\n22-11-12 11:21:45.668 pyfstat.make_sfts INFO : …file contents unmatched, updating /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff.
\n22-11-12 11:21:45.668 pyfstat.make_sfts INFO : Writing config file: /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/PyFstat.cff
\n22-11-12 11:21:45.670 pyfstat.make_sfts INFO : Checking if we can re-use existing SFT data file(s)…
\n22-11-12 11:21:45.671 pyfstat.make_sfts INFO : …no SFT file matching '/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft' found. Will create new SFT file(s).
\n22-11-12 11:21:45.672 pyfstat.utils.cli INFO : Now executing: lalpulsar_Makefakedata_v4 --lineFeature=TRUE --outSingleSFT=TRUE --outSFTbname=\"/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft\" --IFO=\"H1\" --noiseSFTs=\"/media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_H1_band_signal-1238170665-10367456.sft\" --window=\"tukey\" --tukeyBeta=0.001 --startTime=1238170665 --duration=10367456 --fmin=79.18430415750552 --Band=0.08234307313076569 --Tsft=1800 --h0=9.241657937942324e-24 --Freq=79.22547569407091 --phi0=1.850280086984284 --cosi=0 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"
\n22-11-12 11:21:48.905 pyfstat.utils.cli INFO :
\n22-11-12 11:21:48.908 pyfstat.utils.cli INFO : WARNING: for SFT-creation we had to adjust (fmin,Band) to fmin_eff=79.1838888888889 and Band_eff=0.0827777777777778
\n22-11-12 11:21:48.909 pyfstat.utils.cli INFO :
\n22-11-12 11:21:48.911 pyfstat.make_sfts INFO : Successfully wrote SFTs to: /media/sakvaua/Samsung960/LIGO/g2net-detecting-continuous-gravitational-waves/train/synthetic/tmp/tmp1/H-4497_H1_1800SFT_PyFstat-1238170665-10367456.sft
\n22-11-12 11:21:48.912 pyfstat.make_sfts INFO : Now validating each SFT file…
And AFTER the LineWriter the minimum frequency is 79.18 and the maximum is 79.26 (was 78.87 - 79.67 before )
\nWhy does it have to adjust the fmin and Band?
\nThanks!
Hi @sakvaua ,
\nThis is another effect of not having direct access to fmin from Writer and derived classes, together with the fact that you did not specify a Band.
Since you didn't specify Band, MFDv4 assumed it so that you could run a fully-targeted F-statistic search.
\nIf you specify Band, on the other hand, this line applies and you'll get the same behavior as whenever using Writer.
Let me know if this helped.
\nAs a second option, you could as well generate several instances of \"noisy data\" with the same frequency range and then add them together at numpy-array level, as I said in this answer.
", "votes": 1, "replies": [] }, { "id": 2030108, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-11-15T07:46:33.407000", "content": "Hi, Rodrigo. Thank you so very much for your time and answers.
\nWell, the noise stft I'm passing has all the information needed (like frequencies and timestamps) so I'm not sure why this writer needs to have the band specified. I currently use a workaround by providing Band, but the problem is that it is centered around F0 and if F0 in my STFT and F0 in my LineWriter are different - then the generated frequency bands for Line Stfts and my Noise STFTs will also be different. Thus making simple summation impossible. So I have to pick LineWriter F0 from the exact frequencies of the noise SFTS I generated.
Hi @sakvaua ,
\nApologies for the late replay.
\nI am afraid your solution is basically the way of doing this at the moment. IT is far from being perfect, but roughly it's equivalent to the way we sometimes do some of our analyses. (The main difference is that we tend to use LALSuite structures rather than numpy arrays, meaning codes get a bit messier, but that's roughly it).
\nI do agree with you in that we should really rethink what's going on with this class in PyFstat.
", "votes": 1, "replies": [] } ] }, { "id": 2025328, "author_name": "mn", "author_url": "", "post_date": "2022-11-11T06:44:49.913000", "content": "Thanks for such a wonderful project! What books should I read to make sense of data？
", "votes": 0, "replies": [] }, { "id": 2022761, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-11-09T08:50:00.787000", "content": "@rodrigotenorio
\nThank you for the interesting competition!
\nDoes test data is from the same gravitational-wave interferometers as training ones (LIGO Hanford & LIGO Livingston)?
Hi @hanejiyuto ,
\nYes, we only use the Advanced LIGO interferometers (Hanford and Livingstone) in this competition 👍.
", "votes": 1, "replies": [] }, { "id": 2037038, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-11-20T11:49:06.043000", "content": "Thanks for your reply!
", "votes": 0, "replies": [] } ] }, { "id": 2020602, "author_name": "HIROFUMI OHTA", "author_url": "", "post_date": "2022-11-07T16:20:55.667000", "content": "Hello @rodrigotenorio ! Thanks for the informative tutorial and sample code.
\nI used pyfstat.yml from the link below to build my environment in my conda environment.
\n https://github.com/PyFstat/PyFstat/wiki/conda-environments
However, I am unable to import pyfstat by any means due to the following error.
\nSorry for the long post. Could you please give me some good advice?
\n
RuntimeError Traceback (most recent call last)
\nCell In [2], line 7
\n 4 import numpy as np
\n 5 import matplotlib.pyplot as plt
\n----> 7 import pyfstat
\n 9 from scipy import stats
\n 11 get_ipython().run_line_magic('matplotlib', 'inline')
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/init.py:37
\n 32 from .gridcorner import gridcorner
\n 33 from .injection_parameters import (
\n 34 AllSkyInjectionParametersGenerator,
\n 35 InjectionParametersGenerator,
\n 36 )
\n---> 37 from .make_sfts import (
\n 38 BinaryModulatedWriter,
\n 39 FrequencyAmplitudeModulatedArtifactWriter,
\n 40 FrequencyModulatedArtifactWriter,
\n 41 GlitchWriter,
\n 42 LineWriter,
\n 43 Writer,
\n 44 )
\n 45 from .mcmc_based_searches import (
\n 46 MCMCFollowUpSearch,
\n 47 MCMCGlitchSearch,
\n (…)
\n 50 MCMCTransientSearch,
\n 51 )
\n 52 from .snr import DetectorStates, SignalToNoiseRatio
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:21
\n 16 from pyfstat.core import BaseSearchClass, SearchForSignalWithJumps
\n 18 logger = logging.getLogger(name)
\n---> 21 class Writer(BaseSearchClass):
\n 22 \"\"\"The main class for generating data in the form of SFTs.
\n 23
\n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,
\n (…)
\n 35 for more detailed help with some of the parameters.
\n 36 \"\"\"
\n 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/make_sfts.py:38, in Writer()
\n 21 class Writer(BaseSearchClass):
\n 22 \"\"\"The main class for generating data in the form of SFTs.
\n 23
\n 24 Short Fourier Transforms (SFTs) are a standard data format used in LALSuite,
\n (…)
\n 35 for more detailed help with some of the parameters.
\n 36 \"\"\"
\n---> 38 mfd = utils.get_lal_exec(\"Makefakedata_v5\")
\n 39 \"\"\"The executable; can be overridden by child classes.\"\"\"
\n 41 signal_parameter_labels = [
\n 42 \"tref\",
\n 43 \"F0\",
\n (…)
\n 54 \"transientTau\",
\n 55 ]
File ~/.conda/envs/pyfstat/lib/python3.11/site-packages/pyfstat/utils/runlalsuite.py:35, in get_lal_exec(cmd)
\n 33 full_cmd = shutil.which(\"lalpulsar_\" + cmd) or shutil.which(\"lalapps_\" + cmd)
\n 34 if full_cmd is None:
\n---> 35 raise RuntimeError(
\n 36 f\"Could not find either lalpulsar or lalapps version of command {cmd}.\"
\n 37 )
\n 38 return os.path.basename(full_cmd)
RuntimeError: Could not find either lalpulsar or lalapps version of command Makefakedata_v5.
", "votes": 0, "replies": [ { "id": 2020626, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-07T16:41:40.560000", "content": "Hi @kami2suukyi ,
\nThanks for the post.
\nI see you are using Python 3.11…
\ncould you try to re-do your environment using Python 3.10 and see if the issue still persists?
@rodrigotenorio Thanks for the quick reply.
\nI followed your advice and created a clean environment with the following command:
\n$ conda create -n pyfstat python=3.10
Then I installed only pyfstat and jupyter.
\n$ conda activate pyfstat
\n$ conda install -c conda-forge pyfstat jupyter
But I got the exact same error as before. I had the same error with python=3.7 too.
", "votes": 0, "replies": [] }, { "id": 2021862, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-08T14:24:12.287000", "content": "Hi @kami2suukyi ,
\nWhat operative system are you using? Could you open up a terminal with the conda environment activated
\nand run the following command:
$ which lalapps_version\n@rodrigotenorio My operating system is Ubuntu 20.04.5 LTS.
\nThe result of typing is as follows:
\n/home/USER/.local/bin/lalapps_version
Thanks again and again for your advice.
", "votes": 0, "replies": [] }, { "id": 2022158, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-08T19:06:10.873000", "content": "Hi @kami2suukyi , I'm surprised by that result (having that executable under $USER/.local/bin).
\nHave you, by any chance, installed lalsuite or pyfstat using your system's Python? (i.e. running pip install lalsuite or pip install --user lalsuite without any kind of virtual environment)?
Hello @rodrigotenorio, thanks to your advice I understood that there is a problem with the path to lalapps_Makefakedata_v5. I am sure I have pip install lalasuite because I have tried many things and did not work.
\nI pip uninstalled lalasuite and lscsoft-glue from all conda environments. Then I completely rebuilt the conda environment with a new python 3.10 and conda install -c conda-forge pyfstat jupyter.
\nThe return value of the result of \"$ which lalapps_version\" in its current state is nothing, but that of the result of \"$ which lalpulsar_Makefakedata_v5\" is \"~/.conda/envs/pyfstat/bin/lalpulsar_Makefakedata_v5\".
\nIn above situation I got the exact same error as before.
Hi @kami2suukyi ,
\nThanks for your answers.
\n\n\nIn above situation I got the exact same error as before.
\n
Could you paste which command you actually run?
\nCan I ask you to run the following commands?
\n- \n
- Outside and inside the conda environment: \n
echo $PATH \n- \n
- Inside the conda environment: \n
$ python\n>>> import os\n>>> os.environ(\"PATH\")\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nHello @rodrigotenorio, I'm sorry for late reply.
\nThanks very much for all your help and advice.
(1) I recreated clean python 3.10 conda environment, then I registered that environment with jupyter.
\n$ conda create -n pyfstat-py310 python=3.10\n$ conda activate pyfstat-py310\n$ conda install -c conda-forge pyfstat jupyter\n$ ipython kernel install --user --name pyfstat-py310 --display-name pyfstat-py310\n(2) outside conda environment:
\n$ echo $PATH\n/home/USER/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\n(3) inside conda environment:
\n$ python\nPython 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n>>> import os\n>>> os.environ[\"PATH\"]\n'/home/USER/.conda/envs/pyfstat-py310/bin:/home/USER/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Need one of --IFOs, --noiseSFTs or --inFrChannels to determine detectors\n\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Invalid argument\nXLAL Error - main (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:180): Check failed: XLALInitMakefakedata ( &GV, &uvar ) == XLAL_SUCCESS\nXLAL Error - main (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:180): Internal function call failed: Invalid argument\n65280\n>>>\n(4) I ran your notebook. The error occurred in the second cell.
\nimport os\nimport sys\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport pyfstat\n\nfrom scipy import stats\n\n%matplotlib inline\nAnd I got the exact same error as before.
\n22-11-12 22:59:34.406 pyfstat INFO : Running PyFstat version 1.18.1\n22-11-12 22:59:34.725 pyfstat.utils.importing INFO : No $DISPLAY environment variable found, so importing matplotlib.pyplot with non-interactive 'Agg' backend.\n---------------------------------------------------------------------------\nRuntimeError Traceback (most recent call last)\nCell In [2], line 7\n 4 import numpy as np\n 5 import matplotlib.pyplot as plt\n----> 7 import pyfstat\n 9 from scipy import stats\n 11 get_ipython().run_line_magic('matplotlib', 'inline')\n\nFile ~/.conda/envs/pyfstat-py310/lib/python3.10/site-packages/pyfstat/__init__.py:37\n\n...\n\nFile ~/.conda/envs/pyfstat-py310/lib/python3.10/site-packages/pyfstat/utils/runlalsuite.py:35, in get_lal_exec(cmd)\n 33 full_cmd = shutil.which(\"lalpulsar_\" + cmd) or shutil.which(\"lalapps_\" + cmd)\n 34 if full_cmd is None:\n---> 35 raise RuntimeError(\n 36 f\"Could not find either lalpulsar or lalapps version of command {cmd}.\"\n 37 )\n 38 return os.path.basename(full_cmd)\n\nRuntimeError: Could not find either lalpulsar or lalapps version of command Makefakedata_v5.\nHi @kami2suukyi ,
\nThank you very much for being so thorough. I think I know what's going on.
\nIf I'm not mistaken, the following line
\n>>> os.system(\"lalpulsar_Makefakedata_v5\")\nXLAL Error - XLALInitMakefakedata (/home/conda/feedstock_root/build_artifacts/lalpulsar-split_1667605874815/work/bin/MakeData/makefakedata_v5.c:380): Need one of --IFOs, --noiseSFTs or --inFrChannels to determine detectors\ntells us that you have Makefakedata installed in your system.
\nJust to be extra sure: Can you run which lalpulsar_Makefakedata_v5 outside your conda environment as well?
\n\nthen I registered that environment with jupyter.
\n
Ok, I think I've never done that myself.
\nCan you try to star the notebook by just running
\n$ jupyter notebook\nin your conda environment?
", "votes": 1, "replies": [] }, { "id": 2027714, "author_name": "HIROFUMI OHTA", "author_url": "", "post_date": "2022-11-13T06:09:02.367000", "content": "Hi @rodrigotenorio, thanks to your advice I finally solved the problem.
\nSince I have started jupyter as a server in my environment, it was difficult to start jupyter locally as you requested, I added \"env\" to \"~/.local/share/jupyter/kernels/pyfstat-py310/kernel.json\" as follows
{\n \"argv\": [\n \"/home/USER/.conda/envs/pyfstat-py310/bin/python3.10\",\n \"-m\",\n \"ipykernel_launcher\",\n \"-f\",\n \"{connection_file}\"\n ],\n \"env\": {\n \"PATH\": \"${HOME}/.conda/envs/pyfstat-py310/bin:${HOME}/.local/bin:/usr/local/cuda-11.7/bin:/usr/share/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\"\n },\n \"display_name\": \"pyfstat-py310\",\n \"language\": \"python\",\n \"metadata\": {\n \"debugger\": true\n }\n}\nNow I can call the registered environment \"pyfstat-py310\" from jupyter and run the code.
\nThanks again and again for your help.
", "votes": 1, "replies": [] } ] }, { "id": 2020140, "author_name": "BamunParticle", "author_url": "", "post_date": "2022-11-07T07:56:08.007000", "content": "I ran into a error while installing pyfstat in the tutorial notebooks and I am not able to understand what is causing it…
\nI cloned the entire pyfstat in github desktop and from there I accessed the tutorials.
Hi @bamunparticle ,
\nIt looks like you are trying to install PyFstat in a Windows machine, which is unfortunatelly unsupported at the moment.
\nCan you confirm that this is indeed the case?
\nSolutions may include using a linux vm, Kaggle kernels or Google Collab.
", "votes": 2, "replies": [] } ] }, { "id": 2016404, "author_name": "Felipe Loque", "author_url": "", "post_date": "2022-11-04T01:23:58.957000", "content": "Hi @rodrigotenorio Please how can I generate a data like this one:
\n\n\nID: 001121a05
\n\n
\n- \n
L1
\n -- SFTs: (360, 4653)
\n -- timestamps: (4653,)- \n
H1
\n -- SFTs: (360, 4612)
\n -- timestamps: (4612,)- \n
Frequency data: (360,)
I mean, which parameters do I have to set to achieve those shapes?
", "votes": 0, "replies": [ { "id": 2016896, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-04T10:38:55.233000", "content": "Hi @felipefonte99 ,
\nWell, there's many ways in which you could achieve that, and basic tools for that are explained in tutorial 1.
\nEssentially, that 360 corresponds to 0.2Hz with Tsft = 1800, as discussed at different points in this competition.
\nThe best way you could get that is by generating a larger band and slicing out the relevant part of it.
As for timestamps, each timestamp will give you one of those 4653 (or 4612) points, so it's a matter of deciding where to start and how big the space between them.
\nNote that those numbers do not specify a starting frequency (the typical values of which you can get from some amazing EDA notebooks).
", "votes": 3, "replies": [] }, { "id": 2058953, "author_name": "Neeraj Anand", "author_url": "", "post_date": "2022-12-08T10:53:29.447000", "content": "Hi @rodrigotenorio,
\nFor timestamps to be similar like 4653, we have to select appropriate value of duration ?
Hi @neerajanandcoder ,
\nSo, each timestamp corresponds to an SFT lasting for 1800s, meaning you'd need duration = 8375400 (1800 * 4653) to get
\n4653 consecutive timestamps.
You could also create an array of timestamps (start at some GPS time after 2020 and add numbers bigger than 1800 until you fill up 4653) and pass that to Writer. This way, you'll be able to generate non-consecutive timestamps (such as the ones provided in the test set).
Let me know if this helped.
", "votes": 1, "replies": [] }, { "id": 2059824, "author_name": "Neeraj Anand", "author_url": "", "post_date": "2022-12-09T09:00:44.833000", "content": "Thanks for the reply!!
", "votes": 1, "replies": [] } ] }, { "id": 2002494, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-24T21:06:53.333000", "content": "@rodrigotenorio
\nRodrigo, can you also elaborate a bit on the meaning of the SFTWindowBeta parameter? The documentation is kind of lacking \"Optional parameter for some windowing functions\" I see it set at 0.01 or 0.001.
\nWhich one do we need? :)
Hi @sakvaua , sure.
\nSome windows accept a \"tuning parameter\" which adjust their shape.
\nIn these examples, we use a Tukey window, which is one of the typical choices made in continuous wave searches. For this specific window, the SFTWindowBeta parameter then tunes how long is the fall-off of this window near the edges: A value of 0 sets a rectangular window, while a value of 1 gives you a Hann window.
We normally set this value to 0.01 or 0.001, and at that level I don't think it would matter that much which one you pick.
", "votes": 5, "replies": [] }, { "id": 2003274, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-25T11:48:35.003000", "content": "Thanks, Rodrigo
\nNNs are very sensitive to such usually imperceivable differences. For example, switching from CV2 jpeg decoding to PIL can result in ca. 1% of accuracy on imagenet. Amazing!
1 Is the train / test dataset real collected dataset?
\n2 Are the sqrtSX for generating the (train/test) dataset almost fixed parameters?
\nIn the case of actual collected data, are the noise levels intentionally added after data collection equal or zero?
3 What range does h0 have to reproduce the example data (train dataset)?
\n\n\nThe typical amplitudes of the resulting signals are one or two orders of magnitude lower than the amplitude of the detector noise.
\n
Can we assume h0 = sqrtSX * (1 to 0.5) in this case?
\nIf not, does it mean log(h0) = log(sqrtSX) * (1~2) ?
4 I can't understand the following parameters. Is there any related data?
\nF1, Alpha, Delta, cosi, psi, phi
\nI presume these are variables related to astrophysics, but I'd like to understand how they can be adjusted to produce the shape of the signal I want.
For example I would like to know how to create an sft that looks like a cosine function with a shorter period.
", "votes": 0, "replies": [ { "id": 1996595, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-20T10:13:01.597000", "content": "Hi @assign , thanks for taking part of this :)
\n- \n
Yes, part of the data we have provided is taken directly from the Advanced LIGO detectors. In some cases we have included a signal, in some cases we haven't, but noise properties should be essentially those of real data.
\nI'm not sure if I understand this question, but in some cases data is stationary (hence you could use
sqrtSXto describe it) and in some cases, such as real data, it's not, meaning a singlesqrtSXmay not be representative enough. You could actually try yourself using severalsqrtSXfor different periods of time whenever you generate your data. \nParameters like
F0, F1, Alpha, Deltaaffect the frequency evolution of your signal, whereas parameters such ascosi, phi, psiaffect the amplitude of the signal. For the latter, there's this nice visualization which may help in visualizing the angles. \n
Initially, I was baffled by the amount of data/documentation/symbols that seemed confusing to me as a non-expert, but after generating dozens of sample data in PyFstat, I understood to some extent how it affects the graph. I don't know exactly what units and numbers each affects though. It would be nice to be able to see the code it generates, but it's hard to trace because they use system commands.
\nThanks for the reply.
", "votes": 0, "replies": [] } ] }, { "id": 1994273, "author_name": "Cristo JV", "author_url": "", "post_date": "2022-10-18T22:57:44.977000", "content": "Hi @rodrigotenorio !
\nThanks, for such brain-consuming competition.
\nShould we consider that there is signal in the test when the depth sensing is between 10 and 50 Hz**1/2 as stated in the previous notebooks?
\nOr are we being challenged to address deeper depth conditions?
Hi @cristojv ,
\nAll I can say is that there is a broad distribution of SNR / amplitudes / depth in the data.
\nHope that helps 😊.
\n\nIn total there are eight parameters which have all been randomised.
\n
Are they F0, F1, Alpha, Delta, Depth, cosi, psi, phi?
", "votes": 0, "replies": [ { "id": 1992244, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-17T15:44:37.653000", "content": "Hi zzy,
\nYou can also find a list of the randomized parameters in the data page.
\nThe relevant parameters are the ones you mentioned, except for Depth: Data-generation codes work in terms of h0, which is a more physical quantity, while Depth is reserved to interpret the results.
Cheers,
", "votes": 1, "replies": [] }, { "id": 1992250, "author_name": "Leon", "author_url": "", "post_date": "2022-10-17T15:47:12.297000", "content": "Thanks for your reply!
", "votes": 0, "replies": [] }, { "id": 1996241, "author_name": "assign", "author_url": "", "post_date": "2022-10-20T06:33:37.217000", "content": "\n\n\nYou can also find a list of the randomized parameters in the data page.
\n
I can't understand this statement.
\nDoes that mean that 8 parameters are written on the data page?
\nI'd appreciate it if you could explain clearly
Hi @assign, just to clarify things. Rodrigo is referring to this sentence from the data page:
\n\n\nThe signals are characterised by the location and orientation of the hypothetical astrophysical source as well as two intrinsic parameters: frequency and spin-down. In total there are eight parameters which have all been randomised.
\n
We can break this down a bit, the eight parameters are:
\n- \n
- Two parameters for the location on the sky \n
- Three parameters for the orientation of the source \n
- One parameter that describes the \"strength\" of the signal (see Rodrigo's comment above about
h0) \n - Two intrinsic parameters: frequency and spin-down \n
Hope this helps :)
", "votes": 4, "replies": [] }, { "id": 1996600, "author_name": "assign", "author_url": "", "post_date": "2022-10-20T10:15:55.453000", "content": "@michaeljwill Thank you for answer. Now I understand what it means.
\nCould you please check if the parameters used in PyFstat are mapped the same as the following ones?
Two parameters for the location on the sky: Alpha, Delta
\nThree parameters for the orientation of the source: cosi, psi, phi
\nOne parameter that describes the \"strength\" of the signal: h0
\nTwo intrinsic parameters: frequency and spin-down: F0, F1
Thanks again
", "votes": 3, "replies": [] }, { "id": 1996624, "author_name": "Michael J. Williams", "author_url": "", "post_date": "2022-10-20T10:30:19.860000", "content": "@assign Yes, that mapping is correct :)
", "votes": 1, "replies": [] } ] }, { "id": 1987676, "author_name": "Bhaskar Verma", "author_url": "", "post_date": "2022-10-14T19:29:46.260000", "content": "Hey Rodrigo,
\nThanks for the information and tutorials.
I was trying to run the noise generation code.
\nThe first error its giving me is \"No /data/bverma/.pyfstat.conf file found\". This leads to the consiquent errors.
I have checked the pyfstat files and my computer files and I am unable to find the particular config file.
\nI have included the entire error in this thread.
\n\"
\n22-10-14 14:32:14.350 pyfstat.core INFO : Creating Writer object…
\n22-10-14 14:32:14.352 pyfstat.utils.ephemeris INFO : No /data/bverma/.pyfstat.conf file found. Will fall back to lal's automatic path resolution for files [earth00-40-DE405.dat.gz,sun00-40-DE405.dat.gz]. Alternatively, set 'earth_ephem' and 'sun_ephem' class options.
\n22-10-14 14:32:14.354 pyfstat.make_sfts INFO : Generating SFTs with fmin=99.5, Band=1.0
\n22-10-14 14:32:14.355 pyfstat.make_sfts INFO : Got h0=0, not writing an injection .cff file.
\n22-10-14 14:32:14.355 pyfstat.make_sfts INFO : Checking if we can re-use existing SFT data file(s)…
\n22-10-14 14:32:14.356 pyfstat.make_sfts INFO : …no SFT file matching 'PyFstat_example_data/H-240_H1_1800SFT_single_detector_gaussian_noise-1238166018-432000.sft' found. Will create new SFT file(s).
\n22-10-14 14:32:14.356 pyfstat.utils.cli INFO : Now executing: lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"
\n22-10-14 14:32:14.391 pyfstat.utils.cli ERROR : Execution failed: Command 'lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"' returned non-zero exit status 255.
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR : XLAL Error - XLALReadEphemerisFile (LALInitBarycenter.c:482): Failed to find ephemeris-file 'earth00-40-DE405.dat.gz[.gz]'
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR :
\n22-10-14 14:32:14.392 pyfstat.utils.cli ERROR : XLAL Error - XLALReadEphemerisFile (LALInitBarycenter.c:482): Invalid argument
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR : XLAL Error - XLALInitBarycenter (LALInitBarycenter.c:252): XLALReadEphemerisFile('earth00-40-DE405.dat.gz') failed
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR :
\n22-10-14 14:32:14.393 pyfstat.utils.cli ERROR : XLAL Error - XLALInitBarycenter (LALInitBarycenter.c:252): Internal function call failed: Invalid argument
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:369): Check failed: (cfg->edat = XLALInitBarycenter ( uvar->ephemEarth, uvar->ephemSun )) != ((void *)0)
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - XLALInitMakefakedata (makefakedata_v5.c:369): Internal function call failed: Invalid argument
\n22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - main (makefakedata_v5.c:181): Check failed: XLALInitMakefakedata ( &GV, &uvar ) == XLAL_SUCCESS
22-10-14 14:32:14.394 pyfstat.utils.cli ERROR : XLAL Error - main (makefakedata_v5.c:181): Internal function call failed: Invalid argument
\nCalledProcessError Traceback (most recent call last)
\nCell In [6], line 18
\n 15 writer = pyfstat.Writer(**writer_kwargs)
\n 17 # Create SFTs
\n---> 18 writer.make_data()
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/make_sfts.py:707, in Writer.make_data(self, verbose)
\n 705 else:
\n 706 logger.info(\"Got h0=0, not writing an injection .cff file.\")
\n--> 707 self.run_makefakedata()
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/make_sfts.py:721, in Writer.run_makefakedata(self)
\n 719 check_ok = self.check_cached_data_okay_to_use(cl_mfd)
\n 720 if check_ok is False:
\n--> 721 utils.run_commandline(cl_mfd)
\n 722 if not np.all([os.path.isfile(f) for f in self.sftfilenames]):
\n 723 raise IOError(
\n 724 f\"It seems we successfully ran {self.mfd},\"
\n 725 f\" but did not get the expected SFT file path(s): {self.sftfilepath}.\"
\n 726 f\" What we have in the output directory '{self.outdir}' is:\"
\n 727 f\" {os.listdir(self.outdir)}\"
\n 728 )
File ~/.conda/envs/ContGW/lib/python3.10/site-packages/pyfstat/utils/cli.py:41, in run_commandline(cl, raise_error, return_output)
\n 37 logger.warning(
\n 38 \"Pipe ('|') found in commandline, errors may not be properly caught!\"
\n 39 )
\n 40 try:
\n---> 41 completed_process = subprocess.run(
\n 42 cl,
\n 43 check=True,
\n 44 shell=True,
\n 45 capture_output=True,
\n 46 text=True,
\n 47 )
\n 48 msg = completed_process.stdout
\n 49 if msg:
File ~/.conda/envs/ContGW/lib/python3.10/subprocess.py:524, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
\n 522 retcode = process.poll()
\n 523 if check and retcode:
\n--> 524 raise CalledProcessError(retcode, process.args,
\n 525 output=stdout, stderr=stderr)
\n 526 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command 'lalapps_Makefakedata_v5 --outSingleSFT=TRUE --outSFTdir=\"PyFstat_example_data\" --outLabel=\"single_detector_gaussian_noise\" --IFOs=\"H1\" --sqrtSX=\"1e-23\" --SFTWindowType=\"tukey\" --SFTWindowBeta=0.01 --startTime=1238166018 --duration=432000 --fmin=99.5 --Band=1 --Tsft=1800 --ephemEarth=\"earth00-40-DE405.dat.gz\" --ephemSun=\"sun00-40-DE405.dat.gz\"' returned non-zero exit status 255.
\n\"
\nThanks for your help,
\nBhaskar Verma
Hi Bhaskar Verma,
\nThanks for reporting this.
\n\n\nThe first error its giving me is \"No /data/bverma/.pyfstat.conf file found\". This leads to the consiquent errors.
\n
That is sorta expected. The .pyfstat.conf file is something you could create in order to configure PyFstat in a specific way, but you are not required to do so. Rather, PyFstat should already include everything it needs to work.
Can I ask you to report this as an issue so we can follow it up?
\nIt looks like you don't have the required ephemeris files, which should have been pulled together if you were using the latest version of LALSuite. What pip version are you using? Can you paste pip --version and pip list in that issue as well?
Cheers,
", "votes": 0, "replies": [] } ] }, { "id": 1987054, "author_name": "Clément Verrier", "author_url": "", "post_date": "2022-10-14T13:24:03.933000", "content": "Hi Rodrigo,
\nThank you for this tutorial.
\nI have a question about the fact that competitors need to generate additional data for training: why do we have to generate our own samples, i.e. why did not you generate all samples by yourself and then upload them as the full training data? Is it because you except each competitor to have its own training dataset and also except that each competitor tries to generate data using personal approach (e.g. based on some parameter estimations using the available training instances)?
\nThanks in advance!
", "votes": 0, "replies": [ { "id": 1987365, "author_name": "Michael J. Williams", "author_url": "", "post_date": "2022-10-14T17:08:11.790000", "content": "Hi @cverrier,
\nFirstly, thanks for taking part in the competition :)
\nThere were a few reasons for handling the data this way but the main factors were the following:
\n- \n
- There are limits to how much data we as hosts can upload for a competition, the ~200 GB is already pushing the limit, so we couldn't really upload significantly more. \n
- The data has to be split between training and testing. We felt it was more important to maximise the amount of test data since the scores on the leaderboard will be more reliable the more test data is used. This meant that ultimately we could only provide a limited amount of training data. Given the nature of the problem, we think it's likely that you'll need more training data, so we decided it would be fairer if we provide code for everyone to use. \n
- In the last competition we ran (G2Net Gravitational Wave Detection) one of the main points of feedback we received from the community was that having code to generate your own data would have been helpful. In fact, the first-place team won partly because they were one of the few teams that wrote their own code to generate data. \n
I hope this gives you some insight into our thought process when we were deciding how to provide the data.
", "votes": 4, "replies": [ { "id": 1989235, "author_name": "Clément Verrier", "author_url": "", "post_date": "2022-10-15T19:41:25.683000", "content": "Hi @michaeljwill,
\nThank you for your reply and detailed explanations.
\n\n\nI hope this gives you some insight into our thought process when we were deciding how to provide the data.
\n
Crystal clear! Thanks again 👍
", "votes": 0, "replies": [] } ] } ] }, { "id": 1979785, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2022-10-09T17:28:46.987000", "content": "Hi Rodrigo,
\nI am wondering about the SNR of the real data. What range of SNRs do physicist's actually expect?
\nIs it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?
", "votes": 0, "replies": [ { "id": 1980712, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-10T10:35:26.907000", "content": "\n\nWhat range of SNRs do physicist's actually expect?
\n
0 to arbitrarily large, depending on how good your detector is ;)
\nSince we haven't detected any of these signals yet, we have no idea about how strong they actually are. You can check some of the LIGO papers (look for the CW tag) to get a sense of how sensitive our searches are (i.e. how strong could this signals be while remaining undetected); I wouldn't quote any specific numbers, however, since they are highly dependent on search setup and detector configuration.
\n\nIs it right to assume that a SFT given out of blue might have high SNR but might not have the CW signal since signal can be from say electrical interference (60Hz)?
\n
Minor comment on notation: when we talk about SNR, we refer to the SNR of a specific model. That is, given a set of parameters characterizing a signal, I can compute the SNR associated to that specific signal in the data. Data itself doesn't have an \"SNR\" in our language.
\nNow, going to your question: Absolutely. Part of the test set is based on actual data taken by the Advanced LIGO detectors, hence it may contain some instrumental artifacts in there with which you'll have to deal with.
", "votes": 1, "replies": [] } ] }, { "id": 1979255, "author_name": "GodGod3", "author_url": "", "post_date": "2022-10-09T09:00:16.357000", "content": "Hi.Is there a fixed window length for fourier transform？
", "votes": 0, "replies": [ { "id": 1979930, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2022-10-09T20:23:03.160000", "content": "For this kind of signal the recommended lenght is 1800seconds (30 mins). There might be gaps in the timestamp but the considered length is 1800 seconds. Rodrigo can confirm better. :)
", "votes": 2, "replies": [] }, { "id": 1980396, "author_name": "GodGod3", "author_url": "", "post_date": "2022-10-10T06:25:42.070000", "content": "So each timestamp coressponds to a period ?And why are the values for amplitude so small after the fourier transformation?
", "votes": 0, "replies": [] }, { "id": 1980689, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-10T10:12:07.303000", "content": "Hi GodGod3,
\nYes, each timestamp labels a period of time of 1800s (you can check that by noting the frequency resolution of 1/1800 Hz), but timestamps need not to be consecutive (i.e. there may be times at which no data is collected).
", "votes": 0, "replies": [] }, { "id": 1981127, "author_name": "GodGod3", "author_url": "", "post_date": "2022-10-10T15:51:02.497000", "content": "THANKS.To clarify,is the sampling frequency also the window for fourier transformation?How are the gaps handled during fourier transformation?
", "votes": 0, "replies": [] }, { "id": 1984735, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-12T20:36:44.127000", "content": "Ups, sorry GodGod3, seems like I missed this comment.
\nFourier transforms are taken over continuous segments. For every timestamp, you can assume there is a continuous span of time on which we applied the Fourier transform. Whenever we hit a gap, we stop and wait until we have again enough data to compute another Fourier transform.
", "votes": 0, "replies": [] } ] }, { "id": 1973698, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2022-10-05T19:08:58.170000", "content": "For those who are interested this is the main Writer class that generates the data: https://github.com/PyFstat/PyFstat/blob/b173b3d6e39088fa3f50110033e6c7ec2c1d6f54/pyfstat/make_sfts.py#L21
Hey Rodrigo Tenorio, thank you for providing resources for generating more data. I am unclear about the \"target\" part while generating the data. The kenrel provided to generate the data does not seem to the information about the label (target 1, 0, or -1). Or am I missing something? Thanks in advance.
", "votes": 0, "replies": [ { "id": 1973327, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-10-05T15:11:19.807000", "content": "Hi Ayush Thakur,
\nThanks for participating in the challenge.
\n\n\nThe kenrel provided to generate the data does not seem to the information about the label
\n
0/1 labels refer to whether we included a simulated signal to that sample (1) or if it consists only on noise.
\nIn terms of the quantities of the kernel your refer to, a label of 1 would correspond to a signal with an amplitude h0 greater than 0 (i. e. there's a signal), while a label of 0 would correspond to not having a signal (i.e. amplitude h0 = 0).
The same definition can be made in terms of SNR: Label of 1 corresponds to SNR > 0, label 0 corresponds to SNR = 0 (no signal added at all).
", "votes": 1, "replies": [] }, { "id": 1973367, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2022-10-05T15:32:42.577000", "content": "Thanks for the clarification. This helps. :)
", "votes": 0, "replies": [] } ] }, { "id": 2074247, "author_name": "", "author_url": "", "post_date": "2022-12-23T22:13:22.910000", "content": "", "votes": 0, "replies": [] }, { "id": 2032972, "author_name": "", "author_url": "", "post_date": "2022-11-17T00:05:06.057000", "content": "Hi, when I generate new data, I get different shapes of the SFTs. For example I get shape (100, 150) one time and then (123, 150) the other time (so the time is fixed but I get different shapes for the span of amplitudes). Could you please specify if there is a way to generate fixed shape SFTs.
", "votes": 0, "replies": [ { "id": 2033017, "author_name": "chris", "author_url": "", "post_date": "2022-11-17T01:44:54.097000", "content": "The \"height\" is controlled by the frequency band. To get 360 (like the train/test data), you want a band of 0.2 Hz, so you can set: \"Band\": 0.2, or you can make it larger and clip it
", "votes": 1, "replies": [] } ] }, { "id": 2028083, "author_name": "", "author_url": "", "post_date": "2022-11-13T14:12:51.630000", "content": "[EDIT: I just understood that the Delta and Alpha coordinates are in equatorial coordinatees and not in ecliptic coordinates, which answers my question]
\nHello all,
\nIf I understood correctly, due to the relative speed between the detector and the source, there are two main frequency modulations of the signal: one due to the rotation of the earth, which is contained within a frequency band, and one due to the rotation of the earth.
\nThe second frequency modulation should disappear if the source direction is perpendicular to the ecliptic plan? Therefore I thought that by setting Delta = np.pi / 2, I would get a signal with a constant frequency but this is not the case. Such signals still have a yearly modulation of their frequency. Is it because there are other physical phenomenon that takes place? Or because Delta is measure from the equator and not the ecliptic?
\nHere are the parameters I'm using to generate such a signal:
\nparams = {\n \"tstart\": 1238166018,\n \"tref\": 1238166018,\n \"duration\": 2*365 * 24 * 60 * 60, # 2*365 days\n \"detectors\": \"H1,L1\",\n \"Band\": 0.2,\n \"sqrtSX\": 1e-23,\n \"Tsft\": 1800,\n \"SFTWindowType\": \"tukey\",\n \"SFTWindowBeta\": 0.01,\n \"h0\": 1e-23,\n \"F0\": 150.15,\n \"F1\": 0.0,\n \"F2\": 0.0,\n \"Alpha\": 0.0,\n \"Delta\": np.pi / 2.0,\n \"cosi\": 1,\n \"psi\": 0.0,\n \"phi\": 0.0,\n }\nMany thanks for your help!
\nTantto
Hello! Thank you for the competition and the effort you made to set it up! I have few questions on the data, for which I could not find the answer in that thread nor in the description:
\n(i) I'm not sure to understand the meaning of the label -1 ? If I understand it right it only concerns samples from the training dataset and not the test dataset?
\n(ii) Do we know the proportion of data sample in the train/test using simulated noise vs real-noise? As I could not see that information, I'm assuming it's not a public information of that challenge?
\n(iii) Is it at least the same proportion in the train and test dataset?
\n(iv) In the dataset using real-noise, how are we sure that there is no CW source within it? I mean, what differentiate a sample consisting of simulated noise + a very low simulated source VS a sample of real noise with no simulated source added but that could have potentially a real CW sourced that was not detected?
\n(v) A related question: did you set the minimum value of the amplitude of the simulated sources used bigger than what you could detect using your pipeline in order to avoid the situation described in the previous question?
Many thanks for your help! Let me know if my questions are unclear or already answered somewhere.
", "votes": 1, "replies": [ { "id": 2014526, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-02T16:12:54.150000", "content": "Hi @tantto ,
\n(i) Well, we are not sure either :) Due to their exceptional features, we wanted to get some input from the Kaggle community: maybe it's just noise and we failed to understand it, maybe it's a weird signal we don't know how to interpret. In any case, any shared knowledge on it will be very welcome.
\n(ii) The only public information is that there's both classes of noise in the test set.
\n(iii) The training set is a small sample we provided so that Kaggler (such as yourself :) ) would get a sense of how this competition would play out. I'm afraid we did not include any real data, since we wanted to put as much of it as possible in the test set.
\n(iv) We are not. It could well be the case that a real CW signal is buried deep down into the real noise. Of course, for that to be the case, such a signal would have escaped every single one of the LVK searches we have conducted so far.
\nI hope this clarifies your questions.
\nCheers,
", "votes": 1, "replies": [] }, { "id": 2014710, "author_name": "", "author_url": "", "post_date": "2022-11-02T19:15:29.143000", "content": "Thanks for the answers!
", "votes": 0, "replies": [] }, { "id": 2061297, "author_name": "Naren Manikandan", "author_url": "", "post_date": "2022-12-10T23:38:43.060000", "content": "What is this LVK search?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1909452": "Most continuous-wave simulation tools are built around [LALSuite](https://git.ligo.org/lscsoft/lalsuite), \na core software library for the analyses of the LIGO-Virgo-KAGRA Collaboration written in a custom version of C. \nIf required, most of its low-level functions can be access in Python thanks to [LALSWIG](https://arxiv.org/abs/2012.09552)\n\nFor this challenge, we suggest you use [PyFstat](https://github.com/PyFstat/PyFstat), a Python package which, amongst other things, wraps the basic data-generation routines in LALSuite and allows to read binary-format data (such as SFTs) as a simple [numpy](https://numpy.org/) array. PyFstat can be readily installed using `pip`\n```\npip install pyfstat jupyter\n```\nor `conda`\n```\nconda install -c conda-forge pyfstat jupyter\n```\nwhere we included an installation of `jupyter` in order to read the tutorial notebooks.\n\n[Tutorials](https://github.com/PyFstat/PyFstat/tree/master/examples/tutorials) are in the form of [Jupyter](https://jupyter.org/) notebooks, so we recommend you clone the [PyFstat repository](https://github.com/PyFstat/PyFstat) in order to have the required scripts around:\n- [Tutorial 0: Noise generation](https://github.com/PyFstat/PyFstat/blob/master/examples/tutorials/0_generating_noise.ipynb)\n- [Tutorial 1: Signal generation](https://github.com/PyFstat/PyFstat/blob/master/examples/tutorials/1_generating_signals.ipynb)\n\nAdditionally, have a look at [this short Kaggle notebook](https://www.kaggle.com/code/rodrigotenorio/generating-continuous-gravitational-wave-signals) if you to prefer to start right off generating some signals. \nNote that Kaggle notebooks run on Python 3.7, which was dropped by PyFstat a few releases ago. If you use this Kaggle notebook, please stick to the PyFstat version suggested within; do use the latest version of PyFstat if you decide to use your own installation. \n\nIf you ran into any bugs / unexpected behavior whenever using PyFstat, we would really appreciate if you free opened up an issue reporting the problem in the [issue tracker](https://github.com/PyFstat/PyFstat/issues).", "2019429": "Can someone please explain why data generation is needed for this competition? I've read through the description and all effort is on \"how\" data can be generated, not \"why\".\n\nTraditionally, in a supervised learning task, there're a labelled training set and a test set, and you train a model that hopefully predicts well on the test set. Where and why is there a need for data generation? If the reason is that the training set is too small and data collection is too costly/impossible, and if data generation makes sense for this problem, why can't the host just generate a whole lot more data for us, so that we can treat this problem as just a \"traditional supervised learning task\" described above?", "1997129": "Hello, thanks for exciting competition.\nI would like to ask: what are meaningful values of F1 (spindown)? Are the data supposed to simulate isolated neutron stars (not binary systems)?\nIn [this notebook](https://www.kaggle.com/code/rodrigotenorio/generating-continuous-gravitational-wave-signals), positive values from range 1.0e-12 ... 1.0e-8 are used. So, in fact, the frequency is growing with time.\nIn [this tutorial](https://github.com/PyFstat/PyFstat/blob/master/examples/tutorials/1_generating_signals.ipynb), negative value of -1.0e-9 is used. \nIn both cases, the magnitude of spindown is surprisingly high for me. Maybe I'm wrong, but intuitively I would expect the rotation of neutron star to be much more stable.", "1986395": "Hi, Rodrigo\nThanks, for such a powerful brain-stimulant competition. Are the signals we are looking for located in the center of the frequency band?", "2064527": "Hi @rodrigotenorio\n\nI am trying to follow your code but there is something error. \n- Environment: Jupyter Notebook -> Kaggle \n- Run followed code: 'pip install pyfstat jupyter'\n- But still error like: 'No module named 'pyfstat.utils'\n\nHow can I use your code in Kaggle Notebook? \n\n(Additional Info: after i run 'pip install pyfstat jupyter')\nSuccessfully installed bashplotlib-0.6.5 corner-2.2.1 lalsuite-7.5 ligo-segments-1.4.0 lscsoft-glue-3.0.1 peakutils-1.3.4 ptemcee-1.0.0 pyRXP-3.0.1 pyfstat-1.16.0 versioneer-0.28\nWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">\nNote: you may need to restart the kernel to use updated packages.", "2027613": "Hello, thanks for exciting competition.\n\nI was trying to use PyFstat to generate some data. However, I encountered an error that appeared erratically.\nIt appears to occur in the signal injection.\n\nError: injection signal 0:'./band_signal.cff:TSO' needs frequency band [491.947874, 492.109913]Hz, injecting into [491.90000, 492.10000]Hz\n\nI'm confused because I've been setting F0 from np.random.randint(50, 500). where did x.947874 and x.109913 come from?\n\nI suspect that some underlying IO processes are conflicting when using loops to generate data. It's very frustrating.😣", "1976344": "Hi Rodrigo,\n\nThanks for the information and the tutorials. \n\nI am having troubles to read the .sft files written at the end of tutorial 1. I understood that I need to create an instance of Writer and put the .sft file in the \"noiseSFTs\" argument. I did this, I also added \"SFTWindowType\": \"tukey\", which seems to be a mandatory argument according to lalpulsar_Makefakedata_v5 --help\n\nMy code looks like this\n```\nimport os\nimport h5py\nimport numpy as np\nimport pandas as pd\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\n\nimport pyfstat\nfrom pyfstat.utils import get_sft_as_arrays\n\nwriter_kwargs = {\n \"noiseSFTs\": \"PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft\",\n \"SFTWindowType\": \"tukey\"\n }\n\nwriter = pyfstat.Writer(**writer_kwargs)\n```\n\nwhen I try to run it I get a I/O error\n```\nERROR: Failed to open matched file 'PyFstat_example_data_ensemble/Signal_3/H-5760_H1_1800SFT_Signal_3-1238166018-10368000.sft'\n\nXLAL Error - XLALSFTdataFind (SFTfileIO.c:318): I/O error\nTraceback (most recent call last):\n File \"/home/quentin/Documents/MLCompetitions/Kaggle/G2Net/src/train_Conv2D_synthetic_data.py\", line 16, in