{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **L and H distortion v1**","metadata":{}},{"cell_type":"markdown","source":"Good day my Kaggle friend. \n\nI am new here and want to share something for you.\n\nI hope this notebook will help you to improve your score. Possible the idea is not new for you in this case please skip. I am sure that for some kaggle friends it will help.\n\nIdea:\n\nI figered out that aproximately 20% of the gravitation wave spectra are distorted. I think the distortion was made artificially because of it is exatly the same distortion for H and L shoulders of the interferometer. \n\nPossible, they multiplied by exactly the same function to H and exactly the same (but different from H) to L shoulder.\n\nI have identified this fumction by averaging 10 distorted L and 10 distorted H spectra. You can make it better of course and average more spectra.\n\nI have presented my code here below.\n\nHow it can help?\n\nYou can multiply (with some constant) to your generated date or train data by this distortion function and it will be close to 20 percent of the distorted test spectra and train you NN with better accuracy.\n\nMoreover you can train two NN for distorted and non distorted spectra.\n\nYou just need to identify the distorted test spectrum you have or not.\n\nI can help you with it. Check my second code below to identify it.\n\nI hope it will help you.\n\nPlease vote if the information is interesting for you.\n\nIgor.\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# **L and H distortion****","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd\nimport random, glob\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nimport os\nimport h5py\nfrom PIL import Image\nfrom scipy import ndimage\nfrom scipy import signal\n\n\ndatasetfortrainingxmasks = []\n\n\nspectrogram1F = np.zeros(shape=(360, 180), dtype='float')\nspectrogram2F = np.zeros(shape=(360, 180), dtype='float')\n\n\ntrain_df =  pd.read_csv('/kaggle/input/distortionmask/masksDist.csv')\n\nID = train_df[\"id\"]\n\n\n#outputNN=[ 0 for y in range(1000)]\noutputNN=[0]\ni=0\nn=0\nprint(len(ID))\n\ndatasetfortrainingxmask =[]\n\nmaxelements = len(ID)\nsplit = 0.8\n\nTrainDS = True\n\nmaxelem = len(ID)*3\n\nfor i in range (0,9):\n\n    print(i)\n\n\n    n+=1\n    i+=1\n \n    j = i \n    \n\n    filename = ID[j]+\".hdf5\"\n\n\n    \n    f = h5py.File('/kaggle/input/distortionmask/'+ filename, 'r')\n    \n    H1_SFT = np.divide(f[filename[:-5]][\"H1\"]['SFTs'],1e-22)\n    L1_SFT = np.divide(f[filename[:-5]][\"L1\"]['SFTs'],1e-22)\n\n\n\n    spectrogram1 = (np.imag(L1_SFT[:,0:3960])**2 + np.real(L1_SFT[:,0:3960])**2)\n\n    spectrogram2 = (np.imag(H1_SFT[:,0:3960])**2 + np.real(H1_SFT[:,0:3960])**2)\n   \n    \n\n\n    spectrogram1 /= np.mean(spectrogram1)  # normalize\n    spectrogram1 = np.mean(spectrogram1.reshape(360, 180, 22), axis=2)#+v*np.random.randint(0,100,(360, 360))*0.01*0.02\n\n    spectrogram2 /= np.mean(spectrogram2)  # normalize\n    spectrogram2 = np.mean(spectrogram2.reshape(360, 180, 22), axis=2)#+v*np.random.randint(0,100,(360, 360))*0.01*0.02\n\n\n    spectrogram1F=spectrogram1F+spectrogram1\n    spectrogram2F=spectrogram2F+spectrogram2\n\n\n    \n    fig = plt.figure( figsize=(3, 3), dpi=180)\n    plt.imshow(spectrogram1, aspect=1)\n    cbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\n    plt.show()\n    \n    fig = plt.figure( figsize=(3, 3), dpi=180)\n    plt.imshow(spectrogram2, aspect=1)\n    cbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\n    plt.show()\n\n\ndsxmask = np.array(datasetfortrainingxmask)\n\n# averaging\n\nspectrogram1FF = spectrogram1F/9\n\nspectrogram2FF = spectrogram2F/9\n\n\n\n# final function for L and H\n\nprint(\"final averaging function for L\")\nfig = plt.figure( figsize=(3, 3), dpi=180)\nplt.imshow(spectrogram1FF, aspect=1)\ncbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\nplt.show()\n\nprint(\"final averaging for H\")\nfig = plt.figure( figsize=(3, 3), dpi=180)\nplt.imshow(spectrogram2FF, aspect=1)\ncbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-25T13:05:45.399587Z","iopub.execute_input":"2022-11-25T13:05:45.399984Z","iopub.status.idle":"2022-11-25T13:05:55.984373Z","shell.execute_reply.started":"2022-11-25T13:05:45.399953Z","shell.execute_reply":"2022-11-25T13:05:55.983065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is quite obvious that code which identyfy mean value at different x points can identify the distorted spectra. For example: \n diff = abs(np.mean(spectrogram1[:,80:100])-np.mean(spectrogram1[:,110:130]))\n \nFor undistorted case it will be close to 0 because of we will have the noise with no relative change.","metadata":{}},{"cell_type":"markdown","source":"# **Identifying the distorted spectrum**","metadata":{}},{"cell_type":"code","source":"\n\ndatasetfortrainingx = []\nIDj = []\n\ndef InpOut(a):\n    inpt = a.reshape((1,) + a.shape)\n    return inpt\n\n\ntrain_df =  pd.read_csv('../input/g2net-detecting-continuous-gravitational-waves/sample_submission.csv')\n\nID = train_df[\"id\"]\n\n\nsizeIm = 360\n\noutputNN=[ 0 for y in range(1000)]\n\ni=0\nn=0\n\n\nfor i in range (10,25):\n    n+=1\n    \n\n    \n    j = i\n\n    \n    filename = ID[j]+\".hdf5\"\n\n    f = h5py.File('../input/g2net-detecting-continuous-gravitational-waves/test/'+ filename, 'r')\n    \n\n    H1_SFT = np.divide(f[filename[:-5]][\"H1\"]['SFTs'],1e-22)\n    L1_SFT = np.divide(f[filename[:-5]][\"L1\"]['SFTs'],1e-22)\n    \n    if (L1_SFT.size/360)>3960 and (H1_SFT.size/360)>3960:\n        \n\n\n        spectrogram1 = (np.imag(L1_SFT[:,0:3960])**2 + np.real(L1_SFT[:,0:3960])**2)\n        spectrogram2 = (np.imag(H1_SFT[:,0:3960])**2 + np.real(H1_SFT[:,0:3960])**2)\n        spectrogram3 = (np.imag(H1_SFT[:,0:3960])**2 + np.real(H1_SFT[:,0:3960])**2)\n    \n    \n        spectrogram1 /= np.mean(spectrogram1)  # normalize\n        spectrogram1 = np.mean(spectrogram1.reshape(360, 180, 22), axis=2) \n        spectrogram2 /= np.mean(spectrogram2)  # normalize\n        spectrogram2 = np.mean(spectrogram2.reshape(360, 180, 22), axis=2) \n        \n# Here I detect where distorted or mot the spectra\n        \n        diff = abs(np.mean(spectrogram1[:,80:100])-np.mean(spectrogram1[:,110:130]))\n        \n        print(filename)\n        \n        if diff<0.01:\n            print(\"Normal\")\n        else: print(\"Distorted\")\n    \n\n        fig = plt.figure( figsize=(3, 3), dpi=180)\n        plt.imshow(spectrogram1, aspect=1)\n        cbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\n        plt.show()\n        \n        fig = plt.figure( figsize=(3, 3), dpi=180)\n        plt.imshow(spectrogram2, aspect=1)\n        cbar=plt.colorbar(label=\"input NN\", orientation=\"vertical\",shrink=.75)\n        plt.show()\n        \n\n    \n\n\n        \n\n\n    \n\n\n    ","metadata":{"execution":{"iopub.status.busy":"2022-11-25T12:40:11.028013Z","iopub.execute_input":"2022-11-25T12:40:11.028396Z","iopub.status.idle":"2022-11-25T12:40:26.155941Z","shell.execute_reply.started":"2022-11-25T12:40:11.028367Z","shell.execute_reply":"2022-11-25T12:40:26.154709Z"},"trusted":true},"execution_count":null,"outputs":[]}]}