{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🤝 Intro","metadata":{}},{"cell_type":"markdown","source":"Inference notebook: https://www.kaggle.com/chazzer/neural-net-starter-g2net-infer/","metadata":{}},{"cell_type":"markdown","source":"In 2015, gravitational waves were first detected via LIGO and Virgo.  \n~100 years after their theoretical postulation by Einstein in his General Theory of Relativity.  \nThe waves were detected thanks to a collision between to massive black holes.","metadata":{}},{"cell_type":"markdown","source":"![blackholes](https://thumbs.gfycat.com/AbsoluteWeeGreatargus-size_restricted.gif)","metadata":{}},{"cell_type":"markdown","source":"Today, we are trying to develop a machine learning model that can detect these waves, given some signal data.","metadata":{}},{"cell_type":"markdown","source":"Preface: this is a very rough cut and starter version of a potential convolutional neural network solution.  \nThe performance is very bad (extremely close to random guessing).  \nHowever, I hope you find some insights on how the model can be built.  \nPlease, if you have any feedback, let me know in the comments section!","metadata":{}},{"cell_type":"markdown","source":"The main limitation I've encoutered is on the memory side.  \nI've had to shrink down the signal via average pooling to stay within memory constraints.  \nI've also had to only select a sample of the trainin data, to stay within memory constraints.","metadata":{}},{"cell_type":"markdown","source":"# 🚚 Import","metadata":{}},{"cell_type":"code","source":"import h5py\nfrom glob import glob\n\nimport numpy as np\nimport pandas as pd\n\nimport gc  # garbage collection\nimport json  # saving jsons\n\n# neural network\nimport tensorflow as tf\nfrom tensorflow.keras import layers\n\nfrom cv2 import resize\n\n# train test split\nfrom sklearn.model_selection import train_test_split\n\n# visualization\nfrom tensorflow.keras.utils import plot_model\nimport plotly.express as px\nimport plotly.graph_objects as go","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:27.429028Z","iopub.execute_input":"2022-10-09T18:04:27.429764Z","iopub.status.idle":"2022-10-09T18:04:34.942281Z","shell.execute_reply.started":"2022-10-09T18:04:27.42968Z","shell.execute_reply":"2022-10-09T18:04:34.941313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"config = {\n    \"sample\": 1,\n    \"input_shape\": (36, 480, 2),\n    \"epochs\": 128,\n    \"batch_size\": 32,\n    \"learning_rate\": 0.000005\n}","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:34.944466Z","iopub.execute_input":"2022-10-09T18:04:34.944878Z","iopub.status.idle":"2022-10-09T18:04:34.953553Z","shell.execute_reply.started":"2022-10-09T18:04:34.944841Z","shell.execute_reply":"2022-10-09T18:04:34.948804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(42)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:34.955214Z","iopub.execute_input":"2022-10-09T18:04:34.955959Z","iopub.status.idle":"2022-10-09T18:04:34.996472Z","shell.execute_reply.started":"2022-10-09T18:04:34.955911Z","shell.execute_reply":"2022-10-09T18:04:34.995494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🏋️ Data ETL","metadata":{}},{"cell_type":"markdown","source":"## Main ETL Pipeline","metadata":{}},{"cell_type":"code","source":"labels_df = pd.read_csv('../input/g2net-detecting-continuous-gravitational-waves/train_labels.csv')","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:34.999073Z","iopub.execute_input":"2022-10-09T18:04:34.999526Z","iopub.status.idle":"2022-10-09T18:04:35.016655Z","shell.execute_reply.started":"2022-10-09T18:04:34.999489Z","shell.execute_reply":"2022-10-09T18:04:35.015847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.019462Z","iopub.execute_input":"2022-10-09T18:04:35.019724Z","iopub.status.idle":"2022-10-09T18:04:35.036796Z","shell.execute_reply.started":"2022-10-09T18:04:35.0197Z","shell.execute_reply":"2022-10-09T18:04:35.035811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since our training set is huge, let's get a small sample","metadata":{}},{"cell_type":"code","source":"train_hdf5s = glob('../input/g2net-detecting-continuous-gravitational-waves/train/*')\nnp.random.shuffle(train_hdf5s)\n\nfull_train_size = len(train_hdf5s)\ntrain_hdf5s = train_hdf5s[:int(full_train_size * config[\"sample\"])]","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.038312Z","iopub.execute_input":"2022-10-09T18:04:35.038921Z","iopub.status.idle":"2022-10-09T18:04:35.098461Z","shell.execute_reply.started":"2022-10-09T18:04:35.038885Z","shell.execute_reply":"2022-10-09T18:04:35.097222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_hdf5s))\ntrain_hdf5s[0]","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.102896Z","iopub.execute_input":"2022-10-09T18:04:35.10358Z","iopub.status.idle":"2022-10-09T18:04:35.115889Z","shell.execute_reply.started":"2022-10-09T18:04:35.103534Z","shell.execute_reply":"2022-10-09T18:04:35.114845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize the hdf5 structure","metadata":{}},{"cell_type":"code","source":"filename = train_hdf5s[0]\nwith h5py.File(filename, \"r\") as f:\n    for file_key in f.keys():\n        group = f[file_key]\n        print(group)\n        try:\n            for group_key in group.keys():\n                group2 = group[group_key]\n                print(f\"---->{group2}\")\n                for group_key2 in group2.keys():\n                        print(f\"--------->{group2[group_key2]}\")\n        except AttributeError:\n            pass","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.117202Z","iopub.execute_input":"2022-10-09T18:04:35.119635Z","iopub.status.idle":"2022-10-09T18:04:35.157464Z","shell.execute_reply.started":"2022-10-09T18:04:35.119593Z","shell.execute_reply":"2022-10-09T18:04:35.156276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's build our model only using the data from H1","metadata":{}},{"cell_type":"code","source":"def etl(filenames):\n    ret = []\n\n    norm_real, norm_imag = 0, 0        \n    for filename in filenames:\n        with h5py.File(filename, \"r\") as f:\n            ind, *_ = f.keys()\n            \n            # check if label is valid\n            label = labels_df.loc[labels_df[\"id\"] == ind][\"target\"].values[0]\n            if label == -1:\n                continue\n            \n            group1 = f[ind]\n            h1, _, freqs_key = group1.keys()\n\n            # still don't know how to handle frequencies in model\n            freq = np.array(group1[freqs_key])[:, None]\n\n            group2 = group1[h1]\n            signal_key, *_ = group2.keys()\n            signal = np.array(group2[signal_key])\n            \n            # we get the wave's real and imaginary parts\n            signal_real = np.real(signal)\n            signal_imag = np.imag(signal)\n            \n            signal_real = resize(signal_real, dsize=config[\"input_shape\"][:2])\n            signal_imag = resize(signal_imag, dsize=config[\"input_shape\"][:2])\n            \n            norm_real = max(norm_real, np.amax(signal_real), -np.amin(signal_real))\n            norm_imag = max(norm_imag, np.amax(signal_imag), -np.amin(signal_imag))\n            \n            signal = np.stack([signal_real, signal_imag], axis=-1)\n            \n        \n        ret.append(\n            {\n                \"id\": ind,\n                \"signal\": signal,\n                \"freq\": freq,\n                \"label\": label\n            }\n        )\n    \n    # normalizing signal data between -1 and 1\n    for row in ret:\n        row[\"signal\"][:, :, 0] /= norm_real\n        row[\"signal\"][:, :, 1] /= norm_imag\n        \n    return pd.DataFrame(ret), norm_real, norm_imag","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.161481Z","iopub.execute_input":"2022-10-09T18:04:35.164316Z","iopub.status.idle":"2022-10-09T18:04:35.185156Z","shell.execute_reply.started":"2022-10-09T18:04:35.164279Z","shell.execute_reply":"2022-10-09T18:04:35.183542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_data, norm_real, norm_imag = etl(train_hdf5s)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:04:35.198809Z","iopub.execute_input":"2022-10-09T18:04:35.201661Z","iopub.status.idle":"2022-10-09T18:05:51.463436Z","shell.execute_reply.started":"2022-10-09T18:04:35.201619Z","shell.execute_reply":"2022-10-09T18:05:51.462344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:05:51.465064Z","iopub.execute_input":"2022-10-09T18:05:51.465425Z","iopub.status.idle":"2022-10-09T18:06:00.430301Z","shell.execute_reply.started":"2022-10-09T18:05:51.465388Z","shell.execute_reply":"2022-10-09T18:06:00.429175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🪓 Train Test Split","metadata":{}},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(\n    np.stack(train_data['signal'].values), \n    train_data['label'].values,\n    train_size=0.2,\n    random_state=42\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:06:00.432343Z","iopub.execute_input":"2022-10-09T18:06:00.432729Z","iopub.status.idle":"2022-10-09T18:06:00.490136Z","shell.execute_reply.started":"2022-10-09T18:06:00.43269Z","shell.execute_reply":"2022-10-09T18:06:00.489154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🧱 Build Model","metadata":{}},{"cell_type":"code","source":"class G2NetModel(tf.keras.Model):\n    def __init__(self):\n        super(G2NetModel, self).__init__()\n\n        self.conv1 = layers.Conv2D(32, (3,3), activation='relu')\n        self.dropout1 = layers.SpatialDropout2D(0.5)\n        self.pool1 = layers.MaxPool2D((2, 2))\n        self.conv2 = layers.Conv2D(64, (3,3), activation='relu')\n        self.pool2 = layers.MaxPool2D((2, 2))\n        self.conv3 = layers.Conv2D(128, (3,3), activation='relu')\n        self.pool3 = layers.MaxPool2D((2, 2))\n        self.flatten = layers.Flatten()\n        \n        self.dense1 = layers.Dense(1028, activation='relu')\n        self.batchnorm1 = layers.BatchNormalization()\n        self.dense2 = layers.Dense(32, activation='relu')\n        self.batchnorm2 = layers.BatchNormalization()\n        self.classifier = layers.Dense(1, activation='sigmoid')\n    \n    def call(self, inputs, training=False):\n        \n        # signal\n        x = self.conv1(inputs[0])\n        if training:\n            x = self.dropout1(x, training=training)\n        x = self.pool1(x)\n        x = self.conv2(x)\n        x = self.pool2(x)\n        x = self.conv3(x)\n        x = self.pool3(x)\n        x = self.flatten(x)\n        \n        x = self.dense1(x)\n        x = self.batchnorm1(x)\n        x = self.dense2(x)\n        x = self.batchnorm2(x)\n        \n        return self.classifier(x)\n    \n    def build_graph(self):\n        # helper method for plotting architecture\n        inputs = [\n            layers.Input(config[\"input_shape\"])\n        ]\n        return tf.keras.Model(\n            inputs=inputs,\n            outputs=self.call(inputs)\n        )","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-10-09T18:06:00.491632Z","iopub.execute_input":"2022-10-09T18:06:00.49231Z","iopub.status.idle":"2022-10-09T18:06:00.50654Z","shell.execute_reply.started":"2022-10-09T18:06:00.492272Z","shell.execute_reply":"2022-10-09T18:06:00.505562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = G2NetModel()\nmodel(\n    [\n        layers.Input(config[\"input_shape\"])\n    ]\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-09T18:06:00.507873Z","iopub.execute_input":"2022-10-09T18:06:00.510406Z","iopub.status.idle":"2022-10-09T18:06:03.617938Z","shell.execute_reply.started":"2022-10-09T18:06:00.510304Z","shell.execute_reply":"2022-10-09T18:06:03.617077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=config[\"learning_rate\"]),\n    loss=tf.keras.losses.BinaryCrossentropy(),\n    metrics=['AUC']\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:06:03.622092Z","iopub.execute_input":"2022-10-09T18:06:03.624235Z","iopub.status.idle":"2022-10-09T18:06:03.843911Z","shell.execute_reply.started":"2022-10-09T18:06:03.624198Z","shell.execute_reply":"2022-10-09T18:06:03.842445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_model(\n    model.build_graph(),\n    to_file='model.png', \n    dpi=96,  \n    show_shapes=True, \n    show_layer_names=False,\n    expand_nested=False\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:06:03.845882Z","iopub.execute_input":"2022-10-09T18:06:03.846476Z","iopub.status.idle":"2022-10-09T18:06:04.920277Z","shell.execute_reply.started":"2022-10-09T18:06:03.846439Z","shell.execute_reply":"2022-10-09T18:06:04.919158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🧠 Train Model","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    [X_train],\n    y_train,\n    validation_data=([X_test], y_test),\n    batch_size=config[\"batch_size\"],\n    epochs=config[\"epochs\"]\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-09T18:06:04.921758Z","iopub.execute_input":"2022-10-09T18:06:04.923151Z","iopub.status.idle":"2022-10-09T18:08:28.022078Z","shell.execute_reply.started":"2022-10-09T18:06:04.923087Z","shell.execute_reply":"2022-10-09T18:08:28.02098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(\n    go.Scatter(\n        x=[i+1 for i in range(len(history.history['auc']))],\n        y=history.history['auc'],\n        name='AUC'\n    )\n)\nfig.add_trace(\n    go.Scatter(\n        x=[i+1 for i in range(len(history.history['auc']))],\n        y=history.history['val_auc'],\n        name='Validation AUC'\n    )\n)\n\nmax_y = max(history.history['auc'] + history.history['val_auc'])\nfig.update_yaxes(\n    range=[0, 1],\n)\n\nfig.update_layout(\n    title='AUC Evolution over Epochs',\n    height=600\n)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:08:28.027642Z","iopub.execute_input":"2022-10-09T18:08:28.027939Z","iopub.status.idle":"2022-10-09T18:08:28.143685Z","shell.execute_reply.started":"2022-10-09T18:08:28.02791Z","shell.execute_reply":"2022-10-09T18:08:28.142685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💾 Save Model","metadata":{}},{"cell_type":"code","source":"model.save_weights('g2net_starter_weights.h5')","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:08:28.144948Z","iopub.execute_input":"2022-10-09T18:08:28.145827Z","iopub.status.idle":"2022-10-09T18:08:28.320986Z","shell.execute_reply.started":"2022-10-09T18:08:28.145788Z","shell.execute_reply":"2022-10-09T18:08:28.319907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"extra_parameters = {\n    \"input_shape\": list(config[\"input_shape\"]),\n    \"norm_real\": str(norm_real),\n    \"norm_imag\": str(norm_imag)\n}","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:09:58.529671Z","iopub.execute_input":"2022-10-09T18:09:58.530043Z","iopub.status.idle":"2022-10-09T18:09:58.535846Z","shell.execute_reply.started":"2022-10-09T18:09:58.53001Z","shell.execute_reply":"2022-10-09T18:09:58.53467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"extra_parameters","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:13:56.195509Z","iopub.execute_input":"2022-10-09T18:13:56.196178Z","iopub.status.idle":"2022-10-09T18:13:56.206051Z","shell.execute_reply.started":"2022-10-09T18:13:56.196014Z","shell.execute_reply":"2022-10-09T18:13:56.204037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(\"extra_parameters.json\", \"w\") as f:\n    json.dump(extra_parameters, f, indent=4)","metadata":{"execution":{"iopub.status.busy":"2022-10-09T18:13:50.649546Z","iopub.execute_input":"2022-10-09T18:13:50.64993Z","iopub.status.idle":"2022-10-09T18:13:50.655807Z","shell.execute_reply.started":"2022-10-09T18:13:50.649898Z","shell.execute_reply":"2022-10-09T18:13:50.654531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🙇 Conclusion","metadata":{}},{"cell_type":"markdown","source":"Check out this notebook for inference and submission: https://www.kaggle.com/chazzer/neural-net-starter-g2net-infer/","metadata":{}},{"cell_type":"markdown","source":"Thanks for following along.  \nAs I've said at the beginning of the notebook, this is a very rough version of the model.  \nAll feedback is deeply appreciated.  \nTake care!","metadata":{}}]}