{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Convolutional Autoencoder for SFTs\nAutoencoders can do lots of great things, from denoising images to extracting key features. It's not yet clear how they will be useful for this competition, but I defenitely could use a bit of denoising on the SFTs we've been given, and reducing the input size of the data is interesting as well. If you never heard of autoencoders, the idea is to make a model learn to predict... its inputs! I know, doesn't sound super clever like this, but first it means that our label is actually our input, so really we're doing unsupervised learning. And if your model shrinks the input before expanding it, you're actually compressing information, and hopefully extracting the most important features in doing so. \n\n## What will we be doing in this notebook?\nI want to build a convolutional autoencoder. We'll start easy with an autoencoder which latent dimension (the central bit) is the same size as the input. Then we will reduce the size of the latent dimension and see how it impacts the quality of the autoencoder. We will use synthetic square GW signals with pretty high signal-to-noise (SNR) ratio (I used a depth of 1 to generate them). Enough talk, let's load the data!","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n!pip install git+https://github.com/PyFstat/PyFstat@python37","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T09:53:42.795537Z","iopub.execute_input":"2022-10-25T09:53:42.795948Z","iopub.status.idle":"2022-10-25T09:53:57.038809Z","shell.execute_reply.started":"2022-10-25T09:53:42.795915Z","shell.execute_reply":"2022-10-25T09:53:57.037004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%matplotlib inline\nimport os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom pyfstat.utils import get_sft_as_arrays\nimport tensorflow as tf","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:53:57.041192Z","iopub.execute_input":"2022-10-25T09:53:57.041577Z","iopub.status.idle":"2022-10-25T09:53:57.056509Z","shell.execute_reply.started":"2022-10-25T09:53:57.041541Z","shell.execute_reply":"2022-10-25T09:53:57.055326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# function to read synthetic GW sfts, expanded from https://www.kaggle.com/code/chazzer/neural-net-starter-g2net-train\ndef etl_synthetic(filenames, labels, nophase=False, ini_dim1=1920, max_dim1=360, average=False, norm=1e20, offset=0.0):\n    ret = []\n    ids, signals, freqs, label = [], [], [], []\n\n    i = 0\n    for filename in filenames:\n        frequency, timestamps, sft = get_sft_as_arrays(filename)\n\n        freq = np.array(frequency)[1:]\n        sft = np.array(sft['H1'])[1:,:]\n        signal = np.stack([np.absolute(sft*norm)-offset, np.angle(sft)/np.pi], axis=-1)\n\n        if(average):\n            avg_length = ini_dim1//max_dim1\n            ids.append(str(i*10)+'_avg'+str(avg_length))\n            freqs.append(freq)\n            sft = sft[:,-ini_dim1:]\n            sft_avg = sft[:,::avg_length]\n            for k in range(max_dim1):\n                sft_avg[:,k] = np.mean(sft[:,k*avg_length:(k+1)*avg_length],axis=1)\n            signal = np.stack([np.absolute(sft_avg*norm)-offset, np.angle(sft_avg)/np.pi], axis=-1)\n            if(nophase): signal = np.stack([np.absolute(sft_avg*norm)-offset], axis=-1)\n            signals.append(signal)\n            label.append(labels[i])\n        else:\n            for k in range(ini_dim1//max_dim1):\n                ids.append(i*10+k+1)\n                freqs.append(freq)\n                sft_square = sft[:,k*max_dim1:(k+1)*max_dim1]\n                signal = np.stack([np.absolute(sft_square*1e21)-0.5, np.angle(sft_square)/np.pi], axis=-1)\n                if(nophase): signal = np.stack([np.absolute(sft_square*norm)-offset], axis=-1)\n                signals.append(signal)\n                label.append(labels[i])\n        i+=1\n\n    ret = {\n        \"id\": np.array(ids),\n        \"freq\": np.array(freqs),\n        \"signal\": np.stack(signals, axis=0),\n        \"label\" : np.array(label)\n    }\n\n    return ret","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:53:57.057904Z","iopub.execute_input":"2022-10-25T09:53:57.05824Z","iopub.status.idle":"2022-10-25T09:53:57.074483Z","shell.execute_reply.started":"2022-10-25T09:53:57.05821Z","shell.execute_reply":"2022-10-25T09:53:57.073508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import data\nlabels = [] \nfilenames = []\nfor i in range(100):\n    filenames.append(\"../input/g2net-synthetic-gw-sfts/PyFstat_example_data_ensemble_square_D1/Signal_\"+str(i)+\"/H-360_H1_1800SFT_Signal_\"+str(i)+\"-1238166018-648000.sft\")\n    labels.append(1) # we won't use the labels in this example \n    \n# here we discard the phase and only keep the amplitude of the SFTs, which we mupltiply by 1e20 \nall_dataD1 = etl_synthetic(filenames, labels, nophase=True, ini_dim1=360, average=False, norm=1e20, offset=0.0)\n\n# verify that our training data (80 sfts out of 100) has been correctly normalized (values between 0 and 1)\nprint(\"Minimum value of training dataset: \"+str(np.min(all_dataD1['signal'][:80])))\nprint(\"Maximum value of training dataset: \"+str(np.max(all_dataD1['signal'][:80])))\n","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:53:57.077767Z","iopub.execute_input":"2022-10-25T09:53:57.078233Z","iopub.status.idle":"2022-10-25T09:54:00.1701Z","shell.execute_reply.started":"2022-10-25T09:53:57.078188Z","shell.execute_reply":"2022-10-25T09:54:00.168716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# separate data into train, test and evaluation (80/10/10)\nx_train = all_dataD1['signal'][:80]\nx_test = all_dataD1['signal'][80:90]\nx_eval = all_dataD1['signal'][90:100]","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:54:00.171599Z","iopub.execute_input":"2022-10-25T09:54:00.172684Z","iopub.status.idle":"2022-10-25T09:54:00.177926Z","shell.execute_reply.started":"2022-10-25T09:54:00.172646Z","shell.execute_reply":"2022-10-25T09:54:00.176593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Where are we know?\nWe just loaded the data, taking only the amplitude of the SFTs. We normalized it to have values between 0 and 1 and we did split the dataset into three parts: training (80%), testing (10%) and evaluating (10%). Training data is the one with which the model learns (namely computes the loss and backpropagates), testing data is used to choose the best model when connvergence is not smooth, and evaluating data has never been seen by the model during it's training phase or during model picking, hence it's a good candidate to evaluate our model.\n\n## What's next?\nI think we don't have a model yet, maybe it's time to build one. Input size is 360x360x1, let's try to build a three layers convolutional encoder with latent dimension 45x45x64 (you can check, it's egal to input dimension). So what we're aiming at is this kind of pipeline:\n- 360 x 360 x  1 - input \n- 180 x 180 x  4 \n-  90 x  90 x 16 \n-  45 x  45 x 64 - latent space, end of the encoder\n-  90 x  90 x 16\n- 180 x 180 x  4\n- 360 x 360 x  1 - output, which should be equal to input\n ","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import layers\n# create a class for our convolutional autoencoder\nclass G2Net_AE_Conv2D(tf.keras.Model):\n    def __init__(self, nlayers=3, nfilters_start=4, ks=3, act_fun=\"tanh\", latent_dim=64):\n        super(G2Net_AE_Conv2D, self).__init__()\n        self.encoder = tf.keras.Sequential()\n        self.encoder.add(layers.Input(shape=(360, 360, 1)))\n        for i in range(nlayers):\n            self.encoder.add(layers.Conv2D(4**(i+1), kernel_size=ks, activation=act_fun, padding='same', strides=2))\n\n        self.decoder = tf.keras.Sequential()\n        for i in range(nlayers-1):\n            self.decoder.add(layers.Conv2DTranspose(4**(nlayers-1-i), kernel_size=ks, activation=act_fun, padding='same', strides=2))\n        self.decoder.add(layers.Conv2DTranspose(1, kernel_size=ks, activation='tanh', padding='same', strides=2))\n\n    def call(self, x):\n        encoded = self.encoder(x)\n        decoded = self.decoder(encoded)\n        return decoded","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:54:00.180043Z","iopub.execute_input":"2022-10-25T09:54:00.180365Z","iopub.status.idle":"2022-10-25T09:54:00.191902Z","shell.execute_reply.started":"2022-10-25T09:54:00.180336Z","shell.execute_reply":"2022-10-25T09:54:00.190756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# time to build our model\nmodel = G2Net_AE_Conv2D(nlayers=3)\nmodel.decoder.build(model.encoder.output_shape)\nmodel.encoder.summary()\nmodel.decoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:54:00.193282Z","iopub.execute_input":"2022-10-25T09:54:00.193941Z","iopub.status.idle":"2022-10-25T09:54:00.286561Z","shell.execute_reply.started":"2022-10-25T09:54:00.193907Z","shell.execute_reply":"2022-10-25T09:54:00.285622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checkpoint callback to save best model\nid = \"AE_Conv2D_3L_Ldim64_tanh\"\ncheckpoint_filepath = '/kaggle/working/checkpoint_'+id\ncheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n        filepath=checkpoint_filepath,\n        save_weights_only=True,\n        monitor='val_loss',\n        mode='min',\n        save_best_only=True,\n        verbose=1)\n\n# history callback to save loss during training\nhistory_filepath = '/kaggle/working/history_'+id+'.csv'\nhistory_callback = tf.keras.callbacks.History()\n\n# time to train\nmodel.compile(optimizer='adam', loss=tf.keras.losses.MeanSquaredError())\n\nhistory = model.fit(x_train, x_train,\n                    epochs=128,\n                    verbose=2,\n                    validation_data=(x_test, x_test),\n                    callbacks = [history_callback, checkpoint_callback],\n                    batch_size=10)\n\n# save history\ndf = pd.DataFrame.from_dict(history.history)\ndf.to_csv(history_filepath, encoding='utf-8', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T09:54:00.287816Z","iopub.execute_input":"2022-10-25T09:54:00.288286Z","iopub.status.idle":"2022-10-25T09:55:40.470872Z","shell.execute_reply.started":"2022-10-25T09:54:00.288256Z","shell.execute_reply":"2022-10-25T09:55:40.469719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ok nice, but what does it look like?\nGood question, here we take the ten samples from our evaluation set, we plot the input above and the output below.","metadata":{}},{"cell_type":"code","source":"# load best model\nmodel.load_weights(\"./checkpoint_AE_Conv2D_3L_Ldim64_tanh\")\n\n# generate outputs through the autoencoder\nencoded_sfts = model.encoder(x_eval).numpy()\ndecoded_sfts = model.decoder(encoded_sfts).numpy()\n\n# time to plot\nn = 10\nplt.figure(figsize=(30, 6))\nfor i in range(n):\n  # display original\n  ax = plt.subplot(2, n, i + 1)\n  plt.imshow(x_eval[i], cmap='jet')\n  #plt.gray()\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\n\n  # display reconstruction\n  ax = plt.subplot(2, n, i + 1 + n)\n  plt.imshow(decoded_sfts[i], cmap='jet')\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T09:55:40.473507Z","iopub.execute_input":"2022-10-25T09:55:40.473858Z","iopub.status.idle":"2022-10-25T09:55:42.368276Z","shell.execute_reply.started":"2022-10-25T09:55:40.473826Z","shell.execute_reply":"2022-10-25T09:55:42.367074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## It does not look great, can we improve this?\nYes, we can. The convergence is pretty slow because the model is (relatively) deep and the number of trainable parameters is pretty high, especially in the last layer of the encoder (and the first of the decoder). To overcome this issue we will use **transfer learning**. The idea is to reuse the weights of a previous model. Let's train a model with one layer only, use it to train a model with two layers, and use this last one to train a 3 layers model.","metadata":{}},{"cell_type":"code","source":"# time to build our model\nmodel_1L = G2Net_AE_Conv2D(nlayers=1)\nmodel_1L.decoder.build(model_1L.encoder.output_shape)\nmodel_1L.encoder.summary()\nmodel_1L.decoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:07:50.347221Z","iopub.execute_input":"2022-10-25T10:07:50.347955Z","iopub.status.idle":"2022-10-25T10:07:50.387834Z","shell.execute_reply.started":"2022-10-25T10:07:50.347914Z","shell.execute_reply":"2022-10-25T10:07:50.386774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checkpoint callback to save best model\nid = \"AE_Conv2D_1L_Ldim4_tanh\"\ncheckpoint_filepath = '/kaggle/working/checkpoint_'+id\ncheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n        filepath=checkpoint_filepath,\n        save_weights_only=True,\n        monitor='val_loss',\n        mode='min',\n        save_best_only=True,\n        verbose=1)\n\n# history callback to save loss during training\nhistory_filepath = '/kaggle/working/history_'+id+'.csv'\nhistory_callback = tf.keras.callbacks.History()\n\n# time to train\nmodel_1L.compile(optimizer='adam', loss=tf.keras.losses.MeanSquaredError())\n\nhistory = model_1L.fit(x_train, x_train,\n                       epochs=128,\n                       verbose=2,\n                       validation_data=(x_test, x_test),\n                       callbacks = [history_callback, checkpoint_callback],\n                       batch_size=10)\n\n# save history\ndf = pd.DataFrame.from_dict(history.history)\ndf.to_csv(history_filepath, encoding='utf-8', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T10:07:54.631674Z","iopub.execute_input":"2022-10-25T10:07:54.632069Z","iopub.status.idle":"2022-10-25T10:09:16.984411Z","shell.execute_reply.started":"2022-10-25T10:07:54.632038Z","shell.execute_reply":"2022-10-25T10:09:16.98328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now build a 2 layers model using the weights of the first one \nmodel_2L = G2Net_AE_Conv2D(nlayers=2)\nmodel_2L.decoder.build(model_2L.encoder.output_shape)\n\n# Transfer the weights and make the layers non trainable\nmodel_2L.encoder.layers[0].set_weights(model_1L.encoder.layers[0].get_weights())\nmodel_2L.encoder.layers[0].trainable = False\nmodel_2L.decoder.layers[-1].set_weights(model_1L.decoder.layers[-1].get_weights())\nmodel_2L.decoder.layers[-1].trainable = False\n\n# check the number of trainable parameters\nmodel_2L.encoder.summary()\nmodel_2L.decoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:09:17.052648Z","iopub.execute_input":"2022-10-25T10:09:17.053122Z","iopub.status.idle":"2022-10-25T10:09:17.117822Z","shell.execute_reply.started":"2022-10-25T10:09:17.053079Z","shell.execute_reply":"2022-10-25T10:09:17.116902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checkpoint callback to save best model\nid = \"AE_Conv2D_2L_Ldim16_tanh\"\ncheckpoint_filepath = '/kaggle/working/checkpoint_'+id\ncheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n        filepath=checkpoint_filepath,\n        save_weights_only=True,\n        monitor='val_loss',\n        mode='min',\n        save_best_only=True,\n        verbose=1)\n\n# history callback to save loss during training\nhistory_filepath = '/kaggle/working/history_'+id+'.csv'\nhistory_callback = tf.keras.callbacks.History()\n\n# time to train\nmodel_2L.compile(optimizer='adam', loss=tf.keras.losses.MeanSquaredError())\n\nhistory = model_2L.fit(x_train, x_train,\n                       epochs=128,\n                       verbose=2,\n                       validation_data=(x_test, x_test),\n                       callbacks = [history_callback, checkpoint_callback],\n                       batch_size=10)\n\n# save history\ndf = pd.DataFrame.from_dict(history.history)\ndf.to_csv(history_filepath, encoding='utf-8', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T10:09:49.119884Z","iopub.execute_input":"2022-10-25T10:09:49.120872Z","iopub.status.idle":"2022-10-25T10:10:31.225999Z","shell.execute_reply.started":"2022-10-25T10:09:49.120832Z","shell.execute_reply":"2022-10-25T10:10:31.224781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now build a 3 layers model using the weights of the second one \nmodel_3L = G2Net_AE_Conv2D(nlayers=3)\nmodel_3L.decoder.build(model_3L.encoder.output_shape)\n\n# Transfer the weights and make the layers non trainable\nmodel_3L.encoder.layers[0].set_weights(model_2L.encoder.layers[0].get_weights())\nmodel_3L.encoder.layers[0].trainable = False\nmodel_3L.encoder.layers[1].set_weights(model_2L.encoder.layers[1].get_weights())\nmodel_3L.encoder.layers[1].trainable = False\nmodel_3L.decoder.layers[-2].set_weights(model_2L.decoder.layers[-2].get_weights())\nmodel_3L.decoder.layers[-2].trainable = False\nmodel_3L.decoder.layers[-1].set_weights(model_2L.decoder.layers[-1].get_weights())\nmodel_3L.decoder.layers[-1].trainable = False\n\n# check the number of trainable parameters\nmodel_3L.encoder.summary()\nmodel_3L.decoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:10:35.518268Z","iopub.execute_input":"2022-10-25T10:10:35.5187Z","iopub.status.idle":"2022-10-25T10:10:35.608383Z","shell.execute_reply.started":"2022-10-25T10:10:35.518645Z","shell.execute_reply":"2022-10-25T10:10:35.606942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checkpoint callback to save best model\nid = \"AE_Conv2D_3L_Ldim64_tanh_transfer\"\ncheckpoint_filepath = '/kaggle/working/checkpoint_'+id\ncheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n        filepath=checkpoint_filepath,\n        save_weights_only=True,\n        monitor='val_loss',\n        mode='min',\n        save_best_only=True,\n        verbose=1)\n\n# history callback to save loss during training\nhistory_filepath = '/kaggle/working/history_'+id+'.csv'\nhistory_callback = tf.keras.callbacks.History()\n\n# time to train\nmodel_3L.compile(optimizer='adam', loss=tf.keras.losses.MeanSquaredError())\n\nhistory = model_3L.fit(x_train, x_train,\n                       epochs=128,\n                       verbose=2,\n                       validation_data=(x_test, x_test),\n                       callbacks = [history_callback, checkpoint_callback],\n                       batch_size=10)\n\n# save history\ndf = pd.DataFrame.from_dict(history.history)\ndf.to_csv(history_filepath, encoding='utf-8', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T10:10:40.612515Z","iopub.execute_input":"2022-10-25T10:10:40.612927Z","iopub.status.idle":"2022-10-25T10:12:03.110412Z","shell.execute_reply.started":"2022-10-25T10:10:40.612896Z","shell.execute_reply":"2022-10-25T10:12:03.109356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load best model\nmodel_3L.load_weights(\"./checkpoint_AE_Conv2D_3L_Ldim64_tanh_transfer\")\n\n# generate outputs through the autoencoder\nencoded_sfts = model_3L.encoder(x_eval).numpy()\ndecoded_sfts = model_3L.decoder(encoded_sfts).numpy()\n\n# time to plt\nn = 10\nplt.figure(figsize=(30, 6))\nfor i in range(n):\n  # display original\n  ax = plt.subplot(2, n, i + 1)\n  plt.imshow(x_eval[i], cmap='jet')\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\n\n  # display reconstruction\n  ax = plt.subplot(2, n, i + 1 + n)\n  plt.imshow(decoded_sfts[i], cmap='jet')\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:12:12.789687Z","iopub.execute_input":"2022-10-25T10:12:12.790127Z","iopub.status.idle":"2022-10-25T10:12:14.263873Z","shell.execute_reply.started":"2022-10-25T10:12:12.790095Z","shell.execute_reply":"2022-10-25T10:12:14.26276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## WOW!\nI know, way better than before, right? Let's see how the loss compares between learning from scratch and using transfer learning","metadata":{}},{"cell_type":"code","source":"scratch = np.genfromtxt('./history_AE_Conv2D_3L_Ldim64_tanh.csv', skip_header=1, delimiter=\",\")\ntransfL = np.genfromtxt('./history_AE_Conv2D_3L_Ldim64_tanh_transfer.csv', skip_header=1, delimiter=\",\")\n\nplt.plot(scratch[:,1], label=\"from scratch\")\nplt.plot(transfL[:,1], label=\"transfer learning\")\nplt.yscale('log')\nplt.legend(loc=\"best\")\nplt.ylabel(\"validation loss [-]\")\nplt.xlabel(\"epochs [-]\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:14:41.24881Z","iopub.execute_input":"2022-10-25T10:14:41.249176Z","iopub.status.idle":"2022-10-25T10:15:06.919188Z","shell.execute_reply.started":"2022-10-25T10:14:41.249146Z","shell.execute_reply":"2022-10-25T10:15:06.91813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ok, not bad, is that all?\nNo it's not, now the fun part starts. We now want to reduce the size of the latent space. Let's start by dividing it by two, which is a compression factor of 50%. To do this we start from the previous model and we add a 45x45x32 convolutional layer (and the equivalent deconvolutional layer in the decoder).","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import layers\n# let's create a new class\nclass G2Net_AE_Conv2D_cpr(tf.keras.Model):\n    def __init__(self, nlayers=3, nfilters_start=4, ks=3, act_fun=\"tanh\", latent_dim=32):\n        super(G2Net_AE_Conv2D_cpr, self).__init__()\n        self.encoder = tf.keras.Sequential()\n        self.encoder.add(layers.Input(shape=(360, 360, 1)))\n        for i in range(nlayers):\n            self.encoder.add(layers.Conv2D(4**(i+1), kernel_size=ks, activation=act_fun, padding='same', strides=2))\n        self.encoder.add(layers.Conv2D(latent_dim, kernel_size=1, activation=act_fun, padding='same', strides=1))\n        \n        self.decoder = tf.keras.Sequential()\n        self.decoder.add(layers.Conv2DTranspose(4**nlayers, kernel_size=1, activation=act_fun, padding='same', strides=1))\n        for i in range(nlayers-1):\n            self.decoder.add(layers.Conv2DTranspose(4**(2-i), kernel_size=ks, activation=act_fun, padding='same', strides=2))\n        self.decoder.add(layers.Conv2DTranspose(1, kernel_size=ks, activation='tanh', padding='same', strides=2))\n\n    def call(self, x):\n        encoded = self.encoder(x)\n        decoded = self.decoder(encoded)\n        return decoded\n","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:15:56.258829Z","iopub.execute_input":"2022-10-25T10:15:56.259244Z","iopub.status.idle":"2022-10-25T10:15:56.271736Z","shell.execute_reply.started":"2022-10-25T10:15:56.259209Z","shell.execute_reply":"2022-10-25T10:15:56.270578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# time to build our model\nmodel_cpr = G2Net_AE_Conv2D_cpr(nlayers=3, latent_dim=32)\nmodel_cpr.decoder.build(model_cpr.encoder.output_shape)\n\n# Transfer the weights and make the layers non trainable\nmodel_cpr.encoder.layers[0].set_weights(model_3L.encoder.layers[0].get_weights())\nmodel_cpr.encoder.layers[0].trainable = False\nmodel_cpr.encoder.layers[1].set_weights(model_3L.encoder.layers[1].get_weights())\nmodel_cpr.encoder.layers[1].trainable = False\nmodel_cpr.encoder.layers[2].set_weights(model_3L.encoder.layers[2].get_weights())\nmodel_cpr.encoder.layers[2].trainable = False\nmodel_cpr.decoder.layers[-3].set_weights(model_3L.decoder.layers[-3].get_weights())\nmodel_cpr.decoder.layers[-3].trainable = False\nmodel_cpr.decoder.layers[-2].set_weights(model_3L.decoder.layers[-2].get_weights())\nmodel_cpr.decoder.layers[-2].trainable = False\nmodel_cpr.decoder.layers[-1].set_weights(model_3L.decoder.layers[-1].get_weights())\nmodel_cpr.decoder.layers[-1].trainable = False\n\n# check the number of trainable parameters\nmodel_cpr.encoder.summary()\nmodel_cpr.decoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:16:00.629743Z","iopub.execute_input":"2022-10-25T10:16:00.630868Z","iopub.status.idle":"2022-10-25T10:16:00.742481Z","shell.execute_reply.started":"2022-10-25T10:16:00.630695Z","shell.execute_reply":"2022-10-25T10:16:00.741492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# checkpoint callback to save best model\nid = \"AE_Conv2D_3L_Ldim32_tanh_transfer\"\ncheckpoint_filepath = '/kaggle/working/checkpoint_'+id\ncheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n        filepath=checkpoint_filepath,\n        save_weights_only=True,\n        monitor='val_loss',\n        mode='min',\n        save_best_only=True,\n        verbose=1)\n\n# history callback to save loss during training\nhistory_filepath = '/kaggle/working/history_'+id+'.csv'\nhistory_callback = tf.keras.callbacks.History()\n\n# time to train\nmodel_cpr.compile(optimizer='adam', loss=tf.keras.losses.MeanSquaredError())\n\nhistory = model_cpr.fit(x_train, x_train,\n                       epochs=128,\n                       verbose=2,\n                       validation_data=(x_test, x_test),\n                       callbacks = [history_callback, checkpoint_callback],\n                       batch_size=10)\n\n# save history\ndf = pd.DataFrame.from_dict(history.history)\ndf.to_csv(history_filepath, encoding='utf-8', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-10-25T10:16:09.290552Z","iopub.execute_input":"2022-10-25T10:16:09.290956Z","iopub.status.idle":"2022-10-25T10:17:00.189332Z","shell.execute_reply.started":"2022-10-25T10:16:09.290924Z","shell.execute_reply":"2022-10-25T10:17:00.187953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load best model\nmodel_cpr.load_weights(\"./checkpoint_AE_Conv2D_3L_Ldim32_tanh_transfer\")\n\n# generate outputs through the autoencoder\nencoded_sfts = model_cpr.encoder(x_eval).numpy()\ndecoded_sfts = model_cpr.decoder(encoded_sfts).numpy()\n\n# time to plot\nn = 10\nplt.figure(figsize=(30, 6))\nfor i in range(n):\n  # display original\n  ax = plt.subplot(2, n, i + 1)\n  plt.imshow(x_eval[i], cmap='jet')\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\n\n  # display reconstruction\n  ax = plt.subplot(2, n, i + 1 + n)\n  plt.imshow(decoded_sfts[i], cmap='jet')\n  ax.get_xaxis().set_visible(False)\n  ax.get_yaxis().set_visible(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T10:17:05.835597Z","iopub.execute_input":"2022-10-25T10:17:05.836015Z","iopub.status.idle":"2022-10-25T10:17:07.354521Z","shell.execute_reply.started":"2022-10-25T10:17:05.83598Z","shell.execute_reply":"2022-10-25T10:17:07.353663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Should I be happy with his?\nWell, we definitely lost accuracy but the GW signal itself is still present, which means that this information is kept somewehere in the latent space (which is what we wanted to start with). The model can probably reach better convergence (higher number of epochs, different optimizer or learning rate, ...) which could slightly improve the result. \n","metadata":{}}]}