Guys, I am trying to learn more about Seq2Seq, but I am struggling with several points. Here are some questions:
\n1 - I have some knowledge of Autoencoders. Normally, in Autoencoders, the input of the decoder is the output of the encoder (called the latent representation).
\nBut in Seq2Seq, I did some research on Google (and ChatGPT), and it seems that the decoder is usually fed with the previous state. Is this due to the nature of RNN/LSTM? Like in the code below:
\n :\n def :\n super(Seq2Seq, self).\n\n self.encoder = nn.\n self.decoder = nn.\n self.fc = nn.\n\n def forward(self, encoder_input, decoder_input):\n # Codificador\n _, (hidden, cell) = self.encoder(encoder_input)\n\n # Decodificador\n decoder_output, _ = self.decoder(decoder_input, (hidden, cell))\n\n # Previsão\n predictions = self.fc(decoder_output)\n return predictions\n\n\n epoch range(num_epochs):\n encoder_input, target train_loader:\n # Inicializando o estado oculto\n decoder_input = torch.zeros(batch_size, target.size(), output_dim)\n\n # Forward pass\n outputs = model(encoder_input, decoder_input)\nIt makes no sense to me to always start the decoder input with 0 (the code above was built by ChatGPT).
\n2 - Strictly speaking about the shape of the data that will be fed to the model, should the correct shape be (batch_size, 60, 14) or (batch_size, 14, 60) (and (batch_size, 60, 23) for the outputs)? The first one looks correct to me, but I want to confirm.
\n3 - Is Unet a type of Seq2Seq? I asked ChatGPT and it said that it is not Seq2Seq.
\n4 - Can working with bottleneck architectures cause problems with the decoder part due to the \"reconstruction\" of the data? Can using techniques like output_padding to match the shapes cause problems?
\nIf someone can share a simple Seq2Seq model for learning purposes, it would be really helpful.
", "messages": [ { "id": 2843265, "postDate": "2024-05-29T13:38:59.100Z", "content": "I just made this baseline seq2seq notebook public: https://www.kaggle.com/code/abiolatti/keras-baseline-seq2seq
", "rawMarkdown": "I just made this baseline seq2seq notebook public: https://www.kaggle.com/code/abiolatti/keras-baseline-seq2seq", "votes": 5, "replies": [ { "id": 2843370, "postDate": "2024-05-29T14:35:29.083Z", "content": "Good job. A simple model with a lot of room for improvement, yet with a good score. I'm sure it will help a lot of people.
", "rawMarkdown": "Good job. A simple model with a lot of room for improvement, yet with a good score. I'm sure it will help a lot of people.", "votes": 2 }, { "id": 2845503, "postDate": "2024-05-30T15:17:27.547Z", "content": "THX, really helpful. I am learning a lot from you.
\nI'm studying your code, where does this architecture come from?
\nIsn't this a complete seq2seq in the sense, that you don't have an explicit encoder-decoder architecture?
\nAnd you don't have the input shape (batch, seq_len, features).
You're confusing seq2seq (a general term) with causal attention (encoder-decoder).
\nThe notebook above implements seq2seq with a 1d cnn.
The x_to_seq function reshape the \"flat\" input in a sequence of shape (Batch, 60, 25). In that case we don't have time/causal order in the sequence so we don't need to mask data from the future like is possible to do with classical encoder/decoder architecture, so we can use convolutions, (unmasked) transformers and/or bidirectional LSTM (or maybe some other fancy idea)
Amedeo, I studied your notebook and our team is improving ours based on yours.
\nI have one more question, how do you initially find this network architecture? Trial and error? Based on some literature network? Domain knowledge?
", "rawMarkdown": "Amedeo, I studied your notebook and our team is improving ours based on yours.\n\nI have one more question, how do you initially find this network architecture? Trial and error? Based on some literature network? Domain knowledge?" } ] }, { "id": 2841765, "postDate": "2024-05-28T17:26:31.863Z", "content": "'feed previous state' is when you use encoder-decoder. This is for problems where the output length is independent of the input length. For example, your input is a sentence at length X, and the output should be a translation of length Y.
\nBut here, the output should be the same length as the input, i.e., 60. So, there is no need for an encoder-decoder scheme, only an encoder. Hence, there is also no need to 'feed the previous state.' Unless you use an RNN/LSTM, of course, which works on the 'previous state' by design, but if your encoder is transformer or 1d unet etc. you just map 60 to 60. This is the 'seq to seq'. For example, consider the simplest model of 1 layer of 1d conv with 'same' paddings. Well, it might be challenging to build a good model of the sort if you don't have previous experience, there are some tricks to the trade haha. Since no one shared a good baseline, this competition is really not easy to get into.
Guys, I am trying to learn more about Seq2Seq, but I am struggling with several points. Here are some questions:
\n1 - I have some knowledge of Autoencoders. Normally, in Autoencoders, the input of the decoder is the output of the encoder (called the latent representation).
\nBut in Seq2Seq, I did some research on Google (and ChatGPT), and it seems that the decoder is usually fed with the previous state. Is this due to the nature of RNN/LSTM? Like in the code below:
\n :\n def :\n super(Seq2Seq, self).\n\n self.encoder = nn.\n self.decoder = nn.\n self.fc = nn.\n\n def forward(self, encoder_input, decoder_input):\n # Codificador\n _, (hidden, cell) = self.encoder(encoder_input)\n\n # Decodificador\n decoder_output, _ = self.decoder(decoder_input, (hidden, cell))\n\n # Previsão\n predictions = self.fc(decoder_output)\n return predictions\n\n\n epoch range(num_epochs):\n encoder_input, target train_loader:\n # Inicializando o estado oculto\n decoder_input = torch.zeros(batch_size, target.size(), output_dim)\n\n # Forward pass\n outputs = model(encoder_input, decoder_input)\nIt makes no sense to me to always start the decoder input with 0 (the code above was built by ChatGPT).
\n2 - Strictly speaking about the shape of the data that will be fed to the model, should the correct shape be (batch_size, 60, 14) or (batch_size, 14, 60) (and (batch_size, 60, 23) for the outputs)? The first one looks correct to me, but I want to confirm.
\n3 - Is Unet a type of Seq2Seq? I asked ChatGPT and it said that it is not Seq2Seq.
\n4 - Can working with bottleneck architectures cause problems with the decoder part due to the \"reconstruction\" of the data? Can using techniques like output_padding to match the shapes cause problems?
\nIf someone can share a simple Seq2Seq model for learning purposes, it would be really helpful.
", "rawMarkdown": "Guys, I am trying to learn more about Seq2Seq, but I am struggling with several points. Here are some questions:\n\n1 - I have some knowledge of Autoencoders. Normally, in Autoencoders, the input of the decoder is the output of the encoder (called the latent representation).\n\nBut in Seq2Seq, I did some research on Google (and ChatGPT), and it seems that the decoder is usually fed with the previous state. Is this due to the nature of RNN/LSTM? Like in the code below:\n\n```\nclass Seq2Seq(nn.Module):\n def __init__(self, input_dim, output_dim, hidden_dim, num_layers):\n super(Seq2Seq, self).__init__()\n \n self.encoder = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)\n self.decoder = nn.LSTM(output_dim, hidden_dim, num_layers, batch_first=True)\n self.fc = nn.Linear(hidden_dim, output_dim)\n \n def forward(self, encoder_input, decoder_input):\n # Codificador\n _, (hidden, cell) = self.encoder(encoder_input)\n \n # Decodificador\n decoder_output, _ = self.decoder(decoder_input, (hidden, cell))\n \n # Previsão\n predictions = self.fc(decoder_output)\n return predictions\n\n\nfor epoch in range(num_epochs):\n for encoder_input, target in train_loader:\n # Inicializando o estado oculto\n decoder_input = torch.zeros(batch_size, target.size(1), output_dim)\n \n # Forward pass\n outputs = model(encoder_input, decoder_input)\n```\n\nIt makes no sense to me to always start the decoder input with 0 (the code above was built by ChatGPT).\n\n2 - Strictly speaking about the shape of the data that will be fed to the model, should the correct shape be (batch_size, 60, 14) or (batch_size, 14, 60) (and (batch_size, 60, 23) for the outputs)? The first one looks correct to me, but I want to confirm.\n\n3 - Is Unet a type of Seq2Seq? I asked ChatGPT and it said that it is not Seq2Seq.\n\n4 - Can working with bottleneck architectures cause problems with the decoder part due to the \"reconstruction\" of the data? Can using techniques like output_padding to match the shapes cause problems?\n\nIf someone can share a simple Seq2Seq model for learning purposes, it would be really helpful.\n\n", "votes": 1 } ], "comments": [ { "id": 2843265, "author_name": "Amedeo Biolatti", "author_url": "", "post_date": "2024-05-29T13:38:59.100000", "content": "I just made this baseline seq2seq notebook public: https://www.kaggle.com/code/abiolatti/keras-baseline-seq2seq
", "votes": 5, "replies": [ { "id": 2843370, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-29T14:35:29.083000", "content": "Good job. A simple model with a lot of room for improvement, yet with a good score. I'm sure it will help a lot of people.
", "votes": 2, "replies": [] }, { "id": 2845503, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-05-30T15:17:27.547000", "content": "THX, really helpful. I am learning a lot from you.
\nI'm studying your code, where does this architecture come from?
\nIsn't this a complete seq2seq in the sense, that you don't have an explicit encoder-decoder architecture?
\nAnd you don't have the input shape (batch, seq_len, features).
You're confusing seq2seq (a general term) with causal attention (encoder-decoder).
\nThe notebook above implements seq2seq with a 1d cnn.
The x_to_seq function reshape the \"flat\" input in a sequence of shape (Batch, 60, 25). In that case we don't have time/causal order in the sequence so we don't need to mask data from the future like is possible to do with classical encoder/decoder architecture, so we can use convolutions, (unmasked) transformers and/or bidirectional LSTM (or maybe some other fancy idea)
Amedeo, I studied your notebook and our team is improving ours based on yours.
\nI have one more question, how do you initially find this network architecture? Trial and error? Based on some literature network? Domain knowledge?
", "votes": 0, "replies": [] } ] }, { "id": 2841765, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-28T17:26:31.863000", "content": "'feed previous state' is when you use encoder-decoder. This is for problems where the output length is independent of the input length. For example, your input is a sentence at length X, and the output should be a translation of length Y.
\nBut here, the output should be the same length as the input, i.e., 60. So, there is no need for an encoder-decoder scheme, only an encoder. Hence, there is also no need to 'feed the previous state.' Unless you use an RNN/LSTM, of course, which works on the 'previous state' by design, but if your encoder is transformer or 1d unet etc. you just map 60 to 60. This is the 'seq to seq'. For example, consider the simplest model of 1 layer of 1d conv with 'same' paddings. Well, it might be challenging to build a good model of the sort if you don't have previous experience, there are some tricks to the trade haha. Since no one shared a good baseline, this competition is really not easy to get into.