![\"enter](\"https://discuss.pytorch.org/uploads/default/original/1X/b01a9e75313b8c651c7e3cb22c2161abbaa4751c.png\")
With a busy Oct and Nov I jumped into this challenge very late. I really wanted to try something a little different so I started with an RNN based model. Inspired a bit by a Tensorflow example I saw, https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py, I created a 'basic' 1d CNN + LSTM model in PyTorch. I created a more complex one by incorporating SE blocks from the SEResNext network and converting them to 1d with no strides.
\n\nAn ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training. To round out the ensemble and boost the score into the 0.947 range I included some CNN based models that I trained on the side. The CNNs mirrored the approach taken by many others in this challenge. I was hoping to include the best RNN and the best CNN model into a single one and fine tune the combined model to see how good it could get but ran out of time so they were all combined via ensemble.
\n\nI posted a gist of just the cleaned up RNN models: https://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526
\n\nI fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.
\n\nI didn't have much (any) time to experiment with varations on these RNN models, the SE based one was still improving at the time of final submissions. Potential to match the pure CNN models? Perhaps...
", "messages": [ { "id": 434040, "postDate": "2018-12-05T20:19:14.397Z", "content": "With a busy Oct and Nov I jumped into this challenge very late. I really wanted to try something a little different so I started with an RNN based model. Inspired a bit by a Tensorflow example I saw, https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py, I created a 'basic' 1d CNN + LSTM model in PyTorch. I created a more complex one by incorporating SE blocks from the SEResNext network and converting them to 1d with no strides.
\n\nAn ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training. To round out the ensemble and boost the score into the 0.947 range I included some CNN based models that I trained on the side. The CNNs mirrored the approach taken by many others in this challenge. I was hoping to include the best RNN and the best CNN model into a single one and fine tune the combined model to see how good it could get but ran out of time so they were all combined via ensemble.
\n\nI posted a gist of just the cleaned up RNN models: https://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526
\n\nI fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.
\n\nI didn't have much (any) time to experiment with varations on these RNN models, the SE based one was still improving at the time of final submissions. Potential to match the pure CNN models? Perhaps...
", "rawMarkdown": "With a busy Oct and Nov I jumped into this challenge very late. I really wanted to try something a little different so I started with an RNN based model. Inspired a bit by a Tensorflow example I saw, https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py, I created a 'basic' 1d CNN + LSTM model in PyTorch. I created a more complex one by incorporating SE blocks from the SEResNext network and converting them to 1d with no strides.\n\nAn ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training. To round out the ensemble and boost the score into the 0.947 range I included some CNN based models that I trained on the side. The CNNs mirrored the approach taken by many others in this challenge. I was hoping to include the best RNN and the best CNN model into a single one and fine tune the combined model to see how good it could get but ran out of time so they were all combined via ensemble.\n\nI posted a gist of just the cleaned up RNN models: https://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526\n\nI fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset. \n\nI didn't have much (any) time to experiment with varations on these RNN models, the SE based one was still improving at the time of final submissions. Potential to match the pure CNN models? Perhaps...", "votes": 18 }, { "id": 439976, "postDate": "2018-12-16T18:56:05.133Z", "content": "I ran through training of my 'basic sequence' model again from scratch with the fix (as pointed out by Heng) so that the reverse direction outputs are from the correct timestep.
\n\nSingle model scored .9415 private, .9427 public ... so now better than my previous ensemble of two RNN models.
", "rawMarkdown": "I ran through training of my 'basic sequence' model again from scratch with the fix (as pointed out by Heng) so that the reverse direction outputs are from the correct timestep.\n\nSingle model scored .9415 private, .9427 public ... so now better than my previous ensemble of two RNN models.", "votes": 1 }, { "id": 434355, "postDate": "2018-12-06T09:04:58.703Z", "content": "\"I fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.\"
\n\ndo you have code for this? thanks!
", "rawMarkdown": "\"I fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.\"\n\ndo you have code for this? thanks!", "votes": 1 }, { "id": 434350, "postDate": "2018-12-06T08:49:12.163Z", "content": "@ RossWightman
\n\nThis is the first time i used bidirectional lstm in pytorch, so my understanding my be wrong. In your code:
\n\nhttps://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526, for BasicSeqStrokeNet:
\n\nx, (h, c) = self.rnn(x)\n... \nlast_output = xo.gather(1, idx).squeeze(1)\nwhich i think you are taking the last output of forward lstm and first output of backward lstm. But I think we should be taking last output of both forward and backward.
\n\n(Note a simpler way is to use torch.cat ([h[-1], h[-2]])
\n\nNow according to
\n\nhttps://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66
\n\n\"We should take output[-1, :, :hidden_size] (normal RNN) and output[0, :, hidden_size:] (reverse RNN), concatenate them, and feed the result to the subsequent dense neural network.\"
\n\nalso from https://stackoverflow.com/questions/50856936/taking-the-last-state-from-bilstm-bigru-in-pytorch
\n\nlast_forward = torch.gather(output_forward, 0, lengths - 1).squeeze(0)\nlast_backward = output_backward[0, :, :]\nI was lead to believe, after reading some examples and docs, that the gather on output was necessary to grab the valid output at the end of each (possibly variable length) sequence as per the sequence lengths for each input in the packed/padded format. The output was already hidden size * 2 which happens as soon as you enable bidrectional for the LSTM so it's my understanding nothing is needed wrt to that for output.
", "rawMarkdown": "I was lead to believe, after reading some examples and docs, that the gather on output was necessary to grab the valid output at the end of each (possibly variable length) sequence as per the sequence lengths for each input in the packed/padded format. The output was already hidden size * 2 which happens as soon as you enable bidrectional for the LSTM so it's my understanding nothing is needed wrt to that for output." }, { "id": 436673, "postDate": "2018-12-10T18:14:50.887Z", "content": "The output is 2xhidden size. But the time is wrong. You are getting last of forward lstm ( hidden size) and first of backward lstm ( another hidden size)
", "rawMarkdown": "The output is 2xhidden size. But the time is wrong. You are getting last of forward lstm ( hidden size) and first of backward lstm ( another hidden size)" }, { "id": 436675, "postDate": "2018-12-10T18:17:19.643Z", "content": "https://discuss.pytorch.org/t/bilstm-hidden-states-and-output-dont-match/3561
\n\n
Okay, yeah I see now, so, with that in mind, and taking into account that I'm in batch first I have...
\n\n idxf = (xl - 1).view(-1, 1).expand(xl.size(0), xo.size(2) // 2)\n idxf = idxf.unsqueeze(1)\n idxf = idxf.cuda()\n\n max_seq_len = int(x_seq_len[0])\n idxr = (max_seq_len - xl).view(-1, 1).expand(xl.size(0), xo.size(2) // 2)\n idxr = idxr.unsqueeze(1)\n idxr = idxr.cuda()\n\n # Shape: (batch_size, rnn_hidden_dim)\n last_output_forward = xo[:, :, :512].gather(1, idxf).squeeze(1)\n last_output_reverse = xo[:, :, 512:].gather(1, idxr).squeeze(1)\n\n last_output = torch.cat((last_output_forward, last_output_reverse), 1)\nThe one thing that I'm still not at all clear on is whether I can just use 0 for the reverse direction as in
\n\n last_output_reverse = xo[:, 0, 512:]\nI'm not at all clear on how the padding is handled in the reverse direction for bidir when I'm doing the pack/pad. Does padding end up on the left? so I'd have [pad, pad, pad, ...., 3, 2, 1] in the reverse or would it be [...., 3, 2, 1, pad, pad, pad] ... that would change what you need to pick off from the output...
\n\nOr perhaps I could just cat((h[-2], h[-1])) as that is now matching my output. BUT, my seq lengths are uniform since I resampled them. I'm not sure if I can still use h as is if that's not the case....
\n\nEDIT
\n\nI disabled interpolation for inputs < my max sequence to test the behaviour with variable sequence lengths. The last element in the reverse sequence is always at index 0 and no compensation for padding needed. So, last_output_reverse = xo[:, 0, 512:] is indeed correct and my last_output == torch.cat((h[-2], h[-1])) now for both fixed and variable length sequences.
thanks for the post!
\n\n\"An ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training\"
\n\ndo you have LB score of each individual ones?
", "rawMarkdown": "thanks for the post!\n\n\"An ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training\"\n\ndo you have LB score of each individual ones?", "replies": [ { "id": 434109, "postDate": "2018-12-05T22:42:51.453Z", "content": "In interest of saving submissions, I wasn't submitting single models once the local cv was reliable...
\n\nI just submitted solo now to see..
\n\n'basic seq' 0.93979 priv, 0.94029 pub\n'se seq' 0.93112, 0.93419
\n\nSo the basic model was much stronger. I feel that was largely due to it having more time to train. I created and started training the SE model several days later, it was still noteably improving at the point I had to stop.
", "rawMarkdown": "In interest of saving submissions, I wasn't submitting single models once the local cv was reliable...\n\nI just submitted solo now to see..\n\n'basic seq' 0.93979 priv, 0.94029 pub\n'se seq' 0.93112, 0.93419\n\nSo the basic model was much stronger. I feel that was largely due to it having more time to train. I created and started training the SE model several days later, it was still noteably improving at the point I had to stop. \n " } ] }, { "id": 434095, "postDate": "2018-12-05T21:53:23.980Z", "content": "Congrats and thanks for sharing your solution.
", "rawMarkdown": "Congrats and thanks for sharing your solution." } ], "comments": [ { "id": 439976, "author_name": "RossWightman", "author_url": "", "post_date": "2018-12-16T18:56:05.133000", "content": "I ran through training of my 'basic sequence' model again from scratch with the fix (as pointed out by Heng) so that the reverse direction outputs are from the correct timestep.
\n\nSingle model scored .9415 private, .9427 public ... so now better than my previous ensemble of two RNN models.
", "votes": 1, "replies": [] }, { "id": 434355, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-06T09:04:58.703000", "content": "\"I fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.\"
\n\ndo you have code for this? thanks!
", "votes": 1, "replies": [] }, { "id": 434350, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-06T08:49:12.163000", "content": "@ RossWightman
\n\nThis is the first time i used bidirectional lstm in pytorch, so my understanding my be wrong. In your code:
\n\nhttps://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526, for BasicSeqStrokeNet:
\n\nx, (h, c) = self.rnn(x)\n... \nlast_output = xo.gather(1, idx).squeeze(1)\nwhich i think you are taking the last output of forward lstm and first output of backward lstm. But I think we should be taking last output of both forward and backward.
\n\n(Note a simpler way is to use torch.cat ([h[-1], h[-2]])
\n\nNow according to
\n\nhttps://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66
\n\n\"We should take output[-1, :, :hidden_size] (normal RNN) and output[0, :, hidden_size:] (reverse RNN), concatenate them, and feed the result to the subsequent dense neural network.\"
\n\nalso from https://stackoverflow.com/questions/50856936/taking-the-last-state-from-bilstm-bigru-in-pytorch
\n\nlast_forward = torch.gather(output_forward, 0, lengths - 1).squeeze(0)\nlast_backward = output_backward[0, :, :]\nI was lead to believe, after reading some examples and docs, that the gather on output was necessary to grab the valid output at the end of each (possibly variable length) sequence as per the sequence lengths for each input in the packed/padded format. The output was already hidden size * 2 which happens as soon as you enable bidrectional for the LSTM so it's my understanding nothing is needed wrt to that for output.
", "votes": 0, "replies": [] }, { "id": 436673, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-10T18:14:50.887000", "content": "The output is 2xhidden size. But the time is wrong. You are getting last of forward lstm ( hidden size) and first of backward lstm ( another hidden size)
", "votes": 0, "replies": [] }, { "id": 436675, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-10T18:17:19.643000", "content": "https://discuss.pytorch.org/t/bilstm-hidden-states-and-output-dont-match/3561
\n\n
Okay, yeah I see now, so, with that in mind, and taking into account that I'm in batch first I have...
\n\n idxf = (xl - 1).view(-1, 1).expand(xl.size(0), xo.size(2) // 2)\n idxf = idxf.unsqueeze(1)\n idxf = idxf.cuda()\n\n max_seq_len = int(x_seq_len[0])\n idxr = (max_seq_len - xl).view(-1, 1).expand(xl.size(0), xo.size(2) // 2)\n idxr = idxr.unsqueeze(1)\n idxr = idxr.cuda()\n\n # Shape: (batch_size, rnn_hidden_dim)\n last_output_forward = xo[:, :, :512].gather(1, idxf).squeeze(1)\n last_output_reverse = xo[:, :, 512:].gather(1, idxr).squeeze(1)\n\n last_output = torch.cat((last_output_forward, last_output_reverse), 1)\nThe one thing that I'm still not at all clear on is whether I can just use 0 for the reverse direction as in
\n\n last_output_reverse = xo[:, 0, 512:]\nI'm not at all clear on how the padding is handled in the reverse direction for bidir when I'm doing the pack/pad. Does padding end up on the left? so I'd have [pad, pad, pad, ...., 3, 2, 1] in the reverse or would it be [...., 3, 2, 1, pad, pad, pad] ... that would change what you need to pick off from the output...
\n\nOr perhaps I could just cat((h[-2], h[-1])) as that is now matching my output. BUT, my seq lengths are uniform since I resampled them. I'm not sure if I can still use h as is if that's not the case....
\n\nEDIT
\n\nI disabled interpolation for inputs < my max sequence to test the behaviour with variable sequence lengths. The last element in the reverse sequence is always at index 0 and no compensation for padding needed. So, last_output_reverse = xo[:, 0, 512:] is indeed correct and my last_output == torch.cat((h[-2], h[-1])) now for both fixed and variable length sequences.
thanks for the post!
\n\n\"An ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training\"
\n\ndo you have LB score of each individual ones?
", "votes": 0, "replies": [ { "id": 434109, "author_name": "RossWightman", "author_url": "", "post_date": "2018-12-05T22:42:51.453000", "content": "In interest of saving submissions, I wasn't submitting single models once the local cv was reliable...
\n\nI just submitted solo now to see..
\n\n'basic seq' 0.93979 priv, 0.94029 pub\n'se seq' 0.93112, 0.93419
\n\nSo the basic model was much stronger. I feel that was largely due to it having more time to train. I created and started training the SE model several days later, it was still noteably improving at the point I had to stop.
", "votes": 0, "replies": [] } ] }, { "id": 434095, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-12-05T21:53:23.980000", "content": "Congrats and thanks for sharing your solution.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "434040": "With a busy Oct and Nov I jumped into this challenge very late. I really wanted to try something a little different so I started with an RNN based model. Inspired a bit by a Tensorflow example I saw, https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py, I created a 'basic' 1d CNN + LSTM model in PyTorch. I created a more complex one by incorporating SE blocks from the SEResNext network and converting them to 1d with no strides.\n\nAn ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training. To round out the ensemble and boost the score into the 0.947 range I included some CNN based models that I trained on the side. The CNNs mirrored the approach taken by many others in this challenge. I was hoping to include the best RNN and the best CNN model into a single one and fine tune the combined model to see how good it could get but ran out of time so they were all combined via ensemble.\n\nI posted a gist of just the cleaned up RNN models: https://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526\n\nI fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset. \n\nI didn't have much (any) time to experiment with varations on these RNN models, the SE based one was still improving at the time of final submissions. Potential to match the pure CNN models? Perhaps...", "439976": "I ran through training of my 'basic sequence' model again from scratch with the fix (as pointed out by Heng) so that the reverse direction outputs are from the correct timestep.\n\nSingle model scored .9415 private, .9427 public ... so now better than my previous ensemble of two RNN models.", "434355": "\"I fed both of these with sequences that had been interpolated from the original stroke vectors to a consistent length somewhat arbirtarily chose to be 192 with x, y, t, and pen up inputs in the tensor normalized based on a partial pass of the dataset.\"\n\ndo you have code for this? thanks!", "434350": "@ RossWightman\n\nThis is the first time i used bidirectional lstm in pytorch, so my understanding my be wrong. In your code:\n\nhttps://gist.github.com/rwightman/893723e023ffccca4abf67f92cb76526, for BasicSeqStrokeNet:\n\n\n x, (h, c) = self.rnn(x)\n ... \n last_output = xo.gather(1, idx).squeeze(1)\n\n\nwhich i think you are taking the last output of forward lstm and first output of backward lstm. But I think we should be taking last output of both forward and backward.\n\n(Note a simpler way is to use torch.cat ([h[-1], h[-2]])\n\n\n---\n\nNow according to \n\nhttps://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66\n\n\"We should take output[-1, :, :hidden_size] (normal RNN) and output[0, :, hidden_size:] (reverse RNN), concatenate them, and feed the result to the subsequent dense neural network.\"\n\nalso from https://stackoverflow.com/questions/50856936/taking-the-last-state-from-bilstm-bigru-in-pytorch\n\n\n last_forward = torch.gather(output_forward, 0, lengths - 1).squeeze(0)\n last_backward = output_backward[0, :, :]", "434101": "thanks for the post!\n\n\"An ensemble of one 'basic' and one 'se' model ended up scoring 0.941 after a week of training\"\n\ndo you have LB score of each individual ones?", "434095": "Congrats and thanks for sharing your solution." } }