![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/425073/10697/att.png\")
So far, my experiment shows that cnn perform better than lstm. Is it the same as yours?
", "messages": [ { "id": 410996, "postDate": "2018-10-27T06:02:58.487Z", "content": "So far, my experiment shows that cnn perform better than lstm. Is it the same as yours?
", "rawMarkdown": "So far, my experiment shows that cnn perform better than lstm. Is it the same as yours?", "votes": 17 }, { "id": 425073, "postDate": "2018-11-21T05:13:19.667Z", "content": "a new paper today:
\n\nSketch-R2CNN: An Attentive Network for Vector Sketch Recognition - Lei Li, arxiv 2018
\n\nhttps://arxiv.org/pdf/1811.08170.pdf
\n\n
The author of Sketch-R2CNN hasn't released code yet, is there any other implementations?
", "rawMarkdown": "The author of Sketch-R2CNN hasn't released code yet, is there any other implementations?" }, { "id": 431645, "postDate": "2018-12-02T17:03:34.433Z", "content": "I have implemented their solution but it takes long (NRL implemented in TF, not CUDA itself)\n to train and from initial experiments, does not give better results than my own time encoding. However, it certainly is better than no time encoding whatsoever.
", "rawMarkdown": "I have implemented their solution but it takes long (NRL implemented in TF, not CUDA itself)\n to train and from initial experiments, does not give better results than my own time encoding. However, it certainly is better than no time encoding whatsoever.", "votes": 1 } ] }, { "id": 422665, "postDate": "2018-11-16T15:39:15.247Z", "content": "verified the following:
\n\nLB of \"CNN + lstm \" (ensemble, best score) > LB of CNN > LB of lstm
", "rawMarkdown": "verified the following:\n\nLB of \"CNN + lstm \" (ensemble, best score) > LB of CNN > LB of lstm \n", "votes": 4, "replies": [ { "id": 422916, "postDate": "2018-11-17T03:22:32.227Z", "content": "So you already have a LSTM model than better than CNN?
", "rawMarkdown": "So you already have a LSTM model than better than CNN?" }, { "id": 423327, "postDate": "2018-11-17T23:35:08.953Z", "content": "I believe @hengck23 just accidentaly made a mistake in the post and he actually meant this:
\n\n> LB of \"CNN + lstm \" (ensmble) > LB of CNN > LB of lstm
\n\nBut its just my thoughts based on his previous comments about performance of LSTM vs CNN
", "rawMarkdown": "I believe @hengck23 just accidentaly made a mistake in the post and he actually meant this:\n\n> LB of \"CNN + lstm \" (ensmble) **>** LB of CNN **>** LB of lstm\n\nBut its just my thoughts based on his previous comments about performance of LSTM vs CNN", "votes": 2 }, { "id": 423343, "postDate": "2018-11-18T01:03:48.607Z", "content": "Yes, from his previous comments the LSTM is not as good as CNN
", "rawMarkdown": "Yes, from his previous comments the LSTM is not as good as CNN" }, { "id": 423396, "postDate": "2018-11-18T06:13:44.580Z", "content": "I corrected the mistake. Thanks
", "rawMarkdown": "I corrected the mistake. Thanks " } ] }, { "id": 420643, "postDate": "2018-11-13T23:41:40.057Z", "content": "this is actually not a \"pure\" image recognition problem. We are predicting the target, and not what the drawing look like.
\n\nHence you may want to pool results over several strokes and decide how to make the final prediction.\nThere can be incomplete drawing if the drawer hit the target within the first few stroke.
\n\nin theory, LSTM should handle sequence better than CNN. So I expect LSTM to give better scores. But my experiment results show otherwise, which i am puzzled.
\n\nIt is interesting to plot out the scores of CNN over time and see how CNN score changes as the strokes are drawn.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/420643/10675/last_new1.png\")
applying CNN over time.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/425468/10701/drop_strokes.png\")
I tried two branches(CNN feature+LSTM feature: 0.905), but it is worse than only CNN(0.910)
", "rawMarkdown": "I tried two branches(CNN feature+LSTM feature: 0.905), but it is worse than only CNN(0.910)", "votes": 2, "replies": [ { "id": 411935, "postDate": "2018-10-29T08:39:23.027Z", "content": "Do you have an explanation for this? Also, did you first train the branches separately before training them together?
", "rawMarkdown": "Do you have an explanation for this? Also, did you first train the branches separately before training them together?", "votes": -1 }, { "id": 411990, "postDate": "2018-10-29T11:13:09.937Z", "content": "You can check this paper.\nSketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval - Peng Xu
", "rawMarkdown": "You can check this paper.\n[SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval - Peng Xu][1]\n\n\n [1]: http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2763.pdf", "votes": 1 }, { "id": 412719, "postDate": "2018-10-30T16:57:18.290Z", "content": "@Gary Why do they use 224 x 224 x 3, i.e. color images for training? Are they reusing a network pre-trained on color images?
", "rawMarkdown": "@Gary Why do they use 224 x 224 x 3, i.e. color images for training? Are they reusing a network pre-trained on color images?" }, { "id": 415423, "postDate": "2018-11-05T05:21:49.003Z", "content": "Same here, the RNN branch barely added anything to the performance of the CNN alone.
", "rawMarkdown": "Same here, the RNN branch barely added anything to the performance of the CNN alone. " } ] }, { "id": 413328, "postDate": "2018-10-31T17:57:12.807Z", "content": "IDK about using LSTM the image data does not have a temporal dimension.
", "rawMarkdown": "IDK about using LSTM the image data does not have a temporal dimension.", "replies": [ { "id": 413418, "postDate": "2018-10-31T22:49:21.343Z", "content": "The image doesn't, but the pen stroke data does!
", "rawMarkdown": "The image doesn't, but the pen stroke data does!" } ] }, { "id": 412114, "postDate": "2018-10-29T15:21:44.213Z", "content": "same here. CNN tends to perform better
", "rawMarkdown": "same here. CNN tends to perform better" }, { "id": 412006, "postDate": "2018-10-29T11:48:58.347Z", "content": "tried bidirectional RNN model based on the tensorflow sequence classification tutorial, but the result is pretty bad compared to CNNs... still trying to figure out how to make RNN models work..
", "rawMarkdown": "tried bidirectional RNN model based on the tensorflow sequence classification tutorial, but the result is pretty bad compared to CNNs... still trying to figure out how to make RNN models work..", "replies": [ { "id": 423372, "postDate": "2018-11-18T03:55:38.217Z", "content": "hi luyaxin, do you have that implementation of tensorflow sequence classification using this competition test data? Is it ok for you to publish those result in kernel? I would like to try this approach, only if it is possible for you to publish that kernel.. Thanks!
", "rawMarkdown": "hi luyaxin, do you have that implementation of tensorflow sequence classification using this competition test data? Is it ok for you to publish those result in kernel? I would like to try this approach, only if it is possible for you to publish that kernel.. Thanks!", "votes": 1 }, { "id": 423742, "postDate": "2018-11-19T01:04:07.370Z", "content": "hi, you can find the RNN drawing tutorial here:\nhttps://www.tensorflow.org/tutorials/sequences/recurrent_quickdraw
", "rawMarkdown": "hi, you can find the RNN drawing tutorial here:\nhttps://www.tensorflow.org/tutorials/sequences/recurrent_quickdraw" } ] }, { "id": 411257, "postDate": "2018-10-27T17:57:37.613Z", "content": "I got better result with convnets too. But the learning curve for LSTM is better ( much smaller gap between train and validation loss). May be I need to train it more.
", "rawMarkdown": "I got better result with convnets too. But the learning curve for LSTM is better ( much smaller gap between train and validation loss). May be I need to train it more.", "replies": [ { "id": 411368, "postDate": "2018-10-28T00:47:08.787Z", "content": "Hi, Serigne. you used conv1d+lstm?
", "rawMarkdown": "Hi, Serigne. you used conv1d+lstm?" }, { "id": 411566, "postDate": "2018-10-28T13:28:48.690Z", "content": "Yes Conv1d and LSTM for the RNN
", "rawMarkdown": "Yes Conv1d and LSTM for the RNN" }, { "id": 412987, "postDate": "2018-10-31T05:08:47.513Z", "content": "Sergne - Are you using kaggle kernel? any tips how can it be done with so many records?
", "rawMarkdown": "Sergne - Are you using kaggle kernel? any tips how can it be done with so many records?" }, { "id": 413161, "postDate": "2018-10-31T11:57:26.857Z", "content": "I splitted tha data into many files and use generator to feed the NN model
\n\nTake a look at these two Beluga kernels
\n\nhttps://www.kaggle.com/gaborfodor/shuffle-csvs
\n\nhttps://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892
\n\nThey are very helpful
", "rawMarkdown": "I splitted tha data into many files and use generator to feed the NN model\n\nTake a look at these two Beluga kernels \n \nhttps://www.kaggle.com/gaborfodor/shuffle-csvs\n\nhttps://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892\n\nThey are very helpful ", "votes": 4 }, { "id": 413173, "postDate": "2018-10-31T12:17:38.807Z", "content": "Thanks Serigne.\nAre you using any pre trained Model or your custom CNN desIGN.
", "rawMarkdown": "Thanks Serigne.\nAre you using any pre trained Model or your custom CNN desIGN." }, { "id": 413373, "postDate": "2018-10-31T20:19:08.430Z", "content": "Yes I use pre-trained model....but many pre-trained models do not work really well on this data ( at least in reasonnable amount of time) .
\n\nI think if one wanna use pre-trained models, it may be better to pick-up light-weighted model like mobilenet or others and finetune it, instead of very deep and wide resnet.
\n\nImagenet, coco or Pascal-voc pretrained weights can always help for good weight initialization, even though the dataset is quite big.
", "rawMarkdown": "Yes I use pre-trained model....but many pre-trained models do not work really well on this data ( at least in reasonnable amount of time) . \n\nI think if one wanna use pre-trained models, it may be better to pick-up light-weighted model like mobilenet or others and finetune it, instead of very deep and wide resnet. \n\nImagenet, coco or Pascal-voc pretrained weights can always help for good weight initialization, even though the dataset is quite big. ", "votes": 1 }, { "id": 413715, "postDate": "2018-11-01T12:10:16.893Z", "content": "Thanks Serigne. are you doing any kind of augmentation ?
", "rawMarkdown": "Thanks Serigne. are you doing any kind of augmentation ?" }, { "id": 414424, "postDate": "2018-11-02T18:24:16.300Z", "content": "Not yet for training.
\n\nBut Test-time augmentation seems to help a bit. (+0.002 on LB )
", "rawMarkdown": "Not yet for training. \n\nBut Test-time augmentation seems to help a bit. (+0.002 on LB )", "votes": 1 } ] }, { "id": 411252, "postDate": "2018-10-27T17:39:42.970Z", "content": "my LSTM is around 0.1 less than CNN
", "rawMarkdown": "my LSTM is around 0.1 less than CNN" } ], "comments": [ { "id": 425073, "author_name": "hengck23", "author_url": "", "post_date": "2018-11-21T05:13:19.667000", "content": "a new paper today:
\n\nSketch-R2CNN: An Attentive Network for Vector Sketch Recognition - Lei Li, arxiv 2018
\n\nhttps://arxiv.org/pdf/1811.08170.pdf
\n\n
The author of Sketch-R2CNN hasn't released code yet, is there any other implementations?
", "votes": 0, "replies": [] }, { "id": 431645, "author_name": "Lukasz Grad", "author_url": "", "post_date": "2018-12-02T17:03:34.433000", "content": "I have implemented their solution but it takes long (NRL implemented in TF, not CUDA itself)\n to train and from initial experiments, does not give better results than my own time encoding. However, it certainly is better than no time encoding whatsoever.
", "votes": 1, "replies": [] } ] }, { "id": 422665, "author_name": "hengck23", "author_url": "", "post_date": "2018-11-16T15:39:15.247000", "content": "verified the following:
\n\nLB of \"CNN + lstm \" (ensemble, best score) > LB of CNN > LB of lstm
", "votes": 4, "replies": [ { "id": 422916, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-17T03:22:32.227000", "content": "So you already have a LSTM model than better than CNN?
", "votes": 0, "replies": [] }, { "id": 423327, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2018-11-17T23:35:08.953000", "content": "I believe @hengck23 just accidentaly made a mistake in the post and he actually meant this:
\n\n> LB of \"CNN + lstm \" (ensmble) > LB of CNN > LB of lstm
\n\nBut its just my thoughts based on his previous comments about performance of LSTM vs CNN
", "votes": 2, "replies": [] }, { "id": 423343, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-18T01:03:48.607000", "content": "Yes, from his previous comments the LSTM is not as good as CNN
", "votes": 0, "replies": [] }, { "id": 423396, "author_name": "hengck23", "author_url": "", "post_date": "2018-11-18T06:13:44.580000", "content": "I corrected the mistake. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 420643, "author_name": "hengck23", "author_url": "", "post_date": "2018-11-13T23:41:40.057000", "content": "this is actually not a \"pure\" image recognition problem. We are predicting the target, and not what the drawing look like.
\n\nHence you may want to pool results over several strokes and decide how to make the final prediction.\nThere can be incomplete drawing if the drawer hit the target within the first few stroke.
\n\nin theory, LSTM should handle sequence better than CNN. So I expect LSTM to give better scores. But my experiment results show otherwise, which i am puzzled.
\n\nIt is interesting to plot out the scores of CNN over time and see how CNN score changes as the strokes are drawn.
\n\n
applying CNN over time.
\n\n
I tried two branches(CNN feature+LSTM feature: 0.905), but it is worse than only CNN(0.910)
", "votes": 2, "replies": [ { "id": 411935, "author_name": "Kees van Rooijen", "author_url": "", "post_date": "2018-10-29T08:39:23.027000", "content": "Do you have an explanation for this? Also, did you first train the branches separately before training them together?
", "votes": -1, "replies": [] }, { "id": 411990, "author_name": "Gary", "author_url": "", "post_date": "2018-10-29T11:13:09.937000", "content": "You can check this paper.\nSketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval - Peng Xu
", "votes": 1, "replies": [] }, { "id": 412719, "author_name": "Paul Jurczak", "author_url": "", "post_date": "2018-10-30T16:57:18.290000", "content": "@Gary Why do they use 224 x 224 x 3, i.e. color images for training? Are they reusing a network pre-trained on color images?
", "votes": 0, "replies": [] }, { "id": 415423, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-05T05:21:49.003000", "content": "Same here, the RNN branch barely added anything to the performance of the CNN alone.
", "votes": 0, "replies": [] } ] }, { "id": 413328, "author_name": "", "author_url": "", "post_date": "2018-10-31T17:57:12.807000", "content": "IDK about using LSTM the image data does not have a temporal dimension.
", "votes": 0, "replies": [ { "id": 413418, "author_name": "Paul Jurczak", "author_url": "", "post_date": "2018-10-31T22:49:21.343000", "content": "The image doesn't, but the pen stroke data does!
", "votes": 0, "replies": [] } ] }, { "id": 412114, "author_name": "Gdd", "author_url": "", "post_date": "2018-10-29T15:21:44.213000", "content": "same here. CNN tends to perform better
", "votes": 0, "replies": [] }, { "id": 412006, "author_name": "luyaxin", "author_url": "", "post_date": "2018-10-29T11:48:58.347000", "content": "tried bidirectional RNN model based on the tensorflow sequence classification tutorial, but the result is pretty bad compared to CNNs... still trying to figure out how to make RNN models work..
", "votes": 0, "replies": [ { "id": 423372, "author_name": "", "author_url": "", "post_date": "2018-11-18T03:55:38.217000", "content": "hi luyaxin, do you have that implementation of tensorflow sequence classification using this competition test data? Is it ok for you to publish those result in kernel? I would like to try this approach, only if it is possible for you to publish that kernel.. Thanks!
", "votes": 1, "replies": [] }, { "id": 423742, "author_name": "luyaxin", "author_url": "", "post_date": "2018-11-19T01:04:07.370000", "content": "hi, you can find the RNN drawing tutorial here:\nhttps://www.tensorflow.org/tutorials/sequences/recurrent_quickdraw
", "votes": 0, "replies": [] } ] }, { "id": 411257, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-27T17:57:37.613000", "content": "I got better result with convnets too. But the learning curve for LSTM is better ( much smaller gap between train and validation loss). May be I need to train it more.
", "votes": 0, "replies": [ { "id": 411368, "author_name": "Gary", "author_url": "", "post_date": "2018-10-28T00:47:08.787000", "content": "Hi, Serigne. you used conv1d+lstm?
", "votes": 0, "replies": [] }, { "id": 411566, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-28T13:28:48.690000", "content": "Yes Conv1d and LSTM for the RNN
", "votes": 0, "replies": [] }, { "id": 412987, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-10-31T05:08:47.513000", "content": "Sergne - Are you using kaggle kernel? any tips how can it be done with so many records?
", "votes": 0, "replies": [] }, { "id": 413161, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-31T11:57:26.857000", "content": "I splitted tha data into many files and use generator to feed the NN model
\n\nTake a look at these two Beluga kernels
\n\nhttps://www.kaggle.com/gaborfodor/shuffle-csvs
\n\nhttps://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892
\n\nThey are very helpful
", "votes": 4, "replies": [] }, { "id": 413173, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-10-31T12:17:38.807000", "content": "Thanks Serigne.\nAre you using any pre trained Model or your custom CNN desIGN.
", "votes": 0, "replies": [] }, { "id": 413373, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-31T20:19:08.430000", "content": "Yes I use pre-trained model....but many pre-trained models do not work really well on this data ( at least in reasonnable amount of time) .
\n\nI think if one wanna use pre-trained models, it may be better to pick-up light-weighted model like mobilenet or others and finetune it, instead of very deep and wide resnet.
\n\nImagenet, coco or Pascal-voc pretrained weights can always help for good weight initialization, even though the dataset is quite big.
", "votes": 1, "replies": [] }, { "id": 413715, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-11-01T12:10:16.893000", "content": "Thanks Serigne. are you doing any kind of augmentation ?
", "votes": 0, "replies": [] }, { "id": 414424, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-02T18:24:16.300000", "content": "Not yet for training.
\n\nBut Test-time augmentation seems to help a bit. (+0.002 on LB )
", "votes": 1, "replies": [] } ] }, { "id": 411252, "author_name": "Giba", "author_url": "", "post_date": "2018-10-27T17:39:42.970000", "content": "my LSTM is around 0.1 less than CNN
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "410996": "So far, my experiment shows that cnn perform better than lstm. Is it the same as yours?", "425073": "a new paper today:\n\nSketch-R2CNN: An Attentive Network for Vector Sketch Recognition - Lei Li, arxiv 2018\n\n\nhttps://arxiv.org/pdf/1811.08170.pdf\n\n ![enter image description here][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/425073/10697/att.png", "422665": "verified the following:\n\nLB of \"CNN + lstm \" (ensemble, best score) > LB of CNN > LB of lstm \n", "420643": "this is actually not a \"pure\" image recognition problem. We are predicting the target, and not what the drawing look like.\n\nHence you may want to pool results over several strokes and decide how to make the final prediction.\nThere can be incomplete drawing if the drawer hit the target within the first few stroke.\n\nin theory, LSTM should handle sequence better than CNN. So I expect LSTM to give better scores. But my experiment results show otherwise, which i am puzzled.\n\nIt is interesting to plot out the scores of CNN over time and see how CNN score changes as the strokes are drawn.\n\n ![enter image description here][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/420643/10675/last_new1.png", "425468": "applying CNN over time. \n\n ![enter image description here][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/425468/10701/drop_strokes.png", "411007": "I tried two branches(CNN feature+LSTM feature: 0.905), but it is worse than only CNN(0.910)", "413328": "IDK about using LSTM the image data does not have a temporal dimension.", "412114": "same here. CNN tends to perform better", "412006": "tried bidirectional RNN model based on the tensorflow sequence classification tutorial, but the result is pretty bad compared to CNNs... still trying to figure out how to make RNN models work..", "411257": "I got better result with convnets too. But the learning curve for LSTM is better ( much smaller gap between train and validation loss). May be I need to train it more.", "411252": "my LSTM is around 0.1 less than CNN" } }