Since we have a lot of data for training, it may be a limitation for who does not have enough computer resource.
\nI did some experiences and I would like to share how much data I used and corresponding public LB score.
Cheers,
", "messages": [ { "id": 407964, "postDate": "2018-10-22T04:28:36.713Z", "content": "Since we have a lot of data for training, it may be a limitation for who does not have enough computer resource.
\nI did some experiences and I would like to share how much data I used and corresponding public LB score.
Cheers,
", "rawMarkdown": "Since we have a lot of data for training, it may be a limitation for who does not have enough computer resource. \nI did some experiences and I would like to share how much data I used and corresponding public LB score. \n\n - 3k images / class, LB: 0.832\n - 10k images / class, LB: 0.862\n - 20k images / class, LB: 0.900\n\nCheers,", "votes": 31 }, { "id": 421428, "postDate": "2018-11-15T01:53:52.663Z", "content": "Thanks to Nguyen for your motivation and I used my 1080ti for this competition. \nFYI, my current result is:\n- 25K/class, Local 0.887 and Public LB 0.935 without TTA.\n- Will add TTA and I expect a 0.002's boost.\n- By the way, since I haven't tried any fancy techniques, I think 25k/class for 0.940+ is not hard to achieve.
", "rawMarkdown": "Thanks to Nguyen for your motivation and I used my 1080ti for this competition. \nFYI, my current result is:\n- 25K/class, Local 0.887 and Public LB 0.935 without TTA.\n- Will add TTA and I expect a 0.002's boost.\n- By the way, since I haven't tried any fancy techniques, I think 25k/class for 0.940+ is not hard to achieve.", "votes": 3, "replies": [ { "id": 421495, "postDate": "2018-11-15T04:12:14.757Z", "content": "@YourVenn, did you use pretrained model, like ResNet 50?
", "rawMarkdown": "@YourVenn, did you use pretrained model, like ResNet 50?\n" }, { "id": 421514, "postDate": "2018-11-15T05:00:18.620Z", "content": "yes, pre-trained model is useful.
", "rawMarkdown": "yes, pre-trained model is useful." }, { "id": 421778, "postDate": "2018-11-15T12:05:49.153Z", "content": "but I only got local 0.877, weird..
", "rawMarkdown": "but I only got local 0.877, weird.." }, { "id": 421972, "postDate": "2018-11-15T16:12:17.670Z", "content": "But you may not have the same validation set as Venn has, so may be you LB result is close
", "rawMarkdown": "But you may not have the same validation set as Venn has, so may be you LB result is close" } ] }, { "id": 421448, "postDate": "2018-11-15T02:38:35.510Z", "content": "128 * 128 image, all simplified data (but didn't finish one epoch), local val 0.8797, LB 0.935
", "rawMarkdown": "128 * 128 image, all simplified data (but didn't finish one epoch), local val 0.8797, LB 0.935", "votes": 1 }, { "id": 418572, "postDate": "2018-11-10T06:51:33.763Z", "content": "It seems like a case of rapidly diminishing return, for the same model, I got 0.92 for 30k images/class, but only 0.923 for all data T.T
", "rawMarkdown": "It seems like a case of rapidly diminishing return, for the same model, I got 0.92 for 30k images/class, but only 0.923 for all data T.T", "votes": 1 }, { "id": 415421, "postDate": "2018-11-05T05:16:43.703Z", "content": "30k/class, CNN LB: 0.92
", "rawMarkdown": "30k/class, CNN LB: 0.92", "votes": 1, "replies": [ { "id": 415460, "postDate": "2018-11-05T07:05:34Z", "content": "nice work, could you share what model did you use?
", "rawMarkdown": "nice work, could you share what model did you use?" }, { "id": 415461, "postDate": "2018-11-05T07:12:35.477Z", "content": "I can't disclose too much except it's a pretrained model.
", "rawMarkdown": "I can't disclose too much except it's a pretrained model. " }, { "id": 415499, "postDate": "2018-11-05T08:24:48.770Z", "content": "Impressive result with a small dataset!. Nice work!
", "rawMarkdown": "Impressive result with a small dataset!. Nice work!" }, { "id": 415556, "postDate": "2018-11-05T09:59:50.057Z", "content": "Sorry I made a mistake in my calculation, it's actually 30k/class -.-'
", "rawMarkdown": "Sorry I made a mistake in my calculation, it's actually 30k/class -.-' " } ] }, { "id": 407973, "postDate": "2018-10-22T05:04:33.193Z", "content": "5k images / class, LB: 0.892
\n\n50k images / class, LB: 0.929
", "rawMarkdown": "5k images / class, LB: 0.892\n\n50k images / class, LB: 0.929", "votes": 1, "replies": [ { "id": 410705, "postDate": "2018-10-26T13:55:28.923Z", "content": "Hi, you got a nice socre. if you don't mind. Could you share what model did you use?
", "rawMarkdown": "Hi, you got a nice socre. if you don't mind. Could you share what model did you use?" }, { "id": 410729, "postDate": "2018-10-26T15:01:01.130Z", "content": "I used LSTM + Attention
", "rawMarkdown": "I used LSTM + Attention", "votes": 2 }, { "id": 410735, "postDate": "2018-10-26T15:18:16.917Z", "content": "Attention Is All You Need :)
\n\nIt helped a lot on my RNN too :)
", "rawMarkdown": "Attention Is All You Need :)\n\nIt helped a lot on my RNN too :)", "votes": 1 }, { "id": 410743, "postDate": "2018-10-26T15:32:03.570Z", "content": "Thanks for sharing. Just LSTM? no CNN?
", "rawMarkdown": "Thanks for sharing. Just LSTM? no CNN?" }, { "id": 410754, "postDate": "2018-10-26T15:52:15.140Z", "content": "It's a mix of LSTM and CNN, but mainly LSTM
", "rawMarkdown": "It's a mix of LSTM and CNN, but mainly LSTM" }, { "id": 410757, "postDate": "2018-10-26T16:07:19.650Z", "content": "Sure, It is a model worth trying
", "rawMarkdown": "Sure, It is a model worth trying" }, { "id": 410762, "postDate": "2018-10-26T16:13:01.403Z", "content": "@Tam
\nDo you mean 1D CNN on strokes or 2D CNN on images ?
I think there may be some confusion :)
", "rawMarkdown": "@Tam \nDo you mean 1D CNN on strokes or 2D CNN on images ? \n\nI think there may be some confusion :)" }, { "id": 410763, "postDate": "2018-10-26T16:13:16.710Z", "content": "\n\n\nI used LSTM + Attention
\n \nAttention Is All You Need :)
\n
Wow. Amazing!. LSTM/RNN is one of my weakness. I always get the trouble with them :D
", "rawMarkdown": "> I used LSTM + Attention\n\n> Attention Is All You Need :)\n\n\nWow. Amazing!. LSTM/RNN is one of my weakness. I always get the trouble with them :D" }, { "id": 414505, "postDate": "2018-11-02T22:31:23.583Z", "content": "Hi @serigne, I finally manage to get my attention block working. But it gave almost no improvement (in fact it was slightly worst around 0.8 with 20k/class)\nWhere dit you put it ? before/after the batchnorm ? before/after the lstm layers ?\n(Mine was between my 2 lstm layers)
\n\nIf anyone as tips on how to use \"attention\" well I take :)
", "rawMarkdown": "Hi @serigne, I finally manage to get my attention block working. But it gave almost no improvement (in fact it was slightly worst around 0.8 with 20k/class)\nWhere dit you put it ? before/after the batchnorm ? before/after the lstm layers ?\n(Mine was between my 2 lstm layers)\n\nIf anyone as tips on how to use \"attention\" well I take :)", "votes": 1 }, { "id": 414733, "postDate": "2018-11-03T13:30:23.773Z", "content": "@Haral You have to add the Attention layer after your last LSTM layer (with return_sequence = True )
", "rawMarkdown": "@Haral You have to add the Attention layer after your last LSTM layer (with return_sequence = True ) ", "votes": 1 }, { "id": 417480, "postDate": "2018-11-08T10:51:03.227Z", "content": "Like Haral, I added attention and it didn't help either.
", "rawMarkdown": "Like Haral, I added attention and it didn't help either. " } ] }, { "id": 407987, "postDate": "2018-10-22T05:26:42.373Z", "content": "25k images/class, LB: 0.916 (single model)
\n\nYes size of dataset really matters , but 25k seems large enough for me for experiments (particularly because of limitations on kaggle kernels)
\n\nI will focus then on building stronger model before using more data
", "rawMarkdown": "25k images/class, LB: 0.916 (single model)\n\nYes size of dataset really matters , but 25k seems large enough for me for experiments (particularly because of limitations on kaggle kernels)\n\nI will focus then on building stronger model before using more data\n\n", "votes": 2, "replies": [ { "id": 408029, "postDate": "2018-10-22T07:38:21.697Z", "content": "I have a similar result and plan as you. I will spend time on the final weeks to train with big data.
", "rawMarkdown": "I have a similar result and plan as you. I will spend time on the final weeks to train with big data. ", "votes": -1 }, { "id": 408761, "postDate": "2018-10-23T12:18:26.943Z", "content": "Hi Serigne, have you been only using kaggle kernels so far?
", "rawMarkdown": "Hi Serigne, have you been only using kaggle kernels so far? " }, { "id": 408782, "postDate": "2018-10-23T13:05:55.060Z", "content": "Yes kernel only
\n\nHowever the GPU version of kernel has not a good support of Pytorch 4.xx. So I need to stick with keras.
", "rawMarkdown": "Yes kernel only \n\nHowever the GPU version of kernel has not a good support of Pytorch 4.xx. So I need to stick with keras.", "votes": 2 }, { "id": 409464, "postDate": "2018-10-24T10:39:52Z", "content": "I think your model is great.
", "rawMarkdown": "I think your model is great." }, { "id": 410219, "postDate": "2018-10-25T16:10:40.090Z", "content": "Can i bother asking if your single model is more CNN or more like RNN or even different ? Thx Serigne ;)
", "rawMarkdown": "Can i bother asking if your single model is more CNN or more like RNN or even different ? Thx Serigne ;)" }, { "id": 410275, "postDate": "2018-10-25T18:10:17.233Z", "content": "Thanks Gary
\n\n@Haral I use convnets on images now. I used earlier a LSTM model on strokes but slower to train and it scored 0.905 for 18k/class.
\n\nMaybe I need to find a way to combine both.
", "rawMarkdown": "Thanks Gary \n\n@Haral I use convnets on images now. I used earlier a LSTM model on strokes but slower to train and it scored 0.905 for 18k/class.\n\nMaybe I need to find a way to combine both. ", "votes": 1 }, { "id": 410379, "postDate": "2018-10-26T00:14:57.453Z", "content": "yep, I also need to find a way to combine both. maybe this way work: two branches(CNN for pictures, and LSTM for strokes, and then combine the features from them.)
", "rawMarkdown": "yep, I also need to find a way to combine both. maybe this way work: two branches(CNN for pictures, and LSTM for strokes, and then combine the features from them.) ", "votes": 1 }, { "id": 410443, "postDate": "2018-10-26T04:31:48.813Z", "content": "what is the structure of your RNN based model? I was only able to get 0.84 LB.
", "rawMarkdown": "what is the structure of your RNN based model? I was only able to get 0.84 LB." }, { "id": 410451, "postDate": "2018-10-26T04:57:06.707Z", "content": "CNN achieve LB 0.916??
", "rawMarkdown": "CNN achieve LB 0.916??" }, { "id": 410668, "postDate": "2018-10-26T12:31:17.197Z", "content": "@zjucor my score come from the structure ^^
\n\n@Joe yes, CNN layers in a self designed structure
", "rawMarkdown": "@zjucor my score come from the structure ^^\n\n@Joe yes, CNN layers in a self designed structure", "votes": 1 }, { "id": 410698, "postDate": "2018-10-26T13:41:18.847Z", "content": "Awesome, I tried residue block, resnet 50, mobilenetV2, but unable to give a great boost beyond a kernel scored 0.892, can you share with us any method to build a better model? For me, i'm just randomly trying different model...
", "rawMarkdown": "Awesome, I tried residue block, resnet 50, mobilenetV2, but unable to give a great boost beyond a kernel scored 0.892, can you share with us any method to build a better model? For me, i'm just randomly trying different model..." }, { "id": 410704, "postDate": "2018-10-26T13:54:29.950Z", "content": "if you use resnet50, this model is too large. It need so much time to train. I this mobilenetV2 can get 0.90+ score.
", "rawMarkdown": "if you use resnet50, this model is too large. It need so much time to train. I this mobilenetV2 can get 0.90+ score.", "votes": 1 }, { "id": 410734, "postDate": "2018-10-26T15:16:55.503Z", "content": "@Joe \nI am also a relative newbie in Computer vision ( TGS was my first serious competition on it) .
\nAnd just like you many newbies , my NN structure come from trials and errors ( I mean I am not an expert in NN structure design :p )
I second Gary , you should start with light weight architecture and fine tune it, then check your CV/validation score.
", "rawMarkdown": "@Joe \nI am also a relative newbie in Computer vision ( TGS was my first serious competition on it) . \nAnd just like you many newbies , my NN structure come from trials and errors ( I mean I am not an expert in NN structure design :p )\n\nI second Gary , you should start with light weight architecture and fine tune it, then check your CV/validation score. \n\n" }, { "id": 410746, "postDate": "2018-10-26T15:39:03.547Z", "content": "yep, I started with light weight architecture. But it may be need some fantastic layers to get a better socre.
", "rawMarkdown": "yep, I started with light weight architecture. But it may be need some fantastic layers to get a better socre." } ] }, { "id": 408592, "postDate": "2018-10-23T06:02:34.253Z", "content": "6k samples/class, LB: 0.84 (local cv 0.88) using bidirectional lstm
", "rawMarkdown": "6k samples/class, LB: 0.84 (local cv 0.88) using bidirectional lstm", "votes": 1 }, { "id": 413326, "postDate": "2018-10-31T17:55:46.723Z", "content": "very cool, in general the larger the better but in this competition the train data itself contains a lot of noise so not sure how that would effect this
", "rawMarkdown": "very cool, in general the larger the better but in this competition the train data itself contains a lot of noise so not sure how that would effect this " }, { "id": 413255, "postDate": "2018-10-31T15:32:48.320Z", "content": "What is your approach? Creating the pictures and saving them on the drive or creating them on the run?
", "rawMarkdown": "What is your approach? Creating the pictures and saving them on the drive or creating them on the run? \n" }, { "id": 410635, "postDate": "2018-10-26T11:36:49.227Z", "content": "1.5k images / class, LB:0.810
\n\nPlanning to try 5k images and later on maybe 10k images.
", "rawMarkdown": "1.5k images / class, LB:0.810\n\nPlanning to try 5k images and later on maybe 10k images." }, { "id": 410462, "postDate": "2018-10-26T05:18:37.027Z", "content": "Hi Nguyen, is it enough to train the model by one 1080ti?
", "rawMarkdown": "Hi Nguyen, is it enough to train the model by one 1080ti?", "replies": [ { "id": 410494, "postDate": "2018-10-26T06:30:00.383Z", "content": "In my view, If we ues lightweight model and small Input, it is enough for single 1080ti.
", "rawMarkdown": "In my view, If we ues lightweight model and small Input, it is enough for single 1080ti.", "votes": 2 }, { "id": 410504, "postDate": "2018-10-26T06:52:46.700Z", "content": "Thanks, Gary. I'll have a try : )
", "rawMarkdown": "Thanks, Gary. I'll have a try : )" }, { "id": 410706, "postDate": "2018-10-26T13:56:54.443Z", "content": "I am using a 1080Ti. Go ahead, @YourVenn
", "rawMarkdown": "I am using a 1080Ti. Go ahead, @YourVenn ", "votes": 2 }, { "id": 412448, "postDate": "2018-10-30T08:23:39.287Z", "content": "Thanks for your motivation, Nguyen : )
", "rawMarkdown": "Thanks for your motivation, Nguyen : )" } ] }, { "id": 409581, "postDate": "2018-10-24T14:21:59.673Z", "content": "30k samples/class, LB:0.910
", "rawMarkdown": "30k samples/class, LB:0.910" }, { "id": 409463, "postDate": "2018-10-24T10:38:49.517Z", "content": "30k, 0.905
", "rawMarkdown": "30k, 0.905" }, { "id": 409321, "postDate": "2018-10-24T05:47:34.997Z", "content": "Hi all, what do you guys mean are training only several class of object, say owl for each step while training the model?
", "rawMarkdown": "Hi all, what do you guys mean are training only several class of object, say owl for each step while training the model? ", "replies": [ { "id": 409351, "postDate": "2018-10-24T06:50:30.913Z", "content": "Nope. We take 3k, 10k images, etc of each class then shuffle all of them during training.
", "rawMarkdown": "Nope. We take 3k, 10k images, etc of each class then shuffle all of them during training." }, { "id": 409436, "postDate": "2018-10-24T09:41:15.683Z", "content": "I see. Thank. Just curious, if we all of them use different model, how can we measure the correlation between size of dataset and LB score
", "rawMarkdown": "I see. Thank. Just curious, if we all of them use different model, how can we measure the correlation between size of dataset and LB score\n" }, { "id": 409471, "postDate": "2018-10-24T10:56:59.770Z", "content": "Hi Joe, if we all of them use different model, we can't mesure the correlation between size of dataset and LB score. \nThough, when Nguyen says :
\n\n3k images / class, LB: 0.832\n10k images / class, LB: 0.862\n20k images / class, LB: 0.900
\n\nI think it's with the same model. When Data Lu add : \n5k images / class, LB: 0.892\n50k images / class, LB: 0.929
\n\nI think it's safe to conclude that SIZE MATTER ;)
", "rawMarkdown": "Hi Joe, if we all of them use different model, we can't mesure the correlation between size of dataset and LB score. \nThough, when Nguyen says : \n\n3k images / class, LB: 0.832\n10k images / class, LB: 0.862\n20k images / class, LB: 0.900\n\nI think it's with the same model. When Data Lu add : \n5k images / class, LB: 0.892\n50k images / class, LB: 0.929\n\nI think it's safe to conclude that SIZE MATTER ;)" }, { "id": 409530, "postDate": "2018-10-24T12:47:58.940Z", "content": "I see. Thanks for you guys' reply, but if we select the same amount data from each class, the distribution from it will be different from the original distribution. So would it better to examine different portion of data rather than different size, say 10%,20%,30% for each class rather than 5k,10k,25k for each class?
", "rawMarkdown": "I see. Thanks for you guys' reply, but if we select the same amount data from each class, the distribution from it will be different from the original distribution. So would it better to examine different portion of data rather than different size, say 10%,20%,30% for each class rather than 5k,10k,25k for each class?" } ] }, { "id": 408504, "postDate": "2018-10-23T01:50:00.690Z", "content": "GPU makes everything easy.\nMy single model takes 30 Min per epoch at kernel (using k80) whereas it takes about 9 min per epoch at GCP(using p100)
", "rawMarkdown": "GPU makes everything easy.\nMy single model takes 30 Min per epoch at kernel (using k80) whereas it takes about 9 min per epoch at GCP(using p100)", "replies": [ { "id": 413184, "postDate": "2018-10-31T12:33:45.357Z", "content": "wow. mine takes 12 hours for one epoch...
", "rawMarkdown": "wow. mine takes 12 hours for one epoch... " }, { "id": 413689, "postDate": "2018-11-01T11:22:50.057Z", "content": "Did you use cpu ?
", "rawMarkdown": "Did you use cpu ? " } ] }, { "id": 408047, "postDate": "2018-10-22T08:14:07.643Z", "content": "How many models did you use? Single model or ensemble?
", "rawMarkdown": "How many models did you use? Single model or ensemble?", "replies": [ { "id": 408049, "postDate": "2018-10-22T08:25:47.500Z", "content": "Single model only. I think building many models now is not a good plan
", "rawMarkdown": "Single model only. I think building many models now is not a good plan" } ] }, { "id": 417919, "postDate": "2018-11-09T01:24:53.293Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 421428, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-11-15T01:53:52.663000", "content": "Thanks to Nguyen for your motivation and I used my 1080ti for this competition. \nFYI, my current result is:\n- 25K/class, Local 0.887 and Public LB 0.935 without TTA.\n- Will add TTA and I expect a 0.002's boost.\n- By the way, since I haven't tried any fancy techniques, I think 25k/class for 0.940+ is not hard to achieve.
", "votes": 3, "replies": [ { "id": 421495, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-11-15T04:12:14.757000", "content": "@YourVenn, did you use pretrained model, like ResNet 50?
", "votes": 0, "replies": [] }, { "id": 421514, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-11-15T05:00:18.620000", "content": "yes, pre-trained model is useful.
", "votes": 0, "replies": [] }, { "id": 421778, "author_name": "Gary", "author_url": "", "post_date": "2018-11-15T12:05:49.153000", "content": "but I only got local 0.877, weird..
", "votes": 0, "replies": [] }, { "id": 421972, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-15T16:12:17.670000", "content": "But you may not have the same validation set as Venn has, so may be you LB result is close
", "votes": 0, "replies": [] } ] }, { "id": 421448, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-15T02:38:35.510000", "content": "128 * 128 image, all simplified data (but didn't finish one epoch), local val 0.8797, LB 0.935
", "votes": 1, "replies": [] }, { "id": 418572, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-10T06:51:33.763000", "content": "It seems like a case of rapidly diminishing return, for the same model, I got 0.92 for 30k images/class, but only 0.923 for all data T.T
", "votes": 1, "replies": [] }, { "id": 415421, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-05T05:16:43.703000", "content": "30k/class, CNN LB: 0.92
", "votes": 1, "replies": [ { "id": 415460, "author_name": "Gary", "author_url": "", "post_date": "2018-11-05T07:05:34", "content": "nice work, could you share what model did you use?
", "votes": 0, "replies": [] }, { "id": 415461, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-05T07:12:35.477000", "content": "I can't disclose too much except it's a pretrained model.
", "votes": 0, "replies": [] }, { "id": 415499, "author_name": "cab", "author_url": "", "post_date": "2018-11-05T08:24:48.770000", "content": "Impressive result with a small dataset!. Nice work!
", "votes": 0, "replies": [] }, { "id": 415556, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-05T09:59:50.057000", "content": "Sorry I made a mistake in my calculation, it's actually 30k/class -.-'
", "votes": 0, "replies": [] } ] }, { "id": 407973, "author_name": "luudactam", "author_url": "", "post_date": "2018-10-22T05:04:33.193000", "content": "5k images / class, LB: 0.892
\n\n50k images / class, LB: 0.929
", "votes": 1, "replies": [ { "id": 410705, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T13:55:28.923000", "content": "Hi, you got a nice socre. if you don't mind. Could you share what model did you use?
", "votes": 0, "replies": [] }, { "id": 410729, "author_name": "luudactam", "author_url": "", "post_date": "2018-10-26T15:01:01.130000", "content": "I used LSTM + Attention
", "votes": 2, "replies": [] }, { "id": 410735, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-26T15:18:16.917000", "content": "Attention Is All You Need :)
\n\nIt helped a lot on my RNN too :)
", "votes": 1, "replies": [] }, { "id": 410743, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T15:32:03.570000", "content": "Thanks for sharing. Just LSTM? no CNN?
", "votes": 0, "replies": [] }, { "id": 410754, "author_name": "luudactam", "author_url": "", "post_date": "2018-10-26T15:52:15.140000", "content": "It's a mix of LSTM and CNN, but mainly LSTM
", "votes": 0, "replies": [] }, { "id": 410757, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T16:07:19.650000", "content": "Sure, It is a model worth trying
", "votes": 0, "replies": [] }, { "id": 410762, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-26T16:13:01.403000", "content": "@Tam
\nDo you mean 1D CNN on strokes or 2D CNN on images ?
I think there may be some confusion :)
", "votes": 0, "replies": [] }, { "id": 410763, "author_name": "cab", "author_url": "", "post_date": "2018-10-26T16:13:16.710000", "content": "\n\n\nI used LSTM + Attention
\n \nAttention Is All You Need :)
\n
Wow. Amazing!. LSTM/RNN is one of my weakness. I always get the trouble with them :D
", "votes": 0, "replies": [] }, { "id": 414505, "author_name": "Haral", "author_url": "", "post_date": "2018-11-02T22:31:23.583000", "content": "Hi @serigne, I finally manage to get my attention block working. But it gave almost no improvement (in fact it was slightly worst around 0.8 with 20k/class)\nWhere dit you put it ? before/after the batchnorm ? before/after the lstm layers ?\n(Mine was between my 2 lstm layers)
\n\nIf anyone as tips on how to use \"attention\" well I take :)
", "votes": 1, "replies": [] }, { "id": 414733, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-03T13:30:23.773000", "content": "@Haral You have to add the Attention layer after your last LSTM layer (with return_sequence = True )
", "votes": 1, "replies": [] }, { "id": 417480, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-08T10:51:03.227000", "content": "Like Haral, I added attention and it didn't help either.
", "votes": 0, "replies": [] } ] }, { "id": 407987, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-22T05:26:42.373000", "content": "25k images/class, LB: 0.916 (single model)
\n\nYes size of dataset really matters , but 25k seems large enough for me for experiments (particularly because of limitations on kaggle kernels)
\n\nI will focus then on building stronger model before using more data
", "votes": 2, "replies": [ { "id": 408029, "author_name": "cab", "author_url": "", "post_date": "2018-10-22T07:38:21.697000", "content": "I have a similar result and plan as you. I will spend time on the final weeks to train with big data.
", "votes": -1, "replies": [] }, { "id": 408761, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-10-23T12:18:26.943000", "content": "Hi Serigne, have you been only using kaggle kernels so far?
", "votes": 0, "replies": [] }, { "id": 408782, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-23T13:05:55.060000", "content": "Yes kernel only
\n\nHowever the GPU version of kernel has not a good support of Pytorch 4.xx. So I need to stick with keras.
", "votes": 2, "replies": [] }, { "id": 409464, "author_name": "Gary", "author_url": "", "post_date": "2018-10-24T10:39:52", "content": "I think your model is great.
", "votes": 0, "replies": [] }, { "id": 410219, "author_name": "Haral", "author_url": "", "post_date": "2018-10-25T16:10:40.090000", "content": "Can i bother asking if your single model is more CNN or more like RNN or even different ? Thx Serigne ;)
", "votes": 0, "replies": [] }, { "id": 410275, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-25T18:10:17.233000", "content": "Thanks Gary
\n\n@Haral I use convnets on images now. I used earlier a LSTM model on strokes but slower to train and it scored 0.905 for 18k/class.
\n\nMaybe I need to find a way to combine both.
", "votes": 1, "replies": [] }, { "id": 410379, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T00:14:57.453000", "content": "yep, I also need to find a way to combine both. maybe this way work: two branches(CNN for pictures, and LSTM for strokes, and then combine the features from them.)
", "votes": 1, "replies": [] }, { "id": 410443, "author_name": "good good study", "author_url": "", "post_date": "2018-10-26T04:31:48.813000", "content": "what is the structure of your RNN based model? I was only able to get 0.84 LB.
", "votes": 0, "replies": [] }, { "id": 410451, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-26T04:57:06.707000", "content": "CNN achieve LB 0.916??
", "votes": 0, "replies": [] }, { "id": 410668, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-26T12:31:17.197000", "content": "@zjucor my score come from the structure ^^
\n\n@Joe yes, CNN layers in a self designed structure
", "votes": 1, "replies": [] }, { "id": 410698, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-26T13:41:18.847000", "content": "Awesome, I tried residue block, resnet 50, mobilenetV2, but unable to give a great boost beyond a kernel scored 0.892, can you share with us any method to build a better model? For me, i'm just randomly trying different model...
", "votes": 0, "replies": [] }, { "id": 410704, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T13:54:29.950000", "content": "if you use resnet50, this model is too large. It need so much time to train. I this mobilenetV2 can get 0.90+ score.
", "votes": 1, "replies": [] }, { "id": 410734, "author_name": "Serigne ", "author_url": "", "post_date": "2018-10-26T15:16:55.503000", "content": "@Joe \nI am also a relative newbie in Computer vision ( TGS was my first serious competition on it) .
\nAnd just like you many newbies , my NN structure come from trials and errors ( I mean I am not an expert in NN structure design :p )
I second Gary , you should start with light weight architecture and fine tune it, then check your CV/validation score.
", "votes": 0, "replies": [] }, { "id": 410746, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T15:39:03.547000", "content": "yep, I started with light weight architecture. But it may be need some fantastic layers to get a better socre.
", "votes": 0, "replies": [] } ] }, { "id": 408592, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2018-10-23T06:02:34.253000", "content": "6k samples/class, LB: 0.84 (local cv 0.88) using bidirectional lstm
", "votes": 1, "replies": [] }, { "id": 413326, "author_name": "", "author_url": "", "post_date": "2018-10-31T17:55:46.723000", "content": "very cool, in general the larger the better but in this competition the train data itself contains a lot of noise so not sure how that would effect this
", "votes": 0, "replies": [] }, { "id": 413255, "author_name": "Nazim Girach", "author_url": "", "post_date": "2018-10-31T15:32:48.320000", "content": "What is your approach? Creating the pictures and saving them on the drive or creating them on the run?
", "votes": 0, "replies": [] }, { "id": 410635, "author_name": "Fadhli", "author_url": "", "post_date": "2018-10-26T11:36:49.227000", "content": "1.5k images / class, LB:0.810
\n\nPlanning to try 5k images and later on maybe 10k images.
", "votes": 0, "replies": [] }, { "id": 410462, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-10-26T05:18:37.027000", "content": "Hi Nguyen, is it enough to train the model by one 1080ti?
", "votes": 0, "replies": [ { "id": 410494, "author_name": "Gary", "author_url": "", "post_date": "2018-10-26T06:30:00.383000", "content": "In my view, If we ues lightweight model and small Input, it is enough for single 1080ti.
", "votes": 2, "replies": [] }, { "id": 410504, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-10-26T06:52:46.700000", "content": "Thanks, Gary. I'll have a try : )
", "votes": 0, "replies": [] }, { "id": 410706, "author_name": "cab", "author_url": "", "post_date": "2018-10-26T13:56:54.443000", "content": "I am using a 1080Ti. Go ahead, @YourVenn
", "votes": 2, "replies": [] }, { "id": 412448, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-10-30T08:23:39.287000", "content": "Thanks for your motivation, Nguyen : )
", "votes": 0, "replies": [] } ] }, { "id": 409581, "author_name": "LingChen", "author_url": "", "post_date": "2018-10-24T14:21:59.673000", "content": "30k samples/class, LB:0.910
", "votes": 0, "replies": [] }, { "id": 409463, "author_name": "Gary", "author_url": "", "post_date": "2018-10-24T10:38:49.517000", "content": "30k, 0.905
", "votes": 0, "replies": [] }, { "id": 409321, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-24T05:47:34.997000", "content": "Hi all, what do you guys mean are training only several class of object, say owl for each step while training the model?
", "votes": 0, "replies": [ { "id": 409351, "author_name": "cab", "author_url": "", "post_date": "2018-10-24T06:50:30.913000", "content": "Nope. We take 3k, 10k images, etc of each class then shuffle all of them during training.
", "votes": 0, "replies": [] }, { "id": 409436, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-24T09:41:15.683000", "content": "I see. Thank. Just curious, if we all of them use different model, how can we measure the correlation between size of dataset and LB score
", "votes": 0, "replies": [] }, { "id": 409471, "author_name": "Haral", "author_url": "", "post_date": "2018-10-24T10:56:59.770000", "content": "Hi Joe, if we all of them use different model, we can't mesure the correlation between size of dataset and LB score. \nThough, when Nguyen says :
\n\n3k images / class, LB: 0.832\n10k images / class, LB: 0.862\n20k images / class, LB: 0.900
\n\nI think it's with the same model. When Data Lu add : \n5k images / class, LB: 0.892\n50k images / class, LB: 0.929
\n\nI think it's safe to conclude that SIZE MATTER ;)
", "votes": 0, "replies": [] }, { "id": 409530, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-10-24T12:47:58.940000", "content": "I see. Thanks for you guys' reply, but if we select the same amount data from each class, the distribution from it will be different from the original distribution. So would it better to examine different portion of data rather than different size, say 10%,20%,30% for each class rather than 5k,10k,25k for each class?
", "votes": 0, "replies": [] } ] }, { "id": 408504, "author_name": "hahaha", "author_url": "", "post_date": "2018-10-23T01:50:00.690000", "content": "GPU makes everything easy.\nMy single model takes 30 Min per epoch at kernel (using k80) whereas it takes about 9 min per epoch at GCP(using p100)
", "votes": 0, "replies": [ { "id": 413184, "author_name": "yimacs", "author_url": "", "post_date": "2018-10-31T12:33:45.357000", "content": "wow. mine takes 12 hours for one epoch...
", "votes": 0, "replies": [] }, { "id": 413689, "author_name": "hahaha", "author_url": "", "post_date": "2018-11-01T11:22:50.057000", "content": "Did you use cpu ?
", "votes": 0, "replies": [] } ] }, { "id": 408047, "author_name": "Thanh Hau Nguyen", "author_url": "", "post_date": "2018-10-22T08:14:07.643000", "content": "How many models did you use? Single model or ensemble?
", "votes": 0, "replies": [ { "id": 408049, "author_name": "cab", "author_url": "", "post_date": "2018-10-22T08:25:47.500000", "content": "Single model only. I think building many models now is not a good plan
", "votes": 0, "replies": [] } ] }, { "id": 417919, "author_name": "", "author_url": "", "post_date": "2018-11-09T01:24:53.293000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "407964": "Since we have a lot of data for training, it may be a limitation for who does not have enough computer resource. \nI did some experiences and I would like to share how much data I used and corresponding public LB score. \n\n - 3k images / class, LB: 0.832\n - 10k images / class, LB: 0.862\n - 20k images / class, LB: 0.900\n\nCheers,", "421428": "Thanks to Nguyen for your motivation and I used my 1080ti for this competition. \nFYI, my current result is:\n- 25K/class, Local 0.887 and Public LB 0.935 without TTA.\n- Will add TTA and I expect a 0.002's boost.\n- By the way, since I haven't tried any fancy techniques, I think 25k/class for 0.940+ is not hard to achieve.", "421448": "128 * 128 image, all simplified data (but didn't finish one epoch), local val 0.8797, LB 0.935", "418572": "It seems like a case of rapidly diminishing return, for the same model, I got 0.92 for 30k images/class, but only 0.923 for all data T.T", "415421": "30k/class, CNN LB: 0.92", "407973": "5k images / class, LB: 0.892\n\n50k images / class, LB: 0.929", "407987": "25k images/class, LB: 0.916 (single model)\n\nYes size of dataset really matters , but 25k seems large enough for me for experiments (particularly because of limitations on kaggle kernels)\n\nI will focus then on building stronger model before using more data\n\n", "408592": "6k samples/class, LB: 0.84 (local cv 0.88) using bidirectional lstm", "413326": "very cool, in general the larger the better but in this competition the train data itself contains a lot of noise so not sure how that would effect this ", "413255": "What is your approach? Creating the pictures and saving them on the drive or creating them on the run? \n", "410635": "1.5k images / class, LB:0.810\n\nPlanning to try 5k images and later on maybe 10k images.", "410462": "Hi Nguyen, is it enough to train the model by one 1080ti?", "409581": "30k samples/class, LB:0.910", "409463": "30k, 0.905", "409321": "Hi all, what do you guys mean are training only several class of object, say owl for each step while training the model? ", "408504": "GPU makes everything easy.\nMy single model takes 30 Min per epoch at kernel (using k80) whereas it takes about 9 min per epoch at GCP(using p100)", "408047": "How many models did you use? Single model or ensemble?", "417919": "" } }