![\"Pretrained](\"https://image.ibb.co/k8PgTL/Screen-Shot-2018-11-05-at-11-31-42-AM.png\")
Pretrained models available in Keras with model size, parameters and depth information.\n
What dataset are these accuracy values based on?
", "rawMarkdown": "What dataset are these accuracy values based on?" }, { "id": 416709, "postDate": "2018-11-07T06:21:54.997Z", "content": "imagenet
", "rawMarkdown": "imagenet" } ] }, { "id": 415358, "postDate": "2018-11-05T01:24:41.500Z", "content": "this graph tells you which model is most efficient:\nhttps://arxiv.org/pdf/1810.00736.pdf
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/113660/36e5c10e00918f72d404e381d83f1e0c/models.png\")
\n\"Benchmark Analysis of Representative Deep Neural Network Architectures\"
Simone Bianco , Rene Cadene , Luigi Celona and Paolo Napoletano\nhttps:// github.com/CeLuigi/models-comparison.pytorch
", "rawMarkdown": "this graph tells you which model is most efficient:\nhttps://arxiv.org/pdf/1810.00736.pdf\n\n\n ![enter image description here][1]\n\"Benchmark Analysis of Representative Deep Neural Network Architectures\"\n\nSimone Bianco , Rene Cadene , Luigi Celona and Paolo Napoletano\nhttps:// github.com/CeLuigi/models-comparison.pytorch\n\n \n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/113660/36e5c10e00918f72d404e381d83f1e0c/models.png", "votes": 8 }, { "id": 414413, "postDate": "2018-11-02T18:06:19.343Z", "content": "Yes , They helped for me ( at least for good weight initialization ) , thus needed much less epoches to train.
", "rawMarkdown": "Yes , They helped for me ( at least for good weight initialization ) , thus needed much less epoches to train.", "votes": 5 }, { "id": 414031, "postDate": "2018-11-02T01:50:49.243Z", "content": "Well, in my case:\n Pretrained models have a better performance (Resnet50).\n 96x96 got 0.01 LB score better than 64x64, but I haven't tried 128x128.\nI'm wondering whether you use all data or part of it because in my case Resnet50 got at least 0.915 on public LB after only one epoch. BTW, I use Keras w/o any data augmentation right now. Hope these infos help you.
", "rawMarkdown": "Well, in my case:\n Pretrained models have a better performance (Resnet50).\n 96x96 got 0.01 LB score better than 64x64, but I haven't tried 128x128.\nI'm wondering whether you use all data or part of it because in my case Resnet50 got at least 0.915 on public LB after only one epoch. BTW, I use Keras w/o any data augmentation right now. Hope these infos help you. ", "votes": 4, "replies": [ { "id": 414038, "postDate": "2018-11-02T02:08:11.440Z", "content": "Nice work Jingxiao. Is your one epoch's result (0.915) based on all training data?
", "rawMarkdown": "Nice work Jingxiao. Is your one epoch's result (0.915) based on all training data?", "votes": 1 }, { "id": 414055, "postDate": "2018-11-02T02:42:36.563Z", "content": "Yes. All simplified data.
", "rawMarkdown": "Yes. All simplified data.", "votes": 1 }, { "id": 414105, "postDate": "2018-11-02T06:32:46.430Z", "content": "wow, Jingxiao, just one epoch can get 0.915. Did you use keras?
", "rawMarkdown": "wow, Jingxiao, just one epoch can get 0.915. Did you use keras?" }, { "id": 414159, "postDate": "2018-11-02T08:32:31.283Z", "content": "I use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197.
", "rawMarkdown": "I use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197. " }, { "id": 414169, "postDate": "2018-11-02T08:53:10.047Z", "content": "Although Resnet50 has the limit, you could still load pretrained weights by simply removing the limit. It doesn’t matter. If you use Keras, when you use keras.application.resnet50, pass None to the input_shape and train size to the input_tensor. I’m not sure whether the answer makes sense because my English is poor. Haha.
", "rawMarkdown": "Although Resnet50 has the limit, you could still load pretrained weights by simply removing the limit. It doesn’t matter. If you use Keras, when you use keras.application.resnet50, pass None to the input_shape and train size to the input_tensor. I’m not sure whether the answer makes sense because my English is poor. Haha.", "votes": 1 }, { "id": 414176, "postDate": "2018-11-02T09:06:17.240Z", "content": "@Aramis I created a similary topic in TGS competition before, maybe can help you.\nhttps://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65387
", "rawMarkdown": "@Aramis I created a similary topic in TGS competition before, maybe can help you.\nhttps://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65387", "votes": 1 }, { "id": 414200, "postDate": "2018-11-02T10:04:29.193Z", "rawMarkdown": "", "votes": -4, "isDeleted": true }, { "id": 414222, "postDate": "2018-11-02T10:56:53.587Z", "content": "@Jingxiao Gu, how much time it took for completion of 1 epoch with ResNet50?
", "rawMarkdown": "@Jingxiao Gu, how much time it took for completion of 1 epoch with ResNet50? ", "votes": 1 }, { "id": 414326, "postDate": "2018-11-02T14:58:23.530Z", "content": "That's what I just want to ask haha
", "rawMarkdown": "That's what I just want to ask haha" }, { "id": 414363, "postDate": "2018-11-02T15:56:38.337Z", "content": "@Jingxiao, thanks for the hint, I will give a try
", "rawMarkdown": "@Jingxiao, thanks for the hint, I will give a try" }, { "id": 414369, "postDate": "2018-11-02T16:04:59.153Z", "content": "That's the big question
", "rawMarkdown": "That's the big question " }, { "id": 414411, "postDate": "2018-11-02T17:59:52.380Z", "content": "How did you use the pretrained weights for the single channel inputs? Just triple the single channel or you have some other ways to trim the pretrained weight?
", "rawMarkdown": "How did you use the pretrained weights for the single channel inputs? Just triple the single channel or you have some other ways to trim the pretrained weight?" }, { "id": 414419, "postDate": "2018-11-02T18:15:28.857Z", "content": "I tripled the single channel (but just inside the network , to not increase the memory)
", "rawMarkdown": "I tripled the single channel (but just inside the network , to not increase the memory) ", "votes": 4 }, { "id": 415138, "postDate": "2018-11-04T13:50:14.023Z", "content": "Can you guys share your path of finding a good model... I tried a SENet MobileNet with Residual Block, I spent a whole week on designing a better model, but still couldn't find a good model, I think I must followed a wrong approach in finding model...
", "rawMarkdown": "Can you guys share your path of finding a good model... I tried a SENet MobileNet with Residual Block, I spent a whole week on designing a better model, but still couldn't find a good model, I think I must followed a wrong approach in finding model..." }, { "id": 415148, "postDate": "2018-11-04T14:22:38.107Z", "content": "Honestly, there is no trick finding good single model.
\n\nSimple resnet50 with pretrained weight and 25k samples per class is enough to obtain 0.92 in public LB
", "rawMarkdown": "Honestly, there is no trick finding good single model.\n\nSimple resnet50 with pretrained weight and 25k samples per class is enough to obtain 0.92 in public LB", "votes": 3 }, { "id": 415339, "postDate": "2018-11-04T23:51:44.113Z", "content": "Hi wh1te, thanks for your kind advice, did you preprocess the image? As images used in Resnet50 have different mean as the images we use here.
", "rawMarkdown": "Hi wh1te, thanks for your kind advice, did you preprocess the image? As images used in Resnet50 have different mean as the images we use here." }, { "id": 415849, "postDate": "2018-11-05T19:58:44.747Z", "content": "Update your keras, the new min size limit for ResNet50 is only 32x32.
\n\n\n", "rawMarkdown": "Update your keras, the new min size limit for ResNet50 is only 32x32. \n\n> **Aramis wrote**\n> \n> > I use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197. ", "votes": 1 }, { "id": 415899, "postDate": "2018-11-05T22:02:24.690Z", "content": "Aramis wrote
\n \n\n\nI use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197.
\n
Thank you @James, Indeed I was using an older version of Keras and recently, I came to know about this change.
", "rawMarkdown": "Thank you @James, Indeed I was using an older version of Keras and recently, I came to know about this change. " } ] }, { "id": 419684, "postDate": "2018-11-12T11:51:01.630Z", "content": "Pretrained ResNet 50 works for me.
", "rawMarkdown": "Pretrained ResNet 50 works for me.", "votes": 1 }, { "id": 414218, "postDate": "2018-11-02T10:44:20.450Z", "content": "I also wondered 128x128 could be better than 64x64
", "rawMarkdown": "I also wondered 128x128 could be better than 64x64", "votes": 1, "replies": [ { "id": 414415, "postDate": "2018-11-02T18:09:28.180Z", "content": "I got 0.928 on LB with single model and just 64x64 . Never tried bigger size , but I will, to see if it improves.
", "rawMarkdown": "I got 0.928 on LB with single model and just 64x64 . Never tried bigger size , but I will, to see if it improves.", "votes": 7 }, { "id": 414508, "postDate": "2018-11-02T22:40:50.350Z", "content": "Are you still working with your 25k/class @Serigne ? (0.928 looks too damn high)
", "rawMarkdown": "Are you still working with your 25k/class @Serigne ? (0.928 looks too damn high)", "votes": 1 }, { "id": 414574, "postDate": "2018-11-03T04:06:52.957Z", "content": "@Serigne - may I ask if this is still achievable within the restrictions of the kaggle kernels? or did you go outside of kernels?
", "rawMarkdown": "@Serigne - may I ask if this is still achievable within the restrictions of the kaggle kernels? or did you go outside of kernels?", "votes": 1 }, { "id": 414724, "postDate": "2018-11-03T13:20:09.403Z", "content": "@Haral , I use now much bigger size, Thanksfully I have one lightweight model which can be trained reasonnably fast.
", "rawMarkdown": "@Haral , I use now much bigger size, Thanksfully I have one lightweight model which can be trained reasonnably fast. \n\n" }, { "id": 414730, "postDate": "2018-11-03T13:26:57.680Z", "content": "@RDizzl3 Only the data generation (muliple CSVs files ) is done outside kernel ( due to limited 5Gb output size of kernel, although this can be handled with multiple kernels ) . Otherwise all the rest is done on kernels.
\n\nBeware Kernel has k80 (which is not the most powerful ^^) . But, Given it's free and you can start many kernels, close the browser and do something else :)
", "rawMarkdown": "@RDizzl3 Only the data generation (muliple CSVs files ) is done outside kernel ( due to limited 5Gb output size of kernel, although this can be handled with multiple kernels ) . Otherwise all the rest is done on kernels.\n\nBeware Kernel has k80 (which is not the most powerful ^^) . But, Given it's free and you can start many kernels, close the browser and do something else :)", "votes": 2 } ] }, { "id": 414067, "postDate": "2018-11-02T03:28:47.157Z", "content": "I'm training with incremental chunks of the dataset (1% -> 5% -> 10% -> 50%) for 128x128 image size.\nResNet50 has shown a consistent improvement when doing this. \nI'm using the starter kit shared by @radek
", "rawMarkdown": "I'm training with incremental chunks of the dataset (1% -> 5% -> 10% -> 50%) for 128x128 image size.\nResNet50 has shown a consistent improvement when doing this. \nI'm using the [starter kit](https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/69409) shared by @radek ", "votes": 1 }, { "id": 414148, "postDate": "2018-11-02T08:07:15.537Z", "content": "My experiment result:
\n\nsingle Resnet50 with pretrained weights outperform without pretrain.\nsize 64x64 to 128x128 get performance boost about 0.01
", "rawMarkdown": "My experiment result:\n\nsingle Resnet50 with pretrained weights outperform without pretrain.\nsize 64x64 to 128x128 get performance boost about 0.01", "votes": 2, "replies": [ { "id": 414164, "postDate": "2018-11-02T08:43:34.770Z", "content": "Thanks for sharing, I guess am doing something wrong. However, in my last experiment 128x128 also boost by 0.01.
", "rawMarkdown": "Thanks for sharing, I guess am doing something wrong. However, in my last experiment 128x128 also boost by 0.01. " }, { "id": 414969, "postDate": "2018-11-04T01:55:28.510Z", "content": "Resnet50 takes forever to train. I am puzzled I used the data generator for both training and val data (of course from separate sets of CSVs), I achieved something like: val_accuracy 0.83 and top_3_accuracy of 0.96 but when I submitted, the LB score was only 0.88 which is very disappointing!
", "rawMarkdown": "Resnet50 takes forever to train. I am puzzled I used the data generator for both training and val data (of course from separate sets of CSVs), I achieved something like: val_accuracy 0.83 and top_3_accuracy of 0.96 but when I submitted, the LB score was only 0.88 which is very disappointing!" }, { "id": 414999, "postDate": "2018-11-04T05:04:20.357Z", "content": "@HuyenNguyen, check you model structure, hyper-parameters, etc as it looks like you are over fitting. I have similar issues with a couple of my models and fixed one of them by doing exactly that.
", "rawMarkdown": "@HuyenNguyen, check you model structure, hyper-parameters, etc as it looks like you are over fitting. I have similar issues with a couple of my models and fixed one of them by doing exactly that." }, { "id": 415166, "postDate": "2018-11-04T15:02:49.967Z", "content": "Hi @wh1te, did you use pytorch to train your Resnet50?
\nSince I can't find any pretrained weight with image size smaller than 197x197x3 in Keras
Yes, I use pytorch framework.
\n\ntorchvision provide Resnet50 pretrained weights which could fit any image size.
", "rawMarkdown": "Yes, I use pytorch framework.\n\ntorchvision provide Resnet50 pretrained weights which could fit any image size." }, { "id": 416009, "postDate": "2018-11-06T02:59:09.360Z", "content": "Yes I believe it is a mistake too but I can't see why.
\n\ndef image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n\n\n while True:\n for k in np.random.permutation(ks):\n filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n for df in pd.read_csv(filename, chunksize=batchsize):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n df = df[df.recognized == True]\n x = np.zeros((len(df), size, size, 1))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw,\n time_color=time_color)\n x = np.repeat(x,3, axis = 3)\n x = preprocess_input(x).astype(np.float32)\n y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n yield x, y\ntrain_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10))\nval_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10, NCSVS))\n\n\nmodel.fit_generator(\n train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n validation_data= val_datagen, validation_steps=100,\n callbacks = callbacks\n)\nbased on Beluga's kernel. Can you spot what I did wrong?
", "rawMarkdown": "Yes I believe it is a mistake too but I can't see why.\n\n \n\n def image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n \n \n while True:\n for k in np.random.permutation(ks):\n filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n for df in pd.read_csv(filename, chunksize=batchsize):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n df = df[df.recognized == True]\n x = np.zeros((len(df), size, size, 1))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw,\n time_color=time_color)\n x = np.repeat(x,3, axis = 3)\n x = preprocess_input(x).astype(np.float32)\n y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n yield x, y\n train_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10))\n val_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10, NCSVS))\n\n\n model.fit_generator(\n train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n validation_data= val_datagen, validation_steps=100,\n callbacks = callbacks\n )\n\nbased on Beluga's kernel. Can you spot what I did wrong? " }, { "id": 416077, "postDate": "2018-11-06T06:41:08.910Z", "content": "@HuyenNguyen you're using only recognized images .
", "rawMarkdown": "@HuyenNguyen you're using only recognized images ." }, { "id": 416134, "postDate": "2018-11-06T08:42:28.923Z", "content": "@HuyenNguyen as @Data Luu mentioned, why are you using only recognized images?
", "rawMarkdown": "@HuyenNguyen as @Data Luu mentioned, why are you using only recognized images?" }, { "id": 416230, "postDate": "2018-11-06T12:40:36.473Z", "content": "oops, I didn't see that. I used it at some stage, then removed it, but this copying pasting code from one place to another I forgot to remove it here. Thank you!
", "rawMarkdown": "oops, I didn't see that. I used it at some stage, then removed it, but this copying pasting code from one place to another I forgot to remove it here. Thank you!" }, { "id": 416372, "postDate": "2018-11-06T15:46:13.827Z", "content": "I have a question: Does using unrecognized images get better results?
", "rawMarkdown": "I have a question: Does using unrecognized images get better results?" } ] }, { "id": 419196, "postDate": "2018-11-11T13:41:58.203Z", "content": "What is the best pre-trained model did you use?
", "rawMarkdown": "What is the best pre-trained model did you use?", "replies": [ { "id": 419413, "postDate": "2018-11-11T23:34:44.623Z", "content": "Using Xceptionnet instead of mobilenet increased my score from 0.897 to 0.906 on public LB, experiment other models that are supposedly better such as InceptioResNetV2 but my validation score is worse... I'm using keras fyi
", "rawMarkdown": "Using Xceptionnet instead of mobilenet increased my score from 0.897 to 0.906 on public LB, experiment other models that are supposedly better such as InceptioResNetV2 but my validation score is worse... I'm using keras fyi" }, { "id": 419423, "postDate": "2018-11-11T23:59:05Z", "content": "InceptioResNetV2 is too big.
", "rawMarkdown": "InceptioResNetV2 is too big.", "votes": 1 } ] }, { "id": 415993, "postDate": "2018-11-06T02:38:08.690Z", "content": "Hi, kagglers! What is the best loss function did you use?
", "rawMarkdown": "Hi, kagglers! What is the best loss function did you use?", "replies": [ { "id": 416031, "postDate": "2018-11-06T04:05:10.933Z", "content": "Categorical cross entropy isn't the best one, I guess we'll have to devise custom loss in this case,there's too much noise in data. Also @Hengck23 has highlighted this issue here
", "rawMarkdown": "Categorical cross entropy isn't the best one, I guess we'll have to devise custom loss in this case,there's too much noise in data. Also @Hengck23 has highlighted this issue [here](https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/70585)" } ] }, { "id": 414037, "postDate": "2018-11-02T02:05:34.590Z", "content": "Hi Aramis, thanks for sharing. how many folds do you use in your cv?
", "rawMarkdown": "Hi Aramis, thanks for sharing. how many folds do you use in your cv? ", "replies": [ { "id": 414158, "postDate": "2018-11-02T08:27:57.150Z", "content": "Well, I run for 10-folds based on Bulga's shuffling CSV files. In every fold, you can validate on a new shuffled csv.
", "rawMarkdown": "Well, I run for 10-folds based on Bulga's shuffling CSV files. In every fold, you can validate on a new shuffled csv. " }, { "id": 414178, "postDate": "2018-11-02T09:07:00.007Z", "content": "Thanks, Aramis.
", "rawMarkdown": "Thanks, Aramis. " } ] }, { "id": 414393, "postDate": "2018-11-02T17:08:19.563Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 415444, "author_name": "Rajesh Shreedhar", "author_url": "", "post_date": "2018-11-05T06:04:57.580000", "content": "Pretrained models available in Keras with model size, parameters and depth information.\n
What dataset are these accuracy values based on?
", "votes": 0, "replies": [] }, { "id": 416709, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-07T06:21:54.997000", "content": "imagenet
", "votes": 0, "replies": [] } ] }, { "id": 415358, "author_name": "hengck23", "author_url": "", "post_date": "2018-11-05T01:24:41.500000", "content": "this graph tells you which model is most efficient:\nhttps://arxiv.org/pdf/1810.00736.pdf
\n\n\n\"Benchmark Analysis of Representative Deep Neural Network Architectures\"
Simone Bianco , Rene Cadene , Luigi Celona and Paolo Napoletano\nhttps:// github.com/CeLuigi/models-comparison.pytorch
", "votes": 8, "replies": [] }, { "id": 414413, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-02T18:06:19.343000", "content": "Yes , They helped for me ( at least for good weight initialization ) , thus needed much less epoches to train.
", "votes": 5, "replies": [] }, { "id": 414031, "author_name": "seigato", "author_url": "", "post_date": "2018-11-02T01:50:49.243000", "content": "Well, in my case:\n Pretrained models have a better performance (Resnet50).\n 96x96 got 0.01 LB score better than 64x64, but I haven't tried 128x128.\nI'm wondering whether you use all data or part of it because in my case Resnet50 got at least 0.915 on public LB after only one epoch. BTW, I use Keras w/o any data augmentation right now. Hope these infos help you.
", "votes": 4, "replies": [ { "id": 414038, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-11-02T02:08:11.440000", "content": "Nice work Jingxiao. Is your one epoch's result (0.915) based on all training data?
", "votes": 1, "replies": [] }, { "id": 414055, "author_name": "seigato", "author_url": "", "post_date": "2018-11-02T02:42:36.563000", "content": "Yes. All simplified data.
", "votes": 1, "replies": [] }, { "id": 414105, "author_name": "Gary", "author_url": "", "post_date": "2018-11-02T06:32:46.430000", "content": "wow, Jingxiao, just one epoch can get 0.915. Did you use keras?
", "votes": 0, "replies": [] }, { "id": 414159, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-02T08:32:31.283000", "content": "I use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197.
", "votes": 0, "replies": [] }, { "id": 414169, "author_name": "seigato", "author_url": "", "post_date": "2018-11-02T08:53:10.047000", "content": "Although Resnet50 has the limit, you could still load pretrained weights by simply removing the limit. It doesn’t matter. If you use Keras, when you use keras.application.resnet50, pass None to the input_shape and train size to the input_tensor. I’m not sure whether the answer makes sense because my English is poor. Haha.
", "votes": 1, "replies": [] }, { "id": 414176, "author_name": "Gary", "author_url": "", "post_date": "2018-11-02T09:06:17.240000", "content": "@Aramis I created a similary topic in TGS competition before, maybe can help you.\nhttps://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65387
", "votes": 1, "replies": [] }, { "id": 414200, "author_name": "", "author_url": "", "post_date": "2018-11-02T10:04:29.193000", "content": "", "votes": -4, "replies": [] }, { "id": 414222, "author_name": "Rajesh Shreedhar", "author_url": "", "post_date": "2018-11-02T10:56:53.587000", "content": "@Jingxiao Gu, how much time it took for completion of 1 epoch with ResNet50?
", "votes": 1, "replies": [] }, { "id": 414326, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-11-02T14:58:23.530000", "content": "That's what I just want to ask haha
", "votes": 0, "replies": [] }, { "id": 414363, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-02T15:56:38.337000", "content": "@Jingxiao, thanks for the hint, I will give a try
", "votes": 0, "replies": [] }, { "id": 414369, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-02T16:04:59.153000", "content": "That's the big question
", "votes": 0, "replies": [] }, { "id": 414411, "author_name": "Kulbear", "author_url": "", "post_date": "2018-11-02T17:59:52.380000", "content": "How did you use the pretrained weights for the single channel inputs? Just triple the single channel or you have some other ways to trim the pretrained weight?
", "votes": 0, "replies": [] }, { "id": 414419, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-02T18:15:28.857000", "content": "I tripled the single channel (but just inside the network , to not increase the memory)
", "votes": 4, "replies": [] }, { "id": 415138, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-11-04T13:50:14.023000", "content": "Can you guys share your path of finding a good model... I tried a SENet MobileNet with Residual Block, I spent a whole week on designing a better model, but still couldn't find a good model, I think I must followed a wrong approach in finding model...
", "votes": 0, "replies": [] }, { "id": 415148, "author_name": "wh1te", "author_url": "", "post_date": "2018-11-04T14:22:38.107000", "content": "Honestly, there is no trick finding good single model.
\n\nSimple resnet50 with pretrained weight and 25k samples per class is enough to obtain 0.92 in public LB
", "votes": 3, "replies": [] }, { "id": 415339, "author_name": "Joe Ho", "author_url": "", "post_date": "2018-11-04T23:51:44.113000", "content": "Hi wh1te, thanks for your kind advice, did you preprocess the image? As images used in Resnet50 have different mean as the images we use here.
", "votes": 0, "replies": [] }, { "id": 415849, "author_name": "James Requa", "author_url": "", "post_date": "2018-11-05T19:58:44.747000", "content": "Update your keras, the new min size limit for ResNet50 is only 32x32.
\n\n\n", "votes": 1, "replies": [] }, { "id": 415899, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-05T22:02:24.690000", "content": "Aramis wrote
\n \n\n\nI use part of the data, but it would be silly to ask, how did you deal with minimum image size problem with ResNet50? Because, when you use pretrained, at least the image dimension should be 197x197.
\n
Thank you @James, Indeed I was using an older version of Keras and recently, I came to know about this change.
", "votes": 0, "replies": [] } ] }, { "id": 419684, "author_name": "Tommy Jiang", "author_url": "", "post_date": "2018-11-12T11:51:01.630000", "content": "Pretrained ResNet 50 works for me.
", "votes": 1, "replies": [] }, { "id": 414218, "author_name": "hahaha", "author_url": "", "post_date": "2018-11-02T10:44:20.450000", "content": "I also wondered 128x128 could be better than 64x64
", "votes": 1, "replies": [ { "id": 414415, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-02T18:09:28.180000", "content": "I got 0.928 on LB with single model and just 64x64 . Never tried bigger size , but I will, to see if it improves.
", "votes": 7, "replies": [] }, { "id": 414508, "author_name": "Haral", "author_url": "", "post_date": "2018-11-02T22:40:50.350000", "content": "Are you still working with your 25k/class @Serigne ? (0.928 looks too damn high)
", "votes": 1, "replies": [] }, { "id": 414574, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-11-03T04:06:52.957000", "content": "@Serigne - may I ask if this is still achievable within the restrictions of the kaggle kernels? or did you go outside of kernels?
", "votes": 1, "replies": [] }, { "id": 414724, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-03T13:20:09.403000", "content": "@Haral , I use now much bigger size, Thanksfully I have one lightweight model which can be trained reasonnably fast.
", "votes": 0, "replies": [] }, { "id": 414730, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-03T13:26:57.680000", "content": "@RDizzl3 Only the data generation (muliple CSVs files ) is done outside kernel ( due to limited 5Gb output size of kernel, although this can be handled with multiple kernels ) . Otherwise all the rest is done on kernels.
\n\nBeware Kernel has k80 (which is not the most powerful ^^) . But, Given it's free and you can start many kernels, close the browser and do something else :)
", "votes": 2, "replies": [] } ] }, { "id": 414067, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-11-02T03:28:47.157000", "content": "I'm training with incremental chunks of the dataset (1% -> 5% -> 10% -> 50%) for 128x128 image size.\nResNet50 has shown a consistent improvement when doing this. \nI'm using the starter kit shared by @radek
", "votes": 1, "replies": [] }, { "id": 414148, "author_name": "wh1te", "author_url": "", "post_date": "2018-11-02T08:07:15.537000", "content": "My experiment result:
\n\nsingle Resnet50 with pretrained weights outperform without pretrain.\nsize 64x64 to 128x128 get performance boost about 0.01
", "votes": 2, "replies": [ { "id": 414164, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-02T08:43:34.770000", "content": "Thanks for sharing, I guess am doing something wrong. However, in my last experiment 128x128 also boost by 0.01.
", "votes": 0, "replies": [] }, { "id": 414969, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-04T01:55:28.510000", "content": "Resnet50 takes forever to train. I am puzzled I used the data generator for both training and val data (of course from separate sets of CSVs), I achieved something like: val_accuracy 0.83 and top_3_accuracy of 0.96 but when I submitted, the LB score was only 0.88 which is very disappointing!
", "votes": 0, "replies": [] }, { "id": 414999, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-11-04T05:04:20.357000", "content": "@HuyenNguyen, check you model structure, hyper-parameters, etc as it looks like you are over fitting. I have similar issues with a couple of my models and fixed one of them by doing exactly that.
", "votes": 0, "replies": [] }, { "id": 415166, "author_name": "musicmilif", "author_url": "", "post_date": "2018-11-04T15:02:49.967000", "content": "Hi @wh1te, did you use pytorch to train your Resnet50?
\nSince I can't find any pretrained weight with image size smaller than 197x197x3 in Keras
Yes, I use pytorch framework.
\n\ntorchvision provide Resnet50 pretrained weights which could fit any image size.
", "votes": 0, "replies": [] }, { "id": 416009, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-06T02:59:09.360000", "content": "Yes I believe it is a mistake too but I can't see why.
\n\ndef image_generator_xd(size, batchsize, ks, lw=6, time_color=True):\n\n\n while True:\n for k in np.random.permutation(ks):\n filename = os.path.join(DP_DIR, 'train_k{}.csv.gz'.format(k))\n for df in pd.read_csv(filename, chunksize=batchsize):\n df['drawing'] = df['drawing'].apply(ast.literal_eval)\n df = df[df.recognized == True]\n x = np.zeros((len(df), size, size, 1))\n for i, raw_strokes in enumerate(df.drawing.values):\n x[i, :, :, 0] = draw_cv2(raw_strokes, size=size, lw=lw,\n time_color=time_color)\n x = np.repeat(x,3, axis = 3)\n x = preprocess_input(x).astype(np.float32)\n y = keras.utils.to_categorical(df.y, num_classes=NCATS)\n yield x, y\ntrain_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10))\nval_datagen = image_generator_xd(size=96, batchsize=batchsize, ks=range(NCSVS - 10, NCSVS))\n\n\nmodel.fit_generator(\n train_datagen, steps_per_epoch=STEPS, epochs=EPOCHS, verbose=1,\n validation_data= val_datagen, validation_steps=100,\n callbacks = callbacks\n)\nbased on Beluga's kernel. Can you spot what I did wrong?
", "votes": 0, "replies": [] }, { "id": 416077, "author_name": "luudactam", "author_url": "", "post_date": "2018-11-06T06:41:08.910000", "content": "@HuyenNguyen you're using only recognized images .
", "votes": 0, "replies": [] }, { "id": 416134, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-06T08:42:28.923000", "content": "@HuyenNguyen as @Data Luu mentioned, why are you using only recognized images?
", "votes": 0, "replies": [] }, { "id": 416230, "author_name": "HuyenNguyen", "author_url": "", "post_date": "2018-11-06T12:40:36.473000", "content": "oops, I didn't see that. I used it at some stage, then removed it, but this copying pasting code from one place to another I forgot to remove it here. Thank you!
", "votes": 0, "replies": [] }, { "id": 416372, "author_name": "Thanh Hau Nguyen", "author_url": "", "post_date": "2018-11-06T15:46:13.827000", "content": "I have a question: Does using unrecognized images get better results?
", "votes": 0, "replies": [] } ] }, { "id": 419196, "author_name": "Gary", "author_url": "", "post_date": "2018-11-11T13:41:58.203000", "content": "What is the best pre-trained model did you use?
", "votes": 0, "replies": [ { "id": 419413, "author_name": "Brian Lee", "author_url": "", "post_date": "2018-11-11T23:34:44.623000", "content": "Using Xceptionnet instead of mobilenet increased my score from 0.897 to 0.906 on public LB, experiment other models that are supposedly better such as InceptioResNetV2 but my validation score is worse... I'm using keras fyi
", "votes": 0, "replies": [] }, { "id": 419423, "author_name": "Gary", "author_url": "", "post_date": "2018-11-11T23:59:05", "content": "InceptioResNetV2 is too big.
", "votes": 1, "replies": [] } ] }, { "id": 415993, "author_name": "Gary", "author_url": "", "post_date": "2018-11-06T02:38:08.690000", "content": "Hi, kagglers! What is the best loss function did you use?
", "votes": 0, "replies": [ { "id": 416031, "author_name": "Prajjwal", "author_url": "", "post_date": "2018-11-06T04:05:10.933000", "content": "Categorical cross entropy isn't the best one, I guess we'll have to devise custom loss in this case,there's too much noise in data. Also @Hengck23 has highlighted this issue here
", "votes": 0, "replies": [] } ] }, { "id": 414037, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-11-02T02:05:34.590000", "content": "Hi Aramis, thanks for sharing. how many folds do you use in your cv?
", "votes": 0, "replies": [ { "id": 414158, "author_name": "Aramis", "author_url": "", "post_date": "2018-11-02T08:27:57.150000", "content": "Well, I run for 10-folds based on Bulga's shuffling CSV files. In every fold, you can validate on a new shuffled csv.
", "votes": 0, "replies": [] }, { "id": 414178, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2018-11-02T09:07:00.007000", "content": "Thanks, Aramis.
", "votes": 0, "replies": [] } ] }, { "id": 414393, "author_name": "", "author_url": "", "post_date": "2018-11-02T17:08:19.563000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "413984": " - Deep networks like Vgg16 and ResNet50 couldn't get a better result\n than MobileNet. \n - Pretrained nets did not help at all (faster model convergence but stuck after a couple of more epochs).\n - Cross-Validation improved the LB score with 0.04. \n - 128x128 couldn't perform better than 64x64.", "415444": "Pretrained models available in Keras with model size, parameters and depth information.\n![Pretrained models available in Keras with model size, parameters and depth information.][1]\n [1]: https://image.ibb.co/k8PgTL/Screen-Shot-2018-11-05-at-11-31-42-AM.png", "415358": "this graph tells you which model is most efficient:\nhttps://arxiv.org/pdf/1810.00736.pdf\n\n\n ![enter image description here][1]\n\"Benchmark Analysis of Representative Deep Neural Network Architectures\"\n\nSimone Bianco , Rene Cadene , Luigi Celona and Paolo Napoletano\nhttps:// github.com/CeLuigi/models-comparison.pytorch\n\n \n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/113660/36e5c10e00918f72d404e381d83f1e0c/models.png", "414413": "Yes , They helped for me ( at least for good weight initialization ) , thus needed much less epoches to train.", "414031": "Well, in my case:\n Pretrained models have a better performance (Resnet50).\n 96x96 got 0.01 LB score better than 64x64, but I haven't tried 128x128.\nI'm wondering whether you use all data or part of it because in my case Resnet50 got at least 0.915 on public LB after only one epoch. BTW, I use Keras w/o any data augmentation right now. Hope these infos help you. ", "419684": "Pretrained ResNet 50 works for me.", "414218": "I also wondered 128x128 could be better than 64x64", "414067": "I'm training with incremental chunks of the dataset (1% -> 5% -> 10% -> 50%) for 128x128 image size.\nResNet50 has shown a consistent improvement when doing this. \nI'm using the [starter kit](https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/69409) shared by @radek ", "414148": "My experiment result:\n\nsingle Resnet50 with pretrained weights outperform without pretrain.\nsize 64x64 to 128x128 get performance boost about 0.01", "419196": "What is the best pre-trained model did you use?", "415993": "Hi, kagglers! What is the best loss function did you use?", "414037": "Hi Aramis, thanks for sharing. how many folds do you use in your cv? ", "414393": "" } }