This post will be updated as i move alone.
\n\nreference: https://www.kaggle.com/kmader/quickdraw-simple-models
\n\nobjective is to create baseline pytorch code for the following:
\n\nsimple cnn model on image
simple rnn model on strokes (aka, tensorflow tutorial)
refer to: https://drive.google.com/open?id=17fyH9MQ49hz1teCJ_jWZvo78ju52s5ut
\n\nversion: 2018-10-08
\n\nversion: 2018-10-10
\n\nThis post will be updated as i move alone.
\n\nreference: https://www.kaggle.com/kmader/quickdraw-simple-models
\n\nobjective is to create baseline pytorch code for the following:
\n\nsimple cnn model on image
simple rnn model on strokes (aka, tensorflow tutorial)
refer to: https://drive.google.com/open?id=17fyH9MQ49hz1teCJ_jWZvo78ju52s5ut
\n\nversion: 2018-10-08
\n\nversion: 2018-10-10
\n\n\"Sketch Classification with Neural Networks - A Comparative Study of CNN and RNN on the QuickDraw! data set\"
\n\nhttps://uu.diva-portal.org/smash/get/diva2:1218490/FULLTEXT01.pdf
", "rawMarkdown": "\"Sketch Classification with Neural Networks - A Comparative Study of CNN and RNN on the QuickDraw! data set\"\n\nhttps://uu.diva-portal.org/smash/get/diva2:1218490/FULLTEXT01.pdf", "votes": 3, "replies": [ { "id": 429738, "postDate": "2018-11-29T09:27:51.563Z", "content": "Hi CherKeng,\nI just aware that in your starter kit, you didn't normalize images to [0 1], then normalize images using imagenet mean and std, it convert image into strange distribution
\n\nBut what surprise me is that, the network still converge, my questions are:\n1. Its still better to normalize images, which can get better score?\n2. Is that because doodle images are simple so Batch Normalization can do the job itself?\nThanks,
", "rawMarkdown": "Hi CherKeng,\nI just aware that in your starter kit, you didn't normalize images to [0 1], then normalize images using imagenet mean and std, it convert image into strange distribution\n\nBut what surprise me is that, the network still converge, my questions are:\n1. Its still better to normalize images, which can get better score?\n2. Is that because doodle images are simple so Batch Normalization can do the job itself?\nThanks," } ] }, { "id": 400900, "postDate": "2018-10-09T05:33:41.683Z", "content": "I wasted 5 hours to train my model to archive my 0.7LB. Your model takes only 13min. Very impressive!.
\nThank you
Hi, Heng. how much data did you use? ?k images / class
", "rawMarkdown": "Hi, Heng. how much data did you use? ?k images / class", "votes": 1 }, { "id": 400479, "postDate": "2018-10-08T11:44:26.467Z", "content": "latest version: 2018-10-08
\n\nsee readme.ppt attached
", "rawMarkdown": "latest version: 2018-10-08\n\n- simple cnn on image 32x32 (lb 0.723)\n\nsee readme.ppt attached\n ", "votes": 2 }, { "id": 422902, "postDate": "2018-11-17T02:21:20.233Z", "content": "Hi, thanks for your code. when I train resnet50 on 224x224 using batchsize of 240(8*30). I have got very wired validation results. see below
\n\n0.0037 70.0 0.3 | 14.243 0.003 0.009 (0.005)* | 0.865 0.776 0.915 (0.840) | 18 hr 19 min\n0.0036 71.0 0.3 | 6.594 0.003 0.009 (0.005) | 0.819 0.790 0.920 (0.850) | 18 hr 35 min\n0.0036 72.0 0.3 | 9.629 0.003 0.009 (0.005) | 0.850 0.784 0.914 (0.843) | 18 hr 51 min\n0.0035 73.0 0.4 | 6.732 0.003 0.009 (0.005) | 0.810 0.795 0.918 (0.851) | 19 hr 07 min\n0.0035 74.0 0.4 | 11.131 0.003 0.009 (0.005) | 0.825 0.791 0.920 (0.850) | 19 hr 24 min\n0.0034 75.0 0.4 | 6.489 0.003 0.009 (0.006)* | 0.815 0.793 0.926 (0.854) | 19 hr 40 min\n0.0034 76.0 0.4 | 6.794 0.003 0.008 (0.005) | 0.847 0.787 0.913 (0.844) | 19 hr 56 min\n0.0033 77.0 0.4 | 6.468 0.003 0.009 (0.005) | 0.809 0.794 0.918 (0.851) | 20 hr 12 min\n0.0033 78.0 0.4 | 6.231 0.006 0.017 (0.010) | 0.811 0.795 0.915 (0.850) | 20 hr 28 min\n0.0032 79.0 0.4 | 6.483 0.003 0.009 (0.006) | 0.805 0.795 0.922 (0.853) | 20 hr 43 min\n0.0032 80.0 0.4 | 8.119 0.003 0.009 (0.005)* | 0.831 0.789 0.916 (0.847) | 20 hr 59 min\n0.0031 81.0 0.4 | 7.981 0.003 0.009 (0.005) | 0.809 0.791 0.919 (0.849) | 21 hr 15 min\n0.0031 82.0 0.4 | 8.882 0.003 0.009 (0.005) | 0.775 0.794 0.924 (0.854) | 21 hr 30 min\n0.0031 83.0 0.4 | 3.349 0.269 0.460 (0.352) | 0.819 0.789 0.918 (0.849) | 21 hr 46 min\n0.0030 84.0 0.4 | 7.280 0.003 0.009 (0.005) | 0.819 0.800 0.919 (0.855) | 22 hr 02 min\n0.0030 85.0 0.4 | 6.586 0.003 0.009 (0.005)* | 0.789 0.799 0.925 (0.857) | 22 hr 17 min\n0.0029 86.0 0.4 | 7.680 0.004 0.009 (0.006) | 0.763 0.797 0.931 (0.859) | 22 hr 33 min\nThe training accuracy seems right, but validation accuracy is strange.\ntraining resnet50 on 128*128 does not has this kind of issue.\nDo you have any ideas? Thanks, sir.
", "rawMarkdown": "Hi, thanks for your code. when I train resnet50 on 224x224 using batchsize of 240(8*30). I have got very wired validation results. see below\n\n 0.0037 70.0 0.3 | 14.243 0.003 0.009 (0.005)* | 0.865 0.776 0.915 (0.840) | 18 hr 19 min\n 0.0036 71.0 0.3 | 6.594 0.003 0.009 (0.005) | 0.819 0.790 0.920 (0.850) | 18 hr 35 min\n 0.0036 72.0 0.3 | 9.629 0.003 0.009 (0.005) | 0.850 0.784 0.914 (0.843) | 18 hr 51 min\n 0.0035 73.0 0.4 | 6.732 0.003 0.009 (0.005) | 0.810 0.795 0.918 (0.851) | 19 hr 07 min\n 0.0035 74.0 0.4 | 11.131 0.003 0.009 (0.005) | 0.825 0.791 0.920 (0.850) | 19 hr 24 min\n 0.0034 75.0 0.4 | 6.489 0.003 0.009 (0.006)* | 0.815 0.793 0.926 (0.854) | 19 hr 40 min\n 0.0034 76.0 0.4 | 6.794 0.003 0.008 (0.005) | 0.847 0.787 0.913 (0.844) | 19 hr 56 min\n 0.0033 77.0 0.4 | 6.468 0.003 0.009 (0.005) | 0.809 0.794 0.918 (0.851) | 20 hr 12 min\n 0.0033 78.0 0.4 | 6.231 0.006 0.017 (0.010) | 0.811 0.795 0.915 (0.850) | 20 hr 28 min\n 0.0032 79.0 0.4 | 6.483 0.003 0.009 (0.006) | 0.805 0.795 0.922 (0.853) | 20 hr 43 min\n 0.0032 80.0 0.4 | 8.119 0.003 0.009 (0.005)* | 0.831 0.789 0.916 (0.847) | 20 hr 59 min\n 0.0031 81.0 0.4 | 7.981 0.003 0.009 (0.005) | 0.809 0.791 0.919 (0.849) | 21 hr 15 min\n 0.0031 82.0 0.4 | 8.882 0.003 0.009 (0.005) | 0.775 0.794 0.924 (0.854) | 21 hr 30 min\n 0.0031 83.0 0.4 | 3.349 0.269 0.460 (0.352) | 0.819 0.789 0.918 (0.849) | 21 hr 46 min\n 0.0030 84.0 0.4 | 7.280 0.003 0.009 (0.005) | 0.819 0.800 0.919 (0.855) | 22 hr 02 min\n 0.0030 85.0 0.4 | 6.586 0.003 0.009 (0.005)* | 0.789 0.799 0.925 (0.857) | 22 hr 17 min\n 0.0029 86.0 0.4 | 7.680 0.004 0.009 (0.006) | 0.763 0.797 0.931 (0.859) | 22 hr 33 min\n\nThe training accuracy seems right, but validation accuracy is strange.\ntraining resnet50 on 128*128 does not has this kind of issue.\nDo you have any ideas? Thanks, sir.", "replies": [ { "id": 422912, "postDate": "2018-11-17T03:14:26.347Z", "content": "I use 96*96, only get acc3: 0.927
", "rawMarkdown": "I use 96*96, only get acc3: 0.927" } ] }, { "id": 417981, "postDate": "2018-11-09T04:27:53.650Z", "content": "Thank you @hengck23 for the starter kit. \nI'm not sure if I'm doing something wrong. I'm not able to run the starter code even with bs=1. (I have a 8GB-1070 GPU)\nAny pointers/tips?
", "rawMarkdown": "Thank you @hengck23 for the starter kit. \nI'm not sure if I'm doing something wrong. I'm not able to run the starter code even with bs=1. (I have a 8GB-1070 GPU)\nAny pointers/tips?" }, { "id": 417324, "postDate": "2018-11-08T04:50:11.453Z", "content": "Hi Heng, thank you for your code. There is one problem when I ran your code. It takes me almost 1 hour to load the whole dataset and almost 14 GB memory. However, according to the readme file, it only takes you around 4 minutes. And I can't run the model cause the memory issues. I wonder if you can show me some hints to solve this. By the way, the total RAM memory in my computer is 16 GB and one GTX 1080-8G exits.
", "rawMarkdown": "Hi Heng, thank you for your code. There is one problem when I ran your code. It takes me almost 1 hour to load the whole dataset and almost 14 GB memory. However, according to the readme file, it only takes you around 4 minutes. And I can't run the model cause the memory issues. I wonder if you can show me some hints to solve this. By the way, the total RAM memory in my computer is 16 GB and one GTX 1080-8G exits. ", "replies": [ { "id": 419083, "postDate": "2018-11-11T08:37:39.587Z", "content": "You could convert csv to pickle format, and it should speed up about 8X! here's the code below
\n\nimport pandas as pd\nimport pickle\ndef pickle_dump(data, filename):\n with open(filename, 'wb') as f:\n pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL)
\n\ndef csv_2_pickle():\n '''\n convert csv to pickle format\n read csv 10:25\n read pickle 1:31\n '''\n if 0:\n class_name = CLASS_NAME\n for name in tqdm(CLASS_NAME):\n name = name.replace('_', ' ')\n df = pd.read_csv(PATH_DATA_RAW + 'train_simplified/%s.csv' % name)\n # df = pickle_load(PATH_DATA_RAW + \"train_simplified/%s.p\" % name)\n pickle_dump(df, PATH_DATA_RAW + \"train_simplified/%s.p\" % name)
", "rawMarkdown": "You could convert csv to pickle format, and it should speed up about 8X! here's the code below\n\n\nimport pandas as pd\nimport pickle\ndef pickle_dump(data, filename):\n with open(filename, 'wb') as f:\n pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL)\n\ndef csv_2_pickle():\n '''\n convert csv to pickle format\n read csv 10:25\n read pickle 1:31\n '''\n if 0:\n class_name = CLASS_NAME\n for name in tqdm(CLASS_NAME):\n name = name.replace('_', ' ')\n df = pd.read_csv(PATH_DATA_RAW + 'train_simplified/%s.csv' % name)\n # df = pickle_load(PATH_DATA_RAW + \"train_simplified/%s.p\" % name)\n pickle_dump(df, PATH_DATA_RAW + \"train_simplified/%s.p\" % name) ", "votes": 4 }, { "id": 419403, "postDate": "2018-11-11T22:51:21.770Z", "content": "Thanks for the code. I will try it.
", "rawMarkdown": "Thanks for the code. I will try it. " }, { "id": 427397, "postDate": "2018-11-25T12:41:56.720Z", "content": "It helped alot thanks!
", "rawMarkdown": "It helped alot thanks!" } ] }, { "id": 416124, "postDate": "2018-11-06T08:13:46.897Z", "content": "I noticed your code works fine with different image size, for example 32, 64, and 128. But resnet is trained on image size 224x224, I wonder what's the trick in your Net() class defined in resent.py?
\n\nalso, generally, will we get better results if we match the input size to the pre-trained model size?
\n\nThanks
", "rawMarkdown": "I noticed your code works fine with different image size, for example 32, 64, and 128. But resnet is trained on image size 224x224, I wonder what's the trick in your Net() class defined in resent.py? \n\nalso, generally, will we get better results if we match the input size to the pre-trained model size? \n\nThanks", "replies": [ { "id": 420582, "postDate": "2018-11-13T21:07:42.993Z", "content": "I think you can change the last AvgPooling to the AdaptiveAvgPooling
", "rawMarkdown": "I think you can change the last AvgPooling to the AdaptiveAvgPooling" } ] }, { "id": 412373, "postDate": "2018-10-30T05:22:26.297Z", "content": "one problem \n state_dict[key] = pretrain_state_dict[key.replace('resnet.layer1.','layer1.')]\nKeyError: 'layer1.0.bn1.num_batches_tracked'
\n\nI am sure I had download the right weights, I run your code in linux and it's fine, but in win7 this problem comes out~, do u know what's going on?
", "rawMarkdown": "one problem \n state_dict[key] = pretrain_state_dict[key.replace('resnet.layer1.','layer1.')]\nKeyError: 'layer1.0.bn1.num_batches_tracked'\n\nI am sure I had download the right weights, I run your code in linux and it's fine, but in win7 this problem comes out~, do u know what's going on?", "replies": [ { "id": 412377, "postDate": "2018-10-30T05:28:34.033Z", "content": "Pytorch 0.4.0 and 0.4.1 problem. Can google for more information. A quick fix is to ignore and not to load that key for preteain model
", "rawMarkdown": "Pytorch 0.4.0 and 0.4.1 problem. Can google for more information. A quick fix is to ignore and not to load that key for preteain model", "votes": 1 }, { "id": 413291, "postDate": "2018-10-31T16:32:29.043Z", "content": "got it and already fixed this problem, thanks!
", "rawMarkdown": "got it and already fixed this problem, thanks!" } ] }, { "id": 410449, "postDate": "2018-10-26T04:49:36.950Z", "content": "hi @Heng CherKeng, I was wondering why do we put the code in a google drive instead of a github repo?
\n\nThanks for the help.
", "rawMarkdown": "hi @Heng CherKeng, I was wondering why do we put the code in a google drive instead of a github repo?\n\nThanks for the help." }, { "id": 405890, "postDate": "2018-10-18T09:44:43.667Z", "content": "Amazing!
\n\nThanks for the hard work you put in!
", "rawMarkdown": "Amazing!\n\nThanks for the hard work you put in!" }, { "id": 402146, "postDate": "2018-10-11T08:37:43.170Z", "content": "Impressive. Thanks.\nPytorch also has amazing Documentation and tutorials for beginners. Hope this helps.\nhttps://pytorch.org/resources
", "rawMarkdown": "Impressive. Thanks.\nPytorch also has amazing Documentation and tutorials for beginners. Hope this helps.\nhttps://pytorch.org/resources" }, { "id": 402131, "postDate": "2018-10-11T07:55:56.040Z", "content": "Hi CherKeng, My valid results seem normal, but got bad lb, do you know why?
", "rawMarkdown": "Hi CherKeng, My valid results seem normal, but got bad lb, do you know why?", "replies": [ { "id": 409445, "postDate": "2018-10-24T10:12:47.463Z", "content": "I have the same problem on LSTM network. Have you solved the issue?
", "rawMarkdown": "I have the same problem on LSTM network. Have you solved the issue?" }, { "id": 409453, "postDate": "2018-10-24T10:24:29.253Z", "content": "you can use batch_size = 1, then you would get right results. (Because var arrray length must be sorted)
", "rawMarkdown": "you can use batch_size = 1, then you would get right results. (Because var arrray length must be sorted)" }, { "id": 409454, "postDate": "2018-10-24T10:28:32.657Z", "content": "you can unsort to the original order using cache.index values
\n\ndef null_stroke_collate(batch):\n batch_size = len(batch)
\n\n#resort\nlength = np.array([len(batch[b][0]) for b in range(batch_size)])\nargsort = np.argsort(-length)\n\ncache = []\ninput = []\ntruth = []\n#for b in range(batch_size):\nfor b in argsort:\n input.append(batch[b][0])\n truth.append(batch[b][1])\n cache.append(batch[b][2])\nlength = length[argsort]\n\nlength_max = length.max()\ndim = len(batch[b][0][0])\npack = np.zeros((batch_size, length_max, dim), np.float32)\nfor b in range(batch_size):\n pack[b, 0:length[b]] = input[b]\ninput = torch.from_numpy(pack).float()\n\nif truth[0] is not None:\n truth = np.array(truth)\n truth = torch.from_numpy(truth).long()\n\nreturn input, length, truth, cache\nfor example, using SequentialSampler(evaluate_dataset),
\n\n net.set_mode('test')\n for input, length, truth, cache in evaluate_loader:\n print('\\r\\t',evaluate_num, end='', flush=True)\n evaluate_num += len(truth)\n\n with torch.no_grad():\n input = input.cuda()\n logit = data_parallel(net, (input, length) )\n prob = F.softmax(logit,1)\n\n\n #need to unsort\n index = np.array([c.index for c in cache],np.int32)\n argsort = np.argsort(index)\n probs.append( prob.data.cpu().numpy()[argsort] )\nThx, it seems that one batch should be sorted in order
", "rawMarkdown": "Thx, it seems that one batch should be sorted in order" }, { "id": 409474, "postDate": "2018-10-24T11:04:21.603Z", "content": "thx, that works great
", "rawMarkdown": "thx, that works great" } ] }, { "id": 401754, "postDate": "2018-10-10T16:39:17.030Z", "content": "version: 2018-10-10
\n\nadded code snippets for simple rnn model on strokes
\n\nsee readme.ppt attached
", "rawMarkdown": "version: 2018-10-10\n\nadded code snippets for simple rnn model on strokes\n\n\nsee readme.ppt attached" }, { "id": 400970, "postDate": "2018-10-09T08:08:04.013Z", "content": "some improvement I implemented to improve software architecture:
\n\n.... to be udpated ....
", "rawMarkdown": "some improvement I implemented to improve software architecture:\n\n- use parallel load of csv files (using \"from multiprocessing import Pool\"): reduce data load from 4 min to 1.5 min\n\n\n.... to be udpated ...." }, { "id": 400891, "postDate": "2018-10-09T04:58:43.143Z", "content": "That looks really good!
", "rawMarkdown": "That looks really good!" }, { "id": 400598, "postDate": "2018-10-08T16:00:34.133Z", "content": "Impressive work Heng.
\n\nYour (first?) CNN model performance crushed all my RNN attempts.
", "rawMarkdown": "Impressive work Heng.\n\nYour (first?) CNN model performance crushed all my RNN attempts.", "replies": [ { "id": 400665, "postDate": "2018-10-08T17:37:53.897Z", "content": "@Miha Skalic
\n\nI was lucky to choose some correct network and correct image size for my first attempt.
\n\nI study the data and some initial experiment results and believe that stroke-based RNN should be better. I am working on that now. Sequence data is one of my weakest and i hope to achieve some good results on it ^^;
", "rawMarkdown": "@Miha Skalic\n \nI was lucky to choose some correct network and correct image size for my first attempt.\n\nI study the data and some initial experiment results and believe that stroke-based RNN should be better. I am working on that now. Sequence data is one of my weakest and i hope to achieve some good results on it ^^;\n", "votes": 3 }, { "id": 400674, "postDate": "2018-10-08T17:51:06.387Z", "content": "Good luck! I wish you develop some awesome models :)
", "rawMarkdown": "Good luck! I wish you develop some awesome models :)" }, { "id": 400935, "postDate": "2018-10-09T06:52:19.830Z", "content": "good luck to you too! i will share good models in the forum once i can get some good results.
", "rawMarkdown": "good luck to you too! i will share good models in the forum once i can get some good results." } ] }, { "id": 399378, "postDate": "2018-10-05T17:25:12.780Z", "content": "Niceeeeeeeeee, looking forward to your updates!
", "rawMarkdown": "Niceeeeeeeeee, looking forward to your updates! " }, { "id": 421280, "postDate": "2018-11-14T21:02:45.537Z", "rawMarkdown": "", "isDeleted": true }, { "id": 400119, "postDate": "2018-10-07T16:33:37.383Z", "content": "Thanks.. A Nice One !
", "rawMarkdown": "Thanks.. A Nice One !" } ], "comments": [ { "id": 400982, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-09T08:30:24.833000", "content": "\"Sketch Classification with Neural Networks - A Comparative Study of CNN and RNN on the QuickDraw! data set\"
\n\nhttps://uu.diva-portal.org/smash/get/diva2:1218490/FULLTEXT01.pdf
", "votes": 3, "replies": [ { "id": 429738, "author_name": "gody7334", "author_url": "", "post_date": "2018-11-29T09:27:51.563000", "content": "Hi CherKeng,\nI just aware that in your starter kit, you didn't normalize images to [0 1], then normalize images using imagenet mean and std, it convert image into strange distribution
\n\nBut what surprise me is that, the network still converge, my questions are:\n1. Its still better to normalize images, which can get better score?\n2. Is that because doodle images are simple so Batch Normalization can do the job itself?\nThanks,
", "votes": 0, "replies": [] } ] }, { "id": 400900, "author_name": "cab", "author_url": "", "post_date": "2018-10-09T05:33:41.683000", "content": "I wasted 5 hours to train my model to archive my 0.7LB. Your model takes only 13min. Very impressive!.
\nThank you
Hi, Heng. how much data did you use? ?k images / class
", "votes": 1, "replies": [] }, { "id": 400479, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-08T11:44:26.467000", "content": "latest version: 2018-10-08
\n\nsee readme.ppt attached
", "votes": 2, "replies": [] }, { "id": 422902, "author_name": "good good study", "author_url": "", "post_date": "2018-11-17T02:21:20.233000", "content": "Hi, thanks for your code. when I train resnet50 on 224x224 using batchsize of 240(8*30). I have got very wired validation results. see below
\n\n0.0037 70.0 0.3 | 14.243 0.003 0.009 (0.005)* | 0.865 0.776 0.915 (0.840) | 18 hr 19 min\n0.0036 71.0 0.3 | 6.594 0.003 0.009 (0.005) | 0.819 0.790 0.920 (0.850) | 18 hr 35 min\n0.0036 72.0 0.3 | 9.629 0.003 0.009 (0.005) | 0.850 0.784 0.914 (0.843) | 18 hr 51 min\n0.0035 73.0 0.4 | 6.732 0.003 0.009 (0.005) | 0.810 0.795 0.918 (0.851) | 19 hr 07 min\n0.0035 74.0 0.4 | 11.131 0.003 0.009 (0.005) | 0.825 0.791 0.920 (0.850) | 19 hr 24 min\n0.0034 75.0 0.4 | 6.489 0.003 0.009 (0.006)* | 0.815 0.793 0.926 (0.854) | 19 hr 40 min\n0.0034 76.0 0.4 | 6.794 0.003 0.008 (0.005) | 0.847 0.787 0.913 (0.844) | 19 hr 56 min\n0.0033 77.0 0.4 | 6.468 0.003 0.009 (0.005) | 0.809 0.794 0.918 (0.851) | 20 hr 12 min\n0.0033 78.0 0.4 | 6.231 0.006 0.017 (0.010) | 0.811 0.795 0.915 (0.850) | 20 hr 28 min\n0.0032 79.0 0.4 | 6.483 0.003 0.009 (0.006) | 0.805 0.795 0.922 (0.853) | 20 hr 43 min\n0.0032 80.0 0.4 | 8.119 0.003 0.009 (0.005)* | 0.831 0.789 0.916 (0.847) | 20 hr 59 min\n0.0031 81.0 0.4 | 7.981 0.003 0.009 (0.005) | 0.809 0.791 0.919 (0.849) | 21 hr 15 min\n0.0031 82.0 0.4 | 8.882 0.003 0.009 (0.005) | 0.775 0.794 0.924 (0.854) | 21 hr 30 min\n0.0031 83.0 0.4 | 3.349 0.269 0.460 (0.352) | 0.819 0.789 0.918 (0.849) | 21 hr 46 min\n0.0030 84.0 0.4 | 7.280 0.003 0.009 (0.005) | 0.819 0.800 0.919 (0.855) | 22 hr 02 min\n0.0030 85.0 0.4 | 6.586 0.003 0.009 (0.005)* | 0.789 0.799 0.925 (0.857) | 22 hr 17 min\n0.0029 86.0 0.4 | 7.680 0.004 0.009 (0.006) | 0.763 0.797 0.931 (0.859) | 22 hr 33 min\nThe training accuracy seems right, but validation accuracy is strange.\ntraining resnet50 on 128*128 does not has this kind of issue.\nDo you have any ideas? Thanks, sir.
", "votes": 0, "replies": [ { "id": 422912, "author_name": "Gary", "author_url": "", "post_date": "2018-11-17T03:14:26.347000", "content": "I use 96*96, only get acc3: 0.927
", "votes": 0, "replies": [] } ] }, { "id": 417981, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2018-11-09T04:27:53.650000", "content": "Thank you @hengck23 for the starter kit. \nI'm not sure if I'm doing something wrong. I'm not able to run the starter code even with bs=1. (I have a 8GB-1070 GPU)\nAny pointers/tips?
", "votes": 0, "replies": [] }, { "id": 417324, "author_name": "muzifft", "author_url": "", "post_date": "2018-11-08T04:50:11.453000", "content": "Hi Heng, thank you for your code. There is one problem when I ran your code. It takes me almost 1 hour to load the whole dataset and almost 14 GB memory. However, according to the readme file, it only takes you around 4 minutes. And I can't run the model cause the memory issues. I wonder if you can show me some hints to solve this. By the way, the total RAM memory in my computer is 16 GB and one GTX 1080-8G exits.
", "votes": 0, "replies": [ { "id": 419083, "author_name": "🐳鲲(China)", "author_url": "", "post_date": "2018-11-11T08:37:39.587000", "content": "You could convert csv to pickle format, and it should speed up about 8X! here's the code below
\n\nimport pandas as pd\nimport pickle\ndef pickle_dump(data, filename):\n with open(filename, 'wb') as f:\n pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL)
\n\ndef csv_2_pickle():\n '''\n convert csv to pickle format\n read csv 10:25\n read pickle 1:31\n '''\n if 0:\n class_name = CLASS_NAME\n for name in tqdm(CLASS_NAME):\n name = name.replace('_', ' ')\n df = pd.read_csv(PATH_DATA_RAW + 'train_simplified/%s.csv' % name)\n # df = pickle_load(PATH_DATA_RAW + \"train_simplified/%s.p\" % name)\n pickle_dump(df, PATH_DATA_RAW + \"train_simplified/%s.p\" % name)
", "votes": 4, "replies": [] }, { "id": 419403, "author_name": "muzifft", "author_url": "", "post_date": "2018-11-11T22:51:21.770000", "content": "Thanks for the code. I will try it.
", "votes": 0, "replies": [] }, { "id": 427397, "author_name": "Soonhwan Kwon", "author_url": "", "post_date": "2018-11-25T12:41:56.720000", "content": "It helped alot thanks!
", "votes": 0, "replies": [] } ] }, { "id": 416124, "author_name": "yimacs", "author_url": "", "post_date": "2018-11-06T08:13:46.897000", "content": "I noticed your code works fine with different image size, for example 32, 64, and 128. But resnet is trained on image size 224x224, I wonder what's the trick in your Net() class defined in resent.py?
\n\nalso, generally, will we get better results if we match the input size to the pre-trained model size?
\n\nThanks
", "votes": 0, "replies": [ { "id": 420582, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-13T21:07:42.993000", "content": "I think you can change the last AvgPooling to the AdaptiveAvgPooling
", "votes": 0, "replies": [] } ] }, { "id": 412373, "author_name": "🐳鲲(China)", "author_url": "", "post_date": "2018-10-30T05:22:26.297000", "content": "one problem \n state_dict[key] = pretrain_state_dict[key.replace('resnet.layer1.','layer1.')]\nKeyError: 'layer1.0.bn1.num_batches_tracked'
\n\nI am sure I had download the right weights, I run your code in linux and it's fine, but in win7 this problem comes out~, do u know what's going on?
", "votes": 0, "replies": [ { "id": 412377, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-30T05:28:34.033000", "content": "Pytorch 0.4.0 and 0.4.1 problem. Can google for more information. A quick fix is to ignore and not to load that key for preteain model
", "votes": 1, "replies": [] }, { "id": 413291, "author_name": "🐳鲲(China)", "author_url": "", "post_date": "2018-10-31T16:32:29.043000", "content": "got it and already fixed this problem, thanks!
", "votes": 0, "replies": [] } ] }, { "id": 410449, "author_name": "Bo Peng", "author_url": "", "post_date": "2018-10-26T04:49:36.950000", "content": "hi @Heng CherKeng, I was wondering why do we put the code in a google drive instead of a github repo?
\n\nThanks for the help.
", "votes": 0, "replies": [] }, { "id": 405890, "author_name": "Paolo Gottardo", "author_url": "", "post_date": "2018-10-18T09:44:43.667000", "content": "Amazing!
\n\nThanks for the hard work you put in!
", "votes": 0, "replies": [] }, { "id": 402146, "author_name": "Raj Srujan Jalem", "author_url": "", "post_date": "2018-10-11T08:37:43.170000", "content": "Impressive. Thanks.\nPytorch also has amazing Documentation and tutorials for beginners. Hope this helps.\nhttps://pytorch.org/resources
", "votes": 0, "replies": [] }, { "id": 402131, "author_name": "The16thRoute", "author_url": "", "post_date": "2018-10-11T07:55:56.040000", "content": "Hi CherKeng, My valid results seem normal, but got bad lb, do you know why?
", "votes": 0, "replies": [ { "id": 409445, "author_name": "good good study", "author_url": "", "post_date": "2018-10-24T10:12:47.463000", "content": "I have the same problem on LSTM network. Have you solved the issue?
", "votes": 0, "replies": [] }, { "id": 409453, "author_name": "The16thRoute", "author_url": "", "post_date": "2018-10-24T10:24:29.253000", "content": "you can use batch_size = 1, then you would get right results. (Because var arrray length must be sorted)
", "votes": 0, "replies": [] }, { "id": 409454, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-24T10:28:32.657000", "content": "you can unsort to the original order using cache.index values
\n\ndef null_stroke_collate(batch):\n batch_size = len(batch)
\n\n#resort\nlength = np.array([len(batch[b][0]) for b in range(batch_size)])\nargsort = np.argsort(-length)\n\ncache = []\ninput = []\ntruth = []\n#for b in range(batch_size):\nfor b in argsort:\n input.append(batch[b][0])\n truth.append(batch[b][1])\n cache.append(batch[b][2])\nlength = length[argsort]\n\nlength_max = length.max()\ndim = len(batch[b][0][0])\npack = np.zeros((batch_size, length_max, dim), np.float32)\nfor b in range(batch_size):\n pack[b, 0:length[b]] = input[b]\ninput = torch.from_numpy(pack).float()\n\nif truth[0] is not None:\n truth = np.array(truth)\n truth = torch.from_numpy(truth).long()\n\nreturn input, length, truth, cache\nfor example, using SequentialSampler(evaluate_dataset),
\n\n net.set_mode('test')\n for input, length, truth, cache in evaluate_loader:\n print('\\r\\t',evaluate_num, end='', flush=True)\n evaluate_num += len(truth)\n\n with torch.no_grad():\n input = input.cuda()\n logit = data_parallel(net, (input, length) )\n prob = F.softmax(logit,1)\n\n\n #need to unsort\n index = np.array([c.index for c in cache],np.int32)\n argsort = np.argsort(index)\n probs.append( prob.data.cpu().numpy()[argsort] )\nThx, it seems that one batch should be sorted in order
", "votes": 0, "replies": [] }, { "id": 409474, "author_name": "good good study", "author_url": "", "post_date": "2018-10-24T11:04:21.603000", "content": "thx, that works great
", "votes": 0, "replies": [] } ] }, { "id": 401754, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-10T16:39:17.030000", "content": "version: 2018-10-10
\n\nadded code snippets for simple rnn model on strokes
\n\nsee readme.ppt attached
", "votes": 0, "replies": [] }, { "id": 400970, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-09T08:08:04.013000", "content": "some improvement I implemented to improve software architecture:
\n\n.... to be udpated ....
", "votes": 0, "replies": [] }, { "id": 400891, "author_name": "Watanabe Ryunosuke", "author_url": "", "post_date": "2018-10-09T04:58:43.143000", "content": "That looks really good!
", "votes": 0, "replies": [] }, { "id": 400598, "author_name": "Miha Skalic", "author_url": "", "post_date": "2018-10-08T16:00:34.133000", "content": "Impressive work Heng.
\n\nYour (first?) CNN model performance crushed all my RNN attempts.
", "votes": 0, "replies": [ { "id": 400665, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-08T17:37:53.897000", "content": "@Miha Skalic
\n\nI was lucky to choose some correct network and correct image size for my first attempt.
\n\nI study the data and some initial experiment results and believe that stroke-based RNN should be better. I am working on that now. Sequence data is one of my weakest and i hope to achieve some good results on it ^^;
", "votes": 3, "replies": [] }, { "id": 400674, "author_name": "Miha Skalic", "author_url": "", "post_date": "2018-10-08T17:51:06.387000", "content": "Good luck! I wish you develop some awesome models :)
", "votes": 0, "replies": [] }, { "id": 400935, "author_name": "hengck23", "author_url": "", "post_date": "2018-10-09T06:52:19.830000", "content": "good luck to you too! i will share good models in the forum once i can get some good results.
", "votes": 0, "replies": [] } ] }, { "id": 399378, "author_name": "DrHB", "author_url": "", "post_date": "2018-10-05T17:25:12.780000", "content": "Niceeeeeeeeee, looking forward to your updates!
", "votes": 0, "replies": [] }, { "id": 421280, "author_name": "", "author_url": "", "post_date": "2018-11-14T21:02:45.537000", "content": "", "votes": 0, "replies": [] }, { "id": 400119, "author_name": "Ravi", "author_url": "", "post_date": "2018-10-07T16:33:37.383000", "content": "Thanks.. A Nice One !
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "399256": "This post will be updated as i move alone.\n\nreference: https://www.kaggle.com/kmader/quickdraw-simple-models\n\nobjective is to create baseline pytorch code for the following:\n\n- simple cnn model on image\n\n- simple rnn model on strokes (aka, tensorflow tutorial)\n\nrefer to: https://drive.google.com/open?id=17fyH9MQ49hz1teCJ_jWZvo78ju52s5ut\n\n-----\nversion: 2018-10-08\n\n- simple cnn on image 32x32 (lb 0.723)\n\n---\n\nversion: 2018-10-10\n\n- added code snippets for simple rnn model on strokes", "400982": "\"Sketch Classification with Neural Networks - A Comparative Study of CNN and RNN on the QuickDraw! data set\"\n\nhttps://uu.diva-portal.org/smash/get/diva2:1218490/FULLTEXT01.pdf", "400900": "I wasted 5 hours to train my model to archive my 0.7LB. Your model takes only 13min. Very impressive!. \nThank you", "409465": "Hi, Heng. how much data did you use? ?k images / class", "400479": "latest version: 2018-10-08\n\n- simple cnn on image 32x32 (lb 0.723)\n\nsee readme.ppt attached\n ", "422902": "Hi, thanks for your code. when I train resnet50 on 224x224 using batchsize of 240(8*30). I have got very wired validation results. see below\n\n 0.0037 70.0 0.3 | 14.243 0.003 0.009 (0.005)* | 0.865 0.776 0.915 (0.840) | 18 hr 19 min\n 0.0036 71.0 0.3 | 6.594 0.003 0.009 (0.005) | 0.819 0.790 0.920 (0.850) | 18 hr 35 min\n 0.0036 72.0 0.3 | 9.629 0.003 0.009 (0.005) | 0.850 0.784 0.914 (0.843) | 18 hr 51 min\n 0.0035 73.0 0.4 | 6.732 0.003 0.009 (0.005) | 0.810 0.795 0.918 (0.851) | 19 hr 07 min\n 0.0035 74.0 0.4 | 11.131 0.003 0.009 (0.005) | 0.825 0.791 0.920 (0.850) | 19 hr 24 min\n 0.0034 75.0 0.4 | 6.489 0.003 0.009 (0.006)* | 0.815 0.793 0.926 (0.854) | 19 hr 40 min\n 0.0034 76.0 0.4 | 6.794 0.003 0.008 (0.005) | 0.847 0.787 0.913 (0.844) | 19 hr 56 min\n 0.0033 77.0 0.4 | 6.468 0.003 0.009 (0.005) | 0.809 0.794 0.918 (0.851) | 20 hr 12 min\n 0.0033 78.0 0.4 | 6.231 0.006 0.017 (0.010) | 0.811 0.795 0.915 (0.850) | 20 hr 28 min\n 0.0032 79.0 0.4 | 6.483 0.003 0.009 (0.006) | 0.805 0.795 0.922 (0.853) | 20 hr 43 min\n 0.0032 80.0 0.4 | 8.119 0.003 0.009 (0.005)* | 0.831 0.789 0.916 (0.847) | 20 hr 59 min\n 0.0031 81.0 0.4 | 7.981 0.003 0.009 (0.005) | 0.809 0.791 0.919 (0.849) | 21 hr 15 min\n 0.0031 82.0 0.4 | 8.882 0.003 0.009 (0.005) | 0.775 0.794 0.924 (0.854) | 21 hr 30 min\n 0.0031 83.0 0.4 | 3.349 0.269 0.460 (0.352) | 0.819 0.789 0.918 (0.849) | 21 hr 46 min\n 0.0030 84.0 0.4 | 7.280 0.003 0.009 (0.005) | 0.819 0.800 0.919 (0.855) | 22 hr 02 min\n 0.0030 85.0 0.4 | 6.586 0.003 0.009 (0.005)* | 0.789 0.799 0.925 (0.857) | 22 hr 17 min\n 0.0029 86.0 0.4 | 7.680 0.004 0.009 (0.006) | 0.763 0.797 0.931 (0.859) | 22 hr 33 min\n\nThe training accuracy seems right, but validation accuracy is strange.\ntraining resnet50 on 128*128 does not has this kind of issue.\nDo you have any ideas? Thanks, sir.", "417981": "Thank you @hengck23 for the starter kit. \nI'm not sure if I'm doing something wrong. I'm not able to run the starter code even with bs=1. (I have a 8GB-1070 GPU)\nAny pointers/tips?", "417324": "Hi Heng, thank you for your code. There is one problem when I ran your code. It takes me almost 1 hour to load the whole dataset and almost 14 GB memory. However, according to the readme file, it only takes you around 4 minutes. And I can't run the model cause the memory issues. I wonder if you can show me some hints to solve this. By the way, the total RAM memory in my computer is 16 GB and one GTX 1080-8G exits. ", "416124": "I noticed your code works fine with different image size, for example 32, 64, and 128. But resnet is trained on image size 224x224, I wonder what's the trick in your Net() class defined in resent.py? \n\nalso, generally, will we get better results if we match the input size to the pre-trained model size? \n\nThanks", "412373": "one problem \n state_dict[key] = pretrain_state_dict[key.replace('resnet.layer1.','layer1.')]\nKeyError: 'layer1.0.bn1.num_batches_tracked'\n\nI am sure I had download the right weights, I run your code in linux and it's fine, but in win7 this problem comes out~, do u know what's going on?", "410449": "hi @Heng CherKeng, I was wondering why do we put the code in a google drive instead of a github repo?\n\nThanks for the help.", "405890": "Amazing!\n\nThanks for the hard work you put in!", "402146": "Impressive. Thanks.\nPytorch also has amazing Documentation and tutorials for beginners. Hope this helps.\nhttps://pytorch.org/resources", "402131": "Hi CherKeng, My valid results seem normal, but got bad lb, do you know why?", "401754": "version: 2018-10-10\n\nadded code snippets for simple rnn model on strokes\n\n\nsee readme.ppt attached", "400970": "some improvement I implemented to improve software architecture:\n\n- use parallel load of csv files (using \"from multiprocessing import Pool\"): reduce data load from 4 min to 1.5 min\n\n\n.... to be udpated ....", "400891": "That looks really good!", "400598": "Impressive work Heng.\n\nYour (first?) CNN model performance crushed all my RNN attempts.", "399378": "Niceeeeeeeeee, looking forward to your updates! ", "421280": "", "400119": "Thanks.. A Nice One !" } }