{
  "id": 73701,
  "title": " 24th solution",
  "url": "/competitions/quickdraw-doodle-recognition/discussion/73701",
  "author_name": "Daewoo Myoung",
  "post_date": "2018-12-05T00:50:01.451000",
  "votes": 41,
  "comment_count": 14,
  "views": 0,
  "content": "<p>You may not be interested, but our 24th solution is below.</p>\n\n<p>-. data generate</p>\n\n<p>-. Model</p>\n\n<p>-. optimize, loss, etc</p>\n\n<p>-. ensemble</p>\n\n<ol>\n<li><p>data generate</p>\n\n<ul><li>Using simplified file</li></ul></li>\n</ol>\n\n<p>def draw_cv2_color(raw_strokes, size=256, lw=6, time_color=True):</p>\n\n<pre><code>img = np.zeros((BASE_SIZE, BASE_SIZE,3), np.uint8)\n\nfor t, stroke in enumerate(raw_strokes):     \n\n    inertia_x = 0\n\n    inertia_y = 0\n\n    for i in range(len(stroke[0]) - 1):\n\n        color = int(255 - 245*(float(t)/len(raw_strokes))) if time_color else 255 ## strokes order\n        print(color)\n\n        sx = stroke[0][i]\n\n        sy = stroke[1][i]\n\n        ex = stroke[0][i + 1]\n\n        ey = stroke[1][i + 1]\n\n        color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)\n\n        color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)\n\n        _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n        inertia_x = 2*ex -sx\n\n        inertia_y = 2*ey-sy\n\n\n\nif size != BASE_SIZE:\n\n    return cv2.resize(img, (size, size))\n\nelse:\n\n    return img\n</code></pre>\n\n<ul>\n<li>Using raw file</li>\n</ul>\n\n<p>def draw_cv2_color_new(raw_strokes, size=256, lw=6, time_color=True, last_drop_r = 0.0):</p>\n\n<pre><code>stx_min, sty_min = 99999, 99999\n\nstx_max, sty_max = 0,0\n\nett=0  # How fast to complete less than 20 seconds\n\n\n\nfor t, stroke in enumerate(raw_strokes):\n\n    if t == len(raw_strokes) -1:\n\n        ett = int(stroke[2][-1])\n\n    for i in range(len(stroke[0])):\n\n        stx_min = min(stx_min, int(stroke[0][i]))\n\n        stx_max = max(stx_max, int(stroke[0][i]))\n\n        sty_min = min(sty_min, int(stroke[1][i]))\n\n        sty_max = max(sty_max, int(stroke[1][i]))\n\n\n\nlimit_ett = 20*1000   \n\nofs = 15\n\n\n\nif int(sty_max-sty_min+2*ofs) &amp;gt; 6000 or int(stx_max-stx_min+2*ofs)  &amp;gt; 6000:\n\n    img = np.zeros((6000,6000,3), np.uint8)\n\nelse:\n\n    img = np.zeros((int(sty_max-sty_min+2*ofs), int(stx_max-stx_min+2*ofs),3), np.uint8)\n\n\n\nfor t, stroke in enumerate(raw_strokes):     \n\n    inertia_x = 0\n\n    inertia_y = 0\n\n    pre_st_t = 0 \n\n    for i in range(len(stroke[0]) - 1):\n\n        color = int(255 - 245*float(t)/len(raw_strokes)) if time_color else 255 ##  stroke order\n\n        sx = int(stroke[0][i]) - stx_min +ofs\n\n        sy = int(stroke[1][i]) - sty_min +ofs\n\n        st = stroke[2][i]\n\n        ex = int(stroke[0][i + 1])- stx_min +ofs\n\n        ey = int(stroke[1][i + 1])- sty_min +ofs\n\n        et = stroke[2][i+1]\n\n\n\n\n\n        time = et-st\n\n        if time ==0:\n\n            time = 1\n\n\n\n        color_v =  min(int((np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / time)*255.0), 255) ## speed\n\n        color_a = min(int((np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(time*time))*255.0), 255) ## acceleration (1~0)\n\n        _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n\n\n        if i==0:\n\n            color_inter = int((float(et-pre_st_t)/limit_ett)*245)+10\n\n            _ = cv2.circle(img, (sx, sy), lw, (0,0,color_inter), -1) ##interval time\n\n\n\n        if i==len(stroke[0])-2 and t == len(raw_strokes) -1:\n\n            color_end = int((float(ett)/(limit_ett)*245))+10\n\n\n\n            _ = cv2.circle(img, (sx, sy), lw, (0,color_end,0), -1) ##end time\n\n\n\n        inertia_x = 2*ex -sx\n\n        inertia_y = 2*ey-sy\n\n        pre_st_t=et\n\n\n\nreturn cv2.resize(img, (size, size)) #lw reflects how big the picture is drawn, also the aspect ratio is reflected\n</code></pre>\n\n<p>2.. Model Structure</p>\n\n<ul>\n<li><p>Best single model </p>\n\n<p>: InceptionResnetV2 (139,139,3) size input  local valid score is  0.9516.</p>\n\n<p>: using raw file, using 'imagenet' weights, batch size 180</p></li>\n</ul>\n\n<p>base_model = InceptionResNetV2(input_shape=input_shape, weights='imagenet',include_top= False)</p>\n\n<p>x = base_model.output</p>\n\n<p>x = GlobalAveragePooling2D()(x)</p>\n\n<p>x = Dense(1024, activation='rule')(x)</p>\n\n<p>x = Dropout(0.3)(x)</p>\n\n<p>predictions = Dense(340, activation='softmax', name='lastfc')(x)</p>\n\n<p>model = Model(inputs=base_model.input, outputs=predictions)</p>\n\n<p>3.. Optimizer, Loss, etc.</p>\n\n<ul>\n<li><p>In first train about 1epoch(50M set) , I used adam and learning rate 0.002, </p>\n\n<p>categorical cross entropy loss</p></li>\n<li><p>Second train , I used adam accumulation 500 iters and learning rate 0.002, </p>\n\n<p>categorical cross entropy 10% and top3 loss 90%.</p></li>\n<li><p>I did not have enough time to train to the saturation.</p></li>\n</ul>\n\n<h1><a href=\"https://github.com/keras-team/keras/issues/3556\">https://github.com/keras-team/keras/issues/3556</a></h1>\n\n<p>import keras.backend as K</p>\n\n<p>from keras.legacy import interfaces</p>\n\n<p>from keras.optimizers import Optimizer</p>\n\n<p>class AdamAccumulate(Optimizer):</p>\n\n<pre><code>def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n\n             epsilon=None, decay=0., amsgrad=False, accum_iters=1, **kwargs):\n\n    if accum_iters &amp;lt; 1:\n\n        raise ValueError('accum_iters must be &amp;gt;= 1')\n\n    super(AdamAccumulate, self).__init__(**kwargs)\n\n    with K.name_scope(self.__class__.__name__):\n\n        self.iterations = K.variable(0, dtype='int64', name='iterations')\n\n        self.lr = K.variable(lr, name='lr')\n\n        self.beta_1 = K.variable(beta_1, name='beta_1')\n\n        self.beta_2 = K.variable(beta_2, name='beta_2')\n\n        self.decay = K.variable(decay, name='decay')\n\n    if epsilon is None:\n\n        epsilon = K.epsilon()\n\n    self.epsilon = epsilon\n\n    self.initial_decay = decay\n\n    self.amsgrad = amsgrad\n\n    self.accum_iters = K.variable(accum_iters, K.dtype(self.iterations))\n\n    self.accum_iters_float = K.cast(self.accum_iters, K.floatx())\n\n\n\n@interfaces.legacy_get_updates_support\n\ndef get_updates(self, loss, params):\n\n    grads = self.get_gradients(loss, params)\n\n    self.updates = [K.update_add(self.iterations, 1)]\n\n\n\n    lr = self.lr\n\n    completed_updates = K.cast(K.tf.floordiv(self.iterations, self.accum_iters), K.floatx())\n\n    if self.initial_decay &amp;gt; 0:\n\n        lr = lr * (1. / (1. + self.decay * completed_updates))\n\n\n\n    t = completed_updates + 1\n\n\n\n    lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t)))\n\n\n\n    update_switch = K.equal((self.iterations + 1) % self.accum_iters, 0)\n\n    update_switch = K.cast(update_switch, K.floatx())\n\n\n\n    ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    gs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n\n\n    if self.amsgrad:\n\n        vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    else:\n\n        vhats = [K.zeros(1) for _ in params]\n\n\n\n    self.weights = [self.iterations] + ms + vs + vhats\n\n\n\n    for p, g, m, v, vhat, tg in zip(params, grads, ms, vs, vhats, gs):\n\n\n\n        sum_grad = tg + g\n\n        avg_grad = sum_grad / self.accum_iters_float\n\n\n\n        m_t = (self.beta_1 * m) + (1. - self.beta_1) * avg_grad\n\n        v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(avg_grad)\n\n\n\n        if self.amsgrad:\n\n            vhat_t = K.maximum(vhat, v_t)\n\n            p_t = p - lr_t * m_t / (K.sqrt(vhat_t) + self.epsilon)\n\n            self.updates.append(K.update(vhat, (1 - update_switch) * vhat + update_switch * vhat_t))\n\n        else:\n\n            p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n\n\n        self.updates.append(K.update(m, (1 - update_switch) * m + update_switch * m_t))\n\n        self.updates.append(K.update(v, (1 - update_switch) * v + update_switch * v_t))\n\n        self.updates.append(K.update(tg, (1 - update_switch) * sum_grad))\n\n        new_p = p_t\n\n\n\n        # Apply constraints.\n\n        if getattr(p, 'constraint', None) is not None:\n\n            new_p = p.constraint(new_p)\n\n\n\n        self.updates.append(K.update(p, (1 - update_switch) * p + update_switch * new_p))\n\n    return self.updates\n\ndef get_config(self):\n\n    config = {'lr': float(K.get_value(self.lr)),\n\n              'beta_1': float(K.get_value(self.beta_1)),\n\n              'beta_2': float(K.get_value(self.beta_2)),\n\n              'decay': float(K.get_value(self.decay)),\n\n              'epsilon': self.epsilon,\n\n              'amsgrad': self.amsgrad}\n\n    base_config = super(AdamAccumulate, self).get_config()\n\n    return dict(list(base_config.items()) + list(config.items()))\n</code></pre>\n\n<p>4.. Ensemble</p>\n\n<ul>\n<li><p>I used weighed average ensemble by local valid score and argmax corr.</p></li>\n<li><p>I used InceptionResNetV2 (raw, simple), Xception (raw), Resnet50 (simple).</p></li>\n<li><p>I did not have time to train both (raw, simple)</p></li>\n<li><p>Average weight calculation is below code</p></li>\n</ul>\n\n<p>def get_score_w(local_score):</p>\n\n<pre><code>ls = np.array(local_score)\n\nsub = ls - ls.min()\n\ndiv = sub/sub.max()\n\nadd = div + 0.1\n\nnor = add/add.max()\n\nsqr = nor*nor\n\nprint(sqr)\n\nreturn sqr\n</code></pre>\n\n<p>def get_corr_w(clsnp):</p>\n\n<pre><code>corxlist=[]\n\nfor idx1, cls1 in enumerate((clsnp)):\n\n    corylist=[]\n\n    for idx2, cls2 in enumerate((clsnp)):\n\n        cor_max = np.corrcoef(np.argmax(clsnp[idx1],axis=1),np.argmax(clsnp[idx2],axis=1))[0][1]\n\n        corylist.append(cor_max)\n\n    corxlist.append(corylist)\n\n\n\ndf = pd.DataFrame(corxlist,columns=names, index=names)\n\n#print(df)\n\ncorr_w = []\n\nfor i in range(df.shape[0]):\n\n    count = 0\n\n    thr = 0.95\n\n    for v in df.values[i]:\n\n        if v &amp;gt; thr :\n\n            count+=1\n\n    corr_w.append(1.0/count)   \n\nreturn np.array(corr_w), df\n</code></pre>\n\n<p>def getensemble_w(clsnp, local_score):</p>\n\n<pre><code>score_w = get_score_w(local_score)\n\nfor i in range(score_w.shape[0]):\n\n    if score_w[i]==1.0:\n\n        score_w[i] = 1.2 #max score add 20%\n\ncorr_w = get_corr_w(clsnp)[0]\n\nensemble_w = score_w*corr_w\n\nreturn ensemble_w\n</code></pre>\n\n<p>Everybody enjoy the competition~!!</p>",
  "messages": [
    {
      "id": 433298,
      "postDate": "2018-12-05T00:50:01.450Z",
      "content": "<p>You may not be interested, but our 24th solution is below.</p>\n\n<p>-. data generate</p>\n\n<p>-. Model</p>\n\n<p>-. optimize, loss, etc</p>\n\n<p>-. ensemble</p>\n\n<ol>\n<li><p>data generate</p>\n\n<ul><li>Using simplified file</li></ul></li>\n</ol>\n\n<p>def draw_cv2_color(raw_strokes, size=256, lw=6, time_color=True):</p>\n\n<pre><code>img = np.zeros((BASE_SIZE, BASE_SIZE,3), np.uint8)\n\nfor t, stroke in enumerate(raw_strokes):     \n\n    inertia_x = 0\n\n    inertia_y = 0\n\n    for i in range(len(stroke[0]) - 1):\n\n        color = int(255 - 245*(float(t)/len(raw_strokes))) if time_color else 255 ## strokes order\n        print(color)\n\n        sx = stroke[0][i]\n\n        sy = stroke[1][i]\n\n        ex = stroke[0][i + 1]\n\n        ey = stroke[1][i + 1]\n\n        color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)\n\n        color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)\n\n        _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n        inertia_x = 2*ex -sx\n\n        inertia_y = 2*ey-sy\n\n\n\nif size != BASE_SIZE:\n\n    return cv2.resize(img, (size, size))\n\nelse:\n\n    return img\n</code></pre>\n\n<ul>\n<li>Using raw file</li>\n</ul>\n\n<p>def draw_cv2_color_new(raw_strokes, size=256, lw=6, time_color=True, last_drop_r = 0.0):</p>\n\n<pre><code>stx_min, sty_min = 99999, 99999\n\nstx_max, sty_max = 0,0\n\nett=0  # How fast to complete less than 20 seconds\n\n\n\nfor t, stroke in enumerate(raw_strokes):\n\n    if t == len(raw_strokes) -1:\n\n        ett = int(stroke[2][-1])\n\n    for i in range(len(stroke[0])):\n\n        stx_min = min(stx_min, int(stroke[0][i]))\n\n        stx_max = max(stx_max, int(stroke[0][i]))\n\n        sty_min = min(sty_min, int(stroke[1][i]))\n\n        sty_max = max(sty_max, int(stroke[1][i]))\n\n\n\nlimit_ett = 20*1000   \n\nofs = 15\n\n\n\nif int(sty_max-sty_min+2*ofs) &amp;gt; 6000 or int(stx_max-stx_min+2*ofs)  &amp;gt; 6000:\n\n    img = np.zeros((6000,6000,3), np.uint8)\n\nelse:\n\n    img = np.zeros((int(sty_max-sty_min+2*ofs), int(stx_max-stx_min+2*ofs),3), np.uint8)\n\n\n\nfor t, stroke in enumerate(raw_strokes):     \n\n    inertia_x = 0\n\n    inertia_y = 0\n\n    pre_st_t = 0 \n\n    for i in range(len(stroke[0]) - 1):\n\n        color = int(255 - 245*float(t)/len(raw_strokes)) if time_color else 255 ##  stroke order\n\n        sx = int(stroke[0][i]) - stx_min +ofs\n\n        sy = int(stroke[1][i]) - sty_min +ofs\n\n        st = stroke[2][i]\n\n        ex = int(stroke[0][i + 1])- stx_min +ofs\n\n        ey = int(stroke[1][i + 1])- sty_min +ofs\n\n        et = stroke[2][i+1]\n\n\n\n\n\n        time = et-st\n\n        if time ==0:\n\n            time = 1\n\n\n\n        color_v =  min(int((np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / time)*255.0), 255) ## speed\n\n        color_a = min(int((np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(time*time))*255.0), 255) ## acceleration (1~0)\n\n        _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n\n\n        if i==0:\n\n            color_inter = int((float(et-pre_st_t)/limit_ett)*245)+10\n\n            _ = cv2.circle(img, (sx, sy), lw, (0,0,color_inter), -1) ##interval time\n\n\n\n        if i==len(stroke[0])-2 and t == len(raw_strokes) -1:\n\n            color_end = int((float(ett)/(limit_ett)*245))+10\n\n\n\n            _ = cv2.circle(img, (sx, sy), lw, (0,color_end,0), -1) ##end time\n\n\n\n        inertia_x = 2*ex -sx\n\n        inertia_y = 2*ey-sy\n\n        pre_st_t=et\n\n\n\nreturn cv2.resize(img, (size, size)) #lw reflects how big the picture is drawn, also the aspect ratio is reflected\n</code></pre>\n\n<p>2.. Model Structure</p>\n\n<ul>\n<li><p>Best single model </p>\n\n<p>: InceptionResnetV2 (139,139,3) size input  local valid score is  0.9516.</p>\n\n<p>: using raw file, using 'imagenet' weights, batch size 180</p></li>\n</ul>\n\n<p>base_model = InceptionResNetV2(input_shape=input_shape, weights='imagenet',include_top= False)</p>\n\n<p>x = base_model.output</p>\n\n<p>x = GlobalAveragePooling2D()(x)</p>\n\n<p>x = Dense(1024, activation='rule')(x)</p>\n\n<p>x = Dropout(0.3)(x)</p>\n\n<p>predictions = Dense(340, activation='softmax', name='lastfc')(x)</p>\n\n<p>model = Model(inputs=base_model.input, outputs=predictions)</p>\n\n<p>3.. Optimizer, Loss, etc.</p>\n\n<ul>\n<li><p>In first train about 1epoch(50M set) , I used adam and learning rate 0.002, </p>\n\n<p>categorical cross entropy loss</p></li>\n<li><p>Second train , I used adam accumulation 500 iters and learning rate 0.002, </p>\n\n<p>categorical cross entropy 10% and top3 loss 90%.</p></li>\n<li><p>I did not have enough time to train to the saturation.</p></li>\n</ul>\n\n<h1><a href=\"https://github.com/keras-team/keras/issues/3556\">https://github.com/keras-team/keras/issues/3556</a></h1>\n\n<p>import keras.backend as K</p>\n\n<p>from keras.legacy import interfaces</p>\n\n<p>from keras.optimizers import Optimizer</p>\n\n<p>class AdamAccumulate(Optimizer):</p>\n\n<pre><code>def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n\n             epsilon=None, decay=0., amsgrad=False, accum_iters=1, **kwargs):\n\n    if accum_iters &amp;lt; 1:\n\n        raise ValueError('accum_iters must be &amp;gt;= 1')\n\n    super(AdamAccumulate, self).__init__(**kwargs)\n\n    with K.name_scope(self.__class__.__name__):\n\n        self.iterations = K.variable(0, dtype='int64', name='iterations')\n\n        self.lr = K.variable(lr, name='lr')\n\n        self.beta_1 = K.variable(beta_1, name='beta_1')\n\n        self.beta_2 = K.variable(beta_2, name='beta_2')\n\n        self.decay = K.variable(decay, name='decay')\n\n    if epsilon is None:\n\n        epsilon = K.epsilon()\n\n    self.epsilon = epsilon\n\n    self.initial_decay = decay\n\n    self.amsgrad = amsgrad\n\n    self.accum_iters = K.variable(accum_iters, K.dtype(self.iterations))\n\n    self.accum_iters_float = K.cast(self.accum_iters, K.floatx())\n\n\n\n@interfaces.legacy_get_updates_support\n\ndef get_updates(self, loss, params):\n\n    grads = self.get_gradients(loss, params)\n\n    self.updates = [K.update_add(self.iterations, 1)]\n\n\n\n    lr = self.lr\n\n    completed_updates = K.cast(K.tf.floordiv(self.iterations, self.accum_iters), K.floatx())\n\n    if self.initial_decay &amp;gt; 0:\n\n        lr = lr * (1. / (1. + self.decay * completed_updates))\n\n\n\n    t = completed_updates + 1\n\n\n\n    lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t)))\n\n\n\n    update_switch = K.equal((self.iterations + 1) % self.accum_iters, 0)\n\n    update_switch = K.cast(update_switch, K.floatx())\n\n\n\n    ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    gs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n\n\n    if self.amsgrad:\n\n        vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n    else:\n\n        vhats = [K.zeros(1) for _ in params]\n\n\n\n    self.weights = [self.iterations] + ms + vs + vhats\n\n\n\n    for p, g, m, v, vhat, tg in zip(params, grads, ms, vs, vhats, gs):\n\n\n\n        sum_grad = tg + g\n\n        avg_grad = sum_grad / self.accum_iters_float\n\n\n\n        m_t = (self.beta_1 * m) + (1. - self.beta_1) * avg_grad\n\n        v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(avg_grad)\n\n\n\n        if self.amsgrad:\n\n            vhat_t = K.maximum(vhat, v_t)\n\n            p_t = p - lr_t * m_t / (K.sqrt(vhat_t) + self.epsilon)\n\n            self.updates.append(K.update(vhat, (1 - update_switch) * vhat + update_switch * vhat_t))\n\n        else:\n\n            p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n\n\n        self.updates.append(K.update(m, (1 - update_switch) * m + update_switch * m_t))\n\n        self.updates.append(K.update(v, (1 - update_switch) * v + update_switch * v_t))\n\n        self.updates.append(K.update(tg, (1 - update_switch) * sum_grad))\n\n        new_p = p_t\n\n\n\n        # Apply constraints.\n\n        if getattr(p, 'constraint', None) is not None:\n\n            new_p = p.constraint(new_p)\n\n\n\n        self.updates.append(K.update(p, (1 - update_switch) * p + update_switch * new_p))\n\n    return self.updates\n\ndef get_config(self):\n\n    config = {'lr': float(K.get_value(self.lr)),\n\n              'beta_1': float(K.get_value(self.beta_1)),\n\n              'beta_2': float(K.get_value(self.beta_2)),\n\n              'decay': float(K.get_value(self.decay)),\n\n              'epsilon': self.epsilon,\n\n              'amsgrad': self.amsgrad}\n\n    base_config = super(AdamAccumulate, self).get_config()\n\n    return dict(list(base_config.items()) + list(config.items()))\n</code></pre>\n\n<p>4.. Ensemble</p>\n\n<ul>\n<li><p>I used weighed average ensemble by local valid score and argmax corr.</p></li>\n<li><p>I used InceptionResNetV2 (raw, simple), Xception (raw), Resnet50 (simple).</p></li>\n<li><p>I did not have time to train both (raw, simple)</p></li>\n<li><p>Average weight calculation is below code</p></li>\n</ul>\n\n<p>def get_score_w(local_score):</p>\n\n<pre><code>ls = np.array(local_score)\n\nsub = ls - ls.min()\n\ndiv = sub/sub.max()\n\nadd = div + 0.1\n\nnor = add/add.max()\n\nsqr = nor*nor\n\nprint(sqr)\n\nreturn sqr\n</code></pre>\n\n<p>def get_corr_w(clsnp):</p>\n\n<pre><code>corxlist=[]\n\nfor idx1, cls1 in enumerate((clsnp)):\n\n    corylist=[]\n\n    for idx2, cls2 in enumerate((clsnp)):\n\n        cor_max = np.corrcoef(np.argmax(clsnp[idx1],axis=1),np.argmax(clsnp[idx2],axis=1))[0][1]\n\n        corylist.append(cor_max)\n\n    corxlist.append(corylist)\n\n\n\ndf = pd.DataFrame(corxlist,columns=names, index=names)\n\n#print(df)\n\ncorr_w = []\n\nfor i in range(df.shape[0]):\n\n    count = 0\n\n    thr = 0.95\n\n    for v in df.values[i]:\n\n        if v &amp;gt; thr :\n\n            count+=1\n\n    corr_w.append(1.0/count)   \n\nreturn np.array(corr_w), df\n</code></pre>\n\n<p>def getensemble_w(clsnp, local_score):</p>\n\n<pre><code>score_w = get_score_w(local_score)\n\nfor i in range(score_w.shape[0]):\n\n    if score_w[i]==1.0:\n\n        score_w[i] = 1.2 #max score add 20%\n\ncorr_w = get_corr_w(clsnp)[0]\n\nensemble_w = score_w*corr_w\n\nreturn ensemble_w\n</code></pre>\n\n<p>Everybody enjoy the competition~!!</p>",
      "rawMarkdown": "You may not be interested, but our 24th solution is below.\n\n-. data generate\n\n-. Model\n\n-. optimize, loss, etc\n\n-. ensemble\n\n\n\n1. data generate\n\n - Using simplified file\n\n\n\ndef draw_cv2_color(raw_strokes, size=256, lw=6, time_color=True):\n\n    img = np.zeros((BASE_SIZE, BASE_SIZE,3), np.uint8)\n\n    for t, stroke in enumerate(raw_strokes):     \n\n        inertia_x = 0\n\n        inertia_y = 0\n\n        for i in range(len(stroke[0]) - 1):\n\n            color = int(255 - 245*(float(t)/len(raw_strokes))) if time_color else 255 ## strokes order\n            print(color)\n\n            sx = stroke[0][i]\n\n            sy = stroke[1][i]\n\n            ex = stroke[0][i + 1]\n\n            ey = stroke[1][i + 1]\n\n            color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)\n\n            color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)\n\n            _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n            inertia_x = 2*ex -sx\n\n            inertia_y = 2*ey-sy\n\n            \n\n    if size != BASE_SIZE:\n\n        return cv2.resize(img, (size, size))\n\n    else:\n\n        return img\n\n\n\n\n\n - Using raw file\n\n\n\n\n\ndef draw_cv2_color_new(raw_strokes, size=256, lw=6, time_color=True, last_drop_r = 0.0):\n\n    stx_min, sty_min = 99999, 99999\n\n    stx_max, sty_max = 0,0\n\n    ett=0  # How fast to complete less than 20 seconds\n\n    \n\n    for t, stroke in enumerate(raw_strokes):\n\n        if t == len(raw_strokes) -1:\n\n            ett = int(stroke[2][-1])\n\n        for i in range(len(stroke[0])):\n\n            stx_min = min(stx_min, int(stroke[0][i]))\n\n            stx_max = max(stx_max, int(stroke[0][i]))\n\n            sty_min = min(sty_min, int(stroke[1][i]))\n\n            sty_max = max(sty_max, int(stroke[1][i]))\n\n    \n\n    limit_ett = 20*1000   \n\n    ofs = 15\n\n    \n\n    if int(sty_max-sty_min+2*ofs) &gt; 6000 or int(stx_max-stx_min+2*ofs)  &gt; 6000:\n\n        img = np.zeros((6000,6000,3), np.uint8)\n\n    else:\n\n        img = np.zeros((int(sty_max-sty_min+2*ofs), int(stx_max-stx_min+2*ofs),3), np.uint8)\n\n\n\n    for t, stroke in enumerate(raw_strokes):     \n\n        inertia_x = 0\n\n        inertia_y = 0\n\n        pre_st_t = 0 \n\n        for i in range(len(stroke[0]) - 1):\n\n            color = int(255 - 245*float(t)/len(raw_strokes)) if time_color else 255 ##  stroke order\n\n            sx = int(stroke[0][i]) - stx_min +ofs\n\n            sy = int(stroke[1][i]) - sty_min +ofs\n\n            st = stroke[2][i]\n\n            ex = int(stroke[0][i + 1])- stx_min +ofs\n\n            ey = int(stroke[1][i + 1])- sty_min +ofs\n\n            et = stroke[2][i+1]\n\n            \n\n            \n\n            time = et-st\n\n            if time ==0:\n\n                time = 1\n\n            \n\n            color_v =  min(int((np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / time)*255.0), 255) ## speed\n\n            color_a = min(int((np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(time*time))*255.0), 255) ## acceleration (1~0)\n\n            _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n            \n\n            if i==0:\n\n                color_inter = int((float(et-pre_st_t)/limit_ett)*245)+10\n\n                _ = cv2.circle(img, (sx, sy), lw, (0,0,color_inter), -1) ##interval time\n\n            \n\n            if i==len(stroke[0])-2 and t == len(raw_strokes) -1:\n\n                color_end = int((float(ett)/(limit_ett)*245))+10\n\n\n\n                _ = cv2.circle(img, (sx, sy), lw, (0,color_end,0), -1) ##end time\n\n                \n\n            inertia_x = 2*ex -sx\n\n            inertia_y = 2*ey-sy\n\n            pre_st_t=et\n\n            \n\n    return cv2.resize(img, (size, size)) #lw reflects how big the picture is drawn, also the aspect ratio is reflected\n\n\n\n\n\n\n\n\n\n2.. Model Structure\n\n- Best single model \n\n : InceptionResnetV2 (139,139,3) size input  local valid score is  0.9516.\n\n : using raw file, using 'imagenet' weights, batch size 180\n\n\n\n\n\nbase_model = InceptionResNetV2(input_shape=input_shape, weights='imagenet',include_top= False)\n\nx = base_model.output\n\nx = GlobalAveragePooling2D()(x)\n\nx = Dense(1024, activation='rule')(x)\n\nx = Dropout(0.3)(x)\n\npredictions = Dense(340, activation='softmax', name='lastfc')(x)\n\nmodel = Model(inputs=base_model.input, outputs=predictions)\n\n\n\n\n\n\n\n3.. Optimizer, Loss, etc.\n\n - In first train about 1epoch(50M set) , I used adam and learning rate 0.002, \n\n   categorical cross entropy loss\n\n - Second train , I used adam accumulation 500 iters and learning rate 0.002, \n\n   categorical cross entropy 10% and top3 loss 90%.\n\n - I did not have enough time to train to the saturation.\n\n\n\n#https://github.com/keras-team/keras/issues/3556\n\n\n\nimport keras.backend as K\n\nfrom keras.legacy import interfaces\n\nfrom keras.optimizers import Optimizer\n\nclass AdamAccumulate(Optimizer):\n\n    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n\n                 epsilon=None, decay=0., amsgrad=False, accum_iters=1, **kwargs):\n\n        if accum_iters &lt; 1:\n\n            raise ValueError('accum_iters must be &gt;= 1')\n\n        super(AdamAccumulate, self).__init__(**kwargs)\n\n        with K.name_scope(self.__class__.__name__):\n\n            self.iterations = K.variable(0, dtype='int64', name='iterations')\n\n            self.lr = K.variable(lr, name='lr')\n\n            self.beta_1 = K.variable(beta_1, name='beta_1')\n\n            self.beta_2 = K.variable(beta_2, name='beta_2')\n\n            self.decay = K.variable(decay, name='decay')\n\n        if epsilon is None:\n\n            epsilon = K.epsilon()\n\n        self.epsilon = epsilon\n\n        self.initial_decay = decay\n\n        self.amsgrad = amsgrad\n\n        self.accum_iters = K.variable(accum_iters, K.dtype(self.iterations))\n\n        self.accum_iters_float = K.cast(self.accum_iters, K.floatx())\n\n\n\n    @interfaces.legacy_get_updates_support\n\n    def get_updates(self, loss, params):\n\n        grads = self.get_gradients(loss, params)\n\n        self.updates = [K.update_add(self.iterations, 1)]\n\n\n\n        lr = self.lr\n\n        completed_updates = K.cast(K.tf.floordiv(self.iterations, self.accum_iters), K.floatx())\n\n        if self.initial_decay &gt; 0:\n\n            lr = lr * (1. / (1. + self.decay * completed_updates))\n\n\n\n        t = completed_updates + 1\n\n\n\n        lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t)))\n\n\n\n        update_switch = K.equal((self.iterations + 1) % self.accum_iters, 0)\n\n        update_switch = K.cast(update_switch, K.floatx())\n\n\n\n        ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        gs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n\n\n        if self.amsgrad:\n\n            vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        else:\n\n            vhats = [K.zeros(1) for _ in params]\n\n\n\n        self.weights = [self.iterations] + ms + vs + vhats\n\n\n\n        for p, g, m, v, vhat, tg in zip(params, grads, ms, vs, vhats, gs):\n\n\n\n            sum_grad = tg + g\n\n            avg_grad = sum_grad / self.accum_iters_float\n\n\n\n            m_t = (self.beta_1 * m) + (1. - self.beta_1) * avg_grad\n\n            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(avg_grad)\n\n\n\n            if self.amsgrad:\n\n                vhat_t = K.maximum(vhat, v_t)\n\n                p_t = p - lr_t * m_t / (K.sqrt(vhat_t) + self.epsilon)\n\n                self.updates.append(K.update(vhat, (1 - update_switch) * vhat + update_switch * vhat_t))\n\n            else:\n\n                p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n\n\n            self.updates.append(K.update(m, (1 - update_switch) * m + update_switch * m_t))\n\n            self.updates.append(K.update(v, (1 - update_switch) * v + update_switch * v_t))\n\n            self.updates.append(K.update(tg, (1 - update_switch) * sum_grad))\n\n            new_p = p_t\n\n\n\n            # Apply constraints.\n\n            if getattr(p, 'constraint', None) is not None:\n\n                new_p = p.constraint(new_p)\n\n\n\n            self.updates.append(K.update(p, (1 - update_switch) * p + update_switch * new_p))\n\n        return self.updates\n\n    def get_config(self):\n\n        config = {'lr': float(K.get_value(self.lr)),\n\n                  'beta_1': float(K.get_value(self.beta_1)),\n\n                  'beta_2': float(K.get_value(self.beta_2)),\n\n                  'decay': float(K.get_value(self.decay)),\n\n                  'epsilon': self.epsilon,\n\n                  'amsgrad': self.amsgrad}\n\n        base_config = super(AdamAccumulate, self).get_config()\n\n        return dict(list(base_config.items()) + list(config.items()))\n\n\n\n\n\n4.. Ensemble\n\n- I used weighed average ensemble by local valid score and argmax corr.\n\n- I used InceptionResNetV2 (raw, simple), Xception (raw), Resnet50 (simple).\n\n- I did not have time to train both (raw, simple)\n\n- Average weight calculation is below code\n\n\n\ndef get_score_w(local_score):\n\n    ls = np.array(local_score)\n\n    sub = ls - ls.min()\n\n    div = sub/sub.max()\n\n    add = div + 0.1\n\n    nor = add/add.max()\n\n    sqr = nor*nor\n\n    print(sqr)\n\n    return sqr\n\n\n\ndef get_corr_w(clsnp):\n\n    corxlist=[]\n\n    for idx1, cls1 in enumerate((clsnp)):\n\n        corylist=[]\n\n        for idx2, cls2 in enumerate((clsnp)):\n\n            cor_max = np.corrcoef(np.argmax(clsnp[idx1],axis=1),np.argmax(clsnp[idx2],axis=1))[0][1]\n\n            corylist.append(cor_max)\n\n        corxlist.append(corylist)\n\n\n\n    df = pd.DataFrame(corxlist,columns=names, index=names)\n\n    #print(df)\n\n    corr_w = []\n\n    for i in range(df.shape[0]):\n\n        count = 0\n\n        thr = 0.95\n\n        for v in df.values[i]:\n\n            if v &gt; thr :\n\n                count+=1\n\n        corr_w.append(1.0/count)   \n\n    return np.array(corr_w), df\n\n\n\ndef getensemble_w(clsnp, local_score):\n\n    score_w = get_score_w(local_score)\n\n    for i in range(score_w.shape[0]):\n\n        if score_w[i]==1.0:\n\n            score_w[i] = 1.2 #max score add 20%\n\n    corr_w = get_corr_w(clsnp)[0]\n\n    ensemble_w = score_w*corr_w\n\n    return ensemble_w\n\n\n\n\n\nEverybody enjoy the competition~!!",
      "votes": 41
    },
    {
      "id": 433307,
      "postDate": "2018-12-05T00:59:35.660Z",
      "content": "<p>stroke generate result using raw set</p>",
      "rawMarkdown": "stroke generate result using raw set",
      "votes": 1
    },
    {
      "id": 433305,
      "postDate": "2018-12-05T00:58:46.393Z",
      "content": "<p>stroke generate result using simple set </p>",
      "rawMarkdown": "stroke generate result using simple set ",
      "votes": 1
    },
    {
      "id": 433849,
      "postDate": "2018-12-05T15:07:05.907Z",
      "content": "<p>Thank you and congratulation!!\nI have one question regarding AdamAccumulation : we tried to use this modified optimizer too, but often did not success. Many times, it results in ‘OOM : ResourceExhausion Error’. Did you ever encounter this similar kind of error ?  Nevertheless, it is truly an eye opening to me to know that we can accumulate the batch up to almost 100K (in your case, 180*500 samples) to generate the gradient. I could imagine but never dare to try (too much scare of OOM error)</p>",
      "rawMarkdown": "Thank you and congratulation!!\nI have one question regarding AdamAccumulation : we tried to use this modified optimizer too, but often did not success. Many times, it results in ‘OOM : ResourceExhausion Error’. Did you ever encounter this similar kind of error ?  Nevertheless, it is truly an eye opening to me to know that we can accumulate the batch up to almost 100K (in your case, 180*500 samples) to generate the gradient. I could imagine but never dare to try (too much scare of OOM error)",
      "replies": [
        {
          "id": 433883,
          "postDate": "2018-12-05T15:53:24.717Z",
          "content": "<p>Sorry, but I can not give you advice because I have not experienced the same error as you.</p>",
          "rawMarkdown": "Sorry, but I can not give you advice because I have not experienced the same error as you."
        }
      ]
    },
    {
      "id": 433791,
      "postDate": "2018-12-05T13:36:23.860Z",
      "content": "<p>I wanted to know how the top teams encoded velocity ,acceleration. Thanks for sharing. </p>",
      "rawMarkdown": "I wanted to know how the top teams encoded velocity ,acceleration. Thanks for sharing. "
    },
    {
      "id": 433418,
      "postDate": "2018-12-05T03:49:43.463Z",
      "content": "<p>It was very nice time being with you as a team!. Thanks for your passion, work and sharing! </p>",
      "rawMarkdown": "It was very nice time being with you as a team!. Thanks for your passion, work and sharing! ",
      "replies": [
        {
          "id": 433421,
          "postDate": "2018-12-05T03:56:25.453Z",
          "content": "<p>I enjoyed being with you in this competition. Let 's get together next time~~.</p>",
          "rawMarkdown": "I enjoyed being with you in this competition. Let 's get together next time~~.",
          "votes": 1
        }
      ]
    },
    {
      "id": 433359,
      "postDate": "2018-12-05T02:05:31.377Z",
      "content": "<p>Congrats and thanks for sharing. Would you mind share more intuition about speed and acceleration thing in drawcv2color function?</p>\n\n<p><strong>color_v =  (np.sqrt((sx-ex)<em>(sx-ex) + (sy-ey)</em>(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)</strong></p>\n\n<p><strong>color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)</strong></p>",
      "rawMarkdown": "Congrats and thanks for sharing. Would you mind share more intuition about speed and acceleration thing in drawcv2color function?\n\n**color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)**\n\n**color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)**\n",
      "replies": [
        {
          "id": 433416,
          "postDate": "2018-12-05T03:43:49.510Z",
          "content": "<p>Using simple data set\nRed channel : strokes order\nGreen Channel : 1 stoke distance\nBlue Channel : Distance point specified by previous step inertia</p>\n\n<p><a href=\"https://storage.googleapis.com/kaggle-forum-message-attachments/433305/10829/simple%20set%20generater%20info.PNG\">https://storage.googleapis.com/kaggle-forum-message-attachments/433305/10829/simple%20set%20generater%20info.PNG</a></p>",
          "rawMarkdown": "Using simple data set\nRed channel : strokes order\nGreen Channel : 1 stoke distance\nBlue Channel : Distance point specified by previous step inertia\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/433305/10829/simple%20set%20generater%20info.PNG",
          "votes": 1
        },
        {
          "id": 433482,
          "postDate": "2018-12-05T05:53:25.923Z",
          "content": "<p>Thanks!!</p>",
          "rawMarkdown": "Thanks!!"
        }
      ]
    },
    {
      "id": 433302,
      "postDate": "2018-12-05T00:56:52.650Z",
      "content": "<p>Good Job.</p>",
      "rawMarkdown": "Good Job."
    },
    {
      "id": 433739,
      "postDate": "2018-12-05T12:35:02.977Z",
      "rawMarkdown": "",
      "votes": 6,
      "isDeleted": true
    },
    {
      "id": 433327,
      "postDate": "2018-12-05T01:26:05.960Z",
      "content": "<p>Thank you for sharing!</p>",
      "rawMarkdown": "Thank you for sharing!"
    },
    {
      "id": 433300,
      "postDate": "2018-12-05T00:53:06.160Z",
      "content": "<p>Congrats and thanks for sharing.</p>",
      "rawMarkdown": "Congrats and thanks for sharing."
    }
  ],
  "comments": [
    {
      "id": 433307,
      "author_name": "Daewoo Myoung",
      "author_url": "",
      "post_date": "2018-12-05T00:59:35.660000",
      "content": "<p>stroke generate result using raw set</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 433305,
      "author_name": "Daewoo Myoung",
      "author_url": "",
      "post_date": "2018-12-05T00:58:46.393000",
      "content": "<p>stroke generate result using simple set </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 433849,
      "author_name": "Neuron Engineer",
      "author_url": "",
      "post_date": "2018-12-05T15:07:05.907000",
      "content": "<p>Thank you and congratulation!!\nI have one question regarding AdamAccumulation : we tried to use this modified optimizer too, but often did not success. Many times, it results in ‘OOM : ResourceExhausion Error’. Did you ever encounter this similar kind of error ?  Nevertheless, it is truly an eye opening to me to know that we can accumulate the batch up to almost 100K (in your case, 180*500 samples) to generate the gradient. I could imagine but never dare to try (too much scare of OOM error)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 433883,
          "author_name": "Daewoo Myoung",
          "author_url": "",
          "post_date": "2018-12-05T15:53:24.717000",
          "content": "<p>Sorry, but I can not give you advice because I have not experienced the same error as you.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 433791,
      "author_name": "VikasSangwan",
      "author_url": "",
      "post_date": "2018-12-05T13:36:23.860000",
      "content": "<p>I wanted to know how the top teams encoded velocity ,acceleration. Thanks for sharing. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 433418,
      "author_name": "Youhan Lee",
      "author_url": "",
      "post_date": "2018-12-05T03:49:43.463000",
      "content": "<p>It was very nice time being with you as a team!. Thanks for your passion, work and sharing! </p>",
      "votes": 0,
      "replies": [
        {
          "id": 433421,
          "author_name": "Daewoo Myoung",
          "author_url": "",
          "post_date": "2018-12-05T03:56:25.453000",
          "content": "<p>I enjoyed being with you in this competition. Let 's get together next time~~.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 433359,
      "author_name": "Ao Luo",
      "author_url": "",
      "post_date": "2018-12-05T02:05:31.377000",
      "content": "<p>Congrats and thanks for sharing. Would you mind share more intuition about speed and acceleration thing in drawcv2color function?</p>\n\n<p><strong>color_v =  (np.sqrt((sx-ex)<em>(sx-ex) + (sy-ey)</em>(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)</strong></p>\n\n<p><strong>color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)</strong></p>",
      "votes": 0,
      "replies": [
        {
          "id": 433416,
          "author_name": "Daewoo Myoung",
          "author_url": "",
          "post_date": "2018-12-05T03:43:49.510000",
          "content": "<p>Using simple data set\nRed channel : strokes order\nGreen Channel : 1 stoke distance\nBlue Channel : Distance point specified by previous step inertia</p>\n\n<p><a href=\"https://storage.googleapis.com/kaggle-forum-message-attachments/433305/10829/simple%20set%20generater%20info.PNG\">https://storage.googleapis.com/kaggle-forum-message-attachments/433305/10829/simple%20set%20generater%20info.PNG</a></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 433482,
          "author_name": "Ao Luo",
          "author_url": "",
          "post_date": "2018-12-05T05:53:25.923000",
          "content": "<p>Thanks!!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 433302,
      "author_name": "Gary",
      "author_url": "",
      "post_date": "2018-12-05T00:56:52.650000",
      "content": "<p>Good Job.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 433739,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-12-05T12:35:02.977000",
      "content": "",
      "votes": 6,
      "replies": []
    },
    {
      "id": 433327,
      "author_name": "Soonhwan Kwon",
      "author_url": "",
      "post_date": "2018-12-05T01:26:05.960000",
      "content": "<p>Thank you for sharing!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 433300,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2018-12-05T00:53:06.160000",
      "content": "<p>Congrats and thanks for sharing.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "433298": "You may not be interested, but our 24th solution is below.\n\n-. data generate\n\n-. Model\n\n-. optimize, loss, etc\n\n-. ensemble\n\n\n\n1. data generate\n\n - Using simplified file\n\n\n\ndef draw_cv2_color(raw_strokes, size=256, lw=6, time_color=True):\n\n    img = np.zeros((BASE_SIZE, BASE_SIZE,3), np.uint8)\n\n    for t, stroke in enumerate(raw_strokes):     \n\n        inertia_x = 0\n\n        inertia_y = 0\n\n        for i in range(len(stroke[0]) - 1):\n\n            color = int(255 - 245*(float(t)/len(raw_strokes))) if time_color else 255 ## strokes order\n            print(color)\n\n            sx = stroke[0][i]\n\n            sy = stroke[1][i]\n\n            ex = stroke[0][i + 1]\n\n            ey = stroke[1][i + 1]\n\n            color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)\n\n            color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)\n\n            _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n            inertia_x = 2*ex -sx\n\n            inertia_y = 2*ey-sy\n\n            \n\n    if size != BASE_SIZE:\n\n        return cv2.resize(img, (size, size))\n\n    else:\n\n        return img\n\n\n\n\n\n - Using raw file\n\n\n\n\n\ndef draw_cv2_color_new(raw_strokes, size=256, lw=6, time_color=True, last_drop_r = 0.0):\n\n    stx_min, sty_min = 99999, 99999\n\n    stx_max, sty_max = 0,0\n\n    ett=0  # How fast to complete less than 20 seconds\n\n    \n\n    for t, stroke in enumerate(raw_strokes):\n\n        if t == len(raw_strokes) -1:\n\n            ett = int(stroke[2][-1])\n\n        for i in range(len(stroke[0])):\n\n            stx_min = min(stx_min, int(stroke[0][i]))\n\n            stx_max = max(stx_max, int(stroke[0][i]))\n\n            sty_min = min(sty_min, int(stroke[1][i]))\n\n            sty_max = max(sty_max, int(stroke[1][i]))\n\n    \n\n    limit_ett = 20*1000   \n\n    ofs = 15\n\n    \n\n    if int(sty_max-sty_min+2*ofs) &gt; 6000 or int(stx_max-stx_min+2*ofs)  &gt; 6000:\n\n        img = np.zeros((6000,6000,3), np.uint8)\n\n    else:\n\n        img = np.zeros((int(sty_max-sty_min+2*ofs), int(stx_max-stx_min+2*ofs),3), np.uint8)\n\n\n\n    for t, stroke in enumerate(raw_strokes):     \n\n        inertia_x = 0\n\n        inertia_y = 0\n\n        pre_st_t = 0 \n\n        for i in range(len(stroke[0]) - 1):\n\n            color = int(255 - 245*float(t)/len(raw_strokes)) if time_color else 255 ##  stroke order\n\n            sx = int(stroke[0][i]) - stx_min +ofs\n\n            sy = int(stroke[1][i]) - sty_min +ofs\n\n            st = stroke[2][i]\n\n            ex = int(stroke[0][i + 1])- stx_min +ofs\n\n            ey = int(stroke[1][i + 1])- sty_min +ofs\n\n            et = stroke[2][i+1]\n\n            \n\n            \n\n            time = et-st\n\n            if time ==0:\n\n                time = 1\n\n            \n\n            color_v =  min(int((np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / time)*255.0), 255) ## speed\n\n            color_a = min(int((np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(time*time))*255.0), 255) ## acceleration (1~0)\n\n            _ = cv2.line(img, (sx, sy), (ex, ey), (color,color_v,color_a), lw)\n\n            \n\n            if i==0:\n\n                color_inter = int((float(et-pre_st_t)/limit_ett)*245)+10\n\n                _ = cv2.circle(img, (sx, sy), lw, (0,0,color_inter), -1) ##interval time\n\n            \n\n            if i==len(stroke[0])-2 and t == len(raw_strokes) -1:\n\n                color_end = int((float(ett)/(limit_ett)*245))+10\n\n\n\n                _ = cv2.circle(img, (sx, sy), lw, (0,color_end,0), -1) ##end time\n\n                \n\n            inertia_x = 2*ex -sx\n\n            inertia_y = 2*ey-sy\n\n            pre_st_t=et\n\n            \n\n    return cv2.resize(img, (size, size)) #lw reflects how big the picture is drawn, also the aspect ratio is reflected\n\n\n\n\n\n\n\n\n\n2.. Model Structure\n\n- Best single model \n\n : InceptionResnetV2 (139,139,3) size input  local valid score is  0.9516.\n\n : using raw file, using 'imagenet' weights, batch size 180\n\n\n\n\n\nbase_model = InceptionResNetV2(input_shape=input_shape, weights='imagenet',include_top= False)\n\nx = base_model.output\n\nx = GlobalAveragePooling2D()(x)\n\nx = Dense(1024, activation='rule')(x)\n\nx = Dropout(0.3)(x)\n\npredictions = Dense(340, activation='softmax', name='lastfc')(x)\n\nmodel = Model(inputs=base_model.input, outputs=predictions)\n\n\n\n\n\n\n\n3.. Optimizer, Loss, etc.\n\n - In first train about 1epoch(50M set) , I used adam and learning rate 0.002, \n\n   categorical cross entropy loss\n\n - Second train , I used adam accumulation 500 iters and learning rate 0.002, \n\n   categorical cross entropy 10% and top3 loss 90%.\n\n - I did not have enough time to train to the saturation.\n\n\n\n#https://github.com/keras-team/keras/issues/3556\n\n\n\nimport keras.backend as K\n\nfrom keras.legacy import interfaces\n\nfrom keras.optimizers import Optimizer\n\nclass AdamAccumulate(Optimizer):\n\n    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n\n                 epsilon=None, decay=0., amsgrad=False, accum_iters=1, **kwargs):\n\n        if accum_iters &lt; 1:\n\n            raise ValueError('accum_iters must be &gt;= 1')\n\n        super(AdamAccumulate, self).__init__(**kwargs)\n\n        with K.name_scope(self.__class__.__name__):\n\n            self.iterations = K.variable(0, dtype='int64', name='iterations')\n\n            self.lr = K.variable(lr, name='lr')\n\n            self.beta_1 = K.variable(beta_1, name='beta_1')\n\n            self.beta_2 = K.variable(beta_2, name='beta_2')\n\n            self.decay = K.variable(decay, name='decay')\n\n        if epsilon is None:\n\n            epsilon = K.epsilon()\n\n        self.epsilon = epsilon\n\n        self.initial_decay = decay\n\n        self.amsgrad = amsgrad\n\n        self.accum_iters = K.variable(accum_iters, K.dtype(self.iterations))\n\n        self.accum_iters_float = K.cast(self.accum_iters, K.floatx())\n\n\n\n    @interfaces.legacy_get_updates_support\n\n    def get_updates(self, loss, params):\n\n        grads = self.get_gradients(loss, params)\n\n        self.updates = [K.update_add(self.iterations, 1)]\n\n\n\n        lr = self.lr\n\n        completed_updates = K.cast(K.tf.floordiv(self.iterations, self.accum_iters), K.floatx())\n\n        if self.initial_decay &gt; 0:\n\n            lr = lr * (1. / (1. + self.decay * completed_updates))\n\n\n\n        t = completed_updates + 1\n\n\n\n        lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t)))\n\n\n\n        update_switch = K.equal((self.iterations + 1) % self.accum_iters, 0)\n\n        update_switch = K.cast(update_switch, K.floatx())\n\n\n\n        ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        gs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n\n\n        if self.amsgrad:\n\n            vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]\n\n        else:\n\n            vhats = [K.zeros(1) for _ in params]\n\n\n\n        self.weights = [self.iterations] + ms + vs + vhats\n\n\n\n        for p, g, m, v, vhat, tg in zip(params, grads, ms, vs, vhats, gs):\n\n\n\n            sum_grad = tg + g\n\n            avg_grad = sum_grad / self.accum_iters_float\n\n\n\n            m_t = (self.beta_1 * m) + (1. - self.beta_1) * avg_grad\n\n            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(avg_grad)\n\n\n\n            if self.amsgrad:\n\n                vhat_t = K.maximum(vhat, v_t)\n\n                p_t = p - lr_t * m_t / (K.sqrt(vhat_t) + self.epsilon)\n\n                self.updates.append(K.update(vhat, (1 - update_switch) * vhat + update_switch * vhat_t))\n\n            else:\n\n                p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n\n\n            self.updates.append(K.update(m, (1 - update_switch) * m + update_switch * m_t))\n\n            self.updates.append(K.update(v, (1 - update_switch) * v + update_switch * v_t))\n\n            self.updates.append(K.update(tg, (1 - update_switch) * sum_grad))\n\n            new_p = p_t\n\n\n\n            # Apply constraints.\n\n            if getattr(p, 'constraint', None) is not None:\n\n                new_p = p.constraint(new_p)\n\n\n\n            self.updates.append(K.update(p, (1 - update_switch) * p + update_switch * new_p))\n\n        return self.updates\n\n    def get_config(self):\n\n        config = {'lr': float(K.get_value(self.lr)),\n\n                  'beta_1': float(K.get_value(self.beta_1)),\n\n                  'beta_2': float(K.get_value(self.beta_2)),\n\n                  'decay': float(K.get_value(self.decay)),\n\n                  'epsilon': self.epsilon,\n\n                  'amsgrad': self.amsgrad}\n\n        base_config = super(AdamAccumulate, self).get_config()\n\n        return dict(list(base_config.items()) + list(config.items()))\n\n\n\n\n\n4.. Ensemble\n\n- I used weighed average ensemble by local valid score and argmax corr.\n\n- I used InceptionResNetV2 (raw, simple), Xception (raw), Resnet50 (simple).\n\n- I did not have time to train both (raw, simple)\n\n- Average weight calculation is below code\n\n\n\ndef get_score_w(local_score):\n\n    ls = np.array(local_score)\n\n    sub = ls - ls.min()\n\n    div = sub/sub.max()\n\n    add = div + 0.1\n\n    nor = add/add.max()\n\n    sqr = nor*nor\n\n    print(sqr)\n\n    return sqr\n\n\n\ndef get_corr_w(clsnp):\n\n    corxlist=[]\n\n    for idx1, cls1 in enumerate((clsnp)):\n\n        corylist=[]\n\n        for idx2, cls2 in enumerate((clsnp)):\n\n            cor_max = np.corrcoef(np.argmax(clsnp[idx1],axis=1),np.argmax(clsnp[idx2],axis=1))[0][1]\n\n            corylist.append(cor_max)\n\n        corxlist.append(corylist)\n\n\n\n    df = pd.DataFrame(corxlist,columns=names, index=names)\n\n    #print(df)\n\n    corr_w = []\n\n    for i in range(df.shape[0]):\n\n        count = 0\n\n        thr = 0.95\n\n        for v in df.values[i]:\n\n            if v &gt; thr :\n\n                count+=1\n\n        corr_w.append(1.0/count)   \n\n    return np.array(corr_w), df\n\n\n\ndef getensemble_w(clsnp, local_score):\n\n    score_w = get_score_w(local_score)\n\n    for i in range(score_w.shape[0]):\n\n        if score_w[i]==1.0:\n\n            score_w[i] = 1.2 #max score add 20%\n\n    corr_w = get_corr_w(clsnp)[0]\n\n    ensemble_w = score_w*corr_w\n\n    return ensemble_w\n\n\n\n\n\nEverybody enjoy the competition~!!",
    "433307": "stroke generate result using raw set",
    "433305": "stroke generate result using simple set ",
    "433849": "Thank you and congratulation!!\nI have one question regarding AdamAccumulation : we tried to use this modified optimizer too, but often did not success. Many times, it results in ‘OOM : ResourceExhausion Error’. Did you ever encounter this similar kind of error ?  Nevertheless, it is truly an eye opening to me to know that we can accumulate the batch up to almost 100K (in your case, 180*500 samples) to generate the gradient. I could imagine but never dare to try (too much scare of OOM error)",
    "433791": "I wanted to know how the top teams encoded velocity ,acceleration. Thanks for sharing. ",
    "433418": "It was very nice time being with you as a team!. Thanks for your passion, work and sharing! ",
    "433359": "Congrats and thanks for sharing. Would you mind share more intuition about speed and acceleration thing in drawcv2color function?\n\n**color_v =  (np.sqrt((sx-ex)*(sx-ex) + (sy-ey)*(sy-ey)) / np.sqrt(size*size)) * 255 ## strokes distance like speed (1~0)**\n\n**color_a = (np.sqrt((inertia_x-ex)*(inertia_x-ex) + (inertia_y-ey)*(inertia_y-ey)) / np.sqrt(size*size*4)) * 255 ## strokes distance like acceleration (1~0)**\n",
    "433302": "Good Job.",
    "433739": "",
    "433327": "Thank you for sharing!",
    "433300": "Congrats and thanks for sharing."
  }
}