{
  "id": 434382,
  "title": "how to transfer weights from pytorch to keras to tflite",
  "url": "/competitions/asl-fingerspelling/discussion/434382",
  "author_name": "hengck23",
  "post_date": "2023-08-25T03:19:34.124000",
  "votes": 4,
  "comment_count": 6,
  "views": 0,
  "content": "<p>maybe there is going to the a version.3 of the same competition, so the knowledge of using pytorch for tflite is useful.<br>\n(actually 2nd rank solution in previous competition uses pytorch-keras method, similar to here) </p>\n<h2>step one: you must code in keras like this</h2>\n<pre><code>def :\n    def apply(x):\n        x = tf.keras.layers.(x)\n        x = tf.keras.layers.(x)\n        x = tf.keras.layers.(x)\n        return x\n\n    return apply\n\n## !!! write like this instead  tf.keras layer def make:\n    print('xxxx')\n    num_block = \n    embed_dim =   # cfg.model_dim\n\n    input = tf.keras.\n    x = tf.keras.layers.(input)\n    x = (x)\n    x = tf.keras.layers.(x)\n\n     i  range(num_block):\n        x = (x)\n        x = (x)\n        x = (x)\n        x = (x)\n\n    logit = tf.keras.layers.(x)\n    model = tf.keras.\n    return model\n</code></pre>\n<p>because if you write like this, it is easy to dump keras model weights like a list</p>\n<pre><code>/kernel: (, )\n/pe: (, )\n/gamma: (,)\n/beta: (,)\n/moving_mean: (,)\n/moving_variance: (,)\n/kernel: (, , )\n/gamma: (,)\n/beta: (,)\n/moving_mean: (,)\n/moving_variance: (,)\n/depthwise_kernel: (, , )\n</code></pre>\n<p>my code for dumping keras model weight</p>\n<pre><code> ():\n    model = make_keras_model()\n    x = np.random.rand(, cfg.xyz_max_length, cfg.xyz_dim)\n    y = model(x)\n    (y.shape)\n\n    \n    kdict = {}\n    weight = model.get_weights()\n    name = [weight.name  layer  model.layers  weight  layer.weights]\n     n, w  (name, weight):\n        (n, w.shape)\n        kdict[n] = w\n    ((weight))\n    write_pickle_to_file(, kdict)\n</code></pre>\n<p>now it is easy to modify the kdict with pytorch weight, see below.<br>\nyou can load back kdict to keras easily</p>\n<hr>\n<p>why use pytorch? it trains very much faster then tf if you use \"arbitrary dataset pipline\".<br>\nit is easier to debug in pytorch.</p>\n<p>if you want to debug, you can always use</p>\n<pre><code>x = numpymodel = keras model\n\nthen\n\nx = model.layers[0](x)\nx = model.layers[1](x)\nx = model.layers[2](x) ... you can set breakpoint tothe values\n\n----\n\neasy to catch bug just\ninput x = same as pytorch\nmodel.layers[i] =  as pytorch \noutput x = same as pytorch\n</code></pre>\n<p>you can actually make the names of keras  layer same as pytorch, but when i check the tf namescope, <br>\ni find it difficult to use and i give up and give out.</p>",
  "messages": [
    {
      "id": 2407332,
      "postDate": "2023-08-25T03:19:34.123Z",
      "content": "<p>maybe there is going to the a version.3 of the same competition, so the knowledge of using pytorch for tflite is useful.<br>\n(actually 2nd rank solution in previous competition uses pytorch-keras method, similar to here) </p>\n<h2>step one: you must code in keras like this</h2>\n<pre><code>def :\n    def apply(x):\n        x = tf.keras.layers.(x)\n        x = tf.keras.layers.(x)\n        x = tf.keras.layers.(x)\n        return x\n\n    return apply\n\n## !!! write like this instead  tf.keras layer def make:\n    print('xxxx')\n    num_block = \n    embed_dim =   # cfg.model_dim\n\n    input = tf.keras.\n    x = tf.keras.layers.(input)\n    x = (x)\n    x = tf.keras.layers.(x)\n\n     i  range(num_block):\n        x = (x)\n        x = (x)\n        x = (x)\n        x = (x)\n\n    logit = tf.keras.layers.(x)\n    model = tf.keras.\n    return model\n</code></pre>\n<p>because if you write like this, it is easy to dump keras model weights like a list</p>\n<pre><code>/kernel: (, )\n/pe: (, )\n/gamma: (,)\n/beta: (,)\n/moving_mean: (,)\n/moving_variance: (,)\n/kernel: (, , )\n/gamma: (,)\n/beta: (,)\n/moving_mean: (,)\n/moving_variance: (,)\n/depthwise_kernel: (, , )\n</code></pre>\n<p>my code for dumping keras model weight</p>\n<pre><code> ():\n    model = make_keras_model()\n    x = np.random.rand(, cfg.xyz_max_length, cfg.xyz_dim)\n    y = model(x)\n    (y.shape)\n\n    \n    kdict = {}\n    weight = model.get_weights()\n    name = [weight.name  layer  model.layers  weight  layer.weights]\n     n, w  (name, weight):\n        (n, w.shape)\n        kdict[n] = w\n    ((weight))\n    write_pickle_to_file(, kdict)\n</code></pre>\n<p>now it is easy to modify the kdict with pytorch weight, see below.<br>\nyou can load back kdict to keras easily</p>\n<hr>\n<p>why use pytorch? it trains very much faster then tf if you use \"arbitrary dataset pipline\".<br>\nit is easier to debug in pytorch.</p>\n<p>if you want to debug, you can always use</p>\n<pre><code>x = numpymodel = keras model\n\nthen\n\nx = model.layers[0](x)\nx = model.layers[1](x)\nx = model.layers[2](x) ... you can set breakpoint tothe values\n\n----\n\neasy to catch bug just\ninput x = same as pytorch\nmodel.layers[i] =  as pytorch \noutput x = same as pytorch\n</code></pre>\n<p>you can actually make the names of keras  layer same as pytorch, but when i check the tf namescope, <br>\ni find it difficult to use and i give up and give out.</p>",
      "rawMarkdown": "maybe there is going to the a version.3 of the same competition, so the knowledge of using pytorch for tflite is useful.\n(actually 2nd rank solution in previous competition uses pytorch-keras method, similar to here) \n\n## step one: you must code in keras like this\n```\ndef FeedForward(x_dim, hidden_dim):\n\tdef apply(x):\n\t\tx = tf.keras.layers.Dense(hidden_dim)(x)\n\t\tx = tf.keras.layers.ReLU()(x)\n\t\tx = tf.keras.layers.Dense(x_dim)(x)\n\t\treturn x\n\n\treturn apply\n\n## !!! write like this instead of tf.keras layer class ...\n\n\ndef make_keras_model():\n\tprint('xxxx')\n\tnum_block = 6\n\tembed_dim = 192  # cfg.model_dim\n\n\tinput = tf.keras.Input(INPUT_SHAPE)\n\tx = tf.keras.layers.Dense(embed_dim, use_bias=False)(input)\n\tx = PositionalEncoding(cfg.xyz_max_length, embed_dim)(x)\n\tx = tf.keras.layers.BatchNormalization(epsilon=1e-05)(x)\n\n\tfor i in range(num_block):\n\t\tx = ResBlock(embed_dim, kernel_size=11,)(x)\n\t\tx = ResBlock(embed_dim, kernel_size=5, )(x)\n\t\tx = ResBlock(embed_dim, kernel_size=3, )(x)\n\t\tx = TransformerBlock(embed_dim, num_head=8)(x)\n\n\tlogit = tf.keras.layers.Dense(cfg.vocab_size, name='logit')(x)\n\tmodel = tf.keras.Model(logit, x)\n\treturn model\n\n```\n\nbecause if you write like this, it is easy to dump keras model weights like a list\n\n```\ndense/kernel:0 (198, 192)\npositional_encoding/pe:0 (256, 192)\nbatch_normalization/gamma:0 (192,)\nbatch_normalization/beta:0 (192,)\nbatch_normalization/moving_mean:0 (192,)\nbatch_normalization/moving_variance:0 (192,)\nconv1d/kernel:0 (1, 192, 384)\nbatch_normalization_1/gamma:0 (384,)\nbatch_normalization_1/beta:0 (384,)\nbatch_normalization_1/moving_mean:0 (384,)\nbatch_normalization_1/moving_variance:0 (384,)\ndepthwise_conv1d/depthwise_kernel:0 (11, 384, 1)\n\n```\n\nmy code for dumping keras model weight\n\n```\ndef run_dump_weight(dir):\n\tmodel = make_keras_model()\n\tx = np.random.rand(1, cfg.xyz_max_length, cfg.xyz_dim)\n\ty = model(x)\n\tprint(y.shape)\n\n\t# -----\n\tkdict = {}\n\tweight = model.get_weights()\n\tname = [weight.name for layer in model.layers for weight in layer.weights]\n\tfor n, w in zip(name, weight):\n\t\tprint(n, w.shape)\n\t\tkdict[n] = w\n\tprint(len(weight))\n\twrite_pickle_to_file(f'{dir}/kdict.pickle', kdict)\n```\n\nnow it is easy to modify the kdict with pytorch weight, see below.\nyou can load back kdict to keras easily\n\n---\n\nwhy use pytorch? it trains very much faster then tf if you use \"arbitrary dataset pipline\".\nit is easier to debug in pytorch.\n\n \nif you want to debug, you can always use\n\n```\nx = numpy array\nmodel = keras model\n\nthen\n\nx = model.layers[0](x)\nx = model.layers[1](x)\nx = model.layers[2](x) ... you can set breakpoint to check the values\n\n----\n\neasy to catch bug just check\n\ninput x = same as pytorch\nmodel.layers[i] = \"correctly transposed\" as pytorch \noutput x = same as pytorch\n\n```\n\nyou can actually make the names of keras  layer same as pytorch, but when i check the tf namescope, \ni find it difficult to use and i give up and give out.\n\n",
      "votes": 4
    },
    {
      "id": 2407349,
      "postDate": "2023-08-25T03:39:07.197Z",
      "content": "<p>Actually you could use wonderfull lib nocubo</p>",
      "rawMarkdown": "Actually you could use wonderfull lib nocubo",
      "votes": 2
    },
    {
      "id": 2407350,
      "postDate": "2023-08-25T03:40:34.337Z",
      "content": "<h2>step.4 load converted weights</h2>\n<p>finally</p>\n<pre><code>       dir = \n    cdict = (f)\n    weight = \n\n    model = ()\n    x = np(, cfg, cfg.xyz_dim)\n    y = (x)\n    (y.shape)\n    model(weight)\n    model(\n        f\n    )\n</code></pre>\n<p>saving as ckpt or h5 are optional since you can already load from dictionary of numpy array using set_weight() </p>",
      "rawMarkdown": "## step.4 load converted weights\n\nfinally\n\n```\n\n       dir = 'dump'\n\tcdict = read_pickle_from_file(f'{dir}/cdict1.pickle')\n\tweight = [w for n, w in cdict.items()]\n\n\tmodel = make_model()\n\tx = np.random.rand(1, cfg.xyz_max_length, cfg.xyz_dim)\n\ty = model(x)\n\tprint(y.shape)\n\tmodel.set_weights(weight)\n\tmodel.save_weights(\n\t\tf'{dir}/cdict1-from-pytorch.ckpt'\n\t)\n```\n\nsaving as ckpt or h5 are optional since you can already load from dictionary of numpy array using set\\_weight() "
    },
    {
      "id": 2407339,
      "postDate": "2023-08-25T03:29:58.067Z",
      "content": "<h2>step 3. copy the weights.</h2>\n<p>here are example to convert the common layers like nn.linear, conv1d, layernorm etc …<br>\nyou need to transpose because one use NCHW, the other uses NHWC …</p>\n<p>note that batch/layer norm eps values are different. remember to use same values.<br>\nother differences are convolution padding if your kernel size is even number like 2,4,6</p>\n<pre><code>\ne.g.   pytorch is   keras\n\ndef convert_dense(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(1,0)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_conv1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(2,1,0)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_dwconv1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None) ##&lt;todo&gt;\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(2,0,1,)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_batchnorm1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +)\n    pmean = pdict.(pname +)\n    pvar = pdict.(pname +)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    kmean = kdict.(kname +)\n    kvar = kdict.(kname +)\n\n    w,b,mean,var = pw,pb,pmean,pvar\n    assert(w.==kw.shape)\n    assert(b.==kb.shape)\n    assert(mean.==kmean.shape)\n    assert(var.==kvar.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n    cdict[kname +] = np.ascontiguousarray(b)\n    cdict[kname +] = np.ascontiguousarray(mean)\n    cdict[kname +] = np.ascontiguousarray(var)\n\n\ndef convert_layernorm(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n\n    w,b = pw,pb\n    assert(w.==kw.shape)\n    assert(b.==kb.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n    cdict[kname +] = np.ascontiguousarray(b)\n</code></pre>",
      "rawMarkdown": "##step 3. copy the weights.\n\nhere are example to convert the common layers like nn.linear, conv1d, layernorm etc ...\nyou need to transpose because one use NCHW, the other uses NHWC ...\n\nnote that batch/layer norm eps values are different. remember to use same values.\nother differences are convolution padding if your kernel size is even number like 2,4,6\n\n```\n## both keras and python weights are now in numpy and stored as dictionary. so we just need to find/match the correct dictionary key/name.\ne.g. 'weight' in pytorch is 'kernel:0' in keras\n\ndef convert_dense(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'kernel:0')\n\tkb = kdict.get(kname +'bias:0',None)\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(1,0)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_conv1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'kernel:0')\n\tkb = kdict.get(kname +'bias:0',None)\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(2,1,0)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_dwconv1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'depthwise_kernel:0')\n\tkb = kdict.get(kname +'bias:0',None) ##<todo>\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(2,0,1,)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'depthwise_kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_batchnorm1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias')\n\tpmean = pdict.get(pname +'running_mean')\n\tpvar = pdict.get(pname +'running_var')\n\n\tkw = kdict.get(kname +'gamma:0')\n\tkb = kdict.get(kname +'beta:0',None)\n\tkmean = kdict.get(kname +'moving_mean:0')\n\tkvar = kdict.get(kname +'moving_variance:0')\n\n\tw,b,mean,var = pw,pb,pmean,pvar\n\tassert(w.shape==kw.shape)\n\tassert(b.shape==kb.shape)\n\tassert(mean.shape==kmean.shape)\n\tassert(var.shape==kvar.shape)\n\tcdict[kname +'gamma:0'] = np.ascontiguousarray(w)\n\tcdict[kname +'beta:0'] = np.ascontiguousarray(b)\n\tcdict[kname +'moving_mean:0'] = np.ascontiguousarray(mean)\n\tcdict[kname +'moving_variance:0'] = np.ascontiguousarray(var)\n\n#--------------\ndef convert_layernorm(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias')\n\n\tkw = kdict.get(kname +'gamma:0')\n\tkb = kdict.get(kname +'beta:0',None)\n\n\tw,b = pw,pb\n\tassert(w.shape==kw.shape)\n\tassert(b.shape==kb.shape)\n\tcdict[kname +'gamma:0'] = np.ascontiguousarray(w)\n\tcdict[kname +'beta:0'] = np.ascontiguousarray(b)\n\n\n```",
      "replies": [
        {
          "id": 2407345,
          "postDate": "2023-08-25T03:35:35.117Z",
          "content": "<p>this is how i convert for my resnet block and tx block</p>\n<pre><code> ():\n    pdict = read_pickle_from_file()\n    kdict = read_pickle_from_file()\n    cdict = {n:  n,w  kdict.items()}\n\n\n     :\n        num_block=\n         i  (, num_block):\n             j  [,,]:\n                \n                pname=[\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                kname=[\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                 :\n                    kname = [ k.replace(,)  k  kname]\n                convert_res_block( cdict, kdict, pdict, pname, kname)\n\n     :\n        num_block = \n         i  (, num_block):\n                \n                pname = [\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                kname = [\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                 :\n                    kname = [k.replace(, )  k  kname]\n                convert_tx_block(cdict, kdict, pdict, pname, kname)\n</code></pre>\n<pre><code>def convert_tx_block(\n    cdict, kdict, pdict, pname, kname,\n):\n    i=\n    try:\n        #multi head attention (q,k,v, out)\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        #ffn\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=\n    except:\n        print(,pname[i], kname[i])\n\ndef convert_res_block(\n    cdict, kdict, pdict, pname, kname,\n):\n    i=\n    try:\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_dwconv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        #squeeze-excite\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n    except:\n        print(,pname[i], kname[i])\n</code></pre>",
          "rawMarkdown": "this is how i convert for my resnet block and tx block\n\n```\ndef make_weights_for_keras(\n\tdir,\n\tpdict_name='pdict',\n\tkdict_name='kdict',\n\tcdict_name='cdict',\n):\n\tpdict = read_pickle_from_file(f'{dir}/{pdict_name}.pickle')\n\tkdict = read_pickle_from_file(f'{dir}/{kdict_name}.pickle')\n\tcdict = {n:None for n,w in kdict.items()}\n\n\n\tif 1:\n\t\tnum_block=6\n\t\tfor i in range(0, num_block):\n\t\t\tfor j in [0,1,2]:\n\t\t\t\t#print('res_block', i,j)\n\t\t\t\tpname=[\n\t\t\t\t\tf'block.{i}.{j}.conv1.0.',\n\t\t\t\t\tf'block.{i}.{j}.conv1.1.',\n\t\t\t\t\tf'block.{i}.{j}.conv2.0.',\n\t\t\t\t\tf'block.{i}.{j}.conv2.1.',\n\t\t\t\t\tf'block.{i}.{j}.squeeze.mlp.0.',\n\t\t\t\t\tf'block.{i}.{j}.squeeze.mlp.2.',\n\t\t\t\t\tf'block.{i}.{j}.conv3.0.',\n\t\t\t\t\tf'block.{i}.{j}.conv3.1.',\n\t\t\t\t]\n\t\t\t\tkname=[\n\t\t\t\t\tf'conv1d_{(i*3+j)*4}/',\n\t\t\t\t\tf'batch_normalization_{(i*3+j)*3+1}/',\n\t\t\t\t\tf'depthwise_conv1d_{i*3+j}/',\n\t\t\t\t\tf'batch_normalization_{(i*3+j)*3+1+1}/',\n\t\t\t\t\tf'conv1d_{(i*3+j)*4+1}/',\n\t\t\t\t\tf'conv1d_{(i*3+j)*4+2}/',\n\t\t\t\t\tf'conv1d_{(i*3+j)*4+3}/',\n\t\t\t\t\tf'batch_normalization_{(i*3+j)*3+1+2}/',\n\t\t\t\t]\n\t\t\t\tif 1:\n\t\t\t\t\tkname = [ k.replace('_0','') for k in kname]\n\t\t\t\tconvert_res_block( cdict, kdict, pdict, pname, kname)\n\n\tif 1:\n\t\tnum_block = 6\n\t\tfor i in range(0, num_block):\n\t\t\t\t#print('convert_tx_block', i)\n\t\t\t\tpname = [\n\t\t\t\t\tf'block.{i}.3.attn.q.',\n\t\t\t\t\tf'block.{i}.3.attn.k.',\n\t\t\t\t\tf'block.{i}.3.attn.v.',\n\t\t\t\t\tf'block.{i}.3.attn.out.',\n\t\t\t\t\tf'block.{i}.3.ffn.mlp.0.',\n\t\t\t\t\tf'block.{i}.3.ffn.mlp.2.',\n\t\t\t\t\tf'block.{i}.3.norm1.',\n\t\t\t\t\tf'block.{i}.3.norm2.',\n\t\t\t\t]\n\t\t\t\tkname = [\n\t\t\t\t\tf'my_multi_head_attention_{i}/dense_{i*6+1}/',\n\t\t\t\t\tf'my_multi_head_attention_{i}/dense_{i*6+2}/',\n\t\t\t\t\tf'my_multi_head_attention_{i}/dense_{i*6+3}/',\n\t\t\t\t\tf'my_multi_head_attention_{i}/dense_{i*6+4}/',\n\t\t\t\t\tf'dense_{i*6+5}/',\n\t\t\t\t\tf'dense_{i*6+6}/',\n\t\t\t\t\tf'layer_normalization_{2*i}/',\n\t\t\t\t\tf'layer_normalization_{2*i+1}/',\n\t\t\t\t]\n\t\t\t\tif 1:\n\t\t\t\t\tkname = [k.replace('_0', '') for k in kname]\n\t\t\t\tconvert_tx_block(cdict, kdict, pdict, pname, kname)\n\n```\n\n```\ndef convert_tx_block(\n\tcdict, kdict, pdict, pname, kname,\n):\n\ti=0\n\ttry:\n\t\t#multi head attention (q,k,v, out)\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\n\t\t#ffn\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\n\t\tconvert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\texcept:\n\t\tprint('\\t ERROR!!!!',pname[i], kname[i])\n\ndef convert_res_block(\n\tcdict, kdict, pdict, pname, kname,\n):\n\ti=0\n\ttry:\n\t\tconvert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\n\t\tconvert_dwconv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\n\t\t#squeeze-excite\n\t\tconvert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\n\t\tconvert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\t\tconvert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=1\n\texcept:\n\t\tprint('\\t ERROR!!!!',pname[i], kname[i])\n\n```\n\n"
        }
      ]
    },
    {
      "id": 2407338,
      "postDate": "2023-08-25T03:27:47.010Z",
      "content": "<p>Very very useful 👍👍👍<br>\nFor the past 3 months, I've been always using TensorFlow and sometimes feel so miserable. Really really want to use Pytorch next time 🥶</p>",
      "rawMarkdown": "Very very useful 👍👍👍\nFor the past 3 months, I've been always using TensorFlow and sometimes feel so miserable. Really really want to use Pytorch next time 🥶"
    },
    {
      "id": 2407335,
      "postDate": "2023-08-25T03:21:28.217Z",
      "content": "<h2>step.two. dumpy pytorch state dictionary</h2>\n<pre><code>\n (nn.Module):\n     ():\n        ().__init__()\n        .mlp = nn.Sequential(\n            nn.Linear(x_dim, hidden_dim),\n            nn.ReLU(inplace=True),\n            nn.Linear(hidden_dim, x_dim),\n        )\n     ():\n         .mlp(x)\n</code></pre>\n<p>it is easier to dump pytorch dict into numpy as well. there are some bugs to include both\"import torch\" and \"import tensorflow\" in the same python script file.</p>\n<p>so we first dump everything as numpy and work work numpy instead.</p>\n<pre><code>     ():\n        checkpoint_file = \\\n            \n\n\n        f = torch.load(checkpoint_file, map_location= storage, loc: storage)\n        state_dict = f[]\n\n        (, (state_dict))  \n        pdict = {}\n         n, w  state_dict.items():\n               n: \n            w = w.data.numpy()\n            (n, w.shape)\n            pdict[n] = w\n        ((pdict))\n        write_pickle_to_file(, pdict)\n</code></pre>",
      "rawMarkdown": "##step.two. dumpy pytorch state dictionary \n\n\n\n```\n#equivalent pytorch code for the above example:\nclass FeedForward(nn.Module):\n\tdef __init__(self, x_dim, hidden_dim):\n\t\tsuper().__init__()\n\t\tself.mlp = nn.Sequential(\n\t\t\tnn.Linear(x_dim, hidden_dim),\n\t\t\tnn.ReLU(inplace=True),\n\t\t\tnn.Linear(hidden_dim, x_dim),\n\t\t)\n\tdef forward(self, x):\n\t\treturn self.mlp(x)\n```\n\nit is easier to dump pytorch dict into numpy as well. there are some bugs to include both\"import torch\" and \"import tensorflow\" in the same python script file.\n\nso we first dump everything as numpy and work work numpy instead.\n\n```\n\n\n\tdef run_dump_weight(dir='dump', out_name='pdict2'):\n\t\tcheckpoint_file = \\\n\t\t\t'/home/titanx/hengck/share1/kaggle/2022/google-finger-spell/result/MULTI-HEAD/trained/fold-2/00040139.pth'\n\n\n\t\tf = torch.load(checkpoint_file, map_location=lambda storage, loc: storage)\n\t\tstate_dict = f['state_dict']\n\n\t\tprint('state_dict)', len(state_dict))  # 501\n\t\tpdict = {}\n\t\tfor n, w in state_dict.items():\n\t\t\tif 'num_batches_tracked' in n: continue\n\t\t\tw = w.data.numpy()\n\t\t\tprint(n, w.shape)\n\t\t\tpdict[n] = w\n\t\tprint(len(pdict))\n\t\twrite_pickle_to_file(f'{dir}/{out_name}.pickle', pdict)\n\t\n```\n\n"
    }
  ],
  "comments": [
    {
      "id": 2407349,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2023-08-25T03:39:07.197000",
      "content": "<p>Actually you could use wonderfull lib nocubo</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2407350,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2023-08-25T03:40:34.337000",
      "content": "<h2>step.4 load converted weights</h2>\n<p>finally</p>\n<pre><code>       dir = \n    cdict = (f)\n    weight = \n\n    model = ()\n    x = np(, cfg, cfg.xyz_dim)\n    y = (x)\n    (y.shape)\n    model(weight)\n    model(\n        f\n    )\n</code></pre>\n<p>saving as ckpt or h5 are optional since you can already load from dictionary of numpy array using set_weight() </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2407339,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2023-08-25T03:29:58.067000",
      "content": "<h2>step 3. copy the weights.</h2>\n<p>here are example to convert the common layers like nn.linear, conv1d, layernorm etc …<br>\nyou need to transpose because one use NCHW, the other uses NHWC …</p>\n<p>note that batch/layer norm eps values are different. remember to use same values.<br>\nother differences are convolution padding if your kernel size is even number like 2,4,6</p>\n<pre><code>\ne.g.   pytorch is   keras\n\ndef convert_dense(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(1,0)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_conv1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(2,1,0)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_dwconv1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +,None)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None) ##&lt;todo&gt;\n    assert((pb is None) + (kb is None) !=1)\n\n    w = pw.transpose(2,0,1,)\n    assert(w.==kw.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n\n     kb is  None:\n        b = pb\n        assert(b.==kb.shape)\n        cdict[kname + ] = np.ascontiguousarray(b)\n\n\ndef convert_batchnorm1d(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +)\n    pmean = pdict.(pname +)\n    pvar = pdict.(pname +)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n    kmean = kdict.(kname +)\n    kvar = kdict.(kname +)\n\n    w,b,mean,var = pw,pb,pmean,pvar\n    assert(w.==kw.shape)\n    assert(b.==kb.shape)\n    assert(mean.==kmean.shape)\n    assert(var.==kvar.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n    cdict[kname +] = np.ascontiguousarray(b)\n    cdict[kname +] = np.ascontiguousarray(mean)\n    cdict[kname +] = np.ascontiguousarray(var)\n\n\ndef convert_layernorm(\n    cdict, kdict, pdict, pname, kname,\n):\n    pw = pdict.(pname +)\n    pb = pdict.(pname +)\n\n    kw = kdict.(kname +)\n    kb = kdict.(kname +,None)\n\n    w,b = pw,pb\n    assert(w.==kw.shape)\n    assert(b.==kb.shape)\n    cdict[kname +] = np.ascontiguousarray(w)\n    cdict[kname +] = np.ascontiguousarray(b)\n</code></pre>",
      "votes": 0,
      "replies": [
        {
          "id": 2407345,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2023-08-25T03:35:35.117000",
          "content": "<p>this is how i convert for my resnet block and tx block</p>\n<pre><code> ():\n    pdict = read_pickle_from_file()\n    kdict = read_pickle_from_file()\n    cdict = {n:  n,w  kdict.items()}\n\n\n     :\n        num_block=\n         i  (, num_block):\n             j  [,,]:\n                \n                pname=[\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                kname=[\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                 :\n                    kname = [ k.replace(,)  k  kname]\n                convert_res_block( cdict, kdict, pdict, pname, kname)\n\n     :\n        num_block = \n         i  (, num_block):\n                \n                pname = [\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                kname = [\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                    ,\n                ]\n                 :\n                    kname = [k.replace(, )  k  kname]\n                convert_tx_block(cdict, kdict, pdict, pname, kname)\n</code></pre>\n<pre><code>def convert_tx_block(\n    cdict, kdict, pdict, pname, kname,\n):\n    i=\n    try:\n        #multi head attention (q,k,v, out)\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        #ffn\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_dense(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_layernorm(cdict, kdict, pdict, pname[i], kname[i]); i+=\n    except:\n        print(,pname[i], kname[i])\n\ndef convert_res_block(\n    cdict, kdict, pdict, pname, kname,\n):\n    i=\n    try:\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_dwconv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        #squeeze-excite\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n\n        convert_conv1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n        convert_batchnorm1d(cdict, kdict, pdict, pname[i], kname[i]); i+=\n    except:\n        print(,pname[i], kname[i])\n</code></pre>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2407338,
      "author_name": "Yu Wu",
      "author_url": "",
      "post_date": "2023-08-25T03:27:47.010000",
      "content": "<p>Very very useful 👍👍👍<br>\nFor the past 3 months, I've been always using TensorFlow and sometimes feel so miserable. Really really want to use Pytorch next time 🥶</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2407335,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2023-08-25T03:21:28.217000",
      "content": "<h2>step.two. dumpy pytorch state dictionary</h2>\n<pre><code>\n (nn.Module):\n     ():\n        ().__init__()\n        .mlp = nn.Sequential(\n            nn.Linear(x_dim, hidden_dim),\n            nn.ReLU(inplace=True),\n            nn.Linear(hidden_dim, x_dim),\n        )\n     ():\n         .mlp(x)\n</code></pre>\n<p>it is easier to dump pytorch dict into numpy as well. there are some bugs to include both\"import torch\" and \"import tensorflow\" in the same python script file.</p>\n<p>so we first dump everything as numpy and work work numpy instead.</p>\n<pre><code>     ():\n        checkpoint_file = \\\n            \n\n\n        f = torch.load(checkpoint_file, map_location= storage, loc: storage)\n        state_dict = f[]\n\n        (, (state_dict))  \n        pdict = {}\n         n, w  state_dict.items():\n               n: \n            w = w.data.numpy()\n            (n, w.shape)\n            pdict[n] = w\n        ((pdict))\n        write_pickle_to_file(, pdict)\n</code></pre>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2407332": "maybe there is going to the a version.3 of the same competition, so the knowledge of using pytorch for tflite is useful.\n(actually 2nd rank solution in previous competition uses pytorch-keras method, similar to here) \n\n## step one: you must code in keras like this\n```\ndef FeedForward(x_dim, hidden_dim):\n\tdef apply(x):\n\t\tx = tf.keras.layers.Dense(hidden_dim)(x)\n\t\tx = tf.keras.layers.ReLU()(x)\n\t\tx = tf.keras.layers.Dense(x_dim)(x)\n\t\treturn x\n\n\treturn apply\n\n## !!! write like this instead of tf.keras layer class ...\n\n\ndef make_keras_model():\n\tprint('xxxx')\n\tnum_block = 6\n\tembed_dim = 192  # cfg.model_dim\n\n\tinput = tf.keras.Input(INPUT_SHAPE)\n\tx = tf.keras.layers.Dense(embed_dim, use_bias=False)(input)\n\tx = PositionalEncoding(cfg.xyz_max_length, embed_dim)(x)\n\tx = tf.keras.layers.BatchNormalization(epsilon=1e-05)(x)\n\n\tfor i in range(num_block):\n\t\tx = ResBlock(embed_dim, kernel_size=11,)(x)\n\t\tx = ResBlock(embed_dim, kernel_size=5, )(x)\n\t\tx = ResBlock(embed_dim, kernel_size=3, )(x)\n\t\tx = TransformerBlock(embed_dim, num_head=8)(x)\n\n\tlogit = tf.keras.layers.Dense(cfg.vocab_size, name='logit')(x)\n\tmodel = tf.keras.Model(logit, x)\n\treturn model\n\n```\n\nbecause if you write like this, it is easy to dump keras model weights like a list\n\n```\ndense/kernel:0 (198, 192)\npositional_encoding/pe:0 (256, 192)\nbatch_normalization/gamma:0 (192,)\nbatch_normalization/beta:0 (192,)\nbatch_normalization/moving_mean:0 (192,)\nbatch_normalization/moving_variance:0 (192,)\nconv1d/kernel:0 (1, 192, 384)\nbatch_normalization_1/gamma:0 (384,)\nbatch_normalization_1/beta:0 (384,)\nbatch_normalization_1/moving_mean:0 (384,)\nbatch_normalization_1/moving_variance:0 (384,)\ndepthwise_conv1d/depthwise_kernel:0 (11, 384, 1)\n\n```\n\nmy code for dumping keras model weight\n\n```\ndef run_dump_weight(dir):\n\tmodel = make_keras_model()\n\tx = np.random.rand(1, cfg.xyz_max_length, cfg.xyz_dim)\n\ty = model(x)\n\tprint(y.shape)\n\n\t# -----\n\tkdict = {}\n\tweight = model.get_weights()\n\tname = [weight.name for layer in model.layers for weight in layer.weights]\n\tfor n, w in zip(name, weight):\n\t\tprint(n, w.shape)\n\t\tkdict[n] = w\n\tprint(len(weight))\n\twrite_pickle_to_file(f'{dir}/kdict.pickle', kdict)\n```\n\nnow it is easy to modify the kdict with pytorch weight, see below.\nyou can load back kdict to keras easily\n\n---\n\nwhy use pytorch? it trains very much faster then tf if you use \"arbitrary dataset pipline\".\nit is easier to debug in pytorch.\n\n \nif you want to debug, you can always use\n\n```\nx = numpy array\nmodel = keras model\n\nthen\n\nx = model.layers[0](x)\nx = model.layers[1](x)\nx = model.layers[2](x) ... you can set breakpoint to check the values\n\n----\n\neasy to catch bug just check\n\ninput x = same as pytorch\nmodel.layers[i] = \"correctly transposed\" as pytorch \noutput x = same as pytorch\n\n```\n\nyou can actually make the names of keras  layer same as pytorch, but when i check the tf namescope, \ni find it difficult to use and i give up and give out.\n\n",
    "2407349": "Actually you could use wonderfull lib nocubo",
    "2407350": "## step.4 load converted weights\n\nfinally\n\n```\n\n       dir = 'dump'\n\tcdict = read_pickle_from_file(f'{dir}/cdict1.pickle')\n\tweight = [w for n, w in cdict.items()]\n\n\tmodel = make_model()\n\tx = np.random.rand(1, cfg.xyz_max_length, cfg.xyz_dim)\n\ty = model(x)\n\tprint(y.shape)\n\tmodel.set_weights(weight)\n\tmodel.save_weights(\n\t\tf'{dir}/cdict1-from-pytorch.ckpt'\n\t)\n```\n\nsaving as ckpt or h5 are optional since you can already load from dictionary of numpy array using set\\_weight() ",
    "2407339": "##step 3. copy the weights.\n\nhere are example to convert the common layers like nn.linear, conv1d, layernorm etc ...\nyou need to transpose because one use NCHW, the other uses NHWC ...\n\nnote that batch/layer norm eps values are different. remember to use same values.\nother differences are convolution padding if your kernel size is even number like 2,4,6\n\n```\n## both keras and python weights are now in numpy and stored as dictionary. so we just need to find/match the correct dictionary key/name.\ne.g. 'weight' in pytorch is 'kernel:0' in keras\n\ndef convert_dense(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'kernel:0')\n\tkb = kdict.get(kname +'bias:0',None)\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(1,0)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_conv1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'kernel:0')\n\tkb = kdict.get(kname +'bias:0',None)\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(2,1,0)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_dwconv1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias',None)\n\n\tkw = kdict.get(kname +'depthwise_kernel:0')\n\tkb = kdict.get(kname +'bias:0',None) ##<todo>\n\tassert((pb is None) + (kb is None) !=1)\n\n\tw = pw.transpose(2,0,1,)\n\tassert(w.shape==kw.shape)\n\tcdict[kname +'depthwise_kernel:0'] = np.ascontiguousarray(w)\n\n\tif kb is not None:\n\t\tb = pb\n\t\tassert(b.shape==kb.shape)\n\t\tcdict[kname + 'bias:0'] = np.ascontiguousarray(b)\n\n#--------------\ndef convert_batchnorm1d(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias')\n\tpmean = pdict.get(pname +'running_mean')\n\tpvar = pdict.get(pname +'running_var')\n\n\tkw = kdict.get(kname +'gamma:0')\n\tkb = kdict.get(kname +'beta:0',None)\n\tkmean = kdict.get(kname +'moving_mean:0')\n\tkvar = kdict.get(kname +'moving_variance:0')\n\n\tw,b,mean,var = pw,pb,pmean,pvar\n\tassert(w.shape==kw.shape)\n\tassert(b.shape==kb.shape)\n\tassert(mean.shape==kmean.shape)\n\tassert(var.shape==kvar.shape)\n\tcdict[kname +'gamma:0'] = np.ascontiguousarray(w)\n\tcdict[kname +'beta:0'] = np.ascontiguousarray(b)\n\tcdict[kname +'moving_mean:0'] = np.ascontiguousarray(mean)\n\tcdict[kname +'moving_variance:0'] = np.ascontiguousarray(var)\n\n#--------------\ndef convert_layernorm(\n\tcdict, kdict, pdict, pname, kname,\n):\n\tpw = pdict.get(pname +'weight')\n\tpb = pdict.get(pname +'bias')\n\n\tkw = kdict.get(kname +'gamma:0')\n\tkb = kdict.get(kname +'beta:0',None)\n\n\tw,b = pw,pb\n\tassert(w.shape==kw.shape)\n\tassert(b.shape==kb.shape)\n\tcdict[kname +'gamma:0'] = np.ascontiguousarray(w)\n\tcdict[kname +'beta:0'] = np.ascontiguousarray(b)\n\n\n```",
    "2407338": "Very very useful 👍👍👍\nFor the past 3 months, I've been always using TensorFlow and sometimes feel so miserable. Really really want to use Pytorch next time 🥶",
    "2407335": "##step.two. dumpy pytorch state dictionary \n\n\n\n```\n#equivalent pytorch code for the above example:\nclass FeedForward(nn.Module):\n\tdef __init__(self, x_dim, hidden_dim):\n\t\tsuper().__init__()\n\t\tself.mlp = nn.Sequential(\n\t\t\tnn.Linear(x_dim, hidden_dim),\n\t\t\tnn.ReLU(inplace=True),\n\t\t\tnn.Linear(hidden_dim, x_dim),\n\t\t)\n\tdef forward(self, x):\n\t\treturn self.mlp(x)\n```\n\nit is easier to dump pytorch dict into numpy as well. there are some bugs to include both\"import torch\" and \"import tensorflow\" in the same python script file.\n\nso we first dump everything as numpy and work work numpy instead.\n\n```\n\n\n\tdef run_dump_weight(dir='dump', out_name='pdict2'):\n\t\tcheckpoint_file = \\\n\t\t\t'/home/titanx/hengck/share1/kaggle/2022/google-finger-spell/result/MULTI-HEAD/trained/fold-2/00040139.pth'\n\n\n\t\tf = torch.load(checkpoint_file, map_location=lambda storage, loc: storage)\n\t\tstate_dict = f['state_dict']\n\n\t\tprint('state_dict)', len(state_dict))  # 501\n\t\tpdict = {}\n\t\tfor n, w in state_dict.items():\n\t\t\tif 'num_batches_tracked' in n: continue\n\t\t\tw = w.data.numpy()\n\t\t\tprint(n, w.shape)\n\t\t\tpdict[n] = w\n\t\tprint(len(pdict))\n\t\twrite_pickle_to_file(f'{dir}/{out_name}.pickle', pdict)\n\t\n```\n\n"
  }
}