Since we were restricted to use only TFLite ops, it was tricky to use beam search. Below is my implementation:
\nvocab_size = text_featurizer.num_classes\ntop_k = 5\nmax_time=256\n\nvocab_range_tf = tf.cast), dtype=tf.int32)\nvocab_range_tf = tf.reshape\n\n\nfirst_top_paths = tf.zeros\n\nfirst_top_probs = tf.fill)\n\nfirst_hash = tf.ones\nfirst_prefix_len = tf.zeros\nfirst_non_zero_last_char = tf.zeros\n\n\npows = []\np = 67 \np_pow = 1\nmodulo = 2144050663 \nmodulo_tf = tf.cast\n\nfor i in range:\n pows.append\n p_pow = int % modulo)\n\n\npows_tf = tf.cast\n\n\n@tf.function\ndef beam_search_tf:\n with tf.name_scope:\n total = encoded_length\n encoded = tf.math.log\n\n def condition:\n return tf.less\n\n def body:\n\n previous_probs = tf.reshape, [top_k, 1])\n\n \n context = tf.gather_nd)\n context = tf.cast\n\n \n \n current_probs = previous_probs + context\n\n \n \n p_pow_tf = tf.gather_nd)\n p_pow_tf = tf.expand_dims\n\n cur_char_hash = tf.math.mod\n cur_hash = tf.math.mod + cur_char_hash, modulo_tf)\n\n \n prefix_lens = tf.ones + tf.reshape\n\n last_chars = tf.gather_nd, tf.fill], axis=-1))\n last_chars = tf.expand_dims\n\n mesh = tf.expand_dims, axis=0)\n repeating_indices = tf.reshape, [top_k, 2])\n\n lm_rescore:\n lm_probs = tf.gather_nd)\n \n lm_probs = tf.tensor_scatter_nd_update\n )\n\n \n current_probs = current_probs + lm_probs \n\n \n\n new_hash = tf.tensor_scatter_nd_update\n\n new_prefix_len = tf.tensor_scatter_nd_update\n\n \n unique_hash, unique_hash_id = tf.unique)\n\n \n \n merged_probs = tf.tensor_scatter_nd_add,\n indices=tf.reshape,\n updates=tf.reshape, [top_k * vocab_size])\n )\n\n \n best_prob_ids = tf.math.top_k\n next_probs = tf.gather_nd)\n\n \n \n \n\n \n merged_sparse_mapping = tf.transpose], axis=0))\n\n merged_dense_mapping = tf.tensor_scatter_nd_update,\n indices=merged_sparse_mapping,\n updates=tf.ones\n )\n\n merged_ids = tf.math.argmax\n selected_ids = tf.gather_nd)\n\n unravel_index_0 = tf.math.floordiv\n unravel_index_1 = tf.math.mod\n selected_unravel = tf.stack\n\n selected_path_ids = tf.expand_dims\n selected_tokens = unravel_index_1\n selected_paths = tf.gather_nd\n\n \n next_paths = tf.tensor_scatter_nd_update, tf.fill], axis=-1),\n updates=selected_tokens,\n name='scatter_nd_update_4'\n )\n selected_unravel_t = tf.transpose\n\n next_hash = tf.gather_nd\n next_prefix_len = tf.gather_nd\n\n lm_rescore:\n selected_non_zero_ids = tf.boolean_mask, selected_tokens != 0)\n\n \n previous_non_zero_last_char = tf.gather_nd\n\n \n next_non_zero_last_char = tf.tensor_scatter_nd_update,\n updates=tf.gather_nd)\n )\n else:\n next_non_zero_last_char = previous_non_zero_last_char\n\n return time + 1, total, next_paths, next_probs, next_hash, next_prefix_len, next_non_zero_last_char\n\n _, _, top_paths, top_probs, _, _, _ = tf.while_loop,\n total.get_shape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n ]\n )\n\n return top_paths, top_probs\nAlso, I used the code for 'training' char bigram:
\nvocab_size = text_featurizer.num_classes\n\nphrases = train_df[].values\nphrases_encoded = [[character_map[c] + 1 c phrase] phrase phrases]\n\nunigram_counts = np.zeros([vocab_size], =np.int32)\nbigram_counts = np.zeros([vocab_size, vocab_size], =np.int32)\n\n\n phrase phrases_encoded:\n i range(len(phrase)):\n i == 0:\n c = 0\n :\n c = phrase[i - 1]\n\n unigram_counts[c] += 1\n\n d = phrase[i]\n bigram_counts[c][d] += 1\n\n\n\nzero_map = defaultdict(set)\n\n i range(vocab_size):\n j range(vocab_size):\n bigram_counts[i][j] == 0:\n zero_map[i].(j)\n\nw_sum_for_zero_v = {}\n\n v, zero_ws zero_map.items():\n w_sum_for_zero_v[v] = 0\n w zero_ws:\n w_sum_for_zero_v[v] += unigram_counts[w]\n\n\n\nbigram_singlton_num = (bigram_counts == 1).sum()\nchar_num = bigram_counts.sum()\nlambda_discount = bigram_singlton_num / char_num\n\n\n(f)\n(f)\n(f)\n\nbigram_lm = np.zeros([vocab_size, vocab_size], =np.float64)\n\n i tqdm(range(vocab_size), =vocab_size):\n j range(1, vocab_size):\n bigram_counts[i][j] == 0:\n bigram_lm[i][j] = lambda_discount * (unigram_counts[j] / w_sum_for_zero_v[i])\n w_sum_for_zero_v[i] == 0:\n (, i, j)\n :\n bigram_lm[i][j] = (1 - lambda_discount) * (bigram_counts[i][j] / unigram_counts[i])\n unigram_counts[i] == 0:\n (, i, j)\n\n\nbigram_lm_tf = tf.cast(bigram_lm, =tf.float32)\nbigram_lm_tf = tf.math.log(bigram_lm_tf + 1e-12) * tf.cast(0.01, =tf.float32)\nUnfortunately, char bigram lm didn't give me any score boost. Maybe, using higher orders (3grams, 4grams etc.), would give better score.
\nAlso, beam search gave + 0.005 both on private and public LB for all of my models. Also, validation showed, that increasing top_k gives even better results, but such models didn't pass the time limits.
\nYou may also notice a limitation of the beam search implementation, that beam_size equals vocab_size, which makes it much easier to do some calculations like repeating_indices
Since we were restricted to use only TFLite ops, it was tricky to use beam search. Below is my implementation:
\nvocab_size = text_featurizer.num_classes\ntop_k = 5\nmax_time=256\n\nvocab_range_tf = tf.cast), dtype=tf.int32)\nvocab_range_tf = tf.reshape\n\n\nfirst_top_paths = tf.zeros\n\nfirst_top_probs = tf.fill)\n\nfirst_hash = tf.ones\nfirst_prefix_len = tf.zeros\nfirst_non_zero_last_char = tf.zeros\n\n\npows = []\np = 67 \np_pow = 1\nmodulo = 2144050663 \nmodulo_tf = tf.cast\n\nfor i in range:\n pows.append\n p_pow = int % modulo)\n\n\npows_tf = tf.cast\n\n\n@tf.function\ndef beam_search_tf:\n with tf.name_scope:\n total = encoded_length\n encoded = tf.math.log\n\n def condition:\n return tf.less\n\n def body:\n\n previous_probs = tf.reshape, [top_k, 1])\n\n \n context = tf.gather_nd)\n context = tf.cast\n\n \n \n current_probs = previous_probs + context\n\n \n \n p_pow_tf = tf.gather_nd)\n p_pow_tf = tf.expand_dims\n\n cur_char_hash = tf.math.mod\n cur_hash = tf.math.mod + cur_char_hash, modulo_tf)\n\n \n prefix_lens = tf.ones + tf.reshape\n\n last_chars = tf.gather_nd, tf.fill], axis=-1))\n last_chars = tf.expand_dims\n\n mesh = tf.expand_dims, axis=0)\n repeating_indices = tf.reshape, [top_k, 2])\n\n lm_rescore:\n lm_probs = tf.gather_nd)\n \n lm_probs = tf.tensor_scatter_nd_update\n )\n\n \n current_probs = current_probs + lm_probs \n\n \n\n new_hash = tf.tensor_scatter_nd_update\n\n new_prefix_len = tf.tensor_scatter_nd_update\n\n \n unique_hash, unique_hash_id = tf.unique)\n\n \n \n merged_probs = tf.tensor_scatter_nd_add,\n indices=tf.reshape,\n updates=tf.reshape, [top_k * vocab_size])\n )\n\n \n best_prob_ids = tf.math.top_k\n next_probs = tf.gather_nd)\n\n \n \n \n\n \n merged_sparse_mapping = tf.transpose], axis=0))\n\n merged_dense_mapping = tf.tensor_scatter_nd_update,\n indices=merged_sparse_mapping,\n updates=tf.ones\n )\n\n merged_ids = tf.math.argmax\n selected_ids = tf.gather_nd)\n\n unravel_index_0 = tf.math.floordiv\n unravel_index_1 = tf.math.mod\n selected_unravel = tf.stack\n\n selected_path_ids = tf.expand_dims\n selected_tokens = unravel_index_1\n selected_paths = tf.gather_nd\n\n \n next_paths = tf.tensor_scatter_nd_update, tf.fill], axis=-1),\n updates=selected_tokens,\n name='scatter_nd_update_4'\n )\n selected_unravel_t = tf.transpose\n\n next_hash = tf.gather_nd\n next_prefix_len = tf.gather_nd\n\n lm_rescore:\n selected_non_zero_ids = tf.boolean_mask, selected_tokens != 0)\n\n \n previous_non_zero_last_char = tf.gather_nd\n\n \n next_non_zero_last_char = tf.tensor_scatter_nd_update,\n updates=tf.gather_nd)\n )\n else:\n next_non_zero_last_char = previous_non_zero_last_char\n\n return time + 1, total, next_paths, next_probs, next_hash, next_prefix_len, next_non_zero_last_char\n\n _, _, top_paths, top_probs, _, _, _ = tf.while_loop,\n total.get_shape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n tf.TensorShape,\n ]\n )\n\n return top_paths, top_probs\nAlso, I used the code for 'training' char bigram:
\nvocab_size = text_featurizer.num_classes\n\nphrases = train_df[].values\nphrases_encoded = [[character_map[c] + 1 c phrase] phrase phrases]\n\nunigram_counts = np.zeros([vocab_size], =np.int32)\nbigram_counts = np.zeros([vocab_size, vocab_size], =np.int32)\n\n\n phrase phrases_encoded:\n i range(len(phrase)):\n i == 0:\n c = 0\n :\n c = phrase[i - 1]\n\n unigram_counts[c] += 1\n\n d = phrase[i]\n bigram_counts[c][d] += 1\n\n\n\nzero_map = defaultdict(set)\n\n i range(vocab_size):\n j range(vocab_size):\n bigram_counts[i][j] == 0:\n zero_map[i].(j)\n\nw_sum_for_zero_v = {}\n\n v, zero_ws zero_map.items():\n w_sum_for_zero_v[v] = 0\n w zero_ws:\n w_sum_for_zero_v[v] += unigram_counts[w]\n\n\n\nbigram_singlton_num = (bigram_counts == 1).sum()\nchar_num = bigram_counts.sum()\nlambda_discount = bigram_singlton_num / char_num\n\n\n(f)\n(f)\n(f)\n\nbigram_lm = np.zeros([vocab_size, vocab_size], =np.float64)\n\n i tqdm(range(vocab_size), =vocab_size):\n j range(1, vocab_size):\n bigram_counts[i][j] == 0:\n bigram_lm[i][j] = lambda_discount * (unigram_counts[j] / w_sum_for_zero_v[i])\n w_sum_for_zero_v[i] == 0:\n (, i, j)\n :\n bigram_lm[i][j] = (1 - lambda_discount) * (bigram_counts[i][j] / unigram_counts[i])\n unigram_counts[i] == 0:\n (, i, j)\n\n\nbigram_lm_tf = tf.cast(bigram_lm, =tf.float32)\nbigram_lm_tf = tf.math.log(bigram_lm_tf + 1e-12) * tf.cast(0.01, =tf.float32)\nUnfortunately, char bigram lm didn't give me any score boost. Maybe, using higher orders (3grams, 4grams etc.), would give better score.
\nAlso, beam search gave + 0.005 both on private and public LB for all of my models. Also, validation showed, that increasing top_k gives even better results, but such models didn't pass the time limits.
\nYou may also notice a limitation of the beam search implementation, that beam_size equals vocab_size, which makes it much easier to do some calculations like repeating_indices
\n\nSince we were restricted to use only TFLite ops, it was tricky to use beam search
\n
restricted to only TFLite ops made a lot of things tricky!! Congratulations on your beam search implementation and great LB result!
\na few discussions around how to/issues on beam search so good to see your code. Thanks.
Congratulations. Thanks for sharing your code examples.
", "rawMarkdown": "Congratulations. Thanks for sharing your code examples. ", "votes": 1 } ], "comments": [ { "id": 2412222, "author_name": "something4kag", "author_url": "", "post_date": "2023-08-28T06:43:39.810000", "content": "\n\nSince we were restricted to use only TFLite ops, it was tricky to use beam search
\n
restricted to only TFLite ops made a lot of things tricky!! Congratulations on your beam search implementation and great LB result!
\na few discussions around how to/issues on beam search so good to see your code. Thanks.
Congratulations. Thanks for sharing your code examples.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2409483": "Since we were restricted to use only TFLite ops, it was tricky to use beam search. Below is my implementation:\n\n\tvocab_size = text_featurizer.num_classes\n\ttop_k = 5\n\tmax_time=256\n\t\n\tvocab_range_tf = tf.cast(list(range(1, vocab_size + 1)), dtype=tf.int32)\n\tvocab_range_tf = tf.reshape(vocab_range_tf, [1, vocab_size])\n\n\t# initialize with epmpty symbols\n\tfirst_top_paths = tf.zeros([top_k, max_time + 1], dtype=tf.int32)\n\t# uniform prob to the 1st states\n\tfirst_top_probs = tf.fill([top_k], value=tf.cast(1 / top_k, tf.float32))\n\n\tfirst_hash = tf.ones([top_k], dtype=tf.int32)\n\tfirst_prefix_len = tf.zeros([top_k], dtype=tf.int32)\n\tfirst_non_zero_last_char = tf.zeros([top_k], dtype=tf.int32)\n\n\t# precomute pows for string hashing\n\tpows = []\n\tp = 67 # first prime larger than vocab_size\n\tp_pow = 1\n\tmodulo = 2144050663 # prime\n\tmodulo_tf = tf.cast(modulo, dtype=tf.int32)\n\n\tfor i in range(max_time + 4):\n\t\tpows.append(p_pow)\n\t\tp_pow = int((p_pow * p) % modulo)\n\n\t# max_time pows for the hypothetical largest string\n\tpows_tf = tf.cast(pows[4:], dtype=tf.int32)\n\n\n\t@tf.function()\n\tdef beam_search_tf(encoded, encoded_length, lm_rescore=True):\n\t\twith tf.name_scope(f\"cnn_ctc_beam_search\"):\n\t\t\ttotal = encoded_length\n\t\t\tencoded = tf.math.log(encoded + 1e-8)\n\n\t\t\tdef condition(time, total, prev_top_paths, prev_top_probs, previous_hash, previous_prefix_lens, previous_non_zero_last_char):\n\t\t\t\treturn tf.less(time, total)\n\n\t\t\tdef body(time, total, prev_top_paths, prev_top_probs, previous_hash, previous_prefix_lens, previous_non_zero_last_char):\n\n\t\t\t\tprevious_probs = tf.reshape(tf.math.log(prev_top_probs + 1e-8), [top_k, 1])\n\n\t\t\t\t# use log add scheme\n\t\t\t\tcontext = tf.gather_nd(encoded, tf.reshape(time, [1, 1]))\n\t\t\t\tcontext = tf.cast(context, dtype=tf.float32)\n\n\t\t\t\t# current probs is of shape [top_k, vocab_size]\n\t\t\t\t# current_probs_ij means the probability ith path connect to jth context\n\t\t\t\tcurrent_probs = previous_probs + context\n\n\t\t\t\t# use string hashing to find equal prefixes for merging the paths \n\t\t\t\t# more details here https://cp-algorithms.com/string/string-hashing.html \n\t\t\t\tp_pow_tf = tf.gather_nd(pows_tf, tf.expand_dims(previous_prefix_lens, axis=-1))\n\t\t\t\tp_pow_tf = tf.expand_dims(p_pow_tf, axis=-1)\n\n\t\t\t\tcur_char_hash = tf.math.mod(vocab_range_tf * p_pow_tf, modulo_tf)\n\t\t\t\tcur_hash = tf.math.mod(tf.reshape(previous_hash, [top_k, 1]) + cur_char_hash, modulo_tf)\n\n\t\t\t\t# keep track of prefix lens (with merged repeating chars) i.e. prefix_lens for \"o o a a b\" is 3\n\t\t\t\tprefix_lens = tf.ones([1, vocab_size], dtype=tf.int32) + tf.reshape(previous_prefix_lens, [top_k, 1])\n\n\t\t\t\tlast_chars = tf.gather_nd(prev_top_paths, tf.stack([tf.range(top_k), tf.fill([top_k], time)], axis=-1))\n\t\t\t\tlast_chars = tf.expand_dims(last_chars, axis=0)\n\n\t\t\t\tmesh = tf.expand_dims(tf.range(top_k), axis=0)\n\t\t\t\trepeating_indices = tf.reshape(tf.stack([mesh, last_chars], axis=2), [top_k, 2])\n\n\t\t\t\tif lm_rescore:\n\t\t\t\t\tlm_probs = tf.gather_nd(bigram_lm_tf, tf.expand_dims(previous_non_zero_last_char, axis=-1))\n\t\t\t\t\t## make lm prob zero @ repeating_indices i.e. where char repeats no lm contribution\n\t\t\t\t\tlm_probs = tf.tensor_scatter_nd_update(\n\t\t\t\t\t\tlm_probs,\n\t\t\t\t\t\trepeating_indices,\n\t\t\t\t\t\ttf.zeros([top_k], dtype=tf.float32)\n\t\t\t\t\t)\n\n\t\t\t\t\t# lm_probs are log probs \n\t\t\t\t\tcurrent_probs = current_probs + lm_probs # lm factor alreay applied\n\n\t\t\t\t# hashes and lens are old @ repeating_indices\n\n\t\t\t\tnew_hash = tf.tensor_scatter_nd_update(\n\t\t\t\t\ttensor=cur_hash,\n\t\t\t\t\tindices=repeating_indices,\n\t\t\t\t\tupdates=previous_hash\n\t\t\t\t)\n\n\t\t\t\tnew_prefix_len = tf.tensor_scatter_nd_update(\n\t\t\t\t\ttensor=prefix_lens,\n\t\t\t\t\tindices=repeating_indices,\n\t\t\t\t\tupdates=previous_prefix_lens\n\t\t\t\t)\n\n\t\t\t\t# find unique hashes \n\t\t\t\tunique_hash, unique_hash_id = tf.unique(tf.reshape(new_hash, [top_k * vocab_size]))\n\n\t\t\t\t# merge probs for paths with the same unique_hash_id\n\t\t\t\t# tf.math.exp to sum pure probs not logprobs \n\t\t\t\tmerged_probs = tf.tensor_scatter_nd_add(\n\t\t\t\t\ttensor=tf.zeros([top_k * vocab_size], dtype=tf.float32),\n\t\t\t\t\tindices=tf.reshape(unique_hash_id, [top_k * vocab_size, 1]),\n\t\t\t\t\tupdates=tf.reshape(tf.math.exp(current_probs), [top_k * vocab_size])\n\t\t\t\t)\n\n\t\t\t\t# select top_k\n\t\t\t\tbest_prob_ids = tf.math.top_k(merged_probs, k=top_k).indices\n\t\t\t\tnext_probs = tf.gather_nd(merged_probs, tf.expand_dims(best_prob_ids, axis=-1))\n\n\t\t\t\t# we need to pick any id with the same unique_hash_id\n\t\t\t\t# it's pretty easy by applying tensor_scatter_nd_update\n\t\t\t\t# in tflite tensor_scatter_nd_update works as tensor_scatter_nd_add\n\t\t\t\t\n\t\t\t\t# hense use a workaround to achieve the same\n\t\t\t\tmerged_sparse_mapping = tf.transpose(tf.stack([unique_hash_id, tf.range(0, top_k * vocab_size)], axis=0))\n\n\t\t\t\tmerged_dense_mapping = tf.tensor_scatter_nd_update(\n\t\t\t\t\ttensor=tf.zeros([top_k * vocab_size, top_k * vocab_size], dtype=tf.int32),\n\t\t\t\t\tindices=merged_sparse_mapping,\n\t\t\t\t\tupdates=tf.ones([top_k * vocab_size], dtype=tf.int32)\n\t\t\t\t)\n\n\t\t\t\tmerged_ids = tf.math.argmax(merged_dense_mapping, axis=1, output_type=tf.int32)\n\t\t\t\tselected_ids = tf.gather_nd(merged_ids, tf.expand_dims(best_prob_ids, axis=-1))\n\n\t\t\t\tunravel_index_0 = tf.math.floordiv(selected_ids, vocab_size)\n\t\t\t\tunravel_index_1 = tf.math.mod(selected_ids, vocab_size)\n\t\t\t\tselected_unravel = tf.stack([unravel_index_0, unravel_index_1], axis=0)\n\n\t\t\t\tselected_path_ids = tf.expand_dims(unravel_index_0, axis=-1)\n\t\t\t\tselected_tokens = unravel_index_1\n\t\t\t\tselected_paths = tf.gather_nd(prev_top_paths, selected_path_ids)\n\n\t\t\t\t# fill in the paths tensor with tokens @ time step 'time + 1'\n\t\t\t\tnext_paths = tf.tensor_scatter_nd_update(\n\t\t\t\t\ttensor=selected_paths,\n\t\t\t\t\tindices=tf.stack([tf.range(top_k), tf.fill([top_k], time + 1)], axis=-1),\n\t\t\t\t\tupdates=selected_tokens,\n\t\t\t\t\tname='scatter_nd_update_4'\n\t\t\t\t)\n\t\t\t\tselected_unravel_t = tf.transpose(selected_unravel)\n\n\t\t\t\tnext_hash = tf.gather_nd(new_hash, selected_unravel_t)\n\t\t\t\tnext_prefix_len = tf.gather_nd(new_prefix_len, selected_unravel_t)\n\n\t\t\t\tif lm_rescore:\n\t\t\t\t\tselected_non_zero_ids = tf.boolean_mask(tf.range(top_k), selected_tokens != 0)\n\n\t\t\t\t\t# track last non zero char and deal with reshuffled orders\n\t\t\t\t\tprevious_non_zero_last_char = tf.gather_nd(previous_non_zero_last_char, selected_path_ids)\n\n\t\t\t\t\t# update with non zero tokens\n\t\t\t\t\tnext_non_zero_last_char = tf.tensor_scatter_nd_update(\n\t\t\t\t\t\ttensor=previous_non_zero_last_char,\n\t\t\t\t\t\tindices=tf.expand_dims(selected_non_zero_ids, axis=-1),\n\t\t\t\t\t\tupdates=tf.gather_nd(selected_tokens, tf.expand_dims(selected_non_zero_ids, axis=-1))\n\t\t\t\t\t)\n\t\t\t\telse:\n\t\t\t\t\tnext_non_zero_last_char = previous_non_zero_last_char\n\n\t\t\t\treturn time + 1, total, next_paths, next_probs, next_hash, next_prefix_len, next_non_zero_last_char\n\n\t\t\t_, _, top_paths, top_probs, _, _, _ = tf.while_loop(\n\t\t\t\tcondition,\n\t\t\t\tbody,\n\t\t\t\tloop_vars=[0, total, first_top_paths, first_top_probs, first_hash, first_prefix_len, first_non_zero_last_char],\n\t\t\t\tshape_invariants=[\n\t\t\t\t\ttotal.get_shape(),\n\t\t\t\t\ttotal.get_shape(),\n\t\t\t\t\ttf.TensorShape([top_k, max_time + 1]),\n\t\t\t\t\ttf.TensorShape([top_k]),\n\t\t\t\t\ttf.TensorShape([top_k]),\n\t\t\t\t\ttf.TensorShape([top_k]),\n\t\t\t\t\ttf.TensorShape([top_k]),\n\t\t\t\t]\n\t\t\t)\n\n\t\t\treturn top_paths, top_probs\n\n\nAlso, I used the code for 'training' char bigram:\n\n\tvocab_size = text_featurizer.num_classes\n\t## count\n\tphrases = train_df['phrase'].values\n\tphrases_encoded = [[character_map[c] + 1 for c in phrase] for phrase in phrases]\n\n\tunigram_counts = np.zeros([vocab_size], dtype=np.int32)\n\tbigram_counts = np.zeros([vocab_size, vocab_size], dtype=np.int32)\n\n\n\tfor phrase in phrases_encoded:\n\t\tfor i in range(len(phrase)):\n\t\t\tif i == 0:\n\t\t\t\tc = 0\n\t\t\telse:\n\t\t\t\tc = phrase[i - 1]\n\t\t\t\n\t\t\tunigram_counts[c] += 1\n\n\t\t\td = phrase[i]\n\t\t\tbigram_counts[c][d] += 1\n\n\t# aux counts for linear discount\n\n\tzero_map = defaultdict(set)\n\n\tfor i in range(vocab_size):\n\t\tfor j in range(vocab_size):\n\t\t\tif bigram_counts[i][j] == 0:\n\t\t\t\tzero_map[i].add(j)\n\n\tw_sum_for_zero_v = {}\n\n\tfor v, zero_ws in zero_map.items():\n\t\tw_sum_for_zero_v[v] = 0\n\t\tfor w in zero_ws:\n\t\t\tw_sum_for_zero_v[v] += unigram_counts[w]\n\n\t# linear discounting\n\n\tbigram_singlton_num = (bigram_counts == 1).sum()\n\tchar_num = bigram_counts.sum()\n\tlambda_discount = bigram_singlton_num / char_num\n\n\n\tprint(f'{bigram_singlton_num=}')\n\tprint(f'{char_num=}')\n\tprint(f'{lambda_discount=}')\n\n\tbigram_lm = np.zeros([vocab_size, vocab_size], dtype=np.float64)\n\n\tfor i in tqdm(range(vocab_size), total=vocab_size):\n\t\tfor j in range(1, vocab_size):\n\t\t\tif bigram_counts[i][j] == 0:\n\t\t\t\tbigram_lm[i][j] = lambda_discount * (unigram_counts[j] / w_sum_for_zero_v[i])\n\t\t\t\tif w_sum_for_zero_v[i] == 0:\n\t\t\t\t\tprint('zero', i, j)\n\t\t\telse:\n\t\t\t\tbigram_lm[i][j] = (1 - lambda_discount) * (bigram_counts[i][j] / unigram_counts[i])\n\t\t\t\tif unigram_counts[i] == 0:\n\t\t\t\t\tprint('zero', i, j)\n\n\t# log probs with discount\n\tbigram_lm_tf = tf.cast(bigram_lm, dtype=tf.float32)\n\tbigram_lm_tf = tf.math.log(bigram_lm_tf + 1e-12) * tf.cast(0.01, dtype=tf.float32)\n\n\nUnfortunately, char bigram lm didn't give me any score boost. Maybe, using higher orders (3grams, 4grams etc.), would give better score.\nAlso, beam search gave + 0.005 both on private and public LB for all of my models. Also, validation showed, that increasing top_k gives even better results, but such models didn't pass the time limits. \nYou may also notice a limitation of the beam search implementation, that beam_size equals vocab_size, which makes it much easier to do some calculations like `repeating_indices`\n \n", "2412222": ">Since we were restricted to use only TFLite ops, it was tricky to use beam search\n\nrestricted to only TFLite ops made a lot of things tricky!! Congratulations on your beam search implementation and great LB result!\na few discussions around how to/issues on beam search so good to see your code. Thanks. ", "2412155": "Congratulations. Thanks for sharing your code examples. " } }