My notebook saves out a submission.zip file with \"model.tflite\" and \"inference_args.json\". The tflite model can score examples from the parquet files with no problem (see code below). I'm completely stumped on what the problem is - as far as I can tell, this matches with the submission interface perfectly.
\nWhen I run unzip the submission file (to the folder 'submission/'), then run the following code in a notebook, it works fine:
\ndef load:\n return pd.read\n\nsample_file = 'data/train_landmarks/parquet'\n\n ('submission/inference_args.json', ) json_file:\n selected_cols = load.load(json_file)\n\nmeta_df = pd.read\n\n# Grab a random sequence out the metadata file\n\nframes = load\nseq_id = meta_df.sample().sequence_id.iloc\nframes = frames.loc\n\n# Load model into the interpreter\n\ninterpreter = tf.lite.\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n\n (f'data/character_to_prediction_index.json', ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items}\n\nfound_signatures = (interpreter.get.keys)\n\n REQUIRED_SIGNATURE not found_signatures:\n raise \n\nprediction_fn = interpreter.get\noutput = prediction\n\nprint(output.shape)\n\nprint(np.argmax(output, axis=))\n\nprediction_str = .join(, axis=)])\n\n# Output from running the above\n\n>>> (, )\n>>> \n>>> -\nAs far as I can tell, the input is supposed to be rank 2 (sequence length x number of selected columns), and the output it supposed to be rank 2 (predictions length x size of vocabulary).
\nWhat is the problem with this??
", "messages": [ { "id": 2359692, "postDate": "2023-07-26T10:27:52.397Z", "content": "You need to test your model with some corner cases, like the frames in the shape of [0, num_freatures], [1, num_features], all zeros in the frames and all nans in the frames.
\nIn the test data, there are empty frames in the shape of [0, num_freatures].
", "rawMarkdown": "You need to test your model with some corner cases, like the frames in the shape of [0, num_freatures], [1, num_features], all zeros in the frames and all nans in the frames.\n\nIn the test data, there are empty frames in the shape of [0, num_freatures].", "votes": 1, "replies": [ { "id": 2366799, "postDate": "2023-07-31T07:33:05.503Z", "content": "My problem turned out to be that I wasn't using the standalone tflite interpreter to test, but rather the one that comes packaged with Tensorflow… thus, there were functions I was using which aren't available in the base runtime.
\nHowever I'm still having weird issues - my model works perfectly with WonderingAlice's inference notebook, but for whatever reason, my scores are stuck very close to zero (like 0.065). Even stranger still, the better models I've submitted (ones trained for dozens of epochs with much lower validation loss) are performing even worse than the model I first got to submit without errors, which was only trained for 3 epochs and had terrible validation loss.
", "rawMarkdown": "My problem turned out to be that I wasn't using the standalone tflite interpreter to test, but rather the one that comes packaged with Tensorflow... thus, there were functions I was using which aren't available in the base runtime. \n\nHowever I'm still having weird issues - my model works perfectly with WonderingAlice's inference notebook, but for whatever reason, my scores are stuck very close to zero (like 0.065). Even stranger still, the better models I've submitted (ones trained for dozens of epochs with much lower validation loss) are performing even worse than the model I first got to submit without errors, which was only trained for 3 epochs and had terrible validation loss.", "replies": [ { "id": 2368887, "postDate": "2023-08-01T12:40:41.340Z", "content": "You can test your tflite model locally before submission. The reason might be that your preprocess layer differs from what you use for validation.
", "rawMarkdown": "You can test your tflite model locally before submission. The reason might be that your preprocess layer differs from what you use for validation.", "votes": 1 } ] } ] }, { "id": 2358590, "postDate": "2023-07-25T15:53:51.007Z", "content": "Early on in this competition, everyone had a lot of problems with submission. Check out the discussion threads and kernels with successful submissions. An important edge case to deal with is empty input.
", "rawMarkdown": "Early on in this competition, everyone had a lot of problems with submission. Check out the discussion threads and kernels with successful submissions. An important edge case to deal with is empty input.", "votes": 2 }, { "id": 2358539, "postDate": "2023-07-25T15:22:12.867Z", "content": "My notebook saves out a submission.zip file with \"model.tflite\" and \"inference_args.json\". The tflite model can score examples from the parquet files with no problem (see code below). I'm completely stumped on what the problem is - as far as I can tell, this matches with the submission interface perfectly.
\nWhen I run unzip the submission file (to the folder 'submission/'), then run the following code in a notebook, it works fine:
\ndef load:\n return pd.read\n\nsample_file = 'data/train_landmarks/parquet'\n\n ('submission/inference_args.json', ) json_file:\n selected_cols = load.load(json_file)\n\nmeta_df = pd.read\n\n# Grab a random sequence out the metadata file\n\nframes = load\nseq_id = meta_df.sample().sequence_id.iloc\nframes = frames.loc\n\n# Load model into the interpreter\n\ninterpreter = tf.lite.\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n\n (f'data/character_to_prediction_index.json', ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items}\n\nfound_signatures = (interpreter.get.keys)\n\n REQUIRED_SIGNATURE not found_signatures:\n raise \n\nprediction_fn = interpreter.get\noutput = prediction\n\nprint(output.shape)\n\nprint(np.argmax(output, axis=))\n\nprediction_str = .join(, axis=)])\n\n# Output from running the above\n\n>>> (, )\n>>> \n>>> -\nAs far as I can tell, the input is supposed to be rank 2 (sequence length x number of selected columns), and the output it supposed to be rank 2 (predictions length x size of vocabulary).
\nWhat is the problem with this??
", "rawMarkdown": "My notebook saves out a submission.zip file with \"model.tflite\" and \"inference_args.json\". The tflite model can score examples from the parquet files with no problem (see code below). I'm completely stumped on what the problem is - as far as I can tell, this matches with the submission interface perfectly.\n\nWhen I run unzip the submission file (to the folder 'submission/'), then run the following code in a notebook, it works fine:\n\n```\ndef load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=selected_cols)\n\nsample_file = 'data/train_landmarks/152029243.parquet'\n\nwith open('submission/inference_args.json', \"r\") as json_file:\n selected_cols = load.load(json_file)['selected_columns']\n\nmeta_df = pd.read_csv('data/train.csv')\n\n# Grab a random sequence out of the metadata file\n\nframes = load_relevant_data_subset(sample_file)\nseq_id = meta_df[meta_df.file_id == 152029243].sample(1).sequence_id.iloc[0]\nframes = frames.loc[seq_id]\n\n# Load model into the interpreter\n\ninterpreter = tf.lite.Interpreter('submission/model.tflite')\n\nREQUIRED_SIGNATURE = \"serving_default\"\nREQUIRED_OUTPUT = \"outputs\"\n\nwith open (f'data/character_to_prediction_index.json', \"r\") as f:\n character_map = json.load(f)\nrev_character_map = {j:i for i,j in character_map.items()}\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\n\nprint(output[REQUIRED_OUTPUT].shape)\n\nprint(np.argmax(output[REQUIRED_OUTPUT], axis=1))\n\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])\n\n# Output from running the above\n\n>>> (7, 59)\n>>> [21 18 24 15 12 18 16]\n>>> 6390-31\n```\n\nAs far as I can tell, the input is supposed to be rank 2 (sequence length x number of selected columns), and the output it supposed to be rank 2 (predictions length x size of vocabulary). \n\nWhat is the problem with this??" } ], "comments": [ { "id": 2359692, "author_name": "bliao", "author_url": "", "post_date": "2023-07-26T10:27:52.397000", "content": "You need to test your model with some corner cases, like the frames in the shape of [0, num_freatures], [1, num_features], all zeros in the frames and all nans in the frames.
\nIn the test data, there are empty frames in the shape of [0, num_freatures].
", "votes": 1, "replies": [ { "id": 2366799, "author_name": "Tarek Sami", "author_url": "", "post_date": "2023-07-31T07:33:05.503000", "content": "My problem turned out to be that I wasn't using the standalone tflite interpreter to test, but rather the one that comes packaged with Tensorflow… thus, there were functions I was using which aren't available in the base runtime.
\nHowever I'm still having weird issues - my model works perfectly with WonderingAlice's inference notebook, but for whatever reason, my scores are stuck very close to zero (like 0.065). Even stranger still, the better models I've submitted (ones trained for dozens of epochs with much lower validation loss) are performing even worse than the model I first got to submit without errors, which was only trained for 3 epochs and had terrible validation loss.
", "votes": 0, "replies": [ { "id": 2368887, "author_name": "bliao", "author_url": "", "post_date": "2023-08-01T12:40:41.340000", "content": "You can test your tflite model locally before submission. The reason might be that your preprocess layer differs from what you use for validation.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2358590, "author_name": "WalkingMoose", "author_url": "", "post_date": "2023-07-25T15:53:51.007000", "content": "Early on in this competition, everyone had a lot of problems with submission. Check out the discussion threads and kernels with successful submissions. An important edge case to deal with is empty input.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2359692": "You need to test your model with some corner cases, like the frames in the shape of [0, num_freatures], [1, num_features], all zeros in the frames and all nans in the frames.\n\nIn the test data, there are empty frames in the shape of [0, num_freatures].", "2358590": "Early on in this competition, everyone had a lot of problems with submission. Check out the discussion threads and kernels with successful submissions. An important edge case to deal with is empty input.", "2358539": "My notebook saves out a submission.zip file with \"model.tflite\" and \"inference_args.json\". The tflite model can score examples from the parquet files with no problem (see code below). I'm completely stumped on what the problem is - as far as I can tell, this matches with the submission interface perfectly.\n\nWhen I run unzip the submission file (to the folder 'submission/'), then run the following code in a notebook, it works fine:\n\n```\ndef load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=selected_cols)\n\nsample_file = 'data/train_landmarks/152029243.parquet'\n\nwith open('submission/inference_args.json', \"r\") as json_file:\n selected_cols = load.load(json_file)['selected_columns']\n\nmeta_df = pd.read_csv('data/train.csv')\n\n# Grab a random sequence out of the metadata file\n\nframes = load_relevant_data_subset(sample_file)\nseq_id = meta_df[meta_df.file_id == 152029243].sample(1).sequence_id.iloc[0]\nframes = frames.loc[seq_id]\n\n# Load model into the interpreter\n\ninterpreter = tf.lite.Interpreter('submission/model.tflite')\n\nREQUIRED_SIGNATURE = \"serving_default\"\nREQUIRED_OUTPUT = \"outputs\"\n\nwith open (f'data/character_to_prediction_index.json', \"r\") as f:\n character_map = json.load(f)\nrev_character_map = {j:i for i,j in character_map.items()}\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\n\nprint(output[REQUIRED_OUTPUT].shape)\n\nprint(np.argmax(output[REQUIRED_OUTPUT], axis=1))\n\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])\n\n# Output from running the above\n\n>>> (7, 59)\n>>> [21 18 24 15 12 18 16]\n>>> 6390-31\n```\n\nAs far as I can tell, the input is supposed to be rank 2 (sequence length x number of selected columns), and the output it supposed to be rank 2 (predictions length x size of vocabulary). \n\nWhat is the problem with this??" } }