Dear all,
\nWhen I click on the leaderboard, it is empty.
\nIs it just me or have no submissions been made yet?
\nI find it strange that there aren't even any \"hello world\" baselines yet. Is there something wrong with the data maybe?
Dear all,
\nWhen I click on the leaderboard, it is empty.
\nIs it just me or have no submissions been made yet?
\nI find it strange that there aren't even any \"hello world\" baselines yet. Is there something wrong with the data maybe?
Because the training set is too large, we still need some time for model training.
", "rawMarkdown": "Because the training set is too large, we still need some time for model training.", "votes": 3, "replies": [ { "id": 2260446, "postDate": "2023-05-15T16:57:35.657Z", "content": "It makes sense, but what about the fact, that in \"Google Research - Identify Contrails to Reduce Global Warming\" there are 60+ teams, while the dataset is more than twice bigger?
", "rawMarkdown": "It makes sense, but what about the fact, that in \"Google Research - Identify Contrails to Reduce Global Warming\" there are 60+ teams, while the dataset is more than twice bigger?", "votes": 1, "replies": [ { "id": 2260528, "postDate": "2023-05-15T17:45:39.560Z", "content": "It's trivial to make a dummy submission to the contrails competition (or almost any other competition for that matter). The bar for an initial submission is higher in this case since someone will need to prepare a working model first.
", "rawMarkdown": "It's trivial to make a dummy submission to the contrails competition (or almost any other competition for that matter). The bar for an initial submission is higher in this case since someone will need to prepare a working model first.", "votes": 1 }, { "id": 2265537, "postDate": "2023-05-19T10:15:43.510Z", "content": "@sohier it should be quite trivial to make an initial random submission here as well, but as you can read in some of the other discussion threads, all (very well informed) attempts to do so are failing thus far. Could you please clarify the required input and output formats:
\nThank you for raising this concern. We reviewed the metric and identified that a necessary dataset had failed to properly attach. Submissions should now be working.
", "rawMarkdown": "Thank you for raising this concern. We reviewed the metric and identified that a necessary dataset had failed to properly attach. Submissions should now be working.", "votes": 4, "replies": [ { "id": 2266361, "postDate": "2023-05-20T00:58:22.053Z", "content": "However it still doesn’t work on my side. I am wondering if anyone can make successful submission now.
", "rawMarkdown": "However it still doesn’t work on my side. I am wondering if anyone can make successful submission now.", "votes": 2 }, { "id": 2266414, "postDate": "2023-05-20T03:50:43.543Z", "content": "I still can't submit a valid submission. Is it possible to know what are the input shapes by defaults? n_framesx1630?
\nthe output shape should be n_charactersx59?
@sohier Thank you for looking into this. I still can't make any valid submissions. Could you please give the exact formats for the input data passed to the model and the output format expected? Or even better, provide a notebook with a working submission? That would really help us get started on this competition.
", "rawMarkdown": "@sohier Thank you for looking into this. I still can't make any valid submissions. Could you please give the exact formats for the input data passed to the model and the output format expected? Or even better, provide a notebook with a working submission? That would really help us get started on this competition.", "votes": 3 }, { "id": 2266698, "postDate": "2023-05-20T09:23:47.740Z", "content": "Sadly, my submissions keep failing as well. I agree with @mdecoster that providing an example submission notebook would be very helpful.
", "rawMarkdown": "Sadly, my submissions keep failing as well. I agree with @mdecoster that providing an example submission notebook would be very helpful.", "votes": 3 }, { "id": 2269074, "postDate": "2023-05-22T07:48:48.437Z", "content": "can you please fix the doc of the evaluation page and let us know the expected signature (especially please define REQUIRED_OUTPUT variable)
", "rawMarkdown": "can you please fix the doc of the evaluation page and let us know the expected signature (especially please define REQUIRED_OUTPUT variable)", "votes": 1 } ] }, { "id": 2267501, "postDate": "2023-05-21T03:03:49.247Z", "content": "Everyone got what should be in REQUIRED_SIGNATURE and REQUIRED_OUTPUT?
\nAs described in Evaluation part
My understanding is that REQUIRED_SIGNATURE = serving_default and REQUIRED_OUTPUT = outputs.
Yes, at least, that was the interpretation in the previous competition.
", "rawMarkdown": "Yes, at least, that was the interpretation in the previous competition." } ] }, { "id": 2267757, "postDate": "2023-05-21T07:59:46.820Z", "content": "i suppose that REQUIRED_SIGNATURE = serving_default, but REQUIRED_OUTPUT is number of elements in the true string, because our evaluation metric for this contest is the normalized total levenshtein distance and number of characters in the labels be N and the total levenshtein distance be D. The metric equals (N - D) / N. I believe that organizers want to have range of that metric like [0, 1].
\nFor instance if we have 2 strings real and estiamted: y_true = \"i can see the rings on saturn\", and y_pred = \"i can see the rings on saturn s\" N = len(y_true) , the Levinstein distance is calculated on (y_true, y_pred[:N]). But it is on only my guess i could be wrong. By the way i'm also interest what is REQUIRED_OUTPUT.
In our interpretation, REQUIRED_OUTPUT is \"outputs\", i.e. the name you need to give to the last layer in your model (i.e., the one that produces the outputs).
\nI have posted a demo notebook to show how TfLite inference can be done here
", "rawMarkdown": "In our interpretation, REQUIRED_OUTPUT is \"outputs\", i.e. the name you need to give to the last layer in your model (i.e., the one that produces the outputs).\n\nI have posted a demo notebook to show how TfLite inference can be done [here](https://www.kaggle.com/code/wonderingalice/dummy-submission-with-previous-kernel?kernelSessionId=130400757)", "votes": 1 } ] } ] }, { "id": 2264709, "postDate": "2023-05-18T16:55:14.017Z", "content": "That’s very tricky. I created a random model that takes left hand + right hand landmarks data as input for submission. It can make inferences with parquet files. However submission failed after running about one hour.
", "rawMarkdown": "That’s very tricky. I created a random model that takes left hand + right hand landmarks data as input for submission. It can make inferences with parquet files. However submission failed after running about one hour.", "votes": 1, "replies": [ { "id": 2264765, "postDate": "2023-05-18T17:23:39.807Z", "content": "Hi Lonnie,
\nThat's already better than my attempts: this suggests that scoring works for some but not all samples?
", "rawMarkdown": "Hi Lonnie,\n\nThat's already better than my attempts: this suggests that scoring works for some but not all samples?", "replies": [ { "id": 2265084, "postDate": "2023-05-19T01:11:11.760Z", "content": "I am not sure. Everything could be possible. It may be some kinds of pipeline issues or dataset issues. But in my notebook it takes 2 minutes to make inferences and evaluation with tf.edit_distance method on train files and supplemental files with low memory footprint.
Hi @lonnieqin ,
\nI tried to make a dummy solution, using all features, and based on the submission model format from the previous competition. I have tried both:
\ninputs = tf.keras.Input(shape=(543,3), name=\"inputs\")
\nand
\ninputs = tf.keras.Input(shape=(None, 543,3), name=\"inputs\")
\nAs input shapes (using all features for now) and I think I've tried all possible output shape tensor combinations, i.e.:
\nSince all these attempts (at a rate of 5 totally uninformative \"random guesses\" per day) failed after about one minute, this suggests they don't even start getting scored. I am assuming there is still something wrong with my interpretation of the input tensor shape and/or required output tensor shape or properties (e.g. data format)?
\nSo, since it seems that your attempts at least get partially scored: could you clarify what the input and output tensor shapes are in your model?
\nBTW: I do feel that it is the responsibility of competition organisers to clarify the desired interface of submitted models. It is also the responsibility of competition organisers of a code competition to make sure that models can be properly tested in Kaggle kernels that have the same settings as the actual test inference kernel. This is not the case since we can currently NOT run tflite interpreter inference checks in Kaggle kernels!!
", "rawMarkdown": "Hi @lonnieqin ,\n\nI tried to make a dummy solution, using all features, and based on the submission model format from the previous competition. I have tried both:\n\ninputs = tf.keras.Input(shape=(543,3), name=\"inputs\")\n\nand \n\ninputs = tf.keras.Input(shape=(None, 543,3), name=\"inputs\")\n\nAs input shapes (using all features for now) and I think I've tried all possible output shape tensor combinations, i.e.:\n- with and without hacking batch dimension as time dimension (as in the previous competition)\n- I also tried per-frame models (although that would be horribly restrictive)\n\nSince all these attempts (at a rate of 5 totally uninformative \"random guesses\" per day) failed after about one minute, this suggests they don't even start getting scored. I am assuming there is still something wrong with my interpretation of the input tensor shape and/or required output tensor shape or properties (e.g. data format)?\n\nSo, since it seems that your attempts at least get partially scored: could you clarify what the input and output tensor shapes are in your model?\n\n\n\n\n**BTW: I do feel that it is the responsibility of competition organisers to clarify the desired interface of submitted models. It is also the responsibility of competition organisers of a code competition to make sure that models can be properly tested in Kaggle kernels that have the same settings as the actual test inference kernel. This is not the case since we can currently NOT run tflite interpreter inference checks in Kaggle kernels!!**\n\n\n\n", "votes": 1, "replies": [ { "id": 2265539, "postDate": "2023-05-19T10:19:10.363Z", "content": "My Model has input shape (None, 9) and output shape (num_samples * 18, 59). num_samples is calculated by reading frame column. I choose num_samples * 18 because average phrase length of training file is about 18.
def get_inference_model():\n inputs = tf.keras.Input((9), dtype=tf.float32, name=\"inputs\")\n vector = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n frame_value = vector[:, 8]\n num_samples = tf.math.count_nonzero(frame_value == 1, dtype=tf.int32)\n samples = tf.random.normal(shape=(num_samples * 18, 42))\n outputs = tf.keras.layers.Dense(59, activation=\"softmax\", name=\"outputs\")(samples)\n inference_model = tf.keras.Model(inputs=inputs, outputs=outputs) \n inference_model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics=[\"accuracy\"])\n return inference_model\nDuring evaluation, each parquet file is read using code: pd.read_parquet(pq_path, columns=selected_columns) when I save json file named inference_args.json
{'selected_columns': ['x_left_hand_0', 'x_left_hand_1', 'y_left_hand_0', 'y_left_hand_1', 'x_right_hand_0', 'x_right_hand_1', 'y_right_hand_0', 'y_right_hand_1', 'frame']}\n
\nThis code can read parquet file with shape (n, 9). Training parquet files all have 1000 samples of phrases, when making inference with this model, it can get output tensor with shape (18000, 59).
I think we can use less than 1% of the data for training and validation, which will not cost too much, but still verify that the model meets the competition requirements.
", "rawMarkdown": " I think we can use less than 1% of the data for training and validation, which will not cost too much, but still verify that the model meets the competition requirements." }, { "id": 2265627, "postDate": "2023-05-19T11:45:09.903Z", "content": "Thanks @lonnieqin
\nI'd already used up my \"random guesses\" for today, but since your dummy model at least scores partially, you've probably got the dimensions right. What I'd missed, obviously, is that features are loaded as a flat array for each frame (instead of the #kpx3 format of the previous competition).
\nWhat may be happening in yoru model that there are additional constraints on output length, e.g. that it cannot be longer than the original number of frames in the video OR that your \"num_samples\" possibly returns zero at some point ?
", "rawMarkdown": "Thanks @lonnieqin \n\nI'd already used up my \"random guesses\" for today, but since your dummy model at least scores partially, you've probably got the dimensions right. What I'd missed, obviously, is that features are loaded as a flat array for each frame (instead of the #kpx3 format of the previous competition).\n\nWhat may be happening in yoru model that there are additional constraints on output length, e.g. that it cannot be longer than the original number of frames in the video OR that your \"num_samples\" possibly returns zero at some point ?", "votes": 1 }, { "id": 2265634, "postDate": "2023-05-19T11:50:38.100Z", "content": "Btw, my next attempt would be this one:
\nFEATURES = 543*3+1\nNUM_CHARACTERS = 59\n\ndef get_dummy_model():\n inputs = tf.keras.Input(shape=(FEATURES,), dtype=tf.float32, name=\"inputs\")\n print(inputs.shape)\n x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n print(x.shape)\n # just some dummy stuff here to create random output logits\n x = tf.math.reduce_mean(x, axis = 1, keepdims = True)\n print(x.shape)\n x = tf.keras.layers.Dense(NUM_CHARACTERS)(x)\n print(x.shape)\n out = tf.keras.layers.Activation(\"linear\", name=\"outputs\")(x)\n print(out.shape)\n inference_model = tf.keras.Model(inputs=inputs, outputs=out)\n inference_model.compile(loss=\"sparse_categorical_crossentropy\",metrics=\"accuracy\")\n return inference_model\n\ndummy_model_test = get_dummy_model()\ndummy_model_test.summary()\nI tried to restrict num_samples with min and max values, and tried things like num_samples = tf.math.floordiv(tf.shape(inputs)[0], 6), only to find it failed immediately, which may indicate the length of output tensor need to be divided by number of phrases. I can’t even add 1 to num_samples.
", "rawMarkdown": "I tried to restrict num_samples with min and max values, and tried things like num_samples = tf.math.floordiv(tf.shape(inputs)[0], 6), only to find it failed immediately, which may indicate the length of output tensor need to be divided by number of phrases. I can’t even add 1 to num_samples." }, { "id": 2265868, "postDate": "2023-05-19T15:09:30.060Z", "content": "Does this mean that the submitted model is supposed to predict outputs for multiple concatenated samples/phrases? I assumed that it would only receive one phrase at the time and that output length would simply have to be <= input length …
", "rawMarkdown": "Does this mean that the submitted model is supposed to predict outputs for multiple concatenated samples/phrases? I assumed that it would only receive one phrase at the time and that output length would simply have to be <= input length ..." }, { "id": 2266012, "postDate": "2023-05-19T16:30:12.717Z", "content": "I think in mobile devices there are just a few phrases to be predicted at a time, it’s able to handle video streams. However in kaggle competition our model have to make batch inferences since dataset size is so large. The output size can be set to a reasonable value as long as the model’s performance meets requirement.
", "rawMarkdown": "I think in mobile devices there are just a few phrases to be predicted at a time, it’s able to handle video streams. However in kaggle competition our model have to make batch inferences since dataset size is so large. The output size can be set to a reasonable value as long as the model’s performance meets requirement." } ] } ] }, { "id": 2260524, "postDate": "2023-05-15T17:43:29.287Z", "content": "I'm going to review our logs in detail later today to verify that there's no issue on our end, but we expected to see delayed submissions for this competition given the need to train a model first. It took several days for the first successful submission to be made to the previous sign language competition as well. I have already confirmed that we've only gotten a handful of submission attempts so far.
", "rawMarkdown": "I'm going to review our logs in detail later today to verify that there's no issue on our end, but we expected to see delayed submissions for this competition given the need to train a model first. It took several days for the first successful submission to be made to the previous sign language competition as well. I have already confirmed that we've only gotten a handful of submission attempts so far.", "votes": 1, "replies": [ { "id": 2260744, "postDate": "2023-05-15T20:42:52.830Z", "content": "Thanks for taking a look at the logs, I am struggling for two days now to get a successful submission.
\nConversion to TFLite model and inference in TFLite model are working properly and are following the same format as in the Isolated Sign Language competition.
\nWould be helpful to know how the inference differs from the previous Isolated Sign Language competition.
@sohier Could you share any insights from the logs as to why everyone's submissions are failing? Even my simple notebooks with a model which should return outputs in the correct format are failing.
", "rawMarkdown": "@sohier Could you share any insights from the logs as to why everyone's submissions are failing? Even my simple notebooks with a model which should return outputs in the correct format are failing.", "votes": 1 }, { "id": 2265005, "postDate": "2023-05-18T23:18:43.877Z", "content": "Any news on this?
\n I am assuming the dataframe is converted to a numpy array of [frame, 1630] (including the column \"frame\")
also in this line of inference
\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])
\nis REQUIRED_OUTPUT = \"outputs\"
", "rawMarkdown": "Any news on this?\n I am assuming the dataframe is converted to a numpy array of [frame, 1630] (including the column \"frame\")\n\nalso in this line of inference\n\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])\n\nis REQUIRED_OUTPUT = \"outputs\"", "votes": 2 } ] }, { "id": 2261245, "postDate": "2023-05-16T07:42:56.550Z", "content": "Could there be any issues if subset of landmarks is tried? as per Evaluation
\n\n", "rawMarkdown": "Could there be any issues if subset of landmarks is tried? as per Evaluation\n\n>If you want to load only a subset of the landmarks, include a file named inference_args.json in your submission.zip with the field selected_columns containing a list of the landmark columns you want to use. If that is not included we will load all columns.", "replies": [ { "id": 2264648, "postDate": "2023-05-18T15:46:37.117Z", "content": "If you want to load only a subset of the landmarks, include a file named inference_args.json in your submission.zip with the field selected_columns containing a list of the landmark columns you want to use. If that is not included we will load all columns.
\n
I also tried with all the landmarks (without the JSON file) and the submission still failed. But without any error messages it is hard to know what the reason is.
", "rawMarkdown": "I also tried with all the landmarks (without the JSON file) and the submission still failed. But without any error messages it is hard to know what the reason is.", "votes": 1 } ] }, { "id": 2261538, "postDate": "2023-05-16T11:56:01.327Z", "content": "We trying to prepare our first submit. But it's hard to choose right model size without any knowledge about inference time and what is \"35 hours\"
\n@sohier could you please tell ~inference time for one phrase (asked here ) as you did in previous competition? Where we knew the number of samples (phrases here)
Everyone trusts CV :)
", "rawMarkdown": "Everyone trusts CV :) " }, { "id": 2267913, "postDate": "2023-05-21T10:37:20.203Z", "content": "Was comparing the parquet data in this competition compared to GISLR and think that these parquet files are multiple sequence ids in the one parquet (but should all be for the same participant id). Whereas in GISLR, each parquet was a separate folder for participant id and a separate parquet for each sequence id, really the parquet name was the sequence id. So at the moment what comes back from the function
\nload_relevant_data_subset(pq_path)
\n can be like 1000+ entries and the parquet files are huge 1.5+ GB compared to < 2MB in GISLR.
\nThink it is impossible to do anything with these parquet files given the competition constraints, unless what is passed to the models is a further subset, that is unclear from the Evaluation page. Could be wrong but think the parquet files need to be re-done by sequence id.
\n@sohier - what do you think?
I think here we have some test.csv with same structure as train.csv
And we don't see how variable frames preprocessed. I guess it should be like this for one sample. (Or something more efficient, but with the same output):
test_df = pd.read_csv()\n () json_file:\n selected_columns_ = json.load(json_file)\nselected_columns = selected_columns_[]\n\n ():\n pd.read_parquet(pq_path, columns=selected_columns)\n\ni = \nsample = test_df.loc[i] \nloaded = load_relevant_data_subset( + sample[])\nframes = loaded[loaded.index==sample[]].values\noutput = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output[], axis=)])\nBut yeah, I'm also have an error after 1 minute😔
", "rawMarkdown": "I think here we have some `test.csv` with same structure as `train.csv`\n\nAnd we don't see how variable `frames` preprocessed. I guess it should be like this for one sample. (Or something more efficient, but with the same output):\n\n```python\ntest_df = pd.read_csv('test_csv')\nwith open('inference_args.json') as json_file:\n selected_columns_ = json.load(json_file)\nselected_columns = selected_columns_['selected_columns']\n\ndef load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=selected_columns)\n\ni = 0\nsample = test_df.loc[i] \nloaded = load_relevant_data_subset('asl-fingerspelling/' + sample['path'])\nframes = loaded[loaded.index==sample['sequence_id']].values\noutput = prediction_fn(inputs=frames)\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[\"outputs\"], axis=1)])\n```\n\nBut yeah, I'm also have an error after 1 minute😔" } ] }, { "id": 2266931, "postDate": "2023-05-20T13:30:15.340Z", "content": "Hi all,
\nBesides @lonnieqin , has anyone else managed to make a submission that did not crash after ~1min (so at least managed to get partially through scoring)?
\nI think I've tried everything (including running the code Lonnie shared) and my submissions keep failing after 1min.
For those of you who did get past this one minute-barrier: did you generate TfLite from the default Kaggle kernel or did you use a different setup (Python version? Tensorflow version? other??). And does your submission behave the same if you do run t from the Kaggle kernel?
\nI'm just trying to find out whether part of the problems may be due to a version issue …?
", "rawMarkdown": "Hi all,\n\nBesides @lonnieqin , has anyone else managed to make a submission that did not crash after ~1min (so at least managed to get partially through scoring)? \nI think I've tried everything (including running the code Lonnie shared) and my submissions keep failing after 1min.\n\nFor those of you who did get past this one minute-barrier: did you generate TfLite from the default Kaggle kernel or did you use a different setup (Python version? Tensorflow version? other??). And does your submission behave the same if you do run t from the Kaggle kernel?\n\nI'm just trying to find out whether part of the problems may be due to a version issue ...?", "replies": [ { "id": 2267032, "postDate": "2023-05-20T14:35:13.597Z", "content": "Taking into account Sohiers's answer, I think any submission that didn't crash after 1minutes were put in the queue, and spend some times in the queue.
", "rawMarkdown": "Taking into account Sohiers's answer, I think any submission that didn't crash after 1minutes were put in the queue, and spend some times in the queue. ", "replies": [ { "id": 2267115, "postDate": "2023-05-20T15:50:57.673Z", "content": "Indeed, I resubmitted 1 notebook version that could run over 1 hour before, only to find it failed after a few minutes.
", "rawMarkdown": "Indeed, I resubmitted 1 notebook version that could run over 1 hour before, only to find it failed after a few minutes. " }, { "id": 2267137, "postDate": "2023-05-20T16:10:11.030Z", "content": "Thanks, so that means that probably no-one got beyond that initial crash then?
\nI guess there's nothing to be done then but wait for the organisers to finally clarify the requirements and/or fix any remaining issues … I'm wondering whether they even ran a test submission themselves to verify that all is well.
", "rawMarkdown": "Thanks, so that means that probably no-one got beyond that initial crash then?\n\nI guess there's nothing to be done then but wait for the organisers to finally clarify the requirements and/or fix any remaining issues ... I'm wondering whether they even ran a test submission themselves to verify that all is well. ", "votes": 1 }, { "id": 2267803, "postDate": "2023-05-21T08:43:13.390Z", "content": "I think required model
\n input and output shape is very clear now, it’s more likely a pipeline or dataset issue.
Yep. Nothing to do but wait …
\nUpdate:
\nI tried creating and submitting my dummy model from a 'pinned' kernel from the previous competition and I got the same weird behaviour that you reported: first time I got submission scoring error after ~1h, second time after ~25 min. Probably a coincidence but the submissions from the new (current) Kaggle kernel consistently failed after 1 min.
\nIn the old kernel I had no problem doing inference with the TfLite model I submitted (using 'best guesses' for the stuff that is missing or unclear in the evaluation script code on this site).
", "rawMarkdown": "Yep. Nothing to do but wait ... \n\nUpdate: \n\nI tried creating and submitting my dummy model from a 'pinned' kernel from the previous competition and I got the same weird behaviour that you reported: first time I got submission scoring error after ~1h, second time after ~25 min. Probably a coincidence but the submissions from the new (current) Kaggle kernel consistently failed after 1 min.\n\nIn the old kernel I had no problem doing inference with the TfLite model I submitted (using 'best guesses' for the stuff that is missing or unclear in the evaluation script code on this site).", "votes": 1 } ] } ] } ], "comments": [ { "id": 2260397, "author_name": "Dewei Chen", "author_url": "", "post_date": "2023-05-15T16:30:42.423000", "content": "Because the training set is too large, we still need some time for model training.
", "votes": 3, "replies": [ { "id": 2260446, "author_name": "Mykola", "author_url": "", "post_date": "2023-05-15T16:57:35.657000", "content": "It makes sense, but what about the fact, that in \"Google Research - Identify Contrails to Reduce Global Warming\" there are 60+ teams, while the dataset is more than twice bigger?
", "votes": 1, "replies": [ { "id": 2260528, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-05-15T17:45:39.560000", "content": "It's trivial to make a dummy submission to the contrails competition (or almost any other competition for that matter). The bar for an initial submission is higher in this case since someone will need to prepare a working model first.
", "votes": 1, "replies": [] }, { "id": 2265537, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-19T10:15:43.510000", "content": "@sohier it should be quite trivial to make an initial random submission here as well, but as you can read in some of the other discussion threads, all (very well informed) attempts to do so are failing thus far. Could you please clarify the required input and output formats:
\nThank you for raising this concern. We reviewed the metric and identified that a necessary dataset had failed to properly attach. Submissions should now be working.
", "votes": 4, "replies": [ { "id": 2266361, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-20T00:58:22.053000", "content": "However it still doesn’t work on my side. I am wondering if anyone can make successful submission now.
", "votes": 2, "replies": [] }, { "id": 2266414, "author_name": "Jon", "author_url": "", "post_date": "2023-05-20T03:50:43.543000", "content": "I still can't submit a valid submission. Is it possible to know what are the input shapes by defaults? n_framesx1630?
\nthe output shape should be n_charactersx59?
@sohier Thank you for looking into this. I still can't make any valid submissions. Could you please give the exact formats for the input data passed to the model and the output format expected? Or even better, provide a notebook with a working submission? That would really help us get started on this competition.
", "votes": 3, "replies": [] }, { "id": 2266698, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-05-20T09:23:47.740000", "content": "Sadly, my submissions keep failing as well. I agree with @mdecoster that providing an example submission notebook would be very helpful.
", "votes": 3, "replies": [] }, { "id": 2269074, "author_name": "alexis1024", "author_url": "", "post_date": "2023-05-22T07:48:48.437000", "content": "can you please fix the doc of the evaluation page and let us know the expected signature (especially please define REQUIRED_OUTPUT variable)
", "votes": 1, "replies": [] } ] }, { "id": 2267501, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-21T03:03:49.247000", "content": "Everyone got what should be in REQUIRED_SIGNATURE and REQUIRED_OUTPUT?
\nAs described in Evaluation part
My understanding is that REQUIRED_SIGNATURE = serving_default and REQUIRED_OUTPUT = outputs.
Yes, at least, that was the interpretation in the previous competition.
", "votes": 0, "replies": [] } ] }, { "id": 2267757, "author_name": "George Megre", "author_url": "", "post_date": "2023-05-21T07:59:46.820000", "content": "i suppose that REQUIRED_SIGNATURE = serving_default, but REQUIRED_OUTPUT is number of elements in the true string, because our evaluation metric for this contest is the normalized total levenshtein distance and number of characters in the labels be N and the total levenshtein distance be D. The metric equals (N - D) / N. I believe that organizers want to have range of that metric like [0, 1].
\nFor instance if we have 2 strings real and estiamted: y_true = \"i can see the rings on saturn\", and y_pred = \"i can see the rings on saturn s\" N = len(y_true) , the Levinstein distance is calculated on (y_true, y_pred[:N]). But it is on only my guess i could be wrong. By the way i'm also interest what is REQUIRED_OUTPUT.
In our interpretation, REQUIRED_OUTPUT is \"outputs\", i.e. the name you need to give to the last layer in your model (i.e., the one that produces the outputs).
\nI have posted a demo notebook to show how TfLite inference can be done here
", "votes": 1, "replies": [] } ] } ] }, { "id": 2264709, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-18T16:55:14.017000", "content": "That’s very tricky. I created a random model that takes left hand + right hand landmarks data as input for submission. It can make inferences with parquet files. However submission failed after running about one hour.
", "votes": 1, "replies": [ { "id": 2264765, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-18T17:23:39.807000", "content": "Hi Lonnie,
\nThat's already better than my attempts: this suggests that scoring works for some but not all samples?
", "votes": 0, "replies": [ { "id": 2265084, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-19T01:11:11.760000", "content": "I am not sure. Everything could be possible. It may be some kinds of pipeline issues or dataset issues. But in my notebook it takes 2 minutes to make inferences and evaluation with tf.edit_distance method on train files and supplemental files with low memory footprint.
Hi @lonnieqin ,
\nI tried to make a dummy solution, using all features, and based on the submission model format from the previous competition. I have tried both:
\ninputs = tf.keras.Input(shape=(543,3), name=\"inputs\")
\nand
\ninputs = tf.keras.Input(shape=(None, 543,3), name=\"inputs\")
\nAs input shapes (using all features for now) and I think I've tried all possible output shape tensor combinations, i.e.:
\nSince all these attempts (at a rate of 5 totally uninformative \"random guesses\" per day) failed after about one minute, this suggests they don't even start getting scored. I am assuming there is still something wrong with my interpretation of the input tensor shape and/or required output tensor shape or properties (e.g. data format)?
\nSo, since it seems that your attempts at least get partially scored: could you clarify what the input and output tensor shapes are in your model?
\nBTW: I do feel that it is the responsibility of competition organisers to clarify the desired interface of submitted models. It is also the responsibility of competition organisers of a code competition to make sure that models can be properly tested in Kaggle kernels that have the same settings as the actual test inference kernel. This is not the case since we can currently NOT run tflite interpreter inference checks in Kaggle kernels!!
", "votes": 1, "replies": [ { "id": 2265539, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-19T10:19:10.363000", "content": "My Model has input shape (None, 9) and output shape (num_samples * 18, 59). num_samples is calculated by reading frame column. I choose num_samples * 18 because average phrase length of training file is about 18.
def get_inference_model():\n inputs = tf.keras.Input((9), dtype=tf.float32, name=\"inputs\")\n vector = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n frame_value = vector[:, 8]\n num_samples = tf.math.count_nonzero(frame_value == 1, dtype=tf.int32)\n samples = tf.random.normal(shape=(num_samples * 18, 42))\n outputs = tf.keras.layers.Dense(59, activation=\"softmax\", name=\"outputs\")(samples)\n inference_model = tf.keras.Model(inputs=inputs, outputs=outputs) \n inference_model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics=[\"accuracy\"])\n return inference_model\nDuring evaluation, each parquet file is read using code: pd.read_parquet(pq_path, columns=selected_columns) when I save json file named inference_args.json
{'selected_columns': ['x_left_hand_0', 'x_left_hand_1', 'y_left_hand_0', 'y_left_hand_1', 'x_right_hand_0', 'x_right_hand_1', 'y_right_hand_0', 'y_right_hand_1', 'frame']}\n
\nThis code can read parquet file with shape (n, 9). Training parquet files all have 1000 samples of phrases, when making inference with this model, it can get output tensor with shape (18000, 59).
I think we can use less than 1% of the data for training and validation, which will not cost too much, but still verify that the model meets the competition requirements.
", "votes": 0, "replies": [] }, { "id": 2265627, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-19T11:45:09.903000", "content": "Thanks @lonnieqin
\nI'd already used up my \"random guesses\" for today, but since your dummy model at least scores partially, you've probably got the dimensions right. What I'd missed, obviously, is that features are loaded as a flat array for each frame (instead of the #kpx3 format of the previous competition).
\nWhat may be happening in yoru model that there are additional constraints on output length, e.g. that it cannot be longer than the original number of frames in the video OR that your \"num_samples\" possibly returns zero at some point ?
", "votes": 1, "replies": [] }, { "id": 2265634, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-19T11:50:38.100000", "content": "Btw, my next attempt would be this one:
\nFEATURES = 543*3+1\nNUM_CHARACTERS = 59\n\ndef get_dummy_model():\n inputs = tf.keras.Input(shape=(FEATURES,), dtype=tf.float32, name=\"inputs\")\n print(inputs.shape)\n x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n print(x.shape)\n # just some dummy stuff here to create random output logits\n x = tf.math.reduce_mean(x, axis = 1, keepdims = True)\n print(x.shape)\n x = tf.keras.layers.Dense(NUM_CHARACTERS)(x)\n print(x.shape)\n out = tf.keras.layers.Activation(\"linear\", name=\"outputs\")(x)\n print(out.shape)\n inference_model = tf.keras.Model(inputs=inputs, outputs=out)\n inference_model.compile(loss=\"sparse_categorical_crossentropy\",metrics=\"accuracy\")\n return inference_model\n\ndummy_model_test = get_dummy_model()\ndummy_model_test.summary()\nI tried to restrict num_samples with min and max values, and tried things like num_samples = tf.math.floordiv(tf.shape(inputs)[0], 6), only to find it failed immediately, which may indicate the length of output tensor need to be divided by number of phrases. I can’t even add 1 to num_samples.
", "votes": 0, "replies": [] }, { "id": 2265868, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-19T15:09:30.060000", "content": "Does this mean that the submitted model is supposed to predict outputs for multiple concatenated samples/phrases? I assumed that it would only receive one phrase at the time and that output length would simply have to be <= input length …
", "votes": 0, "replies": [] }, { "id": 2266012, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-19T16:30:12.717000", "content": "I think in mobile devices there are just a few phrases to be predicted at a time, it’s able to handle video streams. However in kaggle competition our model have to make batch inferences since dataset size is so large. The output size can be set to a reasonable value as long as the model’s performance meets requirement.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2260524, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-05-15T17:43:29.287000", "content": "I'm going to review our logs in detail later today to verify that there's no issue on our end, but we expected to see delayed submissions for this competition given the need to train a model first. It took several days for the first successful submission to be made to the previous sign language competition as well. I have already confirmed that we've only gotten a handful of submission attempts so far.
", "votes": 1, "replies": [ { "id": 2260744, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-05-15T20:42:52.830000", "content": "Thanks for taking a look at the logs, I am struggling for two days now to get a successful submission.
\nConversion to TFLite model and inference in TFLite model are working properly and are following the same format as in the Isolated Sign Language competition.
\nWould be helpful to know how the inference differs from the previous Isolated Sign Language competition.
@sohier Could you share any insights from the logs as to why everyone's submissions are failing? Even my simple notebooks with a model which should return outputs in the correct format are failing.
", "votes": 1, "replies": [] }, { "id": 2265005, "author_name": "Jon", "author_url": "", "post_date": "2023-05-18T23:18:43.877000", "content": "Any news on this?
\n I am assuming the dataframe is converted to a numpy array of [frame, 1630] (including the column \"frame\")
also in this line of inference
\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])
\nis REQUIRED_OUTPUT = \"outputs\"
", "votes": 2, "replies": [] } ] }, { "id": 2261245, "author_name": "something4kag", "author_url": "", "post_date": "2023-05-16T07:42:56.550000", "content": "Could there be any issues if subset of landmarks is tried? as per Evaluation
\n\n", "votes": 0, "replies": [ { "id": 2264648, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-05-18T15:46:37.117000", "content": "If you want to load only a subset of the landmarks, include a file named inference_args.json in your submission.zip with the field selected_columns containing a list of the landmark columns you want to use. If that is not included we will load all columns.
\n
I also tried with all the landmarks (without the JSON file) and the submission still failed. But without any error messages it is hard to know what the reason is.
", "votes": 1, "replies": [] } ] }, { "id": 2261538, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-16T11:56:01.327000", "content": "We trying to prepare our first submit. But it's hard to choose right model size without any knowledge about inference time and what is \"35 hours\"
\n@sohier could you please tell ~inference time for one phrase (asked here ) as you did in previous competition? Where we knew the number of samples (phrases here)
Everyone trusts CV :)
", "votes": 0, "replies": [] }, { "id": 2267913, "author_name": "something4kag", "author_url": "", "post_date": "2023-05-21T10:37:20.203000", "content": "Was comparing the parquet data in this competition compared to GISLR and think that these parquet files are multiple sequence ids in the one parquet (but should all be for the same participant id). Whereas in GISLR, each parquet was a separate folder for participant id and a separate parquet for each sequence id, really the parquet name was the sequence id. So at the moment what comes back from the function
\nload_relevant_data_subset(pq_path)
\n can be like 1000+ entries and the parquet files are huge 1.5+ GB compared to < 2MB in GISLR.
\nThink it is impossible to do anything with these parquet files given the competition constraints, unless what is passed to the models is a further subset, that is unclear from the Evaluation page. Could be wrong but think the parquet files need to be re-done by sequence id.
\n@sohier - what do you think?
I think here we have some test.csv with same structure as train.csv
And we don't see how variable frames preprocessed. I guess it should be like this for one sample. (Or something more efficient, but with the same output):
test_df = pd.read_csv()\n () json_file:\n selected_columns_ = json.load(json_file)\nselected_columns = selected_columns_[]\n\n ():\n pd.read_parquet(pq_path, columns=selected_columns)\n\ni = \nsample = test_df.loc[i] \nloaded = load_relevant_data_subset( + sample[])\nframes = loaded[loaded.index==sample[]].values\noutput = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output[], axis=)])\nBut yeah, I'm also have an error after 1 minute😔
", "votes": 0, "replies": [] } ] }, { "id": 2266931, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-20T13:30:15.340000", "content": "Hi all,
\nBesides @lonnieqin , has anyone else managed to make a submission that did not crash after ~1min (so at least managed to get partially through scoring)?
\nI think I've tried everything (including running the code Lonnie shared) and my submissions keep failing after 1min.
For those of you who did get past this one minute-barrier: did you generate TfLite from the default Kaggle kernel or did you use a different setup (Python version? Tensorflow version? other??). And does your submission behave the same if you do run t from the Kaggle kernel?
\nI'm just trying to find out whether part of the problems may be due to a version issue …?
", "votes": 0, "replies": [ { "id": 2267032, "author_name": "Mykola", "author_url": "", "post_date": "2023-05-20T14:35:13.597000", "content": "Taking into account Sohiers's answer, I think any submission that didn't crash after 1minutes were put in the queue, and spend some times in the queue.
", "votes": 0, "replies": [ { "id": 2267115, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-20T15:50:57.673000", "content": "Indeed, I resubmitted 1 notebook version that could run over 1 hour before, only to find it failed after a few minutes.
", "votes": 0, "replies": [] }, { "id": 2267137, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-20T16:10:11.030000", "content": "Thanks, so that means that probably no-one got beyond that initial crash then?
\nI guess there's nothing to be done then but wait for the organisers to finally clarify the requirements and/or fix any remaining issues … I'm wondering whether they even ran a test submission themselves to verify that all is well.
", "votes": 1, "replies": [] }, { "id": 2267803, "author_name": "Lonnie", "author_url": "", "post_date": "2023-05-21T08:43:13.390000", "content": "I think required model
\n input and output shape is very clear now, it’s more likely a pipeline or dataset issue.
Yep. Nothing to do but wait …
\nUpdate:
\nI tried creating and submitting my dummy model from a 'pinned' kernel from the previous competition and I got the same weird behaviour that you reported: first time I got submission scoring error after ~1h, second time after ~25 min. Probably a coincidence but the submissions from the new (current) Kaggle kernel consistently failed after 1 min.
\nIn the old kernel I had no problem doing inference with the TfLite model I submitted (using 'best guesses' for the stuff that is missing or unclear in the evaluation script code on this site).
", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2260138": "Dear all, \n\nWhen I click on the leaderboard, it is empty.\nIs it just me or have no submissions been made yet?\nI find it strange that there aren't even any \"hello world\" baselines yet. Is there something wrong with the data maybe?\n", "2260397": "Because the training set is too large, we still need some time for model training.", "2266183": "Thank you for raising this concern. We reviewed the metric and identified that a necessary dataset had failed to properly attach. Submissions should now be working.", "2267501": "Everyone got what should be in `REQUIRED_SIGNATURE` and `REQUIRED_OUTPUT`? \nAs described in Evaluation part", "2264709": "That’s very tricky. I created a random model that takes left hand + right hand landmarks data as input for submission. It can make inferences with parquet files. However submission failed after running about one hour.", "2260524": "I'm going to review our logs in detail later today to verify that there's no issue on our end, but we expected to see delayed submissions for this competition given the need to train a model first. It took several days for the first successful submission to be made to the previous sign language competition as well. I have already confirmed that we've only gotten a handful of submission attempts so far.", "2260225": "Everyone trusts CV :) ", "2267913": "Was comparing the parquet data in this competition compared to GISLR and think that these parquet files are multiple sequence ids in the one parquet (but should all be for the same participant id). Whereas in GISLR, each parquet was a separate folder for participant id and a separate parquet for each sequence id, really the parquet name was the sequence id. So at the moment what comes back from the function \n`load_relevant_data_subset(pq_path)`\n can be like 1000+ entries and the parquet files are huge 1.5+ GB compared to < 2MB in GISLR.\nThink it is impossible to do anything with these parquet files given the competition constraints, unless what is passed to the models is a further subset, that is unclear from the Evaluation page. Could be wrong but think the parquet files need to be re-done by sequence id. \n@sohier - what do you think?", "2266931": "Hi all,\n\nBesides @lonnieqin , has anyone else managed to make a submission that did not crash after ~1min (so at least managed to get partially through scoring)? \nI think I've tried everything (including running the code Lonnie shared) and my submissions keep failing after 1min.\n\nFor those of you who did get past this one minute-barrier: did you generate TfLite from the default Kaggle kernel or did you use a different setup (Python version? Tensorflow version? other??). And does your submission behave the same if you do run t from the Kaggle kernel?\n\nI'm just trying to find out whether part of the problems may be due to a version issue ...?" } }