![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F518134%2Fe51c936752b5cd7bd0689f6ef9faebe9%2Fvalidation_worst_error.png?generation=1606717208803896&alt=media\")
\nThe biggest error in the validation dataset with error=43988.1 !!
We have published our code:
\nhttps://github.com/pfnet-research/kaggle-lyft-motion-prediction-4th-place-solution
Great job.
\nThanks for sharing!
Thanks, please check/try our ensemble code :)
\n" }, { "id": 1107795, "authorName": "Dean Kang", "votes": 1, "postDate": "2020-12-10T01:03:26.083000", "content": "Awesome! thank you! 👍
" }, { "id": 1107809, "authorName": "corochann", "votes": 2, "postDate": "2020-12-10T01:26:49.637000", "content": "😃
\nYou can find various trial implementations of our rasterizer too: (numba jit tuned version of rasterizer etc)
\n" }, { "id": 1096004, "authorName": "corochann", "votes": 6, "postDate": "2020-11-30T06:27:25.403000", "content": "Sorry for the late post, these are some of the items that we tried but not worked.
\nWe tried several hyper parameters, especially for rasterizers. But none of them contributed to improve the model's accuracy.
\nimage size
\npixel_size
\npixel_size from default 0.5 into 0.25 or 0.15 but the accuracy becomes worse.num_history
\nhistory_num_frames=7, 10 with history_step_size=2 instead of default history_num_frames=10 with history_step_size=1. This model’s accuracy was lower than the original model even if we only chose the validation input to have more than 20 history frames.We tried resnet101 & res152 too, but they did not work well.
We tried implementing our own rasterizer to add more rich information to CNN input, but all of them did not work well to improve the accuracy somehow…
\nChannelSemanticRasterizerSemanticRasterizer draws the semantic in RGB space, using 3 channels. We thought this is not always optimal for CNN input and tried to input 6 channels with 1. road, 2. default lane, 3. green signal lane, 4. yellow signal lane, 5. red signal lane, 6. crosswalk.TLSemanticRasterizer:AgentTypeBoxRasterizer:BoxRasterizer. Also agent type information is also dropped when drawing boxes.Speed up rasterizer
\nUse numpy batch operation as much as possible, and replacing implementations with numba jit computations. Even though it becomes faster in single process, its computation speed-up does not contribute so much for multi-process data preparation during training.
Other than trying the AgentTypeBoxRasterizer, we tried inputting agent type one-hot vector explicitly, but it did not contribute to improve accuracy too.
The baseline kernel predicts future movement of only target agent. Instead I considered to build a model which predicts all the agent's future movement within the input image.
\nThe first I thought this idea speed-ups the training since it can predict multiple agent at once. However sometimes the host car detects agent with very far place, like 400 pixels far away. So we noticed it is difficult to align the image size to fixed size. And we suspended its further trial.
The below figure shows the biggest error in the validation dataset. The error is extremely high when the yaw in the dataset was actually opposite and the model predicted the opposite way to go.
\nWe worked hard to check if the test dataset contains this kind of case. Indeed there seems to be some frames whose yaw might be opposite, however most of the time the agent is stopping in this case in the test dataset. Even though we fixed the yaw, the LB score was almost unchanged.
\nThe biggest error in the validation dataset with error=43988.1 !!
When we checked the dataset carefully, we noticed that the timestamp & the map position actually overlaps within the train/validation/test dataset.
\nWe checked if the test dataset information was leaked from another dataset. But it seems that the timestamp is not aligned in scenes (maybe physically other host car is used to collect data, and timestamp record is not calibrated).
Thanks for sharing! I think this clear list of things that don't work is way more important than those that work!
\nFor Lyft, I think they might want to check the \"Yaw correction\" part. Predicting a car to move in a completely opposite direction will probably be a fatal mistake in real life. Even if it only happen once.
Thanks for reply @louis925, yeah I think so too. For considering application, this \"what did not work\" is more important to consider further why.
\nActually \"Yaw correction\" happened many times, and I guess to correct it we need better accuracy for the previous object detection NN part where I guess this module will decide the yaw direction.
\nWhile the pedestrian & cyclist they are more easier to decide direction from real image and indeed there is less mistake for yaw, but car is just \"box\" shape, determining its direction from image sometimes has mistake.
Thank you so much for sharing. Just curious, did you use GCP or a local machine for training? Using v100 in GCP seems quite expansive.
" }, { "id": 1092923, "authorName": "corochann", "votes": 3, "postDate": "2020-11-27T09:44:30.027000", "content": "We used local machine :)
" }, { "id": 1092700, "authorName": "Dieter", "votes": 3, "postDate": "2020-11-27T05:12:27.447000", "content": "Great write-up, thanks for sharing. I like your GMM ensembling.
" }, { "id": 1092755, "authorName": "corochann", "votes": 0, "postDate": "2020-11-27T06:22:29.687000", "content": "Thank you @christofhenkel for comment. I'm looking forward to see your team's solution.
\nI saw that your team used stacking. I also wonder if your team applied GMM ensembling, the score has increased further or not!
" }, { "id": 1092894, "authorName": "Dieter", "votes": 1, "postDate": "2020-11-27T08:52:44.610000", "content": "thats what I am also wondering. Might give it a try today. Thanks for posting the code
" }, { "id": 1092309, "authorName": "Heroseo", "votes": 3, "postDate": "2020-11-26T17:28:17.537000", "content": "Thanks for sharing and well explanation.
\nI totally agree with you!
\nLyft: Training with multi-mode confidence is really strong baseline.
And Congrats 4th @corochann and @zaburo, @qhapaq49, @charmq.
" }, { "id": 1092326, "authorName": "corochann", "votes": 0, "postDate": "2020-11-26T17:42:01.813000", "content": "Thank you! Glad to know that :) @piantic
" }, { "id": 1098852, "authorName": "YaGana Sheriff-Hussaini", "votes": 1, "postDate": "2020-12-01T22:38:03.257000", "content": "Congrats @corochann and team. Thanks for sharing your solution.
" }, { "id": 1098932, "authorName": "corochann", "votes": 1, "postDate": "2020-12-02T00:33:12.727000", "content": "Thanks, congrats @sheriytm too!
" }, { "id": 1092767, "authorName": "hengck23", "votes": 1, "postDate": "2020-11-27T06:37:08.903000", "content": "good work and thanks for the writeup!
\n\"so we also tried implementing own GMM model with fixing covariance to be 1.\"
\ndo you have some code or pseudo-code for that? i would like to implement and try.
\nthanks
@hengck23
\nThanks for your comment.
\nWe used this code, usage is same with sklearn's GMM.
import numba as nb\nimport numpy as np\nfrom sklearn.mixture import GaussianMixture\n# from sklearn.mixture._gaussian_mixture import _estimate_gaussian_parameters\n\n\n@nb.jit(nb.types.Tuple(\n (nb.float64[:], nb.float64[:, :])\n)(nb.float64[:, :], nb.float64[:, :]), nopython=True, nogil=True)\ndef _estimate_gaussian_parameters(X, resp):\n nk = resp.sum(axis=0) + 10 * np.finfo(resp.dtype).eps\n means = np.dot(np.ascontiguousarray(resp.T), X) / np.ascontiguousarray(np.expand_dims(nk, 1))\n return nk, means\n\n\nclass GaussianMixtureIdentity(GaussianMixture):\n def _initialize(self, X, resp):\n n_samples, _ = X.shape\n self.covariances_ = np.zeros(self.n_components)+1.0\n self.precisions_cholesky_ = np.zeros(self.n_components)+1.0\n weights, means = _estimate_gaussian_parameters(X, resp)\n weights /= n_samples\n\n self.weights_ = (weights if self.weights_init is None\n else self.weights_init)\n self.means_ = means if self.means_init is None else self.means_init\n\n def _m_step(self, X, log_resp):\n n_samples, _ = X.shape\n self.covariances_ = np.zeros(self.n_components)+1.0\n self.precisions_cholesky_ = np.zeros(self.n_components)+1.0\n self.weights_, self.means_ = _estimate_gaussian_parameters(X, np.exp(log_resp))\n self.weights_ /= n_samples\nCongrats! Wow! You set min_history=0! Very interesting idea! Does that increase the amount samples in AgentDataset by a lot?
Yes, the dataset becomes 191,177,863 -> 198,474,478.
I guess min_frame_history=0 increases the number, while min_frame_history=10 reduces the number.
Congrats! What hardware have you used to run such a massive experiment? I have trained my models for days and only covered 34M samples, and you did 191M. How many cores have you used, what was the bottleneck? Quite impressive!
" }, { "id": 1092328, "authorName": "corochann", "votes": 3, "postDate": "2020-11-26T17:45:17.960000", "content": "Thank you and congrats to you too! @zaharch
\nWe used 8 V100 GPUs for single model training with about 32 cpu cores.
\nI think disk IO and rasterization process was the bottleneck compared to the normal image training.
Thanks! @prokaggler
" }, { "id": 1102591, "authorName": "", "votes": 1, "postDate": "2020-12-05T05:04:27.047000", "content": "I love this post!
" }, { "id": 1103475, "authorName": "corochann", "votes": 0, "postDate": "2020-12-05T23:56:34.103000", "content": "Thanks @YoungseokJoung !
" } ], "index": { "id": "199657", "title": "4th place solution: Ensemble with GMM", "authorName": "corochann", "commentCount": "26", "votes": "58", "postDate": "2020-12-09 09:06:03.400000" } }