{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on May 11, 2023 by Marília Prata, mpwolke","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport dask.dataframe as dd\nimport dask.array as da\nfrom dask.diagnostics import ProgressBar\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-05-11T01:43:23.752997Z","iopub.execute_input":"2023-05-11T01:43:23.753903Z","iopub.status.idle":"2023-05-11T01:43:26.008976Z","shell.execute_reply.started":"2023-05-11T01:43:23.753864Z","shell.execute_reply":"2023-05-11T01:43:26.007635Z"},"_kg_hide-input":true,"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Citation\n\n@misc{asl-fingerspelling,\n\n    author = {Ashley Chow, Glenn Cameron, HCL-Jevster, Manfred Georg, Mark \n    \n    Sherwood, Phil Culliton, Sam Sepah, Sohier Dane, Thad Starner},\n    \n    title = {Google - American Sign Language Fingerspelling Recognition},\n    \n    publisher = {Kaggle},\n    \n    year = {2023},\n    \n    url = {https://kaggle.com/competitions/asl-fingerspelling}\n}","metadata":{}},{"cell_type":"markdown","source":"![](https://www.signlanguageforum.com/img/fingerspelling/ASL-Fingerspelling-Alphabet.png)https://www.signlanguageforum.com/asl/fingerspelling/alphabet/","metadata":{}},{"cell_type":"markdown","source":"\"Finger spelling is a building block in which you or your child uses hands and fingers to spell out words. Hand shapes represent the letters in the alphabet. Finger spelling is used with many other building blocks; it is almost never used by itself. It is most often used with American Sign Language (ASL), Conceptually Accurate Signed English (CASE), and Manually Coded English (MCE) to spell out words that don’t have a sign — such as the names of places or people.\"\n\nhttps://www.cdc.gov/ncbddd/hearingloss/parentsguide/building/finger-spelling.html#:~:text=Finger%20spelling%20is%20a%20building,almost%20never%20used%20by%20itself.","metadata":{}},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nBASE_DIR = '../input/asl-fingerspelling/'\ntrain = pd.read_csv(f'{BASE_DIR}/train.csv')","metadata":{"execution":{"iopub.status.busy":"2023-05-11T01:48:45.152573Z","iopub.execute_input":"2023-05-11T01:48:45.153367Z","iopub.status.idle":"2023-05-11T01:48:45.314649Z","shell.execute_reply.started":"2023-05-11T01:48:45.153321Z","shell.execute_reply":"2023-05-11T01:48:45.313514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\n# Train.csv has the path to each parquet file, the particpant id, sequence_id and sign.\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:52:00.934969Z","iopub.execute_input":"2023-05-11T02:52:00.936321Z","iopub.status.idle":"2023-05-11T02:52:00.949627Z","shell.execute_reply.started":"2023-05-11T02:52:00.936268Z","shell.execute_reply":"2023-05-11T02:52:00.948342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#How to Spell with one Finger an URL? Probably is with more than one :D","metadata":{}},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nfig, ax = plt.subplots(figsize=(8, 8))\ntrain[\"phrase\"].value_counts().head(10).sort_values(ascending=True).plot(\n    kind=\"barh\", ax=ax, title=\"Top 10 Signs in Training Dataset\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:52:52.959478Z","iopub.execute_input":"2023-05-11T02:52:52.959907Z","iopub.status.idle":"2023-05-11T02:52:53.272203Z","shell.execute_reply.started":"2023-05-11T02:52:52.959873Z","shell.execute_reply":"2023-05-11T02:52:53.270893Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Let's work with Supplemental Metadata \n\nBy the way, that is my style of having Fun and work. Check phrase on line 6:\n\n\"All work and no play.\"","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/asl-fingerspelling/supplemental_metadata.csv\", delimiter=',', encoding='UTF-8')\npd.set_option('display.max_columns', None)\ndf.head(7)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:42:17.737255Z","iopub.execute_input":"2023-05-11T02:42:17.737644Z","iopub.status.idle":"2023-05-11T02:42:17.82177Z","shell.execute_reply.started":"2023-05-11T02:42:17.737614Z","shell.execute_reply":"2023-05-11T02:42:17.820667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#And the winner is: \"Why do ask silly questions?\" Fingerspelt 117 Times\n\nFind a nearby parking spot requires a car. No Fun.","metadata":{}},{"cell_type":"code","source":"df[\"phrase\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:24:15.189095Z","iopub.execute_input":"2023-05-11T02:24:15.189491Z","iopub.status.idle":"2023-05-11T02:24:15.207304Z","shell.execute_reply.started":"2023-05-11T02:24:15.189461Z","shell.execute_reply":"2023-05-11T02:24:15.206212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"silly = df[(df['phrase']==\"why do you ask silly questions\")].reset_index(drop=True)\nsilly.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:31:32.889481Z","iopub.execute_input":"2023-05-11T02:31:32.889875Z","iopub.status.idle":"2023-05-11T02:31:32.912598Z","shell.execute_reply.started":"2023-05-11T02:31:32.889847Z","shell.execute_reply":"2023-05-11T02:31:32.911211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nfig, ax = plt.subplots(figsize=(4, 4))\ndf[\"phrase\"].value_counts().head(3).sort_values(ascending=True).plot(\n    kind=\"barh\", color='g', ax=ax, title=\"Top 3 Fingerspelling: Why do you ask silly questions?\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:49:37.906862Z","iopub.execute_input":"2023-05-11T02:49:37.907288Z","iopub.status.idle":"2023-05-11T02:49:38.088322Z","shell.execute_reply.started":"2023-05-11T02:49:37.907253Z","shell.execute_reply":"2023-05-11T02:49:38.08721Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Pull an example parquet file data\n\nRob Mulla pulled an example landmark file for the phrase \"hentaihubs.com\"","metadata":{}},{"cell_type":"code","source":"example_fn = train.query('phrase == \"hentaihubs.com\"')[\"path\"].values[0]\n\nexample_landmark = pd.read_parquet(f\"{BASE_DIR}/{example_fn}\")\nexample_landmark.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T02:56:48.031049Z","iopub.execute_input":"2023-05-11T02:56:48.031516Z","iopub.status.idle":"2023-05-11T02:57:04.865636Z","shell.execute_reply.started":"2023-05-11T02:56:48.03148Z","shell.execute_reply":"2023-05-11T02:57:04.864662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_frames = example_landmark[\"frame\"].nunique()\n#unique_types = example_landmark[\"type\"].nunique()\n#types_in_video = example_landmark[\"type\"].unique()\nprint(\n    f\"The file has {unique_frames} unique frames\"\n)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:02:03.647337Z","iopub.execute_input":"2023-05-11T03:02:03.650758Z","iopub.status.idle":"2023-05-11T03:02:03.667937Z","shell.execute_reply.started":"2023-05-11T03:02:03.650643Z","shell.execute_reply":"2023-05-11T03:02:03.666497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Really? 672 unique frames. I've No clue how to deal with that ","metadata":{}},{"cell_type":"markdown","source":"#Read only one file. At least.","metadata":{}},{"cell_type":"code","source":"ddf = dd.read_parquet(\"/kaggle/input/asl-fingerspelling/train_landmarks/1019715464.parquet\")","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:11:02.402418Z","iopub.execute_input":"2023-05-11T03:11:02.402961Z","iopub.status.idle":"2023-05-11T03:11:02.699269Z","shell.execute_reply.started":"2023-05-11T03:11:02.402922Z","shell.execute_reply":"2023-05-11T03:11:02.697949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nlen(ddf)","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:11:46.515407Z","iopub.execute_input":"2023-05-11T03:11:46.515854Z","iopub.status.idle":"2023-05-11T03:11:47.107245Z","shell.execute_reply.started":"2023-05-11T03:11:46.515818Z","shell.execute_reply":"2023-05-11T03:11:47.106402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:12:02.59838Z","iopub.execute_input":"2023-05-11T03:12:02.598814Z","iopub.status.idle":"2023-05-11T03:12:24.576034Z","shell.execute_reply.started":"2023-05-11T03:12:02.598779Z","shell.execute_reply":"2023-05-11T03:12:24.575207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.x_face_0.describe().compute()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:12:54.026388Z","iopub.execute_input":"2023-05-11T03:12:54.028334Z","iopub.status.idle":"2023-05-11T03:12:54.211165Z","shell.execute_reply.started":"2023-05-11T03:12:54.028278Z","shell.execute_reply":"2023-05-11T03:12:54.210256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/code/luxcem/dask-on-large-parquet-dataset/notebook\n\nddf.where(ddf.x_face_0 > 50).visualize()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:13:20.842934Z","iopub.execute_input":"2023-05-11T03:13:20.843579Z","iopub.status.idle":"2023-05-11T03:13:21.508697Z","shell.execute_reply.started":"2023-05-11T03:13:20.84353Z","shell.execute_reply":"2023-05-11T03:13:21.507555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/code/luxcem/dask-on-large-parquet-dataset/notebook\n\nh, bins = da.histogram(ddf[[\"x_face_0\"]].to_dask_array(), bins=(0, 1, 2, 5, 10, 20, 50, 100, 1000))","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:14:01.376537Z","iopub.execute_input":"2023-05-11T03:14:01.376978Z","iopub.status.idle":"2023-05-11T03:14:01.392702Z","shell.execute_reply.started":"2023-05-11T03:14:01.376942Z","shell.execute_reply":"2023-05-11T03:14:01.391483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with ProgressBar():\n    hd = h.compute()\n    bins = bins.compute()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:14:16.802567Z","iopub.execute_input":"2023-05-11T03:14:16.803021Z","iopub.status.idle":"2023-05-11T03:14:29.741439Z","shell.execute_reply.started":"2023-05-11T03:14:16.802985Z","shell.execute_reply":"2023-05-11T03:14:29.740213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/code/luxcem/dask-on-large-parquet-dataset/notebook\n\nplt.style.use('fivethirtyeight')\n\nsns.barplot(x=bins[1:], y=hd);","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:14:38.554215Z","iopub.execute_input":"2023-05-11T03:14:38.554663Z","iopub.status.idle":"2023-05-11T03:14:38.853584Z","shell.execute_reply.started":"2023-05-11T03:14:38.554629Z","shell.execute_reply":"2023-05-11T03:14:38.852278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Another way of reading parquet\n\nhttps://www.kaggle.com/code/sohier/reading-the-data-with-python","metadata":{}},{"cell_type":"code","source":"#By Sohier Dane https://www.kaggle.com/code/sohier/reading-the-data-with-python\n\nimport pyarrow.parquet as pq\n\ntrain = pq.read_pandas('../input/asl-fingerspelling/train_landmarks/1019715464.parquet').to_pandas()","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:15:54.47024Z","iopub.execute_input":"2023-05-11T03:15:54.470717Z","iopub.status.idle":"2023-05-11T03:15:56.363333Z","shell.execute_reply.started":"2023-05-11T03:15:54.47068Z","shell.execute_reply":"2023-05-11T03:15:56.362059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Closing Arguments\n\nSince I've already copied all the snippets that I could run again, I'll stop here cause working with such an amount of frames like that isn't for amateurs.\n\nBesides, I prefer not asking any Silly Question.","metadata":{}},{"cell_type":"markdown","source":"![](https://i.pinimg.com/236x/83/bb/7a/83bb7a2dc4c51711a047200b975606e3.jpg)pinterest","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nLuxcem: https://www.kaggle.com/code/luxcem/dask-on-large-parquet-dataset/notebook\n\nRob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nSohier Dane https://www.kaggle.com/code/sohier/reading-the-data-with-python","metadata":{"execution":{"iopub.status.busy":"2023-05-11T03:04:23.847729Z","iopub.execute_input":"2023-05-11T03:04:23.848167Z","iopub.status.idle":"2023-05-11T03:04:23.890556Z","shell.execute_reply.started":"2023-05-11T03:04:23.848122Z","shell.execute_reply":"2023-05-11T03:04:23.888874Z"}}}]}