WELCOME TO Google - American Sign Language Fingerspelling Recognition
\n\n**AUTHOR - SUJAY KAPADNIS**\n\n**DATE - 12/05/2023**\n\n## Description\n### Goal of the Competition\nThe goal of this competition is to **detect** and **translate** American Sign Language **(ASL)** fingerspelling into **text**. You will create a model trained on the largest dataset of its kind, released specifically for this competition. The data includes more than **three million fingerspelled characters** produced by **over 100 Deaf signers** captured via the selfie camera of a smartphone with a variety of backgrounds and lighting conditions.\n\n\n\n## Context\n**Fingerspelling** uses **hand shapes** that represent **individual letters** *to* convey **words**. While fingerspelling is only a part of ASL, it is often **used for communicating names, addresses, phone numbers, and other information commonly entered on a mobile phone**. Many Deaf smartphone users can **fingerspell** words **faster** than they can type on **mobile keyboards**. In fact, ASL **fingerspelling** can be **substantially faster** than typing on a smartphone’s virtual keyboard (57 words/minute average versus 36 words/minute US average). **But** **sign language recognition AI for text** entry lags far behind voice-to-text or even gesture-based typing, as robust datasets didn't previously exist.\nTechnology that understands sign language fits squarely within Google's mission to organize the world's information and make it universally accessible and useful. Google’s AI principles also support this idea and encourage Google to make products that empower people, widely benefit current and future generations, and work for the common good. This collaboration between Google and the Deaf Professional Arts Network will explore AI solutions that can be scaled globally (such as other sign languages), and support individual user experience needs while interacting with products.\nBesides convenient text entry for web search, map directions, and texting, there is potential for an **app** that **can** then **translate this input using sign language-to-speech technology to speak the words**. Such an app would enable the Deaf and Hard of Hearing community to communicate with hearing non-signers more quickly and smoothly.\n\n## Evaluation Metric\nThe evaluation metric for this contest is the normalized total [levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance). Let the **total number of characters in the labels be N** and the **total levenshtein distance be D**. The **metric** equals `(N - D) / N\n`*levenshtein distance - between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other*\n\nLets try to understand it with the example\nFor example, the Levenshtein distance between \"kitten\" and \"sitting\" is 3, since the following 3 edits change one into the other, and there is no way to do it with fewer than 3 edits:\n\n- kitten → sitten (substitution of \"s\" for \"k\"),\n- sitten → sittin (substitution of \"i\" for \"e\"),\n- sittin → sitting (insertion of \"g\" at the end).[source: Wiki]\n\n## Dataset Summary\nThe ASL Fingerspelling Recognition Corpus (version 1.0) is a collection of hand and facial landmarks generated by **Mediapipe version 0.9.0.1** on videos of phrases, addresses, phone numbers, and urls fingerspelling by over 100 Deaf signers.\n\n## Files\n**[train/supplemental_metadata].csv**\n\n`path` - The path to the landmark file.\n\n`file_id` - A unique identifier for the data file.\n\n`participant_id` - A unique identifier for the data contributor.\n\n`sequence_id` - A unique identifier for the landmark sequence. Each data file may contain many sequences.\n\n`phrase` - The labels for the landmark sequence. The train and test datasets contain randomly generated addresses, phone numbers, and urls derived from components of real addresses/phone numbers/urls. Any overlap with real addresses, phone numbers, or urls is purely accidental. The supplemental dataset consists of fingerspelled sentences. Note that some of the urls include adult content. The intent of this competition is to support the Deaf and Hard of Hearing community in engaging with technology on an equal footing with other adults.\n\n**character_to_prediction_index.json**\n\n**[train/supplemental]_landmarks**/ The landmark data. The landmarks were extracted from raw videos with the MediaPipe holistic model. Not all of the frames necessarily had visible hands or hands that could be detected by the model.\nThe landmark files contain the same data as in the ASL Signs competition (minus the row ID column) but reshaped into a wide format. This allows you to take advantage of the Parquet format to entirely skip loading landmarks that you aren't using.\n\n`sequence_id` - A unique identifier for the landmark sequence. Most landmark files contain 1,000 sequences. The sequence ID is used as the dataframe index.\n\n`frame` - The frame number within a landmark sequence.\n\n`[x/y/z]_[type]_[landmark_index]` - There are now 1,629 spatial coordinate columns for the x, y and z coordinates for each of the 543 landmarks. The type of landmark is one of `['face', 'left_hand', 'pose', 'right_hand']`. Details of the hand landmark locations can be found here. The spatial coordinates have already been normalized by MediaPipe. Note that the MediaPipe model is not fully trained to predict depth so you may wish to ignore the z values. The landmarks have been converted to float32.\n\n\n### This is work in Progress, if you find this helpful kindly upvote","metadata":{"papermill":{"duration":0.011905,"end_time":"2023-04-24T10:43:58.469766","exception":false,"start_time":"2023-04-24T10:43:58.457861","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"###Who is this notebook for
\n\nThis notebook can be very helpful for the beginners, it contains\n- Deep Learning concepts\n- NLP concepts\n- Tensorflow apis understanding\n- Transformers\n\nIt was quite a tough task for me too, I have analzed the codes line by line and written comments for understanding everywhere, also how the inbuilt functions of tensorflow(the ones that we do not commonly use) are shown in relevant cells. \n\nIt was a genuine try for help, help for myself and others, I hope you find this notebook helpful and start your journey for kaggle's featured competition from this notebook.\n\nI'll Leave you with the code, Enjoy:)","metadata":{}},{"cell_type":"markdown","source":"###LOAD YOUR DEPENDENCIES
","metadata":{"papermill":{"duration":0.010273,"end_time":"2023-04-24T10:43:58.490826","exception":false,"start_time":"2023-04-24T10:43:58.480553","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# # install nb_black for autoformating\n# !pip install nb_black --quiet\n# %load_ext lab_black","metadata":{"papermill":{"duration":15.177598,"end_time":"2023-04-24T10:44:13.679161","exception":false,"start_time":"2023-04-24T10:43:58.501563","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-07-14T18:39:01.471384Z","iopub.execute_input":"2023-07-14T18:39:01.471836Z","iopub.status.idle":"2023-07-14T18:39:01.478581Z","shell.execute_reply.started":"2023-07-14T18:39:01.47178Z","shell.execute_reply":"2023-07-14T18:39:01.47735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install mediapipe --quiet","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:01.481883Z","iopub.execute_input":"2023-07-14T18:39:01.482463Z","iopub.status.idle":"2023-07-14T18:39:15.877125Z","shell.execute_reply.started":"2023-07-14T18:39:01.48243Z","shell.execute_reply":"2023-07-14T18:39:15.875947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib as mpl\nfrom matplotlib import animation, rc\nimport matplotlib.pyplot as plt\nimport json\nimport mediapipe as mp\nfrom mediapipe.framework.formats import landmark_pb2\nimport plotly.graph_objs as go\n\nfrom colorama import Style, Fore\n\nimport os\nimport shutil\nimport pyarrow.parquet as pq\nimport tensorflow as tf\nimport json\nimport random\n\nfrom skimage.transform import resize\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tqdm.notebook import tqdm\nfrom matplotlib import animation, rc\n\nblk = Style.BRIGHT + Fore.BLACK\nred = Style.BRIGHT + Fore.RED\nblu = Style.BRIGHT + Fore.BLUE\ncyan = Style.BRIGHT + Fore.CYAN\ngreen = Style.BRIGHT + Fore.GREEN\nres = Style.RESET_ALL","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:15.879842Z","iopub.execute_input":"2023-07-14T18:39:15.881567Z","iopub.status.idle":"2023-07-14T18:39:23.996832Z","shell.execute_reply.started":"2023-07-14T18:39:15.881521Z","shell.execute_reply":"2023-07-14T18:39:23.995879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load data\nBASE_DIR = \"/kaggle/input/asl-fingerspelling/\"\ntrain_metadata_df = pd.read_csv(f\"{BASE_DIR}train.csv\")\nsupp_metadata_df = pd.read_csv(f\"{BASE_DIR}supplemental_metadata.csv\")\nchar_to_pred_idx = json.load(open(f\"{BASE_DIR}character_to_prediction_index.json\"))","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:23.998134Z","iopub.execute_input":"2023-07-14T18:39:23.998833Z","iopub.status.idle":"2023-07-14T18:39:24.263323Z","shell.execute_reply.started":"2023-07-14T18:39:23.998797Z","shell.execute_reply":"2023-07-14T18:39:24.26231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the training data\nprint(train_metadata_df.shape)\nprint(train_metadata_df['participant_id'].nunique())\ntrain_metadata_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:24.265967Z","iopub.execute_input":"2023-07-14T18:39:24.266339Z","iopub.status.idle":"2023-07-14T18:39:24.291132Z","shell.execute_reply.started":"2023-07-14T18:39:24.266305Z","shell.execute_reply":"2023-07-14T18:39:24.290248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the supplemental data\nprint(supp_metadata_df.shape)\nprint(supp_metadata_df['participant_id'].nunique())\nsupp_metadata_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:24.292596Z","iopub.execute_input":"2023-07-14T18:39:24.293195Z","iopub.status.idle":"2023-07-14T18:39:24.307294Z","shell.execute_reply.started":"2023-07-14T18:39:24.293162Z","shell.execute_reply":"2023-07-14T18:39:24.306266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 8))\ntrain_metadata_df[\"phrase\"].value_counts().head(20).sort_values(ascending=True).plot(\n kind=\"barh\", ax=ax, title=\"Top 20 phrases\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:24.309118Z","iopub.execute_input":"2023-07-14T18:39:24.309449Z","iopub.status.idle":"2023-07-14T18:39:24.74697Z","shell.execute_reply.started":"2023-07-14T18:39:24.309419Z","shell.execute_reply":"2023-07-14T18:39:24.746058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"surprise az was the phrase the occured the most followed by yonkers new york and so on","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,8))\ntrain_metadata_df[\"phrase\"].value_counts().tail(10).sort_values(ascending = True).plot(\n kind=\"barh\", ax=ax, title=\"Bottom 10 Phrases\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:24.74808Z","iopub.execute_input":"2023-07-14T18:39:24.748406Z","iopub.status.idle":"2023-07-14T18:39:25.039772Z","shell.execute_reply.started":"2023-07-14T18:39:24.748373Z","shell.execute_reply":"2023-07-14T18:39:25.038901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"supp_metadata_df.query(\"phrase == 'surprise az'\")","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:25.041166Z","iopub.execute_input":"2023-07-14T18:39:25.042237Z","iopub.status.idle":"2023-07-14T18:39:25.059998Z","shell.execute_reply.started":"2023-07-14T18:39:25.0422Z","shell.execute_reply":"2023-07-14T18:39:25.059145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12, 12))\nsupp_metadata_df[\"phrase\"].value_counts().head(50).sort_values(ascending=True).plot(\n kind=\"barh\", ax=ax, title=\"Top 50 phrases\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:25.061357Z","iopub.execute_input":"2023-07-14T18:39:25.061762Z","iopub.status.idle":"2023-07-14T18:39:25.991565Z","shell.execute_reply.started":"2023-07-14T18:39:25.061729Z","shell.execute_reply":"2023-07-14T18:39:25.990656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### In the training data provided to us, let us see how many instances are there for surprise az phrase","metadata":{}},{"cell_type":"code","source":"for col in train_metadata_df.columns:\n print(col,end = '\\t')\n print(pd.merge(train_metadata_df,supp_metadata_df, on = [col]).shape)","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:25.996281Z","iopub.execute_input":"2023-07-14T18:39:25.996948Z","iopub.status.idle":"2023-07-14T18:39:30.471553Z","shell.execute_reply.started":"2023-07-14T18:39:25.99691Z","shell.execute_reply":"2023-07-14T18:39:30.470596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Attention Please
\n\nI have analyzed other notebooks that have been made public as of today(15-5-23). They are very good notebooks but due to absence of comments I have a little hard time understanding some code. So I am here making that code easy using comments. kindly upvote the original content as well.\n\nThanks","metadata":{}},{"cell_type":"code","source":"# helper functions\ndef map_new_to_old_style(sequence):\n # there are 4 types of landmarks [face,pose,right_hand,left_hand]\n types = []# list where we'll store landmarks\n landmark_indexes = []\n for column in list(sequence.columns)[1:544]:\n # first column is frame which is not a landmark so counter starts from 1\n # why 544 - because we are given that there are now 1,629 spatial coordinate columns for the x, y and z coordinates for each of the 543 landmarks.\n parts = column.split(\"_\")\n if len(parts) == 4:\n # for x_left_hand_1 - there will be 4 parts and for x_pose_1 there will be 3 only\n types.append(parts[1] + \"_\" + parts[2])\n else:\n types.append(parts[1])\n\n landmark_indexes.append(int(parts[-1]))\n\n data = {\"frame\": [], \"type\": [], \"landmark_index\": [], \"x\": [], \"y\": [], \"z\": []}\n\n for index, row in sequence.iterrows():\n data[\"frame\"] += [int(row.frame)] * 543\n # frame has values from 1,2,3....\n # 1. Firstly we are converting them into integers - 1\n # 2. Then we are making it a list - [1]\n # 3. Multiplying by 543 makes the replicates the same element(1 from our example) 543 times for all the landmarks present\n data[\"type\"] += types\n data[\"landmark_index\"] += landmark_indexes\n\n for _type, landmark_index in zip(types, landmark_indexes):\n data[\"x\"].append(row[f\"x_{_type}_{landmark_index}\"])\n data[\"y\"].append(row[f\"y_{_type}_{landmark_index}\"])\n data[\"z\"].append(row[f\"z_{_type}_{landmark_index}\"])\n\n return pd.DataFrame.from_dict(data)\n\n\ndef assign_colors(row):\n if row == \"face\":\n return \"red\"\n elif \"hand\" in row:\n return \"blue\"\n else:\n return \"green\"\n\n\n# specifies the plotting order\ndef assign_order(row):\n if row.type == \"face\":\n return row.landmark_index + 101\n elif row.type == \"pose\":\n return row.landmark_index + 30\n elif row.type == \"left_hand\":\n return row.landmark_index + 80\n else:\n return row.landmark_index\n\n\n# A function to visualize the landmarks in 2d\ndef visualise2d_landmarks(parquet_df, title=\"\"):\n # we first define a list of landmark connections, which specify which landmarks are connected by a line.\n\n # face landmarks are not connected by lines, you can also see that we have added 101 to face landmark indexs and in connections all values are below 100\n\n connections = [ \n [0, 1, 2, 3, 4,],\n [0, 5, 6, 7, 8],\n [0, 9, 10, 11, 12],\n [0, 13, 14, 15, 16],\n [0, 17, 18, 19, 20],\n\n \n [38, 36, 35, 34, 30, 31, 32, 33, 37],\n [40, 39],\n [52, 46, 50, 48, 46, 44, 42, 41, 43, 45, 47, 49, 45, 51],\n [42, 54, 56, 58, 60, 62, 58],\n [41, 53, 55, 57, 59, 61, 57],\n [54, 53],\n\n \n [80, 81, 82, 83, 84, ],\n [80, 85, 86, 87, 88],\n [80, 89, 90, 91, 92],\n [80, 93, 94, 95, 96],\n [80, 97, 98, 99, 100], ]\n\n parquet_df = map_new_to_old_style(parquet_df)\n frames = sorted(set(parquet_df.frame))\n first_frame = min(frames)\n parquet_df[\"color\"] = parquet_df.type.apply(lambda row: assign_colors(row))\n parquet_df[\"plot_order\"] = parquet_df.apply(lambda row: assign_order(row), axis=1)\n first_frame_df = parquet_df[parquet_df.frame == first_frame].copy()\n first_frame_df = first_frame_df.sort_values([\"plot_order\"]).set_index(\"plot_order\")\n\n frames_l = []\n for frame in frames:\n filtered_df = parquet_df[parquet_df.frame == frame].copy()\n filtered_df = filtered_df.sort_values([\"plot_order\"]).set_index(\"plot_order\")\n traces = [\n go.Scatter(\n x=filtered_df[\"x\"],\n y=filtered_df[\"y\"],\n mode=\"markers\",\n marker=dict(color=filtered_df.color, size=9),\n )\n ]\n\n for i, seg in enumerate(connections):\n trace = go.Scatter(\n x=filtered_df.loc[seg][\"x\"],\n y=filtered_df.loc[seg][\"y\"],\n mode=\"lines\",\n )\n traces.append(trace)\n frame_data = go.Frame(data=traces, traces=[i for i in range(17)])\n frames_l.append(frame_data)\n\n traces = [\n go.Scatter(\n x=first_frame_df[\"x\"],\n y=first_frame_df[\"y\"],\n mode=\"markers\",\n marker=dict(color=first_frame_df.color, size=9),\n )\n ]\n for i, seg in enumerate(connections):\n trace = go.Scatter(\n x=first_frame_df.loc[seg][\"x\"],\n y=first_frame_df.loc[seg][\"y\"],\n mode=\"lines\",\n line=dict(color=\"black\", width=2),\n )\n traces.append(trace)\n fig = go.Figure(data=traces, frames=frames_l)\n\n fig.update_layout(\n width=500,\n height=800,\n scene={\n \"aspectmode\": \"data\",\n },\n updatemenus=[\n {\n \"buttons\": [\n {\n \"args\": [\n None,\n {\n \"frame\": {\"duration\": 100, \"redraw\": True},\n \"fromcurrent\": True,\n \"transition\": {\"duration\": 0},\n },\n ],\n \"label\": \"▶\",\n \"method\": \"animate\",\n },\n {\n \"args\": [\n [None],\n {\n \"frame\": {\"duration\": 0, \"redraw\": False},\n \"mode\": \"immediate\",\n \"transition\": {\"duration\": 0},\n },\n ],\n \"label\": \"▌▌\",\n \"method\": \"animate\",\n },\n ],\n \"direction\": \"left\",\n \"pad\": {\"r\": 100, \"t\": 100},\n \"font\": {\"size\": 20},\n \"type\": \"buttons\",\n \"x\": 0.1,\n \"y\": 0,\n }\n ],\n )\n camera = dict(up=dict(x=0, y=-1, z=0), eye=dict(x=0, y=0, z=2.5))\n fig.update_layout(title_text=title, title_x=0.5)\n fig.update_layout(scene_camera=camera, showlegend=False)\n fig.update_layout(\n xaxis=dict(visible=False),\n yaxis=dict(visible=False),\n )\n fig.update_yaxes(autorange=\"reversed\")\n\n fig.show()\n\n\ndef get_phrase(df, file_id, sequence_id):\n return df[\n np.logical_and(df.file_id == file_id, df.sequence_id == sequence_id)\n ].phrase.iloc[0]","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:52.755642Z","iopub.execute_input":"2023-07-14T18:39:52.755997Z","iopub.status.idle":"2023-07-14T18:39:52.78886Z","shell.execute_reply.started":"2023-07-14T18:39:52.755951Z","shell.execute_reply":"2023-07-14T18:39:52.788241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"file_id = 474255203\nsequence_id = 53122870\n\nsurprise = f\"/kaggle/input/asl-fingerspelling/train_landmarks/{file_id}.parquet\"\nsurprised = pd.read_parquet(surprise)\n\nsequence = surprised[surprised.index == sequence_id]\nsequence","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:39:52.790206Z","iopub.execute_input":"2023-07-14T18:39:52.790809Z","iopub.status.idle":"2023-07-14T18:40:09.26095Z","shell.execute_reply.started":"2023-07-14T18:39:52.790776Z","shell.execute_reply":"2023-07-14T18:40:09.259972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_phrase(train_metadata_df, file_id, sequence_id)\nvisualise2d_landmarks(sequence, \"Surprised\")","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:09.262412Z","iopub.execute_input":"2023-07-14T18:40:09.262744Z","iopub.status.idle":"2023-07-14T18:40:24.326696Z","shell.execute_reply.started":"2023-07-14T18:40:09.262711Z","shell.execute_reply":"2023-07-14T18:40:24.325864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function create animation from images.\n\nmpl.rcParams['animation.embed_limit'] = 2**128\n'''\nanimation.embed_limit determines the maximum size limit for embedding animations in output files or displays.\nIn this case, the value 2**128 is assigned, which is a very large value to ensure that there are no limitations on embedding animations.\n'''\n\nmpl.rcParams['savefig.pad_inches'] = 0\n'''\nsavefig.pad_inches determines the amount of padding (in inches) to be added around the saved figures.\nIn this case, the value 0 is assigned, meaning no padding will be added around the saved figures.'''\n\nrc('animation', html='jshtml')\n'''animation.html determines the HTML representation for displaying animations.\nIn this case, the value 'jshtml' is assigned, which indicates that JavaScript-based HTML animations will be used.\n'''\n\ndef create_animation(images):\n fig = plt.figure(figsize=(6, 9))\n ax = plt.Axes(fig, [0., 0., 1., 1.])#The [0., 0., 1., 1.] argument specifies the position and size of the Axes object, where [left, bottom, width, height] are normalized coordinates ranging from 0 to 1.'''\n ax.set_axis_off()\n fig.add_axes(ax)\n im=ax.imshow(images[0], cmap=\"gray\")# assigns the first image from the images list to the im object.'''\n plt.close(fig)\n \n def animate_func(i): #will be called for each frame of the animation.'''\n im.set_array(images[i])\n return [im]\n\n return animation.FuncAnimation(fig, animate_func, frames=len(images), interval=1000/10)\n\n\n'''FuncAnimation class constructs an animation by repeatedly calling the animate_func function for each frame\nfig is the figure object to be animated, \nanimate_func is the animation function,\nframes specifies the number of frames (equal to the length of the images list), and interval sets the delay between frames in milliseconds.'''","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:24.328232Z","iopub.execute_input":"2023-07-14T18:40:24.328805Z","iopub.status.idle":"2023-07-14T18:40:24.343546Z","shell.execute_reply.started":"2023-07-14T18:40:24.328771Z","shell.execute_reply":"2023-07-14T18:40:24.342172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the landmark data and convert it to an image using medipipe library.\n# This function extracts the data for both hands.\n\nmp_pose = mp.solutions.pose\nmp_hands = mp.solutions.hands\nmp_drawing = mp.solutions.drawing_utils \nmp_drawing_styles = mp.solutions.drawing_styles\n\ndef get_hands(seq_df):\n images = []\n all_hand_landmarks = []\n for seq_idx in range(len(seq_df)):\n img, hand_land = list(),list()\n for side in ['right','left']:\n x_hand = seq_df.iloc[seq_idx].filter(regex=f\"x_{side}_hand.*\").values\n y_hand = seq_df.iloc[seq_idx].filter(regex=f\"y_{side}_hand.*\").values\n z_hand = seq_df.iloc[seq_idx].filter(regex=f\"z_{side}_hand.*\").values\n hand_image = f'{side}_hand_image'\n hand_image = np.zeros((600, 600, 3))\n landmarks = f'{side}_hand_landmarks'\n landmarks = landmark_pb2.NormalizedLandmarkList()\n\n for x, y, z in zip(x_hand, y_hand, z_hand):\n landmarks.landmark.add(x=x, y=y, z=z)\n\n mp_drawing.draw_landmarks(\n hand_image,\n landmarks,\n mp_hands.HAND_CONNECTIONS,\n landmark_drawing_spec=mp_drawing_styles.get_default_hand_landmarks_style())\n img.append(hand_image.astype(np.uint8))\n hand_land.append(landmarks)\n images.append(img)\n all_hand_landmarks.append(hand_land)\n\n return images, all_hand_landmarks","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:24.345094Z","iopub.execute_input":"2023-07-14T18:40:24.345671Z","iopub.status.idle":"2023-07-14T18:40:24.364399Z","shell.execute_reply.started":"2023-07-14T18:40:24.345639Z","shell.execute_reply":"2023-07-14T18:40:24.363285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the images created using mediapipe apis\nhand_images, hand_landmarks = get_hands(sequence)\n# Fetch and show the data for right hand\ncreate_animation(np.array(hand_images)[:, 0])","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:24.366587Z","iopub.execute_input":"2023-07-14T18:40:24.367279Z","iopub.status.idle":"2023-07-14T18:40:34.035525Z","shell.execute_reply.started":"2023-07-14T18:40:24.367245Z","shell.execute_reply":"2023-07-14T18:40:34.03306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## points to consider\n- Up untill now we were taking file path and phrase from training data and then get the landmarks data using file path.\n- We will gather data such that we don't have to juggle between these dataframes and have a single dataframe only.\n- We will store the data in TFRecord format becuase it stores the data efficiently. More on this [here](https://www.tensorflow.org/tutorials/load_data/tfrecord)\n\n- Also This is only fingerspelling challenge and it focuses on hand movements. So what we need is to focus on those corresponding landmarks only. Refer below. [Source - Mediapipe]","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Pose coordinates for hand movement.\nLPOSE = [13, 15, 17, 19, 21]\nRPOSE = [14, 16, 18, 20, 22]\nPOSE = LPOSE + RPOSE","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:34.036973Z","iopub.execute_input":"2023-07-14T18:40:34.03797Z","iopub.status.idle":"2023-07-14T18:40:34.04297Z","shell.execute_reply.started":"2023-07-14T18:40:34.037933Z","shell.execute_reply":"2023-07-14T18:40:34.042223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = [f'x_right_hand_{i}' for i in range(21)] + [f'x_left_hand_{i}' for i in range(21)] + [f'x_pose_{i}' for i in POSE] #20+20+12 = 52\nY = [f'y_right_hand_{i}' for i in range(21)] + [f'y_left_hand_{i}' for i in range(21)] + [f'y_pose_{i}' for i in POSE] #20+20+12 = 52\nZ = [f'z_right_hand_{i}' for i in range(21)] + [f'z_left_hand_{i}' for i in range(21)] + [f'z_pose_{i}' for i in POSE] #20+20+12 = 52\n\nFEATURE_COLUMNS = X + Y + Z","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:34.044353Z","iopub.execute_input":"2023-07-14T18:40:34.044883Z","iopub.status.idle":"2023-07-14T18:40:34.056451Z","shell.execute_reply.started":"2023-07-14T18:40:34.044848Z","shell.execute_reply":"2023-07-14T18:40:34.055466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"x_\" in col]\nY_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"y_\" in col]\nZ_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"z_\" in col]\n\nRHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"right\" in col]\nLHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"left\" in col]\nRPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in RPOSE]\nLPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in LPOSE]","metadata":{"execution":{"iopub.status.busy":"2023-07-14T18:40:34.057938Z","iopub.execute_input":"2023-07-14T18:40:34.058458Z","iopub.status.idle":"2023-07-14T18:40:34.071958Z","shell.execute_reply.started":"2023-07-14T18:40:34.058423Z","shell.execute_reply":"2023-07-14T18:40:34.070928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocess and write the dataset as TFRecords\nUsing the extracted landmarks and phrases let us create new dataset files and write them as TFRecords.","metadata":{}},{"cell_type":"code","source":"from tqdm.notebook import tqdm\nimport os\nimport shutil\nimport tensorflow as tf\n# Set length of frames to 128\nFRAME_LEN = 128\n\n# Create directory to store the new data\nif not os.path.isdir(\"preprocessed\"):\n os.mkdir(\"preprocessed\")\nelse:\n shutil.rmtree(\"preprocessed\")\n os.mkdir(\"preprocessed\")\n\n# Loop through each file_id\nfor file_id in tqdm(train_metadata_df.file_id.unique()):\n # Parquet file name\n pq_file = f\"{BASE_DIR}/train_landmarks/{file_id}.parquet\"\n # Filter train.csv and fetch entries only for the relevant file_id\n file_df = train_metadata_df.loc[train_metadata_df[\"file_id\"] == file_id]\n # Fetch the parquet file\n parquet_df = pd.read_parquet(f\"{BASE_DIR}train_landmarks/{str(file_id)}.parquet\",\n columns=['sequence_id'] + FEATURE_COLUMNS)\n # File name for the updated data\n tf_file = f\"preprocessed/{file_id}.tfrecord\"\n parquet_numpy = parquet_df.to_numpy()\n # Initialize the pointer to write the output of \n # each `for loop` below as a sequence into the file.\n with tf.io.TFRecordWriter(tf_file) as file_writer:\n # Loop through each sequence in file.\n for seq_id, phrase in zip(file_df.sequence_id, file_df.phrase):\n # Fetch sequence data\n frames = parquet_numpy[parquet_df.index == seq_id]\n \n # Calculate the number of NaN values in each hand landmark\n r_nonan = np.sum(np.sum(np.isnan(frames[:, RHAND_IDX]), axis = 1) == 0)\n l_nonan = np.sum(np.sum(np.isnan(frames[:, LHAND_IDX]), axis = 1) == 0)\n no_nan = max(r_nonan, l_nonan)\n \n if 2*len(phrase)References
\n1. [📊 [EDA] ASL - Fingerspelling 2D & 3D Plot 📊](https://www.kaggle.com/code/drmwnnrafi/eda-asl-fingerspelling-2d-3d-plot)\n2. [[EDA] 🤟ASLFR🖖 - Animated visualization📊🧍](https://www.kaggle.com/code/leonidkulyk/eda-aslfr-animated-visualization/notebook)\n3. [ASL Fingerspelling Recognition w/ TensorFlow](https://www.kaggle.com/code/gusthema/asl-fingerspelling-recognition-w-tensorflow/notebook#Google---American-Sign-Language-Fingerspelling-Recognition-with-TensorFlow)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}