{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":52950,"databundleVersionId":5973250,"sourceType":"competition"},{"sourceId":7111805,"sourceType":"datasetVersion","datasetId":4100768}],"dockerImageVersionId":30787,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Translating Silence: Transformer-Powered American Sign Language Fingerspelling Recognition\n\n\n","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"ASL\n
","metadata":{}},{"cell_type":"markdown","source":"# **1. Introduction**\n\nEffective communication is the cornerstone of human interaction. In an effort to enhance accessibility and inclusivity, this project focuses on the development of a robust system for detecting and translating ** American Sign Language (ASL) fingerspelling into text**. The dataset used in this project consists of more than **three million** fingerspelled characters, generously contributed by over 100 Deaf signers. Captured through smartphone selfie cameras, the dataset offers a diverse range of backgrounds and lighting conditions, adding real-world complexity to the academic exploration.\n","metadata":{}},{"cell_type":"markdown","source":"## **1.1. Dataset Overview**\n\nThe dataset is a valuable resource for academic research and experimentation:\n\n- **Metadata File [train/supplemental_metadata.csv]:**\n - **path:** The path to the landmark file.\n - **file_id:** A unique identifier for the data file.\n - **participant_id:** A unique identifier for the data contributor.\n - **sequence_id:** A unique identifier for the landmark sequence.\n - **phrase:** Labels for the landmark sequence, randomly generated for privacy and ethical considerations.\n\n\n- **Landmark Data [train/supplemental_landmarks/]:**\n - Extracted from raw videos using the MediaPipe holistic model.\n - Reshaped into a wide format for efficiency.\n - Contains spatial coordinates (x, y, z) for 543 landmarks, including face, left hand, pose, and right hand.\n - Approximately 1,000 sequences with sequence ID as the dataframe index.","metadata":{}},{"cell_type":"markdown","source":"Landmark data is a crucial aspect of this project, providing detailed spatial information for various key points, such as face, left hand, pose, and right hand. The figure below illustrates the concept of landmark data:\n\n\n
\n \n \"LANDMARK\n
\n\nIn the landmark data, spatial coordinates (x, y, z) are provided for each of the 543 landmarks. It plays a pivotal role in understanding the intricate movements and positions associated with ASL fingerspelling.","metadata":{}},{"cell_type":"markdown","source":"## **1.2. Project Objective**\n\n* **Objective** : Create a transformer-based ASL Language to text application \n\n*Note: The dataset has been anonymized, and privacy measures have been implemented to ensure ethical use within an academic framework.*","metadata":{}},{"cell_type":"markdown","source":"# **2. Setup & Dependencies**\n\nBefore diving into the implementation, let's ensure the notebook environment is properly set up. ","metadata":{}},{"cell_type":"markdown","source":"## **2.1. Mediapipe**\n\n[**Mediapipe**](https://developers.google.com/mediapipe) is an open-source library developed by Google that offers a comprehensive set of pre-built solutions for various computer vision tasks. It simplifies the implementation of complex pipelines for tasks such as hand tracking, pose estimation, face detection, and more.","metadata":{}},{"cell_type":"code","source":"!pip install mediapipe","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:13.332469Z","iopub.execute_input":"2023-12-21T21:47:13.33315Z","iopub.status.idle":"2023-12-21T21:47:29.834982Z","shell.execute_reply.started":"2023-12-21T21:47:13.333117Z","shell.execute_reply":"2023-12-21T21:47:29.833699Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **2.2. Libraries** ","metadata":{}},{"cell_type":"code","source":"import os\nimport shutil\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq\nimport tensorflow as tf\nimport json\nimport mediapipe\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport random\n\nfrom skimage.transform import resize\nfrom mediapipe.framework.formats import landmark_pb2\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tqdm.notebook import tqdm\nfrom matplotlib import animation, rc","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:29.837609Z","iopub.execute_input":"2023-12-21T21:47:29.838402Z","iopub.status.idle":"2023-12-21T21:47:43.844879Z","shell.execute_reply.started":"2023-12-21T21:47:29.838362Z","shell.execute_reply":"2023-12-21T21:47:43.843843Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **2.3. Versions**","metadata":{}},{"cell_type":"code","source":"print(\"TensorFlow v\" + tf.__version__)\nprint(\"Mediapipe v\" + mediapipe.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:43.846222Z","iopub.execute_input":"2023-12-21T21:47:43.846793Z","iopub.status.idle":"2023-12-21T21:47:43.852562Z","shell.execute_reply.started":"2023-12-21T21:47:43.846763Z","shell.execute_reply":"2023-12-21T21:47:43.851458Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **3. Data Loading & Exploration**","metadata":{}},{"cell_type":"markdown","source":"## **3.1. Loading the data**","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/asl-fingerspelling/train.csv')\nprint(\"Shape of Train Set is :\" , df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:43.854134Z","iopub.execute_input":"2023-12-21T21:47:43.854727Z","iopub.status.idle":"2023-12-21T21:47:44.048364Z","shell.execute_reply.started":"2023-12-21T21:47:43.854692Z","shell.execute_reply":"2023-12-21T21:47:44.047311Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:44.051217Z","iopub.execute_input":"2023-12-21T21:47:44.051539Z","iopub.status.idle":"2023-12-21T21:47:44.073788Z","shell.execute_reply.started":"2023-12-21T21:47:44.051511Z","shell.execute_reply":"2023-12-21T21:47:44.072929Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here's a breakdown of the example data:\n\n* `path`: The path to the Parquet file containing landmarks data for a specific sequence.\n\n* `file_id`: A unique identifier for the data file.\n\n* `sequence_id`: A unique identifier for the landmark sequence.\n\n* `participant_id`: A unique identifier for the data contributor.\n\n* `phrase`: The labels for the landmark sequence, which may represent phrases or words associated with the sign language gestures.\n\n\nA **Parquet** file is a columnar storage file format optimized for use with big data processing frameworks.","metadata":{}},{"cell_type":"markdown","source":"Each entry in **train.csv** contains a `phrase`, its `sequence_id`, `path` and `file_id`. The `file_id` indicates the file that holds the **_landmarks_** data for that particular `phrase` and `sequence_id` is the unique index of the landmark sequence within the landmarks data file.\n\nThe following diagram shows an example of how files are connected: \n\n![ASL Data](https://storage.googleapis.com/kagglesdsdata/datasets/4100768/7111805/asl_data.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=databundle-worker-v2%40kaggle-161607.iam.gserviceaccount.com%2F20231220%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20231220T163433Z&X-Goog-Expires=345600&X-Goog-SignedHeaders=host&X-Goog-Signature=38921595ecd416daedb2d58a94ef3e752de193ea3564e5d893b7090fe9f41b755b6b71a5baf2cf921f4ab6e2d91f3027674f1733514c1e8b066d79de0b24c45858f2fd60f70bac57db2fe63f68d843bfc12291bbfd4d3513b068ec94acb91163b158ea93719c02f5a14c1476292b745af5e2f9df2df214ff0ec67d5eca251c414dc8c7377476684d21d3d45b6cd5ed4d2416cd1e3b124f1395081b9ee11f718aebb655c782f38aeff915af44458eca71fb3151add8273436999aa86462bfd9e4d0c4429272eacb28466e58736602bc7d8f7cfb8c7ae9ed18371814d1c70bfb4ba75ff3c53cbc502ea49bb611438eefe030783670e76b05f6f3f667722c0228c3)\n","metadata":{}},{"cell_type":"markdown","source":"## **3.2. Exploring a sample of the Dataset**\n\nAs an example, let us examine the landmarks file for the first row of`train.csv` :","metadata":{}},{"cell_type":"code","source":"sequence_id, file_id, phrase = df.iloc[177][['sequence_id', 'file_id', 'phrase']]\nprint(f\"sequence_id: {sequence_id}, file_id: {file_id}, phrase: {phrase}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:44.074993Z","iopub.execute_input":"2023-12-21T21:47:44.075311Z","iopub.status.idle":"2023-12-21T21:47:44.089483Z","shell.execute_reply.started":"2023-12-21T21:47:44.075284Z","shell.execute_reply":"2023-12-21T21:47:44.088512Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, let us open the parquet file and fetch the data for the particular `sequence_id` :","metadata":{}},{"cell_type":"code","source":"sample = pq.read_table(f\"/kaggle/input/asl-fingerspelling/train_landmarks/{str(file_id)}.parquet\",\n filters=[[('sequence_id', '=', sequence_id)],]).to_pandas()\n\nprint(\"Full sequence dataset shape is :\", sample.shape)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:44.090675Z","iopub.execute_input":"2023-12-21T21:47:44.091057Z","iopub.status.idle":"2023-12-21T21:47:47.565632Z","shell.execute_reply.started":"2023-12-21T21:47:44.091031Z","shell.execute_reply":"2023-12-21T21:47:47.564634Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This particular phrase(***3 creekhouse***) contains 123 entries (or frames) and 1630 columns including 1628 landmark coordinates. Let us print the first 5 entries.","metadata":{}},{"cell_type":"code","source":"sample.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:47.566938Z","iopub.execute_input":"2023-12-21T21:47:47.56722Z","iopub.status.idle":"2023-12-21T21:47:47.595119Z","shell.execute_reply.started":"2023-12-21T21:47:47.567196Z","shell.execute_reply":"2023-12-21T21:47:47.594171Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from ipywidgets import interact\n\n@interact\ndef show_columns(column_start=(0, len(sample.columns), 100), column_end=(100, len(sample.columns), 100)):\n print(sample.columns[column_start:column_end])\n \n\n# Frame 1 \n# Face 0-467 (468) x3\n# Left Hand 0-20 (21) x3\n# Pose 0-32 (33) x3\n# Right Hand 0-20 (21) x3","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:47.596521Z","iopub.execute_input":"2023-12-21T21:47:47.59729Z","iopub.status.idle":"2023-12-21T21:47:47.630651Z","shell.execute_reply.started":"2023-12-21T21:47:47.597252Z","shell.execute_reply":"2023-12-21T21:47:47.629677Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **3.3. Data Visualisation using Mediapipe**","metadata":{}},{"cell_type":"markdown","source":"Let us visualize the hand landmarks data for the phrase ***3 creek house*** using the [Hand landmarker APIs](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) of the [Mediapipe Library](https://developers.google.com/mediapipe).\n\nHand landmarks represent the key points on a human hand.","metadata":{}},{"cell_type":"code","source":"# Function create animation from images.\n\nmatplotlib.rcParams['animation.embed_limit'] = 2**128 #Enables larger embedded animation \nmatplotlib.rcParams['savefig.pad_inches'] = 0 #No extra padding added\nrc('animation', html='jshtml') #Animation through jupyter notebook \n\ndef create_animation(images):\n fig = plt.figure(figsize=(6, 9))\n ax = plt.Axes(fig, [0., 0., 1., 1.])\n ax.set_axis_off()\n fig.add_axes(ax)\n im=ax.imshow(images[0], cmap=\"gray\")\n plt.close(fig)\n \n #This function updates the image displayed on the axes for each animation frame. It is used by animation.FuncAnimation to create the animation.\n def animate_func(i):\n im.set_array(images[i])\n return [im]\n \n #The FuncAnimation class is used to create the animation. It takes the figure (fig), the animation function (animate_func), the number of frames (len(images)), and the interval between frames (1000/10 milliseconds, or 10 frames per second).\n return animation.FuncAnimation(fig, animate_func, frames=len(images), interval=1000/10)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:47.63211Z","iopub.execute_input":"2023-12-21T21:47:47.632679Z","iopub.status.idle":"2023-12-21T21:47:47.643536Z","shell.execute_reply.started":"2023-12-21T21:47:47.632644Z","shell.execute_reply":"2023-12-21T21:47:47.642605Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This image illustrates the 21 keypoints on the hand : \n\n\n
\n \n \"LANDMARK\"\n
","metadata":{}},{"cell_type":"code","source":"# Extract the landmark data and convert it to an image using medipipe library.\n# This function extracts the data for both hands.\n\n#Are used for hand landmark detection\nmp_pose = mediapipe.solutions.pose\nmp_hands = mediapipe.solutions.hands\n\n#Are used for visualizing landmarks on images\nmp_drawing = mediapipe.solutions.drawing_utils \nmp_drawing_styles = mediapipe.solutions.drawing_styles\n\ndef get_hands(seq_df):\n images = []\n all_hand_landmarks = []\n for seq_idx in range(len(seq_df)):\n \n # Extract x, y, z coordinates for the right hand\n x_hand = seq_df.iloc[seq_idx].filter(regex=\"x_right_hand.*\").values\n y_hand = seq_df.iloc[seq_idx].filter(regex=\"y_right_hand.*\").values\n z_hand = seq_df.iloc[seq_idx].filter(regex=\"z_right_hand.*\").values\n\n # Create an empty image for the right hand\n right_hand_image = np.zeros((600, 600, 3))\n\n # Create NormalizedLandmarkList for the right hand\n right_hand_landmarks = landmark_pb2.NormalizedLandmarkList()\n \n # Populate right_hand_landmarks with x, y, z coordinates\n for x, y, z in zip(x_hand, y_hand, z_hand):\n right_hand_landmarks.landmark.add(x=x, y=y, z=z)\n\n # Draw hand landmarks on the right_hand_image\n mp_drawing.draw_landmarks(\n right_hand_image,\n right_hand_landmarks,\n mp_hands.HAND_CONNECTIONS,\n landmark_drawing_spec=mp_drawing_styles.get_default_hand_landmarks_style())\n \n x_hand = seq_df.iloc[seq_idx].filter(regex=\"x_left_hand.*\").values\n y_hand = seq_df.iloc[seq_idx].filter(regex=\"y_left_hand.*\").values\n z_hand = seq_df.iloc[seq_idx].filter(regex=\"z_left_hand.*\").values\n \n left_hand_image = np.zeros((600, 600, 3))\n \n left_hand_landmarks = landmark_pb2.NormalizedLandmarkList()\n for x, y, z in zip(x_hand, y_hand, z_hand):\n left_hand_landmarks.landmark.add(x=x, y=y, z=z)\n\n mp_drawing.draw_landmarks(\n left_hand_image,\n left_hand_landmarks,\n mp_hands.HAND_CONNECTIONS,\n landmark_drawing_spec=mp_drawing_styles.get_default_hand_landmarks_style())\n \n # Append right and left hand images to the images list\n images.append([right_hand_image.astype(np.uint8), left_hand_image.astype(np.uint8)])\n \n # Append right and left hand landmarks to the all_hand_landmarks list\n all_hand_landmarks.append([right_hand_landmarks, left_hand_landmarks])\n \n return images, all_hand_landmarks","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:47.645256Z","iopub.execute_input":"2023-12-21T21:47:47.645614Z","iopub.status.idle":"2023-12-21T21:47:47.661429Z","shell.execute_reply.started":"2023-12-21T21:47:47.645581Z","shell.execute_reply":"2023-12-21T21:47:47.660403Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get the images created using mediapipe apis\nhand_images, hand_landmarks = get_hands(sample)\n\n# Fetch and show the data for right hand\ncreate_animation(np.array(hand_images)[:, 0])","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:47:47.662895Z","iopub.execute_input":"2023-12-21T21:47:47.663246Z","iopub.status.idle":"2023-12-21T21:48:31.81901Z","shell.execute_reply.started":"2023-12-21T21:47:47.663214Z","shell.execute_reply":"2023-12-21T21:48:31.817608Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **4. Data Preprocessing**\n\nFor convenience and efficiency, we will rearrange the data so that each parquet file contains the landmark data along with the phrase it represents. This way we don't have to switch between train.csv and its parquet file. \n\nWe will save the new data in the [TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) format. `TFRecord` format is a simple format for storing a sequence of binary records. Storing and loading the data using `TFRecord` is much more efficient and faster.\n","metadata":{}},{"cell_type":"markdown","source":"## **4.1. Pose Landmark Coordinates**\n\nThis image illustrates the pose coordinates related to the whole body movement.\n\n\n
\n \n \"POSE\"\n
\n\nCoordinates from **13** to **22** represents the hand movement as follows :\n* 13 - left elbow\n* 14 - right elbow\n* 15 - left wrist\n* 16 - right wrist\n* 17 - left pinky\n* 18 - right pinky\n* 19 - left index\n* 20 - right index\n* 21 - left thumb\n* 22 - right thumb","metadata":{}},{"cell_type":"code","source":"# Pose coordinates for hand movement.\nLP = [13, 15, 17, 19, 21]\nRP = [14, 16, 18, 20, 22]\nPOSE = LP + RP","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:48:31.820765Z","iopub.execute_input":"2023-12-21T21:48:31.821294Z","iopub.status.idle":"2023-12-21T21:48:31.82655Z","shell.execute_reply.started":"2023-12-21T21:48:31.821246Z","shell.execute_reply":"2023-12-21T21:48:31.825534Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Create x,y,z label names from coordinates : ","metadata":{}},{"cell_type":"code","source":"X = [f'x_right_hand_{i}' for i in range(21)] + [f'x_left_hand_{i}' for i in range(21)] + [f'x_pose_{i}' for i in POSE]\nY = [f'y_right_hand_{i}' for i in range(21)] + [f'y_left_hand_{i}' for i in range(21)] + [f'y_pose_{i}' for i in POSE]\nZ = [f'z_right_hand_{i}' for i in range(21)] + [f'z_left_hand_{i}' for i in range(21)] + [f'z_pose_{i}' for i in POSE]","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:48:31.831242Z","iopub.execute_input":"2023-12-21T21:48:31.831554Z","iopub.status.idle":"2023-12-21T21:48:31.841915Z","shell.execute_reply.started":"2023-12-21T21:48:31.831528Z","shell.execute_reply":"2023-12-21T21:48:31.840862Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Create feature columns from the extracted coordinates :","metadata":{}},{"cell_type":"code","source":"FEATURE_COLUMNS = X + Y + Z\nlen(FEATURE_COLUMNS)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:48:31.843072Z","iopub.execute_input":"2023-12-21T21:48:31.843385Z","iopub.status.idle":"2023-12-21T21:48:31.859328Z","shell.execute_reply.started":"2023-12-21T21:48:31.843357Z","shell.execute_reply":"2023-12-21T21:48:31.858225Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Store ids of each coordinate labels to lists : ","metadata":{}},{"cell_type":"code","source":"X_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"x_\" in col]\nY_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"y_\" in col]\nZ_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"z_\" in col]\n\nRHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"right\" in col]\nLHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"left\" in col]\nRPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in RP]\nLPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in LP]\n\nlen(LPOSE_IDX)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:48:31.860464Z","iopub.execute_input":"2023-12-21T21:48:31.860755Z","iopub.status.idle":"2023-12-21T21:48:31.873656Z","shell.execute_reply.started":"2023-12-21T21:48:31.860731Z","shell.execute_reply":"2023-12-21T21:48:31.872651Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"In summary, this code snippet provides an example of how to extract information from the dataset, construct a file path based on that information, read a sample sequence dataset from a Parquet file, and display information about the sample dataset. This can be useful for exploring the structure and content of the data during the preprocessing stage.","metadata":{}},{"cell_type":"markdown","source":"## **4.2. TFRecord Preprocessing**","metadata":{}},{"cell_type":"markdown","source":"Using the extracted landmarks and phrases let us create new dataset files and write them as TFRecords.\n\n**Motivation for Using TFRecords:**\n \n* **Efficiency**: TFRecords are a binary format that can be read efficiently by TensorFlow. This format reduces I/O overhead and allows for faster data loading during training.\n \n* **Serialization**: TFRecords enable easy serialization of complex data structures, making it convenient to store and retrieve data, especially when dealing with multiple data types.","metadata":{}},{"cell_type":"code","source":"# Set length of frames to 128\nFRAME_LEN = 128\n\n# Create directory to store the new data\nif not os.path.isdir(\"preprocessed\"):\n os.mkdir(\"preprocessed\")\nelse:\n shutil.rmtree(\"preprocessed\")\n os.mkdir(\"preprocessed\")\n\n# Loop through each file_id\nfor file_id in tqdm(df.file_id.unique()):\n # Parquet file name\n pq_file = f\"/kaggle/input/asl-fingerspelling/train_landmarks/{file_id}.parquet\"\n \n # Filter train.csv and fetch entries only for the relevant file_id\n file_df = df.loc[df[\"file_id\"] == file_id]\n \n # Fetch the parquet file\n parquet_df = pq.read_table(f\"/kaggle/input/asl-fingerspelling/train_landmarks/{str(file_id)}.parquet\",\n columns=['sequence_id'] + FEATURE_COLUMNS).to_pandas()\n \n # File name for the updated data\n tf_file = f\"preprocessed/{file_id}.tfrecord\"\n parquet_numpy = parquet_df.to_numpy()\n \n # Initialize the pointer to write the output of \n # each `for loop` below as a sequence into the file.\n with tf.io.TFRecordWriter(tf_file) as file_writer:\n # Loop through each sequence in file.\n for seq_id, phrase in zip(file_df.sequence_id, file_df.phrase):\n # Fetch sequence data\n frames = parquet_numpy[parquet_df.index == seq_id]\n \n # Calculate the number of NaN values in each hand landmark\n r_nonan = np.sum(np.sum(np.isnan(frames[:, RHAND_IDX]), axis = 1) == 0)\n l_nonan = np.sum(np.sum(np.isnan(frames[:, LHAND_IDX]), axis = 1) == 0)\n no_nan = max(r_nonan, l_nonan)\n \n if 2*len(phrase)\" to mark the start and end of each phrase, and \"P\" for padding.","metadata":{}},{"cell_type":"code","source":"with open (\"/kaggle/input/asl-fingerspelling/character_to_prediction_index.json\", \"r\") as f:\n char_to_num = json.load(f)\n\n# Add pad_token, start pointer and end pointer to the dict\npad_token = 'P'\nstart_token = '<'\nend_token = '>'\npad_token_idx = 59\nstart_token_idx = 60\nend_token_idx = 61\n\nchar_to_num[pad_token] = pad_token_idx\nchar_to_num[start_token] = start_token_idx\nchar_to_num[end_token] = end_token_idx\nnum_to_char = {j:i for i,j in char_to_num.items()}","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:44.561678Z","iopub.execute_input":"2023-12-21T21:59:44.561987Z","iopub.status.idle":"2023-12-21T21:59:44.573948Z","shell.execute_reply.started":"2023-12-21T21:59:44.561961Z","shell.execute_reply":"2023-12-21T21:59:44.573073Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"=> **Purpose**: Reads a JSON file containing a character-to-index mapping and adds special tokens for padding, start, and end of phrases to the character mapping.","metadata":{}},{"cell_type":"code","source":"# Function to resize and add padding.\ndef resize_pad(x):\n # Check if the length of x is less than FRAME_LEN\n if tf.shape(x)[0] < FRAME_LEN:\n # Pad the tensor with zeros to make its length FRAME_LEN\n x = tf.pad(x, ([[0, FRAME_LEN-tf.shape(x)[0]], [0, 0], [0, 0]]))\n else:\n # Resize the tensor to have length FRAME_LEN\n x = tf.image.resize(x, (FRAME_LEN, tf.shape(x)[1]))\n return x\n\n# Detect the dominant hand from the number of NaN values.\n# Dominant hand will have fewer NaN values since it is in frame moving.\ndef pre_process(x):\n # Extract hand and pose coordinates for both right and left hands\n rhand = tf.gather(x, RHAND_IDX, axis=1)\n lhand = tf.gather(x, LHAND_IDX, axis=1)\n rpose = tf.gather(x, RPOSE_IDX, axis=1)\n lpose = tf.gather(x, LPOSE_IDX, axis=1)\n \n # Identify NaN values in each hand\n rnan_idx = tf.reduce_any(tf.math.is_nan(rhand), axis=1)\n lnan_idx = tf.reduce_any(tf.math.is_nan(lhand), axis=1)\n \n # Count the number of NaN values\n rnans = tf.math.count_nonzero(rnan_idx)\n lnans = tf.math.count_nonzero(lnan_idx)\n \n # Determine the dominant hand based on NaN values\n if rnans > lnans:\n hand = lhand\n pose = lpose\n \n # Flip hand coordinates for left hand\n hand_x = hand[:, 0*(len(LHAND_IDX)//3) : 1*(len(LHAND_IDX)//3)]\n hand_y = hand[:, 1*(len(LHAND_IDX)//3) : 2*(len(LHAND_IDX)//3)]\n hand_z = hand[:, 2*(len(LHAND_IDX)//3) : 3*(len(LHAND_IDX)//3)]\n hand = tf.concat([1-hand_x, hand_y, hand_z], axis=1)\n \n pose_x = pose[:, 0*(len(LPOSE_IDX)//3) : 1*(len(LPOSE_IDX)//3)]\n pose_y = pose[:, 1*(len(LPOSE_IDX)//3) : 2*(len(LPOSE_IDX)//3)]\n pose_z = pose[:, 2*(len(LPOSE_IDX)//3) : 3*(len(LPOSE_IDX)//3)]\n pose = tf.concat([1-pose_x, pose_y, pose_z], axis=1)\n else:\n hand = rhand\n pose = rpose\n \n # Extract coordinates for the dominant hand\n hand_x = hand[:, 0*(len(LHAND_IDX)//3) : 1*(len(LHAND_IDX)//3)]\n hand_y = hand[:, 1*(len(LHAND_IDX)//3) : 2*(len(LHAND_IDX)//3)]\n hand_z = hand[:, 2*(len(LHAND_IDX)//3) : 3*(len(LHAND_IDX)//3)]\n hand = tf.concat([hand_x[..., tf.newaxis], hand_y[..., tf.newaxis], hand_z[..., tf.newaxis]], axis=-1)\n \n # Standardize the hand coordinates\n mean = tf.math.reduce_mean(hand, axis=1)[:, tf.newaxis, :]\n std = tf.math.reduce_std(hand, axis=1)[:, tf.newaxis, :]\n hand = (hand - mean) / std\n\n # Extract coordinates for the dominant hand's pose\n pose_x = pose[:, 0*(len(LPOSE_IDX)//3) : 1*(len(LPOSE_IDX)//3)]\n pose_y = pose[:, 1*(len(LPOSE_IDX)//3) : 2*(len(LPOSE_IDX)//3)]\n pose_z = pose[:, 2*(len(LPOSE_IDX)//3) : 3*(len(LPOSE_IDX)//3)]\n pose = tf.concat([pose_x[..., tf.newaxis], pose_y[..., tf.newaxis], pose_z[..., tf.newaxis]], axis=-1)\n \n # Concatenate standardized hand and pose coordinates\n x = tf.concat([hand, pose], axis=1)\n \n # Resize and pad the tensor\n x = resize_pad(x)\n \n # Replace NaN values with zeros\n x = tf.where(tf.math.is_nan(x), tf.zeros_like(x), x)\n \n # Reshape the tensor to final dimensions\n x = tf.reshape(x, (FRAME_LEN, len(LHAND_IDX) + len(LPOSE_IDX)))\n \n return x ","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:44.575256Z","iopub.execute_input":"2023-12-21T21:59:44.575632Z","iopub.status.idle":"2023-12-21T21:59:44.599457Z","shell.execute_reply.started":"2023-12-21T21:59:44.575599Z","shell.execute_reply":"2023-12-21T21:59:44.598587Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Why It's Done in the Context of the Project ?**\n\n* **Normalization**: Normalizing coordinates helps in achieving consistent scales, making it easier for the model to learn patterns effectively.\n* **Handling Missing Values**: NaN values are handled appropriately to avoid introducing noise into the model.\n* **Resizing and Padding**: Ensures that all input sequences have a consistent length, which is crucial for model training.\n* **Special Tokens**: The introduction of special tokens enhances the model's understanding of the beginning, end, and padding of phrases during training.\n\nIn summary, this preprocessing code ensures that the input data is in a suitable format for training the ASL fingerspelling model. It addresses issues such as inconsistent scales, missing values, and sequence length variations, contributing to the overall robustness and effectiveness of the machine learning model.\n \nChara","metadata":{}},{"cell_type":"markdown","source":"**C. Create function to parse data from TFRecord format**","metadata":{}},{"cell_type":"code","source":"def decode_fn(record_bytes):\n # Define the schema for decoding TFRecord data\n schema = {COL: tf.io.VarLenFeature(dtype=tf.float32) for COL in FEATURE_COLUMNS}\n schema[\"phrase\"] = tf.io.FixedLenFeature([], dtype=tf.string)\n \n # Parse a single example from the TFRecord\n features = tf.io.parse_single_example(record_bytes, schema)\n \n # Extract the phrase information\n phrase = features[\"phrase\"]\n \n # Extract landmarks data for each frame\n landmarks = ([tf.sparse.to_dense(features[COL]) for COL in FEATURE_COLUMNS])\n \n # Transpose to maintain the original shape of landmarks data\n landmarks = tf.transpose(landmarks)\n \n return landmarks, phrase","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:44.600679Z","iopub.execute_input":"2023-12-21T21:59:44.601028Z","iopub.status.idle":"2023-12-21T21:59:44.612709Z","shell.execute_reply.started":"2023-12-21T21:59:44.601003Z","shell.execute_reply":"2023-12-21T21:59:44.611757Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* The purpose of this function is to decode the binary data stored in TFRecord format into tensors that can be used for training the model. It ensures that the data is correctly parsed, and the landmarks and phrases are extracted in a format suitable for further processing in the deep learning model. The decoding function is a crucial step in the data pipeline, allowing the model to consume and understand the input data during training.","metadata":{}},{"cell_type":"markdown","source":"**D. Create function to convert the data**\n\nThis function transposes and applies masks to the landmark coordinates. It also vectorizes the phrase corresponding to the landmarks using `character_to_prediction_index.json`.","metadata":{}},{"cell_type":"code","source":"# Create a StaticHashTable for mapping characters to their corresponding indices\ntable = tf.lookup.StaticHashTable(\n initializer=tf.lookup.KeyValueTensorInitializer(\n keys=list(char_to_num.keys()),\n values=list(char_to_num.values()),\n ),\n default_value=tf.constant(-1),\n name=\"class_weight\"\n)\n\n# Define the convert function\ndef convert_fn(landmarks, phrase):\n # Add start and end pointers to the phrase\n phrase = start_token + phrase + end_token\n # Split the phrase into individual characters and map to their indices using the StaticHashTable\n phrase = tf.strings.bytes_split(phrase)\n phrase = table.lookup(phrase)\n \n # Vectorize the phrase and add padding to match the desired sequence length\n phrase = tf.pad(phrase, paddings=[[0, 64 - tf.shape(phrase)[0]]], mode='CONSTANT', constant_values=pad_token_idx)\n \n # Apply the pre_process function to the landmarks\n return pre_process(landmarks), phrase","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:44.614048Z","iopub.execute_input":"2023-12-21T21:59:44.614696Z","iopub.status.idle":"2023-12-21T21:59:47.575929Z","shell.execute_reply.started":"2023-12-21T21:59:44.61466Z","shell.execute_reply":"2023-12-21T21:59:47.575006Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* In summary, this conversion function prepares the data for training by adding start and end tokens to the phrases, mapping characters to numeric indices, vectorizing, and padding the phrases. It also applies the pre_process function to the landmarks data. This function is often used as part of the data input pipeline during the training process.","metadata":{}},{"cell_type":"markdown","source":"**TFRecords Preprocessing Summary:**\n \n* **TFRecord Creation**: TFRecord files are created for each sequence, storing landmarks and phrases together. This is done to improve data loading efficiency.\n\n* **Data Decoding**: TFRecord data is decoded using the decode_fn function, extracting landmarks and phrases in tensor format.\n\n* **Preprocessing of Landmarks**: Landmarks are pre-processed to handle missing values, normalize data, and ensure consistent shape.\n\n* **Conversion of Data**: The `convert_fn` function is applied to convert landmarks and phrases. The phrase is vectorized using character-to-index mapping, and padding is added.\n\nThese preprocessing steps are essential for preparing the data for training the model. They handle data cleaning, normalization, and formatting, ensuring that the input to the model is in a suitable format for effective learning","metadata":{}},{"cell_type":"markdown","source":"## **4.3. Train and Validation Split**","metadata":{}},{"cell_type":"code","source":"batch_size = 64\ntrain_len = int(0.8 * len(tf_records))\n\n# Training dataset\ntrain_ds = (\n tf.data.TFRecordDataset(tf_records[:train_len]) # Selecting training TFRecord files\n .map(decode_fn) # Decoding TFRecord bytes into tensors (landmarks, phrase)\n .map(convert_fn) # Applying preprocessing to get input and label tensors\n .batch(batch_size) # Grouping samples into batches\n .prefetch(buffer_size=tf.data.AUTOTUNE) # Prefetching data for efficiency\n .cache() # Caching the dataset for faster access during training\n)\n\n# Validation dataset\nvalid_ds = (\n tf.data.TFRecordDataset(tf_records[train_len:]) # Selecting validation TFRecord files\n .map(decode_fn) # Decoding TFRecord bytes into tensors (landmarks, phrase)\n .map(convert_fn) # Applying preprocessing to get input and label tensors\n .batch(batch_size) # Grouping samples into batches\n .prefetch(buffer_size=tf.data.AUTOTUNE) # Prefetching data for efficiency\n .cache() # Caching the dataset for faster access during validation\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:47.57734Z","iopub.execute_input":"2023-12-21T21:59:47.577738Z","iopub.status.idle":"2023-12-21T21:59:49.38887Z","shell.execute_reply.started":"2023-12-21T21:59:47.577703Z","shell.execute_reply":"2023-12-21T21:59:49.387994Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for landmarks, phrase in train_ds.take(1): # Taking one batch for example\n # Example of input tensor\n print(\"Training Input Tensor (Landmarks):\")\n print(landmarks.numpy())\n\n # Example of label tensor\n print(\"\\nTraining Label Tensor (Phrase):\")\n print(phrase.numpy())","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:49.390362Z","iopub.execute_input":"2023-12-21T21:59:49.390744Z","iopub.status.idle":"2023-12-21T21:59:49.859252Z","shell.execute_reply.started":"2023-12-21T21:59:49.390705Z","shell.execute_reply":"2023-12-21T21:59:49.858303Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for landmarks, phrase in train_ds.take(1): # Taking one batch for example\n # Print the shape of the input tensor\n print(\"Training Input Tensor (Landmarks) Shape:\", landmarks.shape)\n\n # Print the shape of the label tensor\n print(\"Training Label Tensor (Phrase) Shape:\", phrase.shape)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:49.860585Z","iopub.execute_input":"2023-12-21T21:59:49.860901Z","iopub.status.idle":"2023-12-21T21:59:50.214047Z","shell.execute_reply.started":"2023-12-21T21:59:49.860876Z","shell.execute_reply":"2023-12-21T21:59:50.213078Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The code snippet is using the take(1) method to extract one batch from the training dataset (train_ds). Then, it prints the shape of the input tensor (landmarks) and the label tensor (phrase) for that batch.\n\n* **Training Input Tensor (Landmarks) Shape: `(64, 128, 78)`**\nThis indicates that the input tensor has a shape of `(batch_size, sequence_length, feature_dimension)`. In this case, we have a batch size of 64, each sequence (or sample) has a length of 128 frames, and there are 78 features (landmark coordinates) per frame.\n\n* **Training Label Tensor (Phrase) Shape: `(64, 64)`**\nThis indicates that the label tensor has a shape of `(batch_size, sequence_length)`. In this case, we have a batch size of 64, and each sequence (or label) has a length of 64. The labels represent the phrases or gestures associated with the corresponding landmarks.\n\n\nSo, in summary, each batch consists of 64 samples. Each sample includes a sequence of 128 frames, where each frame has 78 landmark coordinates. The associated labels are sequences of length 64, representing the phrases or gestures corresponding to the landmark sequences. There is a total of **632** batches. \n\nThese datasets (train_ds and valid_ds) are now ready to be used for training and validation in a machine learning model. The batching, prefetching, and caching steps are common practices to optimize the efficiency of the input pipeline during training.","metadata":{}},{"cell_type":"markdown","source":"# **5. Model Creation - Transformer**\n \nWe will use a **[Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model))** to train a model for this dataset. **Transformers** are designed to process sequential input data. The model we are going to design is similar to the one used in the [Automatic Speech Recognition with Transformer](https://keras.io/examples/audio/transformer_asr/) tutorial for **Keras**. We will finetune only a small part of the model since we can treat the ASL Fingerspelling recognition problem similar to speech recognition. In both cases, we have to predict a sentence from a sequence of data.\n\n\n
\n \n \"TRAN\"\n
","metadata":{}},{"cell_type":"markdown","source":"## **5.1. Embedding & Positioning**","metadata":{}},{"cell_type":"markdown","source":"### **5.1.1. Token Embedding**","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"tokemb\"\n
","metadata":{}},{"cell_type":"code","source":"class TokenEmbedding(layers.Layer):\n def __init__(self, num_vocab=1000, maxlen=100, num_hid=64):\n super().__init__()\n # Token Embedding Layer\n self.emb = tf.keras.layers.Embedding(num_vocab, num_hid)\n # Positional Embedding Layer\n self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=num_hid)\n\n def call(self, x):\n # Get the maximum sequence length dynamically\n maxlen = tf.shape(x)[-1]\n # Token Embedding: Convert token indices to dense vectors\n x = self.emb(x)\n # Generate positional embeddings for each position in the sequence\n positions = tf.range(start=0, limit=maxlen, delta=1)\n positions = self.pos_emb(positions)\n return x + positions","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:50.21556Z","iopub.execute_input":"2023-12-21T21:59:50.216353Z","iopub.status.idle":"2023-12-21T21:59:50.224136Z","shell.execute_reply.started":"2023-12-21T21:59:50.216321Z","shell.execute_reply":"2023-12-21T21:59:50.223133Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The `TokenEmbedding` block consists of two main components: the Token Embedding Layer and the Positional Embedding Layer. It is a fundamental part of transformer architectures, addressing the challenge of processing sequential data.","metadata":{}},{"cell_type":"markdown","source":"### **5.1.2. Landmark Embedding**","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"landemb\"\n
","metadata":{}},{"cell_type":"code","source":"class LandmarkEmbedding(layers.Layer):\n def __init__(self, num_hid=64, maxlen=100):\n super().__init__()\n # Convolutional layers with activation function\n self.conv1 = tf.keras.layers.Conv1D(\n num_hid, 11, strides=2, padding=\"same\", activation=\"relu\"\n )\n self.conv2 = tf.keras.layers.Conv1D(\n num_hid, 11, strides=2, padding=\"same\", activation=\"relu\"\n )\n self.conv3 = tf.keras.layers.Conv1D(\n num_hid, 11, strides=2, padding=\"same\", activation=\"relu\"\n )\n # Positional Embedding Layer\n self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=num_hid)\n\n def call(self, x):\n # Apply convolutional layers\n x = self.conv1(x)\n x = self.conv2(x)\n return self.conv3(x)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:50.225265Z","iopub.execute_input":"2023-12-21T21:59:50.225529Z","iopub.status.idle":"2023-12-21T21:59:50.235928Z","shell.execute_reply.started":"2023-12-21T21:59:50.225505Z","shell.execute_reply":"2023-12-21T21:59:50.235093Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* This block applies a series of 1D convolutions to the input landmark coordinates, creating a processed representation that captures relevant features and relationships in the data.\n* In the context of sequential data, like time series or landmark coordinates, Conv1D layers are suitable because they consider the sequential nature of the data. Conv1D filters move along the sequence, capturing local patterns and dependencies. This is crucial for tasks like gesture recognition or any application where the order of data points matters.","metadata":{}},{"cell_type":"markdown","source":"## **5.2. Encoder & Decoder**","metadata":{}},{"cell_type":"markdown","source":"### **5.2.1. Transformer Encoder**","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"encod\"\n
","metadata":{}},{"cell_type":"code","source":"class TransformerEncoder(layers.Layer):\n def __init__(self, embed_dim, num_heads, feed_forward_dim, rate=0.1):\n super().__init__()\n \n # Multi-Head Self-Attention Layer\n self.att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n \n # Feedforward Neural Network\n self.ffn = keras.Sequential(\n [\n layers.Dense(feed_forward_dim, activation=\"relu\"),\n layers.Dense(embed_dim),\n ]\n )\n \n # Layer Normalization for the Self-Attention Output\n self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)\n \n # Layer Normalization for the Feedforward Output\n self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)\n \n # Dropout Layers for Self-Attention and Feedforward\n self.dropout1 = layers.Dropout(rate)\n self.dropout2 = layers.Dropout(rate)\n\n def call(self, inputs, training):\n # Multi-Head Self-Attention\n attn_output = self.att(inputs, inputs)\n \n # Dropout for Self-Attention\n attn_output = self.dropout1(attn_output, training=training)\n \n # Residual Connection and Layer Normalization for Self-Attention\n out1 = self.layernorm1(inputs + attn_output)\n \n # Feedforward Processing\n ffn_output = self.ffn(out1)\n \n # Dropout for Feedforward\n ffn_output = self.dropout2(ffn_output, training=training)\n \n # Residual Connection and Layer Normalization for Feedforward\n return self.layernorm2(out1 + ffn_output)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:50.237417Z","iopub.execute_input":"2023-12-21T21:59:50.237724Z","iopub.status.idle":"2023-12-21T21:59:50.252427Z","shell.execute_reply.started":"2023-12-21T21:59:50.237698Z","shell.execute_reply":"2023-12-21T21:59:50.251578Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### **Explanation**:\n\n* The TransformerEncoder block is a fundamental building block of the Transformer model, commonly used in natural language processing tasks but applicable to sequential data in general.\n\n* It consists of two main sub-blocks: multi-head self-attention (self.att) and a feedforward neural network (self.ffn).\n\n* **Multi-Head Self-Attention**:\n self.att is a multi-head self-attention layer. It allows the model to weigh different parts of the input sequence differently, capturing dependencies between different elements in the sequence.\n* **Feedforward Neural Network**:\n self.ffn is a feedforward neural network. It processes the information from the attention layer and transforms it to a higher-level representation.\n* **Layer Normalization and Dropout**:\n After each of these sub-blocks, layer normalization (self.layernorm1, self.layernorm2) is applied to normalize the outputs.\n Dropout (self.dropout1, self.dropout2) is used during training to prevent overfitting by randomly setting a fraction of input units to zero.\n* **Call Method**:\n In the call method, the input sequence (inputs) is passed through the self-attention layer, normalized, and then passed through the feedforward network.\n Residual connections (inputs + output) are used, and layer normalization is applied again.","metadata":{}},{"cell_type":"markdown","source":"### **5.2.2. Transformer Decoder**","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"decod\"\n
\n\n\n**Difference between Full-Attention and Causal Attention Mechanisms:**\n\n
\n \n \"Causal\"\n
","metadata":{}},{"cell_type":"code","source":"class TransformerDecoder(layers.Layer):\n def __init__(self, embed_dim, num_heads, feed_forward_dim, dropout_rate=0.1):\n super().__init__()\n\n # Layer normalization for self-attention\n self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)\n # Layer normalization for encoder attention\n self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)\n # Layer normalization for feed-forward network\n self.layernorm3 = layers.LayerNormalization(epsilon=1e-6)\n\n # Self-Attention Layer for Decoding\n self.self_att = layers.MultiHeadAttention(\n num_heads=num_heads, key_dim=embed_dim\n )\n # Encoder Attention Layer for Decoding\n self.enc_att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n\n # Dropout layers for self-attention, encoder attention, and feed-forward network\n self.self_dropout = layers.Dropout(0.5)\n self.enc_dropout = layers.Dropout(0.1)\n self.ffn_dropout = layers.Dropout(0.1)\n\n # Feed-Forward Network\n self.ffn = keras.Sequential(\n [\n layers.Dense(feed_forward_dim, activation=\"relu\"),\n layers.Dense(embed_dim),\n ]\n )\n\n def causal_attention_mask(self, batch_size, n_dest, n_src, dtype):\n \"\"\"Masks the upper half of the dot product matrix in self attention.\n\n This prevents the flow of information from future tokens to the current token.\n 1's in the lower triangle, counting from the lower right corner.\n \"\"\"\n i = tf.range(n_dest)[:, None]\n j = tf.range(n_src)\n m = i >= j - n_src + n_dest\n mask = tf.cast(m, dtype)\n mask = tf.reshape(mask, [1, n_dest, n_src])\n mult = tf.concat(\n [batch_size[..., tf.newaxis], tf.constant([1, 1], dtype=tf.int32)], 0\n )\n return tf.tile(mask, mult)\n\n def call(self, enc_out, target, training):\n # Get the shape of the target\n input_shape = tf.shape(target)\n batch_size = input_shape[0]\n seq_len = input_shape[1]\n\n # Generate causal attention mask\n causal_mask = self.causal_attention_mask(batch_size, seq_len, seq_len, tf.bool)\n\n # Self-attention on the target with causal mask\n target_att = self.self_att(target, target, attention_mask=causal_mask)\n\n # Apply layer normalization and dropout to the self-attention output\n target_norm = self.layernorm1(target + self.self_dropout(target_att, training=training))\n\n # Encoder attention using the normalized target and encoder output\n enc_out = self.enc_att(target_norm, enc_out)\n\n # Apply layer normalization and dropout to the encoder attention output and add to the target\n enc_out_norm = self.layernorm2(self.enc_dropout(enc_out, training=training) + target_norm)\n\n # Feed-forward network on the encoder attention output\n ffn_out = self.ffn(enc_out_norm)\n\n # Apply layer normalization and dropout to the feed-forward network output and add to the encoder attention output\n ffn_out_norm = self.layernorm3(enc_out_norm + self.ffn_dropout(ffn_out, training=training))\n\n return ffn_out_norm","metadata":{"execution":{"iopub.status.busy":"2023-12-21T21:59:50.253589Z","iopub.execute_input":"2023-12-21T21:59:50.253929Z","iopub.status.idle":"2023-12-21T21:59:50.270648Z","shell.execute_reply.started":"2023-12-21T21:59:50.253905Z","shell.execute_reply":"2023-12-21T21:59:50.269748Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### **Explanation**:\n\n* **Multi-Head Self-Attention (mha1)**:\n\nmha1 performs self-attention on the input sequence with a causal mask (look_ahead_mask). It allows the model to attend to previous positions while preventing attention to future positions.\n\n* **Layer Normalization and Residual Connection (layernorm1)**:\n\nlayernorm1 normalizes the output of the self-attention layer and adds it to the input, creating a residual connection.\n* **Multi-Head Attention with Encoder Output (mha2)**:\n\nmha2 performs attention on the decoder's output (out1) using the encoder's output. It helps the decoder focus on relevant information from the encoder.\n* **Layer Normalization and Residual Connection (layernorm2)**:\n\nlayernorm2 normalizes the output of the second attention layer and adds it to the output of the first attention layer, creating another residual connection.\n* **Feedforward Network (ffn)**:\n\nffn is a feedforward neural network applied to the output of the second attention layer.\n* **Layer Normalization and Residual Connection (layernorm3)**:\n\nlayernorm3 normalizes the output of the feedforward network and adds it to the output of the second attention layer, creating a final residual connection.\n* **Dropout**:\n\nDropout layers are applied after each attention and feedforward sub-layer to prevent overfitting during training.","metadata":{}},{"cell_type":"markdown","source":"## **5.3. Transformer Building**","metadata":{}},{"cell_type":"markdown","source":"\n
\n \n \"decod\"\n
","metadata":{}},{"cell_type":"code","source":"class Transformer(keras.Model):\n #Block 1: Define the Transformer Model\n def __init__(\n self,\n num_hid=64,\n num_head=2,\n num_feed_forward=128,\n source_maxlen=100,\n target_maxlen=100,\n num_layers_enc=4,\n num_layers_dec=1,\n num_classes=60,\n ):\n super().__init__()\n \n #Block 2: Model Initialization\n # Metrics for tracking loss and edit distance\n self.loss_metric = keras.metrics.Mean(name=\"loss\")\n self.acc_metric = keras.metrics.Mean(name=\"edit_dist\")\n \n # Additional metrics for precision, recall, and F1-score\n #self.precision_metric = keras.metrics.Precision(name=\"precision\")\n #self.recall_metric = keras.metrics.Recall(name=\"recall\")\n # Initialize F1Score metric\n #self.f1_metric = keras.metrics.F1Score(name=\"f1_score\", average=\"weighted\")\n\n \n # Model parameters\n self.num_layers_enc = num_layers_enc\n self.num_layers_dec = num_layers_dec\n self.target_maxlen = target_maxlen\n self.num_classes = num_classes\n \n #Block 3: Input Embedding Layers\n self.enc_input = LandmarkEmbedding(num_hid=num_hid, maxlen=source_maxlen)\n self.dec_input = TokenEmbedding(\n num_vocab=num_classes, maxlen=target_maxlen, num_hid=num_hid\n )\n\n #Block 4: Encoder\n # Encoder: Stack of TransformerEncoder layers\n self.encoder = keras.Sequential(\n [self.enc_input]\n + [\n TransformerEncoder(num_hid, num_head, num_feed_forward)\n for _ in range(num_layers_enc)\n ]\n )\n \n #Block 5: Decoder Layers\n # Decoder layers\n for i in range(num_layers_dec):\n setattr(\n self,\n f\"dec_layer_{i}\",\n TransformerDecoder(num_hid, num_head, num_feed_forward),\n )\n \n #Block 6: Classifier\n\n # Classifier for final predictions\n self.classifier = layers.Dense(num_classes)\n\n #Block 7: Decode Method\n def decode(self, enc_out, target, training):\n y = self.dec_input(target)\n for i in range(self.num_layers_dec):\n y = getattr(self, f\"dec_layer_{i}\")(enc_out, y, training)\n return y\n\n #Block 8: Call Method\n def call(self, inputs, training):\n source = inputs[0]\n target = inputs[1]\n x = self.encoder(source, training)\n y = self.decode(x, target, training)\n return self.classifier(y)\n\n #Block 9: Property for Returning Model Metrics\n @property\n def metrics(self):\n return [self.loss_metric,\n self.acc_metric]\n\n #Block 10: Training Step\n def train_step(self, batch):\n \"\"\"Processes one batch inside model.fit().\"\"\"\n source = batch[0]\n target = batch[1]\n\n input_shape = tf.shape(target)\n batch_size = input_shape[0]\n\n dec_input = target[:, :-1]\n dec_target = target[:, 1:]\n with tf.GradientTape() as tape:\n preds = self([source, dec_input])\n one_hot = tf.one_hot(dec_target, depth=self.num_classes)\n mask = tf.math.logical_not(tf.math.equal(dec_target, pad_token_idx))\n loss = self.compiled_loss(one_hot, preds, sample_weight=mask)\n trainable_vars = self.trainable_variables\n gradients = tape.gradient(loss, trainable_vars)\n self.optimizer.apply_gradients(zip(gradients, trainable_vars))\n\n\n # Computes the Levenshtein distance between sequences\n edit_dist = tf.edit_distance(\n tf.sparse.from_dense(target),\n tf.sparse.from_dense(tf.cast(tf.argmax(preds, axis=1), tf.int32)),\n )\n edit_dist = tf.reduce_mean(edit_dist)\n self.acc_metric.update_state(edit_dist)\n self.loss_metric.update_state(loss)\n #self.precision_metric.update_state(one_hot, preds, sample_weight=mask)\n\n return {\n \"loss\": self.loss_metric.result(),\n \"edit_dist\": self.acc_metric.result()}\n\n #Block 11: Testing Step\n def test_step(self, batch): \n source = batch[0]\n target = batch[1]\n\n input_shape = tf.shape(target)\n batch_size = input_shape[0]\n\n dec_input = target[:, :-1]\n dec_target = target[:, 1:]\n preds = self([source, dec_input])\n\n one_hot = tf.one_hot(dec_target, depth=self.num_classes)\n mask = tf.math.logical_not(tf.math.equal(dec_target, pad_token_idx))\n loss = self.compiled_loss(one_hot, preds, sample_weight=mask)\n\n\n # Computes the Levenshtein distance between sequences\n edit_dist = tf.edit_distance(\n tf.sparse.from_dense(target),\n tf.sparse.from_dense(tf.cast(tf.argmax(preds, axis=1), tf.int32)),\n )\n edit_dist = tf.reduce_mean(edit_dist)\n self.acc_metric.update_state(edit_dist)\n self.loss_metric.update_state(loss)\n #self.precision_metric.update_state(one_hot, preds, sample_weight=mask)\n\n return {\n \"loss\": self.loss_metric.result(),\n \"edit_dist\": self.acc_metric.result()}\n\n #Block 12: Inference method\n def generate(self, source, target_start_token_idx):\n \"\"\"Performs inference over one batch of inputs using greedy decoding.\"\"\"\n bs = tf.shape(source)[0]\n enc = self.encoder(source, training = False)\n dec_input = tf.ones((bs, 1), dtype=tf.int32) * target_start_token_idx\n dec_logits = []\n for i in range(self.target_maxlen - 1):\n dec_out = self.decode(enc, dec_input, training = False)\n logits = self.classifier(dec_out)\n logits = tf.argmax(logits, axis=-1, output_type=tf.int32)\n last_logit = logits[:, -1][..., tf.newaxis]\n dec_logits.append(last_logit)\n dec_input = tf.concat([dec_input, last_logit], axis=-1)\n return dec_input","metadata":{"execution":{"iopub.status.busy":"2023-12-21T22:37:00.163125Z","iopub.execute_input":"2023-12-21T22:37:00.163655Z","iopub.status.idle":"2023-12-21T22:37:00.191495Z","shell.execute_reply.started":"2023-12-21T22:37:00.163622Z","shell.execute_reply":"2023-12-21T22:37:00.190399Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### **Block 1: Define the Transformer Model**:\nHere, the `Transformer` class is defined, inheriting from keras.Model. It includes various parameters for configuring the model, such as the number of hidden units, heads, and feed-forward dimensions, as well as maximum lengths and the number of layers for the encoder and decoder:\n* `num_hid`: Dimensionality of the hidden state in the model.\n* `num_head`: Number of attention heads in multi-head attention layers.\n* `num_feed_forward`: Dimensionality of the feed-forward layer in the transformer blocks.\n* `source_maxlen`: Maximum length of the source sequence.\n* `target_maxlen`: Maximum length of the target sequence.\n* `num_layers_enc`: Number of encoder layers in the model.\n* `num_layers_dec`: Number of decoder layers in the model.\n* `num_classes`: Number of output classes or tokens in the vocabulary.\n\n#### **Block 2: Model Initialization**:\nIn this part, metrics for tracking loss and edit distance are initialized. Model parameters are also set based on the provided arguments during model creation. `edit_dist` refers to the edit distance between two sequences. The edit distance, also known as Levenshtein distance, measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one sequence into the other.\n\n#### **Block 3: Input Embedding Layers**:\nHere, the input embedding layers (``enc_input`` and ``dec_input``) are created using custom classes (``LandmarkEmbedding`` and ``TokenEmbedding``).\n\n#### **Block 4: Encoder**\nThe encoder is defined as a sequential stack of layers, starting with the ``enc_input`` layer followed by multiple instances of the ``TransformerEncoder`` layer.\n\n#### **Block 5: Decoder Layers**\nFor the decoder, a loop creates instances of the ``TransformerDecoder`` layer and sets them as attributes (``dec_layer_i``) of the model. A loop is used to dynamically create and add multiple instances of the TransformerDecoder layer to the model. This is motivated by the desire to create a flexible and customizable architecture that can adapt to different numbers of decoder layers specified by the user.\n\n#### **Block 6: Classifier**\nA dense layer is defined as the classifier for making final predictions. This layer maps the model's output to the number of classes.\n\n#### **Block 7: Decode Method**\nThe decode method takes encoder output (``enc_out``), target sequences (``target``), and a training flag. It processes the target sequences through the decoder layers (``dec_layer_i``).\n\n#### **Block 8: Call Method**\nThe ``call`` method takes input sequences (``source`` and ``target``) and a training flag. It encodes the source sequences, decodes the target sequences, and predicts the final output using the classifier.\n\n#### **Block 9: Property for Returning Model Metrics**\nThis property defines the metrics to be tracked during training and evaluation. In addition to ``loss_metric`` and ``acc_metric`` (edit distance), it includes precision, recall, and F1-score metrics.\n\n#### **Block 10: Training Step**\nIn this block, the ``train_step`` method processes one batch during model training. It calculates the loss, updates model weights, and computes various metrics including precision, recall, F1-score, and Levenshtein distance. The precision, recall, and F1-score metrics are updated using the ground truth (``one_hot``) and predicted values (``preds``). Levenshtein distance is computed using the ``edit_distance`` function.\n\n#### **Block 11: Testing Step**\nSimilar to the training step, the ``test_step`` method processes one batch during model evaluation. It calculates the loss, updates various metrics, and computes precision, recall, F1-score, and Levenshtein distance. The precision, recall, and F1-score metrics are updated using the ground truth (``one_hot``) and predicted values (``preds``). Levenshtein distance is computed using the ``edit_distance`` function.\n\n#### **Block 12: Inference Method**\nIn this block, the ``generate`` method performs inference over one batch of inputs using greedy decoding. It generates predictions for the target sequence. The model is provided with a source sequence, and the target sequence is generated step by step by predicting the next token at each iteration of the loop. The generated sequence (``dec_input``) is then returned. Greedy decoding is a simple strategy used in sequence generation tasks, such as machine translation or text generation, where the model generates one token at a time. In the context of this code, the ``generate`` method is using greedy decoding to generate predictions for the target sequence.","metadata":{}},{"cell_type":"markdown","source":"## **5.4. Model Training & Testing**","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-12-21T22:34:55.193213Z","iopub.execute_input":"2023-12-21T22:34:55.193656Z","iopub.status.idle":"2023-12-21T22:34:55.234357Z","shell.execute_reply.started":"2023-12-21T22:34:55.193628Z","shell.execute_reply":"2023-12-21T22:34:55.23341Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class DisplayOutputs(keras.callbacks.Callback):\n def __init__(\n self, batch, idx_to_token, target_start_token_idx=60, target_end_token_idx=61\n ):\n \"\"\"Displays a batch of outputs after every 4 epoch\n\n Args:\n batch: A test batch\n idx_to_token: A List containing the vocabulary tokens corresponding to their indices\n target_start_token_idx: A start token index in the target vocabulary\n target_end_token_idx: An end token index in the target vocabulary\n \"\"\"\n self.batch = batch\n self.target_start_token_idx = target_start_token_idx\n self.target_end_token_idx = target_end_token_idx\n self.idx_to_char = idx_to_token\n\n def on_epoch_end(self, epoch, logs=None):\n if epoch % 4 != 0:\n return\n source = self.batch[0]\n target = self.batch[1].numpy()\n bs = tf.shape(source)[0]\n preds = self.model.generate(source, self.target_start_token_idx)\n preds = preds.numpy()\n for i in range(bs):\n target_text = \"\".join([self.idx_to_char[_] for _ in target[i, :]])\n prediction = \"\"\n for idx in preds[i, :]:\n prediction += self.idx_to_char[idx]\n if idx == self.target_end_token_idx:\n break\n print(f\"target: {target_text.replace('-','')}\")\n print(f\"prediction: {prediction}\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-12-21T23:37:04.527924Z","iopub.execute_input":"2023-12-21T23:37:04.528355Z","iopub.status.idle":"2023-12-21T23:37:04.53989Z","shell.execute_reply.started":"2023-12-21T23:37:04.528325Z","shell.execute_reply":"2023-12-21T23:37:04.538874Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"batch = next(iter(valid_ds))\n\n# The vocabulary to convert predicted indices into characters\nidx_to_char = list(char_to_num.keys())\ndisplay_cb = DisplayOutputs(\n batch, idx_to_char, target_start_token_idx=char_to_num['<'], target_end_token_idx=char_to_num['>']\n)\n\nmodel = Transformer(\n num_hid=200,\n num_head=4,\n num_feed_forward=400,\n source_maxlen = FRAME_LEN,\n target_maxlen=64,\n num_layers_enc=2,\n num_layers_dec=1,\n num_classes=62\n)\nloss_fn = tf.keras.losses.CategoricalCrossentropy(\n from_logits=True, label_smoothing=0.1,\n)\n\n\noptimizer = keras.optimizers.Adam(0.0001)\nmodel.compile(optimizer=optimizer, loss=loss_fn)\n\n\n\nhistory = model.fit(train_ds, validation_data=valid_ds, callbacks=[display_cb], epochs=20)","metadata":{"execution":{"iopub.status.busy":"2023-12-21T23:37:43.50286Z","iopub.execute_input":"2023-12-21T23:37:43.503819Z","iopub.status.idle":"2023-12-21T23:45:19.489483Z","shell.execute_reply.started":"2023-12-21T23:37:43.503757Z","shell.execute_reply":"2023-12-21T23:45:19.488351Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **5.5. Model Evaluation**","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.plot(history.history['val_edit_dist'])\nplt.legend(['training loss', 'val_loss', 'val_edit_dist'])","metadata":{"execution":{"iopub.status.busy":"2023-12-21T23:52:36.032924Z","iopub.execute_input":"2023-12-21T23:52:36.033382Z","iopub.status.idle":"2023-12-21T23:52:36.334661Z","shell.execute_reply.started":"2023-12-21T23:52:36.033346Z","shell.execute_reply":"2023-12-21T23:52:36.333623Z"},"trusted":true},"outputs":[],"execution_count":null}]}