{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Google ASL Fingerspelling

\n

βœ‹Data Training and Evaluation with Levenshtein Distance βœ‹

\n\n
\"Heat
\n\n\n\n\n\nWelcome back to my notebook for the ASL Fingerspelling Recognition Kaggle competition! This notebook is a continuation from my previous notebook [ASLFR EDA & preprocessing](https://www.kaggle.com/code/hebasaleh00/aslfr-eda-preprocessing?scriptVersionId=133917049), where I processed the data and extracted features for the task. In this notebook, I will train and evaluate a model using the extracted features. Additionally, I will use the Levenshtein distance as a metric to measure the similarity between predicted labels and ground truth labels.\n\nI hope that my notebook will showcase the effectiveness of my approach and provide insights into the development of robust sign language recognition AI. I am excited to present my solution and contribute to the empowerment of the Deaf and Hard of Hearing community through innovative machine learning techniques.\n\nWish me luck as I embark on this journey!\n \n ** If you liked this Notebook, please do forget to upvote, and GooD LucK.**\n\n
\n πŸ‘   IF YOU FORK THIS OR FIND THIS HELPFUL   πŸ‘

PLEASE UPVOTE!

This was a lot of work for me and while it may seem silly, it makes me feel appreciated when others like my work. πŸ˜…\n
\n \n\n\n\n#

TABLE OF CONTENTS

\n\n* [1. Levenshtein Distance Implementation](#1)\n\n - [How to calculate the Levenshtein Distance](#1.1)\n \n - [Calculation Example Levenshtein Distance](#1.2)\n \n - [Import Libraries](#1.3)\n \n* [2. Data Preparation and Splitting](#2) \n\n - [Loading Dataset](#2.1)\n \n - [Splitting Dataset](#2.2)\n \n -[Converting Data to PyTorch tensors](#2.3)\n \n* [3. Data Training](#3)\n\n - [3.1 Model Definition](#3.1)\n\n - [3.2 Optimization Setup](#3.2)\n \n - [3.3 Evaluation Metrics function](#3.1)\n\n - [3.4 Training Loop and Evalutation](#3.3)\n\n* [4. Model Evaluation using Levenshtein Distance](#4)\n\n\n\n\n#

1. Levenshtein Distance Implementation

\n\nThe **Levenshtein distance**, also known as the edit distance, quantifies the dissimilarity between two strings by measuring the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. This metric provides a valuable measure of how well the predicted ASL sequence matches the ground truth or reference sequence.\n\n\n
\n\n#

1.1 How to calculate the Levenshtein Distance ?

\n\n---\n\nTo calculate the Levenshtein distance for ASL recognition, the predicted ASL sequence and the reference or ground truth sequence are compared character by character. Each character is treated as a token, representing a specific sign or gesture. The Levenshtein distance is then computed by determining the minimum number of edit operations needed to transform the predicted sequence into the reference sequence or vice versa.\n\nThe evaluation metric for this contest is the normalized total Levenshtein distance. The formula for calculating the metric is as follows:\n\nMetric = (N - D) / N\n\nWhere:\n\n N is the total number of characters in the labels.\n D is the total Levenshtein distance.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"\n\n#

1.2 Import Libraries

\n\n---","metadata":{}},{"cell_type":"code","source":"# import the desired packages\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport json\nimport plotly.graph_objects as go\nimport plotly.io as pio\npio.templates.default = \"simple_white\"\n\n# import data processing and visualisation libraries\nimport seaborn as sns\n%matplotlib inline\nimport os\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import MultiLabelBinarizer\nimport Levenshtein\n\n\nprint(\"Packages imported...\")","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:11:58.563455Z","iopub.execute_input":"2023-06-29T22:11:58.563823Z","iopub.status.idle":"2023-06-29T22:12:05.129822Z","shell.execute_reply.started":"2023-06-29T22:11:58.563794Z","shell.execute_reply":"2023-06-29T22:12:05.128668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import gc\nfrom tqdm import tqdm\n\nimport multiprocessing as mp\n\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\nfrom sklearn.metrics import accuracy_score\n\nimport warnings\nwarnings.filterwarnings(action='ignore')","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:05.132462Z","iopub.execute_input":"2023-06-29T22:12:05.133935Z","iopub.status.idle":"2023-06-29T22:12:05.42775Z","shell.execute_reply.started":"2023-06-29T22:12:05.133892Z","shell.execute_reply":"2023-06-29T22:12:05.426708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LANDMARK_FILES_DIR = \"/kaggle/input/asl-fingerspelling/train_landmarks\"\nTRAIN_FILE = \"/kaggle/input/asl-fingerspelling/train.csv\"\nlabel_map = json.load(open(\"/kaggle/input/asl-fingerspelling/character_to_prediction_index.json\", \"r\"))","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:05.429299Z","iopub.execute_input":"2023-06-29T22:12:05.429756Z","iopub.status.idle":"2023-06-29T22:12:05.440579Z","shell.execute_reply.started":"2023-06-29T22:12:05.429716Z","shell.execute_reply":"2023-06-29T22:12:05.439565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n#

1.2 How to calculate the Levenshtein Distance ?

\n\n\nTo calculate the metric, you would need the labels data and the predicted sequence. The Levenshtein distance measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one sequence into another.\n\nIn this data the labels are provided in the \"phrase\" column of the [train/supplemental_metadata].csv file. The predicted sequence can be obtained from the landmark data files in the [train/supplemental]_landmarks/ directory.\n\nTo calculate the total Levenshtein distance, I need to compare each character in the labels with the corresponding character in the predicted sequence and count the number of edits required.\n\nFinally, I can plug the values of N (total characters in the labels) and D (total Levenshtein distance) into the formula to compute the metric. The resulting value will give me an indication of the accuracy of the predicted sequence compared to the labels, with higher values indicating better performance. \n\nBelow is a small example of how levenshtein distance work:\n","metadata":{}},{"cell_type":"code","source":"from Levenshtein import distance\n#Using the dynamic programming approach for calculating the Levenshtein distance\n\ndef levenshteinDistanceDP(token1, token2):\n # Create a 2-D matrix \n distances = np.zeros((len(token1) + 1, len(token2) + 1))\n \n #Initialize the first row and column, Row index is fixed to 0 and the variable t1 is used to define the column index. \n for t1 in range(len(token1) + 1):\n distances[t1][0] = t1\n \n #Column index of the distances array is now fixed to 0, while the loop variable t2 is used to define the index of the rows\n for t2 in range(len(token2) + 1):\n distances[0][t2] = t2\n a = 0\n b = 0\n c = 0\n \n #Inside the loops the distances are calculated for all combinations of prefixes from the two words. \n for t1 in range(1, len(token1) + 1):\n for t2 in range(1, len(token2) + 1):\n if (token1[t1-1] == token2[t2-1]):\n distances[t1][t2] = distances[t1 - 1][t2 - 1]\n \n #If the two characters are not equal, then the distance in the current cell is equal to the\n #minimum of the three existing values in the 2 x 2 matrix after adding a cost of 1\n else:\n a = distances[t1][t2 - 1]\n b = distances[t1 - 1][t2]\n c = distances[t1 - 1][t2 - 1]\n \n if (a <= b and a <= c):\n distances[t1][t2] = a + 1\n elif (b <= a and b <= c):\n distances[t1][t2] = b + 1\n else:\n distances[t1][t2] = c + 1\n \n #Print its contents \n printDistances(distances, len(token1), len(token2))\n \n #returning the calculated distance between the two words\n return distances[len(token1)][len(token2)]\n\n\ndef printDistances(distances, token1Length, token2Length):\n for t1 in range(token1Length + 1):\n for t2 in range(token2Length + 1):\n print(int(distances[t1][t2]), end=\" \")\n print()\n \nphase1 = '3 creekhouse'\nphase2 = 'scales/kuhaylah'\n\n#Calling levenshteinDistanceDP function, \n#It returns an integer representing the distance between them\n\nprint(\"Printing The Distance Matrix:\")\nprint(f\" \\nThe Levenshtein distance of phases = {levenshteinDistanceDP(phase1, phase2):.02f} \")\n\n","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:07.198832Z","iopub.execute_input":"2023-06-29T22:12:07.199223Z","iopub.status.idle":"2023-06-29T22:12:07.215061Z","shell.execute_reply.started":"2023-06-29T22:12:07.199193Z","shell.execute_reply":"2023-06-29T22:12:07.214018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n#

2. Pre-Process Model

\n\n\n\n\n#

2.1 Column Extraction

\n\n---\n\n","metadata":{}},{"cell_type":"code","source":"def get_cols(df, words_pos, words_neg=[], ret_names=True):\n cols = []\n names = []\n for col in df.columns:\n # Check if column name contains all words\n if all([w in col for w in words_pos]) and all([w not in col for w in words_neg]):\n cols.append(df[col]) # Append the entire column to the list\n names.append(col)\n\n if ret_names:\n return cols, names\n else:\n return cols","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:08.256314Z","iopub.execute_input":"2023-06-29T22:12:08.256696Z","iopub.status.idle":"2023-06-29T22:12:08.263117Z","shell.execute_reply.started":"2023-06-29T22:12:08.256667Z","shell.execute_reply":"2023-06-29T22:12:08.26219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = pd.read_parquet(\"/kaggle/input/asl-fingerspelling/train_landmarks/1019715464.parquet\")\n\n# Landmark for Left/Right hand without z axis in raw data\nLH_Index, LEFT_HAND_NAME = get_cols(sample, ['left_hand'], ['z'])\nRH_Index,RIGHT_HAND_NAME = get_cols(sample, ['right_hand'], ['z'])\n#RIGHT_HAND_NAMES0.insert(0, \"frame\")\nRIGHT_HAND_NAME.insert(0, \"frame\")\nCOLUMNS = np.concatenate((LEFT_HAND_NAME, RIGHT_HAND_NAME))","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:08.559696Z","iopub.execute_input":"2023-06-29T22:12:08.560371Z","iopub.status.idle":"2023-06-29T22:12:23.736941Z","shell.execute_reply.started":"2023-06-29T22:12:08.560331Z","shell.execute_reply":"2023-06-29T22:12:23.735874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n#

2.2 Data Reduction function

\n\n---","metadata":{}},{"cell_type":"code","source":"# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n \"\"\" iterate through all the columns of a dataframe and modify the data type\n to reduce memory usage. \n \"\"\"\n #start_mem = df.memory_usage().sum() / 1024**2\n #print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n for col in df.columns:\n col_type = df[col].dtype\n\n if col_type != object:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64)\n\n return df","metadata":{"execution":{"iopub.status.busy":"2023-06-29T22:12:23.738605Z","iopub.execute_input":"2023-06-29T22:12:23.739068Z","iopub.status.idle":"2023-06-29T22:12:23.753087Z","shell.execute_reply.started":"2023-06-29T22:12:23.739028Z","shell.execute_reply":"2023-06-29T22:12:23.752069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

2.3 Feature Generator and video processing

\n\n---\n","metadata":{}},{"cell_type":"code","source":"class FeatureGen(torch.nn.Module):\n def __init__(self):\n super(FeatureGen, self).__init__()\n pass\n \n def forward(self, group):\n # Iterate over each landmark index\n #for landmark_index in range(1,22):\n # Generate feature names for x, y, z coordinates\n x_feature = f'x_right_hand_{landmark_index}'\n y_feature = f'y_right_hand_{landmark_index}'\n\n # Get the x, y, z coordinates for the landmark\n x = group[:, :21].float()\n y = group[:, 22:].float()\n #x = group[x_feature].values.astype(np.float16)\n #y = group[y_feature].values.astype(np.float16)\n \n # Perform feature transformations or calculations\n x = torch.tensor(x).contiguous().view(-1, 21)\n y = torch.tensor(y).contiguous().view(-1, 21)\n\n # Replace NaN values with 0\n x = x[~torch.any(torch.isnan(x), dim=1),:]\n y = y[~torch.any(torch.isnan(y), dim=1),:]\n #x = x[~np.isnan(x)] \n #y = y[~np.isnan(y)] \n\n x_mean = torch.mean(x, 0) \n y_mean = torch.mean(y, 0)\n\n x_std = torch.std(x, 0) \n y_std = torch.std(y, 0) \n\n # Add the calculated features to the sequence feature vector\n sequence_features = torch.cat([x_mean, y_mean,x_std,y_std], axis=0)\n sequence_features = torch.where(torch.isnan(sequence_features), torch.tensor(0.0, dtype=torch.float32), sequence_features)\n\n diff = 2158 - sequence_features.shape[0]\n if diff > 0:\n padding = torch.zeros(diff)\n sequence_features = torch.cat((sequence_features, padding))\n\n features = sequence_features[:2158]\n\n\n return features\n \nfeature_converter = FeatureGen()\nfeature_converter","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:22:11.481474Z","iopub.execute_input":"2023-06-30T02:22:11.481874Z","iopub.status.idle":"2023-06-30T02:22:11.49795Z","shell.execute_reply.started":"2023-06-30T02:22:11.481843Z","shell.execute_reply":"2023-06-30T02:22:11.497052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selected_columns = RIGHT_HAND_NAME\n\ndef load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=selected_columns)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-29T23:01:57.841653Z","iopub.execute_input":"2023-06-29T23:01:57.843299Z","iopub.status.idle":"2023-06-29T23:01:57.851041Z","shell.execute_reply.started":"2023-06-29T23:01:57.843255Z","shell.execute_reply":"2023-06-29T23:01:57.850106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import multiprocessing as mp\n\ndef process_parquet(row):\n x = load_relevant_data_subset(os.path.join(\"/kaggle/input/asl-fingerspelling\", row[1].path))\n grouped_landmarks = x.groupby('sequence_id')\n\n # Initialize empty lists to store features and labels\n features = []\n labels = []\n\n # Iterate over each sequence_id\n for sequence_id, group in grouped_landmarks:\n # Get the label for the sequence\n phrase = df.loc[df['sequence_id'] == sequence_id, 'phrase'].iloc[0]\n\n # Map each letter in the phrase using label_map\n mapped_phrase = [letter for letter in phrase]\n\n # Create a new Series with sequence and mapped_phrase\n result_series = pd.DataFrame({'sequence_id': sequence_id, 'mapped_phrase': mapped_phrase}) \n result_series['label'] = result_series['mapped_phrase'].map(label_map).astype(np.int8) \n\n # Initialize an empty feature vector for the sequence\n sequence_features = []\n group = group.values..astype(np.float32)\n x = feature_converter(torch.tensor(group)).cpu().numpy()\n \n return x, result_series['label']\n\ndf = pd.read_csv(TRAIN_FILE)\ndf = reduce_mem_usage(df) \n\nmax_label_length = 30\nall_features = np.zeros((df.shape[0], 2158))\nlabels = np.zeros((df.shape[0], max_label_length))\n\n# Process parquet files in parallel\nwith mp.Pool() as pool:\n results = pool.imap(process_parquet, df.iterrows(), chunksize=250)\n for i, (x, y) in tqdm(enumerate(results), total=df.shape[0]):\n all_features[i, :] = x\n labels[i, :len(y)] = y.values.reshape(1, -1)\n\n# Print the shapes of the tensors\nnp.save(\"feature_data.npy\", all_features)\nnp.save(\"feature_labels.npy\", labels)\n\nprint(\"Features tensor shape:\", all_features.shape)\nprint(\"Labels tensor shape:\", labels.shape)","metadata":{"execution":{"iopub.status.busy":"2023-06-29T23:11:31.664553Z","iopub.execute_input":"2023-06-29T23:11:31.66493Z","iopub.status.idle":"2023-06-30T01:41:11.371268Z","shell.execute_reply.started":"2023-06-29T23:11:31.664899Z","shell.execute_reply":"2023-06-30T01:41:11.369923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s = pd.read_parquet('/kaggle/input/asl-fingerspelling/train_landmarks/1019715464.parquet', columns=RIGHT_HAND_NAME) #selected_columns)\ns = s.groupby('sequence_id')\nfor sequence_id, group in s:\n print(group.shape)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:56:10.080952Z","iopub.execute_input":"2023-06-30T01:56:10.081376Z","iopub.status.idle":"2023-06-30T01:56:10.469215Z","shell.execute_reply.started":"2023-06-30T01:56:10.081343Z","shell.execute_reply":"2023-06-30T01:56:10.468184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import multiprocessing as mp\nimport pandas as pd\nimport torch\nimport numpy as np\n\n# Function to process a single parquet file\ndef process_parquet(row):\n path = os.path.join(\"/kaggle/input/asl-fingerspelling\", row[1].path)\n data_columns = COLUMNS\n landmark_df = pd.read_parquet(path, columns=data_columns)\n\n # Group the landmarks by sequence_id\n grouped_landmarks = landmark_df.groupby('sequence_id')\n\n # Initialize empty lists to store features and labels\n features = []\n labels = []\n\n # Iterate over each sequence_id\n for sequence_id, group in grouped_landmarks:\n # Get the label for the sequence\n phrase = df.loc[df['sequence_id'] == sequence_id, 'phrase'].iloc[0]\n\n # Map each letter in the phrase using label_map\n mapped_phrase = [letter for letter in phrase]\n \n # Create a new Series with sequence and mapped_phrase\n result_series = pd.DataFrame({'sequence_id': sequence_id, 'mapped_phrase': mapped_phrase}) \n result_series['label'] = result_series['mapped_phrase'].map(label_map).astype(np.int8) \n\n # Convert the label Series to a list\n #label_list = result_series['label'].tolist()\n\n # Initialize an empty feature vector for the sequence\n sequence_features = []\n \n # Iterate over each landmark index\n for landmark_index in range(20):\n # Generate feature names for x, y, z coordinates\n x_feature = f'x_right_hand_{landmark_index}'\n y_feature = f'y_right_hand_{landmark_index}'\n\n # Get the x, y, z coordinates for the landmark\n x = group[x_feature].values.astype(np.float16)\n y = group[y_feature].values.astype(np.float16)\n\n # Replace NaN values with 0\n x = x[~np.isnan(x)] \n y = y[~np.isnan(y)] \n \n # Perform feature transformations or calculations\n x = torch.tensor(x).contiguous().view(-1, x.shape[0])\n y = torch.tensor(y).contiguous().view(-1, y.shape[0])\n \n x_mean = torch.mean(x, 0) \n y_mean = torch.mean(y, 0) \n\n x_std = torch.std(x, 1) \n y_std = torch.std(y, 1) \n\n # Add the calculated features to the sequence feature vector\n sequence_features = torch.cat([x_mean, y_mean,x_std,y_std], axis=0)\n sequence_features = torch.where(torch.isnan(sequence_features), torch.tensor(0.0, dtype=torch.float32), sequence_features)\n \n diff = 2158 - sequence_features.shape[0]\n if diff > 0:\n padding = torch.zeros(diff)\n sequence_features = torch.cat((sequence_features, padding))\n \n features = sequence_features[:2158].cpu().numpy()\n \n \n return sequence_features.cpu().numpy(), result_series['label']\n\nif __name__ == '__main__':\n df = pd.read_csv(TRAIN_FILE)\n df = reduce_mem_usage(df) \n df2 = df.head(10)\n\n max_label_length = 30\n all_features = np.zeros((df.shape[0], 2158))\n labels = np.zeros((df.shape[0], max_label_length))\n #all_features = []\n #labels = []\n\n # Process parquet files in parallel\n with mp.Pool() as pool:\n results = pool.imap(process_parquet, df.iterrows(), chunksize=250)\n for i, (x, y) in tqdm(enumerate(results), total=df.shape[0]):\n all_features[i, :] = x\n labels[i, :len(y)] = y.values.reshape(1, -1)\n\n # Print the shapes of the tensors\n np.save(\"feature_data.npy\", all_features)\n np.save(\"feature_labels.npy\", labels)\n\n print(\"Features tensor shape:\", all_features.shape)\n print(\"Labels tensor shape:\", labels.shape)","metadata":{"execution":{"iopub.status.busy":"2023-06-27T20:55:23.197655Z","iopub.execute_input":"2023-06-27T20:55:23.198093Z","iopub.status.idle":"2023-06-27T20:55:34.120062Z","shell.execute_reply.started":"2023-06-27T20:55:23.198061Z","shell.execute_reply":"2023-06-27T20:55:34.118518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n#

2. Data Preparation and Splitting

\n\n\n\n\n

3.1 Loading Dataset

\n\n---","metadata":{}},{"cell_type":"code","source":"datax = np.load(\"/kaggle/working/feature_data.npy\")\ndatay = np.load(\"/kaggle/working/feature_labels.npy\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:21.278993Z","iopub.execute_input":"2023-06-30T01:41:21.279401Z","iopub.status.idle":"2023-06-30T01:41:21.664051Z","shell.execute_reply.started":"2023-06-30T01:41:21.279366Z","shell.execute_reply":"2023-06-30T01:41:21.662958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datax1 = np.load(\"/kaggle/input/aslfr-eda-preprocessing/feature_data.npy\")\ndatay1 = np.load(\"/kaggle/input/aslfr-eda-preprocessing/feature_labels.npy\")","metadata":{"execution":{"iopub.status.busy":"2023-06-29T15:11:17.484594Z","iopub.execute_input":"2023-06-29T15:11:17.485543Z","iopub.status.idle":"2023-06-29T15:11:26.272216Z","shell.execute_reply.started":"2023-06-29T15:11:17.485497Z","shell.execute_reply":"2023-06-29T15:11:26.271267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datax.shape\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:26.02349Z","iopub.execute_input":"2023-06-30T01:41:26.024173Z","iopub.status.idle":"2023-06-30T01:41:26.031774Z","shell.execute_reply.started":"2023-06-30T01:41:26.024132Z","shell.execute_reply":"2023-06-30T01:41:26.030654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datay.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:28.725711Z","iopub.execute_input":"2023-06-30T01:41:28.728266Z","iopub.status.idle":"2023-06-30T01:41:28.734645Z","shell.execute_reply.started":"2023-06-30T01:41:28.728231Z","shell.execute_reply":"2023-06-30T01:41:28.733579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datay = datay.astype(int)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:42:44.585248Z","iopub.execute_input":"2023-06-30T01:42:44.585625Z","iopub.status.idle":"2023-06-30T01:42:44.596499Z","shell.execute_reply.started":"2023-06-30T01:42:44.585596Z","shell.execute_reply":"2023-06-30T01:42:44.595376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

3.2 Splitting Dataset

\n\n---","metadata":{}},{"cell_type":"code","source":"# Data Split\ntrainx, testx, trainy, testy = train_test_split(datax, datay, test_size=0.15, random_state=10)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:34.891501Z","iopub.execute_input":"2023-06-30T01:41:34.892224Z","iopub.status.idle":"2023-06-30T01:41:35.560161Z","shell.execute_reply.started":"2023-06-30T01:41:34.892187Z","shell.execute_reply":"2023-06-30T01:41:35.559128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

3.3 Converting Data to PyTorch tensors

\n\n---","metadata":{}},{"cell_type":"code","source":"# Convert data to PyTorch tensors\ntrainx = torch.from_numpy(trainx).float()\ntrainy = torch.from_numpy(trainy).float()\ntestx = torch.from_numpy(testx).float()\ntesty = torch.from_numpy(testy).float()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:36.364346Z","iopub.execute_input":"2023-06-30T01:41:36.365288Z","iopub.status.idle":"2023-06-30T01:41:36.852838Z","shell.execute_reply.started":"2023-06-30T01:41:36.365247Z","shell.execute_reply":"2023-06-30T01:41:36.851816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ASLDataset(Dataset):\n def __init__(self, datax, datay):\n self.datax = datax\n self.datay = datay\n \n def __getitem__(self, index):\n return self.datax[index,:], self.datay[index]\n \n def __len__(self):\n return len(self.datay)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:37.25148Z","iopub.execute_input":"2023-06-30T01:41:37.252042Z","iopub.status.idle":"2023-06-30T01:41:37.259053Z","shell.execute_reply.started":"2023-06-30T01:41:37.252001Z","shell.execute_reply":"2023-06-30T01:41:37.258044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set device\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# Data Preparation\ntrain_data = ASLDataset(trainx, trainy)\ntest_data = ASLDataset(testx, testy)\n\n# DataLoader\nBATCH_SIZE = 64\ntrain_loader = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True)\ntest_loader = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:37.579137Z","iopub.execute_input":"2023-06-30T01:41:37.581417Z","iopub.status.idle":"2023-06-30T01:41:37.684517Z","shell.execute_reply.started":"2023-06-30T01:41:37.581386Z","shell.execute_reply":"2023-06-30T01:41:37.683488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n#

4. Data Training

\n\n\n\n\n

4.1 Model Definition

\n\n---\n","metadata":{}},{"cell_type":"code","source":"import torch.nn.functional as F\n\n# Model Definition\nclass ASLModel(torch.nn.Module):\n def __init__(self, input_size, output_size):\n super(ASLModel, self).__init__()\n self.fc1 = nn.Linear(input_size, 2048)\n self.fc2 = nn.Linear(2048, 1024)\n self.fc3 = nn.Linear(1024, output_size)\n #self.dropout = nn.Dropout(0.2) \n\n def forward(self, x):\n x = torch.relu(self.fc1(x))\n #x = self.dropout(x)\n x = torch.relu(self.fc2(x))\n #x = self.dropout(x)\n x = self.fc3(x)\n # Apply softmax activation for multi-class classification\n #x = F.softmax(x, dim=0)\n return x","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:39.096318Z","iopub.execute_input":"2023-06-30T01:41:39.096718Z","iopub.status.idle":"2023-06-30T01:41:39.104753Z","shell.execute_reply.started":"2023-06-30T01:41:39.096688Z","shell.execute_reply":"2023-06-30T01:41:39.10362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

4.2 Optimization Setup

\n\n---\n","metadata":{}},{"cell_type":"code","source":"# Model Initialization\nmodel = ASLModel(input_size=trainx.shape[1], output_size=trainy.shape[1]).to(device)\n\n# Optimization Setup\ncriterion = nn.SmoothL1Loss()\noptimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n#optimizer = torch.optim.SGD(model.parameters(), lr=0.001) \n#optimizer = torch.optim.AdamW(model.parameters(), lr=0.005)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:44.820739Z","iopub.execute_input":"2023-06-30T01:41:44.821149Z","iopub.status.idle":"2023-06-30T01:41:48.209265Z","shell.execute_reply.started":"2023-06-30T01:41:44.821116Z","shell.execute_reply":"2023-06-30T01:41:48.208211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

4.3 Evaluation Metrics Function

\n\n---","metadata":{}},{"cell_type":"code","source":"def calculate_levenshtein_distance(pred_labels, target_labels):\n distance = 0\n for pred, target in zip(pred_labels, target_labels):\n distance += Levenshtein.distance(pred, target)\n return distance","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:41:48.211143Z","iopub.execute_input":"2023-06-30T01:41:48.211497Z","iopub.status.idle":"2023-06-30T01:41:48.218222Z","shell.execute_reply.started":"2023-06-30T01:41:48.211463Z","shell.execute_reply":"2023-06-30T01:41:48.217048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n

4.4 Training Loop and Evaluation

\n\n---","metadata":{}},{"cell_type":"code","source":"EPOCHS = 40\nfor epoch in range(EPOCHS):\n model.train()\n train_loss = 0.0\n train_correct = 0\n\n for inputs, targets in train_loader:\n inputs = inputs.to(device)\n targets = targets.to(device)\n \n outputs = model(inputs)\n loss = criterion(outputs, targets)\n loss.backward()\n #torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1) # Adjust the max_norm value as needed\n optimizer.step()\n optimizer.zero_grad()\n\n train_loss += loss.item()\n train_correct += (outputs.round() == targets).sum().item()\n\n train_loss /= len(train_loader)\n train_accuracy = train_correct / len(train_data)\n\n # Evaluation\n model.eval()\n test_loss = 0.0\n test_correct = 0\n levenshtein_distance = 0\n\n with torch.no_grad():\n for inputs, targets in test_loader:\n \n inputs = inputs.to(device)\n targets = targets.to(device)\n \n outputs = model(inputs) \n loss = criterion(outputs, targets)\n\n test_loss += loss.item()\n test_correct += (outputs.round() == targets).sum().item()\n\n # Define the reverse mapping dictionary\n reverse_label_map = {v: k for k, v in label_map.items()}\n outputs_array = outputs.detach().cpu().numpy()\n targets_array = targets.detach().cpu().numpy()\n\n # Convert predictions and targets to letter sequences\n pred_labels = [[reverse_label_map[int(label)] for label in output if int(label) in reverse_label_map ] if len(output) > 0 else [] for output in outputs_array.round()]\n target_labels = [[reverse_label_map[label] for label in target if int(label) in reverse_label_map ] if len(target) > 0 else [] for target in targets_array.round()]\n\n # Calculate Levenshtein distance\n levenshtein_distance += calculate_levenshtein_distance(pred_labels, target_labels)\n \n test_loss /= len(test_loader)\n test_accuracy = test_correct / len(test_data)\n average_levenshtein_distance = levenshtein_distance / len(test_data)\n\n # Print epoch results\n print(f\"Epoch {epoch+1}/{EPOCHS}\")\n print(f\"Train Loss: {train_loss:.4f} | Train Accuracy: {train_accuracy:.4f}\")\n print(f\"Test Loss: {test_loss:.4f} | Test Accuracy: {test_accuracy:.4f}\")\n print(f\"Average Levenshtein Distance: {average_levenshtein_distance:.4f}\")\n print(\"=\" * 50)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:43:55.22232Z","iopub.execute_input":"2023-06-30T01:43:55.2227Z","iopub.status.idle":"2023-06-30T01:47:45.917139Z","shell.execute_reply.started":"2023-06-30T01:43:55.222671Z","shell.execute_reply":"2023-06-30T01:47:45.916064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the PyTorch model's weights\ntorch.save(model.state_dict(), 'pytorch_weights.pth')","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:49:53.922419Z","iopub.execute_input":"2023-06-30T01:49:53.922815Z","iopub.status.idle":"2023-06-30T01:49:53.987819Z","shell.execute_reply.started":"2023-06-30T01:49:53.922782Z","shell.execute_reply":"2023-06-30T01:49:53.986822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n#

5. Model Evaluation using Levenshtein Distance

","metadata":{}},{"cell_type":"code","source":"def evaluate_model(model, test_loader, label_map):\n model.eval()\n levenshtein_distance = 0\n results = []\n\n with torch.no_grad():\n for inputs, targets in test_loader:\n inputs = inputs.to(device)\n targets = targets.to(device)\n\n outputs = model(inputs)\n\n # Define the reverse mapping dictionary\n outputs_array = outputs.detach().cpu().numpy()\n targets_array = targets.detach().cpu().numpy()\n\n # Convert predictions and targets to letter sequences\n pred_labels = [[reverse_label_map[int(label)] for label in output if int(label) in reverse_label_map ] if len(output) > 0 else [] for output in outputs_array.round()]\n target_labels = [[reverse_label_map[label] for label in target if int(label) in reverse_label_map ] if len(target) > 0 else [] for target in targets_array.round()]\n\n # Calculate Levenshtein distance\n levenshtein_distance += calculate_levenshtein_distance(pred_labels, target_labels)\n\n # Append results to the list\n results.extend(list(zip(target_labels, pred_labels)))\n\n average_levenshtein_distance = levenshtein_distance / len(test_loader.dataset)\n\n # Create a dataframe to display the results\n df = pd.DataFrame(results, columns=['Actual Phrase', 'Predicted Phrase'])\n df['Actual Phrase'] = df['Actual Phrase'].apply(lambda x: ' '.join(x))\n df['Predicted Phrase'] = df['Predicted Phrase'].apply(lambda x: ' '.join(x))\n df['Levenshtein_Distance'] = df.apply(lambda row: Levenshtein.distance(row['Actual Phrase'], row['Predicted Phrase']), axis=1)\n\n\n return df, average_levenshtein_distance\n\n# Evaluate the model\nevaluation_df, average_distance = evaluate_model(model, test_loader, label_map)\n\n# Display the results\nprint(f\"Average Levenshtein Distance: {average_distance:.4f}\")\nevaluation_df.head(20)\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:49:57.01085Z","iopub.execute_input":"2023-06-30T01:49:57.01155Z","iopub.status.idle":"2023-06-30T01:49:58.748067Z","shell.execute_reply.started":"2023-06-30T01:49:57.011514Z","shell.execute_reply":"2023-06-30T01:49:58.747026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Value Counts\nOutput_Evaluation = evaluation_df['Levenshtein_Distance'].value_counts().sort_index()\n\nplt.figure(figsize=(15, 8))\nplt.bar(Output_Evaluation.index, Output_Evaluation.values, color='#ce4257')\nplt.title('Output Levenshtein Distance Distribution')\nplt.xlabel('Levenshtein Distance')\nplt.ylabel('Sample Count')\nplt.xlim(-0.5, evaluation_df.Levenshtein_Distance.max() + 0.5)\nplt.grid(axis='y')\nplt.xticks(range(evaluation_df['Levenshtein_Distance'].max() + 1))\n\nplt.show()\nprint(f'Mean Levenshtein Distance Distribution: {evaluation_df.Levenshtein_Distance.mean():.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-06-30T01:50:02.429955Z","iopub.execute_input":"2023-06-30T01:50:02.430357Z","iopub.status.idle":"2023-06-30T01:50:02.893921Z","shell.execute_reply.started":"2023-06-30T01:50:02.430326Z","shell.execute_reply":"2023-06-30T01:50:02.89289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n#

6. Converting model for Submission

\n\n\n\n\n#

6.1 Converting Pytorch to ONNX

\n---\n","metadata":{}},{"cell_type":"code","source":"# install libs\n!pip install onnx_tf","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:31:54.738041Z","iopub.execute_input":"2023-06-30T02:31:54.738782Z","iopub.status.idle":"2023-06-30T02:32:08.837782Z","shell.execute_reply.started":"2023-06-30T02:31:54.738746Z","shell.execute_reply":"2023-06-30T02:32:08.836491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dummy_input = torch.rand((166, 43))\n\nonnx_feat_gen_path = 'feature_gen.onnx'\n\nfeature_converter.eval()\n\ntorch.onnx.export(\n feature_converter, # PyTorch Model\n dummy_input, # Input tensor\n onnx_feat_gen_path, # Output file (eg. 'output_model.onnx')\n opset_version=12, # Operator support version\n input_names=['input'], # Input tensor name (arbitary)\n output_names=['output'], # Output tensor name (arbitary)\n dynamic_axes={\n 'input' : {0: 'input'}\n }\n)\nprint(\"converting done.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:23:24.254301Z","iopub.execute_input":"2023-06-30T02:23:24.254718Z","iopub.status.idle":"2023-06-30T02:23:24.316964Z","shell.execute_reply.started":"2023-06-30T02:23:24.254687Z","shell.execute_reply":"2023-06-30T02:23:24.315795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainx.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:24:11.868987Z","iopub.execute_input":"2023-06-30T02:24:11.869775Z","iopub.status.idle":"2023-06-30T02:24:11.877407Z","shell.execute_reply.started":"2023-06-30T02:24:11.869739Z","shell.execute_reply":"2023-06-30T02:24:11.876366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import onnx\nimport torch\nclass ASLModel(nn.Module):\n def __init__(self, input_size, output_size):\n super(ASLModel, self).__init__()\n self.fc1 = nn.Linear(input_size, 2048)\n self.fc2 = nn.Linear(2048, 1024)\n self.fc3 = nn.Linear(1024, output_size)\n\n def forward(self, x):\n x = torch.relu(self.fc1(x))\n x = torch.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n# Create an instance of the model\nmodel = ASLModel(input_size=trainx.shape[1], output_size=trainy.shape[1])\n\n# Save the model\ntorch.save(model.state_dict(), 'model.pth')\n\n\nprint(\"converting from torch to onnx...\")\nopset_version = 12\n\nsample_input= trainx\nprint(sample_input.shape)\n#sample_input = torch.tensor(sample_input)\nsample_input = torch.randn(57126, 2158) \n\n#model = torch.load(\"/kaggle/working/pytorch_weights.pth\")\n\n\n# Set the model in evaluation mode\n#model.eval()\n\ntorch.onnx.export(\n model, # PyTorch Model\n sample_input, # Input tensor\n \"/kaggle/working/model.onnx\",# Output file (eg. 'output_model.onnx')\n opset_version=opset_version, # Operator support version\n input_names=['inputs'], # Input tensor name (arbitary)\n output_names=['outputs'], # Output tensor name (arbitary)\n dynamic_axes={\n 'inputs': {0: 'length', },\n 'outputs': {0: 'length', },\n }\n)\nprint(\"converting done.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:24:22.058526Z","iopub.execute_input":"2023-06-30T02:24:22.058917Z","iopub.status.idle":"2023-06-30T02:24:31.03835Z","shell.execute_reply.started":"2023-06-30T02:24:22.058885Z","shell.execute_reply":"2023-06-30T02:24:31.037272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n#

6.2 Converting ONNX to TensorFlow

\n---\n\n","metadata":{}},{"cell_type":"code","source":"import onnx_tf\n\n\n\n#tf_feature_generator_path = '/kaggle/working/tf_Feature_generator'\nonnx_feat_gen = onnx.load('/kaggle/working/feature_gen.onnx')\ntf_rep = onnx_tf.backend.prepare(onnx_feat_gen)\ntf_rep.export_graph('/kaggle/working/tf_Feature_generator')\n\n\n#tf_model_path = '/kaggle/working/tf_model'\nonnx_model = onnx.load(\"/kaggle/working/model.onnx\")\ntf_rep = onnx_tf.backend.prepare(onnx_model)\ntf_rep.export_graph('/kaggle/working/tf_model')\n\nprint(\"converting done.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T02:32:17.637445Z","iopub.execute_input":"2023-06-30T02:32:17.63787Z","iopub.status.idle":"2023-06-30T02:32:43.33792Z","shell.execute_reply.started":"2023-06-30T02:32:17.637832Z","shell.execute_reply":"2023-06-30T02:32:43.336686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n#

6.3 Converting TensorFlow to TensorFlow Lite

\n---\n\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\n\nclass Inference_Model(tf.Module):\n def __init__(self):\n self.feature_gen = tf.saved_model.load('/kaggle/working/tf_Feature_generator')\n self.model = tf.saved_model.load('/kaggle/working/tf_model')\n self.feature_gen.trainable = False\n self.model.trainable = False\n \n @tf.function(input_signature=[\n tf.TensorSpec(shape=[166, 43], dtype=tf.float32, name='inputs')\n ])\n def call(self, inputs):\n output_tensors = {}\n features = self.feature_gen(**{'inputs': inputs})['output']\n output_tensors['outputs'] = self.model(**{'inputs': tf.expand_dims(features, 0)})['output'][0, :]\n return output_tensors\n\nmy_tensorflow_model = Inference_Model()\ntf.saved_model.save(my_tensorflow_model, '/kaggle/working/tf_inference_model', signatures={'serving_default': my_tensorflow_model.call})\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T03:33:27.787382Z","iopub.execute_input":"2023-06-30T03:33:27.787844Z","iopub.status.idle":"2023-06-30T03:33:28.643875Z","shell.execute_reply.started":"2023-06-30T03:33:27.787812Z","shell.execute_reply":"2023-06-30T03:33:28.641973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\nclass Inference_Model(tf.Module):\n def __init__(self):\n super(Inference_Model, self).__init__()\n self.feature_gen = tf.saved_model.load('/kaggle/working/tf_Feature_generator')\n self.model = tf.saved_model.load('/kaggle/working/tf_model')\n self.feature_gen.trainable = False\n self.model.trainable = False\n \n @tf.function(input_signature=[\n tf.TensorSpec(shape=[None, 43], dtype=tf.float32, name='inputs')\n ])\n def call(self, input):\n output_tensors = {}\n features = self.feature_gen(**{'input': input})['output']\n output_tensors['outputs'] = self.model(**{'input': tf.expand_dims(features, 0)})['output'][0,:]\n return output_tensors\n \nmy_tensorflow_model = Inference_Model()\ntf.saved_model.save(my_tensorflow_model, '/kaggle/working/tf_inference_model', signatures={'serving_default': my_tensorflow_model.call})","metadata":{"execution":{"iopub.status.busy":"2023-06-30T03:33:38.981977Z","iopub.execute_input":"2023-06-30T03:33:38.982731Z","iopub.status.idle":"2023-06-30T03:33:39.999979Z","shell.execute_reply.started":"2023-06-30T03:33:38.982694Z","shell.execute_reply":"2023-06-30T03:33:39.997905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\nprint(\"converting from tensorflow to tflite\")\n\nconverter = tf.lite.TFLiteConverter.from_saved_model(\"/kaggle/working/tf_model\")\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\nconverter.target_spec.supported_types = [tf.float16]\n# converter.target_spec.supported_types = [tf.float32]\nconverter.experimental_new_converter = True\nconverter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,\n tf.lite.OpsSet.SELECT_TF_OPS]\ntflite_model = converter.convert()\n# Save the model\nwith open(\"/kaggle/working/model.tflite\", 'wb') as f:\n f.write(tflite_model)\nprint(\"converting done.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-29T15:49:14.013522Z","iopub.execute_input":"2023-06-29T15:49:14.013943Z","iopub.status.idle":"2023-06-29T15:49:15.56588Z","shell.execute_reply.started":"2023-06-29T15:49:14.013914Z","shell.execute_reply":"2023-06-29T15:49:15.564806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n#

6.4 Create submission file

\n---\n\n","metadata":{}},{"cell_type":"code","source":"basedir = \"/kaggle/working/\"\n\nSEL_FEATURES = ['x_left_hand_0', 'x_left_hand_1', 'x_left_hand_2', 'x_left_hand_3',\n 'x_left_hand_4', 'x_left_hand_5', 'x_left_hand_6', 'x_left_hand_7',\n 'x_left_hand_8', 'x_left_hand_9', 'x_left_hand_10',\n 'x_left_hand_11', 'x_left_hand_12', 'x_left_hand_13',\n 'x_left_hand_14', 'x_left_hand_15', 'x_left_hand_16',\n 'x_left_hand_17', 'x_left_hand_18', 'x_left_hand_19',\n 'x_left_hand_20', 'y_left_hand_0', 'y_left_hand_1',\n 'y_left_hand_2', 'y_left_hand_3', 'y_left_hand_4', 'y_left_hand_5',\n 'y_left_hand_6', 'y_left_hand_7', 'y_left_hand_8', 'y_left_hand_9',\n 'y_left_hand_10', 'y_left_hand_11', 'y_left_hand_12',\n 'y_left_hand_13', 'y_left_hand_14', 'y_left_hand_15',\n 'y_left_hand_16', 'y_left_hand_17', 'y_left_hand_18',\n 'y_left_hand_19', 'y_left_hand_20', 'frame', 'x_right_hand_0',\n 'x_right_hand_1', 'x_right_hand_2', 'x_right_hand_3',\n 'x_right_hand_4', 'x_right_hand_5', 'x_right_hand_6',\n 'x_right_hand_7', 'x_right_hand_8', 'x_right_hand_9',\n 'x_right_hand_10', 'x_right_hand_11', 'x_right_hand_12',\n 'x_right_hand_13', 'x_right_hand_14', 'x_right_hand_15',\n 'x_right_hand_16', 'x_right_hand_17', 'x_right_hand_18',\n 'x_right_hand_19', 'x_right_hand_20', 'y_right_hand_0',\n 'y_right_hand_1', 'y_right_hand_2', 'y_right_hand_3',\n 'y_right_hand_4', 'y_right_hand_5', 'y_right_hand_6',\n 'y_right_hand_7', 'y_right_hand_8', 'y_right_hand_9',\n 'y_right_hand_10', 'y_right_hand_11', 'y_right_hand_12',\n 'y_right_hand_13', 'y_right_hand_14', 'y_right_hand_15',\n 'y_right_hand_16', 'y_right_hand_17', 'y_right_hand_18',\n 'y_right_hand_19', 'y_right_hand_20'\n ]\nNUM_FEATURES = len(RIGHT_HAND_NAME)\n\n#print(\"number of used features:\",NUM_FEATURES)\n\nd = {\"selected_columns\":RIGHT_HAND_NAME}\n\nwith open(f\"{basedir}/inference_args.json\", \"w\") as f:\n json.dump(d, f)","metadata":{"execution":{"iopub.status.busy":"2023-06-29T17:35:35.580835Z","iopub.execute_input":"2023-06-29T17:35:35.581214Z","iopub.status.idle":"2023-06-29T17:35:35.593507Z","shell.execute_reply.started":"2023-06-29T17:35:35.581184Z","shell.execute_reply":"2023-06-29T17:35:35.592537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!zip submission.zip '/kaggle/working/model.tflite' '/kaggle/working/inference_args.json'\n","metadata":{"execution":{"iopub.status.busy":"2023-06-29T17:35:38.408687Z","iopub.execute_input":"2023-06-29T17:35:38.409058Z","iopub.status.idle":"2023-06-29T17:35:40.47509Z","shell.execute_reply.started":"2023-06-29T17:35:38.409028Z","shell.execute_reply":"2023-06-29T17:35:40.473852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow.lite as tflite","metadata":{"execution":{"iopub.status.busy":"2023-06-29T17:35:40.478395Z","iopub.execute_input":"2023-06-29T17:35:40.47879Z","iopub.status.idle":"2023-06-29T17:35:40.483669Z","shell.execute_reply.started":"2023-06-29T17:35:40.478752Z","shell.execute_reply":"2023-06-29T17:35:40.482705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nCHECKING = True\n\nif CHECKING:\n #!pip install tflite-runtime==2.9.1\n #import tflite_runtime.interpreter as tflite\n\n def load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=RIGHT_HAND_NAME) #selected_columns)\n \n data_path = \"/kaggle/input/asl-fingerspelling/train_landmarks/1019715464.parquet\"\n frames = load_relevant_data_subset(data_path).values\n \n model_path = '/kaggle/working/model.tflite'\n interpreter = tflite.Interpreter(model_path)\n found_signatures = list(interpreter.get_signature_list().keys())\n prediction_fn = interpreter.get_signature_runner(\"serving_default\")\n print(frames.shape)\n with open (\"/kaggle/input/asl-fingerspelling/character_to_prediction_index.json\", \"r\") as f:\n character_map = json.load(f)\n rev_character_map = {j:i for i,j in character_map.items()}","metadata":{"execution":{"iopub.status.busy":"2023-06-29T18:03:39.557548Z","iopub.execute_input":"2023-06-29T18:03:39.557935Z","iopub.status.idle":"2023-06-29T18:03:39.732239Z","shell.execute_reply.started":"2023-06-29T18:03:39.557905Z","shell.execute_reply":"2023-06-29T18:03:39.731102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainx.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-29T18:05:41.07241Z","iopub.execute_input":"2023-06-29T18:05:41.072841Z","iopub.status.idle":"2023-06-29T18:05:41.080435Z","shell.execute_reply.started":"2023-06-29T18:05:41.07281Z","shell.execute_reply":"2023-06-29T18:05:41.079472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CHECKING:\n output = prediction_fn(inputs=frames)\n prediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output['outputs'], axis=1)])\n print(\"\\n\\n\",prediction_str[:100])","metadata":{"execution":{"iopub.status.busy":"2023-06-29T18:03:44.422191Z","iopub.execute_input":"2023-06-29T18:03:44.422619Z","iopub.status.idle":"2023-06-29T18:03:44.466842Z","shell.execute_reply.started":"2023-06-29T18:03:44.422589Z","shell.execute_reply":"2023-06-29T18:03:44.465445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}