\n ![]()
\n
\n\n\n\n"},"metadata":{}}]},{"cell_type":"markdown","source":"# Preprocess the data\nFor convenience and efficiency, we will rearrange the data so that each parquet file contains the landmark data along with the phrase it represents. This way we don't have to switch between train.csv and its parquet file.\n\n\nWe will save the new data in the[TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord?hl=fr) format. TFRecord format is a simple format for storing a sequence of binary records. Storing and loading the data using TFRecord is much more efficient and faster.\n\n\nReference:\n\n\nhttps://www.kaggle.com/code/irohith/aslfr-preprocess-dataset\n\nhttps://www.kaggle.com/code/shlomoron/aslfr-parquets-to-tfrecords-cleaned","metadata":{}},{"cell_type":"markdown","source":"ASL-Fingerspelling mainly focusses on hand movement. So we will take hand land mark coordinates and pose coordinates for hands to train the model.","metadata":{}},{"cell_type":"markdown","source":"[](https://photos.app.goo.gl/tannmrqyKAY7iyf8A)","metadata":{}},{"cell_type":"markdown","source":"# Fetch the pose landmark coordinates related to hand movement.","metadata":{}},{"cell_type":"markdown","source":"This image illustrates the pose coordinates related to hand movement. The pose coordinates also come from the parquet file.","metadata":{}},{"cell_type":"markdown","source":"[](https://photos.app.goo.gl/X9cAaKXWP5pEev9X6)\n\n\n[Image Sources](https://developers.google.com/mediapipe/solutions/vision/pose_landmarker)","metadata":{}},{"cell_type":"code","source":"# Pose coordinates for hand movement.\nLPOSE = [13, 15, 17, 19, 21]\nRPOSE = [14, 16, 18, 20, 22]\nPOSE = LPOSE + RPOSE","metadata":{"execution":{"iopub.status.busy":"2023-10-05T09:54:54.825049Z","iopub.execute_input":"2023-10-05T09:54:54.825413Z","iopub.status.idle":"2023-10-05T09:54:54.831012Z","shell.execute_reply.started":"2023-10-05T09:54:54.825385Z","shell.execute_reply":"2023-10-05T09:54:54.829784Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"# Create x,y,z label names from coordinates","metadata":{}},{"cell_type":"code","source":"X = [f'x_right_hand_{i}' for i in range(21)] + [f'x_left_hand_{i}' for i in range(21)] + [f'x_pose_{i}' for i in POSE]\nY = [f'y_right_hand_{i}' for i in range(21)] + [f'y_left_hand_{i}' for i in range(21)] + [f'y_pose_{i}' for i in POSE]\nZ = [f'z_right_hand_{i}' for i in range(21)] + [f'z_left_hand_{i}' for i in range(21)] + [f'z_pose_{i}' for i in POSE]","metadata":{"execution":{"iopub.status.busy":"2023-10-05T09:55:01.580864Z","iopub.execute_input":"2023-10-05T09:55:01.581226Z","iopub.status.idle":"2023-10-05T09:55:01.588101Z","shell.execute_reply.started":"2023-10-05T09:55:01.581199Z","shell.execute_reply":"2023-10-05T09:55:01.58652Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":"Create feature columns from the extracted coordinates.","metadata":{}},{"cell_type":"code","source":"FEATURE_COLUMNS = X + Y + Z","metadata":{"execution":{"iopub.status.busy":"2023-10-05T09:55:08.327597Z","iopub.execute_input":"2023-10-05T09:55:08.327991Z","iopub.status.idle":"2023-10-05T09:55:08.332728Z","shell.execute_reply.started":"2023-10-05T09:55:08.327964Z","shell.execute_reply":"2023-10-05T09:55:08.331566Z"},"trusted":true},"execution_count":17,"outputs":[]},{"cell_type":"markdown","source":"Store ids of each coordinate labels to lists","metadata":{}},{"cell_type":"code","source":"X_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"x_\" in col]\nY_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"y_\" in col]\nZ_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"z_\" in col]\nRHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"right\" in col]\nLHAND_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"left\" in col]\nRPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in RPOSE]\nLPOSE_IDX = [i for i, col in enumerate(FEATURE_COLUMNS) if \"pose\" in col and int(col[-2:]) in LPOSE]","metadata":{"execution":{"iopub.status.busy":"2023-10-05T09:55:12.346033Z","iopub.execute_input":"2023-10-05T09:55:12.346374Z","iopub.status.idle":"2023-10-05T09:55:12.353767Z","shell.execute_reply.started":"2023-10-05T09:55:12.346348Z","shell.execute_reply":"2023-10-05T09:55:12.35278Z"},"trusted":true},"execution_count":18,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess and write the dataset as TFRecords","metadata":{}},{"cell_type":"markdown","source":"Using the extracted landmarks and phrases let us create new dataset files and write them as TFRecords.\n\n\nThis takes around 10 \nminutes. After this, loading the dataset will be faster for any future experiments.","metadata":{}},{"cell_type":"code","source":"# Set length of frames to 128\nFRAME_LEN = 128\n# Create directory to store the new data\nif not os.path.isdir(\"preprocessed\"):\n os.mkdir(\"preprocessed\")\nelse:\n shutil.rmtree(\"preprocessed\")\n os.mkdir(\"preprocessed\")\n# Loop through each file_id\nfor file_id in tqdm(dataset_df.file_id.unique()):\n # Parquet file name\n pq_file = f\"/kaggle/input/asl-fingerspelling/train_landmarks/{file_id}.parquet\"\n # Filter train.csv and fetch entries only for the relevant file_id\n file_df = dataset_df.loc[dataset_df[\"file_id\"] == file_id]\n # Fetch the parquet file\n parquet_df = pq.read_table(f\"/kaggle/input/asl-fingerspelling/train_landmarks/{str(file_id)}.parquet\",\n columns=['sequence_id'] + FEATURE_COLUMNS).to_pandas()\n # File name for the updated data\n tf_file = f\"preprocessed/{file_id}.tfrecord\"\n parquet_numpy = parquet_df.to_numpy()\n # Initialize the pointer to write the output of \n # each `for loop` below as a sequence into the file.\n with tf.io.TFRecordWriter(tf_file) as file_writer:\n # Loop through each sequence in file.\n for seq_id, phrase in zip(file_df.sequence_id, file_df.phrase):\n # Fetch sequence data\n frames = parquet_numpy[parquet_df.index == seq_id]\n \n # Calculate the number of NaN values in each hand landmark\n r_nonan = np.sum(np.sum(np.isnan(frames[:, RHAND_IDX]), axis = 1) == 0)\n l_nonan = np.sum(np.sum(np.isnan(frames[:, LHAND_IDX]), axis = 1) == 0)\n no_nan = max(r_nonan, l_nonan)\n \n if 2*len(phrase)\n \n \n \n
\n