{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"### CSS for notebook styling ###\nfrom IPython.core.display import HTML\n\nHTML('''\n\n''')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-12T10:01:48.049626Z","iopub.execute_input":"2023-05-12T10:01:48.050034Z","iopub.status.idle":"2023-05-12T10:01:48.084226Z","shell.execute_reply.started":"2023-05-12T10:01:48.050002Z","shell.execute_reply":"2023-05-12T10:01:48.083025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n

🤞 Google - American Sign Language Fingerspelling Recognition 🤞

\n
\n
\n
\n
\n \n
\n
\n\n
\n

\nGoal of the Competition\n
\n The goal of this competition is to detect and translate American Sign Language (ASL) fingerspelling into text. You will create a model trained on the largest dataset of its kind, released specifically for this competition. The data includes more than three million fingerspelled characters produced by over 100 Deaf signers captured via the selfie camera of a smartphone with a variety of backgrounds and lighting conditions.\n
\n\n

\nASL Fingerspelling Alphabet\n
\nFingerspelling is a way of spelling words using hand movements. The fingerspelling manual alphabet is used in sign language to spell out names of people and places for which there is not a sign. Fingerspelling can also be used to spell words for signs that the signer does not know the sign for, or to clarify a sign that is not known by the person reading the signer. Fingerspelling signs are often also incorporated into other ASL signs.\n
Source : Sign Language Forum

\n\n

\nHand Landmarks Detection\n
\nThe MediaPipe Hand Landmarker task lets you detect the landmarks of the hands in an image. You can use this Task to localize key points of the hands and render visual effects over the hands. This task operates on image data with a machine learning (ML) model as static data or a continuous stream and outputs hand landmarks in image coordinates, hand landmarks in world coordinates and handedness(left/right hand) of multiple detected hands. The hand landmark model bundle detects the keypoint localization of 21 hand-knuckle coordinates within the detected hand regions. The model was trained on approximately 30K real-world images, as well as several rendered synthetic hand models imposed over various backgrounds.\n\n

\n\n
Source: Mediapipe \n

\n
\n\n
\n\n
\n

Table of Contents

\n
\n
\n\n- Import Libraries\n- Train Dataset\n- Supplemental Metadata\n- Plot Hands in 2D\n- Plot Hands in 3D\n\n
\n\n
\n\n
\n

Import Library ↑

\n
\n
","metadata":{}},{"cell_type":"code","source":"!pip install mediapipe --quiet","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:01:48.098634Z","iopub.execute_input":"2023-05-12T10:01:48.099711Z","iopub.status.idle":"2023-05-12T10:02:06.232312Z","shell.execute_reply.started":"2023-05-12T10:01:48.09966Z","shell.execute_reply":"2023-05-12T10:02:06.230845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport mediapipe as mp\nimport plotly.graph_objects as go\n\nfrom colorama import Style, Fore\nblk = Style.BRIGHT + Fore.BLACK\nred = Style.BRIGHT + Fore.RED\nblu = Style.BRIGHT + Fore.BLUE\ncyan = Style.BRIGHT + Fore.CYAN\nres = Style.RESET_ALL","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-05-12T10:02:06.237825Z","iopub.execute_input":"2023-05-12T10:02:06.238226Z","iopub.status.idle":"2023-05-12T10:02:16.304093Z","shell.execute_reply.started":"2023-05-12T10:02:06.238188Z","shell.execute_reply":"2023-05-12T10:02:16.303105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_dir = '/kaggle/input/asl-fingerspelling'\ntrain_csv = f'{base_dir}/train.csv'\nsupplemental_csv = f'{base_dir}/supplemental_metadata.csv'","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:16.305456Z","iopub.execute_input":"2023-05-12T10:02:16.306172Z","iopub.status.idle":"2023-05-12T10:02:16.314237Z","shell.execute_reply.started":"2023-05-12T10:02:16.306138Z","shell.execute_reply":"2023-05-12T10:02:16.311633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
\n

Train Dataset ↑

\n
\n
","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(train_csv)\ntrain['path'] = base_dir +'/' + train['path']\ntrain = train.drop(['file_id'], axis=1)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:16.316692Z","iopub.execute_input":"2023-05-12T10:02:16.317032Z","iopub.status.idle":"2023-05-12T10:02:16.556328Z","shell.execute_reply.started":"2023-05-12T10:02:16.317004Z","shell.execute_reply":"2023-05-12T10:02:16.555501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Top 10 Phrase in Training Dataset

\n
","metadata":{}},{"cell_type":"code","source":"sns.set(style=\"whitegrid\")\nfig, ax = plt.subplots(figsize=(8, 8))\n\nsns.barplot(\n y=train[\"phrase\"].value_counts().head(10).sort_values(ascending=False).index,\n x=train[\"phrase\"].value_counts().head(10).sort_values(ascending=False),\n ax=ax,\n)\n\nax.set_title(\"Top 10 Phrase in Training Dataset\")\nax.set_xlabel(\"Number of Training Examples\")\nax.set_ylabel(\"Phrase\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:16.557371Z","iopub.execute_input":"2023-05-12T10:02:16.55809Z","iopub.status.idle":"2023-05-12T10:02:17.095042Z","shell.execute_reply.started":"2023-05-12T10:02:16.558042Z","shell.execute_reply":"2023-05-12T10:02:17.094008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Parquet File of Top 1 Phrase in Train Dataset

\n
","metadata":{}},{"cell_type":"code","source":"select_train = \"surprise az\"\ntrain_example = train.query('phrase == @select_train')['path'].values[0]\nselect_landmark_train = pd.read_parquet(train_example)\nselect_landmark_train","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:17.096445Z","iopub.execute_input":"2023-05-12T10:02:17.097474Z","iopub.status.idle":"2023-05-12T10:02:34.238707Z","shell.execute_reply.started":"2023-05-12T10:02:17.097441Z","shell.execute_reply":"2023-05-12T10:02:34.237624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One parquet path consist of more than one sequence_id, here I want to show a landmark for `surprise az` but in one parquet has many sequence_id. It means one parquet can consist of different labels.","metadata":{}},{"cell_type":"code","source":"seq_target_train = train.query('phrase == @select_train')['sequence_id'].values[0]\nprint(f\"{blu}[+]{blk} Sequence ID : {blu}{seq_target_train}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:34.240116Z","iopub.execute_input":"2023-05-12T10:02:34.240453Z","iopub.status.idle":"2023-05-12T10:02:34.254574Z","shell.execute_reply.started":"2023-05-12T10:02:34.240424Z","shell.execute_reply":"2023-05-12T10:02:34.253406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seq_df_train = select_landmark_train.query('index == @seq_target_train')\nseq_df_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:34.256438Z","iopub.execute_input":"2023-05-12T10:02:34.256826Z","iopub.status.idle":"2023-05-12T10:02:34.640179Z","shell.execute_reply.started":"2023-05-12T10:02:34.256795Z","shell.execute_reply":"2023-05-12T10:02:34.639178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def x_y_z(column_names):\n x = [col for col in column_names if col.startswith('x')]\n y = [col for col in column_names if col.startswith('y')]\n z = [col for col in column_names if col.startswith('z')]\n return x,y,z","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-05-12T10:02:34.641329Z","iopub.execute_input":"2023-05-12T10:02:34.641635Z","iopub.status.idle":"2023-05-12T10:02:34.647898Z","shell.execute_reply.started":"2023-05-12T10:02:34.64161Z","shell.execute_reply":"2023-05-12T10:02:34.646863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def type_of_landmark(example_landmark):\n body_parts = set()\n for column in example_landmark.columns:\n parts = column.split('_')\n if len(parts) >= 2:\n if parts[1] == 'right':\n body_parts.add('right_hand')\n elif parts[1] == 'left':\n body_parts.add('left_hand')\n else :\n body_parts.add(parts[1])\n return body_parts","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-05-12T10:02:34.652567Z","iopub.execute_input":"2023-05-12T10:02:34.657309Z","iopub.status.idle":"2023-05-12T10:02:34.664072Z","shell.execute_reply.started":"2023-05-12T10:02:34.657264Z","shell.execute_reply":"2023-05-12T10:02:34.662715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Check Landmarks, Frames, and (X, Y, Z) Points

\n
","metadata":{}},{"cell_type":"code","source":"unique_frames = seq_df_train[\"frame\"].nunique()\ntype_landmark_train = type_of_landmark(seq_df_train)\n\nface_train = [col for col in seq_df_train.columns if \"face\" in col]\nright_hand_train = [col for col in seq_df_train.columns if \"right_hand\" in col]\nleft_hand_train = [col for col in seq_df_train.columns if \"left_hand\" in col]\npose_train = [col for col in seq_df_train.columns if \"pose\" in col]\n\nx_face_train, y_face_train, z_face_train = x_y_z(face_train)\nx_right_hand_train, y_right_hand_train, z_right_hand_train = x_y_z(right_hand_train)\nx_left_hand_train, y_left_hand_train, z_left_hand_train = x_y_z(left_hand_train)\nx_pose_train, y_pose_train, z_pose_train = x_y_z(pose_train)\n\nprint(f\"{cyan}{'='*20} ( Train Dataset) {'='*20}\")\nprint(f\"{blk}Landmark file for sequence_id {red}{seq_target_train}{blk} has {red}{unique_frames}{blk} frames \")\nprint(f\"{blk}This landmark has {red}{len(type_landmark_train)} {blk}types of landmarks and consists of {red}{type_landmark_train}\")\nprint(f\"\\n{blu}[+]{blk} {blk}Face landmark has {red}{len(face_train)} {blk}points in x : {red}{len(x_face_train)} points, {blk}y : {red}{len(y_face_train)} points, {blk}z : {red}{len(z_face_train)} points\")\nprint(f\"{blu}[+]{blk} {blk}Right hand landmark has {red}{len(right_hand_train)} {blk}points in x : {red}{len(x_right_hand_train)} points, {blk}y : {red}{len(y_right_hand_train)} points, {blk}z : {red}{len(z_right_hand_train)} points\")\nprint(f\"{blu}[+]{blk} {blk}Left hand landmark has {red}{len(left_hand_train)} {blk}points in x : {red}{len(x_left_hand_train)} points, {blk}y : {red}{len(y_left_hand_train)} points, {blk}z : {red}{len(z_left_hand_train)} points\")\nprint(f\"{blu}[+]{blk} {blk}Pose landmark has {red}{len(pose_train)} {blk}points in x : {red}{len(x_pose_train)} points, {blk}y : {red}{len(y_pose_train)} points, {blk}z : {red}{len(z_pose_train)} points\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-12T10:02:34.666113Z","iopub.execute_input":"2023-05-12T10:02:34.666576Z","iopub.status.idle":"2023-05-12T10:02:34.694711Z","shell.execute_reply.started":"2023-05-12T10:02:34.666537Z","shell.execute_reply":"2023-05-12T10:02:34.693515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
\n

Supplemental Metadata ↑

\n
\n
","metadata":{}},{"cell_type":"code","source":"supplemental = pd.read_csv(supplemental_csv)\nsupplemental['path'] = base_dir +'/' + supplemental['path']\nsupplemental = supplemental.drop(['file_id'], axis=1)\nsupplemental.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:34.696482Z","iopub.execute_input":"2023-05-12T10:02:34.696952Z","iopub.status.idle":"2023-05-12T10:02:34.845181Z","shell.execute_reply.started":"2023-05-12T10:02:34.696911Z","shell.execute_reply":"2023-05-12T10:02:34.843895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Top 10 Phrase in Supplemental Metadata

\n
","metadata":{}},{"cell_type":"code","source":"sns.set(style=\"whitegrid\")\n\nfig, ax = plt.subplots(figsize=(8, 8))\n\nsns.barplot(\n y=supplemental[\"phrase\"].value_counts().head(10).sort_values(ascending=False).index,\n x=supplemental[\"phrase\"].value_counts().head(10).sort_values(ascending=False),\n ax=ax,\n)\n\nax.set_title(\"Top 10 Phrase in Supplemental Dataset\")\nax.set_xlabel(\"Number of Examples\")\nax.set_ylabel(\"Phrase\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:34.847162Z","iopub.execute_input":"2023-05-12T10:02:34.847605Z","iopub.status.idle":"2023-05-12T10:02:35.340659Z","shell.execute_reply.started":"2023-05-12T10:02:34.847564Z","shell.execute_reply":"2023-05-12T10:02:35.339473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Parquet File of Top 1 Phrase in Supplemental Metadata

\n
","metadata":{}},{"cell_type":"code","source":"select_supp = \"why do you ask silly questions\"\nsupp_example = supplemental.query('phrase == @select_supp')['path'].values[0]\nselect_landmark_supp = pd.read_parquet(supp_example)\nselect_landmark_supp","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:35.342485Z","iopub.execute_input":"2023-05-12T10:02:35.343147Z","iopub.status.idle":"2023-05-12T10:02:52.502709Z","shell.execute_reply.started":"2023-05-12T10:02:35.343108Z","shell.execute_reply":"2023-05-12T10:02:52.501666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Same as **Train Dataset**, one parquet file also has more than one sequence_id","metadata":{}},{"cell_type":"code","source":"seq_target_supp = supplemental.query('phrase == @select_supp')['sequence_id'].values[0]\nprint(f\"{blu}[+]{blk} Sequence ID : {blu}{seq_target_supp}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:52.504318Z","iopub.execute_input":"2023-05-12T10:02:52.504977Z","iopub.status.idle":"2023-05-12T10:02:52.516453Z","shell.execute_reply.started":"2023-05-12T10:02:52.504939Z","shell.execute_reply":"2023-05-12T10:02:52.51525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seq_df_supp = select_landmark_supp.query('index == @seq_target_supp')\nseq_df_supp.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:52.517596Z","iopub.execute_input":"2023-05-12T10:02:52.517944Z","iopub.status.idle":"2023-05-12T10:02:52.642618Z","shell.execute_reply.started":"2023-05-12T10:02:52.517915Z","shell.execute_reply":"2023-05-12T10:02:52.641504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Check Landmarks, Frames, and (X, Y, Z) Points

\n
","metadata":{}},{"cell_type":"code","source":"unique_frames = seq_df_supp[\"frame\"].nunique()\ntype_landmark_supp = type_of_landmark(seq_df_supp)\n\nface_supp = [col for col in seq_df_supp.columns if \"face\" in col]\nright_hand_supp = [col for col in seq_df_supp.columns if \"right_hand\" in col]\nleft_hand_supp = [col for col in seq_df_supp.columns if \"left_hand\" in col]\npose_supp = [col for col in seq_df_supp.columns if \"pose\" in col]\n\nx_face_supp, y_face_supp, z_face_supp = x_y_z(face_supp)\nx_right_hand_supp, y_right_hand_supp, z_right_hand_supp = x_y_z(right_hand_supp)\nx_left_hand_supp, y_left_hand_supp, z_left_hand_supp = x_y_z(left_hand_supp)\nx_pose_supp, y_pose_supp, z_pose_supp = x_y_z(pose_supp)\n\nprint(f\"{cyan}{'='*20} ( Supplemental Dataset) {'='*20}\")\nprint(f\"{blk}Landmark file for sequence_id {red}{seq_target_supp}{blk} has {red}{unique_frames}{blk} frames \")\nprint(f\"{blk}This landmark has {red}{len(type_landmark_supp)} {blk}types of landmarks and consists of {red}{type_landmark_supp}\")\nprint(f\"\\n{blu}[+]{blk} Face landmark has {red}{len(face_supp)} {blk}points in x : {red}{len(x_face_supp)} points, {blk}y : {red}{len(y_face_supp)} points, {blk}z : {red}{len(z_face_supp)} points\")\nprint(f\"{blu}[+]{blk} Right hand landmark has {red}{len(right_hand_supp)} {blk}points in x : {red}{len(x_right_hand_supp)} points, {blk}y : {red}{len(y_right_hand_supp)} points, {blk}z : {red}{len(z_right_hand_supp)} points\")\nprint(f\"{blu}[+]{blk} Left hand landmark has {red}{len(left_hand_supp)} {blk}points in x : {red}{len(x_left_hand_supp)} points, {blk}y : {red}{len(y_left_hand_supp)} points, {blk}z : {red}{len(z_left_hand_supp)} points\")\nprint(f\"{blu}[+]{blk} Pose landmark has {red}{len(pose_supp)} {blk}points in x : {red}{len(x_pose_supp)} points, {blk}y : {red}{len(y_pose_supp)} points, {blk}z : {red}{len(z_pose_supp)} points\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-12T10:02:52.644111Z","iopub.execute_input":"2023-05-12T10:02:52.644877Z","iopub.status.idle":"2023-05-12T10:02:52.664391Z","shell.execute_reply.started":"2023-05-12T10:02:52.644837Z","shell.execute_reply":"2023-05-12T10:02:52.663049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
\n

Plot Hands in 2D ↑

\n
\n
","metadata":{}},{"cell_type":"code","source":"mp_hands = mp.solutions.hands","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:52.665737Z","iopub.execute_input":"2023-05-12T10:02:52.666046Z","iopub.status.idle":"2023-05-12T10:02:52.679957Z","shell.execute_reply.started":"2023-05-12T10:02:52.666019Z","shell.execute_reply":"2023-05-12T10:02:52.678873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_plot(seq, frame, x_col, y_col, z_col, df):\n x = df.query(\"sequence_id == @seq and frame == @frame\")[x_col].iloc[0].values\n y = df.query(\"sequence_id == @seq and frame == @frame\")[y_col].iloc[0].values\n z = df.query(\"sequence_id == @seq and frame == @frame\")[z_col].iloc[0].values\n landmark_idx = [int(col.split('_')[-1]) for col in df.query(\"sequence_id == @seq and frame == @frame\")[x_col].columns]\n dataframe = pd.DataFrame({'x': x, 'y': y,'z': z, 'landmark_idx': landmark_idx})\n return dataframe","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:52.681586Z","iopub.execute_input":"2023-05-12T10:02:52.681937Z","iopub.status.idle":"2023-05-12T10:02:52.693459Z","shell.execute_reply.started":"2023-05-12T10:02:52.681909Z","shell.execute_reply":"2023-05-12T10:02:52.691036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Training Data Plot Hands of \"surprise az\" Phrase

\n
","metadata":{}},{"cell_type":"code","source":"frame = 12\nleft_hand_train = data_plot(seq_target_train, frame, x_left_hand_train, y_left_hand_train, z_left_hand_train, seq_df_train)\nright_hand_train = data_plot(seq_target_train,frame, x_right_hand_train, y_right_hand_train, z_right_hand_train, seq_df_train)","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:52.694932Z","iopub.execute_input":"2023-05-12T10:02:52.695275Z","iopub.status.idle":"2023-05-12T10:02:53.135633Z","shell.execute_reply.started":"2023-05-12T10:02:52.695247Z","shell.execute_reply":"2023-05-12T10:02:53.134483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\nax.scatter(right_hand_train[\"x\"], right_hand_train[\"y\"])\nax.scatter(left_hand_train[\"x\"], left_hand_train[\"y\"])\n\nfor connection in mp_hands.HAND_CONNECTIONS:\n point_a = connection[0]\n point_b = connection[1]\n x1, y1 = right_hand_train.query(\"landmark_idx == @point_a\")[[\"x\", \"y\"]].values[0]\n x2, y2 = right_hand_train.query(\"landmark_idx == @point_b\")[[\"x\", \"y\"]].values[0]\n plt.plot([x1, x2], [y1, y2], color=\"red\")\n x3, y3 = left_hand_train.query(\"landmark_idx == @point_a\")[[\"x\", \"y\"]].values[0]\n x4, y4 = left_hand_train.query(\"landmark_idx == @point_b\")[[\"x\", \"y\"]].values[0]\n plt.plot([x3, x4], [y3, y4], color=\"red\")\n \nax.set_title(select_train)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:53.13755Z","iopub.execute_input":"2023-05-12T10:02:53.138004Z","iopub.status.idle":"2023-05-12T10:02:53.772498Z","shell.execute_reply.started":"2023-05-12T10:02:53.137966Z","shell.execute_reply":"2023-05-12T10:02:53.771295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Supplement Metadata Plot \"why do you ask silly questions\" Phrase

\n
","metadata":{}},{"cell_type":"code","source":"frame = 140\nleft_hand_supp = data_plot(seq_target_supp, frame, x_left_hand_supp, y_left_hand_supp, z_left_hand_supp, seq_df_supp)\nright_hand_supp = data_plot(seq_target_supp, frame, x_right_hand_supp, y_right_hand_supp, z_right_hand_supp, seq_df_supp)","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:53.773772Z","iopub.execute_input":"2023-05-12T10:02:53.774102Z","iopub.status.idle":"2023-05-12T10:02:54.179759Z","shell.execute_reply.started":"2023-05-12T10:02:53.774075Z","shell.execute_reply":"2023-05-12T10:02:54.178663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\nax.scatter(right_hand_supp[\"x\"], right_hand_supp[\"y\"])\nax.scatter(left_hand_supp[\"x\"], left_hand_supp[\"y\"])\n\nfor connection in mp_hands.HAND_CONNECTIONS:\n point_a = connection[0]\n point_b = connection[1]\n x1, y1 = right_hand_supp.query(\"landmark_idx == @point_a\")[[\"x\", \"y\"]].values[0]\n x2, y2 = right_hand_supp.query(\"landmark_idx == @point_b\")[[\"x\", \"y\"]].values[0]\n plt.plot([x1, x2], [y1, y2], color=\"red\")\n x3, y3 = left_hand_supp.query(\"landmark_idx == @point_a\")[[\"x\", \"y\"]].values[0]\n x4, y4 = left_hand_supp.query(\"landmark_idx == @point_b\")[[\"x\", \"y\"]].values[0]\n plt.plot([x3, x4], [y3, y4], color=\"red\")\n \nax.set_title(select_supp)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:54.18127Z","iopub.execute_input":"2023-05-12T10:02:54.18159Z","iopub.status.idle":"2023-05-12T10:02:54.823058Z","shell.execute_reply.started":"2023-05-12T10:02:54.181563Z","shell.execute_reply":"2023-05-12T10:02:54.82198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
\n

Plot Hands in 3D ↑

\n
\n
","metadata":{}},{"cell_type":"code","source":"def plot_3d(right, left, title):\n right['hand'] = 'right'\n left['hand'] = 'left'\n df = pd.concat([right, left])\n\n traces = []\n scatter_trace = go.Scatter3d(\n x=df['x'],\n y=df['y'],\n z=df['z'],\n mode='markers',\n marker=dict(\n size=5,\n color='blue',\n opacity=0.8\n ),\n name='Landmarks'\n )\n traces.append(scatter_trace)\n\n line_trace = go.Scatter3d(\n x=[],\n y=[],\n z=[],\n mode='lines',\n line=dict(\n color='red',\n width=2\n ),\n name='Connections'\n )\n\n for connection in mp_hands.HAND_CONNECTIONS:\n point_a = connection[0]\n point_b = connection[1]\n x1, y1, z1 = df.query(\"landmark_idx == @point_a\")[[\"x\", \"y\", \"z\"]].values[0]\n x2, y2, z2 = df.query(\"landmark_idx == @point_b\")[[\"x\", \"y\", \"z\"]].values[0]\n line_trace['x'] += (x1, x2, None)\n line_trace['y'] += (y1, y2, None)\n line_trace['z'] += (z1, z2, None)\n\n traces.append(line_trace)\n\n fig = go.Figure(data=traces)\n _text = f'3D Plot of \"{title}\"'\n \n fig.update_layout(\n scene=dict(\n xaxis_title='X',\n yaxis_title='Y',\n zaxis_title='Z'\n ),\n showlegend=True,\n title = {\n 'text' : _text,\n 'xanchor':'center',\n 'yanchor':'top',\n 'x':0.5,\n }\n )\n fig.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:54.824757Z","iopub.execute_input":"2023-05-12T10:02:54.825863Z","iopub.status.idle":"2023-05-12T10:02:54.843111Z","shell.execute_reply.started":"2023-05-12T10:02:54.825818Z","shell.execute_reply":"2023-05-12T10:02:54.842016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Train Dataset Plot in 3D

\n
","metadata":{"execution":{"iopub.status.busy":"2023-05-12T09:51:13.748822Z","iopub.execute_input":"2023-05-12T09:51:13.749358Z","iopub.status.idle":"2023-05-12T09:51:13.760349Z","shell.execute_reply.started":"2023-05-12T09:51:13.749277Z","shell.execute_reply":"2023-05-12T09:51:13.758385Z"}}},{"cell_type":"code","source":"plot_3d(right_hand_train, left_hand_train, select_train)","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:54.84457Z","iopub.execute_input":"2023-05-12T10:02:54.844959Z","iopub.status.idle":"2023-05-12T10:02:55.409601Z","shell.execute_reply.started":"2023-05-12T10:02:54.844927Z","shell.execute_reply":"2023-05-12T10:02:55.408821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Supplemental Metadata Plot in 3D

\n
","metadata":{}},{"cell_type":"code","source":"plot_3d(right_hand_supp, left_hand_supp, select_supp)","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:02:55.410733Z","iopub.execute_input":"2023-05-12T10:02:55.411231Z","iopub.status.idle":"2023-05-12T10:02:55.571318Z","shell.execute_reply.started":"2023-05-12T10:02:55.411203Z","shell.execute_reply":"2023-05-12T10:02:55.569722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n Reference : @robikscube\n
\n\n

\n  ðŸ™‚ PLEASE UPVOTE IF YOU LIKE MY WORK 🙂 

\n

","metadata":{}}]}