Chapter 2: Vision-Based Sign Language Recognition — RGB/Depth/MediaPipe/OpenPose

2.1 Processing Stages of Vision-Based Sign Language Recognition

Vision-Based Sign Language Recognition (V-SLR) recognizes sign-language signals from camera video. Unlike sensor-based approaches that rely on data gloves, inertial sensors, or electromyography sensors, V-SLR allows signers to sign naturally without wearing any special equipment. The WIA standard adopts V-SLR as the primary recommended approach but recommends supplementary modalities for high-stakes domains such as medical care and law. The simulator Data Format panel exposes the vision input data schema for verification.

The standard processing stages of V-SLR are as follows.

1. Capture: Raw frames are acquired from RGB, depth, and infrared cameras. For KSL recognition, the signer's body from head to waist must be within the frame, and the camera position must allow both hands to be visible at all times.
2. Pre-processing: Brightness, hue, and resolution are normalized; noise is removed; background subtraction may be applied; and the signer region is cropped.
3. Body Part Detection: Hands, face, and upper body are detected. MediaPipe Hands, MediaPipe FaceMesh, and MediaPipe Pose are de-facto industry standards.
4. Keypoint Extraction: Hand, face, and body keypoint coordinates are extracted.
5. Feature Vector Construction: Keypoint coordinates are combined with derived quantities — angles, velocities, accelerations — into feature vectors.
6. Sequence Classification: CNN, LSTM, or Transformer time-series models output sign-language lexical sequences.

2.1.1 Recommended Camera Specifications

These specifications are minimum recommendations for accurate sign-language recognition. In medical and legal environments, the signer should face the camera frontally with both hands held within a 50 cm radius in front of the chest for highest accuracy. In actual deployment the signer often moves and the camera is at an oblique angle; the standard recommends that models be trained for stable performance within ±30° of frontal viewing angle.

2.2 Keypoint Extraction with MediaPipe

MediaPipe¹⁾ is an open-source cross-platform real-time perception pipeline framework released by Google in 2019. It operates at 30 fps or higher on mobile GPUs and CPUs and provides four solutions essential to sign language recognition.

2.2.1 MediaPipe Hands — 21 Hand Keypoints

MediaPipe Hands extracts 21 keypoints per hand (wrist, four MCP joints, four PIP joints, four DIP joints, four fingertips, and four additional thumb keypoints) from a single-camera input. Each keypoint is provided as a 3D (x, y, z) coordinate, where z is a relative distance from the camera estimated regressively from monocular input without a depth camera. MediaPipe Hands consists of a two-stage pipeline: BlazePalm Detector (hand-region detection) and Hand Landmark Model (keypoint regression). Inference latency is approximately 4 ms per hand on a Pixel 6.

2.2.2 MediaPipe Pose — 33 Body Keypoints

MediaPipe Pose extracts 33 keypoints from upper and lower body — nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles, and so on. Sign language recognition primarily uses the 13 upper-body keypoints (nose, both shoulders, both elbows, both wrists). MediaPipe Pose uses the BlazePose model and operates at 30 fps or higher on mobile GPUs.

2.2.3 MediaPipe FaceMesh — 468 Face Landmarks

MediaPipe FaceMesh extracts 468 landmarks across the facial surface, including eyebrows, eyes, nose, mouth, chin, and cheeks. It is used in sign language recognition to detect non-manual markers (NMMs); specifically, eyebrow position (interrogative marker), mouth morpheme, and gaze direction are most important. FaceMesh combines BlazeFace + Face Mesh and operates at 30 fps or higher on mobile GPUs.

2.2.4 MediaPipe Holistic — Unified Solution

MediaPipe Holistic integrates Pose, Hands, and FaceMesh into a single pipeline that simultaneously extracts 21 + 21 + 33 + 468 = 543 keypoints. The WIA standard recommends MediaPipe Holistic as the standard input specification for KSL recognition.

2.3 Keypoint Extraction with OpenPose

OpenPose²⁾ is a real-time multi-person pose-estimation library released by Carnegie Mellon University (CMU) in 2017. It estimates multiple people's poses simultaneously using Part Affinity Fields (PAFs). OpenPose provides the BODY_25 model (25 body keypoints), Hand model (21 hand keypoints × two hands), and Face model (70 face keypoints). OpenPose is slightly more accurate than MediaPipe but requires substantial GPU memory and compute and is thus impractical for mobile devices. The WIA standard recommends OpenPose for precision recognition in server environments and research settings, and recommends MediaPipe for mobile environments.

2.3.1 MediaPipe vs. OpenPose Comparison

2.4 Depth Cameras — Leveraging Depth Information

Accurate estimation of fingertip z-coordinates (camera distance) is difficult from monocular RGB alone. Depth cameras directly measure per-pixel depth information, enabling accurate three-dimensional recovery of hand and body positions. The WIA standard recommends the following three depth-camera technologies.

2.4.1 Time-of-Flight (ToF)

ToF sensors emit infrared (IR) pulses and measure their reflection time to compute distance. Microsoft Azure Kinect (released 2019), Intel RealSense L515, and the LiDAR sensors integrated into recent smartphones (iPad Pro LiDAR, iPhone 12 Pro and later) are representative examples. ToF provides 1–5 cm accuracy at 1–5 m distance. ToF is the optimal choice for KSL recognition environments.

2.4.2 Structured Light

Structured-light methods project a known-pattern IR illumination and compute depth from the deformation of the projected pattern on the object's surface. Microsoft Kinect v1 (released 2010) and Intel RealSense D435 are representative. Accuracy is higher at short distances than ToF.

2.4.3 Stereo Vision

Stereo vision places two cameras at a known baseline distance and computes depth from the disparity between the two images. ZED, Stereolabs, and Intel RealSense D435 use this approach. No separate IR illumination is required, so stereo cameras work outdoors, but textureless surfaces reduce accuracy.

2.4.4 KAIST Gesture Recognition Lab Case Study

The Gesture Recognition Lab at the KAIST AI Graduate School built a Korean Sign Language hand-gesture dataset using Azure Kinect (ToF). The use of depth information yielded approximately 35% better fingertip-location accuracy than monocular RGB alone. The WIA standard adopts the KAIST data-collection procedure as the "Depth-Camera-Based KSL Dataset Construction Guideline."

2.5 Data Format — SignLanguageFrame JSON Schema

The WIA Sign Language Recognition Standard's data format is shown below. The format is validated in the simulator Data Format panel.

{
  "frame_id": 1234,
  "timestamp_us": 1700000000000000,
  "ptp_clock_id": "0x001122FFFE334455",
  "sign_language": "KSL",
  "input_types": ["VIDEO", "SKELETON", "DEPTH"],
  "video": {
    "width": 1920, "height": 1080,
    "format": "BGR24", "fps": 60,
    "encoding": "H.264"
  },
  "skeleton": {
    "source": "MEDIAPIPE_HOLISTIC",
    "body": [[x,y,z], ... 33 keypoints],
    "left_hand": [[x,y,z], ... 21 keypoints],
    "right_hand": [[x,y,z], ... 21 keypoints],
    "face": [[x,y,z], ... 468 landmarks]
  },
  "depth": {
    "source": "AZURE_KINECT",
    "width": 640, "height": 480,
    "depth_format": "MM_16BIT",
    "min_mm": 500, "max_mm": 5000
  }
}

The schema is serialized per frame; at 60 fps it generates 60 JSON objects per second. To save bandwidth the standard also recommends Protocol Buffers (protobuf) binary encoding and MessagePack encoding.

2.6 Korea-Specific Vision Infrastructure

The Republic of Korea has the following infrastructure for vision-based sign-language recognition research.

• Electronics and Telecommunications Research Institute (ETRI): Applied MediaPipe Holistic in the "Su-eo Tok" mobile interpretation terminal. As of 2024, achieved 95.2% isolated SLR accuracy on the KSL 419-vocabulary subset.
• National Information Society Agency (NIA) AI Hub: The "AI Training Korean Sign Language Video" dataset was filmed at 1920×1080@30 fps RGB and Azure Kinect 640×480@30 fps depth. Approximately 1,200 signers, 110,000 lexical entries, and 1,000 total hours.
• Korea Advanced Institute of Science and Technology (KAIST): The Gesture Recognition Lab at the AI Graduate School released a KSL dataset based on Azure Kinect together with a Transformer-based recognition model.
• Korea Institute of Science and Technology (KIST): Applied OpenPose in the sign-language recognition module of the "ETRI-Robi" humanoid robot.
• Seoul National University Department of Computer Science and Engineering, Yonsei University Department of Artificial Intelligence: Publish academic research on Korean Sign Language recognition. KCC and KIISE proceedings include annual sign-language recognition papers.
• Telecommunications Technology Association (TTA): Defines the vision-input format in TTAK.KO-10.1156 (Korean Sign Language Data Annotation Standard).
• Korean Standards Association (KSA) and Korean Agency for Technology and Standards (KATS): Manage Korean editions of vision-recognition system conformance test standards.
• Korea Internet and Security Agency (KISA): The "Sign Language Video Personal-Information De-identification Guideline" (2023) defines de-identification procedures using face mosaicking and skeleton extraction.
• Korea Laboratory Accreditation Scheme (KOLAS): Accredits vision-system accuracy testing bodies.
• Korea Communications Agency (KCA): Evaluates broadcast sign-language recognition systems for image-quality and frame-rate conformance.
• Korea Electronics Technology Institute (KETI): Developed the vision module of the "KETI-Hand" real-time interpretation engine.
• National Institute of the Korean Language (NIKL): Provides automatic annotation of the Korean Sign Language Dictionary's video assets (approximately 16,000 lexical entries) using MediaPipe Holistic.
• National Institute of Special Education (NISE): Evaluates students' KSL learning progress through vision analysis of pedagogical video at schools for the Deaf.
• Korean Association of the Deaf (KAD): Operates a Deaf-led user evaluation procedure for vision recognition systems.
• Korea Communications Commission: Recommends a WER 30% or lower accuracy threshold for automatic sign-language captioning systems used in broadcasting.

2.7 Limitations of Vision-Based Recognition and How to Address Them

Vision-based sign-language recognition has the following limitations; the WIA standard offers mitigation guidance for each.

2.7.1 Occlusion

When the two hands overlap or are obscured by parts of the body, it is hard to estimate finger positions from a single camera. The standard recommends three mitigations.

• Multi-camera configuration: Three cameras (front, left, right) operate simultaneously. Regions occluded from one viewpoint are likely visible from another.
• Depth + Skeleton augmentation: Combining depth information with temporal keypoint tracking compensates for transient occlusion.
• Inertial-sensor (IMU) auxiliary input: Wrist-mounted IMUs measure hand motion even when the camera is occluded.

2.7.2 Illumination Variation

Indoor lighting, outdoor sunlight, backlighting, and other lighting conditions vary RGB colour and reduce accuracy. The standard recommends including video captured under diverse illumination conditions (100–5,000 lux) during dataset construction, and activating automatic white balance (AWB) and automatic exposure (AE) at inference time.

2.7.3 Background Noise

Complex backgrounds (crowds, moving objects, skin-coloured surfaces) cause false-positive hand detection. The standard recommends background subtraction and signer-region segmentation (e.g., Mask R-CNN, U-Net).

2.7.4 Signer Variation

The same sign varies by signer in hand size, body proportions, signing speed, and signing style. The standard recommends datasets that include diverse signers (gender, age, body proportions) and signer-independent evaluation protocols. The NIA AI Hub dataset includes approximately 1,200 signers and thus meets this criterion.

2.7.5 Camera Angle Variation

In real deployment, the camera is not always positioned frontally to the signer. Ceiling-mounted conference cameras, driver-monitoring cameras in autonomous vehicles, and hand-held mobile cameras may have non-frontal views. The standard defines ≥ 95% accuracy within ±30° of frontal viewing as the "standard operating range" and ±45° as the "extended operating range." Beyond ±45°, confidence drops sharply and the system reports "out of range."

2.7.6 Fine-Grained Fingertip Distinction

Korean Sign Language contains many lexical pairs distinguished by sub-knuckle (~5 cm) differences in finger position. For example, the signs for SCHOOL and STUDY differ by approximately 5 cm in finger position. Sub-pixel accuracy in hand-keypoint coordinates is required for fine-grained distinction. MediaPipe Hands provides approximately ±0.5 pixel accuracy in image coordinates, corresponding to about 1 mm at 1920×1080 resolution.

2.7.7 Bimanual Coordination

Approximately 60% of Korean Sign Language vocabulary uses both hands simultaneously. Vocabulary in which both hands assume the same shape and movement (e.g., FAMILY, HARMONY) is symmetrical; vocabulary in which the two hands play different roles (e.g., STUDY, where one hand is a book and the other writes) is asymmetrical. Vision recognition systems must extract keypoints from the two hands independently and learn their interaction at the sequence-model stage.

2.8 Real-Time Performance Requirements

2.8.1 Latency Budget

A 150 ms or shorter latency is the threshold at which human signers perceive the interaction as natural. NHK STRL research shows that signers begin to perceive latency above 200 ms. The standard defines "real time" as ≤ 150 ms on mobile and ≤ 80 ms on desktop.

2.8.2 Throughput

Broadcast, education, and medical settings may need to operate many concurrent sign-recognition sessions simultaneously. The standard recommends the following throughput levels:

• Single GPU (NVIDIA A100): 100 concurrent sessions at 30 fps
• Single GPU (RTX 3080): 30 concurrent sessions at 30 fps
• Mobile GPU (Pixel 6 Tensor G1): 1 session at 30 fps
• Mobile NPU (Apple A17 Pro): 1 session at 60 fps

2.8.3 Accuracy Targets

These targets are based on Korean Sign Language. For other sign languages, the values vary by dataset size and diversity. The WIA standard uses signer-independent accuracy as the primary evaluation metric.

2.9 Korean Sign Language Vision Recognition Reference Implementations

2.9.1 ETRI "Su-eo Tok" Mobile Interpretation Terminal

ETRI's "Su-eo Tok" is an Android and iOS Korean sign-language interpretation app that uses MediaPipe Holistic plus an internally trained Transformer model. It reports 95.2% isolated-SLR accuracy on a 419-vocabulary subset and 28% continuous-SLR WER. As of 2024 it has accumulated approximately 120,000 users and is integrated with the Ministry of Health and Welfare's Deaf-only telecommunication relay service "107 Son-Mal-Ieum Center," enabling Deaf users to converse with hearing users in real time from a mobile device.

2.9.2 KAIST Continuous SLR Model

The KAIST AI Graduate School released "TFNet" (Temporal Fusion Network for Continuous KSL) in 2024. On a 1,000-vocabulary KSL dataset captured with Azure Kinect it achieved WER 21.4%, and on RWTH-PHOENIX-Weather 2014T it achieved WER 22.8%, approaching state of the art.

2.9.3 Use of the NIA AI Hub Dataset

The NIA AI Hub Korean Sign Language Video dataset has accumulated approximately 8,500 downloads since its 2021 release. Representative use cases include (i) graduate research at KAIST, Seoul National University, Yonsei University, and Hanyang University; (ii) pretraining of in-house models at Naver Clova and LG AI Research; (iii) the Ministry of Education classroom interpretation system pilot programme; and (iv) automatic captioning assistance tools at KBS and EBS.

2.9.4 KBS 4K UHD Sign Language Broadcasting

KBS introduced sign-language interpretation into 4K UHD terrestrial broadcasts in 2022. A sign-language interpreter's window appears in the lower-right of the screen, with the interpreter filmed against a uniform background colour aligned to the BT.709 colour space. This is a deliberately designed "broadcast standard capture environment" that allows automatic recognition systems to segment the signer easily. The WIA standard adopts the KBS environment as the "Broadcast Grade" input specification.

2.99 Closing Remarks and Korean Infrastructure Cross-Reference

The technical specifications described in this chapter all serve the broader goal of guaranteeing the everyday communication rights of the Deaf community. Korean Sign Language is, under Article 2 of the Korean Sign Language Act, a public language with status equal to Korean, and the WIA Sign Language Recognition Standard provides the technical underpinning for this legal status.

The standard operates on the Korean national infrastructure: the Korean Association of the Deaf (KAD), the National Institute of the Korean Language (NIKL), the National Information Society Agency (NIA), the Electronics and Telecommunications Research Institute (ETRI), the Korea Advanced Institute of Science and Technology (KAIST), the Korea Institute of Science and Technology Information (KISTI), the Telecommunications Technology Association (TTA), the Korean Standards Association (KSA), the Korean Agency for Technology and Standards (KATS), the Korea Internet and Security Agency (KISA), the Korea Laboratory Accreditation Scheme (KOLAS), and the Korea Communications Agency (KCA), as well as government ministries including the Ministry of Culture, Sports and Tourism (MCST), the Ministry of Health and Welfare (MOHW), the Ministry of Education (MOE), the National Institute of Special Education (NISE), the Korea Communications Commission (KCC), the Ministry of Science and ICT (MSIT), the National Human Rights Commission of Korea (NHRCK), the Ministry of Employment and Labor (MOEL), the Ministry of the Interior and Safety (MOIS), and the Ministry of Justice.

Broadcasters KBS, MBC, SBS, EBS, National Assembly Television, and Arirang International Broadcasting plan to adopt the standard's automatic captioning system; 5G operators SK Telecom, KT, and LG U+ apply the standard to Deaf telecommunication relay services; and AI providers Naver Clova, Kakao i, LG AI Research, Kakao Brain, and SK Telecom X publish KSL recognition APIs compatible with the standard.

The eighteen schools for the Deaf nationwide (Seoul School for the Deaf, Daejeon School for the Deaf, Busan Sungsim School, Incheon Sunhwa School, Gwangju Sunmyeong School, Daegu Yeonghwa School, Gangwon Provincial Dowon School, Chungbuk Cheongju Sungsim School, Chungnam Cheonan Inae School, Jeonbuk Iksan Jeil School, Jeonnam Gwangju Yeonghwa School, Gyeongbuk Andong Yeongmyeong School, Gyeongnam Jinju Hyegwang School, Jeju Yeongji School, Incheon Cheonghak School, Gyeonggi Ansan Jahae School, Ulsan Meari School, and Sejong Sarang School) serve as the standard's KSL education hubs.

This standard is published as an open standard under the MIT license; all simulator code, specifications, and example code are openly available at GitHub WIA-Official/wia-standards-public/tree/main/sign-language. Our hope is that this standard enables Deaf people who use Korean Sign Language to communicate more freely, and that this standard becomes the foundation for sign-language recognition standards across the Asia-Pacific region.