The accuracy of sign language recognition depends not only on hand motion but also on how precisely facial expression, gaze, head, and shoulder non-manual markers are aligned with the manual components.
Sign language is intrinsically multi-modal. While the hands carry lexical meaning, facial expression marks sentence type (declarative, interrogative, imperative, negation); gaze indexes pronouns and locations; and head and shoulder movements mark role shift, emphasis, and temporal relations. These multiple channels operate simultaneously, and any AI recognition system must therefore possess multi-modal alignment capability. This chapter covers the WIA standard for hand-face-body multi-modal alignment.
| Channel | Information carried | Primary keypoints | KSL example |
|---|---|---|---|
| Handshape | Lexical meaning | 21 hand keypoints × 2 | SCHOOL vs. STUDY |
| Location | Lexical meaning, spatial grammar | Wrist position + body keypoints | MOTHER (chin) vs. FATHER (forehead) |
| Movement | Verb meaning, tense, aspect | Time-series hand-keypoint trajectory | GO (forward) vs. COME (backward) |
| Palm orientation | Lexical disambiguation | Palm normal vector | LOVE vs. RESPECT |
| Facial expression (NMM) | Interrogative, negation, emphasis | 468 face landmarks (brows, mouth) | Interrogative GA-YO? = raised brows |
| Eye gaze | Pronoun, deixis | Iris keypoints + face direction | THAT = directional gaze |
| Head | Affirmation, negation, emphasis | Nose and ear keypoint rotation | Head shake = negation |
| Shoulders | Role shift, quotation | Both shoulder keypoint rotation | "MOTHER SAID …" = shoulders shifted to mother's locus |
All eight channels must be extracted simultaneously to recover the full meaning of sign-language utterances. For Korean Sign Language, non-manual markers (NMMs) are reported to account for approximately 30% of meaning differentiation (National Institute of the Korean Language, "Study on Non-Manual Markers in Korean Sign Language," 2019). Ignoring facial expression and recognizing only the hands cannot distinguish interrogatives from declaratives or affirmations from negations, producing serious meaning-reversal errors.
Non-manual markers are decomposed into the following five sub-channels.
In Korean Sign Language, raised eyebrows mark yes-no questions, furrowed eyebrows mark wh-questions, and pinched eyebrows mark conditional clauses. Detection uses the vertical-coordinate variation of MediaPipe FaceMesh landmarks 70–76 (left eyebrow) and 296–302 (right eyebrow).
In Korean Sign Language, mouth shape is used to (i) accompany the spoken-Korean mouth shape of the lexical concept (e.g., the SCHOOL sign accompanied by the mouth shape for /hak-gyo/); (ii) convey adverbial meaning (e.g., rounded "pong" mouth = LARGE); and (iii) carry facial-expression cues (smiling, frowning). The system uses MediaPipe FaceMesh outer-lip landmarks 78–95 and inner-lip landmarks 14–17.
Eyes carry two kinds of information: eye gaze (pronoun deixis, spatial reference) and blink (sentence boundary). The MediaPipe Iris model estimates gaze direction as a 3D vector.
Puffed or drawn-in cheeks carry adverbial meaning (FULL, A-LITTLE). Detection uses FaceMesh landmarks 36–40 (left cheek) and 266–270 (right cheek).
Head nod (yes), head shake (no), head tilt (curiosity), and head thrust (attention) all have grammatical functions. Head rotation (pitch, yaw, roll) is estimated from MediaPipe Pose nose and ear keypoints together with the FaceMesh facial-plane normal vector.
Eye gaze plays a central role in the spatial grammar of sign languages. When a signer fixes gaze on an imaginary spatial position (R-locus), references to a "person" or "object" are assigned to that position, and subsequent hand gestures pointing to that position invoke the assigned referent. Gaze estimation follows three stages.
Practical accuracy is approximately ±5°, corresponding to spatial accuracy of about ±9 cm at 1 m signer distance, which is adequate for separating R-loci (typically spaced > 60 cm apart) in sign languages.
The KAIST AI Graduate School released a KSL recognition model "Gaze-SLR" (2024) that integrates gaze information. Compared with a gaze-ignoring baseline, continuous-SLR WER improved from 24.8% to 22.1%, with the largest effect in conversational regions rich in pronouns and spatial deixis.
When RGB camera, depth camera, and IMU sensors operate on different clocks, temporal alignment is required. The standard mandates PTP (IEEE 1588) synchronization, with all modality timestamps aligned to the microsecond level. Timestamp alignment can be verified in the simulator Data Format panel.
When RGB and depth cameras have different coordinate systems, the same hand keypoint can have different coordinates. The standard recommends pre-measuring camera calibration matrices (intrinsics and extrinsics) and converting all modality keypoints to a common "signer coordinate system."
The simplest alignment flattens all keypoints into a 1D vector and concatenates them. MediaPipe Holistic gives 33×3 (body) + 21×3×2 (hands) + 468×3 (face) = 1,629 dimensions. This is too high; the standard recommends the following dimensionality reduction:
The current best practice treats hand, face, and body modalities through independent encoders and learns inter-modality interactions through a cross-attention mechanism. This is referred to as "Late Fusion + Cross-Attention." The "Sign Language Transformer" of Camgöz et al. (2020) is a representative example.
| Dataset | Sign language | NMM annotation | Annotation tool |
|---|---|---|---|
| NIA AI Hub Korean SL | KSL | Interrogative, negation, wh-marker | ELAN + in-house tool |
| RWTH-PHOENIX-2014T | DGS | Mouth morpheme, head pose | ELAN |
| How2Sign | ASL | Speech captions + partial NMM | ELAN |
| BSL Corpus | BSL | Rich NMM annotation | ELAN |
| NHK Sign Language | JSL | Mouth morpheme, gaze | NHK in-house tool |
NMM annotation is the most labour-intensive stage of sign-language corpus work, requiring approximately 30–50 hours of human work per hour of video. The standard recommends automatic NMM annotation tooling and adopts the ELAN NMM template (Korean-language edition by NIKL) as the standard annotation format.
When RGB and depth cameras have different exposure times, frames that appear to be simultaneous can in fact differ by several milliseconds. Even with PTP synchronization, residual error of approximately ±1 ms remains. With fast hand motion (> 2 m/s), this synchronization error becomes pixel-level positional error.
Deployment environments may lack some modalities (e.g., depth). The standard requires "graceful degradation" — the model must produce the most accurate inference possible from whatever modalities are available.
If the visual hand channel recognizes SCHOOL while the mouth channel recognizes STUDY, modality conflict arises. The standard recommends resolving conflicts via confidence-weighted averaging.
Differences as fine as 1 mm in eyebrow position or 2 mm in mouth corner can change meaning. The standard recommends sub-pixel accuracy (±0.3 pixel) for facial keypoints.
The technical specifications described in this chapter all serve the broader goal of guaranteeing the everyday communication rights of the Deaf community. Korean Sign Language is, under Article 2 of the Korean Sign Language Act, a public language with status equal to Korean, and the WIA Sign Language Recognition Standard provides the technical underpinning for this legal status.
The standard operates on the Korean national infrastructure: the Korean Association of the Deaf (KAD), the National Institute of the Korean Language (NIKL), the National Information Society Agency (NIA), the Electronics and Telecommunications Research Institute (ETRI), the Korea Advanced Institute of Science and Technology (KAIST), the Korea Institute of Science and Technology Information (KISTI), the Telecommunications Technology Association (TTA), the Korean Standards Association (KSA), the Korean Agency for Technology and Standards (KATS), the Korea Internet and Security Agency (KISA), the Korea Laboratory Accreditation Scheme (KOLAS), and the Korea Communications Agency (KCA), as well as government ministries including the Ministry of Culture, Sports and Tourism (MCST), the Ministry of Health and Welfare (MOHW), the Ministry of Education (MOE), the National Institute of Special Education (NISE), the Korea Communications Commission (KCC), the Ministry of Science and ICT (MSIT), the National Human Rights Commission of Korea (NHRCK), the Ministry of Employment and Labor (MOEL), the Ministry of the Interior and Safety (MOIS), and the Ministry of Justice.
Broadcasters KBS, MBC, SBS, EBS, National Assembly Television, and Arirang International Broadcasting plan to adopt the standard's automatic captioning system; 5G operators SK Telecom, KT, and LG U+ apply the standard to Deaf telecommunication relay services; and AI providers Naver Clova, Kakao i, LG AI Research, Kakao Brain, and SK Telecom X publish KSL recognition APIs compatible with the standard.
The eighteen schools for the Deaf nationwide (Seoul School for the Deaf, Daejeon School for the Deaf, Busan Sungsim School, Incheon Sunhwa School, Gwangju Sunmyeong School, Daegu Yeonghwa School, Gangwon Provincial Dowon School, Chungbuk Cheongju Sungsim School, Chungnam Cheonan Inae School, Jeonbuk Iksan Jeil School, Jeonnam Gwangju Yeonghwa School, Gyeongbuk Andong Yeongmyeong School, Gyeongnam Jinju Hyegwang School, Jeju Yeongji School, Incheon Cheonghak School, Gyeonggi Ansan Jahae School, Ulsan Meari School, and Sejong Sarang School) serve as the standard's KSL education hubs.
This standard is published as an open standard under the MIT license; all simulator code, specifications, and example code are openly available at GitHub WIA-Official/wia-standards-public/tree/main/sign-language. Our hope is that this standard enables Deaf people who use Korean Sign Language to communicate more freely, and that this standard becomes the foundation for sign-language recognition standards across the Asia-Pacific region.
WIA-Official/wia-standards-public/tree/main/sign-language — All implementations of multi-modal alignment algorithms, gaze estimation models, and NMM classification systems referenced in this chapter are openly published in this repository.