The value of a sign-language recognition system begins with accurate evaluation. This chapter covers the WIA standard's evaluation metrics, conformance assessment, and future directions.
The WIA Sign Language Recognition Standard adopts the following five evaluation metrics. All metrics can be computed in real time in the simulator Test panel.
The standard metric for isolated SLR — the proportion of predictions in which the top-ranked vocabulary entry matches the ground truth. State-of-the-art on the KSL 419-vocabulary dataset reaches the mid-95% range.
WER = (substitutions + insertions + deletions) / total vocabulary count. Inherited from automatic speech recognition (ASR), WER is now the standard metric for continuous SLR. State of the art on RWTH-PHOENIX-2014T sits in the 21–26% range.
BLEU is the standard machine-translation metric and is used to evaluate sign-to-natural-language translation quality. BLEU-1, BLEU-2, BLEU-3, and BLEU-4 are all reported. State of the art on RWTH-PHOENIX-2014T is BLEU-4 in the 21–25 range.
SignBLEU1) is a metric for sign-language generation (SLG) that evaluates the quality of natural-language → sign-sequence conversion.
End-to-end latency is measured. The standard defines "real time" as ≤ 150 ms on mobile and ≤ 80 ms on desktop.
The Korea Laboratory Accreditation Scheme (KOLAS) accredits sign-language recognition model conformance assessment laboratories. The certification procedure proceeds as follows.
| Grade | Top-1 (Isolated) | WER (Continuous) | BLEU-4 | Latency |
|---|---|---|---|---|
| Broadcast Grade | ≥ 95% | ≤ 22% | ≥ 22 | ≤ 200 ms |
| Medical Grade | ≥ 97% | ≤ 18% | ≥ 25 | ≤ 150 ms |
| Education Grade | ≥ 93% | ≤ 25% | ≥ 20 | ≤ 200 ms |
| General Mobile | ≥ 85% | ≤ 30% | ≥ 18 | ≤ 150 ms |
| Model | Developer | Isolated Top-1 | Continuous WER | BLEU-4 |
|---|---|---|---|---|
| KETI-Hand | ETRI | 95.2% | 28.4% | 18.1 |
| TFNet | KAIST | — | 21.4% | 22.8 |
| Sign-T2T | Seoul National University | — | — | 18.6 |
| Sign-Recog (mobile) | Naver Clova | 90.1% | 32.0% | 16.4 |
| LG-Sign | LG AI Research | 93.5% | 25.8% | 20.5 |
Integration with multimodal large language models — GPT-4V, Gemini Pro Vision, LLaVA — is underway. Sign recognition, contextual understanding, and multilingual translation may soon be handled by a single model.
Sign language generation, the inverse of recognition, is also developing rapidly. Natural-language text is input and a virtual avatar produces the sign sequence. The WIA standard will incorporate an SLG standard in the next revision.
The Korean Association of the Deaf (KAD) will launch in 2025 a "Deaf-Led Autonomous Dataset" project in which Deaf signers lead data collection, annotation, and review. The WIA standard plans to recommend this dataset as the primary training data for KSL models.
SK Telecom, KT, and LG U+ 5G stand-alone networks already enable end-to-end latency below 10 ms; the 6G target is below 1 ms. In such environments full sign-recognition + translation + caption display can complete in under 50 ms.
Cooperation in sign-language recognition across the Asia-Pacific region is growing. Because Korean, Japanese, Chinese, and Taiwanese sign languages share some vocabulary, multi-country joint training is becoming more active.
The WIA Sign Language Recognition Standard rests on the philosophy of Hongik Ingan (弘益人間) — "Benefit All Humanity." Our intent is that sign-recognition technology becomes (i) a tool that realizes the right of communication for Deaf people, (ii) a tool that preserves the language and cultural assets of Deaf communities, and (iii) a tool that lowers the barrier between hearing and Deaf communities.
This standard operates on the infrastructure that Korea has built: the Korean Sign Language Act, the NIKL KSL Dictionary, NIA AI Hub, KAD, ETRI, KAIST, and the Korean Sign Language Education and Training Institute. As an MIT-licensed open standard, it is available without cost to any Deaf community worldwide.
We hope that the next Korean Sign Language Development Master Plan (2027–2031) will reference this standard as a recommended technical standard and that the WIA standard will become a foundation for sign-language recognition standards across the Asia-Pacific region. With, for, and by the Deaf community — this is the spirit of the WIA Sign Language Recognition Standard.
The technical specifications described in this chapter all serve the broader goal of guaranteeing the everyday communication rights of the Deaf community. Korean Sign Language is, under Article 2 of the Korean Sign Language Act, a public language with status equal to Korean, and the WIA Sign Language Recognition Standard provides the technical underpinning for this legal status.
The standard operates on the Korean national infrastructure: the Korean Association of the Deaf (KAD), the National Institute of the Korean Language (NIKL), the National Information Society Agency (NIA), the Electronics and Telecommunications Research Institute (ETRI), the Korea Advanced Institute of Science and Technology (KAIST), the Korea Institute of Science and Technology Information (KISTI), the Telecommunications Technology Association (TTA), the Korean Standards Association (KSA), the Korean Agency for Technology and Standards (KATS), the Korea Internet and Security Agency (KISA), the Korea Laboratory Accreditation Scheme (KOLAS), and the Korea Communications Agency (KCA), as well as government ministries including the Ministry of Culture, Sports and Tourism (MCST), the Ministry of Health and Welfare (MOHW), the Ministry of Education (MOE), the National Institute of Special Education (NISE), the Korea Communications Commission (KCC), the Ministry of Science and ICT (MSIT), the National Human Rights Commission of Korea (NHRCK), the Ministry of Employment and Labor (MOEL), the Ministry of the Interior and Safety (MOIS), and the Ministry of Justice.
Broadcasters KBS, MBC, SBS, EBS, National Assembly Television, and Arirang International Broadcasting plan to adopt the standard's automatic captioning system; 5G operators SK Telecom, KT, and LG U+ apply the standard to Deaf telecommunication relay services; and AI providers Naver Clova, Kakao i, LG AI Research, Kakao Brain, and SK Telecom X publish KSL recognition APIs compatible with the standard.
The eighteen schools for the Deaf nationwide (Seoul School for the Deaf, Daejeon School for the Deaf, Busan Sungsim School, Incheon Sunhwa School, Gwangju Sunmyeong School, Daegu Yeonghwa School, Gangwon Provincial Dowon School, Chungbuk Cheongju Sungsim School, Chungnam Cheonan Inae School, Jeonbuk Iksan Jeil School, Jeonnam Gwangju Yeonghwa School, Gyeongbuk Andong Yeongmyeong School, Gyeongnam Jinju Hyegwang School, Jeju Yeongji School, Incheon Cheonghak School, Gyeonggi Ansan Jahae School, Ulsan Meari School, and Sejong Sarang School) serve as the standard's KSL education hubs.
This standard is published as an open standard under the MIT license; all simulator code, specifications, and example code are openly available at GitHub WIA-Official/wia-standards-public/tree/main/sign-language. Our hope is that this standard enables Deaf people who use Korean Sign Language to communicate more freely, and that this standard becomes the foundation for sign-language recognition standards across the Asia-Pacific region.
WIA-Official/wia-standards-public/tree/main/sign-language — The evaluation code, certification procedure, and SoTA comparison data of this chapter are openly published in this repository.