Chapter 5: Neural Network Models (CNN/LSTM/Transformer/CTC)

The heart of any modern sign-language recognition system is a deep neural network. This chapter covers standard model architectures and training procedures that combine CNN, RNN, LSTM, Transformer, and CTC.

5.1 Evolution of Sign-Language Recognition Neural Networks

Over the past decade, sign-language recognition neural networks have evolved as follows.

  1. 2012–2015: CNNs (Convolutional Neural Networks) applied to hand detection. Image classification networks such as AlexNet, VGGNet, and ResNet transferred to handshape classification.
  2. 2015–2018: RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory)1) applied to time-series modeling, learning the temporal flow of signs.
  3. 2017–2020: CTC2) loss applied to sequence-to-sequence learning, predicting lexical sequences directly from unsegmented video.
  4. 2018–2024: Transformer3) architectures introduced. Self-attention learns both long-range dependencies and multi-modal alignment simultaneously.
  5. 2023–present: Combined with multi-modal large language models, sign → natural-language translation accuracy has improved dramatically.

The WIA Sign Language Recognition Standard integrates the cumulative gains of these five stages into a "standard recognition pipeline." Inference results from different models can be compared in the simulator Algorithms panel.

5.1.1 Standard Recognition Pipeline

  1. Feature encoder: MediaPipe Holistic keypoint time series → CNN+LSTM or Transformer Encoder.
  2. Modality alignment: Cross-attention fuses hand, face, and body features.
  3. Sequence decoder: CTC or encoder-decoder attention outputs the lexical sequence.
  4. Translation decoder (optional): Converts the lexical sequence into natural-language sentences (mBART, NLLB-200).

5.2 CNN — Handshape Classification and Feature Extraction

CNNs learn the spatial features of image data. In sign-language recognition they are used for (i) hand-region detection, (ii) handshape classification, and (iii) frame-level feature extraction.

5.2.1 ResNet-50-Based Handshape Classifier

ResNet-504) is a 50-layer CNN that benefits from ImageNet pretraining, making it well suited to sign-language recognition. ETRI reports 97.8% accuracy on the 50-class KSL handshape classification task (2024).

5.2.2 3D-CNN and I3D

I3D (Inflated 3D ConvNet)5) extends 2D CNN into the time dimension. It processes video-frame sequences in a single pass and learns spatio-temporal features. On WLASL, I3D achieves 65% Top-1 accuracy and is widely adopted as a baseline.

5.2.3 MobileNet and EfficientNet

In mobile environments, lightweight CNNs such as MobileNet and EfficientNet are used. Accuracy decreases slightly, but inference at 30 fps or higher is possible on mobile GPUs. ETRI's "Su-eo Tok" uses a MobileNetV3 + LSTM combination.

5.3 LSTM and GRU — Time-Series Modeling

LSTM1) resolves the long-term-dependency problem of vanilla RNNs by controlling information flow through three gates (forget, input, output). In sign-language recognition the keypoint time series is fed to an LSTM to learn temporal patterns.

5.3.1 Bi-directional LSTM

Bi-directional LSTMs process the time series in both forward and backward directions, allowing the current time step to be informed by future context. KSL isolated recognition reports approximately 3.5% accuracy improvement over single-direction LSTM.

5.3.2 GRU (Gated Recurrent Unit)

GRU is a simplified LSTM with fewer parameters and faster training. In mobile environments GRU often replaces LSTM.

5.3.3 LSTM Limitations and the Transition to Transformer

LSTMs process sequences sequentially, limiting long-range dependency modeling and parallel training. Since the Transformer's introduction in 2017, the transition from LSTM to Transformer has accelerated in sign-language recognition.

5.4 Transformer — The Core of Modern Sign-Language Recognition

Transformer3), proposed by Vaswani et al. in "Attention Is All You Need" (2017), learns the relationships between all positions in a sequence directly via self-attention. In sign-language recognition the keypoint time series is fed to a Transformer Encoder, which learns spatio-temporal features and multi-modal alignment in a single pass.

5.4.1 Sign Language Transformer (Camgöz et al. 2020)

Camgöz et al. (2020)6) released the Sign Language Transformer. The encoder learns spatio-temporal features from video frames and the decoder generates natural-language translation. The model achieved BLEU-4 21.8 on RWTH-PHOENIX-2014T.

5.4.2 Multi-Modal Transformer

State-of-the-art models process hand, face, and body modalities through independent Transformer encoders and fuse them via cross-attention. KAIST's "TFNet" (2024) is a representative example and achieves WER 22.8% on RWTH-PHOENIX-2014T.

5.4.3 Combination with Vision Transformer (ViT)

Recent research processes video frames directly through ViT (Vision Transformer), bypassing MediaPipe keypoint extraction. Accuracy is higher but computational cost is significantly greater.

5.5 CTC — Unsegmented Sequence Learning

CTC2) is a loss function that enables learning of output sequences without explicit time alignment in the input sequence. In sign-language recognition this dramatically reduces annotation cost, since lexical boundaries do not need to be marked manually frame by frame.

5.5.1 Principle of CTC Loss

CTC computes loss by summing the probabilities of all alignment paths consistent with the output sequence. At inference, the most probable path is found via beam search.

5.5.2 CTC in Korean Sign Language Continuous SLR

ETRI's KSL continuous SLR model uses Transformer Encoder + CTC loss + Beam Search decoder. As of 2024 it achieves WER 28.4%.

5.6 Training Procedure Standardization

5.6.1 Data Splits

5.6.2 Augmentation

5.6.3 Optimization

The standard recommends AdamW optimizer (learning rate 1e-4, weight decay 0.01) with a cosine-annealing learning-rate schedule. KSL Transformer models are typically trained for 50–100 epochs.

5.6.4 Pre-training

Because KSL datasets are smaller than English and German sign-language datasets, transfer learning is essential. The standard recommends a three-stage training procedure: (i) pretrain on WLASL and How2Sign, (ii) fine-tune on the NIA AI Hub KSL dataset, and (iii) further fine-tune on small domain-specific datasets (medical, educational, legal).

5.7 Korean Sign Language Model Case Studies

5.7.1 ETRI KETI-Hand

ETRI's "KETI-Hand" combines MediaPipe Holistic + ResNet-50 + Bi-LSTM + CTC. On KSL 419-vocabulary isolated SLR it reports 95.2%, and on continuous SLR WER 28%. Deployed in mobile "Su-eo Tok" and used by approximately 120,000 users.

5.7.2 KAIST TFNet

KAIST's "TFNet" (Temporal Fusion Network for Continuous KSL, 2024) combines Multi-modal Transformer + CTC. On KSL 1,000-vocabulary it achieves WER 21.4%, and on PHOENIX-2014T WER 22.8%, approaching SoTA.

5.7.3 Seoul National University Sign-T2T

The Seoul National University CSE department released the KSL → Korean translation model "Sign-T2T," fine-tuning NLLB-200 to achieve BLEU-4 18.6.

5.7.4 Naver Clova Sign-Recog

Naver Clova released the "Sign-Recog" model in 2024, achieving 90% accuracy in mobile environments by combining mobile-NPU optimization with Vision Transformer.

5.8 Korean Neural Network Model Infrastructure

5.99 Closing Remarks and Korean Infrastructure Cross-Reference

The technical specifications described in this chapter all serve the broader goal of guaranteeing the everyday communication rights of the Deaf community. Korean Sign Language is, under Article 2 of the Korean Sign Language Act, a public language with status equal to Korean, and the WIA Sign Language Recognition Standard provides the technical underpinning for this legal status.

The standard operates on the Korean national infrastructure: the Korean Association of the Deaf (KAD), the National Institute of the Korean Language (NIKL), the National Information Society Agency (NIA), the Electronics and Telecommunications Research Institute (ETRI), the Korea Advanced Institute of Science and Technology (KAIST), the Korea Institute of Science and Technology Information (KISTI), the Telecommunications Technology Association (TTA), the Korean Standards Association (KSA), the Korean Agency for Technology and Standards (KATS), the Korea Internet and Security Agency (KISA), the Korea Laboratory Accreditation Scheme (KOLAS), and the Korea Communications Agency (KCA), as well as government ministries including the Ministry of Culture, Sports and Tourism (MCST), the Ministry of Health and Welfare (MOHW), the Ministry of Education (MOE), the National Institute of Special Education (NISE), the Korea Communications Commission (KCC), the Ministry of Science and ICT (MSIT), the National Human Rights Commission of Korea (NHRCK), the Ministry of Employment and Labor (MOEL), the Ministry of the Interior and Safety (MOIS), and the Ministry of Justice.

Broadcasters KBS, MBC, SBS, EBS, National Assembly Television, and Arirang International Broadcasting plan to adopt the standard's automatic captioning system; 5G operators SK Telecom, KT, and LG U+ apply the standard to Deaf telecommunication relay services; and AI providers Naver Clova, Kakao i, LG AI Research, Kakao Brain, and SK Telecom X publish KSL recognition APIs compatible with the standard.

The eighteen schools for the Deaf nationwide (Seoul School for the Deaf, Daejeon School for the Deaf, Busan Sungsim School, Incheon Sunhwa School, Gwangju Sunmyeong School, Daegu Yeonghwa School, Gangwon Provincial Dowon School, Chungbuk Cheongju Sungsim School, Chungnam Cheonan Inae School, Jeonbuk Iksan Jeil School, Jeonnam Gwangju Yeonghwa School, Gyeongbuk Andong Yeongmyeong School, Gyeongnam Jinju Hyegwang School, Jeju Yeongji School, Incheon Cheonghak School, Gyeonggi Ansan Jahae School, Ulsan Meari School, and Sejong Sarang School) serve as the standard's KSL education hubs.

This standard is published as an open standard under the MIT license; all simulator code, specifications, and example code are openly available at GitHub WIA-Official/wia-standards-public/tree/main/sign-language. Our hope is that this standard enables Deaf people who use Korean Sign Language to communicate more freely, and that this standard becomes the foundation for sign-language recognition standards across the Asia-Pacific region.

Chapter 5 Endnotes

  1. Sepp Hochreiter and Jürgen Schmidhuber, "Long Short-Term Memory," Neural Computation, 1997.
  2. Alex Graves et al., "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks," ICML 2006.
  3. Ashish Vaswani et al., "Attention Is All You Need," NeurIPS 2017.
  4. Kaiming He et al., "Deep Residual Learning for Image Recognition," CVPR 2016.
  5. João Carreira and Andrew Zisserman, "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset," CVPR 2017.
  6. Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden, "Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation," CVPR 2020.
  7. Marta R. Costa-jussà et al., "No Language Left Behind: Scaling Human-Centered Machine Translation," Meta AI, 2022. NLLB-200 multilingual translation model.
  8. AI Training Korean Sign Language Video Dataset, National Information Society Agency (NIA) AI Hub, 2021–present.
  9. WIA Standards Public Repository (sign-language folder), MIT License, GitHub: WIA-Official/wia-standards-public/tree/main/sign-language — PyTorch and TensorFlow reference implementations of the CNN/LSTM/Transformer/CTC combinations referenced in this chapter are openly published in this repository.