Hongik Ingan (εΌηδΊΊι)
"Benefit All Humanity"
Real-time emotion streaming enables continuous affective computing for applications that require immediate feedback and time-series analysis. Together with non-real-time processing (Phase 2 REST API), it forms one of the standard's two transport axes; an application can adopt either or both depending on its use case. This chapter covers the four supported streaming protocols (WebSocket, gRPC, MQTT, Server-Sent Events), focusing on the recommended primary choice β WebSocket β with the others covered for the deployments in which they are preferable.
Phase 3 defines a real-time streaming protocol for continuous emotion analysis. It is required for applications such as the following.
Real-time processing has substantively different trade-offs from request-response (Phase 2 REST). In a request-response cycle, retry, logging, and audit are simple because each call is bounded; in a streaming session, the principal quality measures become the stability of the connection, end-to-end latency, and the reconnection strategy. In modern mobile networks, intermittent disruptions arising from cell hand-over, signal shadowing, and vehicle motion must be absorbed gracefully β a sudden ten-millisecond stall during a hand-over should not produce a misclassification at a safety-critical moment.
A second major dimension is consent management. A non-streaming call carries consent and metadata in each request body; a streaming session, by contrast, carries them once at connection setup and then conducts many frames through the same channel. This is efficient but imposes a requirement that consent withdrawal mid-stream be honoured immediately. WIA Phase 3 defines a client-initiated stop message for exactly this purpose; servers must cease processing on receipt and discard frames already received but not yet classified.
A third dimension is user-experience design. A non-streaming response gives a single, easily presented result; a streaming response is a continuous time series whose presentation must avoid distracting the user while still giving meaningful feedback. Real-time gauges, gradient indicators, and explicit text labels are common UI patterns; user-research studies have repeatedly found that explicit text labels combined with a colour-coded gauge are more accessible than colour alone.
| Protocol | Use case | Latency target |
|---|---|---|
| WebSocket | Web and mobile applications, bidirectional | < 100 ms |
| gRPC | Server-to-server, high performance | < 50 ms |
| MQTT | IoT devices, low bandwidth | < 200 ms |
| Server-Sent Events | Unidirectional streaming | < 150 ms |
WebSocket (IETF RFC 6455) is the bidirectional WebSocket protocol that WIA Phase 3 recommends as the primary transport.[1] gRPC suits microservice server-to-server traffic and serialises payloads via Protocol Buffers (Google, 2008βpresent), reducing wire cost.[2] MQTT (OASIS Standard 5.0, 2019) is appropriate for automotive and IoT contexts where bandwidth is constrained and connection persistence matters.[3] Server-Sent Events is appropriate where unidirectional streaming is sufficient (W3C SSE Recommendation, 2015).[4]
Selection criteria are: bidirectionality, bandwidth constraint, server infrastructure, and client environment (browser, mobile, embedded). For typical web and mobile applications, WebSocket is effectively the single answer; for microservice-internal communication, gRPC dominates. For in-vehicle cabin monitoring, where a vehicle's electronic-control unit must communicate with the cloud across an intermittent radio link, MQTT's low-bandwidth and connection-persistence properties matter. WIA Phase 3 over MQTT requires QoS level 1 (at-least-once delivery) at minimum; QoS level 2 (exactly-once delivery) is required for safety-relevant applications where message loss could affect safety.
Endpoint: wss://stream.wiastandards.com/emotion-ai/v1/stream Connect request: GET /emotion-ai/v1/stream HTTP/1.1 Host: stream.wiastandards.com Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Sec-WebSocket-Version: 13 X-WIA-API-Key: your_api_key Connect response: HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
The WebSocket handshake uses the standard HTTP 1.1 upgrade mechanism and is therefore compatible with existing HTTP infrastructure (load balancers, firewalls, proxies). WIA Phase 3 connections terminate on port 443 β the same port as HTTPS β so that most enterprise firewalls accept the connection without configuration changes. The protocol's keep-alive mechanism uses ping/pong frames at thirty-second intervals; if no pong arrives within thirty seconds, the connection is torn down and reconnection is attempted. Many NAT and firewall implementations close idle connections after sixty seconds, so the thirty-second ping interval preserves stability across these intermediaries.
| Type | Direction | Purpose |
|---|---|---|
| config | client β server | Configure the stream parameters |
| frame | client β server | Send a video or audio frame |
| result | server β client | Emotion-analysis result |
| error | server β client | Error notification |
| stop | client β server | Stop processing immediately |
The states a session occupies are recorded in the simulator's π‘ Protocol panel as the ENUM STREAMING (frames flowing) and STOPPED (no frames flowing); see Table 3-5b. The state machine is intentionally narrow β only two states β to keep error handling simple and to make the relationship between consent withdrawal and processing termination immediate.
config (client β server):
{
"type": "config",
"stream_id": "stream_123",
"modality": "facial",
"frame_rate": 30,
"resolution": "720p",
"options": { "return_action_units": true,
"return_dimensions": true }
}
frame (client β server):
{
"type": "frame",
"stream_id": "stream_123",
"frame_id": 1,
"timestamp": "2026-05-01T10:30:00.000Z",
"data": "base64_encoded_image_data...",
"data_format": "jpeg"
}
result (server β client):
{
"type": "result",
"stream_id": "stream_123",
"frame_id": 1,
"result": { /* WIA Phase 1 emotion record */ }
}
stop (client β server):
{ "type": "stop", "stream_id": "stream_123" }
The frame payload may also be transmitted as a binary WebSocket frame for image bytes that need not be base-64 encoded; in that case the metadata accompanies the payload as a JSON header in a separate text frame, in the manner of MIME multipart encoding. The binary form reduces wire cost by approximately 33% for image streams.
| Stage | Budget | Notes |
|---|---|---|
| Capture and encoding | β€ 30 ms | Camera or microphone driver |
| Network transit | β€ 50 ms | One-way; metro fibre or 5G |
| Server inference | β€ 100 ms | GPU-accelerated face or voice model |
| Network return | β€ 50 ms | One-way |
| Client render | β€ 30 ms | UI tick interval |
| End-to-end (provisional) | β€ 300 ms | Per simulator Panel 2 ENUM |
The provisional end-to-end latency target of three hundred milliseconds is the value displayed in the simulator's π‘ Protocol panel and follows from the budget above. It is a guideline rather than a hard limit; safety-critical applications such as automotive driver monitoring require lower budgets at each stage. Per ISO 26262 ASIL-B reliability targets for in-vehicle warning systems, the entire end-to-end loop typically targets less than one hundred and fifty milliseconds.[5]
Streaming sessions encounter transient connection loss in real-world environments. WIA Phase 3 prescribes a back-off and resume strategy similar to the recommendation in Google's Site Reliability Engineering literature: an exponential back-off with full jitter, capped at an upper bound, and a session-resumption token that allows the server to continue from the last acknowledged frame.
def reconnect_backoff(attempt):
base = 1.0 # seconds
cap = 30.0 # seconds
sleep_s = random.uniform(0, min(cap, base * (2 ** attempt)))
return sleep_s
Sessions resume against the original stream_id and a server-issued resume_token issued in the server's last result message. Frames received before the disconnect but not yet acknowledged are re-sent by the client; the server is required to deduplicate by frame_id.
All Phase 3 channels run over TLS 1.2 or TLS 1.3 (TLS 1.3 recommended).[6] WebSocket subprotocols are negotiated at handshake; the WIA subprotocol identifier is wia.emotion-ai.v1. Cross-origin requests are subject to the standard browser CORS policy. Origin-check is mandatory: a server must reject WebSocket upgrade requests whose Origin header is not in the per-API-key allow-list, in line with the OWASP API Security Top 10 (2023).[7]
Authentication uses the same API-key or OAuth 2.0 mechanisms as Phase 2; the credential is presented once at handshake and bound to the session. Tokens that expire mid-session are renewed via a Bearer-Refresh message rather than by tearing down the WebSocket. End-to-end encryption between client and server is the responsibility of TLS and is not duplicated at the application layer; payload encryption beyond TLS is permitted for confidentiality requirements that exceed the network-layer guarantee, but it is not the default.
For server-to-server traffic in microservice architectures, the gRPC streaming variant is recommended. The EmotionAnalyze service exposes a bidirectional streaming RPC that accepts FrameRequest messages and returns EmotionResponse messages, with the same field semantics as the WebSocket message types. Protocol Buffers schemas are published in the public GitHub repository.
For automotive and IoT environments, the MQTT variant is recommended. Topics follow the convention wia/emotion-ai/v1/{stream_id}/{direction}, where direction is frame or result. The MQTT broker is responsible for retention and quality-of-service guarantees. WIA conformance over MQTT requires the broker to support MQTT 5.0 features (per-message expiry, response-topic, content-type), in line with the OASIS MQTT 5.0 specification.
The Korean edition of this volume contains additional sections addressing 5G hand-over patterns observed in named domestic telecom operators, in-vehicle MQTT deployments at named domestic automotive suppliers, and contact-centre WebSocket deployments at named domestic platform companies, together with a survey of Korean public-sector firewall constraints (the relevant national e-government communications-security guidelines).
This English edition deliberately abstracts those passages. Where the Korean edition names specific Korean operators, suppliers, or platforms, the English edition refers to "leading domestic telecom operators", "leading domestic automotive suppliers", and "leading domestic platform companies". The conformance requirements themselves are identical between the two editions.
Seven key takeaways.
STREAMING and STOPPED β keep error handling and consent management simple.stop message ceases processing and discards uncommitted frames.stop message in consent management.Chapter 7 turns to Phase 4 β Integration β which lifts the data-format, API, and streaming layers into named domain contexts (healthcare, education, marketing, automotive, XR). The simulator's π Integration panel (Panel 3) visualises how external systems consume Phase 1β3 output through the Phase 4 adapters. The standard's evolution roadmap is recorded in the public GitHub repository.[99]
WIA-Official/wia-standards-public/tree/main/emotion-ai. The standard's evolution roadmap, revision history, and SDK source code are maintained openly in this repository, where the WIA standards committee records its formal verification of all primary sources cited in this chapter. β