CHAPTER 2

Edge AI Architecture

System Design Principles

Designing edge AI systems requires balancing competing constraints: computational power, memory, energy consumption, latency, and model accuracy. Unlike cloud-based systems with virtually unlimited resources, edge architectures must operate within strict hardware boundaries while delivering responsive, reliable AI capabilities.

The Resource Triangle

Every edge AI system navigates three fundamental constraints:

Optimizing for one dimension often requires trade-offs in others. A highly accurate model may demand more computation and memory, increasing power draw. The art of edge AI architecture lies in finding the optimal balance for your specific use case.

Layered Architecture

Application Layer

The application layer provides the user-facing interface and orchestrates AI functionality. On mobile devices, this might be a native iOS/Android app. On embedded systems, it could be custom firmware. This layer:

// TypeScript example: Application layer inference workflow
import { EdgeAIClient } from '@wia/edge-ai';

const client = new EdgeAIClient({
  model: 'object_detector_v2',
  backend: 'gpu',
  optimizations: ['fp16', 'pruning']
});

async function processFrame(imageData: ImageData): Promise {
  // Preprocess
  const tensor = await preprocessImage(imageData, {
    resize: [300, 300],
    normalize: true,
    colorSpace: 'RGB'
  });

  // Inference
  const outputs = await client.infer(tensor);

  // Postprocess
  const detections = await postprocessDetections(outputs, {
    confidenceThreshold: 0.6,
    nmsThreshold: 0.4,
    maxDetections: 10
  });

  return detections;
}

Inference Engine Layer

The inference engine executes neural network computations. Popular frameworks include:

The inference engine handles:

Hardware Abstraction Layer (HAL)

The HAL provides a uniform interface to diverse hardware accelerators. This allows the same model to run efficiently on different devices without application code changes. The HAL manages:

Hardware Components

Central Processing Unit (CPU)

General-purpose processors handle non-parallelizable tasks and coordinate system operations. While not optimized for neural network math, modern CPUs include:

CPU inference is suitable for small models or when specialized accelerators aren't available. Performance typically ranges from 1-10 GFLOPS.

Graphics Processing Unit (GPU)

Originally designed for graphics rendering, GPUs excel at parallel matrix operations—the core of neural network inference. Mobile GPUs like Adreno (Qualcomm), Mali (ARM), and Apple GPU provide:

GPU inference works well for medium-to-large convolutional and transformer models where parallelism can be fully exploited.

Neural Processing Unit (NPU)

Also called AI accelerators or neural engines, NPUs are purpose-built for neural network inference. Examples include:

NPUs offer:

Digital Signal Processor (DSP)

DSPs specialize in signal processing—audio, video, and sensor data. Qualcomm's Hexagon DSP can run AI workloads efficiently, particularly for:

Software Stack

Operating System Integration

Edge AI frameworks integrate deeply with mobile and embedded operating systems:

Platform Framework Hardware Support
iOS / macOS Core ML CPU, GPU, Neural Engine
Android TensorFlow Lite, NNAPI CPU, GPU, NPU, DSP
Linux Embedded TensorFlow Lite, ONNX Runtime CPU, GPU (OpenCL/Vulkan)
RTOS TensorFlow Lite Micro CPU (Cortex-M)

Model Deployment Pipeline

Getting a trained model from the training environment to edge devices involves several steps:

1. Train Model (PyTorch/TensorFlow)
   ↓
2. Export to Intermediate Format (ONNX, SavedModel)
   ↓
3. Convert to Edge Format (TFLite, CoreML, ONNX)
   ↓
4. Optimize (Quantization, Pruning, Layer Fusion)
   ↓
5. Benchmark on Target Hardware
   ↓
6. Package for Deployment
   ↓
7. Deploy via OTA Update or App Bundle
   ↓
8. Runtime Inference on Device

Deployment Patterns

Fully On-Device Deployment

The entire AI pipeline runs locally with zero cloud dependency. Ideal for:

Pros: Maximum privacy, lowest latency, no bandwidth costs, works offline

Cons: Limited to what device hardware can run, no centralized learning

Edge-Cloud Hybrid

Simple tasks run on-device, complex processing falls back to cloud. For example:

Pros: Balance of performance and privacy, graceful degradation

Cons: Still requires connectivity for advanced features

Federated Edge Architecture

Models train collaboratively across edge devices. Each device:

  1. Downloads the current global model
  2. Trains on local data
  3. Uploads only model updates (gradients) to server
  4. Server aggregates updates from many devices
  5. Broadcasts improved global model

Pros: Privacy-preserving learning, models improve from collective data

Cons: Communication overhead, requires coordination infrastructure

Edge Mesh Networks

Edge devices collaborate directly without central servers. Useful in:

Memory Management

Model Storage

Models are stored compressed on disk and loaded into RAM for execution. Techniques to reduce footprint:

Activation Memory

During inference, intermediate layer outputs (activations) require temporary memory. For a ResNet-50 processing a 224x224 image:

// Activation memory calculation
Input: 224 × 224 × 3 = 150,528 values
Conv layers: ~10 MB peak activation memory
Output: 1000-class probabilities

Total peak memory: ~15 MB (with FP32)
Optimized (FP16): ~7.5 MB
Further optimized (INT8): ~4 MB

Memory optimization strategies:

Power Management

Power Consumption Sources

Edge AI power draw comes from:

Energy-Efficient Design

Strategies to minimize power consumption:

// Adaptive inference example
async function adaptiveInference(image: Tensor): Promise {
  // Try lightweight model first (10ms, 50mW)
  const quickResult = await lightModel.infer(image);

  if (quickResult.confidence > 0.95) {
    return quickResult; // High confidence, use quick result
  }

  // Fall back to heavy model for uncertain cases (50ms, 200mW)
  return await heavyModel.infer(image);
}

Performance Optimization

Model Architecture Selection

Some architectures are inherently more edge-friendly:

Architecture Edge Suitability Use Case
MobileNet Excellent Image classification, detection
EfficientNet Excellent High-accuracy image tasks
SqueezeNet Good Ultra-low footprint imaging
BERT-Tiny Good Language understanding
ResNet-50 Moderate Requires optimization for edge
GPT-3 Poor Too large for edge deployment

Operator Optimization

Inference engines optimize neural network operations through:

Benchmarking and Profiling

Key Metrics

When evaluating edge AI systems, measure:

Profiling Tools

弘益人間 Architecture Principle:

Design edge AI systems that empower users while respecting their privacy, device resources, and autonomy. Optimize for efficiency not just performance—every milliwatt saved extends battery life, every megabyte saved makes models accessible to more devices.

Summary

Edge AI architecture balances computational constraints, memory limits, and power budgets to deliver responsive, efficient AI on resource-constrained devices. Key architectural components include:

  • Layered software stack (application, inference engine, HAL)
  • Specialized hardware accelerators (NPU, GPU, DSP)
  • Optimized inference frameworks (TensorFlow Lite, Core ML, ONNX Runtime)
  • Memory-efficient model deployment and activation management
  • Power-conscious design leveraging hardware acceleration and quantization

Deployment patterns range from fully on-device (maximum privacy, offline capability) to edge-cloud hybrid (balancing performance and resource constraints) to federated edge (collaborative learning while preserving privacy).

Success in edge AI requires holistic optimization—selecting appropriate model architectures, leveraging hardware accelerators, optimizing memory usage, and measuring performance across latency, throughput, power, and accuracy dimensions.

Review Questions

  1. What are the three fundamental constraints in the "resource triangle" of edge AI?
  2. Describe the responsibilities of each layer in the edge AI software stack.
  3. Compare and contrast CPU, GPU, and NPU for edge AI inference.
  4. What advantages do NPUs provide over GPUs for neural network inference?
  5. Explain the model deployment pipeline from training to edge device.
  6. What are the pros and cons of fully on-device deployment versus edge-cloud hybrid?
  7. How does federated edge architecture preserve privacy while enabling model improvement?
  8. What techniques reduce activation memory consumption during inference?
  9. Why do specialized accelerators (NPUs) achieve better energy efficiency than general-purpose processors?
  10. Name five key metrics for benchmarking edge AI systems.