2.1 The Foundation: Tensor Representation

At the heart of every deep learning computation lies the tensor—a multidimensional array of numbers. While conceptually simple, the physical representation of tensors in memory has profound implications for performance, portability, and correctness. WIA-AI-011 Phase 1 establishes comprehensive standards for tensor data formats, ensuring that tensors can be seamlessly shared across different accelerators without conversion overhead or numerical degradation.

A tensor is characterized by several fundamental properties: its shape (dimensions), data type (element format), and layout (memory organization). Consider a simple RGB image tensor representing a batch of 32 images, each 224×224 pixels with 3 color channels. This tensor has shape [32, 3, 224, 224], but how should these 4,816,896 elements be arranged in linear memory?

2.2 Memory Layout Conventions

Different frameworks and hardware vendors have historically adopted different conventions for organizing multidimensional data in memory. The two dominant approaches are row-major (C-style) and column-major (Fortran-style), but for tensors with more than two dimensions, the situation becomes more nuanced.

2.2.1 NCHW Layout (Batch, Channel, Height, Width)

The NCHW layout, also known as channels-first, stores all channels of each spatial location together. For our image example, this means storing all pixels of channel 0, then all pixels of channel 1, then channel 2, for each image in the batch. This layout is favored by NVIDIA's cuDNN library and PyTorch (on GPU) because it enables efficient convolution implementations.

// NCHW memory layout for [2, 3, 2, 2] tensor
Batch 0, Channel 0: [a, b, c, d]
Batch 0, Channel 1: [e, f, g, h]
Batch 0, Channel 2: [i, j, k, l]
Batch 1, Channel 0: [m, n, o, p]
Batch 1, Channel 1: [q, r, s, t]
Batch 1, Channel 2: [u, v, w, x]

Linear memory: [a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x]

2.2.2 NHWC Layout (Batch, Height, Width, Channel)

The NHWC layout, or channels-last, stores all channels of each pixel together. For the same tensor, this interleaves the channels: pixel (0,0) gets R,G,B values, then pixel (0,1), and so on. This layout is preferred by TensorFlow, ARM processors, and many mobile NPUs because it exhibits better cache locality for certain operations and matches how images are typically stored.

// NHWC memory layout for [2, 3, 2, 2] tensor
Batch 0, Pixel (0,0): [a, e, i]
Batch 0, Pixel (0,1): [b, f, j]
Batch 0, Pixel (1,0): [c, g, k]
Batch 0, Pixel (1,1): [d, h, l]
Batch 1, Pixel (0,0): [m, q, u]
...

Linear memory: [a,e,i,b,f,j,c,g,k,d,h,l,m,q,u,n,r,v,o,s,w,p,t,x]

2.2.3 WIA-AI-011 Layout Specification

The WIA-AI-011 standard mandates that implementations support both NCHW and NHWC layouts as first-class citizens. Additionally, it defines a stride-based representation that can describe arbitrary layouts, including blocked and padded formats commonly used for hardware optimization.

{
  "shape": [32, 3, 224, 224],
  "layout": "NCHW",
  "strides": [150528, 50176, 224, 1],
  "offset": 0,
  "element_size_bytes": 4
}

// Strides specify the number of elements to skip
// to move to the next position along each dimension
// Element at [n, c, h, w] is at:
// offset + n*strides[0] + c*strides[1] + h*strides[2] + w*strides[3]

2.3 Data Types and Precision

Neural networks can operate at various numerical precisions, trading accuracy for performance and energy efficiency. WIA-AI-011 defines a comprehensive type system covering the full spectrum of precision formats used in modern AI accelerators.

2.3.1 Floating-Point Formats

Standard IEEE 754 floating-point formats provide familiar, well-defined numerical behavior but consume significant memory and bandwidth.

Type	Bits	Exponent	Mantissa	Range	Use Case
FLOAT64	64	11	52	±10^308	Scientific computing
FLOAT32	32	8	23	±10^38	Default training
FLOAT16	16	5	10	±65504	Mixed precision
BFLOAT16	16	8	7	±10^38	TPU, modern training

2.3.2 Integer and Fixed-Point Formats

Integer formats, particularly INT8, have become essential for efficient inference. They require careful calibration through quantization but offer 4× memory savings and significant computational speedups.

// Quantization parameters in WIA-AI-011
{
  "dtype": "INT8",
  "quantization": {
    "scheme": "per_tensor_symmetric",
    "scale": 0.0235,
    "zero_point": 0,
    "range": [-128, 127]
  }
}

// Dequantized value = (quantized_value - zero_point) * scale
// For symmetric quantization, zero_point = 0

2.3.3 Specialized Formats

Several specialized formats have emerged for specific use cases:

• TensorFloat-32 (TF32): NVIDIA's format with FP32 range but reduced precision (19 bits total)
• INT4/UINT4: Sub-byte quantization for extreme compression in LLM inference
• FP8: Emerging 8-bit floating-point formats with different exponent/mantissa splits
• Posit: Alternative to IEEE 754 with dynamic precision (rarely used in practice)

2.4 Tensor Descriptor Format

The WIA-AI-011 tensor descriptor is a comprehensive metadata structure that fully specifies a tensor's properties. This descriptor serves as the contract between different system components, ensuring unambiguous interpretation.

{
  "version": "WIA-AI-011-v1.0",
  "tensor_id": "uuid-string",
  "name": "input_images",

  "shape": {
    "dimensions": [32, 3, 224, 224],
    "layout": "NCHW",
    "symbolic": false
  },

  "dtype": {
    "base_type": "FLOAT16",
    "quantization": null,
    "byte_order": "little_endian"
  },

  "memory": {
    "strides": [150528, 50176, 224, 1],
    "offset_bytes": 0,
    "total_bytes": 9633792,
    "alignment": 256,
    "device": {
      "type": "NPU",
      "id": 0,
      "memory_space": "device"
    }
  },

  "properties": {
    "requires_grad": false,
    "is_pinned": true,
    "is_contiguous": true,
    "zero_copy_compatible": true
  },

  "metadata": {
    "created_at": "2025-01-15T10:30:00Z",
    "framework": "pytorch",
    "producer": "data_loader_v2"
  }
}

2.5 Memory Alignment and Padding

Modern accelerators impose alignment requirements on memory allocations for optimal performance. Unaligned access can cause significant slowdowns or even hardware exceptions. WIA-AI-011 specifies alignment requirements and padding rules.

2.5.1 Alignment Requirements

Different accelerators have different alignment requirements:

• GPUs: Typically require 256-byte alignment for coalesced memory access
• TPUs: Often require 128-byte alignment for systolic array operations
• NPUs: May require 64-byte or 128-byte alignment
• CPU (SIMD): 32-byte (AVX2) or 64-byte (AVX-512) alignment

WIA-AI-011 mandates a minimum alignment of 64 bytes for all tensor allocations, with devices free to request stricter alignment through capability queries.

2.5.2 Padding Strategies

Padding adds extra elements to dimensions to meet hardware requirements. For example, many accelerators perform best when channel counts are multiples of 8 or 16.

// Original tensor: [32, 11, 224, 224]
// Padded for optimal performance: [32, 16, 224, 224]
// 5 channels of padding added

{
  "shape": [32, 11, 224, 224],
  "physical_shape": [32, 16, 224, 224],
  "padding": {
    "dims": [false, true, false, false],
    "values": [0, 5, 0, 0],
    "mode": "zero_fill"
  }
}

2.6 Zero-Copy Data Sharing

One of the most significant performance optimizations in WIA-AI-011 is zero-copy data sharing. Traditional approaches require copying data when moving between devices or framework boundaries. Zero-copy mechanisms eliminate these copies by sharing the underlying memory buffer.

2.6.1 Shared Memory Protocol

WIA-AI-011 defines a buffer sharing protocol based on file descriptors (POSIX) or handles (Windows) that allows processes and devices to map the same physical memory:

// Zero-copy buffer sharing
{
  "buffer_handle": {
    "type": "dma_buf_fd",  // Linux DMA-BUF
    "fd": 42,
    "size_bytes": 9633792,
    "permissions": "read_write"
  },
  "mapping": {
    "device_address": "0x7f4a2000",
    "cpu_address": null,
    "access_mode": "device_only"
  }
}

2.6.2 Pinned Memory

Pinned (or page-locked) memory cannot be swapped to disk, ensuring stable physical addresses that hardware DMA engines can use directly. WIA-AI-011 provides APIs for allocating pinned memory:

// Pinned memory allocation
wia_memory_desc_t desc = {
    .size = 9633792,
    .alignment = 256,
    .flags = WIA_MEM_PINNED | WIA_MEM_ZERO_COPY,
    .device_affinity = WIA_DEVICE_NPU_0
};

wia_buffer_t buffer;
wia_status_t status = wia_allocate_memory(&desc, &buffer);

2.7 Handling Dynamic Shapes

Many modern neural networks have dynamic shapes that vary based on input. Transformer models process variable-length sequences, object detectors handle different image sizes, and beam search generates outputs of unknown length. WIA-AI-011 supports dynamic shapes through symbolic dimensions.

{
  "shape": {
    "dimensions": ["batch", "sequence_length", 768],
    "symbolic": true,
    "constraints": {
      "batch": {"min": 1, "max": 256},
      "sequence_length": {"min": 1, "max": 2048, "multiple_of": 8}
    }
  },
  "allocation_strategy": "overallocate",
  "max_physical_shape": [256, 2048, 768]
}

2.8 Endianness and Cross-Platform Compatibility

While most modern systems use little-endian byte ordering, ensuring cross-platform compatibility requires explicit specification. WIA-AI-011 tensors carry endianness metadata:

{
  "dtype": {
    "base_type": "FLOAT32",
    "byte_order": "little_endian",  // or "big_endian", "network_order"
    "bit_order": "lsb_first"
  }
}

When transferring tensors between systems with different endianness, WIA-AI-011 implementations must perform byte swapping automatically or raise an error if in-place conversion is impossible.

2.9 Sparse Tensor Formats

Many neural network tensors are sparse, containing mostly zeros. Storing sparse tensors efficiently requires specialized formats. WIA-AI-011 supports several sparse formats:

2.9.1 Coordinate (COO) Format

{
  "format": "COO",
  "shape": [1000, 1000],
  "nnz": 5000,  // number of non-zero elements
  "indices": {
    "dtype": "INT32",
    "data": [[0,5], [0,12], [1,3], ...]  // [row, col] pairs
  },
  "values": {
    "dtype": "FLOAT32",
    "data": [1.5, 2.3, 0.7, ...]
  }
}

2.9.2 Compressed Sparse Row (CSR) Format

CSR is more efficient for sparse matrix operations, particularly for GPUs:

{
  "format": "CSR",
  "shape": [1000, 1000],
  "row_pointers": [0, 3, 7, 12, ...],  // 1001 elements
  "column_indices": [5, 12, 45, ...],  // nnz elements
  "values": [1.5, 2.3, 0.7, ...]       // nnz elements
}

2.10 Practical Implementation Considerations

Implementing WIA-AI-011 tensor formats requires careful attention to several practical concerns:

Performance Validation

Implementations should include benchmarks verifying that the standardized formats don't introduce overhead compared to vendor-native formats. Layout conversions, when necessary, should be lazy and cached.

Backward Compatibility

Providing adapters for existing framework tensor formats eases adoption. PyTorch's torch.Tensor, TensorFlow's tf.Tensor, and NumPy's ndarray should all have efficient conversion paths to/from WIA-AI-011 descriptors.

Versioning

The tensor descriptor includes a version field, allowing future extensions while maintaining compatibility with existing implementations. Readers must gracefully handle newer versions by ignoring unknown fields.