Chapter 2

Document Data Formats and Standards

Standardized data formats are the foundation of interoperable digital document systems, enabling seamless exchange and verification across agencies, platforms, and borders while preserving semantic meaning and legal validity.

Core Document Schema Design

The WIA-SOCIAL Public Document standard defines a hierarchical schema architecture that balances flexibility with structure. At the top level, all documents share common metadata fields including unique identifiers, issuance information, and cryptographic signatures. Document type-specific schemas extend this base schema with additional fields relevant to particular document categories.

This approach provides several advantages. Common tooling can process any document type for authentication, storage, and routing. Type-specific validation can enforce business rules and data quality. New document types can be added without breaking existing systems. Partial schemas allow documents to be extended with additional attributes while maintaining backward compatibility.

Base Document Schema

Every public document, regardless of type, includes a core set of metadata fields. The document ID provides a globally unique identifier, typically combining an issuing authority code, document type code, and sequential or random identifier. Issuer information identifies the government agency or authorized entity that created the document. Issuance and expiration dates define the document's temporal validity. The subject identifies the person, organization, or entity to whom the document pertains.

{
  "documentId": "EE-BC-2025-001234",
  "documentType": "BirthCertificate",
  "version": "2.0",
  "issuer": {
    "countryCode": "EE",
    "organizationName": "Estonian Ministry of Interior",
    "organizationId": "70000329",
    "signerName": "Registrar Maria Tamm",
    "signerTitle": "Chief Registrar, Tallinn District"
  },
  "issuedDate": "2025-01-15T10:30:00Z",
  "validFrom": "2025-01-10",
  "validUntil": null,
  "status": "active",
  "subject": {
    "subjectType": "person",
    "personalCode": "52501104567"
  },
  "metadata": {
    "language": "et",
    "jurisdiction": "EE",
    "documentFormat": "WIA-SOCIAL-v2.0",
    "qrCode": "https://verify.gov.ee/doc/EE-BC-2025-001234",
    "accessLevel": "public-with-consent"
  },
  "signature": {
    "algorithm": "ECDSA-P256-SHA256",
    "certificateChain": ["..."],
    "signatureValue": "...",
    "timestamp": "2025-01-15T10:30:15Z"
  }
}

Document Type-Specific Schemas

Type-specific schemas extend the base schema with additional fields. A birth certificate includes parent information, birth weight and length, hospital details, and registration information. A driver's license includes vehicle classes, restrictions, and endorsements. A property deed includes legal descriptions, parcel numbers, and chain of ownership. Each schema is defined using JSON Schema, allowing automatic validation and documentation generation.

Encoding Formats

While the logical schema defines document structure and semantics, the encoding format determines how documents are serialized for storage and transmission. The WIA-SOCIAL standard supports multiple encoding formats to accommodate different use cases and integration requirements.

JSON (JavaScript Object Notation)

JSON serves as the primary encoding format for API interactions and web applications. Its human-readable structure, native JavaScript support, and extensive library ecosystem make it ideal for modern web services. JSON's flexibility allows easy extension with custom attributes while maintaining backward compatibility. However, JSON's verbosity and lack of binary data support make it less suitable for bandwidth-constrained environments.

XML (Extensible Markup Language)

XML remains important for integration with legacy government systems and enterprise applications. Many existing document standards, including those from the United Nations and various government agencies, are defined in XML Schema. XML's namespace support allows mixing vocabularies from multiple standards in a single document. XML digital signatures (XMLDSig) provide standardized cryptographic protection. The primary drawbacks are verbosity and parsing complexity.

CBOR (Concise Binary Object Representation)

CBOR provides a binary encoding with a data model similar to JSON but much more compact. It's particularly valuable for constrained environments like smart cards, mobile applications with limited bandwidth, and IoT devices. CBOR supports binary data natively, unlike JSON's base64 encoding. CBOR is used extensively in the W3C Web Authentication standard and is gaining adoption for mobile identity credentials.

PDF/A (PDF for Long-term Archival)

For human-readable documents requiring long-term preservation, PDF/A provides a standardized format that embeds all necessary fonts, images, and formatting information. Unlike regular PDF, PDF/A prohibits features that depend on external resources or could prevent future rendering. PDF/A-3 allows embedding structured data (XML, JSON) within the PDF, enabling both human viewing and machine processing from a single file.

Metadata Standards and Vocabularies

Consistent metadata enables discovery, classification, and lifecycle management of documents. The WIA-SOCIAL standard incorporates established metadata vocabularies where possible, ensuring compatibility with library systems, content management platforms, and international standards.

Dublin Core

Dublin Core provides 15 core metadata elements widely used across digital libraries and repositories: title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage, and rights. These elements provide basic bibliographic information applicable to any resource. Extended Dublin Core adds domain-specific terms while maintaining core element compatibility.

ISO 23081 Records Management

ISO 23081 defines metadata for managing records throughout their lifecycle. It specifies metadata for creation, classification, access control, storage, use, and disposition. Government agencies implementing digital document systems typically adopt ISO 23081 to ensure compliance with records management regulations and enable proper retention scheduling.

PREMIS Preservation Metadata

For documents requiring long-term preservation (birth certificates, property deeds, court records), PREMIS (PREservation Metadata: Implementation Strategies) provides a comprehensive framework. PREMIS captures technical metadata about file formats, software dependencies, and preservation actions. This ensures documents remain accessible even as technologies evolve over decades.

{
  "dublinCore": {
    "title": "Birth Certificate - Anna Kask",
    "creator": "Estonian Ministry of Interior",
    "date": "2025-01-15",
    "type": "Text/Official Document",
    "format": "application/json",
    "identifier": "EE-BC-2025-001234",
    "language": "et",
    "rights": "Personal data - subject consent required"
  },
  "iso23081": {
    "classification": "Personal Identity Document",
    "securityClassification": "Protected B",
    "retentionPeriod": "Permanent",
    "disposalAction": "Retain indefinitely",
    "accessRights": "Subject, parents, legal guardians, authorized officials"
  },
  "premis": {
    "originalFormat": "application/json",
    "creatingApplication": "DocGen v3.2.1",
    "preservationLevel": "Full",
    "migrationPath": ["JSON 1.0 → JSON 2.0 → CBOR if needed"]
  }
}

Biometric Data Encoding

Modern identity documents increasingly incorporate biometric data for authentication. Standardized biometric encodings ensure interoperability across capture devices, storage systems, and verification systems.

ISO/IEC 19794 Biometric Data Formats

ISO/IEC 19794 defines standard formats for various biometric modalities. Part 2 covers finger minutiae data (unique points on fingerprints). Part 4 specifies finger image data. Part 5 defines face image data including requirements for pose, lighting, and resolution. Part 6 covers iris image data. These standards ensure biometric data can be shared across systems from different vendors.

Biometric Template Protection

Raw biometric data poses privacy risks—fingerprints and iris scans cannot be changed if compromised. Template protection techniques create one-way transformations of biometric data suitable for matching but not reconstruction. Fuzzy extractors generate cryptographic keys from biometric data, allowing authentication without storing biometric templates. These techniques are increasingly required by privacy regulations.

Machine-Readable Zones and Visual Encodings

To bridge physical and digital worlds, documents often include machine-readable encodings that allow rapid data capture using cameras or specialized readers.

QR Codes

QR codes can encode up to 4,296 alphanumeric characters in a two-dimensional barcode easily scanned by smartphones. For documents, QR codes typically encode either the full document data (for small documents) or a URL linking to verification services. Digital signatures can be included in the QR code, allowing offline verification. The EU Digital COVID Certificate famously used signed QR codes containing full credential data plus cryptographic proof.

NFC and RFID Chips

Near Field Communication (NFC) chips embedded in cards or documents can store larger amounts of data (typically 4-64 KB) and perform cryptographic operations. E-passports contain NFC chips storing biometric data and performing active authentication to prevent cloning. The chips support various security protocols including Basic Access Control (BAC), Password Authenticated Connection Establishment (PACE), and Extended Access Control (EAC) to prevent unauthorized reading.

Machine-Readable Zones (MRZ)

The machine-readable zone, developed by ICAO for passports and travel documents, uses two or three lines of OCR-readable text containing key document data. While older than QR codes, MRZ remains the international standard for travel documents. The format includes check digits computed from data fields, allowing detection of transcription errors. All international passports include an MRZ, and many national ID cards do as well.

P<ESTTEST<<ANNA<<<<<<<<<<<<<<<<<<<<<<<<<<<
ZE0123456<4EST9501101F3112315<<<<<<<<<<<<<02

Line 1: P < Country < Name (with filler <)
Line 2: Document number < Country < DOB < Sex < Expiry < Personal number < Check digits

Multi-Format Representation

Real-world document systems must support multiple representations of the same logical document. A birth certificate might exist as a PDF for printing, JSON for API access, XML for legacy system integration, and a QR code for mobile verification. Ensuring consistency across these representations requires careful architecture.

Canonical Form

Systems should designate one format as the canonical representation—the authoritative source from which all other formats are derived. Typically, this is a structured format like JSON or XML. Digital signatures are applied to the canonical form, not derived formats. When documents are retrieved, they can be transformed into requested formats, but signature verification always operates on canonical data.

Format Conversion

Automated conversion between formats requires careful handling of semantic equivalence. Some data may be lost or transformed during conversion—XML attributes might become JSON properties, date formats might change, binary data encodings differ. Conversion systems should be tested extensively, and round-trip conversion (A → B → A) should preserve all information when possible.

Format Negotiation

RESTful APIs use content negotiation to serve documents in multiple formats from a single URL. Clients specify preferred formats using the Accept HTTP header. Servers respond with the best available format and indicate the format using the Content-Type header. This allows the same API endpoint to serve JSON to web applications, XML to enterprise systems, and PDF to human users.

GET /api/documents/EE-BC-2025-001234 HTTP/1.1
Host: api.gov.ee
Accept: application/json, application/xml;q=0.9, application/pdf;q=0.8
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 4523
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

{
  "documentId": "EE-BC-2025-001234",
  "documentType": "BirthCertificate",
  ...
}

Data Quality and Validation

Ensuring data quality is critical for public documents that serve as authoritative records. Validation occurs at multiple stages: during data entry, before document generation, at system boundaries, and during verification.

Schema Validation

JSON Schema and XML Schema provide formal specifications of valid document structure. Validation engines check that documents conform to schemas, rejecting malformed data before it enters systems. Schemas define required fields, data types, value constraints, and structural relationships. Modern schema languages support conditional requirements, pattern matching, and cross-field validation.

Business Rule Validation

Beyond structural correctness, documents must satisfy business rules. Birth dates must be in the past. Document expiration dates must be after issuance dates. Referenced entities (parent documents, issuing officials) must exist. Some validation requires external data—checking that a social security number is validly issued, verifying that an address exists, confirming that a license applicant has no disqualifying violations. Rule engines and workflow systems encode and enforce these requirements.

Data Consistency Checking

Computed fields like check digits must match their source data. Derived values must be correctly calculated. Cryptographic signatures must verify successfully. These checks detect data corruption, transmission errors, and tampering attempts. Critical documents should be validated every time they're accessed, not just at creation.

Summary

Review Questions

1. Explain the advantages of hierarchical schema design with base and type-specific schemas. How does this support extensibility?
2. Compare JSON, XML, and CBOR encodings. What are the trade-offs and when would you choose each?
3. Why is canonical form important for multi-format documents? How does it relate to digital signatures?
4. Describe the purpose and structure of Machine-Readable Zones (MRZ) in passports. Why do they include check digits?
5. What is biometric template protection and why is it important? How does it differ from simply encrypting biometric data?
6. Explain how content negotiation works in RESTful APIs. How does it enable serving multiple document formats from a single endpoint?
7. What metadata is captured by PREMIS and why is it important for long-term document preservation?
8. Describe the difference between schema validation and business rule validation. Give examples of rules that each would enforce.