Chapter 03: Metadata Management

Three Types of Metadata

1. Technical Metadata: Table names, column names, data types, constraints, indexes, partitions.

2. Business Metadata: Descriptions, owners, business terms, purpose, SLA.

3. Operational Metadata: Last updated, query frequency, quality score, popularity, access logs.

Automated Collection

Extract metadata from databases using JDBC/ODBC, cloud APIs (AWS Glue, GCP Data Catalog), data processing tools (dbt manifest, Spark listeners).

Schema Management

Change Detection: Polling, event-based, CDC (Change Data Capture)
Version Control: Track schema evolution over time
Impact Analysis: Identify affected downstream systems

Quality Metrics

Quality Score = Description (30%) + Owner (20%) + Column Docs (20%) + Tags (15%) + Freshness (15%)

Gold Tier: 90-100% quality
Silver Tier: 70-89% quality
Bronze Tier: 50-69% quality

Governance

Approval workflows for metadata changes
Naming conventions and standards
Regular metadata quality reviews
User feedback collection

Reference Standards

Metadata management within WIA-DATA-007 conforms to:

ISO/IEC 11179-3:2023 — Metadata registries — Part 3: Registry metamodel and basic attributes. Defines Data Element, Value Domain, Conceptual Domain, Classification Scheme, and Object Class.
ISO/IEC 19763 — Information technology — Metamodel framework for interoperability (MFI). Multi-part series covering ontology registration, model mappings, and process model registration.
Dublin Core Metadata Terms (DCMI Metadata Terms) — Refinement of the fifteen-element ISO 15836-1:2017 set.
W3C DCAT v3 — Catalog and dataset vocabulary.
W3C SKOS — Simple Knowledge Organization System for taxonomies and controlled vocabularies.

The Three-Layer Metadata Model

The catalog represents metadata as three orthogonal layers, with cross-references rather than mixing concerns:

Technical metadata conforms to the originating system's schema language (SQL DDL, Avro, Protobuf, JSON Schema). The catalog stores the canonical representation and the originating dialect.
Business metadata uses SKOS concept schemes for business terms, with skos:prefLabel for the canonical name and skos:altLabel for synonyms, allowing free-text discovery without ambiguity.
Operational metadata is captured automatically from query logs, lineage events, and access events, then aged into rolling windows (24 h, 7 d, 30 d, 365 d).

Governance Hooks

Every metadata change emits a W3C PROV-O activity record so the catalog itself produces an auditable trail of who changed what and when. Approval workflows are out-of-band webhooks that pause the activity until a designated owner approves.

Practical Application

The three-layer model collapses into a single review checklist for every published asset:

Technical layer — Does the schema match the source? Is the dialect declared (SQL, Avro, Protobuf, JSON Schema)?
Business layer — Is there a description in the active business term? Is the owner reachable? Are sensitive columns labelled with the SKOS concept from the organisation's privacy taxonomy?
Operational layer — Is the asset receiving query traffic? Is freshness within SLA?

Owners who cannot answer all three within five minutes of opening the asset page are signalled as needing support; the catalog raises a soft assistance ticket rather than blocking publication. Over time, the average time-to-answer trends downward, and that figure becomes the leading indicator that catalog adoption is succeeding.