Chapter 04: Data Lineage Tracking

What is Data Lineage?

Data lineage tracks the complete journey of data from source to destination, showing transformations, dependencies, and flow.

Three Levels of Lineage

1. Dataset-Level: Table-to-table dependencies

2. Column-Level: Column transformation tracking

3. Field-Level: Individual value transformation logic

Lineage Graph Structure

Nodes: Datasets (tables, views, files)
Edges: Data flows (transformations, copies)
Processes: ETL jobs, SQL queries, scripts
Graph Type: DAG (Directed Acyclic Graph)

Auto-Tracking Methods

SQL Parsing: Extract dependencies from queries
Execution Log Analysis: Spark, Airflow logs
API Hooking: Real-time capture
Metastore Integration: Hive, Glue, Unity Catalog

Impact Analysis

When a dataset changes, automatically identify all affected downstream systems: reports, dashboards, ML models, other tables.

Use Cases

Root cause analysis for data quality issues
GDPR data deletion tracking
Schema change impact assessment
Debugging data pipelines

Reference Standards

Discovery in WIA-DATA-007 is built on web-native standards so that catalog content is reachable by general search engines and by domain-specific tools alike:

W3C DCAT v3 — RDF vocabulary for catalogs, datasets, and data services. Used as the export format for the public catalog API.
schema.org Dataset — Used for structured-data markup of dataset landing pages so they appear in Google Dataset Search and similar discovery surfaces.
OpenSearch 1.1 — A simple syntax for exposing catalog search to the wider web.
OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) — Pull-based harvesting protocol for federation across organisations.
ResourceSync (NISO RP-9-2017) — Modern alternative to OAI-PMH using sitemap.xml conventions.

Search Strategies

Discovery quality is dominated by three lever points:

Term frequency — Standard BM25 ranking on title, description, column documentation, and tags.
Authority signals — Owner reputation, certification badge, freshness, query frequency.
User personalisation — Team affinity, recently viewed assets, role-based boosting.

Faceted Browsing

The reference UI exposes facets for domain, owner, certification tier, last refresh, sensitivity class, and PII flag. Each facet is backed by a SKOS concept scheme so the same vocabulary drives both data entry and discovery, eliminating the gap between how an owner labels an asset and how a consumer searches for it.

Practical Application

Discovery is judged by the time from a question ("is there a dataset of...?") to a confident answer. Reference deployments measure four moments:

First search — Median time from query to first result rendered. Target: under 500 ms at the 95th percentile.
First click — Time from query to opening an asset detail page. Target: under three seconds.
First decision — Time from opening detail to a clear yes/no on whether the asset matches the need. Target: under sixty seconds for silver and gold tier assets.
First adoption — Time from decision to actual use (query, download, pipeline integration). Target: under one business day for self-service assets.

These four are tracked alongside accessibility conformance (WCAG 2.2 Level AA) so that the catalog remains usable across the full diversity of consumers.