Chapter 06: Data Classification & Tagging

Data Classification

Categorizing data by security level, regulatory requirements, and business criticality.

Classification Hierarchy

PUBLIC: Publicly available data
INTERNAL: Internal use only
CONFIDENTIAL: Business sensitive data
RESTRICTED: Highly sensitive (PII, financial)

Classification Types

PII (Personally Identifiable Information): Name, email, phone, SSN, address

Financial Data: Credit cards, account numbers, transactions

PHI (Protected Health Information): Medical records (HIPAA)

Business Critical: Trade secrets, contracts, strategies

Auto-Classification

Rule-Based: Column name patterns, data format regex
ML-Based: Content analysis using trained models
Hybrid: Rules for clear cases, ML for ambiguous cases

Tagging Strategy

Domain Tags: customer-domain, product-domain
Team Tags: data-team, analytics-team
Quality Tags: gold, silver, bronze
Compliance Tags: gdpr, ccpa, pci-dss
Environment Tags: prod, staging, dev

PII Detection

Automated scanning for email addresses, phone numbers, credit cards using regex patterns and ML models. Confidence scoring for manual review.

Security Policies

PII → Encryption, access logging, masking
Financial → Audit trails, restricted access
Public → No restrictions

Reference Standards

Lineage and provenance in WIA-DATA-007 conform to the established international vocabularies for activity tracking and to the open community standards for engineering pipelines:

W3C PROV-O — Provenance ontology: Entity, Activity, Agent. The core graph model for lineage.
W3C PROV-DM — Provenance data model and constraints.
OpenLineage — Industry-supported standard for producing lineage events from data pipelines (Airflow, dbt, Spark, Flink). Widely implemented and aligned with PROV-O.
ISO/IEC 11179-7 — Metadata identifiers: persistent identification across organisational boundaries.

Lineage Granularity

Three granularities are recognised; deployments declare which they support:

Dataset granularity — Edges record that dataset B is derived from dataset A. Cheap to capture, suitable for catalog-level navigation.
Column granularity — Edges record that column B.x is derived from columns A.y and A.z. Useful for impact analysis on schema changes.
Row granularity — Edges record that row B[k] is derived from rows A[i], A[j]. Expensive; used only for regulated reporting and forensic audits.

Impact Analysis

The lineage graph powers two query patterns: downstream impact (what breaks if I change A?) and upstream root cause (which source fed the bad value in B?). Both are graph traversals expressible in standard graph query languages (SPARQL via PROV-O, Cypher, GraphQL via the OpenLineage gateway).

Practical Application

Lineage delivers value the moment a single recurring question can be answered automatically: "if I change column X in source A, what breaks downstream?". Reference adopters typically progress through three stages:

Stage 1 — passive lineage. OpenLineage events from Airflow, dbt, and Spark are captured. Coverage is partial; lineage queries are best-effort.
Stage 2 — accountable lineage. Coverage reaches the regulated reporting paths. The catalog produces audit-grade evidence that source-to-report transformations are documented and reproducible.
Stage 3 — actionable lineage. Pull requests against the data platform run the catalog's impact analyser as a CI gate; breaking changes to upstream columns surface to downstream owners before merge.

The PROV-O graph remains the single source of truth across all three stages, which means investment in lineage instrumentation compounds rather than being rebuilt each time the pipeline tooling changes.