Data Provenance and Tracking

Data provenance is the detailed record of the origins, custody, and transformations of data throughout its lifecycle. While lineage shows the "what" and "where," provenance adds the "when," "who," "why," and "how."

Understanding Data Provenance

Think of data provenance as a complete audit trail for your data. Just as tracking a package through delivery shows every scan, location, and handler, data provenance records every touch point in the data's journey.

Core Elements of Provenance

1. Origin Information

• Source System: Where did the data originate?
• Collection Method: API, file upload, database replication?
• Collection Time: When was the data captured?
• Collection Agent: What system/process collected it?
• Original Format: JSON, CSV, Parquet, etc.

{
  "dataset": "raw.customer_events",
  "origin": {
    "source_system": "mobile-app-v2.3.1",
    "collection_method": "REST API",
    "endpoint": "https://api.company.com/v1/events",
    "collected_at": "2025-12-26T10:15:23.456Z",
    "collected_by": "event-collector-service",
    "format": "JSON",
    "schema_version": "2.1",
    "record_count": 15247
  }
}

2. Transformation History

Every change to data should be recorded:

• Transformation Type: Cleaning, aggregation, enrichment, etc.
• Transformation Logic: The actual code or SQL used
• Transformation Time: When it was applied
• Transformation Agent: Which job/process performed it
• Input/Output State: Before and after snapshots

{
  "transformation": {
    "id": "trans-8765",
    "type": "data_cleaning",
    "job": "clean-customer-data-v1.2",
    "executed_at": "2025-12-26T10:16:00.000Z",
    "executed_by": "etl-service-pod-42",
    "duration_ms": 3456,
    "logic": {
      "sql": "UPDATE staging.customers SET email = LOWER(TRIM(email))",
      "git_commit": "abc123def456",
      "config_version": "1.2.0"
    },
    "impact": {
      "records_read": 15247,
      "records_modified": 847,
      "records_rejected": 3
    }
  }
}

3. Custody Chain

Track who has accessed or modified the data:

• Access Logs: Who read the data and when
• Modification Logs: Who changed the data
• Export Logs: Who downloaded or shared the data
• Permission Changes: Access control modifications

4. Quality Metadata

Provenance should include quality checks and validation:

• Validation Rules: What checks were performed
• Quality Scores: Data quality metrics
• Anomaly Detection: Unusual patterns detected
• Completeness: Missing value statistics

Provenance Capture Methods

Manual Annotation

Developers explicitly record provenance information in code:

-- SQL with provenance annotations
-- @provenance: source=crm_api, version=2.1, owner=sales_team
-- @quality: completeness_threshold=0.95
-- @transformation: type=aggregation, purpose=daily_summary
SELECT
    DATE(created_at) as event_date,
    COUNT(*) as event_count,
    'crm-aggregation-v1' as provenance_job
FROM raw.events
GROUP BY event_date;

Pros: Precise, intentional
Cons: Labor-intensive, easily forgotten

Automatic Capture

Systems automatically record provenance through instrumentation:

• Query Parsers: Extract metadata from SQL
• Runtime Hooks: Intercept data operations
• Logging Systems: Capture execution metadata
• Event Streams: Publish provenance events

# Python decorator for provenance capture
from wia_lineage import track_provenance

@track_provenance(
    operation="customer_enrichment",
    sensitivity="high",
    owner="data_team"
)
def enrich_customer_data(df):
    """Enrich customer data with demographic info"""
    # Function automatically records:
    # - Input schema and row count
    # - Execution time and resources
    # - Output schema and row count
    # - Any errors or warnings
    enriched = df.merge(demographics, on='customer_id')
    return enriched

Hybrid Approach

Combine automatic capture with manual enrichment:

• Auto-capture technical metadata (timing, resources, schemas)
• Manually add business context (purpose, owner, SLAs)
• Best of both worlds: comprehensive without excessive burden

Provenance Standards

W3C PROV

The W3C Provenance standard defines a model for representing provenance information:

• Entities: Things that exist (datasets, documents)
• Activities: Actions that happened (transformations, processes)
• Agents: Who/what performed activities (users, services)

{
  "@context": "http://www.w3.org/ns/prov#",
  "entity": {
    "@id": "dataset:cleaned_customers",
    "prov:wasGeneratedBy": "activity:cleaning_job_123"
  },
  "activity": {
    "@id": "activity:cleaning_job_123",
    "prov:startedAtTime": "2025-12-26T10:00:00Z",
    "prov:endedAtTime": "2025-12-26T10:05:00Z",
    "prov:wasAssociatedWith": "agent:etl_service"
  },
  "agent": {
    "@id": "agent:etl_service",
    "prov:actedOnBehalfOf": "agent:data_engineering_team"
  }
}

OpenLineage

OpenLineage extends basic lineage with provenance-like metadata:

• Job execution metadata
• Dataset schema evolution
• Data quality metrics
• Error information

Provenance for Compliance

GDPR Requirements

GDPR Article 30 requires organizations to maintain records of processing activities. Provenance provides this:

• Purpose of Processing: Why data was collected and used
• Categories of Data: What types of personal data
• Recipients: Who received the data
• Retention: How long data is kept
• Transfers: Where data was sent (especially cross-border)

{
  "data_subject_id": "user-12345",
  "processing_activity": {
    "purpose": "service_improvement",
    "legal_basis": "legitimate_interest",
    "data_collected": ["email", "usage_patterns"],
    "collected_at": "2025-01-15T08:30:00Z",
    "consent_id": "consent-789",
    "retention_period": "2_years",
    "recipients": ["analytics_team", "ml_service"],
    "transfers": [
      {
        "to": "EU_data_center",
        "mechanism": "standard_contractual_clauses",
        "date": "2025-01-15T09:00:00Z"
      }
    ]
  }
}

HIPAA Audit Trails

Healthcare data requires detailed access and modification logs:

• Every access to patient data must be logged
• Modifications must be traceable to individuals
• Audit logs must be tamper-proof
• Retention requirements: minimum 6 years

Provenance for Data Science

ML Model Provenance

For machine learning, provenance is critical for reproducibility:

• Training Data: Exact dataset versions used
• Feature Engineering: How features were created
• Model Parameters: Hyperparameters and configurations
• Training Environment: Framework versions, hardware
• Evaluation Metrics: Model performance over time

{
  "model_id": "churn-predictor-v2.3",
  "training_provenance": {
    "training_data": {
      "dataset": "warehouse.customer_features",
      "version": "2025-12-20",
      "snapshot_id": "snap-8876",
      "row_count": 1247893,
      "feature_count": 47
    },
    "code": {
      "git_repo": "github.com/company/ml-models",
      "commit": "abc123def456",
      "notebook": "notebooks/churn_model_v2.ipynb"
    },
    "environment": {
      "python_version": "3.11.5",
      "tensorflow_version": "2.14.0",
      "hardware": "8x NVIDIA A100"
    },
    "hyperparameters": {
      "learning_rate": 0.001,
      "batch_size": 128,
      "epochs": 50
    },
    "metrics": {
      "accuracy": 0.87,
      "precision": 0.84,
      "recall": 0.89,
      "auc": 0.91
    },
    "trained_by": "data_scientist@company.com",
    "trained_at": "2025-12-26T08:00:00Z"
  }
}

Challenges in Provenance Tracking

Storage Overhead

Detailed provenance can generate massive amounts of metadata:

• Strategy: Aggregate and summarize where appropriate
• Keep detailed provenance for critical/sensitive data
• Archive old provenance to cold storage

Performance Impact

Capturing provenance can slow down data operations:

• Strategy: Asynchronous capture (emit events, process later)
• Sampling for high-volume operations
• Optimize storage and indexing

Complex Transformations

Some operations are hard to track precisely:

• Black-box UDFs (user-defined functions)
• External API calls
• Complex multi-step processes

Strategy: Provide hooks for manual annotation, use conservative approximations

Best Practices

1. Start Simple: Begin with basic origin and transformation tracking
2. Automate Everything Possible: Reduce manual effort
3. Make It Searchable: Provenance is useless if you can't find it
4. Think Long-Term: Design for retention and archival
5. Integrate with Workflows: Provenance capture should be seamless
6. Validate Completeness: Regularly check for gaps