Chapter 02: Data Discovery & Search

Search Architecture

Effective data catalog search consists of: Crawler (metadata collection), Indexer (search indexing), Search Engine (query processing), and Ranker (relevance scoring).

Elasticsearch Implementation

Most modern data catalogs use Elasticsearch for: full-text search, autocomplete, fuzzy matching, faceted search, and real-time indexing.

Search Query Types

Keyword Search: Basic text matching
Fuzzy Search: Handles typos and variations
Wildcard Search: Pattern matching
Filtered Search: Narrow results by platform, owner, tags, date
Semantic Search: ML-based meaning understanding

Autocomplete

Real-time suggestions as users type, implemented using: Prefix Matching, N-Grams, or Elasticsearch Completion Suggester.

Relevance Ranking (BM25)

Results ordered by: Text Similarity (40%), Popularity (20%), Freshness (15%), Quality Score (15%), Personalization (10%).

Advanced Features

Faceted Search: Filter by categories in sidebar
Column-Level Search: Search individual columns
Personalized Results: Based on user role and history

Reference Architectures

Three reference architectures dominate contemporary catalog deployments. Each is an instantiation of the ISO/IEC 19763-3 framework for metamodel-based federation.

Push (producer-driven) — Each data system publishes a manifest to the catalog on schema change. Low latency, requires producer cooperation. Conforms to W3C DCAT publishing model.
Pull (catalog-driven) — The catalog crawls registered systems on a schedule. No producer changes needed; latency equals crawl interval.
Federated (hybrid) — Local catalogs aggregate within domains; a global catalog harvests via OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) or ResourceSync (NISO RP-9-2017).

Component Layers

A reference WIA-DATA-007 deployment has six layers, each independently substitutable:

Connector layer — JDBC, OData, AWS Glue, GCP Data Catalog, BigQuery INFORMATION_SCHEMA, dbt manifest, Snowflake INFORMATION_SCHEMA.
Metadata store — Relational store conforming to ISO/IEC 11179-3 metamodel.
Search index — Elasticsearch / OpenSearch with WIA mapping templates.
Lineage graph — Property graph following W3C PROV-DM.
API gateway — REST per OpenAPI 3.1, GraphQL per W3C draft.
UI layer — WCAG 2.2 Level AA compliant interface.

Scaling Considerations

The catalog is not a transactional database, so the dominant scaling constraint is usually search latency rather than write throughput. Practical deployments sustain millions of asset records on commodity hardware when the search tier is sized for the working-set of recently queried assets.

Practical Application

The architecture choice is reversible only at significant cost, so most organisations select a topology by answering three questions in order:

Where does the metadata live? If sovereignty or contractual constraints require metadata to remain inside the organisation's perimeter, hybrid or air-gapped deployment per ISO/IEC 27018 is mandatory.
How fast must metadata be? If schema changes must be reflected within minutes, the push topology with producer manifests is preferred. If a daily refresh is acceptable, the pull topology offers far lower producer integration cost.
How many producer systems exist? Above roughly fifty heterogeneous producer systems, federation per ISO/IEC 19763 with domain-level catalogs becomes operationally cheaper than a single global crawler.

The reference deployment publishes its conformance statement against ISO/IEC 25010 (functional suitability, reliability, security, maintainability) so adopters can match a topology to their non-functional priorities.