A dive does not end when the vehicle is back on deck. Its CTD casts have to reach a national archive, its video annotations have to reach the people who identify animals, its multibeam grid needs a citable identifier, and some of its coordinates must stay hidden for years. Phase 4 of WIA-ENE-040 (PHASE-4-INTEGRATION.md) is the part of the standard that governs this afterlife of the data. It does not invent new message formats. It explains how the Phase 1 messages and the Chapter 4 science records are handed to systems that WIA does not control: research networks, data repositories, cloud platforms and institutional databases.
This chapter follows the specification in the order an integrator meets it: the reference architecture and its four integration patterns (7.1), research networks (7.2), repositories and persistent identifiers (7.3), cloud storage and analytics (7.4), access control and data sharing (7.5), and finally the migration of legacy archives (7.6). Throughout, the specification's examples are corrected where they would mislead an implementer. People, institutions and identifiers in the examples are fictional, and hosts are written as https://api.example.org, because the specification itself uses several different example hosts.
Phase 4 §1.2 lists six integration principles: interoperability with existing oceanographic standards, federation (distributed data with central discovery), security through role-based access control and encryption, scalability to very large archives, complete provenance, and open access by default with controlled exceptions. The last principle is worth keeping in mind, because §7.2 of the same document sets a two-year default embargo. Section 7.5 reconciles the two.
§2.1 places a single integration layer between WIA-conformant systems and the outside world. The layer has four components. The specification names them without defining their interfaces, so the table below states the responsibility each one must carry for the rest of Phase 4 to work. The "inputs" and "outputs" columns are this book's reading, not normative text.
| Component | Responsibility | Inputs | Outputs |
|---|---|---|---|
| Data Gateway | Accepts Phase 1 messages and archives, verifies checksums, de-duplicates, and converts to target formats (CF-NetCDF, CSV, PANGAEA tab files) | Envelopes, JSONL files, mission archives | Converted files, ingest receipts |
| Metadata Service | Builds discovery metadata (ISO 19115, DataCite, Darwin Core) from the Chapter 4 records and the archive manifest; keeps the identifier cross-reference | expeditionId, diveId, metadata.json | Metadata records, DOI requests, search index entries |
| Security Manager | Authenticates users and partner systems, evaluates RBAC roles, embargoes and access tiers, writes the access log | Tokens, role grants, tier labels | Allow/deny decisions, access_log rows |
| Orchestration Engine | Schedules and retries submissions, tracks their state, and releases embargoed data on the due date | Submission jobs, embargo dates | Job status, release events |
Below the layer the specification draws three families of external system: research networks, data archives and cloud services. The separation matters in practice. Research networks such as OOI and MBARI's annotation system are working systems with their own data models; archives such as NCEI, PANGAEA and BCO-DMO are long-term custodians with curator review; cloud services are infrastructure that the operator rents and configures. Each family has a different failure mode: a network rejects a malformed record, an archive rejects incomplete metadata weeks later, and a misconfigured cloud bucket silently publishes what should have been embargoed.
§2.2 defines four patterns. They are not alternatives; a single expedition usually uses all four for different data.
| Pattern | What the spec says | Typical data | Design requirement |
|---|---|---|---|
| 1. Real-time streaming | Live telemetry, WebSocket connections to shore stations, event-driven updates | Vehicle position, alerts, selected sensor channels | Tolerate gaps; never treat the stream as the archive of record |
| 2. Batch synchronization | Periodic uploads from vessels, scheduled repository sync, bulk transfer during port calls | Full-rate sensor series, video, raw sonar | Resumable transfers; file digests checked at both ends |
| 3. Federated query | Distributed search across institutions with a unified interface and aggregated results | Discovery metadata, record summaries | Common vocabulary for fields and units; access rules applied by the data holder |
| 4. Message queue | Asynchronous processing with guaranteed delivery, decoupled producers and consumers | Ingest and conversion jobs, notifications | Idempotent consumers, because "guaranteed" delivery in practice means at-least-once |
The last row deserves emphasis. Queue systems that promise guaranteed delivery almost always deliver at least once, so a consumer will sometimes see the same message twice. Phase 1 already provides the key for de-duplication: the pair (sourceId, sequenceNumber) identifies a message uniquely, because the sequence number increases monotonically per source. Every consumer in the integration layer should store that pair and ignore repeats. Federated query has its own subtlety: results aggregated from several institutions are only comparable if each source reports the same quantity in the same unit, which is exactly what the field mappings in 7.2 are for.
The specification's diagram shows the integration layer from above. The path a single measurement travels is easier to reason about from the side, as below. The stages and link types are an illustration assembled from Phases 1, 3 and 4; the specification does not prescribe this exact topology.
VEHICLE SHIP SHORE ARCHIVE
─────── ──── ───── ───────
sensors ship server WIA Integration Layer NCEI / PANGAEA /
│ Phase 1 envelope │ writes JSONL ┌──────────────────┐ BCO-DMO / OOI
▼ │ builds manifest │ Data Gateway │──► (7.2, 7.3)
onboard log ──acoustic──►│ (Ch.6, low rate) │ Metadata Service │
(full rate) │ │ Security Manager │──► cloud storage
│ │──satellite, live──────►│ Orchestration │ + warehouse
│ ROV: fiber tether ──►│ (pattern 1) └──────────────────┘ (7.4)
│ │ ▲
└──recovery on deck───►│──port call, bulk─────────────┘
(pattern 2: disks or bulk upload)Three observations follow from this picture. First, the version of a measurement that reaches shore in real time is usually a subset: an AUV speaking over an acoustic modem sends a few status values, and only after recovery does the full-rate log reach the ship. The archive must therefore be built from the recovered log, and the streamed copy treated as provisional. Second, the ship is where the Chapter 3 archive layout and its metadata.json manifest should be produced, because that is the last place where operators still remember what each sensor was. Third, the integration layer is where embargo and tier decisions are made, so nothing should bypass it on the way to an archive or a public bucket.
Phase 4 §3 describes three research-network integrations: the Ocean Observatories Initiative (OOI), NOAA's National Centers for Environmental Information (NCEI) and the Monterey Bay Aquarium Research Institute (MBARI). In all three cases the endpoints shown are endpoints on the WIA integration service, not public interfaces of those organisations. The specification does not describe how the WIA service hands the data on, so an implementer must follow each organisation's own current submission procedure. What the specification does define, usefully, is the field mapping.
The OOI submission (§3.1) is a JSON document that describes a set of files rather than carrying the measurements inline. A corrected example, with fictional names and a real-looking but invented grant number, looks like this:
POST https://api.example.org/v1/ooi/data-submission
Content-Type: application/json
Authorization: Bearer <token>
{
"submission": {
"dataType": "CTD",
"platform": "ROV-EXAMPLE-01",
"mission": "EXP-2025-011",
"startTime": "2025-01-15T08:00:00Z",
"endTime": "2025-01-15T16:00:00Z",
"dataFormat": "CF-NetCDF",
"qualityLevel": "VALIDATED",
"files": [
{
"filename": "ctd_2025-01-15_001.nc",
"size": 15728640,
"checksum": "sha256:9b2f…",
"url": "https://data.example.org/datasets/ctd_2025-01-15_001.nc"
}
],
"metadata": {
"chiefScientist": "Dr. Hana Seo",
"institution": "Example Ocean Institute",
"fundingAgency": "Example Science Foundation",
"grantNumber": "EX-0001"
}
}
}The file's checksum should be the same SHA-256 digest recorded in the archive manifest, so the receiver can verify the transfer against the value the ship computed. The qualityLevel should match the dataQuality value carried in the Phase 1 envelopes it summarises.
The mapping table in §3.1 is the most reusable part of Phase 4, because the same mapping serves NCEI and any CF-based consumer. The table below extends it with the CF standard names and adds two corrections. The specification converts temperature from Celsius to Kelvin; CF does not require that, since it accepts any unit convertible to the canonical one, and ocean datasets conventionally publish temperature in degree_Celsius. The rule should be to carry units explicitly and convert only to what the target declares. The pressure conversion from bar to decibar (multiply by 10) is correct, but the table does not say whether the pressure is absolute or relative to the sea surface, and CF has a separate standard name for each.
| WIA field | CF standard name (target) | Target units | Transformation |
|---|---|---|---|
timestamp | time | e.g. seconds since 1970-01-01T00:00:00Z | Parse ISO 8601 UTC; keep milliseconds as a fractional part |
location.depth | depth | m, attribute positive="down" | None; state the positive direction |
environment.temperature | sea_water_temperature | degree_Celsius (or K if the target declares it) | None by default; add 273.15 only if the target requires kelvin |
environment.pressure (absolute) | sea_water_pressure | dbar | bar × 10 |
environment.pressure (sea pressure) | sea_water_pressure_due_to_sea_water | dbar | bar × 10; or from absolute: subtract 10.1325 dbar |
environment.salinity | sea_water_practical_salinity | 1 (dimensionless) | None numerically; drop the "PSU" label, which is not a unit |
environment.dissolvedOxygen | mass_concentration_of_oxygen_in_sea_water | kg m-3 | mg/L × 0.001 |
environment.soundVelocity | speed_of_sound_in_sea_water | m s-1 | None |
waterCurrent.direction | direction_of_sea_water_velocity | degree | CF means "towards"; the spec does not say whether its direction is "towards" or "from", so the source must document it |
A quick sanity check follows from the physics in Chapter 1. Sea pressure in decibars is numerically close to depth in metres; at 3,547 m, hydrostatics with ρ ≈ 1025 kg/m³ gives about 3,565 dbar of sea pressure. A converted pressure column that differs from depth by a factor of ten, or by more than a few percent, points to a unit error, and a validator can check this automatically. The spec's own sample message, which gives 354.72 bar at 3,547.2 m, is simply depth divided by ten and should not be used as a test value (see Chapter 3).
§3.2 describes the NCEI path as a pipeline: WIA data, quality check, format conversion to CF-NetCDF, filling an NCEI metadata template, and submission, after which NCEI assigns an accession number. The required metadata is listed only in general terms: global attributes such as title, institution, source and references; cruise and platform information; instrument descriptions; quality-control procedures; processing history; and citation information. In practice NCEI's NetCDF templates build on the CF Conventions and on the Attribute Convention for Data Discovery (ACDD), so the global attributes should follow ACDD names (title, summary, keywords, creator_name, time_coverage_start and so on). The specification's Python example imports a wia_dse package that is not part of the shipped reference code (which provides the TypeScript SDK @wia/ene-040), so it should be read as pseudocode.
The export itself is straightforward once the archive manifest from Chapter 3 describes the columns of each compact JSONL series. The example below reads one CTD series, renames the columns to CF variables, converts units according to the manifest, and writes a CF-NetCDF file with ACDD global attributes. It uses pandas and xarray; column names, units and the absolute-pressure assumption are those declared in this example manifest.
import json
import pandas as pd
import xarray as xr
ARCHIVE = "mission-MISSION-2025-003"
SERIES = "oceanographic/ctd-data-001.jsonl"
manifest = json.load(open(f"{ARCHIVE}/metadata.json", encoding="utf-8"))
cols = manifest["series"][SERIES]["columns"]
df = pd.read_json(f"{ARCHIVE}/{SERIES}", lines=True)
df["time"] = pd.to_datetime(df["timestamp"], utc=True).dt.tz_localize(None)
df = df.drop(columns="timestamp").set_index("time").sort_index()
# column -> (CF variable, standard_name, target units, converter)
CF = {
"depth": ("depth", "depth", "m", lambda v: v),
"temp": ("TEMP", "sea_water_temperature", "degree_Celsius", lambda v: v),
"press": ("PRES", "sea_water_pressure", "dbar", lambda v: v * 10.0), # bar -> dbar
"sal": ("PSAL", "sea_water_practical_salinity", "1", lambda v: v),
}
ds = xr.Dataset()
for col, (name, std, units, conv) in CF.items():
if col not in df:
continue
src = cols[col]
if col == "press" and src.get("unit") != "bar":
raise ValueError(f"expected bar for {col}, manifest says {src.get('unit')}")
ds[name] = ("time", conv(df[col].to_numpy(dtype="float64")))
ds[name].attrs = {"standard_name": std, "units": units,
"sensor_id": src.get("sensorId", ""),
"calibration_date": src.get("calibrationDate", "")}
ds = ds.assign_coords(time=df.index.values)
ds["depth"].attrs["positive"] = "down"
ds.attrs = {
"Conventions": "CF-1.8, ACDD-1.3",
"title": "CTD time series, dive DIVE-ROV-2025-003 (example)",
"summary": "Example export of a WIA-ENE-040 JSONL series to CF-NetCDF.",
"institution": "Example Ocean Institute",
"source": "ROV-EXAMPLE-01 CTD",
"creator_name": "Hana Seo",
"history": f"converted from {SERIES} by the WIA Data Gateway",
"wia_mission": manifest["missionId"],
"time_coverage_start": str(df.index.min()) + "Z",
"time_coverage_end": str(df.index.max()) + "Z",
}
encoding = {"time": {"units": "seconds since 1970-01-01T00:00:00Z",
"dtype": "float64"}}
ds.to_netcdf("ctd_2025-01-15_001.nc", encoding=encoding)Two design points are embedded in this code. The converter refuses to run when the manifest does not declare the unit it expects, which is better than producing a plausible file with wrong numbers. And each variable keeps the sensorId and calibration date from the manifest, so the provenance that Phase 1 carries per value survives the conversion, if only at the variable level.
§3.3 addresses two MBARI systems. VARS (the Video Annotation and Reference System) records what observers see in video: a concept (usually a taxon name), a timecode, the observer, a position, and free key-value observations. The WIA example posts a video reference and a list of annotations:
{
"video": {
"videoReferenceUuid": "4f6c2a1e-0000-4000-8000-000000000001",
"missionId": "MISSION-2025-003",
"vehicleId": "ROV-EXAMPLE-01",
"startTime": "2025-01-15T10:30:00Z",
"duration": 7200,
"videoUrl": "https://data.example.org/video/exp-2025-011-v01.mp4"
},
"annotations": [
{
"timestamp": "2025-01-15T10:35:23.450Z",
"timecode": "00:05:23.450",
"concept": "Riftia pachyptila",
"observer": "J. Example",
"location": { "latitude": 46.0512, "longitude": -130.0523, "depth": 2234.5 },
"observations": [
{ "key": "colony-size", "value": "large", "unit": null },
{ "key": "tube-length", "value": 1.2, "unit": "m" }
]
}
]
}Three consistency rules make these annotations joinable with everything else. The timestamp must equal startTime plus timecode; in the example, 10:30:00 plus 5 minutes 23.450 seconds gives 10:35:23.450, and a validator should reject a mismatch larger than one video frame. The position should come from the navigation record at that timestamp, not be typed in by the observer. And the concept should be carried into the Chapter 4 OrganismObservation record as scientificName, with the annotation's identifier kept as a cross-reference, so the same animal does not appear twice in an OBIS export. Numeric observations should be numbers with a unit symbol ("m"), not strings with "meters".
For M3, MBARI's information management system for dive metadata, the specification says only that dive metadata, sample tracking and image references are synchronised automatically. There is no field-level definition, so an implementation should start from the DiveLog and sample records of Chapter 4 and agree the mapping with the receiving institution.
Research networks use data; repositories keep it and make it citable. Phase 4 §4 and §6 cover PANGAEA, BCO-DMO, ISO 19115 and DataCite. The common thread is the persistent identifier: once a dataset has a DOI, its citation must keep working even if hosts, file names and formats change.
§4.1 describes a three-stage workflow, summarised below with the checks each stage needs on the WIA side.
| Stage | Spec content | What the WIA side must supply |
|---|---|---|
| 1. Preparation | Convert to PANGAEA's data format, generate DOI-ready metadata, create the citation | Parameter names with units, an author list with ORCIDs, spatial and temporal coverage including minimum and maximum depth |
| 2. Quality assurance | Automated checks, manual curator review, DOI reservation | Answers to curator questions; the processing history from the manifest |
| 3. Publication | Upload, DOI assignment, public release | The release date, which must respect any embargo (7.5) |
The specification's XML template has one misleading detail: it fills in a <doi> element before submission. A DOI is minted by the repository during stages 2 and 3, not chosen by the submitter, so the WIA record should hold a placeholder until PANGAEA returns the identifier, and the Metadata Service should then write the DOI back into the Chapter 4 records. The template also gives spatial coverage as a bounding box with minimumDepth and maximumDepth; these should be computed from the data, not copied from the expedition plan's target depth range.
BCO-DMO works project-first: a project is registered, then datasets are attached to it. The §4.2 registration carries title, acronym, dates, lead PI (name, institution, e-mail, ORCID), funding awards, study area and keywords. Datasets follow as CSV or spreadsheet files converted from WIA data, with parameter descriptions, method documentation, quality flags and processing history. The WIA expeditionId maps naturally to a BCO-DMO project or to a deployment within one, and should be recorded in the project's description so the two systems can be joined later. When converting to CSV, units belong in a parameter description, not in cell values, and the Phase 1 OUT_OF_RANGE flag should become a quality-flag column rather than a deleted row.
§6.2 gives a DataCite registration request. The mandatory properties of the DataCite metadata schema are the identifier, creators, titles, publisher, publication year and resource type, and the example includes all of them, plus an abstract and a geographic bounding box. A corrected version with fictional values:
POST https://api.test.datacite.org/dois
Content-Type: application/vnd.api+json
{
"data": {
"type": "dois",
"attributes": {
"doi": "10.5072/wia-exp-2025-011",
"creators": [{
"name": "Seo, Hana", "nameType": "Personal",
"givenName": "Hana", "familyName": "Seo",
"nameIdentifiers": [{ "nameIdentifier": "https://orcid.org/0000-0000-0000-0000",
"nameIdentifierScheme": "ORCID" }]
}],
"titles": [{ "title": "Expedition EXP-2025-011: CTD, bathymetry and dive records" }],
"publisher": "Example Ocean Data Repository",
"publicationYear": 2027,
"types": { "resourceTypeGeneral": "Dataset" },
"descriptions": [{ "description": "Example dataset record.", "descriptionType": "Abstract" }],
"geoLocations": [{ "geoLocationBox": {
"westBoundLongitude": -130.1, "eastBoundLongitude": -130.0,
"southBoundLatitude": 46.0, "northBoundLatitude": 46.1 } }]
}
}
}Three points about this example deserve attention. The prefix 10.5072, used here as in the specification, is DataCite's reserved test prefix and never resolves; a production DOI uses the repository's own prefix, and drafts should first be created against the test service shown here. The resource type is carried in the types object of the current REST API. And the publication year is the year the data become public, which under a two-year embargo is not the collection year. For a confidential-tier dataset (7.5), the geoLocations box must itself be generalised, because DOI metadata is always public even when the data are not.
§6.1 claims full compliance with ISO 19115 and lists ten "required" elements. That claim cannot be verified from the list alone. ISO 19115-1 defines which elements are mandatory, conditional or optional, and conformance is shown by producing a valid encoding (commonly the ISO 19115-3 XML schemas) that a validator accepts. It is safer to read the table as a WIA profile: the elements a WIA implementation must always fill. The table below maps each one to its source in the WIA records.
| Element | Meaning | Source in WIA data |
|---|---|---|
| Title | Dataset name | Written by the data manager; include expedition and dive identifiers |
| Abstract | Description | Expedition objectives plus dive summary |
| Purpose | Why the data were collected | Expedition.objectives |
| Status | Completed, ongoing, planned | Mission state; "completed" after recovery |
| Point of contact | Responsible party | Chief scientist and data manager, with ORCID |
| Maintenance frequency | Update schedule | "not planned" for a finished dive; "as needed" for curated collections |
| Keywords | Subject terms | Controlled vocabulary preferred (e.g. CF standard names for parameters) |
| Geographic extent | Bounding box | Computed from navigation, generalised for confidential data |
| Temporal extent | Time coverage | First and last timestamps in the series |
| Lineage | Processing history | Manifest history: raw, QC, calibration, conversion, archive |
Phase 4 §5 gives one example per major cloud provider: object storage and a serverless trigger on AWS, blob storage with tiering and soft delete on Azure, and an analytics warehouse on Google Cloud. The standard is vendor-neutral in intent, and the patterns underneath the examples apply on any platform. This section presents them that way, using the provider names only to show what each pattern is called in practice.
A mission archive is large, written once and read rarely, which is exactly what object storage with lifecycle rules is built for. The specification's AWS example uploads a whole mission directly to a Glacier storage class, and its Azure example uses the Cool tier. The choice matters more than the examples suggest, because some archive tiers (for instance S3 Glacier Flexible Retrieval or Azure's Archive tier) require a restore step, which can take hours, before an object can be read. A sensible layout keeps recent and frequently used data online and moves the rest down as it ages.
| Tier | Use for | Examples of provider names | Caution |
|---|---|---|---|
| Hot / standard | Current expedition, data under active processing, public derived products | S3 Standard; Azure Hot; GCS Standard | Highest storage cost |
| Cool / infrequent | Data from recent expeditions, embargoed data awaiting release | S3 Standard-IA; Azure Cool; GCS Nearline or Coldline | Per-read retrieval charges; minimum storage durations |
| Archive | Raw video and sonar kept for preservation, superseded versions | S3 Glacier classes; Azure Archive; GCS Archive | Restore delay on some services; do not put the only copy of a dataset users need here |
Two further settings from the Azure example apply everywhere: object versioning and soft-delete retention, which protect against accidental overwrites. The specification's AWS snippet passes an access key and secret key as literals; real deployments should use the platform's workload identity or role mechanism and never embed credentials in code. The same snippet creates a public content-delivery distribution in front of the bucket. That is appropriate only for a bucket that holds nothing but public-tier data, because a CDN cannot evaluate embargo dates.
The AWS Lambda example in §5.1 runs on every upload: it parses the file, generates thumbnails for images and indexes the content for search. The pattern (object created, then function runs) exists on every platform. The example has three defects worth avoiding. It reads only the first record of the event, while one event may carry several objects. It has no idempotency check, so a retried event creates duplicate index entries. And it derives the thumbnail name with a string replacement on .jpg, which silently overwrites the original when the key has a different extension. A robust handler loops over all records, uses the object's key and version (or the Phase 1 source and sequence pair) as an idempotency key, and writes derived products under a separate prefix. The index it feeds should contain only metadata and public-tier content, because a search index is a place where access control is easy to forget.
The GCP example creates a telemetry table in BigQuery, partitioned by day and clustered by vehicle, and loads JSONL from object storage. Partitioning by date and clustering by the vehicle identifier is the right shape for this data, because almost every query selects a time window and a vehicle. The version below keeps that shape and adds what the original lacks: the de-duplication key, the quality flag, and explicit units in column names. It avoids the column name timestamp, which is also a type name, and it uses the dialect of BigQuery; other warehouses need small syntax changes.
CREATE SCHEMA IF NOT EXISTS wia_deep_sea
OPTIONS (location = 'US', description = 'WIA-ENE-040 telemetry (example)');
CREATE TABLE IF NOT EXISTS wia_deep_sea.vehicle_telemetry (
obs_time TIMESTAMP NOT NULL,
source_id STRING NOT NULL, -- Phase 1 sourceId
sequence_number INT64 NOT NULL, -- Phase 1 sequenceNumber
mission_id STRING,
latitude_deg FLOAT64,
longitude_deg FLOAT64,
depth_m FLOAT64,
temperature_c FLOAT64,
pressure_dbar FLOAT64,
salinity FLOAT64, -- practical salinity, dimensionless
battery_pct INT64,
data_quality STRING, -- e.g. VALIDATED, OUT_OF_RANGE
access_tier STRING -- PUBLIC / RESTRICTED / CONFIDENTIAL
)
PARTITION BY DATE(obs_time)
CLUSTER BY source_id;
-- Deepest point and mean temperature per vehicle on one day,
-- counting each (source_id, sequence_number) once.
SELECT
source_id,
MAX(depth_m) AS max_depth_m,
AVG(temperature_c) AS mean_temperature_c,
COUNT(*) AS samples
FROM (
SELECT * FROM wia_deep_sea.vehicle_telemetry
WHERE DATE(obs_time) = '2025-01-15'
AND data_quality != 'OUT_OF_RANGE'
QUALIFY ROW_NUMBER() OVER (PARTITION BY source_id, sequence_number
ORDER BY obs_time) = 1
)
WHERE depth_m > 3000
GROUP BY source_id;The specification loads the files directly with a LOAD DATA statement. That works only if the JSONL keys already match the column names, and the compact rows of Phase 1 §5.2 use short keys such as temp. A mapping step, driven by the manifest as in the NetCDF exporter above, has to sit between the archive and the warehouse. Two other items from §5 are out of date or out of scope. Azure Data Lake Analytics and its U-SQL language have been retired by Microsoft, so they should not be used for new work; a current Spark- or SQL-based service fills the same role. And the Earth Engine paragraph describes a possible use, not an integration the standard defines.
Chapter 4 classified every record into one of three access tiers taken from ENE-040 §14.1: public (low-resolution bathymetry on a 50 m grid, species-level observation lists, expedition metadata), restricted (high-resolution bathymetry below 10 m under a two-year embargo, genetic data requiring researcher consent, precise mineral locations), and confidential (commercial and military data, exact coordinates of sensitive habitats). Phase 4 §7 adds the mechanism that enforces those tiers: roles, embargoes, licences and an access log.
| Role | Spec permissions | Tier access (this book's reading) |
|---|---|---|
| PUBLIC | Read public datasets | Public tier only; embargoed data invisible, including file listings |
| RESEARCHER | Read all, write own projects | Public tier plus restricted data of projects the user is granted; embargoed data only for members of the owning project |
| CURATOR | Read all, write all, approve submissions | All tiers needed for curation; confidential data only for datasets assigned to them |
| ADMIN | Full access, user management | Manages grants; access to confidential content should itself be logged and justified |
The specification's "read all" for researchers cannot be taken literally, because it would expose restricted and confidential data to every registered user and make the §14.1 tiers meaningless. The third column shows the interpretation this book recommends: roles grant capabilities, and the tier, the project grant and the embargo date together decide what a particular user may read. The permission model in §7.1 already has the parts needed for this: per-project permissions with an expiry date, a list of institutions, and an embargoOverride flag. A compact decision function makes the rule explicit:
type Role = "PUBLIC" | "RESEARCHER" | "CURATOR" | "ADMIN";
type Tier = "PUBLIC" | "RESTRICTED" | "CONFIDENTIAL";
interface Dataset {
id: string; projectId: string; tier: Tier;
embargoUntil?: string; // ISO 8601; absent = no embargo
assignedCurators: string[];
}
interface User {
id: string; roles: Role[];
projects: { projectId: string; permissions: string[]; expiresAt: string }[];
embargoOverride: boolean;
}
function canRead(u: User, d: Dataset, now = new Date()): boolean {
const grant = u.projects.find(p => p.projectId === d.projectId
&& p.permissions.includes("READ") && new Date(p.expiresAt) > now);
const curator = u.roles.includes("CURATOR") && d.assignedCurators.includes(u.id);
const embargoed = d.embargoUntil !== undefined && new Date(d.embargoUntil) > now;
if (embargoed && !grant && !curator && !u.embargoOverride) return false;
switch (d.tier) {
case "PUBLIC": return true;
case "RESTRICTED": return Boolean(grant) || curator;
case "CONFIDENTIAL": return curator || (Boolean(grant) && u.roles.includes("ADMIN"));
}
}The function is deliberately strict: the default for anything not matched is denial, and embargoOverride lifts only the embargo, never the tier. Who may set that override is an organisational decision that the specification leaves open; it should be a curator action recorded in the access log.
§7.2 sets the default embargo at two years from data collection, configurable per dataset and waivable for open-science projects. This is how the "open access by default" principle of §1.2 and the two-year default coexist: the default applies to restricted-tier data, while public-tier products are open immediately. The embargo date should be computed from the collection date in the Chapter 4 records, stored on the dataset, and enforced by the Orchestration Engine, which publishes the data and sends DOI and repository updates on that date. Metadata describing an embargoed dataset can usually be published at once, so other researchers know the data exist, as long as the metadata itself respects the tier (generalised coordinates for confidential data).
The specification offers four licence options: CC0, CC BY 4.0, CC BY-NC 4.0 and custom institutional licences. For data meant to be combined with other datasets, CC0 or CC BY 4.0 is the practical choice. A non-commercial clause is hard to interpret for data that ends up in aggregators and derived products, and it can stop a dataset from being included in collections that require open licences. The licence must be recorded in the DataCite and ISO 19115 metadata as well as on the files, so it travels with every copy.
The access_log table in §7.2 mixes SQL dialects: UUID and INET are PostgreSQL types, while an inline ENUM(...) column is MySQL syntax. A PostgreSQL version that keeps the specification's columns:
CREATE TYPE access_type AS ENUM ('VIEW', 'DOWNLOAD', 'API');
CREATE TABLE access_log (
access_id UUID PRIMARY KEY,
accessed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
user_id VARCHAR(255) NOT NULL,
dataset_id VARCHAR(255) NOT NULL,
access_type access_type NOT NULL,
ip_address INET,
user_agent TEXT,
bytes_transferred BIGINT CHECK (bytes_transferred >= 0)
);
CREATE INDEX access_log_dataset_time ON access_log (dataset_id, accessed_at);
-- Downloads per dataset over the last 30 days
SELECT dataset_id,
COUNT(DISTINCT user_id) AS unique_users,
COUNT(*) AS downloads,
SUM(bytes_transferred) / 1024.0 ^ 3 AS gib
FROM access_log
WHERE access_type = 'DOWNLOAD'
AND accessed_at >= now() - INTERVAL '30 days'
GROUP BY dataset_id
ORDER BY unique_users DESC;IP addresses and user agents are personal data in many jurisdictions, so the log needs a stated retention period after which those columns are removed or truncated, while the counts used for reporting are kept. Access to confidential-tier data should be logged even when it is denied, because repeated refused attempts are exactly the signal a data manager needs to see.
Most institutions adopting WIA-ENE-040 already hold years of data in NetCDF, vendor formats and spreadsheets. §9.1 describes a four-stage migration. The table restates it with an exit criterion for each stage, which the specification does not give.
| Stage | Spec content | Exit criterion (recommended) |
|---|---|---|
| 1. Assessment | Inventory formats, identify transformations, estimate effort | Every legacy format has a named converter or a documented decision to leave it as is |
| 2. Pilot | Pick a representative dataset, write conversion scripts, validate output | Round trip reproduces the original values within stated tolerances; a data manager has reviewed the result |
| 3. Bulk migration and parallel run | Legacy and WIA systems run side by side; incremental conversion; verification | For an agreed period, both systems answer the same queries with the same results |
| 4. Cutover | Deprecate legacy formats, update integrations, archive legacy systems | Legacy archive frozen read-only with digests; DOIs and links redirected to the new locations |
The specification's migration example calls a LegacyConverter from the same unshipped wia_dse package. The conversion it describes, NetCDF to Phase 1 records followed by validation, can be written directly. The sketch below reads a legacy CF-NetCDF CTD file, converts it into Phase 1 OCEANOGRAPHIC_DATA envelopes written as JSONL, applies the Phase 1 §3.2 range limits, and flags rather than drops values outside them. The checksum step is left to the envelope library from Chapter 3.
import json
import numpy as np
import xarray as xr
RANGES = {"depth": (0, 11000), "temperature": (-2, 400),
"pressure": (0, 1100), "salinity": (0, 50)} # Phase 1 section 3.2
ds = xr.open_dataset("legacy_ctd.nc")
t = ds["time"].values.astype("datetime64[ms]")
depth = ds["depth"].values
temp = ds["TEMP"].values # degree_Celsius assumed; check attrs
pres_dbar = ds["PRES"].values
sal = ds["PSAL"].values
assert ds["TEMP"].attrs.get("units") in ("degree_Celsius", "degC", "Celsius")
def measured(v, unit, sensor):
return {"value": round(float(v), 4), "unit": unit, "sensorId": sensor,
"calibrationDate": ds.attrs.get("calibration_date", "unknown")}
with open("ctd-legacy-001.jsonl", "w", encoding="utf-8") as out:
for i in range(len(t)):
env = {"temperature": measured(temp[i], "celsius", "LEGACY-T"),
"pressure": measured(pres_dbar[i] / 10.0, "bar", "LEGACY-P"),
"salinity": measured(sal[i], "PSU", "LEGACY-S")}
checks = {"depth": depth[i], "temperature": temp[i],
"pressure": pres_dbar[i] / 10.0, "salinity": sal[i]}
bad = [k for k, v in checks.items()
if np.isnan(v) or not RANGES[k][0] <= v <= RANGES[k][1]]
msg = {
"wiaVersion": "1.0",
"messageType": "OCEANOGRAPHIC_DATA",
"timestamp": np.datetime_as_string(t[i], unit="ms") + "Z",
"sequenceNumber": i,
"sourceId": "LEGACY-IMPORT-001",
"priority": "LOW",
"payload": {"location": {"latitude": float(ds["latitude"]),
"longitude": float(ds["longitude"]),
"depth": float(depth[i]),
"coordinateSystem": "WGS84"},
"environment": env},
"metadata": {"mission": "LEGACY-IMPORT-001",
"dataQuality": "OUT_OF_RANGE" if bad else "LEGACY",
"flaggedFields": bad,
"source": ds.attrs.get("title", "legacy archive")},
}
out.write(json.dumps(msg, ensure_ascii=False) + "\n")Three choices in this sketch are deliberate. The unit assertion stops the run if the legacy file's temperature is not in Celsius, instead of guessing. The value LEGACY in dataQuality marks records whose calibration history is incomplete; it is a local extension, and like any extension it must be documented in the conformance statement (Chapter 8). And the validation is followed by the reverse direction: exporting the new JSONL back to CF-NetCDF with the 7.2 exporter and comparing the arrays with the original file is the simplest proof that the pilot stage has succeeded.
Phase 4 describes where WIA data go after a dive, but many of its examples are sketches that an implementer must complete. The table collects the decisions this chapter has argued for.
| Topic | Specification | Decision to make |
|---|---|---|
| Delivery semantics | "Guaranteed delivery" queues | De-duplicate on (sourceId, sequenceNumber) everywhere |
| Units in mappings | Celsius to Kelvin; bar to dbar | Carry units explicitly; convert only to the target's declared units; state absolute or sea pressure |
| External endpoints | Examples on several WIA hosts | Follow each organisation's current submission procedure; treat hosts as examples |
| DOIs | DOI written into the template | Repository mints it; write it back into the records; publicationYear is the release year |
| ISO 19115 | "Fully compliant" | Treat the list as a WIA profile; validate a real encoding |
| Cloud | Provider-specific snippets | Lifecycle tiers, no embedded credentials, idempotent triggers, public CDN only for public data |
| Access | Researcher "reads all" | Tier plus project grant plus embargo decide; deny by default |
| Migration | Four stages | Exit criteria per stage; round-trip comparison before cutover |
Phase 4 also defines its own certification levels, which differ from those in Phase 1. Chapter 8 sets the two side by side and explains how an implementation can test its conformance to both.