Phase 1 of WIA-ENE-040 is the part of the standard that everything else leans on. The Phase 2 API returns Phase 1 messages, the Phase 3 acoustic stack carries them in fragments, and Phase 4 repositories ingest them. If two systems disagree about what a depth of 3547.2 means, or which bytes a checksum covers, no later API design will reconcile their data. This chapter reads PHASE-1-DATA-FORMAT.md field by field, shows how to produce and validate messages, and marks each place where the text leaves an implementer to guess, with a recommendation labelled as this book's rather than the standard's.
Phase 1 has four layers, and the sections below follow them in order: a common JSON envelope; four typed payloads (oceanographic, bathymetric, sample and telemetry); an optional binary encoding for acoustic links; and an archive layout for a finished mission. Validation rules cut across all four and get their own section.
Every Phase 1 message, whatever it carries, is a JSON object with the same nine top-level members. The specification's field table (§2.1) is short, so the table below adds the rule an implementer actually needs for each field. The rules come from §2.1 and §3.1 where the specification states them. Where it does not, the table gives this book's recommendation and marks it as such.
| Field | Type | Required | Rule |
|---|---|---|---|
wiaVersion | string | Yes | Must match the specification version. The table calls it "semantic versioning" while the examples use "1.0" and the binary header has room only for major and minor. Recommendation: send MAJOR.MINOR; a patch release never changes the wire format. |
messageType | enum | Yes | One of the enumerated types. The payloads defined in §2.2–2.5 are OCEANOGRAPHIC_DATA, BATHYMETRIC_DATA, SAMPLE_METADATA and VEHICLE_TELEMETRY; the table also names DATA_PACKET, TELEMETRY and COMMAND followed by "etc." |
timestamp | ISO 8601 string | Yes | UTC with a Z suffix and millisecond precision, for example 2025-01-15T14:30:00.000Z. It is the time of measurement, not of transmission. |
sequenceNumber | integer | Yes | Increases monotonically per sourceId. The binary header holds it in 32 bits, so the JSON value should stay within 0–4,294,967,295 (recommendation). |
sourceId | string | Yes | Letters, digits, hyphens and underscores only (§3.1). The specification sets no length limit. Recommendation: at most 64 characters, unique within the operating organisation. |
priority | enum | Yes | LOW, NORMAL, HIGH or CRITICAL. In the binary header these become 0–3. |
payload | object | Yes | Structure is determined by messageType. |
metadata | object | No | Free-form context: mission, institution, vessel, quality state, time-sync record. It is not covered by the checksum (see 3.5). |
checksum | string | Yes | SHA256: followed by the hex digest of the payload. The canonical form of the input is not specified; 3.5 fixes it. |
Three fields together say who produced a record and where it sits in that producer's history. The pair (sourceId, sequenceNumber) is the natural duplicate key. An acoustic link that retransmits a frame, or a ship-to-shore sync that replays a day of telemetry, will deliver the same message more than once, and a receiver that stores by this pair gets idempotent ingestion without any extra machinery. The same pair exposes loss: a jump from 1041 to 1045 means three messages are missing, and the receiver can request them or record the gap.
"Monotonically increasing" is easy to break. A vehicle computer that reboots mid-dive and restarts its counter at zero will have every later message rejected by a strict receiver. Either persist the counter to non-volatile storage before each increment, or change the sourceId suffix on each power cycle (ROV-EXAMPLE-01_B7 for boot 7). Record the choice in the conformance documentation that §6.2 asks for. Receivers should never parse meaning out of a sourceId; the vehicle type belongs in the payload.
The field table allows DATA_PACKET, TELEMETRY and COMMAND "etc.", the payload sections use VEHICLE_TELEMETRY, and Phase 3 (§6) adds RESPONSE. The closed list that §3.1 requires validators to check against is never published. The safer receiver accepts only the types it knows, which is what the validator in 3.5 does. This book treats TELEMETRY as an unresolved alias of VEHICLE_TELEMETRY and proposes a published registry in a v1.x revision (Chapter 8).
priority is a hint to the transport, not a statement about data quality. Phase 3 maps it onto its QoS classes, so a CRITICAL leak alarm overtakes a backlog of NORMAL CTD records on a slow acoustic link. Marking routine data HIGH defeats that.
metadata is optional and free-form; the specification's example carries mission, institution, vessel, chief scientist and a dataQuality state. Because it sits outside the checksum, it suits information that downstream systems add, and it is the wrong home for anything that must be protected against change. Two members are worth standardising locally: the expeditionId and diveId that link a message to the Chapter 4 science records (the convention recommended in Section 2.4), and a timeSync block (3.5).
The oceanographic packet carries a snapshot of the water around a sensor platform: where it is, what the water is like, and how the water is moving. The example below follows the structure of §2.2. The identifiers and institution are fictitious, and the measured values are illustrative but physically consistent with one another. The checksum is real: it is the SHA-256 digest of this payload in the canonical form defined in 3.5.
{
"wiaVersion": "1.0",
"messageType": "OCEANOGRAPHIC_DATA",
"timestamp": "2025-01-15T14:30:00.000Z",
"sequenceNumber": 12345,
"sourceId": "ROV-EXAMPLE-01",
"priority": "NORMAL",
"payload": {
"location": {
"latitude": 36.7977, "longitude": -121.8472,
"depth": 3547.2, "altitude": 12.5,
"coordinateSystem": "WGS84",
"accuracy": { "horizontal": 2.3, "vertical": 0.8, "unit": "meters" }
},
"environment": {
"temperature": { "value": 2.47, "unit": "celsius", "sensorId": "CTD-07-T", "calibrationDate": "2024-11-20" },
"pressure": { "value": 357.6, "unit": "bar", "sensorId": "CTD-07-P", "calibrationDate": "2024-11-20" },
"salinity": { "value": 34.68, "unit": "PSU", "sensorId": "CTD-07-C", "calibrationDate": "2024-11-20" },
"dissolvedOxygen": { "value": 3.1, "unit": "mg/L", "sensorId": "DO-02", "calibrationDate": "2024-10-05" },
"soundVelocity": { "value": 1513.6, "unit": "m/s", "sensorId": "SV-01", "calibrationDate": "2024-09-30" }
},
"waterCurrent": { "speed": 0.08, "direction": 147.5, "unit": "m/s", "directionReference": "true-north" }
},
"metadata": {
"expeditionId": "EXP-2025-011",
"diveId": "DIVE-ROV-2025-003",
"institution": "Example Ocean Institute",
"researchVessel": "R/V Example",
"dataQuality": "VALIDATED",
"timeSync": { "source": "GPS", "lastSync": "2025-01-15T11:02:10.000Z", "estimatedDriftSeconds": 0.04 }
},
"checksum": "SHA256:d24fca65e101a56d472ccbc1f9cd031dd519b1f237009ab0c458ec1df64b9bac"
}latitude and longitude are decimal degrees on WGS84, as coordinateSystem states. depth is in metres, positive downward from the sea surface, and it describes the sensor or vehicle, not the seafloor. altitude is the height of the vehicle above the seabed, typically measured by a DVL or an altimeter. Their sum is therefore an estimate of water depth at that position (3547.2 + 12.5 ≈ 3,560 m in the example). That makes a cheap cross-check against the bathymetry of the same area.
The accuracy block gives horizontal and vertical uncertainty in metres but does not say at what confidence. A USBL fix quoted at one standard deviation and the same fix quoted at 95% differ by roughly a factor of two. Recommendation: state the confidence level explicitly, for example "confidence": 0.95. Hydrographic survey practice under IHO S-44 reports uncertainty at 95%, and following it avoids a conversion when data reaches a charting office.
Every entry in environment is an object, not a bare number: value, unit, sensorId and calibrationDate. This is the most useful design decision in Phase 1. A value without a unit is ambiguous; a value without a sensor cannot be corrected when that sensor is later found to have drifted; a value without a calibration date cannot be matched to a certificate that arrives after the cruise. The provenance costs bytes, but it lets an archive correct exactly the affected values years later.
Two refinements make the object work in practice. First, the unit strings need a controlled vocabulary. The specification uses celsius, bar, PSU, mg/L, pH, NTU and m/s. The table below maps each to the form commonly used by CF-convention NetCDF, which is what Phase 4 exports to, and notes the traps. Second, a quality flag belongs on the value itself, because a range violation is a property of one measurement, not of the whole message. The schema fragment in 3.5 therefore adds an optional qualityFlag member.
| Quantity | Phase 1 unit string | Common CF/UDUNITS form | Note |
|---|---|---|---|
| Temperature | celsius | degC | Keep Celsius. Converting to kelvin for exchange adds nothing and is error-prone (see Chapter 7). |
| Pressure | bar | dbar | 1 bar = 10 dbar. State whether the value is absolute or sea pressure (absolute minus one standard atmosphere). |
| Salinity | PSU | 1 (dimensionless) | Practical salinity (PSS-78) has no unit; "PSU" is a label. Absolute salinity under TEOS-10 is in g/kg and is a different quantity. |
| Dissolved oxygen | mg/L | mg L-1 or umol kg-1 | Oceanographic archives often prefer µmol/kg; conversion needs density, so never convert silently. |
| pH | pH | 1 | Also record the pH scale (total, seawater, NBS), because values on different scales differ. |
| Turbidity | NTU | no standard CF unit | Instrument-relative; keep the sensor model with the data. |
| Sound velocity | m/s | m s-1 | Feeds ray tracing for multibeam processing (3.3). |
The specification's example puts a sensor at 3547.2 m with a pressure of 354.72 bar: exactly depth divided by ten, the Chapter 1 rule of thumb, not a measurement. Hydrostatics with ρ ≈ 1025 kg/m³ gives about 356.6 bar of sea pressure, or about 357.6 bar absolute, which is the figure used above. Two things must be settled. Is the value absolute or sea pressure (CTDs conventionally report sea pressure in decibar)? And is it measured or derived from depth? A derived value must never carry a pressure sensor's sensorId.
Current is a separate block rather than a set of measured values: speed, direction, a shared unit and a directionReference. Two conventions must be pinned down. The first is the reference: true-north is stated, but a magnetic reference would silently rotate every vector by the local declination. The second is whether direction means the way the water flows to (the usual oceanographic convention) or comes from (the meteorological one). The specification does not say. Recommendation: flow-to, and write it into the conformance documentation. The block also lacks sensorId and calibrationDate; add them in the same form as the environment entries.
The bathymetric message carries seafloor depth measurements from a sonar, usually a multibeam echosounder, together with enough configuration to interpret them. It has four blocks.
surveyArea gives a bounding box as northWest and southEast corners, a gridResolution with gridUnit, and the coordinate system. The corner form breaks down for a survey that crosses the 180° meridian, where the western corner has a larger longitude than the eastern one. Receivers should accept that case and interpret it as a crossing, as the GeoJSON bounding-box rules in RFC 7946 do, rather than reject it as an inverted box.
sonarConfiguration records the sonar type (MULTIBEAM in the example), model, frequency, beam width and swath width, each with its unit. The swath width is an angle (165° in the example), not a distance. The ground width it covers grows with altitude, roughly 2·h·tan(θ/2), which for 165° is about fifteen times the height above the seabed. Frequency and platform must also agree. A 400 kHz sonar reaches only a few hundred metres, so a 400 kHz sounding at 3,547 m makes sense only from a vehicle near the bottom, as with the AUV source in the specification's example.
soundings[] is the payload proper. Each sounding has latitude, longitude, depth, uncertainty, intensity, beamAngle and quality. Here depth is the depth of the seafloor at that footprint, computed as the depth of the sonar plus the vertical component of the ray-traced range. It is not the vehicle's depth, and mixing the two is a common import bug. Three fields need conventions the specification leaves open:
intensity (142 in the example) has no unit. Backscatter can be reported in dB or in raw counts, and the two cannot be merged into one mosaic without calibration. Require a unit and a note on whether the value is calibrated.beamAngle needs a sign convention. Recommendation: degrees from vertical, positive to starboard.quality uses VERIFIED, while the oceanographic metadata uses VALIDATED. Neither vocabulary is defined. Choose a scheme and state which one; the IOOS QARTOD and IODE flag schemes are established options.processingParameters ties the soundings to the processing that produced them: the identifier of the sound-velocity profile used (SVP-…), whether tidal correction was applied, the ray-tracing method (SNELL) and whether artifact removal was run. The SVP reference matters most: sound speed varies with temperature, salinity and pressure, a wrong profile bends every outer beam, and the reference lets a processor rerun the soundings when a better profile becomes available.
The sample message records the moment a physical sample is taken: sampleId, sampleType, collectionMethod (such as MANIPULATOR_ARM), location with habitat and substrate, an environment snapshot, a specimen description with media references, the container and its preservation state, the collector, and permits. Phase 1 captures the event; the custody record of Chapter 4 tracks the sample from then on.
Four details of the specification's example should not be copied as they stand.
environment. Here the environment is a set of bare numbers ("temperature": 2.47), unlike the measured-value objects of OCEANOGRAPHIC_DATA. Either use the same objects, or reference the oceanographic message that describes the conditions, by sourceId and sequenceNumber.0000-0002-1234-5678, fails that check: the digit computed from the first fifteen is 7, not 8.permits is a list of strings. For the legal checks in Chapter 2 (research consent within an EEZ, access-and-benefit-sharing terms), a structured object is more useful: issuing authority, permit number, validity dates, and the activities it covers."temperature": "AMBIENT" is a string where other temperatures are numbers. Keep it as a category (ambient, chilled, frozen), but call the field something like storageCondition so it is never read as a measurement.Telemetry is the highest-volume message type and the one operators watch live. Its payload has a static vehicle block (type, model, manufacturer, serial number, maxDepthRating), a position block with attitude and altitude, per-thruster status, a power block, a systems block (hydraulics, cameras, lights, manipulators, sonar, communications) and an alerts array.
Four practical points follow from the example.
vehicle block does not change during a dive. Over a fibre tether its repetition is harmless, but over an acoustic link it can cost more than the dynamic data. Send it in the first message of a dive, or in the archive's metadata.json, and omit it afterwards.hydraulics.pressure is 3000 with no unit (psi and kPa are both plausible readings), and thruster power reads like a percentage but is not labelled. Apply the same measured-value discipline as in 3.2.maxDepthRating: a vehicle approaching its rating should raise an alert, and a report beyond it points either to an emergency or to a sensor error.alerts is an empty array in the specification, with no element schema. A workable element is shown below; severity reuses the envelope's priority vocabulary so that an alert can raise the priority of the message that carries it.{
"alertId": "ALR-0007",
"severity": "HIGH",
"code": "GROUND_FAULT",
"subsystem": "power",
"message": "Insulation resistance below threshold on thruster bus",
"raisedAt": "2025-01-15T14:29:58.412Z",
"value": { "value": 0.8, "unit": "Mohm", "sensorId": "GFD-01", "calibrationDate": "2024-12-01" },
"acknowledged": false
}Validation in Phase 1 has three distinct outcomes, and implementations go wrong when they merge them. A structural failure, such as a missing required field, a malformed timestamp or a checksum mismatch, means the message cannot be trusted as a message. It should be rejected and the rejection logged. A range violation means the message is intact but a value is outside the expected envelope. The specification's instruction is to flag it with OUT_OF_RANGE, not to drop it. A plausibility concern, such as depth disagreeing with pressure or power disagreeing with voltage times current, is advisory and belongs in a quality report.
Every message must have a valid wiaVersion, an ISO 8601 UTC timestamp, a sequenceNumber that increases per source, a sourceId in the allowed character set, a messageType from the list, and a checksum that matches the payload. The monotonicity rule can only be checked with state: the validator needs the last sequence number seen from that source, which is why the code below takes it as a parameter.
| Parameter | Minimum | Maximum | Unit | Why this limit, and its caveat |
|---|---|---|---|---|
| Depth | 0 | 11,000 | m | Covers the deepest trenches with a small margin. |
| Temperature | −2 | 400 | °C | The lower end is near the freezing point of seawater (about −1.9 °C at 35 salinity); the upper end admits hydrothermal vent fluid. |
| Pressure | 0 | 1,100 | bar | A constant-density estimate at 11,000 m gives about 1,107 bar absolute, so this limit is slightly tighter than the depth limit. Near full ocean depth, flag with care; see below. |
| Salinity | 0 | 50 | PSU | Deep-sea brine pools can exceed 50. Those readings are real and must be flagged, not dropped. |
| pH | 0 | 14 | pH | Only catches gross errors; open-ocean values sit in a narrow band. |
| Dissolved oxygen | 0 | 20 | mg/L | Zero is legitimate in anoxic basins. |
| Latitude | −90 | 90 | degrees | WGS84. |
| Longitude | −180 | 180 | degrees | WGS84; ±180 are the same meridian. |
The limits are generous by design: they catch unit errors (a temperature in kelvin, a depth in centimetres) rather than unusual oceanography. Two consequences follow. First, a value inside the range is not therefore good. A 25 °C reading at 3,500 m passes the range check and is almost certainly wrong unless the sensor is in a vent plume, which is the job of the plausibility layer. Second, the pressure and depth limits do not agree exactly at full ocean depth. A vehicle legitimately near 11,000 m can report a pressure above 1,100 bar. Implementations should either derive the pressure limit from the depth limit or document a slightly higher local limit. Chapter 8 lists this as a proposed erratum.
No ranges are given for turbidity, sound velocity, heading, pitch, roll or altitude. A local profile may add them (for example, heading from 0 to below 360); document such additions as deviations under §6.2.
§3.2 says out-of-range values "MUST be flagged with data quality indicator 'OUT_OF_RANGE'" but does not say where the indicator lives. Putting it in metadata.dataQuality would label the entire message, and with it the six other environment values that were fine. The better place is the value itself: "qualityFlag": "OUT_OF_RANGE" inside the measured-value object, plus a summary list in the validator's report. For plain numbers, such as location.depth, the flag goes in the report keyed by the value's JSON path. The flag is added by the receiver and must not change the checksummed payload. A validator therefore reports flags alongside the message, and the storage layer records them next to the stored payload rather than rewriting it.
Timestamps must be within ±1 s of UTC, synchronised by NTP or GPS. A vehicle without a surface link must document its drift in metadata. The difficulty is that a free-running clock drifts steadily. A crystal oscillator with an error of 20 parts per million, used here as an example, gains or loses about 1.7 s per day, so a long autonomous mission can leave the ±1 s window unless it is corrected. The usual method is to compare the vehicle clock with GPS time just before launch and just after recovery, and to spread the difference linearly over the dive. That only works if both comparisons are recorded, which is what the timeSync block does:
"timeSync": {
"source": "GPS",
"preDive": { "at": "2025-01-14T21:10:03.000Z", "offsetSeconds": -0.012 },
"postDive": { "at": "2025-01-16T02:47:40.000Z", "offsetSeconds": -1.861 },
"correction": "LINEAR",
"note": "offset = vehicle clock minus GPS-derived UTC; timestamps in this archive are corrected"
}One more trap: GPS system time is not UTC. It does not apply leap seconds and is currently 18 seconds ahead of UTC. Receivers normally output UTC, but a logger that reads raw GPS time is off by far more than the ±1 s allowed.
§3.1 requires a "valid checksum matching payload", and §2.1 shows the format SHA256:<hex>. What it does not say is which bytes are hashed. JSON has many serialisations of the same object: different key orders, different whitespace, 2.0 or 2, é or \u00e9. A producer that hashes its pretty-printed output and a receiver that re-serialises the parsed object will almost never agree. This is the most common reason two correct implementations fail to interoperate.
This book's recommendation is to hash the UTF-8 bytes of the payload member only, serialised with the JSON Canonicalization Scheme (JCS, RFC 8785). JCS sorts keys and fixes number and string formatting to the ECMAScript rules, so any conforming library produces the same bytes. The digest is written in lowercase hex after the prefix SHA256:. Receivers should accept the prefix case-insensitively, because the Phase 4 examples write sha256:. Two cautions apply. The general-purpose JSON writers in other languages are not JCS; Python's json.dumps(sort_keys=True), for instance, writes 2.0 where JCS writes 2, so use a JCS library. And the digest in the specification's own oceanographic example, e3b0c442…b855, is the SHA-256 of empty input. A validator should reject it outright; accepting it means the checksum was never computed.
The JSON Schema below (draft 2020-12) captures the structural rules for the envelope and a reusable definition of the measured-value object. The messageType list reflects this book's reading of the open list in 3.1. The schema cannot express monotonicity, checksum correctness or range flags, so a code layer does those.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://example.org/wia/ene-040/phase1/envelope.json",
"type": "object",
"required": ["wiaVersion", "messageType", "timestamp", "sequenceNumber",
"sourceId", "priority", "payload", "checksum"],
"properties": {
"wiaVersion": { "type": "string", "pattern": "^1\\.[0-9]+$" },
"messageType": { "enum": ["OCEANOGRAPHIC_DATA", "BATHYMETRIC_DATA",
"SAMPLE_METADATA", "VEHICLE_TELEMETRY",
"DATA_PACKET", "COMMAND", "RESPONSE"] },
"timestamp": { "type": "string",
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\\.[0-9]{3}Z$" },
"sequenceNumber": { "type": "integer", "minimum": 0, "maximum": 4294967295 },
"sourceId": { "type": "string", "pattern": "^[A-Za-z0-9_-]{1,64}$" },
"priority": { "enum": ["LOW", "NORMAL", "HIGH", "CRITICAL"] },
"payload": { "type": "object" },
"metadata": { "type": "object" },
"checksum": { "type": "string", "pattern": "^(SHA256|sha256):[0-9a-f]{64}$" }
},
"$defs": {
"measuredValue": {
"type": "object",
"required": ["value", "unit", "sensorId", "calibrationDate"],
"properties": {
"value": { "type": "number" },
"unit": { "type": "string", "minLength": 1 },
"sensorId": { "type": "string", "pattern": "^[A-Za-z0-9_-]+$" },
"calibrationDate": { "type": "string", "format": "date" },
"qualityFlag": { "type": "string" }
}
}
}
}The TypeScript validator applies the schema-level rules itself so it can run without a schema library, checks the checksum with the canonicalisation above, and returns the three outcomes separately: errors make the message unacceptable, and flags accompany an accepted message. It runs on Node.js with only the built-in crypto module.
import { createHash } from "node:crypto";
type Flag = { path: string; code: "OUT_OF_RANGE"; value: number; min: number; max: number };
type Issue = { path: string; code: string; detail: string };
export type Result = { accepted: boolean; errors: Issue[]; flags: Flag[] };
// Phase 1 §3.2 — limits keyed by the envelope path where each quantity lives
const RANGES: Record<string, [number, number]> = {
"payload.location.depth": [0, 11000],
"payload.location.latitude": [-90, 90],
"payload.location.longitude": [-180, 180],
"payload.environment.temperature.value": [-2, 400],
"payload.environment.pressure.value": [0, 1100],
"payload.environment.salinity.value": [0, 50],
"payload.environment.pH.value": [0, 14],
"payload.environment.dissolvedOxygen.value": [0, 20],
};
const EXPECTED_UNIT: Record<string, string> = {
"payload.environment.temperature": "celsius",
"payload.environment.pressure": "bar",
"payload.environment.dissolvedOxygen": "mg/L",
};
const REQUIRED = ["wiaVersion", "messageType", "timestamp", "sequenceNumber",
"sourceId", "priority", "payload", "checksum"];
const PRIORITIES = new Set(["LOW", "NORMAL", "HIGH", "CRITICAL"]);
const TYPES = new Set(["OCEANOGRAPHIC_DATA", "BATHYMETRIC_DATA", "SAMPLE_METADATA",
"VEHICLE_TELEMETRY", "DATA_PACKET", "COMMAND", "RESPONSE"]); // working list, see 3.1
const TS_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
const EMPTY_SHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
// RFC 8785 (JCS)-compatible for ordinary JSON data: sorted keys, ES number/string form
export function canonical(v: unknown): string {
if (v === null || typeof v !== "object") return JSON.stringify(v);
if (Array.isArray(v)) return "[" + v.map(canonical).join(",") + "]";
const o = v as Record<string, unknown>;
return "{" + Object.keys(o).sort()
.filter((k) => o[k] !== undefined)
.map((k) => JSON.stringify(k) + ":" + canonical(o[k])).join(",") + "}";
}
export function payloadChecksum(payload: unknown): string {
return "SHA256:" + createHash("sha256").update(canonical(payload), "utf8").digest("hex");
}
function get(obj: any, path: string): any {
return path.split(".").reduce((o, k) => (o == null ? undefined : o[k]), obj);
}
export function validate(msg: any, lastSeq?: number): Result {
const errors: Issue[] = [];
const flags: Flag[] = [];
for (const f of REQUIRED)
if (msg?.[f] === undefined) errors.push({ path: f, code: "MISSING_FIELD", detail: "required" });
if (errors.length) return { accepted: false, errors, flags };
if (!/^\d+\.\d+$/.test(msg.wiaVersion) || msg.wiaVersion.split(".")[0] !== "1")
errors.push({ path: "wiaVersion", code: "UNSUPPORTED_VERSION", detail: msg.wiaVersion });
if (!TS_RE.test(msg.timestamp) || Number.isNaN(Date.parse(msg.timestamp)))
errors.push({ path: "timestamp", code: "BAD_TIMESTAMP", detail: msg.timestamp });
if (!/^[A-Za-z0-9_-]{1,64}$/.test(msg.sourceId))
errors.push({ path: "sourceId", code: "BAD_SOURCE_ID", detail: msg.sourceId });
if (!TYPES.has(msg.messageType))
errors.push({ path: "messageType", code: "UNKNOWN_TYPE", detail: String(msg.messageType) });
if (!PRIORITIES.has(msg.priority))
errors.push({ path: "priority", code: "BAD_PRIORITY", detail: String(msg.priority) });
if (!Number.isInteger(msg.sequenceNumber) || msg.sequenceNumber < 0)
errors.push({ path: "sequenceNumber", code: "BAD_SEQUENCE", detail: String(msg.sequenceNumber) });
else if (lastSeq !== undefined && msg.sequenceNumber <= lastSeq)
errors.push({ path: "sequenceNumber", code: "NOT_MONOTONIC", detail: `${msg.sequenceNumber} <= ${lastSeq}` });
const sum = String(msg.checksum);
if (!/^sha256:[0-9a-f]{64}$/i.test(sum) || sum.toLowerCase().endsWith(EMPTY_SHA256))
errors.push({ path: "checksum", code: "BAD_CHECKSUM_FORMAT", detail: sum });
else if (sum.slice(7).toLowerCase() !== payloadChecksum(msg.payload).slice(7))
errors.push({ path: "checksum", code: "CHECKSUM_MISMATCH", detail: "payload altered or non-canonical" });
for (const [prefix, unit] of Object.entries(EXPECTED_UNIT)) {
const m = get(msg, prefix);
if (m && m.unit !== unit)
errors.push({ path: prefix + ".unit", code: "UNEXPECTED_UNIT", detail: `${m.unit} (expected ${unit})` });
}
// Range rules flag, they never reject: a 380 °C vent reading is data, not an error
for (const [path, [min, max]] of Object.entries(RANGES)) {
const v = get(msg, path);
if (typeof v === "number" && (v < min || v > max))
flags.push({ path, code: "OUT_OF_RANGE", value: v, min, max });
}
return { accepted: errors.length === 0, errors, flags };
}Given a lander message reporting a depth of 11,020.4 m with a correct checksum, the validator returns accepted: true with one flag, {path: "payload.location.depth", code: "OUT_OF_RANGE", value: 11020.4, min: 0, max: 11000}. The same message with the empty-input digest is rejected with BAD_CHECKSUM_FORMAT. This split is the point of the design. The depth may be a sensor fault or a very deep trench, and deciding which is a scientist's job. The broken checksum means the data cannot be trusted at all.
JSON is readable and self-describing but expensive. The oceanographic example above is well over a kilobyte, and acoustic links (Chapter 6) carry at most a few kilobits per second, often much less. §4 therefore defines an optional binary form: a fixed 32-byte header followed by a payload encoded with Protocol Buffers version 3.
byte 0 4 6 7 8 16 20 28 32
+--------+------+----+----+----------------+--------+----------------+--------+
| magic | ver |type|prio| timestamp | seq | sourceId hash | length |
| 4 B | 2 B | 1B | 1B | 8 B (ns) | 4 B | 8 B | 4 B |
+--------+------+----+----+----------------+--------+----------------+--------+
"WIAE" maj.min since Unix epoch SHA-256 prefix payload bytesThe table in §4.1 gives offsets and sizes but not byte order, signedness, or how the version, type and source hash are computed. Without those, two conforming encoders can produce incompatible bytes. This book fixes them as follows.
0x57494145 appears on the wire as the ASCII bytes W I A E, which is plainly the intent. A little-endian writer would produce E A I W.0x01 0x00. This is also why the JSON version should be MAJOR.MINOR.LOW, NORMAL, HIGH, CRITICAL.sourceId, read as a big-endian unsigned integer. The specification says only "hash". A hash cannot be reversed, so the receiver needs a registry that maps hashes back to identifiers and that refuses to register a second identifier with the same hash.| Code | messageType | Status |
|---|---|---|
0x00 | (invalid) | Reserved so that a zeroed buffer is never mistaken for a message. |
0x01 | OCEANOGRAPHIC_DATA | Payload defined in §2.2 |
0x02 | BATHYMETRIC_DATA | Payload defined in §2.3 |
0x03 | SAMPLE_METADATA | Payload defined in §2.4 |
0x04 | VEHICLE_TELEMETRY | Payload defined in §2.5 |
0x05 | DATA_PACKET | Named in §2.1, payload undefined |
0x10, 0x11 | COMMAND, RESPONSE | Carried by Phase 3 (Chapter 6) |
0xF0–0xFF | private use | Local experiments; never exchanged between organisations. |
The header has no integrity field. On a live acoustic link that is acceptable, because each Phase 3 frame carries a CRC-32. For binary messages that are stored and forwarded, such as files written on the vehicle and uploaded later, carry the payload's SHA-256 in the payload itself or in the archive manifest (3.7).
Python's struct format >IHBBqIQI expresses the whole layout in one line: > for big-endian, then a 4-byte magic, 2-byte version, two single bytes, a signed 8-byte timestamp, a 4-byte sequence number, an 8-byte hash and a 4-byte length, 32 bytes in all.
import struct, hashlib
from datetime import datetime, timezone
HEADER = struct.Struct('>IHBBqIQI') # big-endian, 32 bytes
MAGIC = 0x57494145 # b"WIAE"
PRIORITY = {"LOW": 0, "NORMAL": 1, "HIGH": 2, "CRITICAL": 3}
def source_hash(source_id: str) -> int:
"""First 8 bytes of SHA-256(UTF-8 sourceId), as unsigned big-endian."""
return int.from_bytes(hashlib.sha256(source_id.encode("utf-8")).digest()[:8], "big")
def iso_to_ns(ts: str) -> int:
dt = datetime.strptime(ts, "%Y-%m-%dT%H:%M:%S.%fZ").replace(tzinfo=timezone.utc)
return (int(dt.timestamp()) * 1_000_000_000) + dt.microsecond * 1_000
def encode_header(version, msg_type, priority, ts_iso, seq, source_id, payload_len):
major, minor = (int(x) for x in version.split(".")[:2])
return HEADER.pack(MAGIC, (major << 8) | minor, msg_type, PRIORITY[priority],
iso_to_ns(ts_iso), seq, source_hash(source_id), payload_len)
def decode_header(buf: bytes) -> dict:
if len(buf) < HEADER.size:
raise ValueError("short header")
magic, ver, mtype, prio, ts_ns, seq, src, plen = HEADER.unpack_from(buf)
if magic != MAGIC:
raise ValueError(f"bad magic 0x{magic:08X}")
if prio > 3:
raise ValueError("priority out of range")
return {"version": f"{ver >> 8}.{ver & 0xFF}", "messageType": mtype,
"priority": prio, "timestampNs": ts_ns, "sequenceNumber": seq,
"sourceHash": f"{src:016x}", "payloadLength": plen}
if __name__ == "__main__":
h = encode_header("1.0", 0x01, "NORMAL", "2025-01-15T14:30:00.000Z",
12345, "ROV-EXAMPLE-01", 118)
assert len(h) == 32
print(h.hex(" "))
print(decode_header(h))Run as a script, it encodes a header for the source ROV-EXAMPLE-01 and prints the bytes below, grouped here by field:
57 49 41 45 magic "WIAE"
01 00 version 1.0
01 type 0x01 OCEANOGRAPHIC_DATA (working code)
01 prio 1 = NORMAL
18 1a e4 18 75 1e 10 00 time 1736951400000000000 ns = 2025-01-15T14:30:00Z
00 00 30 39 seq 12345
7f fa 1d 74 27 53 2e 15 source SHA-256("ROV-EXAMPLE-01")[0:8]
00 00 00 76 length 118 payload bytes followThe magic reads as text, 01 00 is version 1.0, the timestamp decodes to 2025-01-15T14:30:00Z and 00 00 30 39 is sequence 12345. A decoder should check the magic first, and must check that the stated payload length does not exceed the bytes received; that check keeps a corrupted length field from becoming a buffer over-read.
§4.2 says payloads use Protocol Buffers v3 and points to schema definitions in /schemas/protobuf/. Those files were not included in the specification package examined for this book, so an implementer today has to write the .proto files and document them. The sketch below shows the approach for the oceanographic payload. It keeps JSON property names as map keys and field names, so a mechanical mapping between the two forms is possible. It is illustrative, not the normative schema.
syntax = "proto3";
package wia.ene040.v1; // illustrative sketch, not the normative schema
message MeasuredValue {
double value = 1;
string unit = 2; // same vocabulary as the JSON form
string sensor_id = 3;
string calibration_date = 4; // YYYY-MM-DD
string quality_flag = 5; // e.g. "OUT_OF_RANGE"; empty when none
}
message Location {
double latitude = 1;
double longitude = 2;
double depth = 3; // metres, positive down
double altitude = 4; // metres above seabed
}
message OceanographicData {
Location location = 1;
map<string, MeasuredValue> environment = 2; // key = JSON property name
}Protobuf saves space by dropping field names and encoding integers compactly, but doubles still take 8 bytes and the unit and calibration strings repeat in every message. A practical compromise is to send units, sensor IDs and calibration dates once per dive and refer to them by index afterwards. Phase 3's own compact records, and their limits, are examined in Chapter 6.
When a dive ends, its messages become an archive. §5.1 recommends one directory per mission:
mission-<id>/
├── metadata.json
├── telemetry/
│ ├── vehicle-telemetry-001.jsonl
│ └── ...
├── oceanographic/
│ ├── ctd-data-001.jsonl
│ └── ...
├── bathymetry/
│ ├── multibeam-raw/
│ ├── multibeam-processed/
│ └── bathymetry-grid.tiff
├── samples/
│ ├── biological/
│ ├── geological/
│ └── chemical/
├── imagery/
│ ├── photos/
│ ├── video/
│ └── imagery-index.json
└── logs/
├── mission-log.txt
└── event-log.jsonltelemetry/ and oceanographic/ hold time series as JSON Lines. bathymetry/ keeps vendor raw files, processed soundings and a GeoTIFF grid that GIS tools open directly. imagery/ holds media with an imagery-index.json linking each file to a time and position, and logs/ keeps the human mission log beside a machine-readable event log. Raw sonar files stay in their vendor formats: the archive packages them; it does not replace them.
§5.2 recommends JSON Lines: one complete JSON object per line, no enclosing array. A file can be appended to during the dive, a crash loses at most the last line, and tools can stream it without loading gigabytes into memory. The specification's example lines are compact rows, not full envelopes:
{"timestamp":"2025-01-15T14:30:00.000Z","depth":3547.2,"temp":2.47}
{"timestamp":"2025-01-15T14:30:01.000Z","depth":3547.3,"temp":2.46}
{"timestamp":"2025-01-15T14:30:02.000Z","depth":3547.4,"temp":2.45}These rows have lost what the envelope provided: units, sensor identity, calibration dates and a checksum. Either store full envelopes, one per line, which is self-describing but roughly an order of magnitude larger, or keep compact rows and describe their columns once in metadata.json. The second suits high-rate data, provided the description is actually written. In both cases write UTF-8 without a byte-order mark, one line feed per record, in timestamp order.
The specification names metadata.json but does not define its content. This book recommends using it as the archive's manifest. It should record the mission, expedition and dive identifiers that link to the Phase 2 mission and the Chapter 4 records, the standard version, the sources in the archive, the time-sync record, the column definitions of each compact series, and a SHA-256 digest of every file. The digests close the integrity gap noted in 3.6 and let a repository verify an upload without trusting the transport. This is the idea behind the BagIt packaging format (RFC 8493), which libraries and archives use for the same purpose. The example below is illustrative; the digest is shortened here.
{
"wiaVersion": "1.0",
"missionId": "MISSION-2025-003",
"expeditionId": "EXP-2025-011",
"diveId": "DIVE-ROV-2025-003",
"sources": [
{ "sourceId": "ROV-EXAMPLE-01", "type": "ROV", "maxDepthRating": 6000 }
],
"timeSync": { "source": "GPS", "correction": "LINEAR", "maxResidualSeconds": 0.3 },
"series": {
"oceanographic/ctd-data-001.jsonl": {
"columns": {
"timestamp": { "type": "ISO8601", "timezone": "UTC" },
"depth": { "unit": "m", "positive": "down", "sensorId": "CTD-07-P" },
"temp": { "unit": "celsius", "sensorId": "CTD-07-T", "calibrationDate": "2024-11-20" }
}
}
},
"files": {
"oceanographic/ctd-data-001.jsonl": "sha256:3f9c…",
"bathymetry/bathymetry-grid.tiff": "sha256:b41e…"
}
}Phase 1 gets the important things right. It has a common envelope with identity and ordering, per-value units and calibration provenance, range rules that flag rather than discard, a compact binary path, and an archive layout built on open formats. It also leaves a number of decisions open, and interoperability depends on making them the same way. The table collects the ones raised in this chapter, with the recommendation used throughout this book. Chapter 8 turns them into a proposed errata list.
| Gap in the specification | Recommendation in this book |
|---|---|
| Checksum input not defined; example digest is of empty input | SHA-256 of JCS (RFC 8785) bytes of payload; reject the empty-input digest |
| Binary byte order, version packing and source hash unspecified | Big-endian; major/minor bytes; first 8 bytes of SHA-256(sourceId) |
No binary message-type codes; no closed messageType list | Working registry (3.6), recorded as a deviation until published |
Location of the OUT_OF_RANGE flag unspecified | Per-value qualityFlag plus a validator report; never rewrite the payload |
| Pressure limit slightly below the hydrostatic value at the depth limit; pressure reference unstated | Derive the limit from depth or raise it locally; declare absolute vs sea pressure |
| Units missing (backscatter intensity, hydraulic pressure, sample environment) | Measured-value objects everywhere |
| Accuracy confidence level and current direction convention unstated | State confidence (95% as in IHO S-44); current direction as flow-to |
| Protobuf schemas referenced but not supplied | Write and publish .proto files; map names one to one |
metadata.json content undefined | Manifest with column units, time sync and file digests |
The next chapter moves from messages to records: the expedition, dive, observation, sample, vent and survey objects that give these messages their scientific context.