Chapter 8: Long-Term Data Preservation

Extended Discussion — Further Topics in OAIS, BagIt, PREMIS, METS, 30+ year recovery

8.1 Why Cryobiology Data Has a Long Tail

A cryopreserved specimen is meant to outlast the experiment that produced it. The WIA-CRYO-010 experiment record reflects this. Its technical.storageDuration field is an ISO-8601 duration, LONG_TERM_STORAGE is one of the researchType values, and the lifecycle.status enumeration ends in ARCHIVED. The tank may be opened years after the lab notebook is closed. The investigator may have moved, the LIMS may have been replaced, and the original analysis software may no longer run.

Version 2.0 of the standard lists "long-term data archiving and preservation" in its scope. This chapter treats that line as an engineering problem: what must travel with a record so that a stranger in thirty years can still interpret a viability figure, repeat the cooling profile, and trust that nothing changed in storage.

8.2 OAIS Vocabulary Applied to a CRYO-010 Record

The OAIS reference model (ISO 14721) describes an archive as a set of functional entities: Ingest, Archival Storage, Data Management, Administration, Preservation Planning and Access. Information moves as packages: the Submission Information Package (SIP), the Archival Information Package (AIP) and the Dissemination Information Package (DIP). The table maps OAIS concepts onto elements the standard already defines.

OAIS conceptWhere it lives in WIA-CRYO-010
Content InformationThe JSON-LD experiment record, its measurements and temperature_profile
Representation InformationThe @context URL, enumerations, declared unit, version
ProvenanceResearcher ORCID, institution, protocol_deviations, audit log entries
ReferenceexperimentId, relatedPublications (DOIs), rawDataLinks
ContextirbApproval, fundingSource, keywords
Access rightsdata_classification, consent version and withdrawal status
FixityNot defined by the standard; supplied by the packaging layer (8.6)

The last row shows the main gap. The data format describes meaning well but says nothing about bit-level integrity, so the archive must add it.

8.3 Making the Record Self-Describing

The Designated Community of a cryopreservation archive is future cryobiologists who share the vocabulary but not the lab's habits. The standard helps in several ways:

An archive should also store a frozen copy of the JSON-LD context and the JSON Schema alongside the data. A URL such as the @context address is a pointer, and pointers rot. Representation Information that exists only at a remote address is not preserved.

8.4 Version Drift Across v1.0 to v2.0

A thirty-year archive will hold records from every revision. The revisions changed shape in ways that affect interpretation:

Aspectv1.0v1.1v1.2 / v2.0
ResearcherPlain stringObject with name, ORCID (recommended), institutionORCID, institution and email required
CryoprotectantSingle type and concentrationcomponents arrayComponents with grade, manufacturer, preparation date (v2.0)
Temperature pointsMinimum 3Minimum 5, time seriesMinimum 3 in the schema; v2.0 adds sample temperature
File formatsJSON, CSVAdds XMLAdds HDF5

Because the point-count rule was not constant across versions, a validator must apply the rules of the version recorded in the record's own version field, never the newest rules. The v1.1-to-v1.2 migration tool is named wia-cryo-migrate-v1.1-to-v1.2. Migration produces a new representation, so the original should be kept as well. Identifiers should not be rewritten either: the standard shows CRYO-YYYY-NNN, UUID-style identifiers and exp-2025-001-style examples in different places, and an archive keeps whichever the record carries.

8.5 From Export to Submission Information Package

The protocol layer already supplies most of the ingest path. A bulk export request names experiment IDs, a format such as json-ld and a compression such as gzip, and it returns a job ID for asynchronous processing. CSV export is available per experiment. The DELETE /experiments/{id} endpoint is a soft delete that archives the record instead of destroying it, which suits the ARCHIVED lifecycle state.

POST /bulk-export
{
  "experimentIds": ["exp-2025-001", "exp-2025-002"],
  "format": "json-ld",
  "compression": "gzip"
}

A reasonable ingest rule is to build a package when an experiment reaches COMPLETED or when the experiment.completed webhook event fires. Ingest then validates the record. Schema failures come back from the API with status 422, and those records are repaired before packaging. Bad records should not enter the archive for later cleanup.

8.6 BagIt Packaging

BagIt (RFC 8493) wraps a payload in a directory with checksum manifests. Its simplicity suits archives: a bag can be verified with nothing more than a hash tool. A bag for one experiment might look like this:

CRYO-2025-007/
  bagit.txt
  bag-info.txt
  manifest-sha256.txt
  tagmanifest-sha256.txt
  data/
    experiment.jsonld
    temperature_profile.h5
    viability.csv
    raw/sample001.fcs
    schema/cryo-research-v1.context.json
Source-Organization: Example University Cryobiology Lab
Bagging-Date: 2025-03-01
External-Identifier: CRYO-2025-007
Payload-Oxum: (computed at bagging time)

The tag manifest protects the bag's own metadata files. The raw/sample001.fcs path echoes the flow-cytometer integration, where the FCS file and its metadata are exported together. The standard's rawDataLinks array should then point to bag-relative paths rather than only to lab servers.

8.7 PREMIS Events and the Audit Log

PREMIS 3.0 models preservation metadata with Object, Event, Agent and Rights entities. The audit log defined for CRYO-010 systems already records who did what to which resource. Its fields are action, resourceType, resourceId, userId, timestamp and changes.

Audit log fieldPREMIS counterpart
actionEvent type (for example, an update or a modification)
resourceId, resourceTypeLinked Object identifier and type
userIdLinked Agent
timestampEvent date and time
changesEvent detail

Archive-side actions need their own events beyond what the lab system logs: fixity check, format migration, replication and ingest. Each should name the software agent that performed it and the outcome. When a migration such as v1.1 to v1.2 is run, the PREMIS event links the source object to the derived one, which tells a later reader that the two represent the same experiment.

8.8 METS as the Container

METS (Metadata Encoding and Transmission Standard) binds descriptive metadata, administrative metadata, a file inventory and a structural map in one XML document. It is useful when an experiment is a set of related files rather than one record. Version 1.1 of the standard added XML for regulatory compatibility, so the format is already familiar to this community. A skeleton for one experiment:

<mets:mets xmlns:mets="http://www.loc.gov/METS/">
  <mets:dmdSec ID="dmd1">  <!-- experiment descriptive record -->  </mets:dmdSec>
  <mets:amdSec ID="amd1">  <!-- PREMIS events, rights, calibration -->  </mets:amdSec>
  <mets:fileSec>
    <mets:fileGrp USE="raw">
      <mets:file ID="f1" MIMETYPE="application/octet-stream"/>
    </mets:fileGrp>
  </mets:fileSec>
  <mets:structMap>
    <mets:div TYPE="experiment"/>
  </mets:structMap>
</mets:mets>

This skeleton illustrates packaging and is not part of the WIA schema. The practical division of labor is that the JSON-LD record stays the authoritative scientific description, while METS records which files belong to the experiment and how they relate, such as time series, raw cytometry output and statistical CSV.

8.9 Large and Raw Payloads

The standard accepts JSON as primary, XML for regulatory submission, CSV for statistical analysis and HDF5 for large time series. From an archival view these differ in risk. JSON, XML and CSV are plain text, and CSV exports must have a header row and UTF-8 encoding, so they are safe to keep for decades. HDF5 is self-describing but needs a specific library to read, so archives should record the library version in the PREMIS environment description.

Raw instrument output such as FCS files is the evidence behind total_cells and viable_cells. It should be kept even when derived values exist, because a later analysis may apply a different gating or a corrected method. Deprecation notices in v1.2 already point the same way: legacy date formats and non-SI temperature units are retired in favor of ISO 8601 and Celsius or Kelvin, which removes two common causes of ambiguity.

8.10 Privacy and Consent Over Decades

Clinical records carry duties that persist after storage. The standard forbids direct identifiers in research databases, uses study-specific patient IDs, limits geography to region level and reports age in ranges. Consent version, consent date, special permissions and withdrawal status travel with the record, and data_classification takes public, restricted or confidential.

8.11 The 3-2-1 Rule in Practice

Phase 4 states the rule: three copies, two media types, one off-site. Its sample script dumps the PostgreSQL database, compresses the dump and copies it to object storage. That covers the database, which holds each experiment as a JSONB document keyed by experiment_id. It does not cover verification, and it does not preserve the bags themselves. A hardened variant records a checksum before the copy and tests it after:

#!/bin/bash
DATE=$(date +%Y%m%d)
pg_dump cryo_research | gzip > /backup/cryo_${DATE}.sql.gz
sha256sum /backup/cryo_${DATE}.sql.gz > /backup/cryo_${DATE}.sha256
aws s3 cp /backup/cryo_${DATE}.sql.gz s3://wia-backups/
aws s3 cp /backup/cryo_${DATE}.sha256 s3://wia-backups/

Keeping the three copies on different media types protects against a single class of failure. The off-site copy should also sit under a different administrative domain, so that one credential compromise cannot touch every copy.

8.12 Fixity and Semantic Verification

Checksums prove that bits are unchanged. They cannot show that the data still means the same thing. The standard supplies a second layer of checks that can be re-run at any time:

Recomputation makes a good regression test for an archive. Phase 4's own unit test expects 87 viable cells out of 100 to give exactly 87.0 with a two-element confidence interval. If a migrated record's stored viability disagrees with the value recomputed from its raw counts, the discrepancy is itself a preservation finding to log as a PREMIS event.

8.13 A 30-Year Recovery Rehearsal

A recovery guarantee is a rehearsed procedure and cannot rest on the promise that backups exist. A workable rehearsal for a CRYO-010 archive:

  1. Pick a sample of bags across all standard versions, including at least one v1.0 record.
  2. Restore them from the off-site copy only, with no access to the original lab systems.
  3. Verify the bag manifests and tag manifests.
  4. Open each record with only the frozen schema and context stored inside the bag, and run the validation rules of the record's own version.
  5. Recompute viability statistics from raw counts and compare them with stored values.
  6. Read the HDF5 and FCS payloads with the software recorded in the environment metadata, and record anything that no longer opens.
  7. Log every step as a PREMIS event and feed failures into preservation planning, for example by queuing a format migration.

The tiered conformance scheme attached to the protocol phase requires evidence retention of at least seven years at its highest tier. That figure is a floor for audit evidence and is far shorter than the horizon discussed here, so a research archive should set its own, longer retention policy.

8.14 Summary and Bridge to the Next Chapter

The standard describes a cryopreservation experiment precisely, and this chapter has shown how to keep that description usable. OAIS supplies the roles and package types. BagIt adds fixity that the data format lacks. PREMIS gives the audit log a durable, standard form, and METS ties related files together. Version drift, self-describing units, frozen schemas, consent that outlives the experiment and key custody are the cryobiology-specific risks, and the 3-2-1 rule with recomputation checks addresses them. A rehearsed restore is the only proof of a recovery guarantee.

The packages built here are closed, verified and described, but they are not yet connected to the wider body of research data. The next chapter builds on them: once a record can be trusted to survive, the question becomes how bags and their metadata are discovered, cited and combined with data from other laboratories.