solprop.output.hdf5#

The streaming HDF5 output handler, with cache, resume and read-only reopening.

  • HDF5Output streams a run to one file, kept open between optional snapshot transactions; it preallocates the planes the run reserved, writes the saved-plane count n_saved last so that it never covers a partly written plane, follows the run’s field retention, and with append=True continues a file it wrote earlier.

  • check_cache() is the resume gate: the file’s configuration hash must equal the requested one (ADR-066); read_resume_state() returns a ResumeState: the record, the terminal checkpoint or else the last plane, back in the internal convention, the whole step history, the outcome and any Adams march snapshot.

  • open_file() opens a finished run read-only through the same protocol, which is how solprop.processing reads it; read_recipe() and write_recipe() read and write the recipe group (SEP 41).

  • COMPRESSIONS is what h5py ships without a filter plugin.

The on-disk layout, preallocation, the saved-plane count and field retention are in docs/guide/results/output.md (“What is on disk”); the cache and resume rules in docs/guide/results/resuming.md.

Module Attributes

COMPRESSIONS

Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

Functions

check_cache(path, metadata)

Say whether a file may be continued as the run metadata describes.

open_file(path)

Open a finished run for reading, through the protocol it was written with.

read_recipe(file)

Return the recipe an open file records, or None for a run from a script.

read_resume_state(path)

Read a file's resumable state and cumulative work, preferring its checkpoint.

write_recipe(file, record)

Write the recipe group of an open file.

Classes

HDF5Output(path, *[, compression, ...])

Stream a run to one HDF5 file, opened once and extended plane by plane.

ResumeState(metadata, plane, n_saved, history)

Everything a driver needs to carry on from a file it wrote earlier.

solprop.output.hdf5.COMPRESSIONS = (None, 'gzip', 'lzf')#

Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

class solprop.output.hdf5.HDF5Output(path, *, compression=None, compression_opts=None, append=False, readonly=False)[source]#

Bases: PlaneStore

Stream a run to one HDF5 file, opened once and extended plane by plane.

Reading through __getitem__, fields() and plane_field() works throughout. While the handler is open the reads go through its handle; once the run has closed it, each read opens the file for reading, reads, and closes it again, so a finished run returned by prop_*(..., output="run.h5") or Simulation.run reads like one in memory and holds no handle between reads. open_file() keeps one handle open for many reads.

Parameters:
pathpathlib.Path or str

Where to write. An existing file is replaced, with a warning, when the run opens.

compression{None, ‘gzip’, ‘lzf’}, optional

Dataset compression. 'lzf' is fast and modest; 'gzip' is slower and smaller. Nothing else: a Blosc filter would need a plugin that solprop does not depend on.

compression_optsint or None, optional

Level for 'gzip', 0 to 9; not accepted for the other choices.

appendbool, optional

Continue the run the file already holds instead of replacing the file: the resume path, after read_resume_state() has handed the driver the last plane. The record given to open is refused when its configuration hash differs from the file’s; the file’s own record is kept, creation time included, and becomes the handler’s metadata, and the run’s layout is read from the file so that the next plane is checked against the ones already there.

readonlybool, optional

Open the file at once, for reading only: the handler comes back open, with the file’s own record as its metadata, and refuses open and save. This is what open_file() builds.

Raises:
OutputError

If the compression request is not one h5py can serve, if both append and readonly are asked for, or if a read-only file cannot be read.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.metadata import RunMetadata
>>> from solprop.output.records import GridRecord, SavedPlane, StepperRecord
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> metadata = RunMetadata(
...     grid=GridRecord.from_grid(grid),
...     modes=(),
...     simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... )
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5", compression="lzf")
...     handler.open(metadata)
...     start = np.ones((1, 1, grid.n_omega), complex)
...     handler.save(SavedPlane(z=0.0, field=start))
...     shape = handler["field"].shape
...     handler.close()
>>> shape
(1, 1, 1, 2560)
close()[source]#

Close the owned handle, including a failed post-run annotation handle.

Repeated calls are harmless. A failed close retains the handle and writer ownership for a later retry, even if propagation already closed the store.

Raises:
OutputError

If the handle cannot be closed safely.

open(metadata, *, force=False)[source]#

Begin a run, recording what produced it; refused on a read-only handler.

Parameters:
metadataRunMetadata

The record of the run about to start.

forcebool, optional

Whether to overwrite metadata already held for a different run.

Raises:
OutputError

If the handler was built with readonly=True.

property path: Path[source]#

Where the file is written.

reopen()[source]#

Rebind a suspended writer without beginning a new execution.

Raises:
OutputError

If the file changed while suspended or cannot be reopened safely.

save(plane)[source]#

Store one saved plane and flush it to disk; refused on a read-only handler.

The flush keeps the file’s n_saved count consistent through a hard stop – a SIGKILL, an out-of-memory kill, a lost node: the plane’s data is flushed before the count is raised, so a plane cut off half-way is not counted, and readers do not see it. The flush does not sync to disk and the file is not opened in SWMR mode, so HDF5 does not promise that a file stopped mid-write can be opened at all; snapshots= on a run publishes synced, checksummed copies that a hard stop cannot damage. An ordinary interruption (KeyboardInterrupt, an error) never needed the flush: the run loop closes the handler in a finally and closing writes everything. One flush per plane costs nothing against the steps between planes.

Parameters:
planeSavedPlane

The position, field, statistics and step history to store.

Raises:
OutputError

If the handler was built with readonly=True.

save_march(snapshot)[source]#

Keep the Adams march’s state at the position just saved, for a resume.

One snapshot is kept, rewritten in place at every call, beside the planes and outside the record: it says where the march stood, not what was computed, and a file without one resumes by starting the history afresh.

Parameters:
snapshotsolprop.output.records.MarchSnapshot

The march’s state at the last saved plane or terminal checkpoint.

Raises:
OutputError

If the handler was built with readonly=True or is not open.

suspend()[source]#

Close the HDF5 handle while retaining the live writer and its ownership.

Raises:
OutputError

If the writer is not open or its handle cannot be closed safely.

class solprop.output.hdf5.ResumeState(metadata, plane, n_saved, history, outcome=None, checkpoint=None, march=None)[source]#

Bases: Displayed

Everything a driver needs to carry on from a file it wrote earlier.

Equality is by identity (eq=False) because the plane holds arrays.

Attributes:
metadatasolprop.output.metadata.RunMetadata

The record of the interrupted run, including its step schedule.

planesolprop.output.records.SavedPlane

The accepted terminal checkpoint when present, otherwise the last regular plane, with its field back in the internal FFT convention. Modal amplitudes remain in output units until the driver applies the weight at z.

n_savedint

Number of regular planes stored. The next scheduled index is determined from the actual position, independently of this count.

historysolprop.output.records.StepHistory

Every attempt of the whole run so far, concatenated.

outcomesolprop.output.records.RunOutcome or None

Completion status and cumulative call ledgers; None when the file records none.

checkpointsolprop.output.records.TerminalCheckpoint or None

Separate accepted terminal state, preferred over the last regular save.

marchsolprop.output.records.MarchSnapshot or None

The Adams march’s state where the run stopped, when the file kept one there.

Examples

>>> from solprop.output.hdf5 import ResumeState
>>> ResumeState.__doc__.splitlines()[0]
'Everything a driver needs to carry on from a file it wrote earlier.'
checkpoint: TerminalCheckpoint | None = None#
history: StepHistory#
property interpolated: bool[source]#

Whether the plane the file resumes from was interpolated inside a step.

A landed plane is the end of an accepted step of the run; an interpolated one lies inside a step, and a resume from it needs resume="interpolated". The entrance is the initial field.

Whether the plane is not a state the march passed through.

march: MarchSnapshot | None = None#
metadata: RunMetadata#
n_saved: int#
outcome: RunOutcome | None = None#
plane: SavedPlane#
solprop.output.hdf5.check_cache(path, metadata)[source]#

Say whether a file may be continued as the run metadata describes.

The resume gate (ADR-066): the two records’ configuration hashes must be equal. A refusal names the fields that differ. The checks that depend on the file’s state, such as whether any work is left, are the driver’s.

Parameters:
pathpathlib.Path or str or solprop.output.store.Output

The file to check; an output handler is refused by name.

metadataRunMetadata

The record of the run about to start.

Returns:
bool

True when the file may be continued; False when there is no file to resume from.

Raises:
OutputError

If the hashes differ, or the file was written at another output schema; or if path is an output handler.

Examples

>>> from solprop.output.hdf5 import check_cache
>>> from solprop.output.metadata import _example_metadata
>>> check_cache("no-such-file.h5", _example_metadata())
False
solprop.output.hdf5.open_file(path)[source]#

Open a finished run for reading, through the protocol it was written with.

The handler comes back open and read-only: __getitem__, fields, stat_names, n_saved and metadata answer from the file, open and save are refused, and close releases the file – a with block does that on the way out. It is how solprop.processing reads a file after the run that wrote it has ended; read_resume_state() reads the last plane only, and a handler built with append=True opens the file read-write to continue the run.

Parameters:
pathpathlib.Path or str

The file to read.

Returns:
HDF5Output

An open, read-only handler.

Raises:
OutputError

If the file cannot be opened, or carries no metadata.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import MemoryOutput, open_file
>>> from solprop.output.metadata import RunMetadata
>>> from solprop.output.records import GridRecord, SavedPlane, StepperRecord
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> memory = MemoryOutput()
>>> memory.open(RunMetadata(
...     grid=GridRecord.from_grid(grid), modes=(), simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... ))
>>> memory.save(SavedPlane(z=0.0, field=np.ones((1, 1, grid.n_omega), complex)))
>>> with tempfile.TemporaryDirectory() as folder:
...     memory.to_file(Path(folder) / "run.h5")
...     with open_file(Path(folder) / "run.h5") as stored:
...         shape, planes = stored["field"].shape, stored.n_saved
>>> shape, planes, stored.closed
((1, 1, 1, 2560), 1, True)
solprop.output.hdf5.read_recipe(file)[source]#

Return the recipe an open file records, or None for a run from a script.

The recipe group of an optical file or a scan file (SEP 41, S2).

Parameters:
fileh5py.File

The open file.

Returns:
RecipeRecord or None

The record.

solprop.output.hdf5.read_resume_state(path)[source]#

Read a file’s resumable state and cumulative work, preferring its checkpoint.

The last plane comes back in the internal FFT convention, through solprop.fourier.spectrum_to_internal(), so a driver can hand it straight to a backend without conjugating anything itself. A separate terminal checkpoint, or a running checkpoint written between saved planes beside a running outcome at its position (SEP 47 section 5.5), contributes no regular observation but supplies its actual position, pending attempt history and cumulative call ledgers. Reading preserves its landing classification; the driver requires explicit resume="interpolated" consent before restarting from an interpolated field.

A run that keeps no field on its last regular plane (SEP 22) comes back with that plane’s field=None unless a terminal checkpoint, which always carries the field, follows it; the driver refuses to continue from it by name.

Parameters:
pathpathlib.Path or str or solprop.output.store.Output

The file to read; an output handler is refused by name.

Returns:
ResumeState

The metadata, the last plane, how many planes the file holds, and every attempt the stepper made.

Raises:
OutputError

If the file cannot be read, carries no metadata, or holds no plane; or if path is an output handler.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.hdf5 import read_resume_state
>>> from solprop.output.metadata import _example_metadata
>>> from solprop.output.records import SavedPlane
>>> metadata = _example_metadata()
>>> n_omega = metadata.grid.n_omega
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.25, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     state = read_resume_state(Path(folder) / "run.h5")
>>> state.n_saved, state.plane.z
(1, 0.25)

Carry on in the same file with a handler built with append=True:

>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.0, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     again = HDF5Output(Path(folder) / "run.h5", append=True)
...     again.open(metadata)
...     again.save(SavedPlane(z=0.5, field=np.ones((1, 1, n_omega), complex)))
...     again.close()
...     read_resume_state(Path(folder) / "run.h5").n_saved
2
solprop.output.hdf5.write_recipe(file, record)[source]#

Write the recipe group of an open file.

Parameters:
fileh5py.File

The file, open for writing, without a recipe group.

recordRecipeRecord

The recipe.