solprop.output.hdf5#
The streaming HDF5 output handler, with cache, resume and read-only reopening.
HDF5Outputstreams a run to one file, kept open between optional snapshot transactions; it preallocates the planes the run reserved, writes the saved-plane countn_savedlast so that it never covers a partly written plane, follows the run’s field retention, and withappend=Truecontinues a file it wrote earlier.check_cache()is the resume gate: the file’s configuration hash must equal the requested one (ADR-066);read_resume_state()returns aResumeState: the record, the terminal checkpoint or else the last plane, back in the internal convention, the whole step history, the outcome and any Adams march snapshot.open_file()opens a finished run read-only through the same protocol, which is howsolprop.processingreads it;read_recipe()andwrite_recipe()read and write the recipe group (SEP 41).COMPRESSIONSis what h5py ships without a filter plugin.
The on-disk layout, preallocation, the saved-plane count and field retention are in
docs/guide/results/output.md (“What is on disk”); the cache and resume rules in
docs/guide/results/resuming.md.
Module Attributes
Compression filters h5py ships with; no plugin, so no Blosc ( |
Functions
|
Say whether a file may be continued as the run |
|
Open a finished run for reading, through the protocol it was written with. |
|
Return the recipe an open file records, or |
|
Read a file's resumable state and cumulative work, preferring its checkpoint. |
|
Write the recipe group of an open file. |
Classes
|
Stream a run to one HDF5 file, opened once and extended plane by plane. |
|
Everything a driver needs to carry on from a file it wrote earlier. |
- solprop.output.hdf5.COMPRESSIONS = (None, 'gzip', 'lzf')#
Compression filters h5py ships with; no plugin, so no Blosc (
dependencies.md).
- class solprop.output.hdf5.HDF5Output(path, *, compression=None, compression_opts=None, append=False, readonly=False)[source]#
Bases:
PlaneStoreStream a run to one HDF5 file, opened once and extended plane by plane.
Reading through
__getitem__,fields()andplane_field()works throughout. While the handler is open the reads go through its handle; once the run has closed it, each read opens the file for reading, reads, and closes it again, so a finished run returned byprop_*(..., output="run.h5")orSimulation.runreads like one in memory and holds no handle between reads.open_file()keeps one handle open for many reads.- Parameters:
- pathpathlib.Path or str
Where to write. An existing file is replaced, with a warning, when the run opens.
- compression{None, ‘gzip’, ‘lzf’}, optional
Dataset compression.
'lzf'is fast and modest;'gzip'is slower and smaller. Nothing else: a Blosc filter would need a plugin that solprop does not depend on.- compression_optsint or None, optional
Level for
'gzip', 0 to 9; not accepted for the other choices.- appendbool, optional
Continue the run the file already holds instead of replacing the file: the resume path, after
read_resume_state()has handed the driver the last plane. The record given toopenis refused when its configuration hash differs from the file’s; the file’s own record is kept, creation time included, and becomes the handler’smetadata, and the run’s layout is read from the file so that the next plane is checked against the ones already there.- readonlybool, optional
Open the file at once, for reading only: the handler comes back open, with the file’s own record as its
metadata, and refusesopenandsave. This is whatopen_file()builds.
- Raises:
- OutputError
If the compression request is not one h5py can serve, if both
appendandreadonlyare asked for, or if a read-only file cannot be read.
Examples
>>> import numpy as np, tempfile >>> from pathlib import Path >>> from solprop.grid import EnvelopeGrid >>> from solprop.output import HDF5Output >>> from solprop.output.metadata import RunMetadata >>> from solprop.output.records import GridRecord, SavedPlane, StepperRecord >>> from solprop.stepper import DOPRI5, PIController >>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12) >>> metadata = RunMetadata( ... grid=GridRecord.from_grid(grid), ... modes=(), ... simulation_kind="mode_averaged", ... configuration={}, ... stepper=StepperRecord.from_settings( ... DOPRI5, PIController(rtol=1e-8, atol=0.0) ... ), ... ) >>> with tempfile.TemporaryDirectory() as folder: ... handler = HDF5Output(Path(folder) / "run.h5", compression="lzf") ... handler.open(metadata) ... start = np.ones((1, 1, grid.n_omega), complex) ... handler.save(SavedPlane(z=0.0, field=start)) ... shape = handler["field"].shape ... handler.close() >>> shape (1, 1, 1, 2560)
- close()[source]#
Close the owned handle, including a failed post-run annotation handle.
Repeated calls are harmless. A failed close retains the handle and writer ownership for a later retry, even if propagation already closed the store.
- Raises:
- OutputError
If the handle cannot be closed safely.
- open(metadata, *, force=False)[source]#
Begin a run, recording what produced it; refused on a read-only handler.
- Parameters:
- metadataRunMetadata
The record of the run about to start.
- forcebool, optional
Whether to overwrite metadata already held for a different run.
- Raises:
- OutputError
If the handler was built with
readonly=True.
- reopen()[source]#
Rebind a suspended writer without beginning a new execution.
- Raises:
- OutputError
If the file changed while suspended or cannot be reopened safely.
- save(plane)[source]#
Store one saved plane and flush it to disk; refused on a read-only handler.
The flush keeps the file’s
n_savedcount consistent through a hard stop – a SIGKILL, an out-of-memory kill, a lost node: the plane’s data is flushed before the count is raised, so a plane cut off half-way is not counted, and readers do not see it. The flush does not sync to disk and the file is not opened in SWMR mode, so HDF5 does not promise that a file stopped mid-write can be opened at all;snapshots=on a run publishes synced, checksummed copies that a hard stop cannot damage. An ordinary interruption (KeyboardInterrupt, an error) never needed the flush: the run loop closes the handler in afinallyand closing writes everything. One flush per plane costs nothing against the steps between planes.- Parameters:
- planeSavedPlane
The position, field, statistics and step history to store.
- Raises:
- OutputError
If the handler was built with
readonly=True.
- save_march(snapshot)[source]#
Keep the Adams march’s state at the position just saved, for a resume.
One snapshot is kept, rewritten in place at every call, beside the planes and outside the record: it says where the march stood, not what was computed, and a file without one resumes by starting the history afresh.
- Parameters:
- snapshotsolprop.output.records.MarchSnapshot
The march’s state at the last saved plane or terminal checkpoint.
- Raises:
- OutputError
If the handler was built with
readonly=Trueor is not open.
- class solprop.output.hdf5.ResumeState(metadata, plane, n_saved, history, outcome=None, checkpoint=None, march=None)[source]#
Bases:
DisplayedEverything a driver needs to carry on from a file it wrote earlier.
Equality is by identity (
eq=False) because the plane holds arrays.- Attributes:
- metadatasolprop.output.metadata.RunMetadata
The record of the interrupted run, including its step schedule.
- planesolprop.output.records.SavedPlane
The accepted terminal checkpoint when present, otherwise the last regular plane, with its field back in the internal FFT convention. Modal amplitudes remain in output units until the driver applies the weight at z.
- n_savedint
Number of regular planes stored. The next scheduled index is determined from the actual position, independently of this count.
- historysolprop.output.records.StepHistory
Every attempt of the whole run so far, concatenated.
- outcomesolprop.output.records.RunOutcome or None
Completion status and cumulative call ledgers;
Nonewhen the file records none.- checkpointsolprop.output.records.TerminalCheckpoint or None
Separate accepted terminal state, preferred over the last regular save.
- marchsolprop.output.records.MarchSnapshot or None
The Adams march’s state where the run stopped, when the file kept one there.
Examples
>>> from solprop.output.hdf5 import ResumeState >>> ResumeState.__doc__.splitlines()[0] 'Everything a driver needs to carry on from a file it wrote earlier.'
- checkpoint: TerminalCheckpoint | None = None#
- history: StepHistory#
- property interpolated: bool[source]#
Whether the plane the file resumes from was interpolated inside a step.
A landed plane is the end of an accepted step of the run; an interpolated one lies inside a step, and a resume from it needs
resume="interpolated". The entrance is the initial field.Whether the plane is not a state the march passed through.
- march: MarchSnapshot | None = None#
- metadata: RunMetadata#
- outcome: RunOutcome | None = None#
- plane: SavedPlane#
- solprop.output.hdf5.check_cache(path, metadata)[source]#
Say whether a file may be continued as the run
metadatadescribes.The resume gate (ADR-066): the two records’ configuration hashes must be equal. A refusal names the fields that differ. The checks that depend on the file’s state, such as whether any work is left, are the driver’s.
- Parameters:
- pathpathlib.Path or str or solprop.output.store.Output
The file to check; an output handler is refused by name.
- metadataRunMetadata
The record of the run about to start.
- Returns:
- bool
Truewhen the file may be continued;Falsewhen there is no file to resume from.
- Raises:
- OutputError
If the hashes differ, or the file was written at another output schema; or if path is an output handler.
Examples
>>> from solprop.output.hdf5 import check_cache >>> from solprop.output.metadata import _example_metadata >>> check_cache("no-such-file.h5", _example_metadata()) False
- solprop.output.hdf5.open_file(path)[source]#
Open a finished run for reading, through the protocol it was written with.
The handler comes back open and read-only:
__getitem__,fields,stat_names,n_savedandmetadataanswer from the file,openandsaveare refused, andclosereleases the file – awithblock does that on the way out. It is howsolprop.processingreads a file after the run that wrote it has ended;read_resume_state()reads the last plane only, and a handler built withappend=Trueopens the file read-write to continue the run.- Parameters:
- pathpathlib.Path or str
The file to read.
- Returns:
- HDF5Output
An open, read-only handler.
- Raises:
- OutputError
If the file cannot be opened, or carries no metadata.
Examples
>>> import numpy as np, tempfile >>> from pathlib import Path >>> from solprop.grid import EnvelopeGrid >>> from solprop.output import MemoryOutput, open_file >>> from solprop.output.metadata import RunMetadata >>> from solprop.output.records import GridRecord, SavedPlane, StepperRecord >>> from solprop.stepper import DOPRI5, PIController >>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12) >>> memory = MemoryOutput() >>> memory.open(RunMetadata( ... grid=GridRecord.from_grid(grid), modes=(), simulation_kind="mode_averaged", ... configuration={}, ... stepper=StepperRecord.from_settings( ... DOPRI5, PIController(rtol=1e-8, atol=0.0) ... ), ... )) >>> memory.save(SavedPlane(z=0.0, field=np.ones((1, 1, grid.n_omega), complex))) >>> with tempfile.TemporaryDirectory() as folder: ... memory.to_file(Path(folder) / "run.h5") ... with open_file(Path(folder) / "run.h5") as stored: ... shape, planes = stored["field"].shape, stored.n_saved >>> shape, planes, stored.closed ((1, 1, 1, 2560), 1, True)
- solprop.output.hdf5.read_recipe(file)[source]#
Return the recipe an open file records, or
Nonefor a run from a script.The
recipegroup of an optical file or a scan file (SEP 41, S2).- Parameters:
- fileh5py.File
The open file.
- Returns:
- RecipeRecord or None
The record.
- solprop.output.hdf5.read_resume_state(path)[source]#
Read a file’s resumable state and cumulative work, preferring its checkpoint.
The last plane comes back in the internal FFT convention, through
solprop.fourier.spectrum_to_internal(), so a driver can hand it straight to a backend without conjugating anything itself. A separate terminal checkpoint, or a running checkpoint written between saved planes beside arunningoutcome at its position (SEP 47 section 5.5), contributes no regular observation but supplies its actual position, pending attempt history and cumulative call ledgers. Reading preserves its landing classification; the driver requires explicitresume="interpolated"consent before restarting from an interpolated field.A run that keeps no field on its last regular plane (SEP 22) comes back with that plane’s
field=Noneunless a terminal checkpoint, which always carries the field, follows it; the driver refuses to continue from it by name.- Parameters:
- pathpathlib.Path or str or solprop.output.store.Output
The file to read; an output handler is refused by name.
- Returns:
- ResumeState
The metadata, the last plane, how many planes the file holds, and every attempt the stepper made.
- Raises:
- OutputError
If the file cannot be read, carries no metadata, or holds no plane; or if path is an output handler.
Examples
>>> import numpy as np, tempfile >>> from pathlib import Path >>> from solprop.grid import EnvelopeGrid >>> from solprop.output import HDF5Output >>> from solprop.output.hdf5 import read_resume_state >>> from solprop.output.metadata import _example_metadata >>> from solprop.output.records import SavedPlane >>> metadata = _example_metadata() >>> n_omega = metadata.grid.n_omega >>> with tempfile.TemporaryDirectory() as folder: ... handler = HDF5Output(Path(folder) / "run.h5") ... handler.open(metadata) ... handler.save(SavedPlane(z=0.25, field=np.ones((1, 1, n_omega), complex))) ... handler.close() ... state = read_resume_state(Path(folder) / "run.h5") >>> state.n_saved, state.plane.z (1, 0.25)
Carry on in the same file with a handler built with
append=True:>>> with tempfile.TemporaryDirectory() as folder: ... handler = HDF5Output(Path(folder) / "run.h5") ... handler.open(metadata) ... handler.save(SavedPlane(z=0.0, field=np.ones((1, 1, n_omega), complex))) ... handler.close() ... again = HDF5Output(Path(folder) / "run.h5", append=True) ... again.open(metadata) ... again.save(SavedPlane(z=0.5, field=np.ones((1, 1, n_omega), complex))) ... again.close() ... read_resume_state(Path(folder) / "run.h5").n_saved 2