solprop.driver.executors.queue#

Drain durable point work across controllers sharing a filesystem.

A Queue directory holds an immutable manifest (the scan, its retention settings and its builder’s identity), one state row per point and the published artifacts. Each controller claims points under a lock with a refreshed lease (_Claim, _Heartbeat; the lock and row helpers are solprop.driver.executors._queue_state), runs them in its own Pool, and publishes each result as a checked artifact only while its claim token still holds. collect() then validates the artifacts and writes them, idempotently, into the collected scan file through one locked writer. Leases, takeover and the publication order are in docs/dev/scans.md (“Durable executor implementation”).

Functions

collect(path, output)

Idempotently collect validated artifacts through one locked HDF5 writer.

Classes

Queue(path[, stale_age, pool, chunk])

Drain a shared durable queue using a local persistent process pool.

class solprop.driver.executors.queue.Queue(path, stale_age=600.0, pool=<factory>, chunk=None)[source]#

Bases: object

Drain a shared durable queue using a local persistent process pool.

Attributes:
pathpathlib.Path, str or None

Queue directory. None derives it from the collected output path for the CLI.

stale_agefloat

Positive remote lease duration, s. Live local owners remain protected.

poolPool

Resource policy for workers within this controller.

chunktuple of int or None

Optional zero-based round-robin chunk number and count.

chunk: tuple[int, int] | None = None#
path: Path | str | None#
pool: Pool#
prepare(request)[source]#

Create or validate the manifest under an automatically released lock.

Parameters:
requestScanRequest

Scan identity, retention settings and worker to bind before dispatch.

run(points, work)[source]#

Run claimed points while a separate thread refreshes their leases.

Parameters:
pointsIterable[T]

Lazily supplied point descriptions retaining global indices.

workCallable[[T], R]

Pickleable point worker with its memory probe.

Yields:
object

Completed result from the prepared execution policy.

stale_age: float = 600.0#
validate(work)[source]#

Validate serialisation before preparing the durable queue.

Parameters:
workCallable

Pickleable point worker with its memory probe.

solprop.driver.executors.queue.collect(path, output)[source]#

Idempotently collect validated artifacts through one locked HDF5 writer.

Parameters:
pathpathlib.Path or str

Queue directory containing its manifest and published point artifacts.

outputpathlib.Path or str

Collected HDF5 destination.

Returns:
ScanOutput

Reader masking unfinished points.

Raises:
OutputError

If a scan, point, configuration or artifact digest conflicts.