Skip to main content

Module runner

Module runner 

Expand description

The runner above the port: one edge, driven by its caller (#244).

§What is here

An Edge is one replication relationship as the local side sees it: a local store that reads its write groups (WriteGroupStore) and ingests (IngestStore), exchanging with one SyncPeer. Edge::step does one exchange — pull from the far side and ingest locally, then read the local log above the confirmed watermark and push it — and updates the edge’s SyncStatus, which Edge::status hands back without a round trip. Edge::run steps until the edge is synced, quiet or halted, waiting out each transient failure through a Backoff and a Sleep the caller supplies. Quiet means a step moved nothing new and the far side’s count of what is waiting did not fall, so a far side that never says nothing is waiting still lets a run end.

§What each failure class does

A failed step records its phase and class in SyncStatus::last_failure, and the class decides what the edge does next (SY-36). The runner reads the class; it never infers one.

  • Transient: nothing changes, and the same step can succeed later. Edge::run waits out a seeded, jittered, capped delay and retries, and returns the error once the backoff’s attempts are spent, with the edge still running. It never escalates a transient failure to a stop by counting: a Durable Object that was reset comes back either as itself, and the edge carries on, or under a new identity, and the incarnation check stops it.
  • Split on a push of several write groups: the edge halves the events it offers per push, for the rest of its life, and the next step offers fewer groups. A split refusing one group parks the edge, because a group is never cut (SY-30); so does a split outside the push, where the runner holds no batch it could halve.
  • Park: the edge parks (EdgeState::Parked).
  • Stop: the edge stops (EdgeState::Stopped).
  • Reseed: the edge stops. From a pull or a push it stops with StopCause::RetentionGap, which the next section describes; from any other phase, with StopCause::Failure.

A parked or stopped edge does no more I/O: every later step returns StepError::Parked or StepError::Stopped at once.

§Retention gaps

A far side that has forgotten history this edge’s token has not reached refuses the pull with class Reseed (SY-32). The edge stops with StopCause::RetentionGap, carrying the far side’s retention_floor as it read at the refusal, and SyncStatus::retention_gap reads it back. Nothing from the refused pull lands, and Edge::run returns without waiting: a retry fails the same way for ever, and pulling from the start would converge on a strict subset in silence. The runner never compares a token with the floor itself; RetentionGap’s module says why only the peer can.

§The incarnation check

An answer naming a store other than the one the edge synced with stops the edge before the answer is acted on, and leaves the local log alone (SY-34, ADR-0104). IncarnationChanged says which store it knew and which answered; the module it lives in says why the check is shaped as it is.

§Groups: a push carries the writes the local log was written in

A push reads the local log through read_groups and sends one EventGroup per local write group, each whole, with an ingested group’s guard back on it (SY-30, SY-39). So a multi-event append is never cut across two pushes, independent appends are never merged into one atomic unit, and a foreign group this edge relays keeps its boundary and its guard. After an acknowledged push, every local event at or below SyncStatus::confirmed has been pushed.

A group the peer’s limits cannot carry is offered alone, never split, and the peer’s refusal states its class (SY-36).

§A bounded outbound buffer

A step holds at most one push’s worth of local events in memory, and the push is capped by the edge’s own buffer as well as by the peer’s declared limits: a peer that declares usize::MAX events per batch does not make the edge read the whole unconfirmed tail. The local log is the queue; nothing is copied out of it ahead of the push that carries it.

§Caller-driven, and nothing spawned

step and run borrow the peer and the store for as long as they run and keep neither. The edge owns only what is about the link: the far side’s resume token, its incarnation, the status, and the merge rule it was given for what it pulls. Nothing here spawns a task or holds a timer, so the same edge is driven by a test, by a Durable Object’s alarm, or by a task the caller spawns on its own runtime; dropping run’s future between steps loses nothing a later step does not redo.

The bounds are the bare flavours (SyncPeer, WriteGroupStore, IngestStore, Sleep), the weaker requirement both flavours meet, so the Durable Object’s !Send store and peer drive an edge unchanged (SY-17). For a peer and store of the Send flavours the step’s future is Send too, given a Send + Sync resume token; tests/flavours.rs proves both, the second in generic code under tokio::spawn.

§Adjudication: a rule for what the edge pulls

An edge lands what it pulls unjudged unless it is given a merge rule: Edge::with_adjudicator takes any Adjudicate, and each pulled group then lands in one write with the compensation the rule answers for it (SY-2, SY-7). Accept, the default, judges nothing. The rule judges only the pull; what the edge pushes is the far side’s to judge, which a StoreSyncPeer does through its own with_adjudicator. So the two directions of one edge take two rules, with no role parameter (SY-9, SY-10). A rule that cannot decide fails the step in Phase::Adjudicate: nothing in the batch lands, the token stays, and the edge reacts to the rule’s class like any other failure (SY-36).

§Where the watermark lives

SY-31: the per-peer confirmation watermark is written through ProjectionStore under the reserved id ProjectionId::sync_watermark names, never as an event. The edge holds the watermark, the token and the far side’s incarnation in memory and does no projection I/O of its own. Edge::watermark hands the first and the last back as a Watermark, whose commit lands it in the caller’s batch beside whatever read model the caller wrote there, and Edge::restored builds an edge back from one Watermark::load read. The token stays the caller’s to keep or drop (Edge::resume_token, Edge::resuming): an edge that lost it pulls from the start, and ingest skips what it holds.

Structs§

Accept
The rule that judges nothing: every group lands unjudged.
Backoff
A seeded, jittered, capped exponential backoff, with a limit on attempts.
Edge
One replication edge: the local side’s view of one peer relationship.
Failure
Which part of a step failed, and the class the failure stated.
IncarnationChanged
The far side answered as a store other than the one this edge has synced with.
RetentionGap
The far side refused this edge’s resume token because it has forgotten history the token has not reached (SY-32).
Stepped
What Edge::step moved.
SyncStatus
One edge’s status, as of its last step.
Watermark
One edge’s confirmation watermark: the far side, and the first position of the last local write group it acknowledged.

Enums§

BackoffError
Why Backoff::new refused its arguments.
EdgeState
Whether an edge still exchanges.
LoadWatermarkError
Why Watermark::load could not read a watermark.
Pending
How many local events sit above the confirmed watermark.
Phase
The parts of a step.
StepError
Why Edge::step failed: the phase, and the failure it met.
StopCause
Why an edge stopped.

Traits§

Adjudicate
A merge rule: what one direction of an edge writes in answer to a group it ingests (SY-7).
SendAdjudicate
A merge rule: what one direction of an edge writes in answer to a group it ingests (SY-7).
SendSleep
Waits out a duration. The runner’s only contact with time.
Sleep
Waits out a duration. The runner’s only contact with time.

Type Aliases§

StepFailure
The error Edge::step returns for a peer P, a store S and the edge’s merge rule A.