Module runner
Expand description
The runner above the port: one edge, driven by its caller (#244).
§What is here
An Edge is one replication relationship as the local side sees it: a
local store that reads its write groups (WriteGroupStore) and ingests
(IngestStore), exchanging with one SyncPeer. Edge::step does one
exchange — pull from the far side and ingest locally, then read the local
log above the confirmed watermark and push it — and updates the edge’s
SyncStatus, which Edge::status hands back without a round trip.
Edge::run steps until the edge is synced, quiet or halted, waiting out
each transient failure through a Backoff and a Sleep the caller
supplies. Quiet means a step moved nothing new and the far side’s count of
what is waiting did not fall, so a far side that never says nothing is
waiting still lets a run end.
§What each failure class does
A failed step records its phase and class in SyncStatus::last_failure,
and the class decides what the edge does next (SY-36). The runner reads the
class; it never infers one.
Transient: nothing changes, and the same step can succeed later.Edge::runwaits out a seeded, jittered, capped delay and retries, and returns the error once the backoff’s attempts are spent, with the edge still running. It never escalates a transient failure to a stop by counting: a Durable Object that was reset comes back either as itself, and the edge carries on, or under a new identity, and the incarnation check stops it.Spliton a push of several write groups: the edge halves the events it offers per push, for the rest of its life, and the next step offers fewer groups. A split refusing one group parks the edge, because a group is never cut (SY-30); so does a split outside the push, where the runner holds no batch it could halve.Park: the edge parks (EdgeState::Parked).Stop: the edge stops (EdgeState::Stopped).Reseed: the edge stops. From a pull or a push it stops withStopCause::RetentionGap, which the next section describes; from any other phase, withStopCause::Failure.
A parked or stopped edge does no more I/O: every later step returns
StepError::Parked or StepError::Stopped at once.
§Retention gaps
A far side that has forgotten history this edge’s token has not reached
refuses the pull with class Reseed
(SY-32). The edge stops with StopCause::RetentionGap, carrying the far
side’s retention_floor as it read
at the refusal, and SyncStatus::retention_gap reads it back. Nothing
from the refused pull lands, and Edge::run returns without waiting:
a retry fails the same way for ever, and pulling from the start would
converge on a strict subset in silence. The runner never compares a token
with the floor itself; RetentionGap’s module says why only the peer
can.
§The incarnation check
An answer naming a store other than the one the edge synced with stops the
edge before the answer is acted on, and leaves the local log alone (SY-34,
ADR-0104). IncarnationChanged says which store it knew and which
answered; the module it lives in says why the check is shaped as it is.
§Groups: a push carries the writes the local log was written in
A push reads the local log through
read_groups and sends one
EventGroup per local write group, each whole, with an
ingested group’s guard back on it (SY-30, SY-39). So a multi-event append is
never cut across two pushes, independent appends are never merged into one
atomic unit, and
a foreign group this edge relays keeps its boundary and its guard. After an
acknowledged push, every local event at or below
SyncStatus::confirmed has been pushed.
A group the peer’s limits cannot carry is offered alone, never split, and the peer’s refusal states its class (SY-36).
§A bounded outbound buffer
A step holds at most one push’s worth of local events in memory, and the
push is capped by the edge’s own buffer as well as by
the peer’s declared limits: a peer that declares
usize::MAX events per batch does not make the edge read the whole
unconfirmed tail. The local log is the queue; nothing is copied out of it
ahead of the push that carries it.
§Caller-driven, and nothing spawned
step and run borrow the peer and the store for as long as they run and
keep neither. The edge owns only what is about the link: the far side’s
resume token, its incarnation, the status, and the merge rule it was given
for what it pulls. Nothing here spawns a task or
holds a timer, so the same edge is driven by a test, by a Durable Object’s
alarm, or by a task the caller spawns on its own runtime; dropping
run’s future between steps loses nothing a later step does not redo.
The bounds are the bare flavours (SyncPeer, WriteGroupStore,
IngestStore, Sleep), the weaker requirement both flavours meet, so
the Durable Object’s !Send store and peer drive an edge unchanged (SY-17).
For a peer and store of the Send flavours the step’s future is Send too,
given a Send + Sync resume token; tests/flavours.rs proves both, the
second in generic code under tokio::spawn.
§Adjudication: a rule for what the edge pulls
An edge lands what it pulls unjudged unless it is given a merge rule:
Edge::with_adjudicator takes any Adjudicate, and each pulled group
then lands in one write with the compensation the rule answers for it
(SY-2, SY-7). Accept, the default, judges nothing. The rule judges only
the pull; what the edge pushes is the far side’s to judge, which a
StoreSyncPeer does through its own
with_adjudicator. So the two
directions of one edge take two rules, with no role parameter (SY-9,
SY-10). A rule that cannot decide fails the step in
Phase::Adjudicate: nothing in the batch lands, the token stays, and
the edge reacts to the rule’s class like any other failure (SY-36).
§Where the watermark lives
SY-31: the per-peer confirmation watermark is written through
ProjectionStore under the reserved id ProjectionId::sync_watermark
names, never as an event. The edge holds the watermark, the token and the
far side’s incarnation in memory and does no projection I/O of its own.
Edge::watermark hands the first and the last back as a Watermark,
whose commit lands it in the caller’s batch beside
whatever read model the caller wrote there, and Edge::restored builds an
edge back from one Watermark::load read. The token stays the caller’s
to keep or drop (Edge::resume_token, Edge::resuming): an edge that
lost it pulls from the start, and ingest skips what it holds.
Structs§
- Accept
- The rule that judges nothing: every group lands unjudged.
- Backoff
- A seeded, jittered, capped exponential backoff, with a limit on attempts.
- Edge
- One replication edge: the local side’s view of one peer relationship.
- Failure
- Which part of a step failed, and the class the failure stated.
- Incarnation
Changed - The far side answered as a store other than the one this edge has synced with.
- Retention
Gap - The far side refused this edge’s resume token because it has forgotten history the token has not reached (SY-32).
- Stepped
- What
Edge::stepmoved. - Sync
Status - One edge’s status, as of its last step.
- Watermark
- One edge’s confirmation watermark: the far side, and the first position of the last local write group it acknowledged.
Enums§
- Backoff
Error - Why
Backoff::newrefused its arguments. - Edge
State - Whether an edge still exchanges.
- Load
Watermark Error - Why
Watermark::loadcould not read a watermark. - Pending
- How many local events sit above the confirmed watermark.
- Phase
- The parts of a step.
- Step
Error - Why
Edge::stepfailed: the phase, and the failure it met. - Stop
Cause - Why an edge stopped.
Traits§
- Adjudicate
- A merge rule: what one direction of an edge writes in answer to a group it ingests (SY-7).
- Send
Adjudicate - A merge rule: what one direction of an edge writes in answer to a group it ingests (SY-7).
- Send
Sleep - Waits out a duration. The runner’s only contact with time.
- Sleep
- Waits out a duration. The runner’s only contact with time.
Type Aliases§
- Step
Failure - The error
Edge::stepreturns for a peerP, a storeSand the edge’s merge ruleA.