Skip to main content

Crate happenstance_sync_testkit

Crate happenstance_sync_testkit 

Source
Expand description

The conformance suite for happenstance-sync peers, and for the stores they ingest into.

§Status: the peer, ingest, write-group, restore, watermark and convergence families

Published since #108 part E, with happenstance-sync, and grown rule by rule through phase 13, which ran it against the network peers. It is the bar a peer or ingest adapter must clear before it is considered to exist, exactly as happenstance-testkit is for a store, and an adapter outside this workspace mounts it the same way. Pin it exactly: a new rule is a minor of this crate’s number.

Publishing settled the two questions this paragraph used to defer. Its version is the workspace’s, in lockstep until 1.0 — ADR-0066 §4 makes a new rule a minor of its number, and lockstep keeps that true while the workspace is 0.y. And happenstance-testkit’s RuleOutcome is part of this crate’s public API, through every rule’s signature.

§Mounting it

happenstance_sync_testkit::sync_peer_conformance!(
    happenstance_sync_testkit::MemorySyncFixture::new()
);

The expression builds a SyncFixture: one isolated backing peer per instance. It is evaluated inside the generated module, so a fixture type defined beside the invocation is named super::MyFixture; the macro does not glob-import its parent, because an unused glob would be a warning in every crate that named its fixture by full path.

§What is checked

RuleClauseWhat it rejects
ingest_never_rejectsSY-1a peer that evaluates the travelling guard and refuses or drops a conflicting fact
compensation_is_atomic_with_the_losing_eventSY-2a peer that splits a losing event from the compensation that rode with it
event_id_order_is_identical_on_every_peerSY-23a peer that dedupes by the highest origin position it has seen, and so drops an origin’s lower event that arrives late
peer_declares_its_own_limitsSY-18a peer whose declared limits admit an event its ingest then refuses, or that declares below VT-21’s floor
peer_takes_a_batch_at_its_declared_limitsSY-18a peer that refuses a push its own batch limits admit: more events, or more bytes, than it really takes
push_envelope_preserves_group_boundariesSY-30a peer that merges the groups of a page into one, or strips the guard a group was pushed with
every_refusal_states_its_classSY-36a peer whose errors all state one failure class, whatever the failure
nothing_waiting_means_the_next_pull_is_emptySY-37a peer that says nothing is waiting when its page ran out, not its log
ack_never_confirms_a_non_durable_eventSY-31a peer that confirms an event before it is durable
round_trip_preserves_the_whole_logSY-28, SY-27a peer that withholds from its counterpart the events a type or tag filter matches, among those the rule’s log carries
retention_gap_is_reported_not_silentSY-32a peer that compacts and answers a token below its floor, or the start, with a strict subset in silence, or refuses it as anything but Reseed; skipped unless the fixture declares COMPACTION

Each rule names a wrong implementation, and that implementation lives in this crate’s tests/wrong_peers.rs — SY-18’s in tests/wrong_limits.rs, over KvCappedFixture, its batch rule’s in tests/wrong_peers.rs, SY-23’s in tests/wrong_orders.rs, SY-28’s in tests/wrong_scopes.rs, SY-31’s across a reopen in tests/wrong_reopens.rs, and SY-32’s in tests/wrong_retention.rs — where the rule is shown to fail it. A rule no peer can fail is decorative. The mount also emits every_declined_capability_is_stated_by_this_fixture, which fails a fixture that leaves REOPEN at this crate’s default declension (sync_declensions_are_stated).

No peer or ingest rule decodes a payload (SY-35), and no rule in rules holds two peers (SY-8). rules says how the rules keep both. The model family below is the one exception to the second, and its module says why.

§The model family

Behind the off-by-default proptest feature, and absent on wasm32 because proptest does not build there. sync_model generates peer topologies of three to six peers — chains whose ends never meet included — and drives them from one seed through writes, partitions, lost acks, duplicated pushes and delayed pages, then checks every peer at quiescence against a model of its own. It opens one fixture per node, so the fixture expression must build a fresh, isolated peer each time it is evaluated.

RuleClauseWhat it rejects
transitive_convergence_over_a_partial_meshSY-24, SY-4a peer that quarantines what it accepted, or never forwards it
redelivery_of_an_accepted_group_is_a_no_opSY-11a peer that dedupes per batch (SF-03), or miscounts its acknowledgement
forwarding_order_is_arrival_orderSY-25a peer that re-sorts what it forwards

The names are written out rather than linked, because the module they live in does not exist without the feature. A failure names its seed in hex and the smallest scenario it shrank to; tests/wrong_topologies.rs pins every rule’s verdict against four wrong peers.

#[cfg(not(target_arch = "wasm32"))]
happenstance_sync_testkit::sync_model_conformance!(
    happenstance_sync_testkit::MemorySyncFixture::new()
);

§The ingest family

The other side of a peer: the store an IngestStore writes into. A store adapter that implements the port mounts it beside the event-store suite:

happenstance_sync_testkit::ingest_store_conformance!(
    happenstance_sync_testkit::MemoryIngestFixture::new()
);

The expression builds an IngestFixture, whose store is both an EventStore and an IngestStore. The reference is MemoryIngestFixture, over happenstance-sync’s MemoryIngestStore; happenstance-sqlite’s spike and the Durable Object mount the family too.

RuleClauseWhat it rejects
ingest_preserves_origin_identityVT-5a store that re-mints an ingested event
ingest_preserves_the_recorded_timeVT-9a store that restamps an ingested event with its arrival time, keeps whole seconds, holds a time before 1970 as the epoch, holds times non-decreasing with position, binds one time per group, or clamps a future time to its own clock
ingested_events_land_above_the_local_headSY-5an order-preserving merge, which hides an event below a checkpoint already read past
dedupe_reaches_identity_without_decodingSY-12, SY-11, SY-3content-keyed dedupe, and a compensation written again on redelivery
an_unheld_own_identity_stops_the_ingestSY-38a dropped claim, a check of the batch’s head only, a group-by-group commit, a claim hidden by the call’s own compensation, and a refused held echo
ingest_group_is_atomic_under_a_mid_group_faultSY-30a group written in chunks without a transaction, a compensation committed apart, a fault answered Ok, and an armed fault that never fires
ingest_validates_every_event_not_the_headSY-18per-event ceilings checked on each group’s head only, a compensation never checked, and a ceiling off by one
ingest_holds_the_call_to_its_tags_per_batch_ceilingVT-36, SY-18a tags-per-batch ceiling held to each group apart, a compensation left out of the total, a ceiling off by one, and a total refused as per-event, Park rather than Split
only_the_adjudicator_compensatesSY-7a compensation minted under an incarnation the store never read, so its log names an author nobody assigned
directional_merge_rules_composeSY-10a store with the merge rule built in, which answers a group whose travelling guard its log breaks with a reversal of its own, and so judges, as a spoke, what the hub sends; and one that re-mints its own claim when a peer’s claim on the seat arrives
bulk_ingest_is_idempotent_in_bounded_round_tripsSY-14, VT-24a store that consults what it holds for the first 64 events of a call only and writes a held event past them again, and one that refuses a call of VT-24’s 128 events as over its batch limit

Each wrong store lives in tests/wrong_ingest_stores.rs, where a test runs the whole family against it and asserts it fails exactly its rule.

SY-7’s rule opens two stores, so a fixture must give each store it opens a fresh incarnation, and the rule fails one that does not. It plays the runner above them and holds the adjudicator assignment itself, mounting happenstance-sync’s shipped runner::Adjudicate hook: a seat rule on the adjudicator and runner::Accept on the other (#450). What it holds the store to is the identity a compensation is filed under; that the shipped runner writes none under Accept is happenstance-sync‘s own tests’. Every peer adjudicating, the configuration SY-7 forbids, is run by the clause’s falsification test in this crate’s own unit tests, which records the two compensations it leaves for one conflict.

SY-10’s rule opens two stores for each of its two walks, hub and spoke and peer to peer, under the same requirement. It plays a runner that mounts one merge rule per direction of the edge through the same hook: the hub adjudicates what the spoke pushes, and the spoke takes what the hub sends unjudged. What it holds the store to is writing what each direction hands it and nothing of its own. The hub’s rule run on the spoke as well is the clause’s falsification test, in this crate’s unit tests, and the shipped runner::Edge is held to its half in happenstance-sync’s own tests, against wrong runners that each mount a rule where they should not. SY-33’s rule, which must exempt SY-38’s refusal (ADR-0098), is not written yet; the exemption is carried forward to it.

§Capabilities

Three rules need the fixture’s co-operation, and say so rather than pass when they do not get it. SY-30’s needs MID_GROUP_FAULT and arm_mid_group_fault, because only the adapter can fail its own write between two rows. SY-18’s reads MAX_EVENT_DATA_LEN, MAX_TAGS_PER_EVENT and MAX_METADATA_LEN, each mirrored from the adapter’s own constant. VT-36’s reads MAX_TAGS_PER_BATCH, mirrored the same way; only a store whose call has a tags wall of its own states one, today happenstance-cloudflare’s. MemoryIngestFixture declines the fault in its own words and declares no ceiling, so it reports all three rules skipped; the faithful store in tests/wrong_ingest_stores.rs runs them, and happenstance-sqlite’s ingest spike runs the first two. The mount also emits every_declined_capability_is_stated_by_this_fixture, which fails a fixture whose declension is this crate’s default (ingest_declensions_are_stated).

§The write-group family

A store that a runner or a StoreSyncPeer reads from must hand its log back in the writes that made it (SY-39), so a relay forwards each write as one group with its guard (SY-30). The family mounts on the same IngestFixture, whose store must also implement WriteGroupStore:

happenstance_sync_testkit::write_group_store_conformance!(
    happenstance_sync_testkit::MemoryIngestFixture::new()
);
RuleClauseWhat it rejects
an_append_reads_back_as_one_groupSY-39a store that records no groups, and one that merges adjacent appends
an_ingested_group_keeps_its_boundary_and_guardSY-39, SY-30a store that drops an ingested group’s guard, or merges what one call ingested
compensation_joins_its_groupSY-39, SY-2a store that records a compensation as a group of its own
a_group_after_a_redelivery_is_named_by_its_first_landed_rowSY-39, SY-11a store that names a group by the first position it allocated, not the first row it wrote
a_read_never_cuts_a_groupSY-39a page filled by event, a first group skipped for being too large, and a resume above a row position
pages_chain_on_first_and_end_with_nothingSY-39, SY-37a page its budget cut that says nothing is waiting

Each wrong store lives in tests/wrong_write_group_stores.rs, where a test runs the whole family against it and asserts the exact set it fails.

§The convergence family

Not an adapter’s bar but a projection’s: a fold that declares itself convergent, by implementing ConvergentProjection, is run under two interleavings of one set of events from two origins, and its read models must be byte-identical; and run over one arrival order numbered twice, densely and offset with gaps, and its read models must again be byte-identical. The rules are called directly over a ConvergenceFixture; there is no mounting macro, because there is no adapter to mount them beside and no capability to decline.

RuleClauseWhat it rejects
convergent_projection_is_interleaving_independentSY-20unit-ledger, and any declared fold whose answer depends on arrival order
convergent_projection_cannot_observe_local_positionSY-21a declared fold that writes its checkpoint, a local position, into its read model
non_convergent_projections_are_excluded_not_failedSY-22a check with no exclusion, which fails FIFO cost layers, and a check that does not really reorder

A payload is decoded here, and only by the projection under test: a fold reads its events, which is what makes it a fold. The check itself reads no event at all: it compares the bytes of two read models. What the runner reads to select and route events is their type and the projection’s scope tags, which is what SY-35 requires. The wrong projections live in tests/wrong_projections.rs. StockOnHandFixture and CostLayersFixture are the reference cases: one domain, a sum that converges and a FIFO queue that cannot.

§The restore family

VT-6’s restore half: a peer whose store is restored from a backup must not hand back an identity it already issued to a different event. Mounted on a RestoreFixture, which supplies what no peer port has, a writer of the store’s own events, a backup and a restore, and runs the adapter’s documented post-restore procedure (VT-6’s re-mint) as part of the restore:

happenstance_sync_testkit::restore_conformance!(
    happenstance_sync_testkit::MemoryRestoreFixture::new()
);
RuleClauseWhat it rejects
restored_peer_does_not_reissue_identitiesVT-6a store whose identity survives a restore, so a new event takes an identity a lost one was issued; a restore that restores nothing; and a store that stamps its current incarnation on what it reads back, so a re-mint rewrites restored history

A family of its own rather than a capability on SyncFixture: a peer that issues no identity of its own, as MemorySyncPeer does not, has nothing a restore could reissue, and every rule of the peer family is held against peers built on it. MemoryRestoreFixture, a store peer over the in-memory oracle that restores into a fresh incarnation, is the reference; the wrong peers live in tests/wrong_restores.rs, and happenstance-postgres mounts the family with its re-mint and runs the same sequence without it as its negative control.

§The watermark family

A projection store’s bar, not a peer’s: the store a runner keeps its per-peer confirmation watermark in, beside the read model whose rows depend on it (SY-31). The rule takes the ProjectionFixture a projection adapter already mounts the projection suite with, and is called directly, as the convergence family’s are:

#[tokio::test]
async fn watermark_advances_transactionally_with_a_read_model()
-> Result<(), happenstance_sync_testkit::Violation> {
    happenstance_sync_testkit::watermark_rules::watermark_advances_transactionally_with_a_read_model(
        async || happenstance_testkit::fixtures::MemoryProjectionFixture::new(),
    )
    .await?
    .report("watermark_advances_transactionally_with_a_read_model");
    Ok(())
}
RuleClauseWhat it rejects
watermark_advances_transactionally_with_a_read_modelSY-31a store that refuses the reserved sync/ prefix, keeps a watermark in its handle, keys every peer’s watermark alike, or upserts one around the port’s regression check

The rule plays the runner, with the shipped Edge and its Watermark, so what it holds the store to is what that runner asks of it. The wrong stores live in tests/wrong_watermark_stores.rs.

§Harnesses

The emitter is a parameter, as in the event-store family, so the suite never decides which runtime a peer is tested on: emit_sync_tokio! (the default), emit_sync_blocking! — no runtime and no Send bound, which is what a !Send peer runs under — and emit_sync_wasm!. The ingest family has emit_ingest_tokio! (the default), emit_ingest_blocking! and emit_ingest_wasm!, the Durable Object’s. The write-group family has emit_write_group_tokio! (the default), emit_write_group_blocking! and emit_write_group_wasm!. The restore family has emit_restore_tokio! (the default) and emit_restore_blocking!.

§Failure is a value

A rule returns Result<RuleOutcome, Violation> and an emitted test returns Result<(), Violation>, so a breach reaches the test runner as an error naming its clause. Violation says why this family does not panic as the other two do.

§Features

  • proptest, off by default: the model family above. Absent on wasm32 whatever the feature says, because proptest does not build there.

Modules§

convergence_rules
The convergence rules (SY-20, SY-21, SY-22).
ingest_rules
The ingest rules. See the crate documentation for what each covers.
restore_rules
The restore rules. See the crate documentation for what each covers.
rules
The sync rules. See the crate documentation for what each covers.
sync_modelproptest
The model family: seeded peer topologies, driven through push and pull from one seed, and checked at quiescence against a model of their own.
watermark_rules
The watermark rule. See the crate documentation for what it covers.
write_group_rules
The write-group rules. See the crate documentation for what each covers.

Macros§

emit_ingest_blocking
Emits one plain #[test] per ingest rule, driven by happenstance_testkit::block_on.
emit_ingest_tokio
Emits one #[tokio::test] per ingest rule.
emit_ingest_wasm
Emits one #[wasm_bindgen_test] per ingest rule.
emit_restore_blocking
Emits one plain #[test] per restore rule, driven by happenstance_testkit::block_on: no runtime and no Send bound.
emit_restore_tokio
Emits one #[tokio::test] per restore rule.
emit_sync_blocking
Emits one plain #[test] per sync rule, driven by happenstance_testkit::block_on.
emit_sync_model_blockingproptest
Emits one plain #[test] per model rule, driven by happenstance_testkit::block_on: no runtime and no Send bound.
emit_sync_model_tokioproptest
Emits one #[tokio::test] per model rule.
emit_sync_tokio
Emits one #[tokio::test] per sync rule.
emit_sync_wasm
Emits one #[wasm_bindgen_test] per sync rule.
emit_write_group_blocking
Emits one plain #[test] per write-group rule, driven by happenstance_testkit::block_on: no runtime and no Send bound.
emit_write_group_tokio
Emits one #[tokio::test] per write-group rule.
emit_write_group_wasm
Emits one #[wasm_bindgen_test] per write-group rule.
for_each_convergence_rule
Hands the complete convergence rule set to $callback.
for_each_ingest_store_rule
Hands the complete ingest rule set to $callback.
for_each_restore_rule
Hands the complete restore rule set to $callback.
for_each_sync_model_ruleproptest
Hands the complete model rule set to $callback.
for_each_sync_peer_rule
Hands the complete sync rule set to $callback.
for_each_watermark_rule
Hands the complete watermark rule set to $callback.
for_each_write_group_rule
Hands the complete write-group rule set to $callback.
ingest_store_conformance
Generates the ingest conformance suite for a store adapter.
restore_conformance
Generates the restore conformance suite for a peer adapter.
sync_model_conformanceproptest
Generates the model family for a peer adapter.
sync_peer_conformance
Generates the sync conformance suite for a peer adapter.
write_group_store_conformance
Generates the write-group conformance suite for a store adapter.

Structs§

CostLayers
The cost of every issue, consuming the oldest units received first.
CostLayersFixture
CostLayers over the same two sites: the honest non-convergent case SY-22 excludes.
KvCappedFixture
A SyncFixture whose peer stores each event as one KV value of at most KV_VALUE_CAP bytes: Turnstile’s peer D, the shape SY-18’s marker names.
KvCappedPeer
One handle onto a KvCappedFixture’s peer.
MemoryIngestFixture
The reference IngestFixture, over [MemoryIngestStore].
MemoryIngestHandle
One handle onto a MemoryIngestFixture’s store.
MemoryRestoreFixture
The reference RestoreFixture: a [StoreSyncPeer] over a [MemoryIngestStore].
MemorySyncFixture
The reference SyncFixture, over [MemorySyncPeer].
MemorySyncHandle
One handle onto a MemorySyncFixture’s peer.
Refusal
One refusal a fixture drove its peer into, and the class it declares for it.
StockOnHand
Units received and units issued, as two sums.
StockOnHandFixture
StockOnHand over two sites: the declared case SY-20’s rule is proven against before it may fail anyone’s projection.
Violation
A peer did something a sync rule forbids.

Enums§

KvPeerError
How a KvCappedPeer fails.
Stock
A stock movement at one site: the reference cases’ domain event.

Constants§

BYTE_LOG_REFUSED_REASON
The reason SY-37’s rule reports when the peer refused its byte-cut log, and the count-cut log showed no page.
COUNT_LOG_REFUSED_REASON
The reason SY-37’s rule reports when the peer refused its count-cut log.
KV_VALUE_CAP
The per-value cap of Turnstile peer D’s KV store: 128 KiB.
LOG_REFUSED
The capability SY-37’s rule reports when the peer showed no page boundary and refused a push of one of its logs that its own declaration admits.
MAX_PROVOKED_BATCH_BYTES
The largest max_batch_bytes batch_over_the_limit will build a batch over: 64 MiB.
MAX_PROVOKED_BATCH_EVENTS
The most events one push a rule builds will carry: 4,096.
MAX_PROVOKED_TAGS
The most tags SY-18’s ingest rule puts on one event, whatever ceiling the fixture declares (H-13).
MAX_PROVOKED_TAGS_PER_BATCH
The most tags VT-36’s ingest rule puts in one call, whatever ceiling the fixture declares (H-13).
NO_CONFIRMATION
The capability SY-31’s rule reports when the peer’s acknowledgement confirmed none of what the rule pushed.
NO_CONFIRMATION_REASON
The reason SY-31’s rule reports when the peer confirmed none of what it pushed.
NO_HIDDEN_CLAIM
The capability SY-38’s rule reports when the store did not put the rule’s compensation where its arrangement needed it.
NO_HIDDEN_CLAIM_REASON
The reason SY-38’s rule reports when it could not arrange a claim hidden by the call’s own compensation.
NO_PER_EVENT_LIMITS
The capability SY-18’s ingest rule reports when the fixture declares no per-event ceiling it can exercise.
NO_PER_EVENT_LIMITS_REASON
The reason SY-18’s ingest rule reports when it exercised no ceiling.
NO_SECOND_PAGE
The capability SY-37’s rule reports when the peer never showed it a second page: the two declared limits that decide how much one pull carries.
NO_SECOND_PAGE_REASON
The reason SY-37’s rule reports when the peer never showed it a second page.
NO_TAGS_PER_BATCH_LIMIT
The capability VT-36’s ingest rule reports when the fixture declares no tags-per-batch ceiling it can exercise: the IngestFixture constant it reads.
NO_TAGS_PER_BATCH_LIMIT_REASON
The reason VT-36’s ingest rule reports when it exercised no ceiling.
UNSTATED_COMPACTION
What SyncFixture::COMPACTION says when a fixture says nothing.
UNSTATED_MID_GROUP_FAULT
What IngestFixture::MID_GROUP_FAULT says when a fixture says nothing.
UNSTATED_SYNC_REOPEN
What SyncFixture::REOPEN says when a fixture says nothing.

Traits§

ConvergenceFixture
One projection, the events two origins authored, and a way to read what it committed.
IngestFixture
One isolated backing store that both appends and ingests, and a way to reach it.
RestoreFixture
One isolated far side whose storage can be backed up and restored, and a way to reach it.
SyncFixture
One isolated far side, and a way to reach it.

Functions§

batch_over_the_limit
A batch one byte over limits.max_batch_bytes, built from events no larger than limits.max_event_bytes, so a peer that checks both refuses it for the batch and not for any one event.
ingest_declensions_are_stated
CF-18’s machine-checked half for the ingest family: no capability F declines carries the reason it would inherit by saying nothing.
oversized_push_refusal
The oversized-push refusal, provoked on peer and declared [FailureClass::Split].
sync_declensions_are_stated
CF-18’s machine-checked half for the sync-peer family, for SyncFixture::REOPEN only: F must not decline it with the reason it would inherit by saying nothing.

Type Aliases§

EventOf
The domain event a fixture’s projection folds.
RefusalsOf
What SyncFixture::refusals resolves to for a peer P.
StoreOf
The store a fixture’s projection commits into.