Do · how-to
Pass the conformance suite
How do I make my adapter pass the conformance suite?
This page is for you if you have written a store adapter, a type that
implements the EventStore contract over your own storage, and you now want to
prove it is correct. By the end you will have a test file that runs the
conformance suite against your store, and you will know how to read what it
reports.
Why it matters: the conformance suite is the shared set of tests that defines what a correct store does. Application code written against happenstance trusts that every store behaves the same way, and the suite is how that trust is earned. An adapter counts as an adapter only once it has run the suite against its own storage and passed. Compiling is not enough, and the testkit's crate documentation says so at the top.
If you have not yet built the adapter, start with what to read before you write an adapter. This page picks up where that one stops.
What you write
You write one type and one line of test code.
The type is a fixture. The suite never takes your store directly; it asks the fixture for one. The fixture tells the suite how to open your store, and declares what your store can and cannot do. The line hands the fixture to the suite, which expands into one test per rule.
A few words used on the rest of this page:
- A rule is one test in the suite, with a descriptive name such as
two_handles_observe_each_others_appends. - A family is a group of rules with its own macro. The main one checks the event store; others race threads, generate random operations, or check projections.
- A capability is something the suite wants to test but your store may honestly be unable to support, such as surviving a restart. You declare each one as supported, or declined with a reason.
The rest of this page covers what to put in the fixture, how to read the suite's output, and which other families to add.
Write the fixture
One instance, one store
Each fixture instance must have a backing store of its own, such as a fresh
temporary file or a throwaway database schema. Each call to connect opens a new
handle onto that same store
(CF-15, checked by
two_fixture_instances_observe_none_of_each_others_appends).
This separation is what lets the suite run rules side by side without them seeing each other's events. Two instances that point at one file fail that rule.
The two capabilities you must declare
A fixture declares nine constants. Two have no default, so the compiler makes you answer them:
SECOND_HANDLEis not really a choice. The suite opens two handles onto one store, to catch a store that is only correct within a single connection (CF-16, checked bytwo_handles_observe_each_others_appends). Declining it does not skip that rule: the rule fails and quotes your reason.REOPENasks whether your fixture can close the store and open it again, so the suite can check that acknowledged writes survive a restart. If your store keeps its data somewhere that outlives the process, support it and overridereopen; the clause says what the override must do. An in-memory store declines, and says why (CF-17, provisional;acknowledged_writes_survive_a_reopen).
The two fault capabilities, which you should still write
MID_BATCH_FAULT and READ_FAULT ask whether the suite can make your store fail
part way through writing a batch, or part way through a read. Both default to
declined, with a reason written by the testkit.
Do not leave the default. The suite adds a test,
every_declined_capability_is_stated_by_this_fixture, that fails while either
one still carries the testkit's wording
(CF-18). The point is that
the reason should be about your store. Write why it cannot be faulted there,
or support the capability and override the matching arm_ method.
Supporting MID_BATCH_FAULT is a promise about where your fault fires
(CF-39, checked by
arming_a_mid_batch_fault_makes_the_append_fail), so read the clause before you
claim it.
Read-your-own-writes
READ_YOUR_OWN_WRITES defaults the other way, to supported. It means an append
you were told succeeded shows up in your next read through the same handle.
A store whose reads can briefly lag an acknowledged append should decline it,
with its own reason. The PostgreSQL and Neon fixtures do, because a read there
only shows rows below a server-wide snapshot point. Two parts of the suite read
this capability (Fixture::READ_YOUR_OWN_WRITES).
The first is the 84 named rules that read back their own appends. When your
fixture declines the capability, each of them calls
Fixture::settle_frontier after its writes and before it reads them back
(CF-42). Override it to
wait until a read shows every row your store holds, then return
Settled::Visible. The PostgreSQL and Neon fixtures poll the read path's own
predicate, with a deadline. A rule never waits on its own, so the waiting
belongs in your fixture. If you keep the provided body, it returns
Settled::Nothing and those 84 rules report a skip under
READ_YOUR_OWN_WRITES with your reason. A fixture that supports the capability
is never asked to settle.
The second is the model family (ops_agree_with_the_model). Without the
capability the family does not skip: it runs a
weaker check. Each full read must be a prefix of what you accepted that never
shrinks, and each condition verdict is judged over what your store holds,
deferred while the positions that decide it are unknown. The rule's rustdoc
lists what that check does not catch. It also assumes your store's
visibility never moves backward, so a later read never shows less than an
earlier one.
Rows held above a visibility frontier
HELD_ABOVE_FRONTIER asks whether your fixture can leave a committed row where
read does not yet show it. That is possible only on a store whose reads trail its
commits, such as the PostgreSQL and Neon stores, where a row stays invisible until
a server-wide snapshot point passes it. On such a store, contains_event_id must
still answer true for that row, because ES-41 means held, not visible
(ADR-0095;
contains_event_id_reports_a_held_row_above_the_frontier).
It defaults to declined, and the default's reason says your store has no frontier.
If your fixture supports READ_YOUR_OWN_WRITES, that is usually true, and you may
leave it. If your fixture declines READ_YOUR_OWN_WRITES, the default is false
about your store, and every_declined_capability_is_stated_by_this_fixture fails
until you either state your own reason or support the capability. To support it,
override arm_held_above_frontier so that the next append commits a row the
frontier has not passed, and return Armed::Staged. The provided body returns
Armed::Nothing, and the rule fails on that answer, so a forgotten override fails
on every run. The rule also checks that read does not show the row before it
asks the probe, so an override that answers Staged and arms nothing fails too. The PostgreSQL fixture in
its support module does it
with a trigger that lifts the new row above any frontier.
Removing events outside the port
REMOVAL asks whether your fixture can delete events from your store's own
storage, the way a retention job's raw DELETE does. EventStore has no delete,
and through 1.x it will not get one
(ADR-0028), so the deletion has
to come from your fixture's side of the storage. The rules about a store that has
lost events, ES-38's and ES-40's, stand on it.
It defaults to declined, and you may leave it: the default's reason says your
fixture has not answered, which is true. To support it, override
remove_outside_port to delete exactly the events at the positions it is given,
from the store a handle already open can see, and return Removed::Deleted. The
provided body returns Removed::Nothing, and
removal_outside_the_port_removes_exactly_the_named_events fails on that answer.
The rule also reads the store back, so an override that deletes nothing, deletes
everything from the lowest named position up, or rebuilds the log from its
survivors fails too.
Granting it also runs positions_are_not_reused_after_removal (ES-38), which
removes the tail and then everything, and requires every later append to land
above every position your store has ever assigned. An event table without
AUTOINCREMENT fails it, and so does a fixture whose removal of everything
rewinds the allocator (TRUNCATE … RESTART IDENTITY). It also runs
condition_over_removed_history_does_not_reject (ES-40): once the only event
a condition matches is removed, the append must be admitted, so a soft delete
your condition query does not filter fails it.
A removal leaves the store's identity and its position allocator where they were.
A wipe does not: TRUNCATE … RESTART IDENTITY on Postgres, or a Durable Object
cleared with delete_all(), makes a different store under VT-6, with its
positions restarting, and is not what this capability models. The SQLite,
PostgreSQL, Neon and Durable Object fixtures support it with a raw DELETE; the
SQLite support module is the
shortest.
The four size limits
Last come four ceilings: MAX_EVENT_DATA_LEN, MAX_TAGS_PER_EVENT,
MAX_EVENTS_PER_BATCH and MAX_METADATA_LEN. These are facts about your store
rather than trade-offs, so each takes a number, or None for no limit, and no
reason. The metadata ceiling is the one with no floor: any number is legal.
When you state a number, the suite tests both sides of it: an append within the
ceiling must be accepted, and one beyond it must be refused
(CF-40, provisional;
append_reports_exceeded_store_limits). Use your adapter's own constants rather
than typing the numbers again, as the SQLite fixture does in
its support module, so the
two can never disagree.
A whole fixture
The program below is a complete fixture for a store that is deliberately not the
reference in-memory store. GappyMemoryStore hands out positions seven apart,
so it leaves gaps between them. Your handle would wrap a connection or a pool
where this one wraps an Arc.
The single macro line, event_store_conformance!, is everything that goes in
your tests/conformance.rs:
use core::num::NonZeroU64;
use std::collections::BTreeSet;
use std::sync::Arc;
use futures_core::Stream;
use happenstance_core::{AppendCondition, AppendError, Event, EventId, Query};
use happenstance_core::{ReadOptions, SendEventStore, SequencePosition, SequencedEvent};
use happenstance_testkit::{Capability, Fixture, GappyMemoryStore, NO_STORE_LIMITS};
use happenstance_testkit::{RuleOutcome, UNSTATED_HELD_ABOVE_FRONTIER};
use happenstance_testkit::{UNSTATED_MID_BATCH_FAULT, UNSTATED_READ_FAULT, UNSTATED_REMOVAL};
use happenstance_testkit::{block_on, rules};
/// One handle onto a fixture's store.
#[derive(Clone)]
struct Handle(Arc<GappyMemoryStore>);
impl SendEventStore for Handle {
type Error = <GappyMemoryStore as SendEventStore>::Error;
fn read(
&self,
query: &Query,
options: ReadOptions,
) -> impl Stream<Item = Result<SequencedEvent, Self::Error>> + Send {
self.0.read(query, options)
}
async fn append(
&self,
events: &[Event],
condition: Option<&AppendCondition>,
) -> Result<SequencePosition, AppendError<Self::Error>> {
self.0.append(events, condition).await
}
async fn head(&self) -> Result<Option<SequencePosition>, Self::Error> {
self.0.head().await
}
async fn contains_event_id(&self, id: EventId) -> Result<bool, Self::Error> {
self.0.contains_event_id(id).await
}
}
/// One instance is one store. Every `connect` is a handle onto it.
struct StridedFixture(Arc<GappyMemoryStore>);
impl StridedFixture {
fn new() -> Self {
let stride = NonZeroU64::new(7).expect("seven is not zero");
Self(Arc::new(GappyMemoryStore::with_stride(stride)))
}
}
impl Fixture for StridedFixture {
type Store = Handle;
const SECOND_HANDLE: Capability = Capability::SUPPORTED;
const REOPEN: Capability = Capability::declined(
"the events live in a Vec in this process, so nothing outlives a reopen",
);
const MID_BATCH_FAULT: Capability = Capability::declined(
"a batch is written under one lock, so no fault can fire between two rows",
);
const READ_FAULT: Capability = Capability::declined(
"a read iterates memory the store already holds, so there is no fetch to fail",
);
const HELD_ABOVE_FRONTIER: Capability = Capability::declined(
"a read shows every row from the moment its append returns, so there is no frontier",
);
const REMOVAL: Capability = Capability::declined(
"the events live in a Vec only the store can reach, so nothing outside it can delete one",
);
fn connect(&self) -> impl Future<Output = Self::Store> {
core::future::ready(Handle(Arc::clone(&self.0)))
}
}
happenstance_testkit::event_store_conformance!(StridedFixture::new());
fn main() {
// Every rule the suite enumerates, driven the way `emit_blocking` does.
// A rule that fails panics, so reaching the end means none failed.
macro_rules! run_each {
($($rule:ident),* $(,)?) => {
[$((stringify!($rule), block_on(rules::$rule(async || StridedFixture::new())))),*]
};
}
let outcomes = happenstance_testkit::for_each_event_store_rule!(run_each);
let mut skipped = BTreeSet::new();
for (rule, outcome) in outcomes {
if let RuleOutcome::Skipped { capability, reason } = outcome {
skipped.insert(capability);
// Each reason is this fixture's own, never the testkit's default.
let defaults = [
UNSTATED_HELD_ABOVE_FRONTIER,
UNSTATED_MID_BATCH_FAULT,
UNSTATED_READ_FAULT,
UNSTATED_REMOVAL,
];
assert!(!defaults.contains(&reason), "{rule}");
if capability == "REOPEN" {
assert_eq!(Some(reason), StridedFixture::REOPEN.reason(), "{rule}");
}
}
}
let declined = [
"HELD_ABOVE_FRONTIER",
"MID_BATCH_FAULT",
NO_STORE_LIMITS,
"READ_FAULT",
"REMOVAL",
"REOPEN",
];
assert_eq!(skipped, BTreeSet::from(declined));
}
In your crate, cargo test turns that macro line into one test per rule. This
page's example is a doctest, and a doctest is not built as a test binary, so the
generated tests do not exist here. Instead, main runs the same rules itself,
the way the emit_blocking harness does.
main then checks three things. Every rule either passed or was skipped, for a
capability the fixture declined or a ceiling it left as None. No skip carries
the testkit's default wording. And one generated test,
every_declined_capability_is_stated_by_this_fixture, is left to your own
cargo test, because it has no rule behind it to call here.
Gaps in positions are fine
Positions may have gaps
(VT-11),
and no rule checks for a literal position
(CF-6, checked by
mutation_coverage::conformant_variants_pass_everything). The testkit runs the
whole suite against this exact store, stride and all, in its own tests.
If you write tests of your own beside the suite, hold them to the same rule:
compare against positions your store returned, never against 1, 2, 3.
Real fixtures to copy from
- SQLite's, over a temporary file, which supports
REOPENand states all three ceilings, in its support module. - The reference
MemoryFixture, in the testkit's fixtures. - PostgreSQL's, which gives each instance a throwaway schema, in its support module.
Read a skip
When a rule needs a capability your fixture declined, it still runs as a test.
It passes, and prints one line naming the capability and your reason
(CF-18, checked by
mutation_coverage::capability_skips_are_reported).
The suite never removes a rule from the test binary instead. In CI, a missing test looks exactly like a passing one, so removing it would hide the gap.
You have to ask to see those lines. Rust's test runner hides what a passing test
prints, so a plain cargo test shows only the pass count. Run it with
-- --show-output to read them, and do that in CI if you want anyone else to
see them. RuleOutcome::report notes this limit where it is implemented.
Each declined capability is a trade-off, and your reason is the only record of
it. Write it for someone deciding whether to use your adapter, not for yourself.
Nothing checks that a reason is any good: "n/a" passes.
Read a failure
A failing rule panics, and the failing test carries the rule's name. Look the
rule up in the testkit's rules module. Its documentation lists the wrong
implementations it is designed to catch, and yours is often one of them.
The program below shows a common mistake. Every connect opens a brand-new
store, the way every connection to SQLite's :memory: opens a database of its
own. The isolation rule passes, because two fixture instances really do share
nothing. The second-handle rule fails, because two handles onto one instance
share nothing either:
use std::panic::{AssertUnwindSafe, catch_unwind};
use happenstance_core::MemoryEventStore;
use happenstance_testkit::{Capability, Fixture, RuleOutcome, block_on, rules};
// WRONG: every `connect` opens a new, empty store.
struct StorePerConnection;
impl Fixture for StorePerConnection {
type Store = MemoryEventStore;
const SECOND_HANDLE: Capability = Capability::SUPPORTED;
const REOPEN: Capability = Capability::declined("nothing outlives this process");
fn connect(&self) -> impl Future<Output = Self::Store> {
core::future::ready(MemoryEventStore::new())
}
}
// END WRONG.
fn main() {
let open = async || StorePerConnection;
let isolated = block_on(rules::two_fixture_instances_observe_none_of_each_others_appends(open));
assert_eq!(isolated, RuleOutcome::Ran);
let shared = catch_unwind(AssertUnwindSafe(|| {
block_on(rules::two_handles_observe_each_others_appends(open))
}));
let failure = shared.expect_err("the second handle saw nothing, so the rule fails");
let message = failure.downcast_ref::<String>().map_or("", String::as_str);
assert!(message.contains("second handle"), "{message}");
}
Decline a capability only when your store genuinely lacks it, and say why.
Declining one just to turn a failing rule green is a different thing.
SECOND_HANDLE refuses it outright, as above. The others record it in every skip
line, where the people choosing your adapter can read it.
If you believe a rule is wrong, fix it in the testkit, in the same change as an
example of the wrong implementation the rule must still catch. Never skip it.
CONTRIBUTING.md, under
Writing an adapter, says the same.
Choose the harness
Each rule is an async function, so something has to run it. The thing that
wraps each rule into a test is called an emitter, and you choose it to suit
your runtime
(CF-23).
The one-line form uses tokio, wrapping each rule in #[tokio::test], so your
dev-dependencies need tokio with the macros and rt features. The long
form names the emitter:
event_store_conformance!(mod_name = …, emit = …, fixture = …).
emit_blocking(happenstance_testkit::emit_blocking) needs no async runtime at all.emit_wasm(happenstance_testkit::emit_wasm) runs underwasm-bindgen-test, for a store that targetswasm32. The Cloudflare adapter uses it in its conformance target.
Those names are a stable promise: renaming one would be a major release of the
testkit
(CF-41,
checked by the_promised_emitters_are_exactly_the_pinned_list).
You can also write an emitter of your own, but it is not covered by that promise.
The PostgreSQL adapter writes one to mark every rule #[ignore] until a database
server is available, in
its conformance target.
It has to name __conformance_fixture, which CF-41 leaves out of the promise, so
you keep it in step with each testkit release yourself.
Add the other families
The one line runs the event-store family. Three more exist, each with its own macro. Add the ones that fit your store.
-
Concurrency.
event_store_concurrency_conformance!racesconcurrency::CONTENDERSthreads against one store. Among its rules,exactly_one_of_n_contenders_commitschecks that when many writers race on the same condition, exactly one wins (ES-25).It is opt-in, and only a store whose handle is
Send(safe to move between threads) can run it. Its default emitter also needstokio'srt-multi-threadfeature.Your fixture must be able to open one more handle than there are contenders. A connection pool smaller than that hangs rather than failing, because no rule may read a clock (CF-33), so there is no timeout to fail it.
-
Model.
event_store_model_conformance!generates random sequences of operations and checks your store against a simple model of the log. It needs the testkit'sproptestfeature on your dev-dependency. On a fixture that declinesREAD_YOUR_OWN_WRITESit runs a weaker check rather than skipping (see Read-your-own-writes above). The SQLite adapter runs it beside the event-store family in its conformance target. -
Projections. If your crate also implements
ProjectionStore, the store for read models,projection_store_conformance!takes aProjectionFixtureand checks that contract. It needsProjectionProbeimplemented beside your store, behind aconformancefeature of your own crate.The testkit's crate documentation walks the six steps and the manifest under Writing a projection adapter from outside this workspace. The outside projection adapter was written from those steps alone, and runs the family in one line.
Pin the suite
A new rule can arrive in a minor release of the testkit
(CF-29), and
it can turn your passing build red. So pin the testkit exactly, with =, in your
dev-dependencies. Then you choose when to take on a new bar.
That advice is CF-30, which is a recommendation rather than a requirement. It costs your users nothing, because Cargo never resolves a library's dev-dependencies for the people who depend on it.
When an existing rule's meaning changes, the testkit takes a major release that links the decision behind it (CF-31).
What a green run does not tell you
A passing suite is strong evidence, but it has blind spots you should know about.
- That
reopenis honest. Make yourreopenreally close what the fixture holds and read the storage again, because nothing else will catch it. An emptyreopenover a store that loses everything on restart passes every rule. The testkit records that blind spot inreopen_over_claiming_is_undetectable_and_this_is_the_record. - That your store is fast enough. Performance is measured by a separate benchmark harness, which no adapter has to pass (CF-34).
- That the rules you skipped hold. A skipped rule passed without running. Read the skip lines before you quote the pass count.