0.7.0
GitHub

Do · how-to

Pass the conformance suite

How do I make my adapter pass the conformance suite?

This page is for you if you have written a store adapter, a type that implements the EventStore contract over your own storage, and you now want to prove it is correct. By the end you will have a test file that runs the conformance suite against your store, and you will know how to read what it reports.

Why it matters: the conformance suite is the shared set of tests that defines what a correct store does. Application code written against happenstance trusts that every store behaves the same way, and the suite is how that trust is earned. An adapter counts as an adapter only once it has run the suite against its own storage and passed. Compiling is not enough, and the testkit's crate documentation says so at the top.

If you have not yet built the adapter, start with what to read before you write an adapter. This page picks up where that one stops.

What you write

You write one type and one line of test code.

The type is a fixture. The suite never takes your store directly; it asks the fixture for one. The fixture tells the suite how to open your store, and declares what your store can and cannot do. The line hands the fixture to the suite, which expands into one test per rule.

A few words used on the rest of this page:

  • A rule is one test in the suite, with a descriptive name such as two_handles_observe_each_others_appends.
  • A family is a group of rules with its own macro. The main one checks the event store; others race threads, generate random operations, or check projections.
  • A capability is something the suite wants to test but your store may honestly be unable to support, such as surviving a restart. You declare each one as supported, or declined with a reason.

The rest of this page covers what to put in the fixture, how to read the suite's output, and which other families to add.

Write the fixture

One instance, one store

Each fixture instance must have a backing store of its own, such as a fresh temporary file or a throwaway database schema. Each call to connect opens a new handle onto that same store (CF-15, checked by two_fixture_instances_observe_none_of_each_others_appends).

This separation is what lets the suite run rules side by side without them seeing each other's events. Two instances that point at one file fail that rule.

The two capabilities you must declare

A fixture declares nine constants. Two have no default, so the compiler makes you answer them:

  • SECOND_HANDLE is not really a choice. The suite opens two handles onto one store, to catch a store that is only correct within a single connection (CF-16, checked by two_handles_observe_each_others_appends). Declining it does not skip that rule: the rule fails and quotes your reason.
  • REOPEN asks whether your fixture can close the store and open it again, so the suite can check that acknowledged writes survive a restart. If your store keeps its data somewhere that outlives the process, support it and override reopen; the clause says what the override must do. An in-memory store declines, and says why (CF-17, provisional; acknowledged_writes_survive_a_reopen).

The two fault capabilities, which you should still write

MID_BATCH_FAULT and READ_FAULT ask whether the suite can make your store fail part way through writing a batch, or part way through a read. Both default to declined, with a reason written by the testkit.

Do not leave the default. The suite adds a test, every_declined_capability_is_stated_by_this_fixture, that fails while either one still carries the testkit's wording (CF-18). The point is that the reason should be about your store. Write why it cannot be faulted there, or support the capability and override the matching arm_ method.

Supporting MID_BATCH_FAULT is a promise about where your fault fires (CF-39, checked by arming_a_mid_batch_fault_makes_the_append_fail), so read the clause before you claim it.

Read-your-own-writes

READ_YOUR_OWN_WRITES defaults the other way, to supported. It means an append you were told succeeded shows up in your next read through the same handle.

A store whose reads can briefly lag an acknowledged append should decline it, with its own reason. The PostgreSQL and Neon fixtures do, because a read there only shows rows below a server-wide snapshot point. Two parts of the suite read this capability (Fixture::READ_YOUR_OWN_WRITES).

The first is the 84 named rules that read back their own appends. When your fixture declines the capability, each of them calls Fixture::settle_frontier after its writes and before it reads them back (CF-42). Override it to wait until a read shows every row your store holds, then return Settled::Visible. The PostgreSQL and Neon fixtures poll the read path's own predicate, with a deadline. A rule never waits on its own, so the waiting belongs in your fixture. If you keep the provided body, it returns Settled::Nothing and those 84 rules report a skip under READ_YOUR_OWN_WRITES with your reason. A fixture that supports the capability is never asked to settle.

The second is the model family (ops_agree_with_the_model). Without the capability the family does not skip: it runs a weaker check. Each full read must be a prefix of what you accepted that never shrinks, and each condition verdict is judged over what your store holds, deferred while the positions that decide it are unknown. The rule's rustdoc lists what that check does not catch. It also assumes your store's visibility never moves backward, so a later read never shows less than an earlier one.

Rows held above a visibility frontier

HELD_ABOVE_FRONTIER asks whether your fixture can leave a committed row where read does not yet show it. That is possible only on a store whose reads trail its commits, such as the PostgreSQL and Neon stores, where a row stays invisible until a server-wide snapshot point passes it. On such a store, contains_event_id must still answer true for that row, because ES-41 means held, not visible (ADR-0095; contains_event_id_reports_a_held_row_above_the_frontier).

It defaults to declined, and the default's reason says your store has no frontier. If your fixture supports READ_YOUR_OWN_WRITES, that is usually true, and you may leave it. If your fixture declines READ_YOUR_OWN_WRITES, the default is false about your store, and every_declined_capability_is_stated_by_this_fixture fails until you either state your own reason or support the capability. To support it, override arm_held_above_frontier so that the next append commits a row the frontier has not passed, and return Armed::Staged. The provided body returns Armed::Nothing, and the rule fails on that answer, so a forgotten override fails on every run. The rule also checks that read does not show the row before it asks the probe, so an override that answers Staged and arms nothing fails too. The PostgreSQL fixture in its support module does it with a trigger that lifts the new row above any frontier.

Removing events outside the port

REMOVAL asks whether your fixture can delete events from your store's own storage, the way a retention job's raw DELETE does. EventStore has no delete, and through 1.x it will not get one (ADR-0028), so the deletion has to come from your fixture's side of the storage. The rules about a store that has lost events, ES-38's and ES-40's, stand on it.

It defaults to declined, and you may leave it: the default's reason says your fixture has not answered, which is true. To support it, override remove_outside_port to delete exactly the events at the positions it is given, from the store a handle already open can see, and return Removed::Deleted. The provided body returns Removed::Nothing, and removal_outside_the_port_removes_exactly_the_named_events fails on that answer. The rule also reads the store back, so an override that deletes nothing, deletes everything from the lowest named position up, or rebuilds the log from its survivors fails too.

Granting it also runs positions_are_not_reused_after_removal (ES-38), which removes the tail and then everything, and requires every later append to land above every position your store has ever assigned. An event table without AUTOINCREMENT fails it, and so does a fixture whose removal of everything rewinds the allocator (TRUNCATE … RESTART IDENTITY). It also runs condition_over_removed_history_does_not_reject (ES-40): once the only event a condition matches is removed, the append must be admitted, so a soft delete your condition query does not filter fails it.

A removal leaves the store's identity and its position allocator where they were. A wipe does not: TRUNCATE … RESTART IDENTITY on Postgres, or a Durable Object cleared with delete_all(), makes a different store under VT-6, with its positions restarting, and is not what this capability models. The SQLite, PostgreSQL, Neon and Durable Object fixtures support it with a raw DELETE; the SQLite support module is the shortest.

The four size limits

Last come four ceilings: MAX_EVENT_DATA_LEN, MAX_TAGS_PER_EVENT, MAX_EVENTS_PER_BATCH and MAX_METADATA_LEN. These are facts about your store rather than trade-offs, so each takes a number, or None for no limit, and no reason. The metadata ceiling is the one with no floor: any number is legal.

When you state a number, the suite tests both sides of it: an append within the ceiling must be accepted, and one beyond it must be refused (CF-40, provisional; append_reports_exceeded_store_limits). Use your adapter's own constants rather than typing the numbers again, as the SQLite fixture does in its support module, so the two can never disagree.

A whole fixture

The program below is a complete fixture for a store that is deliberately not the reference in-memory store. GappyMemoryStore hands out positions seven apart, so it leaves gaps between them. Your handle would wrap a connection or a pool where this one wraps an Arc.

The single macro line, event_store_conformance!, is everything that goes in your tests/conformance.rs:

use core::num::NonZeroU64;
use std::collections::BTreeSet;
use std::sync::Arc;

use futures_core::Stream;
use happenstance_core::{AppendCondition, AppendError, Event, EventId, Query};
use happenstance_core::{ReadOptions, SendEventStore, SequencePosition, SequencedEvent};
use happenstance_testkit::{Capability, Fixture, GappyMemoryStore, NO_STORE_LIMITS};
use happenstance_testkit::{RuleOutcome, UNSTATED_HELD_ABOVE_FRONTIER};
use happenstance_testkit::{UNSTATED_MID_BATCH_FAULT, UNSTATED_READ_FAULT, UNSTATED_REMOVAL};
use happenstance_testkit::{block_on, rules};

/// One handle onto a fixture's store.
#[derive(Clone)]
struct Handle(Arc<GappyMemoryStore>);

impl SendEventStore for Handle {
    type Error = <GappyMemoryStore as SendEventStore>::Error;

    fn read(
        &self,
        query: &Query,
        options: ReadOptions,
    ) -> impl Stream<Item = Result<SequencedEvent, Self::Error>> + Send {
        self.0.read(query, options)
    }

    async fn append(
        &self,
        events: &[Event],
        condition: Option<&AppendCondition>,
    ) -> Result<SequencePosition, AppendError<Self::Error>> {
        self.0.append(events, condition).await
    }

    async fn head(&self) -> Result<Option<SequencePosition>, Self::Error> {
        self.0.head().await
    }

    async fn contains_event_id(&self, id: EventId) -> Result<bool, Self::Error> {
        self.0.contains_event_id(id).await
    }
}

/// One instance is one store. Every `connect` is a handle onto it.
struct StridedFixture(Arc<GappyMemoryStore>);

impl StridedFixture {
    fn new() -> Self {
        let stride = NonZeroU64::new(7).expect("seven is not zero");
        Self(Arc::new(GappyMemoryStore::with_stride(stride)))
    }
}

impl Fixture for StridedFixture {
    type Store = Handle;

    const SECOND_HANDLE: Capability = Capability::SUPPORTED;
    const REOPEN: Capability = Capability::declined(
        "the events live in a Vec in this process, so nothing outlives a reopen",
    );
    const MID_BATCH_FAULT: Capability = Capability::declined(
        "a batch is written under one lock, so no fault can fire between two rows",
    );
    const READ_FAULT: Capability = Capability::declined(
        "a read iterates memory the store already holds, so there is no fetch to fail",
    );
    const HELD_ABOVE_FRONTIER: Capability = Capability::declined(
        "a read shows every row from the moment its append returns, so there is no frontier",
    );
    const REMOVAL: Capability = Capability::declined(
        "the events live in a Vec only the store can reach, so nothing outside it can delete one",
    );

    fn connect(&self) -> impl Future<Output = Self::Store> {
        core::future::ready(Handle(Arc::clone(&self.0)))
    }
}

happenstance_testkit::event_store_conformance!(StridedFixture::new());

fn main() {
    // Every rule the suite enumerates, driven the way `emit_blocking` does.
    // A rule that fails panics, so reaching the end means none failed.
    macro_rules! run_each {
        ($($rule:ident),* $(,)?) => {
            [$((stringify!($rule), block_on(rules::$rule(async || StridedFixture::new())))),*]
        };
    }
    let outcomes = happenstance_testkit::for_each_event_store_rule!(run_each);

    let mut skipped = BTreeSet::new();
    for (rule, outcome) in outcomes {
        if let RuleOutcome::Skipped { capability, reason } = outcome {
            skipped.insert(capability);
            // Each reason is this fixture's own, never the testkit's default.
            let defaults = [
                UNSTATED_HELD_ABOVE_FRONTIER,
                UNSTATED_MID_BATCH_FAULT,
                UNSTATED_READ_FAULT,
                UNSTATED_REMOVAL,
            ];
            assert!(!defaults.contains(&reason), "{rule}");
            if capability == "REOPEN" {
                assert_eq!(Some(reason), StridedFixture::REOPEN.reason(), "{rule}");
            }
        }
    }
    let declined = [
        "HELD_ABOVE_FRONTIER",
        "MID_BATCH_FAULT",
        NO_STORE_LIMITS,
        "READ_FAULT",
        "REMOVAL",
        "REOPEN",
    ];
    assert_eq!(skipped, BTreeSet::from(declined));
}

In your crate, cargo test turns that macro line into one test per rule. This page's example is a doctest, and a doctest is not built as a test binary, so the generated tests do not exist here. Instead, main runs the same rules itself, the way the emit_blocking harness does.

main then checks three things. Every rule either passed or was skipped, for a capability the fixture declined or a ceiling it left as None. No skip carries the testkit's default wording. And one generated test, every_declined_capability_is_stated_by_this_fixture, is left to your own cargo test, because it has no rule behind it to call here.

Gaps in positions are fine

Positions may have gaps (VT-11), and no rule checks for a literal position (CF-6, checked by mutation_coverage::conformant_variants_pass_everything). The testkit runs the whole suite against this exact store, stride and all, in its own tests.

If you write tests of your own beside the suite, hold them to the same rule: compare against positions your store returned, never against 1, 2, 3.

Real fixtures to copy from

Read a skip

When a rule needs a capability your fixture declined, it still runs as a test. It passes, and prints one line naming the capability and your reason (CF-18, checked by mutation_coverage::capability_skips_are_reported).

The suite never removes a rule from the test binary instead. In CI, a missing test looks exactly like a passing one, so removing it would hide the gap.

You have to ask to see those lines. Rust's test runner hides what a passing test prints, so a plain cargo test shows only the pass count. Run it with -- --show-output to read them, and do that in CI if you want anyone else to see them. RuleOutcome::report notes this limit where it is implemented.

Each declined capability is a trade-off, and your reason is the only record of it. Write it for someone deciding whether to use your adapter, not for yourself. Nothing checks that a reason is any good: "n/a" passes.

Read a failure

A failing rule panics, and the failing test carries the rule's name. Look the rule up in the testkit's rules module. Its documentation lists the wrong implementations it is designed to catch, and yours is often one of them.

The program below shows a common mistake. Every connect opens a brand-new store, the way every connection to SQLite's :memory: opens a database of its own. The isolation rule passes, because two fixture instances really do share nothing. The second-handle rule fails, because two handles onto one instance share nothing either:

use std::panic::{AssertUnwindSafe, catch_unwind};

use happenstance_core::MemoryEventStore;
use happenstance_testkit::{Capability, Fixture, RuleOutcome, block_on, rules};

// WRONG: every `connect` opens a new, empty store.
struct StorePerConnection;

impl Fixture for StorePerConnection {
    type Store = MemoryEventStore;

    const SECOND_HANDLE: Capability = Capability::SUPPORTED;
    const REOPEN: Capability = Capability::declined("nothing outlives this process");

    fn connect(&self) -> impl Future<Output = Self::Store> {
        core::future::ready(MemoryEventStore::new())
    }
}
// END WRONG.

fn main() {
    let open = async || StorePerConnection;

    let isolated = block_on(rules::two_fixture_instances_observe_none_of_each_others_appends(open));
    assert_eq!(isolated, RuleOutcome::Ran);

    let shared = catch_unwind(AssertUnwindSafe(|| {
        block_on(rules::two_handles_observe_each_others_appends(open))
    }));
    let failure = shared.expect_err("the second handle saw nothing, so the rule fails");
    let message = failure.downcast_ref::<String>().map_or("", String::as_str);
    assert!(message.contains("second handle"), "{message}");
}

Decline a capability only when your store genuinely lacks it, and say why. Declining one just to turn a failing rule green is a different thing. SECOND_HANDLE refuses it outright, as above. The others record it in every skip line, where the people choosing your adapter can read it.

If you believe a rule is wrong, fix it in the testkit, in the same change as an example of the wrong implementation the rule must still catch. Never skip it. CONTRIBUTING.md, under Writing an adapter, says the same.

Choose the harness

Each rule is an async function, so something has to run it. The thing that wraps each rule into a test is called an emitter, and you choose it to suit your runtime (CF-23).

The one-line form uses tokio, wrapping each rule in #[tokio::test], so your dev-dependencies need tokio with the macros and rt features. The long form names the emitter: event_store_conformance!(mod_name = …, emit = …, fixture = …).

  • emit_blocking (happenstance_testkit::emit_blocking) needs no async runtime at all.
  • emit_wasm (happenstance_testkit::emit_wasm) runs under wasm-bindgen-test, for a store that targets wasm32. The Cloudflare adapter uses it in its conformance target.

Those names are a stable promise: renaming one would be a major release of the testkit (CF-41, checked by the_promised_emitters_are_exactly_the_pinned_list).

You can also write an emitter of your own, but it is not covered by that promise. The PostgreSQL adapter writes one to mark every rule #[ignore] until a database server is available, in its conformance target. It has to name __conformance_fixture, which CF-41 leaves out of the promise, so you keep it in step with each testkit release yourself.

Add the other families

The one line runs the event-store family. Three more exist, each with its own macro. Add the ones that fit your store.

  • Concurrency. event_store_concurrency_conformance! races concurrency::CONTENDERS threads against one store. Among its rules, exactly_one_of_n_contenders_commits checks that when many writers race on the same condition, exactly one wins (ES-25).

    It is opt-in, and only a store whose handle is Send (safe to move between threads) can run it. Its default emitter also needs tokio's rt-multi-thread feature.

    Your fixture must be able to open one more handle than there are contenders. A connection pool smaller than that hangs rather than failing, because no rule may read a clock (CF-33), so there is no timeout to fail it.

  • Model. event_store_model_conformance! generates random sequences of operations and checks your store against a simple model of the log. It needs the testkit's proptest feature on your dev-dependency. On a fixture that declines READ_YOUR_OWN_WRITES it runs a weaker check rather than skipping (see Read-your-own-writes above). The SQLite adapter runs it beside the event-store family in its conformance target.

  • Projections. If your crate also implements ProjectionStore, the store for read models, projection_store_conformance! takes a ProjectionFixture and checks that contract. It needs ProjectionProbe implemented beside your store, behind a conformance feature of your own crate.

    The testkit's crate documentation walks the six steps and the manifest under Writing a projection adapter from outside this workspace. The outside projection adapter was written from those steps alone, and runs the family in one line.

Pin the suite

A new rule can arrive in a minor release of the testkit (CF-29), and it can turn your passing build red. So pin the testkit exactly, with =, in your dev-dependencies. Then you choose when to take on a new bar.

That advice is CF-30, which is a recommendation rather than a requirement. It costs your users nothing, because Cargo never resolves a library's dev-dependencies for the people who depend on it.

When an existing rule's meaning changes, the testkit takes a major release that links the decision behind it (CF-31).

What a green run does not tell you

A passing suite is strong evidence, but it has blind spots you should know about.

  • That reopen is honest. Make your reopen really close what the fixture holds and read the storage again, because nothing else will catch it. An empty reopen over a store that loses everything on restart passes every rule. The testkit records that blind spot in reopen_over_claiming_is_undetectable_and_this_is_the_record.
  • That your store is fast enough. Performance is measured by a separate benchmark harness, which no adapter has to pass (CF-34).
  • That the rules you skipped hold. A skipped rule passed without running. Read the skip lines before you quote the pass count.