Do · how-to
Sync a device to a central store
How do I keep a device's event store in sync with a central one?
This page is for you if each device in your system keeps its own event log, so it can take decisions while it has no signal, and a central store collects what every device wrote. By the end you will have a device that syncs through a transport you control, a failed sync that loses nothing, a sync position that survives a restart, and a plan for the case where a device and central both decided something the other did not see.
Everything here comes from
happenstance-sync's README and the
documentation of the items named. The programs run in memory, on
MemoryIngestStore from that crate's memory feature. The lines that differ
on a real database are in the last sections, in fences marked as not compiled
here.
The shape
One relationship between two stores is an edge. A device has one edge to central, and three things make it up:
- On the device: its own store, a
runner::Edgethat remembers how far the two sides got, and aRemoteSyncPeerthat stands for central. - On central: a
StoreSyncPeerover central's store, andserve_sync, which turns one request's bytes into one reply's bytes. - Between them: an
Exchange, which is yours. It carries the bytes of one request toserve_syncand brings the reply back, over HTTP or whatever your devices speak.
Each step of the edge pulls what central holds and ingests it, then pushes what
the device holds. A device pushes its whole log, the events it pulled from
central included, and central skips what it already holds
(SY-11). The port
and the runner ask for no Send
(SY-17), so a single-threaded
device runtime is a valid home for all of it.
Write the exchange, and step the edge
The program below is a whole edge. Uplink is the exchange: here it calls
serve_sync directly, where yours would send the request over the network.
Its error states a failure class for each way it fails, and the runner acts on
the class it is given rather than guessing
(SY-36).
No signal is Transient: nothing was sent, and the same request will do later.
A real store's handle is cheap to clone, and StoreSyncPeer takes two: one to
read central's log, one to ingest into it. MemoryIngestStore cannot be
cloned, so the program shares it through a small wrapper, Shared, whose
forwarding code is hidden.
use std::cell::Cell;
use std::collections::BTreeSet;
use happenstance_core::{Event, EventStore, SequencedEvent};
use happenstance_sync::runner::Edge;
use happenstance_sync::wire::Route;
use happenstance_sync::{
Classify, Exchange, FailureClass, MemoryIngestStore, RemoteSyncPeer, ServeError,
StoreSyncPeer, SyncPeer, serve_sync,
};
/// The device's link to central, and whether it has signal.
struct Uplink {
far: StoreSyncPeer<Shared, Shared>,
signal: Cell<bool>,
}
#[derive(Debug, thiserror::Error)]
enum UplinkError {
#[error("no signal: the request never left the device")]
Offline,
#[error(transparent)]
Serve(#[from] ServeError<GroupOrderError, MemoryStoreError>),
}
impl Classify for UplinkError {
fn class(&self) -> FailureClass {
match self {
Self::Offline => FailureClass::Transient,
Self::Serve(error) => error.class(),
}
}
}
impl Exchange for Uplink {
type Error = UplinkError;
async fn exchange(&self, route: Route, request: Vec<u8>) -> Result<Vec<u8>, UplinkError> {
if !self.signal.get() {
return Err(UplinkError::Offline);
}
Ok(serve_sync(&self.far, route, &request).await?)
}
}
let central = Shared::default();
let far = StoreSyncPeer::new(central.clone(), central.clone());
let limits = far.limits();
let signal = Cell::new(true);
let peer = RemoteSyncPeer::new(Uplink { far, signal }, limits);
let tablet = MemoryIngestStore::new();
central.append(&[Event::new("ShowScheduled", &b"{}"[..])?], None).await?;
tablet.append(&[Event::new("SeatSold", &b"{}"[..])?], None).await?;
let mut edge = Edge::new();
edge.step(&peer, &tablet).await?;
let ids = |log: Vec<SequencedEvent>| -> BTreeSet<_> { log.iter().map(|e| e.id).collect() };
assert_eq!(ids(tablet.snapshot()), ids(central.snapshot()));
assert_eq!(central.snapshot().len(), 2);
assert!(edge.status().is_synced());
The two logs are compared by EventId, the identity the writing store gave
each event. Positions are local to each store, so the same event sits at a
different position on each side.
Build the edge once per device and keep it for the life of the relationship. It holds the token central handed it, so stepping it against a different peer would hand that peer a token it never minted.
When the signal drops
A step that fails is reported with the phase it failed in and its class, and
the edge decides what to do next from the class. A Transient failure leaves
the edge running and loses nothing: the events it did not push are pushed by
the next step, and a refused push leaves the device's log as it was
(SY-34).
use happenstance_sync::FailureClass;
use happenstance_sync::runner::{Edge, EdgeState};
let mut edge = Edge::new();
peer.exchange().signal.set(false);
let sold = tablet.append(&[Event::new("SeatSold", &b"{}"[..])?], None).await?;
let failed = edge.step(&peer, &tablet).await;
let failure = failed.err().ok_or("a step with no signal fails")?.failure();
assert_eq!(failure.class, FailureClass::Transient);
assert_eq!(edge.status().state, EdgeState::Running);
assert!(central.snapshot().is_empty());
peer.exchange().signal.set(true);
edge.step(&peer, &tablet).await?;
let landed: Vec<EventId> = central.snapshot().iter().map(|e| e.id).collect();
assert_eq!(landed, [EventId::new(tablet.store_id(), sold)]);
The loop that keeps stepping is yours to choose. Edge::run steps until the
edge is synced, waiting between attempts by a backoff you supply. A device
that syncs when the user asks, or when the network comes back, calls step
itself.
Keep the sync position across a restart
The edge's watermark says which of the device's events central has confirmed. It is not an event, since no peer should replicate it. It is kept through the device's projection store, in a batch of yours, so it lands in the same commit as whatever read model that batch carries (SY-31).
use happenstance_core::MemoryProjectionStore;
use happenstance_sync::runner::{Edge, Watermark};
let models = MemoryProjectionStore::new();
let mut edge = Edge::new();
edge.step(&peer, &tablet).await?;
let kept = edge.watermark().ok_or("central confirmed a push and named itself")?;
let batch = models.open();
kept.commit(&models, batch).await?;
// The device restarts: the edge is rebuilt from what was kept.
let loaded = Watermark::load(&models, kept.peer()).await?;
let mut edge = Edge::restored(loaded.ok_or("kept above")?);
let stepped = edge.step(&peer, &tablet).await?;
assert_eq!(stepped.pushed, 0, "nothing above the watermark to push");
assert_eq!(stepped.ingested.appended, 0, "a re-pull lands nothing twice");
A restored edge pulls from the start of central's log, because the pull token is not kept. That costs a re-read and nothing more: ingest skips what the device already holds.
On SQLite
SqliteEventStore is both halves a sync edge needs once its sync feature
is on, and a clone of it is a second handle onto the same file. Neither line
below is compiled on this page:
happenstance-sqlite = { version = "0.7", features = ["sync"] }
happenstance-sync = "0.7"let central = SqliteEventStore::open("central.db")?;
let far = StoreSyncPeer::new(central.clone(), central.clone());
let tablet = SqliteEventStore::open("tablet.db")?;
edge.step(&peer, &tablet).await?;
For a Postgres central, PostgresEventStore takes SqliteEventStore's place
behind StoreSyncPeer and serve_sync, with the sync feature of
happenstance-postgres. Neon is a different shape: NeonSyncPeer is the peer
itself, so the device talks to Neon over one-shot HTTP with no server between,
and takes it in place of RemoteSyncPeer.
When both sides wrote
Two devices offline can each sell the last seat, and each decision is right about the history that device held. When they sync, central ingests both: ingest never refuses an event for a domain reason (SY-1). The conflict is answered afterwards, by an ordinary append that compensates (SY-2), written by one peer only (SY-7).
StoreSyncPeer writes no compensation unless you give it a merge rule. From
0.7 the runner has a hook for one
(#450): a
runner::Adjudicate, mounted with Edge::with_adjudicator for what an edge
pulls, or with StoreSyncPeer::with_adjudicator, served through
serve_sync_adjudicated, for what a far side is pushed. For central, that is
the second: mount the rule on central's peer, and leave each device's edge on
the default, which judges nothing, so central is the one peer that writes a
compensation. The rule is asked about each pushed group before the push
lands, reads what it needs from its own handle onto central, and returns the
compensation; it writes nothing itself. The runner lands each group and its
compensation in one write, and the compensation is an event, so it syncs
back to every device. Not compiled on this page: CentralRule is the
example's:
let far = StoreSyncPeer::new(central.clone(), central.clone())
.with_adjudicator(CentralRule::new(central.clone(), show));
serve_sync_adjudicated(&far, route, &request).await?
examples/two-devices-one-seat
is that arrangement end to end on three SQLite files: both devices sell the
last seat, central's rule refunds the sale that reached it last as that sale
lands, and the three files converge on the same log and the same read model.