Do · how-to
Rebuild a read model
How do I rebuild a read model from the log?
A read model is built entirely from the events in the log, so you can always throw it away and build it again. You will need to when its rows go wrong, or when you change the code that builds it. This page shows how to do either without showing your users a wrong answer and without applying any event twice.
A few words first, because this page uses them throughout:
- A projection is the code that builds one read model. The function that turns each event into row changes is its fold.
- Each projection has a
ProjectionId, a name that identifies it. - A checkpoint is the position of the last event a projection has applied. It is stored next to the read model's rows, so that after a restart the projection carries on from where it stopped instead of starting over.
- A batch is a set of row changes that the projection store commits together with the checkpoint, as one transaction.
Two reasons to rebuild
How you rebuild depends on why.
- The fold is right, but the rows are wrong. A table was dropped, a bad deploy wrote nonsense, or you just want a clean copy. Reset the view and replay it under the same id.
- The fold itself changed. It counts something different, or it now reads events it used to ignore. Build the new version beside the old one under a second id, switch your readers to it, then retire the old one.
The first is quicker. The second keeps a correct answer on screen the whole time, as long as you promote it in the order below gives.
Both use two methods of the ProjectionStore trait, reset and commit, plus
the projection runner, run_projection, which reads events from the log and
feeds them to your projection in chunks.
Before you start: what the programs below call
Since 0.5.0, run_projection needs no feature: it is in every build of
happenstance, and covered by semver like the rest of the crate. The apply
method your projection implements is async and is handed a Delivered event
(ADR-0074 says why).
The programs on this page run in memory, using MemoryProjectionStore. The
first drives a real projection with run_projection and names its checkpoint
with checkpoint_id, as an application does. The second calls the trait
directly, because what it shows is a rule the store enforces whoever calls
it.
The rebuilding example runs
every step below with the runner, against SQLite. It is the program to read
alongside this page.
Choose the rebuild
Two facts decide which rebuild you can use.
First: a partly rebuilt view looks current. The runner always records its
checkpoint as Authority::Live, meaning "this view is up to date". It has no
way to say that the rows it is writing are only half rebuilt. If a replay stops
after one chunk of two, the checkpoint still says Live, up to the last chunk
it finished.
The ProjectionStore trait can record a "rebuilding" state
(PS-24), but whether the
runner will ever use it is still undecided. Until it does, anyone reading a view
while it replays sees a partial answer that claims to be current.
So: if nobody reads the view during the replay, or a partial answer does no harm, reset it in place. If not, build under a second id.
Second: a changed fold always needs one of the two. The runner stores the
checkpoint under checkpoint_id(&projection): the projection's name and a
digest of the query it derives from its event types and scope
(PS-25). When the query
changes, so does the id, and the next run starts from the beginning. A
projection that starts reading a new event type therefore sees every older
event of that type.
Nothing notices when what a projection does with its events has changed, though. A changed fold over an unchanged query resumes from its old checkpoint, so choosing a new name, or resetting the view, is up to you.
A checkpoint written before the id was derived sits under the bare name, so
after upgrading every view rebuilds once. To keep a view's position instead,
adopt its old checkpoint before the new runner first runs, and only if its
query did not change. The documentation of checkpoint_id shows how.
Reset the view
A reset clears a view's rows and its checkpoint together, so the next run starts from the beginning.
reset takes a batch, the same kind commit takes, and you fill it with your
own deletes. The store does not know which rows belong to which view, so it
cannot clear them for you.
What you get in return is safety against a crash halfway through. The deletes and the checkpoint change commit together, so a crash cannot leave an empty table whose checkpoint says it is up to date (PS-16). That rule is frozen for 1.0: it would reopen only for a store that cannot clear rows in the same transaction as its checkpoint, and every projection adapter in this workspace can.
A reset touches only the view you name. Every other view in the same store
stays where it was, so you can rebuild one view without pausing the rest
(PS-17). For a view the runner maintains,
name it by checkpoint_id(&projection), not projection.id(): the runner
never writes under the bare name.
The same goes for protecting a view from a reset. On SQLite,
SqliteProjectionStore::protect_from_reset takes the id to protect, and a
view the runner maintains is protected only under checkpoint_id(&projection).
Protecting projection.id() marks a name the runner never writes, and the
reset you meant to refuse goes through
(PS-18).
Reset, not a commit at the first position
After a reset, the checkpoint reads Checkpoint::NeverRun, so the next run
applies every event again, the first one included
(PS-20).
There is a shortcut that looks the same and is not: committing an empty batch at the first position. That records "applied up to position 1", so the next run starts after the first event and never applies it (PS-19).
The program below does both, with the runner. Its projection writes one row per order, keyed by the order, so it keeps nothing in its own fields:
use happenstance::{bytes::Bytes, Codec, CodecError, DomainEvent, EventType};
use happenstance::{Authority, Checkpoint, Chunk, Delivered, Event, EventStore};
use happenstance::{Json, MemoryEventStore, MemoryProjectionBatch, Tags};
use happenstance::{MemoryProjectionStore, MemoryProjectionStoreError};
use happenstance::{Projection, ProjectionId, ProjectionStore, SequencePosition};
use happenstance::{checkpoint_id, run_projection};
type Failure = Box<dyn core::error::Error>;
#[derive(serde::Serialize, serde::Deserialize)]
struct OrderPlaced {
order: String,
}
const ORDER_PLACED: EventType = EventType::from_static("OrderPlaced");
impl DomainEvent for OrderPlaced {
const EVENT_TYPES: &'static [EventType] = &[ORDER_PLACED];
fn event_type(&self) -> EventType {
ORDER_PLACED
}
fn tags(&self) -> Tags {
Tags::empty()
}
fn encode<C: Codec>(&self, codec: &C) -> Result<Bytes, CodecError> {
codec.encode(self)
}
fn decode<C: Codec>(codec: &C, _: &EventType, data: &Bytes) -> Result<Self, CodecError> {
codec.decode(data)
}
}
/// One row per order. A replay writes the same rows whatever it replays from.
struct Orders {
id: ProjectionId,
scope: Tags,
}
impl Projection for Orders {
type Event = OrderPlaced;
type Store = MemoryProjectionStore;
type Error = MemoryProjectionStoreError;
fn id(&self) -> &ProjectionId {
&self.id
}
fn scope(&self) -> &Tags {
&self.scope
}
async fn apply(
&mut self,
event: Delivered<OrderPlaced>,
batch: &mut MemoryProjectionBatch,
) -> Result<(), MemoryProjectionStoreError> {
batch.write(event.into_event().order, 1);
Ok(())
}
}
#[tokio::main]
async fn main() -> Result<(), Failure> {
let log = MemoryEventStore::new();
for order in ["o-1", "o-2", "o-3"] {
let placed = OrderPlaced { order: order.into() };
let event = Event::new("OrderPlaced", Json.encode(&placed)?)?;
log.append(&[event], None).await?;
}
let mut orders = Orders { id: ProjectionId::new("orders")?, scope: Tags::empty() };
// The id the runner keeps this view's checkpoint under.
let key = checkpoint_id(&orders)?;
let chunk = Chunk::of(2.try_into()?);
// A view that has caught up with the log.
let views = MemoryProjectionStore::new();
run_projection(&log, &views, &mut orders, &Json, chunk).await?;
assert_eq!(views.len(), 3);
// The reset: your deletes and the checkpoint, as one unit.
let mut deletes = views.begin().await?;
deletes.delete_all();
views.reset(deletes, &key).await?;
assert!(views.is_empty(), "the rows went with the checkpoint");
assert_eq!(views.checkpoint(&key).await?, Checkpoint::NeverRun);
// The replay applies every event again, the first one included.
let replayed = run_projection(&log, &views, &mut orders, &Json, chunk).await?;
assert_eq!(replayed.applied, 3);
assert_eq!(views.get("o-1"), Some(1));
// WRONG - "start again" spelled as a commit at the first position, here
// on a second store standing in for an emptied view that never ran.
let shortcut = MemoryProjectionStore::new();
let empty = shortcut.begin().await?;
let first = SequencePosition::FIRST;
shortcut.commit(empty, &key, first, Authority::Live).await?;
// END WRONG.
// The count below assumes the log starts at the first position.
let start = log.snapshot().first().ok_or("the log is empty")?.position;
assert_eq!(start, first);
let resumed = run_projection(&log, &shortcut, &mut orders, &Json, chunk).await?;
assert_eq!(resumed.applied, 2);
assert_eq!(shortcut.get("o-1"), None, "the first event is lost");
Ok(())
}
MemoryProjectionStore keeps all its rows in one map, so delete_all is the
only delete it offers. On a real database, the batch carries a DELETE for the
view's own table and nothing else. The example's retire function does exactly
that on SQLite when it prepares a rebuild.
Do not wind the checkpoint back
A checkpoint only moves forward. A commit at an earlier position than the
current checkpoint is refused with CommitError::CheckpointRegression
(PS-22).
That protects you from an old runner left over after a redeploy: it cannot drag a view backwards and apply events a second time. The same guard means you cannot rebuild by moving the checkpoint back over rows you kept. A reset is the only way back:
use happenstance::{Authority, Checkpoint, CommitError, Event, EventStore};
use happenstance::{MemoryEventStore, MemoryProjectionStore, ProjectionId};
use happenstance::ProjectionStore;
const STOCK: ProjectionId = ProjectionId::from_static("stock_on_hand");
#[tokio::main]
async fn main() -> Result<(), Box<dyn core::error::Error>> {
let log = MemoryEventStore::new();
let placed = Event::new("OrderPlaced", &b"{}"[..])?;
log.append(&[placed.clone(), placed], None).await?;
let positions: Vec<_> =
log.snapshot().iter().map(|event| event.position).collect();
let [first, last] = positions[..] else {
return Err("expected two events".into());
};
let views = MemoryProjectionStore::new();
let caught_up = views.begin().await?;
views.commit(caught_up, &STOCK, last, Authority::Live).await?;
// Moving it back is refused: its row is not written, and the checkpoint
// stays where it was.
let mut back = views.begin().await?;
back.write("sku-1", 99);
let back = views.commit(back, &STOCK, first, Authority::Live).await;
assert!(matches!(
back,
Err(CommitError::CheckpointRegression { current, attempted })
if current == last && attempted == first
));
assert_eq!(views.get("sku-1"), None, "the refused row was not written");
let kept = views.checkpoint(&STOCK).await?;
assert_eq!(kept, Checkpoint::Live { through: last });
// A reset is the way back, and after it any position is accepted.
let mut deletes = views.begin().await?;
deletes.delete_all();
views.reset(deletes, &STOCK).await?;
let again = views.begin().await?;
views.commit(again, &STOCK, first, Authority::Live).await?;
let restarted = views.checkpoint(&STOCK).await?;
assert_eq!(restarted, Checkpoint::Live { through: first });
Ok(())
}Replay it
After a reset, call run_projection again with the same projection. It finds
NeverRun and replays every event its query selects, a chunk at a time,
committing the checkpoint after each chunk.
Keep the rows independent of the chunk size
The chunk size changes how often the checkpoint moves. It must not change the rows you end up with (PS-14).
Your projection's part in that is to write everything into the batch, and keep no running totals in the projection's own fields. The runner resumes from the checkpoint, so a total held in memory starts from zero on every run, while the rows in the database do not.
That is why the example's StockOnHand writes a
change against the stored row rather than a total. Its reset_and_rebuild
rebuilds at a chunk size of 1 and of 500, and refuses to finish if the rows
differ.
Run one runner per view
Run exactly one runner per projection id in a given store. Nothing enforces this. Two runners moving forward together each apply the same events, and because neither moves the checkpoint backwards, the regression check above does not catch them. The runner's own documentation says so under Exactly one runner per projection, and the caller owns that.
Build the new version beside the old
When the fold changes, give the new version its own ProjectionId and its own
table, and run it from NeverRun while the old view carries on answering
readers. This is often called a blue/green rebuild.
The two views move forward on their own schedules, so a long backfill never holds the live view back (PS-23). One thing to watch: a reader that joins the two views sees two different moments of the log. The example prints that difference rather than hiding it.
Promote the new view
When the new view has caught up, promote it. Whichever way you switch your readers over, start the same way:
- Stop the old view's runner, and take it off any schedule. The old id is
about to be reset to
NeverRun. A runner still scheduled under it would replay the whole log into whatever table then has the name it writes to.
Then take one of two routes.
Route A: repoint your readers. The new view keeps its own table.
- Point every reader at the new table.
- Retire the old view: reset its id with a batch that deletes its rows or drops its table. Nobody reads it any more, so nobody sees it empty.
Route B: rename the new table into the old one's place. Readers keep using the table name they already know.
- Pause the new view's runner.
- Reset the old id with a batch that carries the swap: a
DROP TABLEon the old table, then anALTER TABLE … RENAME TOgiving the new table the old name. Push each statement on its own, because the batch runs one statement perpush. - Redeploy the new view's runner under its own id, writing to the renamed table.
On SQLite, route B's swap is safe. The batch runs inside the one transaction
reset opens, and SQLite's DROP and ALTER are transactional, so readers see
the old table until the reset commits and the new one after it. A swap that
fails leaves the old table and its checkpoint in place. No conformance test puts
table changes like these in a batch, though, so check that your own database
behaves the same before you rely on it.
Two mistakes to avoid when promoting
Keep the new id, whichever route you took. The checkpoint belongs to the id, not to the table. If you change the new view's id to match the table it now lives in, you throw its resume point away and it replays the whole log.
Never retire the old view with a statement that names a table after the
swap. Once the rename commits, the old name belongs to the new view, and a
DELETE against it clears the rows you just promoted.
The rebuilding example's
a_third_view_added_late takes route B and keeps the new view's id. It first
tries the swap with the new table's name mistyped, and the old view keeps its
rows and its checkpoint. Then it promotes for real. After the promotion the view
applies one new event rather than the whole log.
Its tests check both from
the program's output: a_failed_promotion_leaves_the_old_view_whole and
the_binary_operates_every_view.
When the replay fails
A replay can stop on an event the fold cannot read, such as one missing a field
the fold expects. The runner stops at the first failure. It throws away the
chunk it was building and returns a ProjectionError naming the position where
it stopped. The checkpoint stays at the last chunk it committed
(PS-28).
Fix the projection and run it again. It resumes from that checkpoint and applies nothing twice. That holds only if your fix leaves the fold itself alone. A fix that changes which events the fold reads, or how earlier events fold, is a changed fold: go back to Choose the rebuild.
A failing view stops only itself
Each call to run_projection drives one projection, so a failing view stops
only itself. Your other views keep moving, as long as you keep calling them
(PS-29). The example's
one_poisoned_view breaks one view and runs the other three past it, and
the_poisoned_view_stops_and_the_others_advance checks the output. No
conformance test checks this against the runner yet.
Noticing that a view has stopped is your job. The runner returns the error to whoever called it, and reports it nowhere else.
What you cannot see while it runs
A long replay and a hung one look the same from outside, because the runner
reports progress only once, when it returns. What you can watch is the
checkpoint. It moves once per chunk, and another handle on the same store can
read it with ProjectionStore::checkpoint.
To show how far along a replay is, compare that position with the newest position in the log yourself. The runner's documentation says this under What can be seen while it runs. Nothing in the specification promises it, so treat it as current behaviour rather than a guarantee.
How this page is checked
Both programs above are compiled and run by this repository's tests. The rules they rely on are checked by the projection conformance suite, which every projection store passes:
rebuilding_is_distinguishable_from_live(PS-24)reset_clears_rows_and_checkpoint_together(PS-16)reset_is_scoped_to_one_projection(PS-17)reset_is_not_commit_at_first(PS-19)commit_rejects_a_regressing_position(PS-22)rebuild_is_chunk_size_invariant(PS-14)distinct_projections_advance_independently(PS-23)
The runner's tests cover
the runner itself: first_run_starts_from_the_beginning (PS-20), and
decode_failure_names_its_position_and_rolls_back, whose chunk-of-two case is
the half-built replay that still reads Live (PS-28). The rule that would check
PS-29 against a runner, one_poisoned_projection_does_not_stall_the_others, is
not written yet.