Skip to main content

Module read_stream

Module read_stream 

Available on crate feature event-store only.
Expand description

The pooled-cursor read stream.

§The constraint this type exists to reconcile

Two requirements pull in opposite directions, and neither is negotiable.

EventStore::read is not async: the stream sits at the top level of the return type, because that is what lets the Send flavour mark the stream Send rather than merely the future that produces it (ADR-0001, ADR-0008). So read cannot await, and it cannot acquire a connection.

sqlx’s fetch borrows its executor:

pub fn fetch<'e, 'c: 'e, E>(self, executor: E) -> BoxStream<'e, Result<DB::Row, Error>>
where E: Executor<'c, Database = DB>

The returned stream is tied to a connection that, at read time, does not exist yet. Writing the obvious thing — acquire, then hand the borrow to fetch, then return the result — produces a self-referential value, and the compiler says so:

error[E0597]: `conn` does not live long enough
   |
   |         sqlx::query("SELECT position FROM event").fetch(&mut *conn)
   |         ------------------------------------------------------^^^^-
   |         |                                                     |
   |         |                                                     borrowed value does not live long enough
   |         argument requires that `conn` is borrowed for `'1`
   | }
   | - `conn` dropped here while still borrowed

§The shape that works

Do not hold a borrowed stream at all. Hold the owner and re-borrow it for the duration of each step:

  1. read captures the pool and the query and returns immediately, in state ReadState::Unstarted. Nothing is acquired, so nothing is borrowed.
  2. The first poll_next starts a future that acquires a connection, opens a transaction and DECLAREs a server-side cursor. That future owns the [Transaction], so there is no borrow to outlive.
  3. Every subsequent chunk is a FETCH, run by a future that takes the cursor by value and hands it back with the rows. The &mut borrow of the transaction lives entirely inside one await, which is a borrow the compiler can see the end of.

That is why the state machine is written by hand instead of with futures_util::stream::unfold: unfold would do, but it makes the type unnameable, and this is precisely the type the phase-2 portfolio exists to record.

§A server-side cursor, not keyset pagination

The cheaper-looking alternative — re-issuing WHERE position > $last ORDER BY position LIMIT n per chunk against a connection borrowed only for that statement — sidesteps the borrow problem entirely and is wrong. The specification’s ES-11 requires one read to be evaluated against a single consistent state of the store, fixed no later than the first poll; a self-paginating read grows under the caller’s feet. BEGIN ISOLATION LEVEL REPEATABLE READ plus DECLARE CURSOR buys that snapshot from the server, which is the whole reason the transaction is held open across the stream’s life rather than the connection alone.

The cost is honest and belongs in the capability table: a reader occupies a pooled connection and an open transaction for as long as the caller holds the stream. A slow consumer pins a connection and holds back VACUUM.

§ES-11, and a wrong conclusion that is worth recording

Both read_result_is_stable_under_concurrent_append and query_items_share_one_snapshot pass. They did not always, and the reason they now do is worth writing down, because the first diagnosis was wrong in a way that would have cost a specification amendment.

ES-11 requires a read to be evaluated against a state fixed no later than the first poll. Both rules exploit the gap deliberately: poll once, append, then drain. With the cursor’s opening held as an inline future, that future makes progress only while someone is polling it — so the snapshot was taken during the drain, strictly after the append, and both rules failed five times in five runs.

The conclusion drawn from that was that an async driver cannot satisfy the clause at all, since its first poll can only start the round trip. That was wrong, and consistent failure is what made it look right.

What fixes it is that the work is handed to the runtime at the first poll and every other operation is too. open_cursor is spawned here; append is spawned through PostgresEventStore::on_runtime. Both land in the same queue, the read’s task was enqueued first, and so the snapshot is taken before the append runs. “Fixed no later than the first poll” becomes true in the only sense available to an async adapter: the work was committed to at that poll and no longer depends on the caller polling again.

The bridge was not built for this. It was built because sqlx panics without a runtime in thread-local scope and the conformance suite’s concurrency family drives its contenders on raw threads. That it also settles ES-11 is why the earlier diagnosis survived as long as it did: the two failures looked structural because the thing that would have disproved them had not been written yet.

Structs§

PgReadStream
The stream PostgresEventStore::read returns.