Straum

Straum keeps aggregates of PostgreSQL tables in memory and answers asks about them over HTTP. Counts and sums come back in microseconds.

A dashboard runs the same GROUP BY over the same tables all day, and PostgreSQL reads the rows again each time. Straum reads them once, keeps each group as a cell in memory, and follows every commit. You declare what to keep as a view, in a small subset of PostgreSQL’s CREATE VIEW, here over an events table and the accounts it belongs to:

CREATE VIEW usage WITH (buckets = '1 hour, 1 day, 7 days') AS
SELECT date_bin('1 hour', e.at, timestamptz 'epoch') AS bucket,
       e.kind AS kind,
       e.account_id AS account_id,
       a.region AS region,
       count(*) AS events,
       sum(e.amount) AS amount,
       avg(e.amount) AS avg_amount
FROM public.events AS e
LEFT OUTER JOIN public.accounts AS a ON a.id = e.account_id
GROUP BY 1, 2, 3, 4;

An ask is a SELECT over that view, with every value as a parameter. This one asks for purchases by region on one day:

curl --json '{
  "query": "SELECT region, events, amount FROM usage WHERE kind = $1 AND bucket >= $2 AND bucket < $3 GROUP BY region",
  "parameters": ["purchase", "2026-11-13T00:00:00Z", "2026-11-14T00:00:00Z"]
}' http://localhost:8080/ask

{
  "as_of": "2026-11-14T09:12:40.118204573Z",
  "rows": [
    {"region": "eu", "events": 412, "amount": 18240.50},
    {"region": "us", "events": 655, "amount": 30117.25}
  ]
}

Every commit made before as_of is in the answer, and an ask never touches PostgreSQL. In the benchmark counts and sums answer in 166 to 413 µs where PostgreSQL takes 22.3 to 2,376.2 ms, and a median fare in 32 ms where it takes 6.6 s, from 113 MiB of memory. The target is under 1 ms for every ask. The two percentile asks over every trip miss it.

A table owner runs the SQL straum plan prints, once. straum run loads the tables and follows them. Apps ask over HTTP.

What it is good for

A question asked often, about tables that keep changing, whose answer is counts, sums, averages, minima or maxima by a few keys over time:

• Dashboard tiles and charts: events per hour, amounts by region, the same GROUP BY on every page load.
• Numbers an app shows its own users, such as usage this month or orders today. Make the tenant a key and each ask filters on it.
• A summary you maintain by hand today: a materialized view refreshed on a schedule, or a rollup table kept by triggers. Straum follows inserts, updates and deletes itself, and every answer is exact.

What it is not

Straum is not a SQL engine, a replica, a cache or a durable store. It answers only what its spec declares, so it has no ad hoc queries, no rows and no sketch. It keeps nothing on disk: every start loads every table again, and a crash leaves no slot behind holding WAL. PostgreSQL stays the source of truth.

Straum supports PostgreSQL 17 and newer as shipped, with no extension and no superuser, so managed services work. Where wal_level = logical can’t be had, --wal physical reads the WAL replica writes (setup). The name is Nynorsk for a stream or current: Straum’s input is PostgreSQL’s replication stream.

Warning

Straum is not stable yet. The spec, the asks and the flags may change.

The quick start runs Straum in Docker in four steps, on a month of New York taxi trips. After it, a spec and an ask are the two things you write.

Quick start

Straum against PostgreSQL 18, both in Docker, on one month of New York taxi trips: the TLC’s yellow trips of January 2024, 2,964,624 of them, and the 265 taxi zones. You need Docker, uv and curl, and the download is 50 MB. Four steps end in an answer. Every other page uses these tables and this spec.

1. Start PostgreSQL with the trips

docker network create straum
docker run -d --name straum-pg --network straum \
  -e POSTGRES_PASSWORD=postgres postgres:18 -c wal_level=logical

A few seconds later, load the trips:

uv run https://raw.githubusercontent.com/janbjorge/straum/main/bench/load_taxi_trips.py |
  docker exec -i straum-pg psql -U postgres -q

load_taxi_trips.py downloads the trips into data/ and prints them as SQL: the tables taxi_zones and yellow_trips, then their rows. The TLC’s times are New York wall clock, so it reads them in that time zone. Money and distance are numeric, so sums are exact.

2. Write a spec

Save as trips.sql:

CREATE VIEW trips WITH (buckets = '1 hour, 1 day, 7 days') AS
SELECT date_bin('1 hour', trips.pickup_at, timestamptz 'epoch') AS bucket,
       trips.payment_type AS payment_type,
       trips.pu_location_id AS pu_location_id,
       pickup.borough AS borough,
       count(*) AS trips,
       sum(trips.fare_amount) AS fare,
       avg(trips.fare_amount) AS avg_fare
FROM public.yellow_trips AS trips
LEFT OUTER JOIN public.taxi_zones AS pickup ON pickup.location_id = trips.pu_location_id
GROUP BY 1, 2, 3, 4;

PostgreSQL would give the same rows for this view. Straum never creates it; it keeps one cell per hour, payment type, pickup zone and borough in memory. WITH (buckets = ...) lists every bucket width it keeps: the hour of the date_bin, then days and weeks for long ranges. Writing a spec explains each line.

3. Set up PostgreSQL and run Straum

Straum connects as the user in STRAUM_DSN. straum plan reads it too, and prints the SQL that makes that role with that password, lets it read the tables and publishes them. Run the SQL as the tables’ owner:

export STRAUM_DSN='postgres://straum:straum@straum-pg:5432/postgres?sslmode=disable'
docker run --rm -e STRAUM_DSN -v "$PWD:/spec" ghcr.io/janbjorge/straum \
  plan /spec/trips.sql | docker exec -i straum-pg psql -U postgres

This container has no TLS, hence sslmode=disable. Otherwise Straum always connects with TLS. Then run it:

docker run --rm --name straum --network straum -p 8080:8080 \
  -e STRAUM_DSN -v "$PWD:/spec" ghcr.io/janbjorge/straum \
  run /spec/trips.sql

straum: listening on 0.0.0.0:8080
straum: loaded every table; streaming from 0/1933EF18

The load takes about a second.

4. Ask

In another terminal, ask for trips and fares by pickup borough on Wednesday 17 January, New York time:

curl --json '{
  "query": "SELECT borough, trips, fare, avg_fare FROM trips WHERE bucket >= $1 AND bucket < $2 GROUP BY borough",
  "parameters": ["2024-01-17T00:00:00-05:00", "2024-01-18T00:00:00-05:00"]
}' http://localhost:8080/ask

{
  "as_of": "2026-11-13T07:52:38.699695916Z",
  "rows": [
    {"borough": "Bronx", "trips": 220, "fare": 7196.07, "avg_fare": 32.709409},
    {"borough": "Brooklyn", "trips": 924, "fare": 27585.30, "avg_fare": 29.854221},
    {"borough": "EWR", "trips": 10, "fare": 1001.50, "avg_fare": 100.15},
    {"borough": "Manhattan", "trips": 99225, "fare": 1502063.48, "avg_fare": 15.137954},
    {"borough": "N/A", "trips": 68, "fare": 6436.59, "avg_fare": 94.655735},
    {"borough": "Queens", "trips": 9388, "fare": 454949.34, "avg_fare": 48.460731},
    {"borough": "Staten Island", "trips": 3, "fare": 107.40, "avg_fare": 35.8},
    {"borough": "Unknown", "trips": 527, "fare": 9738.86, "avg_fare": 18.47981}
  ]
}

Every commit made before as_of is in the answer, and every value in an ask is a parameter. Asking explains the query, the parameters and the answer.

Change the data

Add a card trip from Newark Airport at noon, and move JFK and LaGuardia, zones 132 and 138, out of Queens into a borough of their own, then ask for the day again:

docker exec straum-pg psql -U postgres -c "
  INSERT INTO yellow_trips (vendor_id, pickup_at, pu_location_id, do_location_id,
                            payment_type, fare_amount, tip_amount, trip_distance)
  VALUES (2, '2024-01-17 12:00-05', 1, 132, 1, 120.00, 0, 17.50);
  UPDATE taxi_zones SET borough = 'Airports' WHERE location_id IN (132, 138);"
curl --json '{
  "query": "SELECT borough, trips, fare FROM trips WHERE bucket >= $1 AND bucket < $2 GROUP BY borough",
  "parameters": ["2024-01-17T00:00:00-05:00", "2024-01-18T00:00:00-05:00"]
}' http://localhost:8080/ask

{
  "as_of": "2026-11-13T07:53:13.818919375Z",
  "rows": [
    {"borough": "Airports", "trips": 7975, "fare": 409124.98},
    {"borough": "Bronx", "trips": 220, "fare": 7196.07},
    {"borough": "Brooklyn", "trips": 924, "fare": 27585.30},
    {"borough": "EWR", "trips": 11, "fare": 1121.50},
    {"borough": "Manhattan", "trips": 99225, "fare": 1502063.48},
    {"borough": "N/A", "trips": 68, "fare": 6436.59},
    {"borough": "Queens", "trips": 1413, "fare": 45824.36},
    {"borough": "Staten Island", "trips": 3, "fare": 107.40},
    {"borough": "Unknown", "trips": 527, "fare": 9738.86}
  ]
}

The new trip added one to EWR, and the two airports’ trips of the day left Queens for Airports, as PostgreSQL’s view would show them (Dims). Straum follows commits asynchronously, so if the answer’s as_of predates the write, ask once more.

Look around, then clean up

docker exec -it straum /straum shell is a prompt at the running Straum, with Tab completion: \d lists the views with the names and types an ask may use, and \rows trips prints a view’s rows. Running Straum has the shell and the straum status, straum views and straum ask commands beside it.

Stop straum with Ctrl-C and remove PostgreSQL and the network:

docker rm -f straum-pg
docker network rm straum

Your own tables

The same steps work on a database you have:

1. Take the GROUP BY your app runs most and write it as a view: the time column in date_bin, the columns it filters and groups by as keys, its aggregates as measures (Writing a spec).
2. straum plan FILE prints the setup SQL, or what is wrong with the spec. A table owner runs the SQL once. PostgreSQL 17 and 18 also need wal_level = logical, which takes a restart, or --wal physical (Setting up PostgreSQL).
3. straum run FILE with STRAUM_DSN set, then ask.

Start with few keys. Memory follows cells, one per distinct combination of keys and bucket (Cells).

Writing a spec

A spec is a file of CREATE VIEW statements, one per table you aggregate, in a small subset of PostgreSQL’s syntax. PostgreSQL would give the same rows for each view. Straum never creates it, and nothing in the spec runs in PostgreSQL: to load a view, Straum copies only the columns it reads.

Straum SQL has one spelling per meaning and infers nothing (ADR-0029). The synopsis is the whole language: anything it does not show is an error that names the construct and says what to write.

Synopsis

CREATE VIEW view WITH (buckets = 'width, ...') AS
SELECT key AS name, ...
       measure AS name, ...
FROM schema.table AS alias
[ INNER JOIN | LEFT OUTER JOIN
    schema.table AS alias ON alias.column = alias.column [AND ...] ] ...
[ WHERE condition [AND ...] ]
GROUP BY position, ...

key        date_bin('width', alias.column, timestamptz 'epoch')
           alias.column
           alias.column::text
           (alias.column IS [NOT] NULL)

measure    count(*)
           count([DISTINCT] alias.column)
           sum(alias.column)       min(alias.column)
           max(alias.column)       avg(alias.column)
           bool_and(alias.column)  bool_or(alias.column)
           percentile_disc(fraction) WITHIN GROUP (ORDER BY alias.column)
           each of these [FILTER (WHERE condition [AND ...])]

condition  alias.column = literal
           alias.column IN (literal, literal, ...)
           alias.column IS [NOT] NULL
           [NOT] EXISTS (SELECT 1 FROM schema.table AS alias
                         WHERE alias.column = alias.column [AND ...])

width      15 minutes, 1 hour, 1 day: seconds, minutes, hours or days
literal    42, -7, 'text', true, false
fraction   0, 0.5, 0.99, 1: digits from 0 to 1, at most 19 past the point

Brackets hold what may be left out, | separates a choice, and ... repeats what is before it. Keys and measures come in any order. EXISTS goes in the view’s WHERE only, not in a FILTER. A parenthesis is written where the synopsis has one and nowhere else: around a null-ness key, never around a condition or a column. A spec is one or more views, with ; between them. Identifiers are unquoted: lowercase letters, digits and _, at most 63 bytes.

A view over an events table and the accounts it belongs to, with a null-ness key and two filtered measures:

CREATE VIEW usage WITH (buckets = '1 hour, 1 day, 7 days') AS
SELECT date_bin('1 hour', e.at, timestamptz 'epoch') AS bucket,
       e.kind AS kind,
       e.account_id AS account_id,
       a.region AS region,
       (e.device_id IS NOT NULL) AS has_device,
       count(*) AS events,
       sum(e.amount) AS amount,
       avg(e.amount) AS avg_amount,
       count(*) FILTER (WHERE e.kind = 'purchase') AS purchases,
       count(*) FILTER (WHERE e.device_id IS NOT NULL) AS from_devices
FROM public.events AS e
LEFT OUTER JOIN public.accounts AS a ON a.id = e.account_id
GROUP BY 1, 2, 3, 4, 5;

Straum keeps one cell per distinct hour, kind, account, region and has_device, holding what the measures need. An ask merges cells and never reads a row.

The tables

FROM names the fact table, the table whose rows the view counts. A table the view joins is a dim. straum plan and the setup check call them fact and dim.

Each name goes through three layers, and each layer uses only its own:

The schema is never left out, even for public: Straum has no search path, and FROM events is refused with table events must be schema.table. The alias is never left out either, so every column says which table it is from. Setup SQL and start checks print tables as public.events, the way PostgreSQL resolves them.

The fact table needs no primary key. On the logical stream the old row of an update or delete comes from REPLICA IDENTITY FULL, which the setup SQL sets on every spec table. --wal physical rebuilds that row from the page and does not set FULL.

Keys

Every SELECT item that is not a measure is a key, and each cell is one distinct combination of the keys. A key is a column of the fact table (e.kind), a column of a dim (a.region), a null-ness key or the time bucket.

The AS name is what asks use, and it is the only way an ask reaches the item: it says region, never a.region. Names are unique within a view. GROUP BY lists the keys by their positions in SELECT, in order: a position is what PostgreSQL resolves without a lookup, where an output name it may read as an input column.

A key column is an integer type, numeric(p, s), boolean or text, or an enum written ::text. Fewer keys, or keys with fewer values, mean fewer cells, and memory follows cells (Cells).

Time

Every view has exactly one bucket, named bucket:

date_bin('1 hour', e.at, timestamptz 'epoch') AS bucket

• The column is a timestamptz or a timestamp. The origin is 'epoch' of the column’s type, so buckets start on midnight. Straum refuses an origin of the other type, since PostgreSQL would then cast the column through the session zone and the view would not mean what Straum computes.
• The width is at most 1 day and divides a day: 15 minutes does, 7 minutes doesn’t.
• The name is bucket, which is what asks call it, so nothing else in the view may be named bucket.
• A timestamp has no zone, and Straum never gives it one. Its buckets and ask bounds are wall clock, and answers print them with a Z that means nothing.
• A row whose time is null, infinity or -infinity is kept but never answered. Any other value Straum can’t read, such as a NaN, stops the load or the stream with the table and column named.

WITH (buckets = '1 hour, 1 day, 7 days') lists every bucket width Straum keeps, narrowest first, and the first must be the date_bin width. Each wider width is a whole multiple of the one before and starts at the epoch, so 7 days buckets start on Thursdays. An ask over a long range reads its middle from the widest buckets that fit and its ends from narrower ones, so it reads far fewer cells, as drawn under How Straum works. Answers stay the same, and each width costs memory.

Joining a dim

INNER JOIN public.accounts AS a ON a.id = e.account_id
LEFT OUTER JOIN public.accounts AS referrer ON referrer.id = e.referrer_id

A join points from a column of the fact, or of a dim already joined, to the key of a dim. The dim columns the view selects, here a.region AS region, are keys like any other, read as the dim is now: an account moved from eu to us moves all its events to us in the next answer, as PostgreSQL’s view would, and nothing starts over (Dims).

• INNER JOIN keeps a fact row only while a dim row matches it; LEFT OUTER JOIN keeps every fact row, with null dim columns while none matches. A RIGHT or FULL join would add dim rows no fact row points at, whose bucket is null and which no ask reads, so it answers as INNER or LEFT OUTER; write that.

• An ON equality has the joined table’s column on the left. Its right side names the fact’s alias or the alias of a dim joined before it, a chain:

  INNER JOIN public.accounts AS a ON a.id = e.account_id
  LEFT OUTER JOIN public.regions AS r ON r.name = a.region
  

  Straum then holds region in its projection of accounts and reads regions through it. A chain through a semi or anti join is refused: those keep no row to read through.

• Any number of dim rows may match a key. A key the dim holds twice counts the fact row twice, as PostgreSQL’s join does: count(*), count(c) and sum double, min, max, count(DISTINCT ...), bool_and and bool_or don’t, avg stays, and percentile_disc counts the row’s value twice. A dim needs no index on its key.

• The fact column and the dim key column must be of the same kind, column for column.

• A join must select at least one dim column. The fact column joined on, e.account_id, may be selected too or left out; either way it keys the cells, so a view with a join has one cell per distinct join id, bucket and other keys, and a dim with many rows means many cells.

• Two joins can point at one dim table, each reading its own columns, as a and referrer do above. A dim table is one dim however many views join it, and all must join it on the same key columns.

• The join itself has no value: asks group and filter by region or account_id, never by a.

Semi and anti joins

WHERE EXISTS (SELECT 1 FROM public.accounts AS known WHERE known.id = e.account_id)

EXISTS keeps a fact row while at least one dim row matches it, once however many match: here an event whose account is in accounts. NOT EXISTS keeps it while none does. Its equalities are those of ON. No column of the subquery’s table can be selected, filtered on or joined through. Both follow the dim as it is now, like any join.

The view’s WHERE

WHERE e.kind IN ('purchase', 'refund') AND a.region = 'eu'

WHERE drops rows from every answer, as the view’s WHERE does in PostgreSQL. Its conditions are those of a FILTER, below, on columns the view selects as keys, so a condition on a dim column follows the dim. A FILTER that every measure repeats is refused: write it here.

Null-ness keys

(e.device_id IS NOT NULL) AS has_device

A null-ness key is a boolean, true for the rows with a device_id, grouped and filtered like a column under its name. IS NULL makes the opposite key. The column may be of any type Straum reads, a time included, but not the view’s time column: a row with no time is never answered, so the key could never be true. The view body stays what PostgreSQL runs, since the key’s name is no column there.

Enum columns

e.kind::text AS kind

Had kind been an enum, the view would read it so. An enum column is read as its label, the text PostgreSQL prints for it, by writing ::text wherever the view reads the column: on a key, in a null-ness key, (e.kind::text IS NULL), in a FILTER and in WHERE, spelled alike in every place and in every view that reads the column. ::text is the one cast, and it goes nowhere else: not in ON, EXISTS, date_bin or a measure. Straum refuses an enum column written without it, and the cast on a column of any other type, so a spec says what it reads. A FILTER literal must be one of the type’s labels, which Straum reads at start. The label is a text key: asks filter it with a string, and answers sort labels as text, not in the type’s order (ADR-0037). Which types take the cast is listed here; today it is enums alone.

Measures

Nulls are skipped, as in SQL. Sums are exact: Straum adds integers, with numeric values scaled by their column’s scale. count(*) counts rows and count(c) the rows where c is not null. bool_and(c) is true when every c that is not null is true, and bool_or(c) when any is; both are null when no c is (ADR-0041). percentile_disc(f) is the value at rank ceil(f * n) of the n values that are not null, at least the first, and null when there are none; ascending is its one order, so ASC and DESC are not written (Median and percentiles).

FILTER makes a measure count only some rows:

count(*) FILTER (WHERE e.kind = 'purchase') AS purchases
count(*) FILTER (WHERE e.device_id IS NOT NULL) AS from_devices

= and IN take a column the view selects as a key, IS NULL and IS NOT NULL a column with a null-ness key. The second filter is why the view needs the has_device key: Straum keeps nothing else about device_id. A literal must fit its column.

Checking a spec

straum plan FILE prints the setup SQL, or what is wrong and exits

1. Syntax outside the subset comes first, with its line, then every problem between declarations at once:

straum: usage.sql: line 6: JOIN, write INNER JOIN, `JOIN public.accounts AS a ON a.id = e.account_id` is not in Straum SQL, the PostgreSQL subset made only for Straum
usage.sql: line 3: unknown table alias a

straum: usage.sql: usage declared twice
usage.sql: view usage: width of 420s does not divide a day

straum run checks the tables themselves, such as a missing column or a wrong type, when it starts.

Asking

An ask is a JSON object sent to POST /ask: a query and its parameters. The examples ask the usage view of Writing a spec. This one asks for purchases in two regions per hour, from 9:00 to 12:00 UTC:

curl --json '{
  "query": "SELECT bucket, region, events, amount FROM usage WHERE kind = $1 AND region IN ($2, $3) AND bucket >= $4 AND bucket < $5 GROUP BY bucket, region",
  "parameters": ["purchase", "eu", "us", "2026-11-13T09:00:00Z", "2026-11-13T12:00:00Z"]
}' http://localhost:8080/ask

The query

SELECT [key, ...] measure, ...
FROM view
WHERE [condition AND ...] bucket >= $n AND bucket < $n
[GROUP BY key, ...]
[HAVING comparison [AND ...]]

condition   key = $n
            key IN ($n, $n, ...)
comparison  measure { = | <> | < | <= | > | >= } $n

The notation is that of the spec’s synopsis, and this is the whole language: anything it does not show is an error that names the construct and says what to write. The shortest ask:

SELECT events, amount FROM usage WHERE bucket >= $1 AND bucket < $2

• A key or a measure goes by the name the view gives it with AS: region, kind, account_id, amount. a.region is not a name an ask can use; neither is a, the join, nor a table name. bucket is the view’s time bucket, and a key like any other in SELECT and GROUP BY.
• Each name is selected at most once, since the names become the fields of each answer row. GROUP BY names exactly the selected keys.
• bucket >= $n and bucket < $n bound the range, once each, and both are written: an ask says what it reads. They and the conditions come in any order. No condition is put in parentheses.

HAVING

WHERE picks the rows by their keys; HAVING picks the groups by their measures, after Straum has added them up. This asks for the regions with more than 40 purchases in an hour, and their amounts:

SELECT bucket, region, events, amount FROM usage WHERE kind = $1 AND bucket >= $2 AND bucket < $3 GROUP BY bucket, region HAVING events > $4

• A comparison names a measure the ask selects, so the answer shows every value a group was kept for. A key goes in WHERE, a measure in HAVING; each refuses the other and says where it goes.
• Not equal is <>; != is refused. Comparisons join by AND.
• A measure that is null meets no comparison, <> included, as in PostgreSQL, so its group is left out.
• A count, sum, min, max, percentile or average compares with a number parameter, exactly: avg_amount > 44.27 is true of 44.273058, and amount = 1875.7 of 1875.70. Up to 38 digits may follow the point.
• bool_and and bool_or compare with a boolean, by = or <> only.
• An ask without keys answers its one row only if it meets every comparison; otherwise none.

Parameters

Every value is a parameter: a value written into the query, such as region = 'eu', is an error, so a value can never change what the query says. Parameters are numbers, strings or booleans, $1 first. Each must be used, and one can be used more than once. A number with a fraction or exponent is read exactly as written, never as a float: 44.27 is 4427 hundredths.

A parameter must have its column’s type, with no conversion: an integer for an integer column or a numeric(p, 0), a string for text, a boolean for a boolean or a null-ness key. account_id = $1 with "7" is an error, not an answer with no rows. A numeric key with a scale above 0 can’t be filtered on: at scale 2, 1 could mean 1 or 0.01. A HAVING comparison takes a number of any scale.

The bucket bounds are RFC 3339 times, such as "2026-11-13T00:00:00-05:00", or Unix seconds. Both must fall on an edge of the view’s date_bin buckets: Straum never answers for part of a bucket.

The answer

{
  "as_of": "2026-11-14T09:15:02.530117842Z",
  "rows": [
    {"bucket": "2026-11-13T09:00:00Z", "region": "eu", "events": 38, "amount": 1875.70},
    {"bucket": "2026-11-13T09:00:00Z", "region": "us", "events": 61, "amount": 2980.00},
    {"bucket": "2026-11-13T10:00:00Z", "region": "eu", "events": 44, "amount": 1530.20},
    {"bucket": "2026-11-13T10:00:00Z", "region": "us", "events": 57, "amount": 2210.45},
    {"bucket": "2026-11-13T11:00:00Z", "region": "eu", "events": 29, "amount": 980.00},
    {"bucket": "2026-11-13T11:00:00Z", "region": "us", "events": 70, "amount": 3125.60}
  ]
}

• as_of: every commit made before this time is in the answer. It is Straum’s wall time when it last confirmed that it had applied all WAL PostgreSQL had written, so it keeps moving on a quiet database. A transaction is applied whole, so an answer never sees half of one. An answer can predate a client’s commit (Seeing your own write).
• rows: one object per group, keyed by the names in SELECT, sorted by key, nulls first. An ask without keys answers one row even when nothing matches, as SQL does without GROUP BY: counts are 0, the rest null.

timestamptz buckets are their start in UTC. timestamp buckets are wall clock readings formatted with Z. Measures print as the measures table says: 1875.70 for a numeric(12, 2) sum, and 44.273058 for an avg_amount of 18240.50 over 412 events, at four places past the scale. A row whose dim row does not exist has null dim columns, as a LEFT OUTER JOIN gives, until the dim row arrives.

Two headers help debugging. Straum-LSN names the last commit the answer reflects. Server-Timing says where the time went in Straum, in milliseconds: parse, answer, json and total; what a client measures beyond total is the network and the client itself.

As a table

An ask sent with Accept: text/plain is answered in plain text, for people: the same rows as a table, then the row count and as_of, as straum ask prints it. Its errors are plain text too, a line each, under the same statuses. The layout is for reading and may change; a program reads the JSON.

Seeing your own write

Straum-LSN is written as PostgreSQL writes a pg_lsn, such as 0/1A2B3C8. A client that needs its own write in the answer can read pg_current_wal_lsn() on the same connection right after its commit, then ask until the header reaches it:

COMMIT;
SELECT pg_current_wal_lsn();  -- 0/1A2B3C8

SELECT '0/1A2B3D0'::pg_lsn >= '0/1A2B3C8'::pg_lsn;  -- Straum-LSN >= mine

Compare as pg_lsn or as two hexadecimal numbers, never as strings. Once the header is at or past the LSN read, the answer contains the write. Under synchronous_commit = off the commit may not be written yet, so read pg_current_wal_insert_lsn() instead.

The test can only say yes. pg_current_wal_lsn() counts all WAL, including tables Straum doesn’t follow, while the header moves only when Straum applies a commit. On a quiet database the header can stay below the LSN read even though the write is in, so give up after a timeout or fall back to as_of.

Errors

{"errors": ["unknown measure x", "zip is not a key of the view"]}
{"errors": ["line 1: an ask writes both bounds; bucket < $n is missing"]}
{"errors": ["unknown field \"tz\""]}
{"errors": ["line 1: value, use a parameter such as $1, `'eu'` is not in Straum SQL, the PostgreSQL subset made only for Straum"]}
{"errors": ["bound 1794558600 is not on a 3600s bucket edge"]}
{"errors": ["account_id is an integer, so it can't be the text \"7\""]}

Setting up PostgreSQL

Straum supports PostgreSQL 17 and newer, self-hosted or managed; its tests run on 17, 18 and the 19 release candidate. It uses logical replication with pgoutput, a temporary slot and an exported snapshot, with no extension and no superuser. Setup is three steps, done once. Where wal_level = logical can’t be had, Straum can read the physical WAL instead: see Without logical decoding.

1. Turn on logical decoding

PostgreSQL 17 and 18 need wal_level = logical, which takes a restart. On your own server, run ALTER SYSTEM SET wal_level = logical; and restart. From PostgreSQL 19, replica, the default, is enough: a logical slot turns logical decoding on while it exists. On a managed service, set the provider’s parameter and restart there:

2. Run the setup SQL

straum plan FILE prints the SQL that lets Straum read the spec’s tables. With STRAUM_DSN or --dsn-file set, the role is the DSN’s user, and its password goes in as a SCRAM-SHA-256 verifier, so the password itself appears in neither the SQL nor the server’s log. Without a DSN the role is straum and the SQL says how to give it a password. For the usage view of Writing a spec:

-- Straum setup: run once, as an owner of the tables.
-- wal_level takes a restart. On Azure, set the server parameter instead.
ALTER SYSTEM SET wal_level = logical;
CREATE ROLE "straum" LOGIN REPLICATION PASSWORD 'SCRAM-SHA-256$4096:...';
GRANT USAGE ON SCHEMA "public" TO "straum";
GRANT SELECT ON "public"."accounts", "public"."events" TO "straum";
CREATE PUBLICATION "straum" FOR TABLE "public"."accounts", "public"."events";
-- fact usage: Straum needs the old row of every update and delete; inserts cost no more.
ALTER TABLE "public"."events" REPLICA IDENTITY FULL;
-- dim accounts: Straum holds its key and the columns the views read, and follows it.
ALTER TABLE "public"."accounts" REPLICA IDENTITY FULL;

Run it as an owner of the tables, since only an owner can add a table to a publication. Straum never asks for FOR ALL TABLES, which needs a superuser. The setup lives in the catalog and survives restarts.

• Some managed services grant replication through a role of their own, such as rds_replication on Amazon RDS. --role NAME names the role.
• --publication NAME names the publication; give straum run the same name.
• Dims are published and get REPLICA IDENTITY FULL like facts, since Straum holds and follows them.

3. Connect

Straum reads its DSN from STRAUM_DSN or from a file named by --dsn-file, never from the command line, where other users could see it:

postgres://straum:PASSWORD@db.example.com:5432/app

The port defaults to 5432 and the database to the user’s name. Percent-encode reserved characters in the password, such as @ or /. Use the server’s direct endpoint, not a pooler: the replication connection needs a session of its own.

Straum connects with TLS and checks that the server’s certificate chains to a trusted root and names the host. Unlike libpq, it does this for every sslmode but disable:

The built-in roots are Mozilla’s. If your provider signs with a CA of its own, download its bundle and name it with sslrootcert:

postgres://straum:PASSWORD@db.example.com/app?sslrootcert=/etc/straum/ca.pem

What Straum checks at start

Before it loads, Straum compares the server with the spec and reports every problem at once, each with its fix, in its log, /readyz and straum status. It stays not ready and tries again after a wait. Some of the checks:

wal_level is replica; Straum needs logical, which takes a restart
publication straum does not exist; run the SQL straum plan prints
usage: publication straum leaves out public.events: ALTER PUBLICATION straum ADD TABLE public.events
usage: Straum's role can't read public.events: GRANT SELECT ON public.events
usage: column amount of public.events is double precision; Straum needs an integer type or numeric(p, s) with p at most 18 there
usage: column kind of public.events is event_kind, an enum; write kind::text
usage: column region of public.accounts is text, not an enum; drop the ::text
view usage: measure purchases compares kind with 'purchse', which is no label of event_kind
fact usage: Straum needs the whole old row of every update and delete: ALTER TABLE public.events REPLICA IDENTITY FULL
dim accounts: Straum needs the whole old row of every update and delete: ALTER TABLE public.accounts REPLICA IDENTITY FULL
view usage: join a joins account_id (an integer) to id (text); they must read alike

What Straum costs the server

• One replication connection and one slot per running Straum, out of max_wal_senders and max_replication_slots. Managed services often keep some of each for themselves.
• WAL, until Straum confirms it, which it does every second. If the server holds more than --max-retained-wal for the slot (4 GiB by default), Straum starts over, so a Straum that can’t keep up can’t fill the disk.
• Nothing after Straum is gone: the slot is temporary, so PostgreSQL drops it when the connection closes, crashes and deleted pods included. Temporary slots don’t move to a standby either, so after a failover Straum loads again from the new primary.
• A snapshot during each load, which holds back vacuum for the whole database and makes ALTER TABLE on its tables wait; Straum ends it right after the last table loads.
• One plain connection beside the replication connection, for the WAL position every second. Nothing else is read after the load.
• WAL decoding per replication connection. Run one Straum per database; a second needs another slot and load.

Without logical decoding

PostgreSQL 17 and 18 refuse a logical slot under wal_level = replica, the default, and changing it takes a restart. straum run --wal physical needs neither: it streams the physical WAL through a temporary physical slot and rebuilds each change from the pages it holds (ADR-0049). The commits it yields equal the logical stream’s; the tests check that on a server with both.

straum plan FILE --wal physical prints the setup: the role and its grants, with no wal_level change, publication or REPLICA IDENTITY FULL:

-- Straum setup: run once, as an owner of the tables.
-- --wal physical: wal_level replica and full_page_writes on, the defaults.
CREATE ROLE "straum" LOGIN REPLICATION PASSWORD 'SCRAM-SHA-256$4096:...';
GRANT USAGE ON SCHEMA "public" TO "straum";
GRANT SELECT ON "public"."accounts", "public"."events" TO "straum";
-- Lets Straum start without waiting for the next checkpoint; a superuser grants it.
GRANT EXECUTE ON FUNCTION pg_log_standby_snapshot() TO "straum";

The server needs full_page_writes = on and a wal_compression of off, pglz or lz4; the start says so when it isn’t:

full_page_writes is off; --wal physical needs on
wal_compression is zstd; --wal physical reads off, pglz and lz4

A start waits until every transaction open when it began has ended, as a logical slot’s creation does, and for PostgreSQL to log which transactions run. The last checkpoint logs that, so the wait is short, except on a server that just restarted, where pg_log_standby_snapshot() logs it at once if the role may run it. After a minute Straum says what it waits for and tries again:

--wal physical waited 60 s for PostgreSQL to log which transactions run; GRANT EXECUTE ON FUNCTION pg_log_standby_snapshot() TO Straum's role
--wal physical waited 60 s for transactions open before it started to end: xid 7512; a long or prepared one (pg_prepared_xacts) holds the start back
--wal physical waited 60 s for transactions it read the commits of to end: xid 7512; a synchronous standby that doesn't answer holds them

The last one: PostgreSQL logs a commit, waits for its synchronous standbys, then lets snapshots see it. Straum’s load must see every commit it read before its start, so it takes the snapshot again until then.

What it costs beyond the logical stream:

• The stream carries the whole server’s WAL, every database, though Straum reads only its tables’ pages.
• Memory: the rows of every page of the spec’s tables changed since the last checkpoint, and the changes of transactions not yet ended. A page goes at the next checkpoint, since its next change brings it whole. A shorter checkpoint_timeout lowers it and raises the WAL. Past --max-page-memory (1 GiB by default) Straum starts over.

What starts Straum over in physical mode, besides what starts the logical stream over:

• A value of a column the spec reads that PostgreSQL stored out of line in its TOAST table: the page holds only a pointer. Values over about 2 KB go there. Each change to such a row starts Straum over.
• TRUNCATE, VACUUM FULL, CLUSTER and an ALTER TABLE that rewrites one of the spec’s tables: its rows move to a new file. ADD COLUMN, DROP COLUMN and other DDL that keeps the file are followed.

Running Straum

straum is one static binary, run on its own or from the Helm chart:

straum run FILE [--listen ADDRESS] [--wal logical|physical]
                [--publication NAME] [--dsn-file FILE] [--max-lag SECONDS]
                [--max-retained-wal BYTES] [--max-transaction-rows ROWS]
                [--max-page-memory BYTES]
straum plan FILE [--wal logical|physical] [--publication NAME]
                 [--role NAME] [--dsn-file FILE]
straum status [--url URL]
straum views [--url URL]
straum ask [--url URL] QUERY [PARAMETER...]
straum shell [--url URL]
straum version
straum help

straum <command> --help lists the command’s flags and their defaults. straum plan prints the setup SQL.

Getting straum

Each release is an image, ghcr.io/janbjorge/straum:X.Y.Z, and the same binary on its own for Linux on x86-64 and arm64, with no libraries it needs. The image tagged latest is the newest stable release. To install the newest binary on x86-64:

base=https://github.com/janbjorge/straum/releases/latest/download
curl -fL -O "$base/straum-x86_64-unknown-linux-musl" -O "$base/SHA256SUMS"
sha256sum -c --ignore-missing SHA256SUMS
sudo install -m 755 straum-x86_64-unknown-linux-musl /usr/local/bin/straum

gh attestation verify straum-x86_64-unknown-linux-musl --repo janbjorge/straum checks it was built by this repository’s CI, and gh attestation verify oci://ghcr.io/janbjorge/straum:X.Y.Z --repo janbjorge/straum does the same for an image.

straum run

FILE is the spec. Straum loads every table it names, follows their changes and answers asks over HTTP.

It logs to stderr, one line per event. SIGTERM or SIGINT stops it: it stops accepting, finishes the asks in flight and exits, and PostgreSQL drops the slot as the connection closes.

States

Loading turns Ready once every table is in. Ready turns Lagging past --max-lag and back when it catches up. Any failure leads to Waiting, then Loading from a new slot. SIGTERM leads to Stopping.

Straum reads the server’s WAL position every second. Lag is the time since Straum last caught up with a position the server had reached. It can overstate the true delay by up to one second, never understate it.

Any failure leads to Waiting: a lost connection, a change Straum can’t apply, a table altered under it, or too much WAL retained for its slot. Straum closes the connection, which drops the slot, and waits: 1 second after the first failure, twice as long after each failure in a row, up to 60 seconds. Then it loads every table again from a new slot. A load that finishes resets the wait.

straum: PostgreSQL holds 5120.0 MiB of WAL for straum's slot, past the limit of 4096.0 MiB
starting over in 1 s

straum status

Prints what the Straum at --url has done, from its GET /status.

$ straum status
ready:        yes
applied:      0/17B2958, committed 2026-11-09T17:12:57.346371Z
as of:        2026-11-09T17:14:02.118250Z, every commit before it applied
lag:          0.0 s, at most 30 s
WAL retained: 0.0 MiB, at most 4096.0 MiB; PostgreSQL is at 0/17B2958
starts:       1
failures:     0
last failure: none
sealed:       no spill

• ready: yes, or no and why, as /readyz says.
• applied: the last commit applied and when it committed.
• as of: the same as_of answers carry. It keeps moving while the database is quiet.
• lag and WAL retained: against --max-lag and --max-retained-wal.
• starts and failures: loads that finished, and failures, since the process started.
• sealed: no spill. The shipped binary has no spill configuration.

straum views

Prints every view of the Straum at --url and how much of it Straum holds, from its GET /views.

$ straum views
view trips
  buckets:  3600s, 86400s, 604800s
  holds:    173660 cells, 2003-01-01T03:00:00Z to 2024-02-01T06:00:00Z
  keys:     bucket timestamptz, payment_type integer, pu_location_id integer, borough text
  measures: trips, fare, avg_fare

dim public.taxi_zones
  holds:    265 rows

• holds: the cells at the narrowest width, and the bound cells of a view that keeps them. Then the range of its buckets, which are the bounds an ask takes to read everything. A few trips in the month carry a wrong year, hence the range from 2003.
• keys: every name an ask may group or filter by, each with the type a parameter for it must have.
• measures: the names an ask may select after the keys.

Until every table is loaded it prints not ready: and why, and exits with 1.

straum ask

Sends a query to the Straum at --url and prints the answer as a table. The words after the query are the values of $1, $2 and on:

$ straum ask 'SELECT borough, trips, fare, avg_fare
    FROM trips WHERE bucket >= $1 AND bucket < $2
    GROUP BY borough' 2024-01-17T05:00:00Z 2024-01-18T05:00:00Z
borough       | trips | fare       | avg_fare
--------------+-------+------------+----------
Bronx         |   220 |    7196.07 | 32.709409
Brooklyn      |   924 |   27585.30 | 29.854221
EWR           |    10 |    1001.50 |    100.15
Manhattan     | 99225 | 1502063.48 | 15.137954
N/A           |    68 |    6436.59 | 94.655735
Queens        |  9388 |  454949.34 | 48.460731
Staten Island |     3 |     107.40 |      35.8
Unknown       |   527 |    9738.86 |  18.47981
8 rows, as of 2026-11-13T07:52:57.7760555Z

Quote the query with single quotes, so $1 reaches Straum as written. The table holds what the JSON answer holds: values print the same, text without its quotes, a null as null.

A command line holds only text, so a parameter’s type is read from it:

The shell takes single quotes for itself, so the string 7 is written "'7'". An ask Straum can’t answer prints the reply’s first line and each error, and exits with 1:

$ straum ask 'SELECT trips FROM trips WHERE payment_type = $1
    AND bucket >= $2 AND bucket < $3' card 0 3600
straum: http://127.0.0.1:8080 answered "HTTP/1.1 400 Bad Request"
payment_type is an integer, so it can't be the text "card"

straum shell

A prompt at the Straum at --url. Each line is a command:

$ straum shell
Straum at http://127.0.0.1:8080. Tab completes; \? lists the commands, \q leaves.
straum> \rows trips
bucket               | payment_type | pu_location_id | borough       | trips | fare    | avg_fare
---------------------+--------------+----------------+---------------+-------+---------+-----------
2003-01-01T03:00:00Z |            3 |            170 | Manhattan     |     2 |    0.00 |          0
2009-01-01T05:00:00Z |            2 |            138 | Airports      |     1 |   50.60 |       50.6
2009-01-02T04:00:00Z |            2 |            137 | Manhattan     |     1 |    4.40 |        4.4
...
2024-01-01T07:00:00Z |            2 |             87 | Manhattan     |     4 |  104.40 |       26.1
1000 of 173660 rows, as of 2026-11-13T07:58:59.965466167Z
straum> SELECT payment_type, trips FROM trips WHERE bucket >= $1 AND bucket < $2 GROUP BY payment_type; 2003-01-01T03:00:00Z 2024-02-01T06:00:00Z
payment_type | trips
-------------+--------
           0 |  140162
           1 | 2319047
           2 |  439191
           3 |   19597
           4 |   46628
5 rows, as of 2026-11-13T07:59:07.988040041Z

\rows reads every cell of the view, so on a large view it takes as long as an ask over all its buckets.

At a terminal, Tab completes by where the cursor is: keys and measures after SELECT, views after FROM, a whole condition with the next free parameter after WHERE or AND, a selected measure and then a comparison after HAVING, and past the semicolon the start or end of the buckets held for a bound. The ask above is SEL Tab pa Tab , t Tab FR Tab t Tab W Tab bu Tab G Tab Tab Tab. Completion offers what is likely next, not only what is valid: Straum still checks the ask.

The lines entered are kept in $XDG_STATE_HOME/straum/history, ~/.local/state/straum/history unless that variable is set; without a home directory, as in the image, they are not kept. NO_COLOR turns off the dimmed hint of the next parameter.

With its input or its output anywhere but a terminal, the shell prints no prompt and completes nothing, so a script can drive it:

printf '%s\n' '\d' '\rows trips' | straum shell

Answers go to stdout. A line that fails says why on stderr and the next line is read, as in psql; if any failed the shell exits with 1 after the last.

Kubernetes

The Helm chart, deploy/helm/straum, runs Straum as one pod next to the apps that ask it, with the spec in a ConfigMap and the DSN in a Secret. Build the image from the repository’s Dockerfile:

docker build -t registry.example.com/straum:0.0.0 .
docker push registry.example.com/straum:0.0.0

The image holds the static binary and nothing else: no shell, no package manager. Its entrypoint is straum.

Set up PostgreSQL first. Then put the DSN in a Secret, since the chart never holds it, or have your secret store sync one, such as the External Secrets Operator or the Secrets Store CSI driver:

kubectl create secret generic straum-dsn \
  --from-literal=dsn='postgres://straum:PASSWORD@HOST:5432/DATABASE'
helm upgrade --install straum deploy/helm/straum \
  --set image.repository=registry.example.com/straum \
  --set image.tag=0.0.0 \
  --set dsn.existingSecret=straum-dsn \
  --set-file spec=spec.sql
kubectl exec deployment/straum -- /straum status
kubectl exec -it deployment/straum -- /straum shell

The chart makes a Deployment, a ConfigMap with the spec, a ClusterIP Service on port 8080 and a ServiceAccount. For a release named straum, apps in the cluster ask at http://straum:8080/ask.

values.yaml documents these and the rest: the image tag and digest, the Service, the ServiceAccount and pod placement. values.schema.json refuses values Straum can’t take, such as a maxLag of 0.

Readiness is /readyz and liveness is /livez, so a load or lag takes the pod out of the Service but never restarts it. A new spec rolls the pod. The new pod loads every table before the old one stops, so asks keep answering, and PostgreSQL holds two slots until the old pod is gone.

The pod meets the Restricted Pod Security Standard: user 65532, no capabilities, no privilege escalation, a read-only root filesystem and the RuntimeDefault seccomp profile. It mounts no ServiceAccount token, since Straum never calls the Kubernetes API, and its only volume is the spec.

Note

The chart mounts no CA bundle, so a server whose certificate needs sslrootcert can’t be reached through the chart yet.

How Straum works

Straum keeps cells, not rows. This page is the model behind every other page: how the tables get in, what a cell is, how a dim reaches it, how a commit changes cells, and what an ask reads. Design is the same model with the code’s names, for contributors.

Load once, follow every commit

straum run opens a temporary logical replication slot. The slot names a point in the WAL, an LSN called L, and gives a snapshot of the database at L. Straum loads every table the spec names from that snapshot, then streams every commit after L through the slot. No commit is missed between the load and the stream, and none is applied twice.

Every commit up to L comes in with the load, every commit after it through the slot. None comes in both ways, and none in neither.

A load is one binary COPY of the columns the spec names, grouped into cells as the rows come, so Straum never holds a fact table. Nothing of Straum runs in PostgreSQL. The slot is temporary, so PostgreSQL drops it when the connection closes, and nothing is on disk: every start loads every table again.

Cells

A cell is one distinct combination of a view’s keys and its bucket. For the usage view of the introduction that is one cell per hour, kind and account. Each width in WITH (buckets = ...) keeps cells of its own, so a view with hours and days holds every row in two cells.

A row’s time is floored to each width the view keeps, all aligned to 1970-01-01, so it lands in one bucket per width in the same commit.

A cell holds its row count and what each measure needs, always counts and sums, so cells merge and a row can leave one as exactly as it came. Aggregates 101 says why each measure keeps what it does, and how a median reads the counts min and max keep. A FILTER keeps nothing of its own: it names parts of the cell key, so a cell counts whole or not at all.

The bucket and the keys pick the cell, and the row’s count and amount add to it. The row is not kept, so a delete subtracts the same numbers. Each width the view keeps gets the same add.

A row whose time is null or infinite is kept in a bucket no ask reads. A cell goes with its last row.

Memory follows cells. Per width:

cells <= min(rows, distinct key combinations x buckets)

Keys that are nearly unique per row keep nearly a cell per row. Fewer keys, keys with fewer values and wider buckets collapse more rows into each cell. The benchmark has what real tables measured.

Dims

A view that joins a dim keys its cells by the join column, e.account_id, not by the dim column, a.region. Beside the cells Straum holds a projection of the dim: its key and the columns the views read, nothing else, kept current from the stream. An ask reads region through each cell’s account_id, as the account rows are now.

So a dim change moves no cell. An account moved from eu to us moves all its events to us in the next answer, as PostgreSQL’s view would, and nothing starts over. It also means a join to a dim with many rows means many cells, one per distinct join id in the fact.

For a view whose joins all read fact columns, one of them a column that is not a key of the view, Straum also keeps bound cells: the same cells keyed by the dim columns’ values, which are few next to the ids, so an ask reads no projection. A dim change moves the bound cells that name its key. A view that keeps its join columns as keys, as usage does, has none: they would be as many as its cells.

Following

Changes arrive after their transaction commits, and Straum applies the transaction whole, under one write lock, with nothing read from PostgreSQL in between:

• An insert into a fact table adds the row to its cell; one into a dim table adds the row to the projection.
• An update takes the old row out and adds the new one. The logical stream has the old row because every spec table has REPLICA IDENTITY FULL. The physical stream rebuilds that row from the page, so it does not need FULL.
• A delete takes the old row out.
• A truncate empties the view’s cells, or the dim’s projection.

The stream thread hands the writer whole commits and tells PostgreSQL what the writer has applied. The writer holds the write lock for one commit at a time; asks read under the read lock.

Asks share the store with the writer behind a read-write lock, which the writer takes only to apply a commit it has already read. A long ask delays the writer, and that shows as lag. The stream thread reads ahead of the writer, up to 64 commits, and confirms what the writer has applied, so PostgreSQL can recycle WAL.

A change that no longer fits makes Straum start over from a new slot: an update without its old row, a row never added, a dim row it never held, a value it can’t read, a table whose columns now read differently, or a transaction above --max-transaction-rows. The last is on purpose: a transaction waits in memory until its commit, and the new slot’s snapshot is after the big commit, so its rows come in through the bulk COPY instead; the benchmark sets the default. A lost connection or WAL past --max-retained-wal starts Straum over too (States).

The physical stream

With --wal physical there is no logical decoding to send rows (ADR-0049). A temporary physical slot streams the server’s WAL from the last checkpoint’s redo point, and Straum turns it into the same messages the logical slot sends, so one decoder serves both:

• Pages. The first change to a page after a checkpoint carries a full image of it; Straum keeps the rows of each page of the spec’s tables from there and applies the inserts, updates, deletes, pruning and vacuuming that follow, as PostgreSQL’s redo does. A delete names only a slot on a page, and an update may log only the bytes that differ from the old row: the page has the rest.
• Transactions. Changes wait by transaction until its commit record, with its subtransactions’, and go in WAL order; an abort drops them.
• Rows. A row’s bytes become the values the load reads from its binary COPY, then the pgoutput messages: the old row of an update or delete is whole, as under REPLICA IDENTITY FULL.
• The load. Straum waits until every transaction open when the stream began has ended, so each later one is whole in the WAL it reads, then takes the load’s snapshot. Commits the snapshot saw are in the load and dropped; the rest stream. A commit read before that point must be in the load; PostgreSQL logs a commit a moment before snapshots see it, longer while a synchronous standby answers, so Straum takes the snapshot again until it does. A transaction past the row limit that the load saw doesn’t stop the stream. The stream reads on during the load.
• Memory. A page goes at the next checkpoint record whose redo point is past its last change: PostgreSQL logs it whole the next time. So Straum holds the pages changed in about one checkpoint cycle, not the tables, and starts over past --max-page-memory.
• DDL. A commit that changes a table’s catalog entry makes Straum read the table’s columns again, and the decoder sees what changed, as with pgoutput. A table rewritten or truncated starts Straum over.

Answering an ask

This ask reads the purchases of two regions over the hour from 9:00 UTC:

SELECT region, events, amount FROM usage
WHERE kind = $1 AND region IN ($2, $3)
  AND bucket >= $4 AND bucket < $5
GROUP BY region

some cells of usage at 2026-11-13T09:00:00Z, one per hour,
kind and account

  kind      account_id | events   amount
  purchase  7          |      2   220.00   account 7 is eu -> eu
  signup    7          |      1     0.00   kind is not purchase
  purchase  18         |      2    70.00   account 18 is apac, not asked for
  purchase  132        |    101  7111.19   account 132 is us -> us
  refund    132        |     45  2229.10   kind is not purchase
  purchase  138        |    172  7658.10   account 138 is us -> us

accounts, projected: 7 -> eu, 18 -> apac, 132 -> us, 138 -> us, ...

answer: eu 2 220.00, us 273 14769.29

An ask reads the cells of the buckets in its range, adds each to its group and to each measure whose filter holds, keeps the groups that meet its HAVING, and sorts the rows by key. It never touches PostgreSQL and never reads a row, so its cost follows the number of cells it reads.

Each width takes the whole buckets inside the range, and what is left at the ends falls to the next narrower width. An ask grouped by bucket reads the date_bin width only.

Answers don’t change with the widths a view keeps; only how many cells an ask reads does.

Every answer carries as_of, and every commit made before it is in the answer. Following is asynchronous, so an answer can predate a client’s own commit; Asking says how to wait for it.

Aggregates 101

Straum never holds a row. It holds cells, and that asks two things of every aggregate:

1. Can cells merge? An ask reads many cells and merges them into one answer per group.
2. Can a row leave? A DELETE or UPDATE in PostgreSQL takes a row out of a cell, and the cell must then read as if the row had never been there.

Which aggregates Straum has, and what each cell keeps for them, follow from these two questions.

Merging

Take the amounts of two hours of events:

              09:00         10:00                 both hours
amounts       10 20 30      40 50 60 70 500

min           10            40                    10
max           30            500                   500
sum           60            720                   780
count         3             5                     8
avg           20            144                   97.5
median        20            60                    45

Merge the two hours from their own answers and compare with the last column:

• min: the smaller of 10 and 40 is 10. Right.
• sum: 60 and 720 add to 780. Right.
• count: 3 and 5 add to 8. Right. The merge of two counts is a sum, not a count.
• avg: the average of 20 and 144 is 82, not 97.5. Wrong: the second hour has more rows and must weigh more.
• median: the median of 20 and 60 is 40, not 45. Wrong, and no weight fixes it.

Aggregates fall in three classes by what a part must keep for the merge to come out right:

avg is right once each hour keeps its sum and count instead of its average: 780 over 8 is 97.5. A median has no such handful of numbers. To merge medians exactly each part must keep its values, or a count per value.

Removing a row

Delete the event with an amount of 10 from the first hour.

• sum and count subtract: 60 less 10 is 50, 3 less 1 is 2. The cell needs nothing it does not already hold.
• min was 10. The new minimum is 20, and a cell that kept only “10” has no way to know that. PostgreSQL would scan the rows again; Straum has no rows.

So a second split, apart from the first: an aggregate whose state can subtract takes a delete, and one that keeps only the winner does not. min and max merge well and delete badly.

Straum’s way out is to keep more than the answer. For min and max a cell keeps a count per value, 10: 1, 20: 1, 30: 1. Deleting 10 drops its count to zero and 20 is the smallest value left. The same count per value is how count(DISTINCT c) knows when a value’s last row has gone, and bool_and and bool_or keep a count of true and a count of false for the same reason.

What a cell keeps

Every state is a count or a sum, so every one both merges and takes a delete, and every answer is exact. Sums are integers for that reason: a float sum gives a different answer by the order it was added in and can’t be subtracted from exactly, so Straum refuses float columns.

Two things follow that a spec’s author can see:

• Memory. The first three rows cost a few numbers a cell. min, max, percentile_disc and count(DISTINCT c) cost by the distinct values in each cell; a percentile beside a min or max on its column costs nothing more. A min or max keeps a value in as few bytes as its cell’s values need: max(fare_amount) per hour and dropoff zone keeps 14,943,826 fares of the taxi trips in 76 MiB. A count(DISTINCT c) over a column with many values holds every one of them.
• Rows counted twice. A join whose dim holds a key twice counts the fact row twice (Joining a dim). count and sum double, because adding a row twice adds it twice, and percentile_disc counts the row’s value twice, as PostgreSQL does over the joined rows. min, max, count(DISTINCT c), bool_and and bool_or stay, because the same value merged twice changes nothing.

Median and percentiles

An exact median needs each cell’s values or a count per value, and a min or max already keeps the counts. So Straum has percentile_disc, exact and following deletes:

CREATE VIEW usage WITH (buckets = '1 hour, 1 day') AS
SELECT date_bin('1 hour', e.at, timestamptz 'epoch') AS bucket,
       e.region AS region,
       max(e.amount) AS top_amount,
       percentile_disc(0.5) WITHIN GROUP (ORDER BY e.amount) AS median_amount,
       percentile_disc(0.99) WITHIN GROUP (ORDER BY e.amount) AS p99_amount
FROM public.events AS e
GROUP BY 1, 2;

max and both percentiles read one count per value of e.amount. Take the two hours above. An ask over both gathers the counts of their cells, 10 20 30 40 50 60 70 500, and counts up to rank ceil(fraction * n): the median is the 4th of 8, 40, and p99 the 8th, 500. That 40 is a value the rows hold, where the 45 in the table above, which percentile_cont would give, lies between two. Delete the 500 and p99 is the 7th of 7, 70.

The rank is taken in float8, as PostgreSQL takes it, so the answer is PostgreSQL’s to the row: 0.07 of 100 values is the 8th there, not the 7th. Nulls are skipped, no values give null, and the answer keeps the column’s type.

Where PostgreSQL takes more, Straum refuses the spec:

• Column: an integer type or numeric(p, s), p at most 18; not text, times or floats.
• Fraction: digits from 0 to 1, such as 0.5; not .5, 5e-1, a parameter or an expression.
• One fraction: no array; write one measure per fraction.
• Order: ORDER BY alias.column only; no DESC, USING, NULLS or expression.
• Others: no percentile_cont, mode or hypothetical-set aggregates.
• FILTER: the conditions every measure takes.

The cost is in the ask. A sum merges a number a cell; a percentile gathers every value of every cell in its range and finds the one at its rank, splitting the values left at their middle one each step without sorting them all. An ask over a long range of many values takes milliseconds where the others take microseconds: the median fare over the six months of benchmark trips takes about 30 ms.

Other systems answer with a sketch: a small summary that merges and gives an approximate answer, t-digest, KLL and DDSketch for percentiles, HyperLogLog for distinct counts. The common sketches merge but can’t take a delete, and their answers are estimates. Straum’s answers are exact and follow deletes, so it has no sketch (ADR-0048).

The sources behind this page, and how PostgreSQL and other systems handle the same two questions, are in AGGREGATES.md.

Benchmark

Straum’s benchmark sends Straum and PostgreSQL the same asks about six months of New York taxi trips, checks that the answers are equal, and times both. Memory at the end is what two real tables measured.

The data is the TLC yellow trips of January to June 2024, 20,332,093 trips, and the 265 rows of taxi_zone_lookup.csv.

Results

Ten fixed asks over the benchmark’s two views, each sent to PostgreSQL as SQL over the raw tables and to Straum as an ask over HTTP. One run on an Apple M2 Pro with 16 GB, on 2026-11-15: PostgreSQL 18.6 in Docker Desktop with 10 CPUs, and Straum’s release build on the same machine. Each time is the median over at least a second, from one Python client: asyncpg for PostgreSQL, aiohttp and orjson for Straum. Rows are the answer’s.

The target is under 1 ms for every ask (ADR-0050); the two percentile asks over every trip miss it.

Straum was ready 16.8 s after it started and held 113 MiB, the 265 zone rows’ projection included. In PostgreSQL the table and its indexes take 2,475 MiB.

• Reads run with PostgreSQL’s defaults, two parallel workers per query, except shared_buffers: 2 GB. The rest of the table comes from the OS cache. The other settings in bench/taxi.py only speed the load. yellow_trips has an index on pickup_at, which the week and day asks use, and the gap is smallest there, where PostgreSQL reads least.
• PostgreSQL’s time follows the rows it reads: every trip, but for the week and day asks, which read a week’s or a day’s. Straum’s hardly changes from ask to ask: most of it is the HTTP round trip, reading the query and writing JSON.
• The percentile asks are the exception: Straum gathers every fare of the cells in range, so their time follows the fares the range holds, as Median and percentiles says.
• Straum answers only what the views keep: these keys, on hour edges. PostgreSQL answers any query over the raw rows. Straum keeps its cells current as trips change; a summary table in PostgreSQL would have to be rebuilt or maintained.

Running it yourself

bench/taxi.py

bench/taxi.py shell sets up the same, then leaves you in straum shell at the trips in place of running the asks. Leaving the shell removes PostgreSQL.

It needs Docker, a Rust toolchain and uv, which fetches the script’s Python packages. The first run downloads the six months into bench/data/, which git ignores. Then it builds Straum, starts a throwaway postgres:18, loads the months with bench/load_taxi_trips.py, runs the SQL straum plan prints and starts straum run. Each ask goes to PostgreSQL as SQL over the raw tables and to Straum as an ask over HTTP, and a run stops at the first answer that differs. A run takes about 80 seconds once the data is there.

bench/taxi.py transactions does the same setup, then one transaction of 1,000 to 1,000,000 changed rows while a client asks Straum back to back, under Transactions.

Each mode takes --wal physical to have Straum read the physical WAL in place of logical decoding. It turns full_page_writes back on after the load and checkpoints, since the physical stream needs page images.

The views

bench/trips.sql opens with trips, the quick start’s view with a median and a p99 fare added: what a dashboard over these trips would declare. dropoffs is the other end of the same trips: tips, the average distance and fares per hour and dropoff zone.

CREATE VIEW dropoffs WITH (buckets = '1 hour, 1 day, 7 days') AS
SELECT date_bin('1 hour', dropoffs.pickup_at, timestamptz 'epoch') AS bucket,
       dropoffs.do_location_id AS do_location_id,
       dropoff.borough AS borough,
       count(*) AS trips,
       sum(dropoffs.tip_amount) AS tips,
       avg(dropoffs.trip_distance) AS avg_distance,
       sum(dropoffs.fare_amount) AS fare
FROM public.yellow_trips AS dropoffs
LEFT OUTER JOIN public.taxi_zones AS dropoff ON dropoff.location_id = dropoffs.do_location_id
GROUP BY 1, 2, 3;

Over the 20,332,093 trips, trips keeps 1,157,581 cells at its hour width and dropoffs 713,457. Neither keeps bound cells: each joins on one of its own keys, so they would be as many again.

The asks

PostgreSQL gets the query anyone would write over the raw tables. The session’s time zone is New York’s, as the TLC’s times are.

-- Trips
SELECT count(*) FROM yellow_trips;

-- Card trips
SELECT count(*) FROM yellow_trips WHERE payment_type = 1;

-- Trips by pickup borough
SELECT z.borough, count(*)
  FROM yellow_trips t
  LEFT JOIN taxi_zones z ON z.location_id = t.pu_location_id
 GROUP BY z.borough;

-- Fares by payment type
SELECT payment_type, count(*), sum(fare_amount), avg(fare_amount)
  FROM yellow_trips
 GROUP BY payment_type;

-- Median fare
SELECT percentile_disc(0.5) WITHIN GROUP (ORDER BY fare_amount) FROM yellow_trips;

-- p99 fare by payment type
SELECT payment_type,
       percentile_disc(0.99) WITHIN GROUP (ORDER BY fare_amount)
  FROM yellow_trips
 GROUP BY payment_type;

-- Median fare per hour, one day
SELECT date_trunc('hour', pickup_at),
       percentile_disc(0.5) WITHIN GROUP (ORDER BY fare_amount)
  FROM yellow_trips
 WHERE pickup_at >= '2024-01-17' AND pickup_at < '2024-01-18'
 GROUP BY 1;

-- Manhattan pickups, one week
SELECT count(*)
  FROM yellow_trips t
  JOIN taxi_zones z ON z.location_id = t.pu_location_id
 WHERE z.borough = 'Manhattan'
   AND t.pickup_at >= '2024-01-08' AND t.pickup_at < '2024-01-15';

-- Trips per hour, one day
SELECT date_trunc('hour', pickup_at), count(*)
  FROM yellow_trips
 WHERE pickup_at >= '2024-01-17' AND pickup_at < '2024-01-18'
 GROUP BY 1;

-- Tips by dropoff borough
SELECT z.borough, count(*), sum(t.tip_amount), avg(t.trip_distance),
       sum(t.fare_amount)
  FROM yellow_trips t
  LEFT JOIN taxi_zones z ON z.location_id = t.do_location_id
 GROUP BY z.borough;

Straum gets the same asks over the views, with every value a parameter; bench/taxi.py holds both. Where the SQL has no range, the ask’s is 2000 to 2030, which holds every trip:

-- Trips by pickup borough
SELECT borough, trips FROM trips
 WHERE bucket >= $1 AND bucket < $2
 GROUP BY borough

Transactions

Straum holds a transaction until its commit and applies it whole under its write lock, so its rows wait in memory and every ask waits for the apply. bench/taxi.py transactions updates the fare of the first N trips, which both views follow, in one transaction while a client asks for one day of hour buckets as fast as the answers come. It measures three things: the time from the commit until Straum has applied it, the longest ask meanwhile, which is the lock hold as a client feels it, and how much Straum’s resident set grew. One run on 2026-11-14, the machine, build and PostgreSQL of the Results; Straum held 114 MiB after the load and an idle ask took 177 µs:

Most of the time goes to PostgreSQL decoding the transaction and Straum reading it, outside the lock; the lock itself is held for about 1.1 ms per 1,000 rows, half of it moving each changed fare between the percentiles’ value counts. Memory is what grows: the rows of a transaction wait in Straum until its commit, about 630 bytes each, so a million-row transaction holds over five times what both views do. That sets the default of --max-transaction-rows at 100,000: a transaction at the limit holds the lock for 108 ms, and anything larger is cheaper to start over, which takes the rows through the bulk COPY instead.

Physical WAL

cargo test -p straum-pg --release physical_wal_bench -- --ignored --nocapture

It measures how fast Straum reads physical WAL. It loads the taxi month with bench/load_taxi_trips.py into a PostgreSQL under wal_level = replica, so it needs uv. Then it takes a checkpoint and opens a physical slot. Four phases each change 1,000,000 trips of yellow_trips: COPY in transactions of 250,000 rows, then INSERT, UPDATE and DELETE in transactions of 10,000. The slot streams the WAL, and Straum keeps it in memory. Straum then reads it again three times each way (STRAUM_BENCH_RUNS), with no socket in the way, and the fastest read counts. “Records read” splits pages into records only. “Pipeline read” goes all the way to the commits for crates/straum-notation/tests/taxi.sql. Every row changed must come out, and nothing may be unfit. Autovacuum is off and the checkpoint interval is an hour, so each run writes the same records. STRAUM_TEST_POSTGRES picks the major.

One run on 2026-11-15, the machine of the Results, one thread, PostgreSQL 18:

“Against PostgreSQL” divides the time PostgreSQL took to write a phase’s WAL by the time the pipeline took to read it. Below 1x, as for the deletes, a writer that keeps that pace for long leaves Straum behind, and once the lag passes --max-lag, /readyz turns off. Images are the full-page images: the first change to a page after the checkpoint brings one. The pipeline held at most 429.5 MiB of rows, most of them the open COPY transactions’ rows, which wait for their commit.

Memory

Memory follows cells, as How Straum works says. The rollup key, the columns in GROUP BY, decides how many there are. Two runs on 2026-11-11 measure it: the taxi month under a wide key, and a warehouse’s tables under a narrow one whose dims carry the ids. The resident set came to 177 and 104 bytes a cell, measures and projections included, before bound cells. A cell has taken fewer bytes since (ADR-0044): the benchmark’s two views held 65 MiB for their 2,078,586 cells, 33 bytes a cell, before trips kept its fares for the percentiles; with them they hold 113 MiB.

The taxi month under a wide key

cargo test -p straum-pg --release taxi_month -- --ignored --nocapture

It loads the month with bench/load_taxi_trips.py, so it needs uv, and reads it with crates/straum-notation/tests/taxi.sql, a wider view than the benchmark’s: 15 minute buckets with day and week widths, keyed by payment type, vendor, pickup zone, dropoff zone and the pickup zone’s borough, zone and service zone, with two filtered counts. The test compares asks with the view body in PostgreSQL six times: after the load, after 200 streamed commits, after an update, after a zone is renamed, after a zone moves to a new key, and after a trip in the renamed zone changes. Nothing starts over. Release build, Docker on the same laptop:

The ask over every key reads every 15 minute cell, maps each through the zones’ projection and answers nearly as many groups, so it is only 2.0 to 2.4 times faster than the view in PostgreSQL. Cells per width, each a count(DISTINCT ...) of that key and the bucket over the month:

Pickup and dropoff zone together are nearly unique per trip, so this spec collapses little at 15 minutes. Without the dropoff zone it collapses 5.2 times.

A warehouse’s tables

Tasks keyed by organization, whether the task is live, its group’s category and its storage location’s type, over hour buckets with a day width beside them. The category and the type are columns of two dims, so the cells are keyed by the group and location ids and Straum holds each dim’s projection beside them: its key and the one column the view reads. The tables are a copy of a production database and are not in the repository: 13,116,878 tasks, 5,690,651 task groups and 851,964 storage locations, 2,645 MiB of tasks in PostgreSQL. The machine and build of the results.

A group holds one task or two, and every task has its own hour, so both widths hold about a cell a task; the day width collapses nothing, since the group id is in its key. Keyed by the category and the type instead, as a view that never followed a dim change could be, the widths would hold 311,969 and 66,107 cells.

Straum was ready 10 s after it started and held 2,378 MiB: the cells and the two projections, 6,542,615 dim rows. A run right before held 1,998 MiB; 88 to 104 bytes a cell, projections included.

Every ask equalled PostgreSQL after the load and after each streamed change. Each change is one transaction; the time runs from the commit until Straum had applied it:

The last three change dims. Each is applied to the projection, no cell moves, and the next ask reads the new category or type; Straum never started over.

The asks, PostgreSQL run twice on the loaded tables and Straum the better of two:

PostgreSQL has an index on completed here, and these are the plain queries the asks mirror, not a dashboard’s own. Straum reads the cells of one width in the range, about a cell a task, and maps each through the projections, so an ask over a week costs more than one over a day.

Design

How Straum works is the model in the guide’s words. This page is the same model for contributors: who takes part, what each use case does, how the crates depend on each other, what the code holds true, and which decision shaped what. A name in code font, such as Writer::start, is the type or function that holds a rule, under crates/; the glossary gives each name its type in the code.

Four participants:

• PostgreSQL: the source of truth. Passive: it publishes the stream and serves the snapshot loads. Nothing of Straum runs in it.
• Straum (straum run): active toward PostgreSQL, reactive toward clients. One writer thread opens the slot, loads the tables and applies every commit; tokio threads answer asks meanwhile.
• Clients: the apps that POST /ask. They never see a table, a join or a row, only the names a view gives its keys and measures.
• People: a table owner, who runs the setup SQL straum plan prints once, and an operator, who runs the binary and reads /readyz, /status, /views and the log. In Kubernetes the chart is the operator.

Architecture

Six crates, dependencies inward only, enforced by scripts/check-layers.py.

The binary depends on the three adapters. straum-pg and straum-http depend on the use cases in straum-app; straum-notation needs only the domain.

• straum-domain has no dependencies and no I/O. Everything an ask or a change needs is a plain struct, enum or function.
• straum-app holds the use cases and the two ports. Source is the one trait PostgreSQL sits behind; Spill is where sealed buckets go. No adapter for Spill ships; the binary passes None.
• The adapters never meet: straum-http is handed a query reader by the binary (ReadAsk in straum-http), so HTTP never knows Straum SQL.

Data moves the other way at run time: PostgreSQL feeds straum-pg, which hands commits to the writer, which applies them to the store; apps post asks to straum-http, which reads the store through the app.

Use cases

Run Straum is run in the binary: read the spec and the DSN, bind the listener, make an App with an empty Store and the Limits, make a Writer over a Postgres source, spawn the writer thread, serve HTTP until a signal. A panic aborts the process (the hook run sets), since the store may be half written. Start over is Writer::step and run_writer: the failure is recorded and Following removed, so asks answer 503; the thread sleeps the backoff; the next step is a start, whose open drops the old stream first, and PostgreSQL drops its slot as the connection closes.

Start

Writer::start:

1. Drop the old store and empty the spill, so two stores never take memory at once.
2. Source::open: connect, begin a read-only repeatable-read transaction, create a temporary logical slot with pgoutput and SNAPSHOT 'use', read L. Read the catalog and check it against the spec.
3. Source::load each dim, then each fact: set the column types from the catalog, run one binary COPY of the columns the spec names, add rows by the batch (Store::add_rows). The snapshot transaction commits with the last load.
4. Read the server’s WAL once; start the Lag from when the slot was opened.
5. Take the write lock, swap the store in, set Following at L, count the start. Straum is ready.
6. Run a seal pass. The stream thread starts at the first next_commit: it reads the stream, answers the server’s keepalives and confirms what the writer has applied, so the server can recycle WAL.

Any error is a Failure, and the next step is a start over. The load fills a new store outside the lock, so asks never wait on PostgreSQL; they answer 503 until step 5.

With --wal physical, step 2 differs (ADR-0049):

1. Read the catalog on a plain connection and check it.
2. Create a temporary physical slot; it reserves WAL from the last checkpoint’s redo point. A checkpoint in progress may set that past the WAL flushed, so wait for pg_current_wal_flush_lsn() to reach it: the server streams only flushed WAL. Read each spec table’s Layout (physical::read_layouts): its file, its attributes, dropped ones too, and its enums’ labels by OID. After the slot, so a rewrite between the two is in the WAL read. Start the stream thread at once, with a Pipeline (wal/pipeline.rs): Records cut the bytes into records, Heap keeps the pages, Transactions holds changes by xid, Publisher turns commits into pgoutput messages for the same Decoder.
3. Try pg_log_standby_snapshot(), then wait until the stream is at its Consistent point: past the first RUNNING_XACTS and every transaction it names.
4. Begin the repeatable-read transaction, read pg_current_snapshot() and the WAL position, check the catalog again, and hand the snapshot to the stream thread (Stream::see). Commits it saw are dropped. L is the LSN before the first commit it didn’t see. PostgreSQL logs a commit before snapshots see it, and waits for synchronous standbys in between, so a snapshot may miss a commit the stream dropped as before its start. The thread then answers Sight::Blind, and Straum rolls back and takes another.

The thread reads on through the load, so the walsender never waits on a reply. The pure parts, wal/ and the pipeline, read no socket: a table’s layout read again after DDL, with its enums’ labels, comes in through a closure.

The store

What the store holds, mutable, one per process, owned by App.

One keyed rollup per fact, a bound rollup for a fact whose joins all read fact columns, one projection per dim. A rollup is one level per width; a hot bucket is columnar cells with one state per input.

• A Rollup is every cell of one fact: one Level per width, the date_bin width first, and one Codes per rollup-key column. The bucket is not coded; it keys the level’s map. Rows without a time go under None (Level::bucket).
• A Bucket is Hot, holding Cells, or Sealed, holding only its cell count. Only date_bin buckets are sealed (Rollup::sealable).
• Cells are columnar: keys, row counts and one State per input column, side by side (Cells). A column of counts or sums is as wide as its widest entry needs, and none at all while every entry is zero (Counts, Sums, ADR-0044). An Index finds a cell by its key codes: a table of positions, the keys themselves staying in their column. A cell goes with its last row (Rollup::remove). Aggregate::state_kind picks which State a measure reads, so measures over one column share it.
• A Projection maps key values to the attribute codes of every row under them. Null is code 0 in every attribute, so no row and a null read alike.
• Bound cells are a second rollup of a fact whose joins all read fact columns, one of them a column no key reads: keyed by the keys and the dim columns’ values, so they are fewer than the fact’s cells and an ask reads no projection. A fact row goes into both rollups; into the bound one once per way its joins resolve now (Bound::rows). A change to a dim the fact reads moves the ones whose join columns name the changed key (Store::retarget); a dim truncate or load makes them again from the keyed cells (Store::rebuild_dim). They hold the hot buckets only: a seal takes its bucket out of them and an unseal binds it back.

Rollup::add puts a row in its cells, here the row of Cells:

The key values become codes, the time picks a bucket per level, and the bucket’s index finds the cell, which adds the row.

1. Each key value gets the code of its column’s first add of it (Codes::code), so a cell key is a few integers.
2. Each level floors the time to its width (Level::bucket) and finds the bucket in its BTreeMap, making a hot one if there is none.
3. The bucket’s Index hashes the codes to a home slot and walks on until the cell with those keys, or appends a new cell (Cells::entry).
4. The cell adds the row: one more row, and each State its input.

Steps 2 to 4 repeat for every level.

Applying a commit

Writer::apply_next, one step:

1. If the WAL was read a second or more ago: read it on the plain connection, fail past --max-retained-wal, feed the Lag, run a seal pass.
2. Source::next_commit, waiting at most until the next WAL reading is due.
3. Unseal every sealed bucket a fact row of the commit falls in.
4. Under the write lock, apply the commit: each change to its rollup or projection, tables the spec doesn’t name skipped, then move Following to the commit’s LSN and time.
5. Forget the sealed buckets of a truncated fact in the spill.

A change to a dim a view reads moves its bound cells: each keyed cell whose join columns name the changed key leaves the bound cells it was in and goes into the ones the dim gives it now.

The stream thread reads ahead of the writer, up to 64 commits, and answers keepalives while the writer is busy (StreamThread).

Answering an ask

The HTTP handler in straum-http (routes.rs) and App::ask:

1. Read at most 1 MiB of body; parse the query with the parameters bound to $n; refuse a range jiff can’t print.
2. On a blocking thread, take the read lock.
3. Check the ask against the spec and refuse any ask it can’t answer exactly; name the sealed buckets it reads (Store::sealed_read). An invalid ask is 400 even while loading. Once a load has read the column types, a filter value of another type than its column is refused too.
4. Without a Following, 503 not ready.
5. Get each sealed bucket from the spill and thaw it against the store; a bucket that can’t be read is 500.
6. Store::ask_with: read the cells of the buckets in the range, the widest the view keeps that fit and narrower ones at the ends (Rollup::cover). A key the ask names is in the cell key or is a dim’s column: while the view’s bound cells hold, a dim column is a part of their key, else it is read from the dim’s projection through the join id in the cell key, as the dim is now, and a cell whose join keeps no row is passed over. Add each cell to its group, and to each measure whose filter holds; filters name only parts of the cell key, so a cell counts whole or not at all. Keep the groups whose measures meet every HAVING comparison, exactly, a null meeting none. Sort rows by key.
7. Answer with as_of from the Lag and the applied LSN in the Straum-LSN header, Cache-Control: no-store: JSON, or a table in plain text for an ask sent with Accept: text/plain.

GET /views takes the read lock once and reports what the store holds (App::held): per view its widths, cells, range of buckets, keys with the types the load set, and measures; per dim its rows. straum views and straum ask print what the server wrote, as straum status does.

GET /views/NAME answers an ask the app builds (App::rows_ask): every measure by the bucket and every key over the buckets the view holds, through the same steps as any ask, written as a table of at most 1000 rows. straum shell is a client of these endpoints: it reads the views as JSON to complete names and knows no Straum SQL.

In the binary, main.rs reads the command line and wires the crates together. The commands that talk to a running Straum go through client.rs, one request on a plain socket. complete.rs is what Tab offers, plain functions over a line and the views, and shell.rs is the prompt around both, the only code that touches the terminal.

Model

Arrows denote dependency, not communication flow.

The spec: what Straum keeps

Immutable once Spec::new has checked it. Every node is a value object compared by value.

A spec is dims and facts. A fact is one view: its table, one time, by keys, refs in join order, measures and the view’s WHERE. A ref joins through one via per key column and reads attributes.

An Aggregate is count, count of a column, sum, min, max, avg, count distinct, bool_and, bool_or or percentile_disc; all but count name one input column (Aggregate::input).

• A Spec holds dims and facts in one namespace (Spec::new). Spec::tables lists each table once, dims first.
• One Dim per table, whatever the number of views joining it; Dim::attributes is the union of what they read.
• A Fact holds its refs in join order: a ref joined through another dim comes after it (Via::Attribute::through is an index into Fact::refs before it).
• A Ref points at a dim by name, with as many Via columns as the dim’s key has (JoinError::KeyLength). Left and inner refs read at least one Attribute; semi and anti read none (JoinError::ReadsNothing, JoinError::ReadsColumns).
• Keys, attributes and measures share one namespace per view, and nothing may be named bucket (SpecError::NamedBucket). An ask reaches any of them by that name alone (Fact::resolve).
• Width divides a day; each wider width is a whole multiple of the one before (SpecError::BadWidth, SpecError::BadBucket).

Derived from a fact, not stored:

The ask and the answer

An Ask names one fact and, by name, its measures, keys and conditions; its range is two bucket edges with from < to. An Answer is one column per key and measure and one row per group.

• Spec::check_ask resolves names, refuses two time keys, a name asked twice, an empty range and a bound off a bucket edge. The store adds the type check once a load has set the types (AskError::WrongType).
• Answered wraps an answer with the applied LSN and as_of. The domain never knows time or LSNs beyond the bucket.

The running system

A writer owns the source port and an app. The app holds the limits, a spilling with the spill port when one is configured, and a state behind a read-write lock: the store, a following while every table is loaded, and the last failure.

• App is an Arc<RwLock<State>> beside the limits and the spilling, which never change; clones share it. Asks take the read lock, the writer the write lock only to apply a commit it has read or to swap a store in (State::apply, Writer::start).
• Following exists only while every table is loaded; a failure removes it (Writer::step). Its absence is “not ready”.
• Lag keeps the WAL readings the slot hasn’t confirmed, at most an hour of them, and the instant it last had; as_of and readiness derive from it.
• A Commit is one transaction, whole, with the end of its commit record as LSN. The stream delivers nothing before the commit, and the store sees it as one.

Invariants

What the code holds true, and where. Planting a bug in any of these makes a property test fail.

1. A commit applies whole. The decoder holds a transaction until its Commit message (Decoder), and the store takes it under one write lock (State::apply). An answer never sees half of one.
2. No commit is missed or applied twice. The snapshot is at the slot’s consistent point L and the stream starts at L (Postgres::open and the first next_commit); the_snapshot_and_the_stream_split_exactly races 2,000 commits against it.
3. Only the writer writes. Every Spill method takes &self; the store is reached through App::read by asks and App::write by the writer alone.
4. A row removed was added. RowError::Missing from a cell, a value count or a projection entry ends the step (Rollup::remove, Projection::remove), and Straum starts over.
5. Types come from the catalog only. The load sets them (Store::set_types); a filter value must fit its column (ColumnType::fits); a Relation message that changes them is Unfit (Decoder).
6. Null joins nothing. A dim row with a null key is never held and a lookup with a null finds nothing (Projection::add, Projection::matches).
7. A row with no time is kept, never answered (Level::bucket).
8. Every width divides a day and nests in the next (SpecError::BadWidth); every ask bound is on a date_bin edge (AskError::RangeUnaligned); an ask never reads part of a bucket.
9. Levels agree. Every row goes into one cell per width in the same add (Rollup::add); Rollup::cover reads each instant of a range from exactly one level.
10. A sealed bucket takes no change. A keyed rollup refuses a row into one (RowError::Sealed); the writer unseals before it applies (Writer::unseal). The bound cells hold nothing of a sealed bucket: a seal takes it out of them at every width and an unseal binds it again, as the dims are then (Store::seal, Store::unseal).
11. The spill holds what the store has sealed and no more. Put before seal (Writer::seal), removed on unseal and truncate (Writer::forget), cleared at every start (Writer::start).
12. Bound cells are the hot keyed cells, bound as the dims are now. They exist only for a fact whose joins all read fact columns, one of them no key: any other fact’s would be as many as its keyed cells (Bound::new). A dim change moves the ones that name its key (Store::retarget) and a dim truncate or load makes them again (Store::rebuild_dim). An ask whose range holds a sealed date_bin bucket reads the keyed cells (Store::reading).
13. Lag never understates. Readings merge only toward older (Lag::read); matches_every_reading_kept checks it against a reference that keeps every reading.
14. Two stores never take memory at once. The old store goes before the new load (Writer::start).
15. One name, one thing, per view. Keys, attributes and measures share a namespace, bucket is reserved, and an answer’s columns are its names (Spec::new, Spec::check_ask).

State machines

The running states, Loading, Ready, Lagging, Waiting and Stopping, are drawn under States. There is no enum: Following is None while loading or waiting, Failure says which, and lag against --max-lag tells Ready from Lagging (App::status, why_not in straum-http). Three smaller machines:

The source connection, a date_bin bucket, and a bucket’s bound cells.

• The source: Postgres moves Closed to Loading on open, Loading to Streaming on the first next_commit, and back to Closed on the next open (Postgres in straum-pg). A load outside Loading and a wal while Closed are Error::Stopped.
• A bucket: only date_bin buckets seal. A truncate drops every bucket of the fact, sealed ones from the spill too.
• The bound cells of a bucket: a dim change moves them and never drops them (a_dim_change_under_a_sealed_bucket_reaches_it_once_unsealed); a dim update that reads the same changes nothing (Store::update_row).

Which decision shaped what

The reasoning and the tradeoffs are in the ADRs; this maps each part of the model to the records that settled it.

The PostgreSQL mechanics behind the load, the apply and the slot, with their citations, are in research/POSTGRESQL.md; what joins replaced and the sources behind them in research/JOINS.md.

Where the code and the records differ

ADR-0034’s adapter is not built: main.rs passes None. The open names are in the glossary’s Clashes table.

Reading the code

This page is for a reader who knows programming but not Rust. It gives the order to read the crates in, then the Rust the code uses, each with an example in Straum’s own words. Design is the model behind the code; the glossary names each thing.

Where to start

Read the crates from the inside out. Each one only uses the ones before it.

1. straum-domain: what a view is and how its cells add up. No I/O. Start with spec.rs (the parsed views), then store/mod.rs (where rows land), store/ask.rs and store/cell_loop.rs (how an ask reads the cells). lib.rs lists the rest in reading order.
2. straum-notation: reads Straum SQL, the CREATE VIEW statements and the asks, into the domain’s types.
3. straum-app: the use cases. app.rs holds App, which answers asks, and Writer, the loop that loads the tables and applies the stream. source.rs is the one trait PostgreSQL sits behind.
4. straum-pg: talks to PostgreSQL. source.rs is where it meets straum-app; wal/ reads the physical WAL. Read wal.rs first, then the files in the order bytes pass through them: reader.rs (pages into records), record.rs (one record), heap.rs and heap/ (row changes), xact.rs and transactions.rs (commits), tuple.rs and messages.rs (values and messages), pipeline.rs (all of it in a row).
5. straum-http: serves asks and status over HTTP. lib.rs is the server loop; routes.rs picks the handler for each request.
6. straum: the binary. main.rs starts it all.

Every file opens with a few lines saying what it holds. Read those first; they tell you whether you need the rest.

The Rust you’ll meet

& and &mut: borrowing. A value has one owner. &rows lends it out to read; &mut rows lends it out to change, and while it is lent that way nothing else may touch it. A function that takes table: &str reads the name and gives it back. &[Datum] is a borrowed list: a row, read in place, never copied.

String and &str, Vec<T> and &[T]. The first of each pair owns its text or list; the second borrows one. Box<[u32]> is a list that owns its items and never grows.

Option<T>: maybe a value. Some(value) or None, never null. Option<usize> is “a position, if there is one”.

Result<T, E>: a value or an error. Ok(value) or Err(error). Every error type is an enum listing what can go wrong, such as RowError or ClientError, with the text a person reads written in its Display.

?: return the error. self.remove_row(table, old)?; means: if that returned Err, return the same error from here; otherwise carry on with the value. Most functions read as a straight line of steps because of it.

let ... else: stop early.

#![allow(unused)]
fn main() {
let Some(bound) = &mut self.bound[fact] else {
    return Ok(());
};
}

reads: take the value out of Some; if there is none, return. After it, bound is the value itself.

if let and match. if let Some(dim) = self.dim_index(table) runs its block only when there is a value. match picks one arm per case of an enum, and the compiler refuses a match that forgets one. Several tests can share one if: if let Ok(key) = key && let Ok(value) = value runs its block only when both hold. matches!(value, Pattern) is true when the value has that shape.

Errors kept in a list: Reported. The Straum SQL walker in straum-notation reports every error in a spec at once, so it pushes each one onto a list instead of returning it. A walk that failed returns Err(Reported): an empty marker saying the error is already on the list. The caller stops there and adds nothing of its own. A plain Option is left for “there is none”, which is no error.

if let ... && ...: several conditions at once.

#![allow(unused)]
fn main() {
if let Some(scale) = key.get("scale")
    && let Some(scale) = scale.as_u64()
{
    column_type = format!("{column_type} scale {scale}");
}
}

runs its block only when every part holds; a later part may use what an earlier let took out.

enum with data. An enum case can carry values: Direction::Add carries none, Step::Started(lsn) carries a position in the WAL. match takes them out again.

Named index types. struct FactIndex(usize); is a struct with one unnamed field, read as fact.0. It costs nothing at run time and makes a signature say what a number points at: fn bind(&mut self, fact: FactIndex, ..) takes a position among the facts, and the compiler refuses a DimIndex there. PartIndex, AttributeIndex and InputIndex do the same for a part of the rollup key, a dim’s attribute and a cell state.

Iterators. for row in rows walks anything that is an Iterator: a type with a next function that hands out one item at a time until it returns None. impl Iterator<Item = CoveredBucket> as a return type means “some iterator of buckets”, without naming it. The domain returns one where a Vec would allocate on a path a test counts allocations on, such as Rollup::cover; elsewhere it returns a Vec. Entries in value_counts/ is an iterator written out as a struct, impl Iterator for Entries its next.

Closures. |cell, group, count| partials[..] += count is a function without a name, written where it is passed. The code binds none to a let. A few stay where a named function won’t do: Counts::each hands each count to one because a loop calling Counts::get per cell measured half again slower, and thread::scope(|scope| ..) runs the wider widths of a load on threads of their own. move before a closure makes it keep copies of what it uses rather than borrow them.

Cow: borrowed or owned. Cow<'_, Datum> holds either a borrowed value or one of its own. A rollup key part reads a column’s value in place, and only a null-ness key makes a new Datum, so no row’s text is copied.

impl blocks and self. impl Store { .. } holds the functions of a Store. A first parameter &self reads the store, &mut self changes it, self uses it up.

Traits. A trait is a set of functions a type promises to have. Straum has two of its own, at its edges: Source (PostgreSQL, in straum-app/src/source.rs) and Spill. impl Display for X and impl From<A> for B are the standard ones: how to print an X, and how to turn an A into a B, which is what lets ? convert one error into another.

#[derive(..)]. Asks the compiler to write the obvious functions: Debug to print for a developer, Clone to copy, PartialEq to compare.

impl Trait in a parameter. input: impl BufRead takes any type that can read lines, without naming it. straum shell passes the terminal, its tests pass bytes in memory. Like a generic, it is kept only where callers really pass different types.

Generics: <T>. Writer<S> is a writer over any source S. The binary uses Writer<Postgres>; tests use a fake source. A generic is kept only where two types really fill it. A parameter error: impl fmt::Display is the short form: any type that can be printed.

Lifetimes: 'a. A name for how long borrowed data lives. Visitor<'de> in straum-http’s json.rs reads a request body that lives as long as 'de; serde’s trait asks for it. A lifetime is kept only where copying would cost or a library asks for one; the comment beside it says so. The walker’s 'tree is one such: its structs point into the tree sqlparser built, which lives as long as the walk, so nothing is copied out of it.

Closures: |value| .... A function without a name, written where it is used, which may use the names around it. Straum writes one only where code is handed over to run later. thread::spawn(move || run_writer(writer)) runs run_writer on a new thread, and move hands the closure the values it uses. load::copy takes one as each: impl FnMut(&[Datum]), “any function of a batch”, and calls it on every batch as it comes, so a table never sits in memory whole. Everywhere else a for loop or a named function does the work.

type and one-field structs. type Oid = u32; gives a number its PostgreSQL meaning; it is still a u32. struct TableIndex(usize); is a new type: a usize that only fits where a TableIndex is asked for, read back as .0.

Threads: Arc and RwLock. App keeps its state in Arc<RwLock<State>>: Arc lets the writer thread and the HTTP threads share it, RwLock lets many asks read at once while one commit writes. read() and write() in app.rs take the lock.

Modules. mod store; pulls in store.rs, or store/mod.rs when the module is a folder of files. pub is visible to other crates, pub(crate) only inside this one, pub(super) only in the folder’s other files, nothing at all only inside the file. #[cfg(test)] mod tests at the end of a file holds its unit tests; tests/ beside src/ holds the tests that use the crate from outside.

Type aliases. type TransactionId = u32; gives u32 a second name, so xid: TransactionId says what the number is. The two are the same type: the compiler doesn’t stop a slot number going where an xid should, it only tells the reader.

Reading bytes. The WAL code reads PostgreSQL’s structs from raw bytes. A field’s place is a named constant, often a range: RECORD_XID: Range<usize> = 4..8 is bytes 4 to 7, and &bytes[RECORD_XID] is those four bytes, borrowed. Indexing past the end panics, so code that reads input it hasn’t checked uses bytes.get(4..8), which gives None instead. u32::from_le_bytes turns four bytes, least significant first (“little-endian”), into a u32; read_u32 in wal.rs wraps it. first_chunk() takes the first bytes as a fixed-size array, if there are enough, and as_chunks::<2>() cuts a slice into two-byte arrays.

Shared and maybe-copied bytes. Arc<[u8]> is bytes several owners share without copying: a row on a page and the change that names it. Cow<'_, [u8]> is bytes that are either borrowed (Cow::Borrowed) or, when they had to be changed, such as decompressed, owned (Cow::Owned).

Dense code. Where a loop would be slower in a way a benchmark measured, the code keeps a denser form and a comment names the benchmark. Everywhere else it is a plain for loop. #[inline] above a function is such a case: it asks the compiler to copy the function into each caller, which four functions an ask calls per cell need.

Glossary

One name per thing, in the guide, the ADRs, the code and its errors. The code’s name follows in parentheses where it differs; “also” lists spellings still found elsewhere. AGGREGATES.md, JOINS.md and POSTGRESQL.md are the notes under research/. The clashes at the end are open.

Straum

• Straum: the product, capitalised in prose. straum is every identifier: the binary, the crate prefix, the default role and publication, the slot prefix, the application_name, the chart and the command (ADR-0026).
• Straum SQL: the subset of PostgreSQL CREATE VIEW syntax that specs and asks are written in, one spelling per meaning, nothing inferred (ADR-0029). Also “the subset”, “the notation” (the crate name, ADR-0028), “the canon” (ADR-0029).
• Guide: the mdBook under docs/src. Its words win in prose. Also “user guide”, “the docs”.
• ADR: one settled decision in docs/adr/, numbered and never reused (ADR-0001). ADRs win over the guide and this glossary; one written before a rename keeps its old words.
• Port and adapter: a port is a trait at the edge of straum-app: Source for PostgreSQL, Spill for sealed buckets. An adapter is the crate that implements one: straum-pg, straum-http (ADR-0003).

Spec

• Spec (Spec): the file of CREATE VIEW statements Straum reads at start, and the parsed whole. straum run and straum plan take that file as FILE.
• View (Fact): one CREATE VIEW in the spec, and the name asks read FROM. Never created in PostgreSQL. Errors say “view”; straum plan and the setup check say “fact” and “dim” for a table’s role.
• Fact table (Fact::table): the table in FROM, whose rows the view counts. Needs no primary key. The logical stream needs REPLICA IDENTITY FULL; the physical stream does not.
• Dim (Dim): a table the view joins, held projected to its key and the columns the views read, and followed like a fact. One dim per table however many views join it. Also “joined table”; JOINS.md says “dimension” when quoting others.
• Dim key (Dim::key): the dim’s key columns, which a join’s columns must equal. Any number of rows may share one; no index is required. Not a view key.
• Spec table (Member): a fact or a dim when the role does not matter: what a load copies. On the logical stream, what the publication holds and what gets REPLICA IDENTITY FULL. Also “every table the spec names”.
• Table (straum_domain::Table): a schema.table, always written whole and with an alias. catalog::Checked is a spec table as the setup check read it. Relation is PostgreSQL’s word, in pgoutput and the catalog.
• Alias (Ref::alias): the AS name after a table. Asks never see one.
• Name (Identifier): the AS name after a SELECT item, the only way an ask reaches a key or measure. Unique in a view; an unquoted lowercase PostgreSQL identifier, at most 63 bytes. Also “output name” (ADR-0029), “select name” (JOINS.md).
• View option: WITH (buckets = '...'), the only one: every bucket width the view keeps, narrowest first (ADR-0028).
• Flag: a command-line option of the binary (ADR-0025). The chart calls the same setting a value (maxLag).

Keys and joins

• Key: a selected column that is not a measure: a fact column, a dim column, a null-ness key or the bucket. An ask names one by its name (Fact::resolve).
• By key (ByKey, Fact::by_keys): a key read straight from a fact column, or from whether it is null. Code and ADRs say “by key” to tell it from a dim column; the guide says key.
• Null-ness key (ByKey::IsNull): (f.col IS NULL) AS name or IS NOT NULL, a boolean key, and what a FILTER with IS NULL counts through (ADR-0019). Also Purpose::Nullness in the setup check, which a count(c) column shares.
• Dim column (Attribute, Ref::reads, Dim::attributes): a column of a dim the view selects. A key, read at ask time through the cell’s join columns, as the dim is now (ADR-0030). Code, the ADRs and JOINS.md say “attribute”.
• Join: how a view reaches a dim: INNER JOIN, LEFT OUTER JOIN, or EXISTS and NOT EXISTS in WHERE. It has no name an ask can use.
• Ref (Ref, RefKind): the join as the domain holds it. ADR-0030 calls a view “a tree of refs rooted at the fact”. Errors and the guide say join. What is wrong with one join of a view is a JoinError, inside SpecError::Join.
• Join kind (RefKind): left, inner, semi or anti.
• Join column (Via::Fact): the fact column a join reads its key from, e.account_id. It keys the cells whether selected or not (ADR-0030). In code a “via”; also “join id” (ADR-0030).
• Chain (Via::Attribute): a join whose key comes from a dim joined before it, r.name = a.region.
• Join row (JoinRow, Bound::join_rows): one way a cell’s or a fact row’s joins resolve: per join the dim row it found, or none. A join that finds several rows gives several join rows.
• Projection (Projection): the shape Straum holds a dim in: a map from dim key to the coded columns of every row under it.

Time

• Time column (Time::column): the timestamptz or timestamp column inside date_bin.
• Bucket: one span of the time column, floored to a width from the epoch, and bucket, the name of the time key. In rollup/, Bucket is also the cells of one such span, hot or sealed.
• Bucket width (Width): <n> seconds, minutes, hours or days, at most a day and dividing it; each wider one a whole multiple of the one before. ADR-0017 says “grain”.
• Narrowest width (Time::width): the date_bin width, first in buckets. The others are the wider widths (Time::wider). Also “finest” (ADR-0017, ADR-0028).
• Level (Level): the cells of one width (ADR-0017, design.md). spec.md says only width.
• Origin (Origin): the third date_bin argument, timestamptz 'epoch' or timestamp 'epoch', matching the column’s type (ADR-0018).
• Bucket edge: a whole multiple of the date_bin width. Both bounds of an ask must be on one.
• Wall clock: how a timestamp column reads: taken as UTC, no zone ever applied, printed with a Z that means nothing (ADR-0018).

Measures and cells

• Measure (Measure): a named aggregate in the view, with an optional FILTER. Asks name measures after the keys.
• Aggregate (Aggregate): count, count of a column, sum, min, max, avg, count distinct, bool_and, bool_or, percentile_disc. The walker says “aggregate” for the function and “an aggregated column” for its argument.
• Input column (Aggregate::input, Input): the column an aggregate reads. Measures over one column share its state (ADR-0016). What the aggregate reads of it (Reads) decides the types the column may have, and the setup check’s Purpose follows it. Also Purpose::Number.
• Cell: one distinct combination of the rollup key and bucket, holding its row count and one state per input column. Cells is every cell of one bucket, side by side.
• Cell state (State, StateKind): a count of the nulls among a cell’s rows, a sum and that count, a count per value for min, max and percentile_disc (the code’s “extent”), a count per distinct value, or the counts of true and false (ADR-0041). Filtered measures keep no state (ADR-0016).
• Merge: what an ask does with the cells it reads: one answer per group from their cell states. Every cell state merges.
• Cell loop (CellLoop, ask_once, ask_expanded): the pass an ask makes over the cells it reads, dropping the ones its filter and joins fail and merging the rest into their groups.
• Distributive, algebraic, holistic: what a part must keep for its aggregate to merge: its own answer (sum, min), a fixed few numbers (avg), or more as it holds more values (median, count(DISTINCT c)). Aggregates 101.
• Fraction (Fraction): the f of percentile_disc(f), from 0 to 1, kept as its digits and places.
• Rank: the position, counting from 1, of the value a percentile answers among the values in its range: ceil(f * n) in float8.
• Sketch: a small summary that merges and answers approximately. Straum has none.
• Rollup (Rollup): every cell of one fact, per width, with the codes of its key columns.
• Rollup key (Fact::rollup_key): the by keys, then each join column once, then the bucket. The guide says “the cell key” or “keys, join columns and bucket”.
• Bound cells (Bound): a second rollup of a fact whose joins all read fact columns, one of them a column no key reads, keyed by the dim columns’ values: fewer than its cells, and an ask reads no projection (ADR-0038, ADR-0042). The fact’s own cells, keyed by its join columns, are its keyed cells (Read::Keyed).
• Pour (pour, Store::pour_bucket): move every cell of a hot keyed bucket into the bound cells its join rows resolve to now, or back out of them, as a dim is replaced or a bucket sealed or unsealed.
• Overlay (Overlay): the rows one dim key held before a change, read in place of the projection so the bound cells can drop what those rows joined.
• Store (Store): the rollups of every fact and the projection of every dim, behind one RwLock (ADR-0009).

Asks and answers

• Ask (Ask): the request to POST /ask: a query and its parameters. straum-http calls the parsed form asked.
• Query: the ask’s query field, one SELECT over a view in Straum SQL (ADR-0006).
• Parameter (Literal): a $n value of an ask: number, string or boolean, with its column’s type and no conversion. A number with a fraction is a Literal::Decimal, exact as written. straum ask reads each one’s type from its text (ADR-0039).
• Literal (Literal): a constant in a spec’s FILTER or WHERE: an integer, 'text', true or false.
• Condition (Condition): key = value, key IN (values), or in a view column IS NULL and IS NOT NULL. A filter is conditions joined by AND: a measure’s FILTER, the view’s WHERE (Fact::filter) or an ask’s WHERE. store/lookup.rs evaluates one as a ConditionCheck, and an inner, semi or anti join that keeps or drops whole cells as a JoinCheck.
• Comparison (MeasureCondition, Comparison): measure op $n in an ask’s HAVING, op one of = <> < <= > >=. It keeps or drops a whole group of the answer by a measure the ask selects (ADR-0055).
• Range (Ask::from, Ask::to): the half-open [from, to) an ask reads, on bucket edges. Its bounds are bucket >= $n and bucket < $m, both required. The tests say “window” for a range that cuts a day bucket.
• Cover (Rollup::cover): the buckets an ask reads: the widest that fit in the middle, narrower ones at the ends, or date_bin buckets alone when grouped by bucket.
• Limit: what a flag caps: --max-lag, --max-retained-wal (Limits), --max-transaction-rows (Transaction::limit) and --max-page-memory (Replication::Physical).
• Answer: the JSON {"as_of", "rows"}, or the same as a table in plain text for Accept: text/plain and straum ask (ADR-0039). Answer is its columns and rows; Answered adds as_of and the applied LSN. straum-http and its tests say “reply” for the HTTP message around it.
• Row (AnswerRow): one object of an answer’s rows, sorted by key. A fact row and a dim row are rows too.
• as_of: when the slot was last known to have confirmed all the WAL there was, by Straum’s clock (ADR-0007, Lag::caught_up). straum status prints “as of”.
• Straum-LSN and Server-Timing: response headers: the last commit the answer reflects, and where the time went.
• Column type (ColumnType): integer, numeric with a scale, boolean, text, or time with or without a zone. Learned from the catalog at load; a filter value must fit it (AskError::WrongType).
• Scale: the digits past the point of a numeric(p, s). Sums and extremes print in it; averages at four places past it (ADR-0007).

Running

• Start (Writer::start, Step::Started): opening a new slot, loading every table in its snapshot and streaming from its consistent point. A load is one table’s binary COPY; “the load” is all of them.
• Start over: after any failure: close the stream, which drops the slot, wait, then start again (ADR-0015). A restart is PostgreSQL’s or a pod’s; ADR-0033 says restart for the process.
• Backoff: the wait before a new start, 1 s doubling to 60 s, reset by a load that finishes.
• States: Loading, Ready, Lagging, Waiting, Stopping (running.md). No enum; following is None while loading or waiting, and why_not says which.
• Ready (Status::ready, /readyz): every table loaded and lag within --max-lag. Otherwise “not ready” and why.
• Lag (Lag): the time since Straum last caught up with a WAL position PostgreSQL had reached, read every second; never understated (ADR-0020). Also “behind”.
• Caught up (Lag::caught_up): the last WAL reading the slot had confirmed all of; the source of as_of.
• Snapshot (Snapshot): the database as of a new slot’s consistent point, seen by a read-only repeatable-read transaction on the replication connection. It ends with the last load (ADR-0010). With --wal physical it is on a plain connection, and the stream drops the commits it saw (TransactionSnapshot, read from pg_current_snapshot()).
• Consistent point (Snapshot::at): the LSN a new slot reports, where the snapshot is taken and the stream starts. The guide says “LSN L”. For the physical stream (Consistency): past the first RUNNING_XACTS record and every transaction it names, so every later commit is whole in the WAL read.
• Stream (Stream): the commits pgoutput sends from the consistent point on. Following is where the stream stands.
• Applied (Following::applied, Status::applied, Answered::applied): the last commit in the store. Committed (Commit::committed, Status::committed): when it committed, by PostgreSQL’s clock.
• Step (Writer::step): one turn of the writer: a start, or one commit applied. A step that fails is a failure (Failure, Status::last_failure).
• Commit (Commit): one transaction’s changes, delivered and applied whole. “Transaction” only where PostgreSQL’s word is needed, as in --max-transaction-rows.
• Change (Change): one insert, update, delete or truncate of a spec table, or Unfit. The physical stream reads a HeapChange first: the change as the WAL holds it, of a HeapTuple, a row as it lies on its page.
• Unfit (Change::Unfit, Error::Unfit): a change the store can’t take, which starts Straum over. The error says “no longer fits”; ADR-0015 says “can’t apply”.
• Apply (State::apply): putting a commit into the store under the write lock.
• Source (Source, Postgres): the port to PostgreSQL: open a stream, load a table, next commit, WAL reading. “Source of truth” means PostgreSQL itself.
• Plain connection (Postgres::plain, Connection): the second, non-replication connection that reads the WAL position each second (ADR-0010).
• WAL retained (Wal::retained, --max-retained-wal, maxRetainedWal): the bytes PostgreSQL keeps for Straum’s slot, its written position less the slot’s restart_lsn.
• Status (Status): what GET /status and straum status show, read at one moment. Not telemetry (ADR-0023). The stream thread’s report of the applied LSN to PostgreSQL (StreamThread::report) is a different message.
• Held (Held, App::held): what GET /views and straum views show, read at one moment: per view its widths, cells, range of buckets, keys with their types and measures; per dim its rows (ADR-0039). None until every table is loaded. As JSON for a program that asks for it (ADR-0040).
• Shell (straum shell): a prompt at a running Straum: \d for the views, \rows VIEW for a view’s rows, an ask as QUERY; PARAMETER..., with Tab completion at a terminal (ADR-0040).
• A view’s rows (App::rows_ask, GET /views/NAME, \rows): the rows PostgreSQL would give for the view, which is the answer to an ask for every measure by the bucket and every key over the buckets held.
• Health check: /livez, up while the process serves, and /readyz, up while ready.

PostgreSQL

• Setup: the SQL a table owner runs once, which straum plan prints. The setup check (Error::Setup, catalog::check) compares the catalog with the spec at every start. Also “start check” (ADR-0014, ADR-0024, ADR-0031), “catalog check” (ADR-0018, ADR-0037).
• Problem: one line of a failed setup check, naming what is wrong and what to run.
• Role: the PostgreSQL role Straum connects as, the DSN’s user or straum (--role). Unrelated: catalog::Purpose is what a column is for in the setup check.
• DSN (Dsn): the connection URL, from STRAUM_DSN or --dsn-file, never the command line (ADR-0021). Its Display leaves the password out.
• Publication: the publication holding every spec table, straum by default (--publication).
• Replication slot: the temporary logical slot each start creates, straum_<pid>_<nanos>_<n>, dropped when the connection closes, or with --wal physical a temporary physical one. “The slot” in the guide.
• Replication (Replication, --wal): which WAL Straum streams: logical decoding’s through the publication, or the physical WAL (ADR-0049).
• Physical stream (Physical, Replay, Reader::Physical): the physical WAL from a physical slot, turned into pgoutput messages for the same decoder.
• Redo point: where recovery from a checkpoint starts, and where a physical slot reserves WAL from. The first change to a page after it carries a full-page image: the page whole, after the change.
• Page state (Heap, Page): the rows Straum keeps of each page of the spec’s tables changed since the last redo point, by slot. A page goes at a checkpoint record whose redo point is past its last change.
• Layout (Layout, physical::read_layouts): how a table’s rows lie on its pages: its file, every attribute by attnum with dropped ones, and each column added with a default’s value.
• Pipeline (wal::pipeline::Pipeline): the physical stream’s pure path from records to commits: page state, held transactions, the publisher and the decoder.
• WAL record (wal::Record, Records): one entry of the physical WAL, written by one resource manager (xl_rmid), the part of PostgreSQL that writes a kind of record and replays it in recovery: heap, transaction, standby, storage, xlog. Records run across 8 KiB WAL pages; Records cuts them out.
• Block reference (Block): a page a record changes, named by its relation file (RelationFile, PostgreSQL’s RelFileLocator: tablespace, database and file node) and block number (BlockNumber), with the record’s data for it and maybe a full-page image.
• Slot on a page (SlotNumber, PostgreSQL’s OffsetNumber): where a row sits on a heap page, counted from 1. Its line pointer says where the row’s bytes lie. Not a replication slot.
• xid (TransactionId, Xid): a transaction id, 32 bits that wrap. TransactionId is the WAL reader’s name; Xid is the name in the errors that say which transactions a start waits on. A subtransaction (subxact) is a savepoint’s own xid; it commits with its top transaction. xact is PostgreSQL’s word for transaction, kept as the name of the module that reads its records.
• Running transactions (xact::Running): a standby record, RUNNING_XACTS, listing the transactions running when it was written. The physical stream holds each transaction’s changes until it ends, as logical decoding’s reorder buffer does (Transactions).
• Relcache invalidation (Invalidated): a commit’s note that a table’s catalog entry changed, which DDL leaves. The physical stream reads the table’s layout again after one.
• Varlena and TOAST: a value of varying length, such as text or numeric, with a one- or four-byte length header. A long one may be compressed in the row, with pglz or lz4, or moved out of it to the table’s TOAST table, leaving a pointer (tuple::Value::External).
• WAL problem (wal::Problem): what is wrong with WAL Straum can’t read, said after “WAL at LSN has”. It stops the stream.
• Replication connection: the connection that creates the slot, loads in its snapshot and then streams, all on one session (ADR-0010).
• LSN (Lsn): a WAL position, printed as PostgreSQL does. A Wal reading holds three: written, confirmed (the slot’s confirmed_flush_lsn) and restart (its restart_lsn).
• OID (Oid): an object id, how PostgreSQL names a row of its catalog: a table, a type, an enum label. pgoutput names a table by its OID, the relation id. Code writes oid, PostgreSQL’s word.
• Catalog (Catalog, Class, Column): PostgreSQL’s system tables as the setup check reads them.
• Old row (Change::Update::old): the row before an update or delete. The logical stream has it whole under REPLICA IDENTITY FULL (ADR-0031). The physical stream rebuilds it from the page.
• Label (EnumLabel): an enum value’s text, read by writing ::text (ADR-0037).
• Poison value (ValueError::at): a value Straum can’t read, such as a NaN. It stops the load or the stream, naming table and column.

Sealing

Old date_bin buckets can leave memory through a port (ADR-0033, ADR-0035).

• Seal (Store::seal): hand a date_bin bucket that ended seal_after ago to the spill and drop its cells, keeping its place in the level. A sealed bucket is Bucket::Sealed; one in memory is hot (Bucket::Hot). ADR-0033’s title says “cold”.
• Spill (Spill, Spilling): the port sealed buckets go to. Every start empties it. The flags --spill-dir and --seal-after and the crate straum-arrow that ADR-0033 and ADR-0034 name do not exist yet; main.rs passes None.
• Sealed (Sealed, SealedId): a bucket’s cells as plain data. SealedId is the fact, the width and bucket_start.

Testing

• Reference: what a test compares Straum’s answer with: a plain computation over raw rows in the domain tests, a store loaded from the fake at the same LSN in the app tests, SQL run in PostgreSQL in the straum-pg tests (Reference). Planting a bug must make the comparison fail.
• Fake (Fake): a scripted PostgreSQL in the app and HTTP tests. PostgresContainer is a real one in Docker. MemorySpill and Lost are spills.
• Taxi trips: the TLC yellow trips of January to June 2024 in bench/data, which the benchmark loads. The taxi month is their January, which the quick start, taxi.sql and the ignored taxi_month test load. The warehouse copy is the benchmark’s production-shaped tables, not in the repository.

Clashes