Rebuilds

A rebuild re-processes the full event history to reconstruct a projection table from scratch. Because projections are derived entirely from events and derive expressions are deterministic, a rebuild always produces the same final state as the original projection — given the same events.

Rebuilds are a normal operational tool, not an emergency procedure.

When to rebuild

Note: Adding a column to fields: does not automatically fill existing rows. The new column will be NULL for all rows written before the rebuild. If NULL is not acceptable (e.g., the column is used in a where filter), you must rebuild.

Rebuild procedure

1. Stop the projection worker (or pause the consumer for the target projection)

Do not reset offsets while the consumer is running. A live consumer will overwrite the reset on the next commit.

kubectl scale deployment causet-projection-worker --replicas=0

2. Optionally truncate the target projection table

For a clean rebuild, truncate the table. This ensures no stale rows persist.

TRUNCATE TABLE {platformId}_{applicationId}_{forkId}.artist_leaderboard;

If you are doing a partial rebuild (replaying only recent events to fix recent rows), skip the truncate.

3. Reset the Kafka consumer group offset

To replay from the beginning of Kafka retention:

kafka-consumer-groups.sh \
  --bootstrap-server $KAFKA_BOOTSTRAP \
  --group causet-projection-worker \
  --topic causet.projection-events.v1 \
  --reset-offsets \
  --to-earliest \
  --execute

To replay from a specific offset (partial rebuild):

kafka-consumer-groups.sh \
  --bootstrap-server $KAFKA_BOOTSTRAP \
  --group causet-projection-worker \
  --topic causet.projection-events.v1 \
  --reset-offsets \
  --to-offset 12345 \
  --execute

4. Restart the projection worker

kubectl scale deployment causet-projection-worker --replicas=3

The worker will start consuming from the reset offset and begin applying UPSERTs.

5. Monitor consumer lag until caught up

kafka-consumer-groups.sh \
  --bootstrap-server $KAFKA_BOOTSTRAP \
  --group causet-projection-worker \
  --describe

Watch the LAG column. When lag reaches 0, the projection has caught up to the current head of the topic.

Monitoring rebuild progress

The most reliable signal is Kafka consumer lag:

GROUP                       TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
causet-projection-worker    causet.projection-events.v1    0          8421            24100           15679
causet-projection-worker    causet.projection-events.v1    1          7833            23450           15617
causet-projection-worker    causet.projection-events.v1    2          9102            24800           15698

Total lag across partitions gives you a rough estimate of remaining work. At known throughput (UPSERTs per second), you can estimate time to completion.

Additional signals:

• projection_upserts_total metric — cumulative UPSERT count per projection
• projection_lag_seconds metric — how far behind (in event timestamp terms) each projection is
• Worker logs: structured log line per committed batch

Zero-downtime rebuild

For production projections that serve live traffic, you can rebuild without downtime by building the new table in parallel:

1. Create a new projection with a temporary name (e.g., artist_leaderboard_v2) targeting the same events.
2. Deploy the updated IR with both the old and new projection.
3. Let the new projection catch up (monitor lag).
4. Once caught up, update your queries to read from artist_leaderboard_v2.
5. Remove the old projection and rename.

This adds complexity. For most non-critical projections, a brief period of stale data during a normal rebuild is acceptable. Reserve zero-downtime rebuilds for projections serving customer-facing queries with strict freshness requirements.

Rebuild vs. migration

Schema migrations (column type changes, column renames) cannot be handled by rebuild alone. They require explicit DDL and may require a rebuild after the DDL is applied. Additive schema changes (adding a column, adding an index) can be applied as DDL and then backfilled via rebuild.

Safety

Because derive expressions are deterministic, the same event always produces the same row. This means a rebuild is safe to run multiple times — the final state of the projection table converges to the correct result regardless of how many times you replay the event history.

The only risk is with aggregate projections under at-least-once delivery: if events are delivered more than once during the rebuild, aggregate counters will be inflated. For aggregate projections, truncate the table before rebuilding to ensure a clean starting state.