Rebuilding Memory

Because all Causet state is derived from the ledger, entity snapshots and projection tables can be rebuilt at any time. This is the mechanism for backfilling new fields, correcting buggy rules, and recovering from infrastructure failures.

When to Rebuild

Rebuilding Entity Snapshots

Entity snapshots are rebuilt by replaying ledger_events through the updated rules engine. The rebuild:

1. Reads events from the ledger in order for each entity
2. Applies preflight and core rules — this includes saga step transitions, since sagas: compiles to ordinary core rules (see Defining Sagas). Side effects (submit, schedule, cross-entity emit) are not re-fired during rebuild — see Durable Execution
3. Writes the resulting state to entity_snapshots

Procedure

Step 1: Deploy the updated IR

Compile your updated .causet files:

causet build compile --runtime . --out dist/

This produces causet.runtime.json (updated with new state fields and rules) and causet.projections.json.

Deploy the updated IR to your Causet environment before triggering the rebuild. The rebuild uses the deployed IR version.

Step 2: Trigger the snapshot rebuild

causet recovery replay --stream user_stream --fork main

This replays the user stream from the beginning (--from-cursor 0, the default) and recomputes every entity snapshot on that stream against the currently deployed rules.

To replay a bounded range instead of the full stream, add --from-cursor / --to-cursor:

causet recovery replay --stream user_stream --fork main --from-cursor 12000

Step 3: Monitor progress

Rebuild progress is exposed via metrics and logs from the causet-projection-worker:

[causet-projection-worker] Rebuilding user snapshots: 12,450 / 85,000 (14.6%)
[causet-projection-worker] Rebuild rate: 2,100 entities/sec
[causet-projection-worker] ETA: ~34 minutes

Step 4: Verify rebuilt state

After rebuild completes, spot-check a sample of entities:

const user = await causet.getEntity("user", "user_abc123");
console.log(user.favorite_venue);  // Should now be populated
console.log(user.concert_count);   // Should match expected value

Query the snapshots table directly for aggregate verification:

SELECT
  COUNT(*) FILTER (WHERE state->>'favorite_venue' != '') AS with_venue,
  COUNT(*) FILTER (WHERE state->>'favorite_venue' = '') AS without_venue,
  AVG((state->>'concert_count')::int) AS avg_concerts
FROM entity_snapshots
WHERE entity_type = 'user';

Rebuilding Projection Tables

Projections are rebuilt by resetting the Kafka consumer group offset to the beginning of the topic and re-processing all events.

Step 1: Deploy updated projections IR

Deploy causet.projections.json with the updated projection definitions.

Step 2: Truncate the target table

TRUNCATE TABLE user_artist_affinity;

Warning: Truncating a projection table removes all current data. Ensure downstream queries can tolerate empty results during the rebuild. Use a staging table if you need zero-downtime projection rebuilds.

Step 3: Reset the consumer group offset

Reset the causet-projection-worker consumer group to the earliest offset for the affected topics:

kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --group causet-projection-worker \
  --topic causet-events \
  --reset-offsets \
  --to-earliest \
  --execute

Step 4: Restart the projection worker

The worker reads from the beginning of the topic and re-processes all events. The UPSERT logic ensures idempotent re-processing.

Step 5: Monitor lag

Monitor the consumer group lag until it reaches zero:

kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --group causet-projection-worker \
  --describe

Determinism and Safety

The rebuild is safe because:

• Rules are deterministic: given the same events and rules, you always get the same state
• Side effects are not replayed: op: submit, op: emit, and op: schedule do not fire during snapshot rebuild — only state mutations in core rules are applied
• UPSERT idempotency: if the worker processes an event it has already processed (due to at-least-once delivery), the UPSERT produces the same result

The determinism guarantee means you can run rebuild in a staging environment against a copy of production ledger data to verify the results before running it in production.

Caution: Side Effects During Original Processing

When events were originally processed, side effects fired — submitting intents, emitting events, scheduling timers. During replay, those side effects do not re-run.

This is correct behavior. You do not want to re-send emails, re-submit intents to downstream entities, or re-schedule timers just because you are rebuilding state.

However, it means that if your memory fields depended on state that was set by a side effect (rather than by the originating intent’s core rules), those fields may not be rebuilt correctly. Design memory fields to accumulate from core rules, not from side effects.

Partial Rebuilds

For large entity populations, a full-stream rebuild can be time-consuming. causet recovery replay scopes by cursor, not by timestamp or entity type — use --from-cursor to skip everything before a known-good point:

causet recovery replay --stream user_stream --fork main --from-cursor 48210

Look up the cursor for a known timestamp with causet inspect timeline before replaying, since the CLI does not accept a --since timestamp directly.

For a single entity, causet recovery rewind (also sandbox-only) rewinds one entity to a specific timeline item instead of replaying a whole stream:

causet recovery rewind --entity user-42 --stream user_stream --to-timeline 48210