Recovery

This page is a practical playbook for recovering from each category of Causet failure. Use it during incidents.

Projection stopped updating

Symptoms: Queries return stale data. projection_lag_seconds is increasing. /actuator/health shows a projection as DEGRADED or DOWN.

Recovery:

1. Check the health endpoint:

   curl -s https://projection-worker.internal/actuator/health | jq .components.projections

2. List open failure records:

   causet failures list --env production --status open

3. Examine the failure record for the affected projection:

   causet failures get --id {failure_id}

   Read errorName, errorMessage, handlerName. Note whether occurredAt coincides with the last deploy.

4. Diagnose and fix the .causet projection definition. Common fixes:

   • Null-safe derive expression
   • Correct field path reference
   • Fix type in fields:
   • Add missing column

5. Compile, validate, and deploy:

   causet build compile --runtime . --out dist/
   causet deploy --fork production --runtime .

6. Replay DLQ messages:

   causet dlq retry --projection {name} --env production

7. Monitor lag until it returns to near-zero:

   kafka-consumer-groups.sh \
     --bootstrap-server $KAFKA_BOOTSTRAP \
     --group causet-projection-worker \
     --describe

8. Verify query results against expected data.

If lag is > 0 but no failures: The worker is behind but processing. Check worker pod health, resource limits, and whether Kafka partition count is adequate for the event volume. The worker will catch up on its own unless it’s resource-constrained.

Saga stalled

Symptoms: An entity is stuck in an intermediate state. Downstream business operations that depend on the entity being in a specific state cannot proceed.

Recovery:

1. Inspect the entity via gRPC:

   grpcurl \
     -d '{"platform": "...", "application": "...", "fork": "...", "entityId": "..."}' \
     runtime.internal:9090 \
     causet.EntityService/GetEntitySnapshot

   Identify the current state and the last event.

2. Determine the correct recovery action:

   • Can the saga be advanced to the next valid state with a corrective intent?
   • Should the entity be cancelled/reset?
   • Is the original error in the rule still present, or was it transient?

3. Fix the rule if it caused the stall (recompile, deploy).

4. Submit a corrective intent to advance or reset the entity:

   curl -X POST \
     "http://runtime/v1/.../intents" \
     -d '{"type": "CorrectiveIntent", "entity_id": "...", "_corrective": true, "reason": "..."}'

5. Verify the entity reaches a valid terminal state via the entity browser or saga health projection.

Query returning stale data

Symptoms: A query returns data that doesn’t reflect recent events. The event was committed (write path succeeded) but the projection hasn’t caught up.

Recovery:

First, determine whether this is expected eventual consistency lag or a projection failure.

1. Check projection lag:

   kafka-consumer-groups.sh \
     --bootstrap-server $KAFKA_BOOTSTRAP \
     --group causet-projection-worker \
     --describe

   If lag is small and decreasing, this is normal lag. Wait.

2. Check for projection failures:

   causet failures list --projection {name} --env production --status open

   If failures exist, follow the “Projection stopped updating” playbook above.

3. Check Redis cache TTL: If the query has a Redis cache configured and the TTL is long, cached stale responses may be served even after the projection catches up. Check the cache TTL and whether a cache invalidation is needed.

4. If stale data persists after lag reaches zero and cache is cleared, a rebuild may be needed:

   • The projection may have been in a failed state long enough to miss events
   • Follow the Rebuilds procedure

Intent rejected unexpectedly

Symptoms: An intent that should succeed returns a 400 error. Preflight rules are rejecting a valid operation.

Recovery:

1. Read the error response body — it includes which rule failed and why.

2. Inspect the entity’s current state via gRPC — the rule may be correctly rejecting based on the current entity state:

   grpcurl \
     -d '{"platform": "...", ..., "entityId": "..."}' \
     runtime.internal:9090 \
     causet.EntityService/GetEntitySnapshot

3. Determine whether:

   • The rule is correct and the intent is invalid (caller needs to fix the intent)
   • The entity is in an unexpected state (saga recovery needed)
   • The rule has a bug (fix the rule, compile, deploy)

4. If the rule has a bug, fix it:

   causet build compile --runtime . --out dist/
   causet deploy --fork production --runtime .

5. Verify the intent succeeds after the fix.

DB connection pool exhausted

Symptoms: The projection worker or query service logs Connection pool exhausted or Unable to acquire connection within timeout. Latency spikes. Queries time out.

Recovery:

1. Identify the immediate cause:

   • Are there long-running queries blocking connections? Check pg_stat_activity.
   • Is the connection pool size too small for the current load?
   • Is there a projection rebuild running that’s consuming connections faster than normal?

2. Immediate relief:

   -- Identify and terminate blocking queries
   SELECT pid, query, state, wait_event_type, wait_event
   FROM pg_stat_activity
   WHERE state != 'idle'
   ORDER BY query_start;
    
   SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid = {blocking_pid};

3. Scale connections if the load has genuinely grown:

   # projection worker config
   causet.r2dbc.pool.max-size: 50  # increase from default

4. Check query patterns — a missing index can cause queries to hold connections for seconds instead of milliseconds. Run EXPLAIN ANALYZE on slow queries.

5. If a rebuild is the cause, pause the rebuild during peak traffic hours and resume during low-traffic windows.

Recovery prioritization

Not all projections are equally critical. When multiple projections fail simultaneously (e.g., after a bad deploy), recover in this order:

1. Customer-facing projections — projections that power live user queries
2. Billing and financial projections — data loss here has direct business impact
3. Internal analytics projections — important but tolerate more lag
4. Development/staging projections — lowest priority

Maintain a documented criticality tier for each projection in your production application.

Data integrity verification after recovery

After recovering a projection, verify data integrity before declaring the incident resolved:

1. Query the projection for a set of known entities
2. Compare against the ledger event history (what should be true based on events)
3. If using an external source of truth, cross-check against it

# Example: verify artist leaderboard row against ledger
causet inspect entity artist_1 \
  --stream artist_stream \
  --fork production
 
# Compare entity.ranking.popularity_score against projection query result
causet query run \
  --query artist_by_id \
  --params '{"entity_id": "artist_1"}'

Communication during incidents

When a projection failure causes user-visible data staleness:

• Communicate data freshness clearly — users seeing old data should be informed that the system is catching up, not that the data is wrong
• Give an ETA based on lag metric — if lag is 10,000 messages at 2,000 msg/s throughput, ETA is ~5 minutes
• Update status page — mark the affected feature as degraded, not down
• Avoid manual data patches — do not manually update the projection table. Fix the handler and replay. Manual patches will be overwritten on the next UPSERT.