Projection Failure Handling

Projection failures are the most dangerous class of failures in an event-driven system. They are silent by default. The projection stops updating. The read model serves stale data. The service reports healthy. Nobody knows.

Causet treats projection failures as first-class data: captured with full context, tracked with status, surfaced via health endpoints, and never silently discarded.

Why projection failures are different

A typical exception gets logged, maybe retried, and potentially surfaced as a 500 to a client. Someone notices.

A projection failure is different:

1. It happens asynchronously, after the write path has already returned success.
2. The affected table silently stops updating.
3. Existing rows remain — they just go stale.
4. Queries return data that looks correct but is wrong.
5. The service health endpoint reports UP.
6. No user-facing error occurs.

The failure is invisible until someone compares query output to expected reality, which might be days later.

Causet’s position: every projection failure must have an owner, a status, and a retry path. Nothing disappears silently.

What causes a projection to fail

Schema and type mismatches

• An event payload field is null but the derive expression doesn’t handle null
• A field type changed in the DSL (string → int) but the existing projection data is incompatible
• A column was removed from the state/event definition but projection still references it
• fields: block omitted → all columns inferred as TEXT → boolean/bigint queries break

Missing infrastructure

• Target table does not exist (DDL not applied from IR)
• Required index missing
• Tenant schema not created (fork not deployed)
• IR version mismatch — projection worker running stale IR from S3/Redis cache

Data problems

• DB constraint violation (PK conflict from non-deterministic ID)
• Unique constraint violation
• Null value in NOT NULL column

Handler bugs

• Derive expression throws (null reference, type error, array index out of bounds)
• Wrong event type in source_events — unhandled event arrives
• Aggregate op overflow

Infrastructure failures

• DB connection pool exhausted
• R2DBC timeout
• PostgreSQL write conflict (transient)

The failure record

Every captured failure produces a record:

type ProjectionFailure = {
  id: string;
  projectionName: string;
  eventId: string;
  eventType: string;
  eventVersion: number;
  handlerName: string;
  errorName: string;
  errorMessage: string;
  stack?: string;
  retryCount: number;
  status: "open" | "retrying" | "resolved" | "ignored";
  occurredAt: string;
  lastRetriedAt?: string;
  deploymentId?: string;
  environment: "local" | "staging" | "production";
};

This record is the authoritative statement that something failed, what it was, and what state it is in. It is not just a log line.

Structured failure log

Every failure also produces a structured log entry:

{
  "level": "error",
  "message": "Projection handler failed",
  "projection": "user_concert_memory",
  "eventType": "CONCERT_ATTENDED",
  "eventId": "evt_abc123",
  "handler": "updateUserConcertMemory",
  "retryCount": 3,
  "correlationId": "corr_xyz",
  "deploymentId": "deploy_v2.1.4",
  "environment": "production",
  "error": "column venue_name does not exist"
}

What happens when a handler fails

1. Error captured with full context (projection name, event ID, event type, error, stack trace).
2. Failure record written with status: "open".
3. Retry scheduled with exponential backoff.
4. Retry count incremented on each attempt.
5. After max retries: event sent to causet.projection-dlq.v1 Kafka topic.
6. Status updated to reflect current retry state.
7. Metrics emitted: causet_projection_failures_total incremented.
8. Health endpoint updated: projection marked degraded or down.

The projection worker does not stop. Other events continue processing. The failing event is isolated so it does not block the entire pipeline.

Retry policy

Retries use exponential backoff with jitter:

Retry count and lastRetriedAt are tracked in the failure record.

Dead letter queue

After exhausting retries, the event is sent to causet.projection-dlq.v1.

DLQ message structure:

{
  "originalEvent": { ... },
  "failureMetadata": {
    "projectionName": "user_concert_memory",
    "errorMessage": "column venue_name does not exist",
    "retryCount": 5,
    "firstFailedAt": "2026-06-25T14:32:00Z"
  }
}

DLQ messages must never be auto-discarded. Every DLQ message represents a real data gap — the projection table is missing at least one row or update.

DLQ should trigger alerting. Any message in DLQ is a production incident.

Poison events

A poison event is an event that will always fail a specific projection handler — not because of a transient infrastructure issue, but because the event data or the handler logic is fundamentally incompatible.

Detection: same event failing repeatedly with the same error.

Risk: a poison event that keeps retrying can delay processing of subsequent events on the same Kafka partition.

Resolution options (in order of preference):

1. Fix the projection handler (recompile + deploy + replay)
2. Fix the event data if the payload is genuinely malformed
3. Mark the failure as "ignored" — explicit decision, acknowledged data gap

Never skip poison events without a deliberate, documented decision.

Deployment gates

The best time to catch a projection failure is before it reaches users.

What to validate at deploy time

• All target projection tables exist in the tenant schema
• All declared indexes exist
• Checkpoint store (Kafka consumer group) is reachable
• IR version matches deployed artifacts on S3
• Fixture events pass through each projection handler without error

Proposed CLI command

causet projections doctor --env production --fail-on-error

Example output when healthy:

✓ Loaded 24 projections
✓ Verified checkpoint store
✓ Verified projection tables (24/24 exist)
✓ Verified required indexes (47/47 exist)
✓ Fixture replay passed (24/24 projections)

All checks passed.

Example output when there are problems:

✓ Loaded 24 projections
✓ Verified checkpoint store
✓ Verified projection tables
✕ user_concert_memory: fixture replay failed
  Error: column venue_name does not exist in table user_concert_memory
✕ venue_stats: missing index idx_venue_stats_venue_id

Deployment blocked. Fix the above errors before deploying.

This command should run in the CI/CD pipeline before every production deploy. A non-zero exit code should block the deploy.

Projection health endpoint

Every service exposes /actuator/health. The projection worker includes projection health in its response:

{
  "status": "UP",
  "components": {
    "projectionHealth": {
      "status": "DEGRADED",
      "projections": {
        "user_concert_memory": { "status": "FAILING", "openFailures": 3 },
        "artist_popularity": { "status": "UP" },
        "venue_stats": { "status": "UP" }
      }
    }
  }
}

Status meanings:

• UP — processing normally, no open failures
• DEGRADED — some failures but still processing
• DOWN — stopped processing or max failures exceeded

Monitoring

Key metrics

Alert on DLQ depth

Any DLQ message should page on-call. DLQ messages represent events that could not be processed after all retries — they are gaps in your projection tables.

Recovery runbook

When a projection fails:

1. Identify the failing projection

# Check projection worker logs
docker compose logs -f causet-projection-worker | grep '"level":"error"'
 
# Or check health endpoint
curl -s http://localhost:8083/actuator/health | jq .components.projectionHealth

2. Get error details from failure record

# Query failure records (replace with your actual DB access)
psql -h localhost -d causet \
  "SELECT * FROM projection_failures WHERE status = 'open' ORDER BY occurred_at DESC LIMIT 10;"

3. Determine root cause

Common causes and fixes:

• column X does not exist → column added to DSL but DDL not applied → redeploy release
• null value in column X → derive expression not null-safe → fix DSL expression
• ClassCastException → type mismatch in derive → fix field type in DSL
• duplicate key value → non-deterministic ID → fix to use deterministic ID

4. Fix the DSL

# Fix .causet files
# Recompile
causet build compile --runtime my-app --out build/out
 
# Deploy new release via control plane

5. Verify fix against fixture event

causet projections doctor --projection user_concert_memory --fail-on-error

6. Process DLQ messages

# After fix is deployed, replay DLQ messages
causet dlq retry --projection user_concert_memory  # proposed CLI

7. Monitor recovery

# Watch consumer lag
watch -n 5 'curl -s http://localhost:8083/actuator/health | jq .components.projectionHealth'

8. Verify query output is correct

# Run expected queries and verify output matches reality
curl -X POST http://localhost:8082/.../queries/user_concert_memory \
  -d '{"params": {"user_id": "user-1"}}'

Common mistakes

Ignoring failure records because queries still return data. Stale data is not correct data. An open failure record means some rows are missing or out of date.

Manually fixing projection tables instead of fixing the handler. Manual fixes are overwritten on the next projection rebuild. Fix the DSL, redeploy, replay.

Silently dropping DLQ messages. Every discarded DLQ message is a permanent data gap. If you decide to ignore an event, make it an explicit, documented decision with status: "ignored".

Not running deployment gates. The deployment gate is how you catch column X does not exist errors in staging instead of production.

Deploying without verifying fixture replay. If your fixture events don’t pass projection handlers before deploy, you will find out the hard way in production.

Next steps

• Rebuilding Projections — fix and replay after a projection bug
• Concepts: Projections — projection model overview
• Testing Projections — validate handlers before deploy