Projection Failures

⚠️

causet failures list/get, causet dlq list/retry/ignore, and causet projections doctor shown below are proposed — none have an equivalent in causet-cli today. See CLI Overview for what’s actually shipped.

This page is a reference summary of the projection failure model. For the complete deep-dive, see Projection Failure Handling.

Types of projection failures

Failure type	Cause	Recoverable?
Invalid event payload	Missing or null field referenced in derive expression	Yes — fix derive expression, replay
Schema mismatch	Column missing from projection table (DDL not applied)	Yes — apply DDL, replay
Bad derive expression	Type error, null dereference	Yes — fix expression, deploy, replay
Missing projection table	Table not created at deploy	Yes — apply DDL, replay
DB constraint violation	PK conflict, unique violation, not-null constraint	Usually yes — fix derive or schema
DB timeout / connection failure	Infrastructure transient failure	Yes — auto-retried, usually resolves
Poison event	Event that always fails this handler	Yes (fix handler) or acknowledged loss
IR version mismatch	Stale IR loaded from cache	Yes — clear cache, redeploy

The ProjectionFailure record

type ProjectionFailure = {
  id: string;
  projectionName: string;
  eventId: string;
  eventType: string;
  eventVersion: number;
  handlerName: string;
  errorName: string;
  errorMessage: string;
  stack?: string;
  retryCount: number;
  status: "open" | "retrying" | "resolved" | "ignored";
  occurredAt: string;
  lastRetriedAt?: string;
  deploymentId?: string;
  environment: "local" | "staging" | "production";
};

Query failure records:

# All open failures in production
causet failures list --env production --status open
 
# Failures for a specific projection
causet failures list --projection artist_leaderboard --env production
 
# Failures since a deploy
causet failures list --since-deployment deploy_abc123

Health endpoint

The projection worker exposes health at /actuator/health:

{
  "status": "DEGRADED",
  "components": {
    "projections": {
      "status": "DEGRADED",
      "details": {
        "artist_leaderboard": "DEGRADED",
        "user_concert_history": "UP",
        "show_attendance": "UP"
      }
    }
  }
}

Status	Meaning
`UP`	No open failures, processing normally
`DEGRADED`	Open failures exist, projection may be falling behind
`DOWN`	No successful event processed within configured threshold

DLQ integration

After exhausting retries, events route to causet.projection-dlq.v1. See Dead Letter Queues for full DLQ documentation.

Quick reference:

# List DLQ messages for a projection
causet dlq list --projection artist_leaderboard --env production
 
# Replay after fixing the handler
causet dlq retry --projection artist_leaderboard --env production
 
# Acknowledge data loss (last resort)
causet dlq ignore --id pfail_01j2abc --reason "..."

Recovery steps

Identify — Check /actuator/health, causet failures list
Examine — causet failures get --id {id} — read errorMessage, handlerName
Diagnose — Match error to root cause table above
Fix — Edit .causet file, recompile, deploy
Validate — causet projections doctor --fail-on-error
Replay — causet dlq retry --projection {name}
Monitor — Watch projection_lag_seconds return to normal
Verify — Query projection, assert correct data

Full recovery runbook: Projection Failure Handling → Recovery runbook.

Key metrics

Metric	Alert threshold
`projection_failures_total`	Any new failure
`projection_dlq_total`	Any new message
`projection_lag_seconds`	> 30s warning, > 300s critical
`projection_upserts_total`	Sudden drop → investigate

What is Causet?