Dead Letter Queues

The DLQ is causet.projection-dlq.v1. An event lands there after exhausting all configured retries in the projection worker. A DLQ message represents a real event that was never successfully processed into a projection table.

Why DLQs exist

Without a DLQ, a projection worker stuck on an unprocessable event faces two options: retry forever (blocking the Kafka partition and all subsequent events) or drop the event (silent data loss). Neither is acceptable.

The DLQ provides a third option: acknowledge the failure, move the event to a safe holding space, and allow subsequent events to continue processing. The failed event is not lost — it’s in a known, queryable location. A human can intervene, fix the handler, and replay it.

What goes into the DLQ

An event enters causet.projection-dlq.v1 when the projection worker has:

1. Attempted to process the event
2. Failed on every attempt
3. Exhausted causet.projection.retry.max-attempts retries

The DLQ message contains:

{
  "originalEvent": {
    "type": "ArtistUpdated",
    "entity_id": "artist_1",
    "ts": 1719340800000,
    "version": 2,
    "payload": {
      "info": { "artist_name": "The Cure" },
      "ranking": null
    }
  },
  "failureMetadata": {
    "projectionName": "artist_leaderboard",
    "handlerName": "artist_leaderboard:ArtistUpdated",
    "errorName": "NullPointerException",
    "errorMessage": "Cannot evaluate derive expression: event.ranking.popularity_score — ranking is null",
    "retryCount": 3,
    "lastAttemptAt": "2026-06-25T20:20:00Z",
    "deploymentId": "deploy_abc123",
    "environment": "production"
  }
}

The original event envelope is preserved exactly. This is critical — the DLQ message contains everything needed to replay the event through a fixed handler.

Never auto-discard DLQ messages

This cannot be overstated: do not configure auto-discard on the DLQ topic.

Every message in causet.projection-dlq.v1 represents a row that was never written to a projection table. Discarding it means:

• That row may never exist in the projection
• Queries that depend on the projection will return incorrect or incomplete data
• The discarded event is gone — even if you fix the handler later, there is no row to replay from

Configure Kafka topic retention for causet.projection-dlq.v1 to be much longer than for the main topic:

# Main topic: 7 days
causet.projection-events.v1.retention.ms=604800000

# DLQ: 30 days minimum — do not reduce
causet.projection-dlq.v1.retention.ms=2592000000

DLQ monitoring and alerting

Any message landing in causet.projection-dlq.v1 is an incident. Set up:

Kafka alert: Consumer lag on causet.projection-dlq.v1 > 0 → page on-call. DLQ messages should never sit unread.

Dashboard metric: projection_dlq_total — cumulative count. In steady state this should be 0 or increasing only during known incidents.

Log alert: The projection worker emits a log line on DLQ routing:

{
  "level": "error",
  "message": "Event routed to DLQ after exhausting retries",
  "projection": "artist_leaderboard",
  "eventId": "evt_abc123",
  "retryCount": 3,
  "dlqTopic": "causet.projection-dlq.v1"
}

Alert on "message": "Event routed to DLQ after exhausting retries" from any environment.

DLQ processing workflow

Step 1: Inspect the DLQ message

causet dlq list \
  --env production \
  --projection artist_leaderboard

Output:

ID             PROJECTION          EVENT_TYPE      EVENT_ID     RETRY_COUNT  OCCURRED_AT
pfail_01j2abc  artist_leaderboard  ArtistUpdated   evt_abc123   3            2026-06-25T20:20:00Z
pfail_01j2def  artist_leaderboard  ArtistUpdated   evt_def456   3            2026-06-25T20:21:00Z

Step 2: Get full error detail

causet dlq inspect --id pfail_01j2abc

Step 3: Fix the handler

Edit the .causet projection. Common fixes:

• Add null guard to a derive expression
• Correct a field path that changed in a new event schema version
• Fix a field type declaration in fields:

Step 4: Deploy and validate

causet build compile --runtime . --out dist/
causet deploy --fork production --runtime .

Step 5: Replay DLQ messages

After the fix is deployed and validated:

causet dlq retry \
  --projection artist_leaderboard \
  --env production

This replays all DLQ messages for the specified projection through the updated handler. Resolved messages are marked status: resolved in the failure record.

Bulk replay after a fix

If many events accumulated in the DLQ during an incident:

causet dlq retry \
  --projection artist_leaderboard \
  --env production \
  --since "2026-06-25T20:00:00Z" \
  --batch-size 100

Monitor the replay progress. If the handler fails again on any message, it re-enters the retry cycle.

Marking events as ignored

If a DLQ message cannot be replayed — the original event payload is genuinely invalid and cannot be fixed — you can mark it as ignored:

causet dlq ignore \
  --id pfail_01j2abc \
  --reason "Event contained corrupted payload from upstream system v1. Field 'ranking' was null due to pre-v2 schema. No recovery possible."

This is an explicit acknowledgment that the data is lost for this event. It must be:

• A documented decision
• Reviewed and approved by an engineer familiar with the downstream impact
• Never the default action

The failure record transitions to status: "ignored" with the reason attached.