Dead Letter Queues

When a projection event fails all retry attempts in the projection-worker, it is written to the dead letter queue (DLQ) topic: causet.projection-dlq.v1.

Purpose

The DLQ prevents a single failing event from blocking projection processing indefinitely. Without a DLQ, a “poison event” — one that always fails a specific handler — would stop the consumer at that offset, causing the worker to stall.

The DLQ allows:

Processing to continue past the failing event.
Failing events to be inspected, diagnosed, and recovered.
Root cause analysis with full error context.

What Ends Up in the DLQ

An event is sent to the DLQ when:

A projection handler throws an exception for this event.
The worker retries the event with exponential backoff up to the configured maximum.
All retry attempts are exhausted.

The event is then published to causet.projection-dlq.v1 and the consumer offset for causet.projection-events.v1 advances past it.

DLQ Message Structure

{
  "originalEvent": {
    "eventType": "ARTIST_FOLLOWED",
    "entityId": "user-123",
    "forkId": "production",
    "tenantSchema": "acme_concertapp_production",
    "irVersion": "v42",
    "sequenceNumber": 1891,
    "ts": 1719331200000,
    "payload": {
      "user_id": "user-123",
      "artist_id": "artist-456"
    }
  },
  "projectionName": "artist_followers",
  "tenantSchema": "acme_concertapp_production",
  "error": "ERROR: column \"followed_at\" of relation \"artist_followers\" does not exist",
  "stackTrace": "org.postgresql.util.PSQLException: ...",
  "retryCount": 5,
  "firstFailedAt": 1719331210000,
  "failedAt": 1719331280000
}

Field	Description
`originalEvent`	The full projection event that failed
`projectionName`	The specific projection handler that failed
`tenantSchema`	The PostgreSQL schema the handler was writing to
`error`	The error message from the last failure
`stackTrace`	Full stack trace from the last failure
`retryCount`	Number of retry attempts before DLQ
`firstFailedAt`	Timestamp of the first failure
`failedAt`	Timestamp of the final failure (DLQ submission)

Common DLQ Causes

Error Pattern	Likely Cause	Fix
`column "X" does not exist`	New projection field added to DSL but DDL not applied to this fork	Deploy new IR version to activate DDL migration
`null value in column "X"`	Required NOT NULL constraint but `derive` expression returns null	Fix `derive` expression or make column nullable
`relation "X" does not exist`	Projection table not created (DDL not applied)	Deploy IR version to create the table
`division by zero`	`derive` expression divides by a field that is zero	Guard expression with null/zero check
`connection refused`	Projection DB unreachable	Infrastructure issue — events will DLQ if retries exhaust

Inspecting the DLQ

Consume causet.projection-dlq.v1 to inspect failing events:

# Using kafkacat / kcat
kcat -b localhost:9092 -t causet.projection-dlq.v1 -C -o beginning

Or via the Causet CLI (proposed):

causet dlq list --fork-id production
causet dlq list --fork-id production --projection artist_followers

Recovery Procedure

Identify the root cause from the DLQ message’s error field.
Fix the issue — deploy a corrected projection definition (new IR version) or fix infrastructure.
Verify the fix by submitting a test intent and confirming the projection materializes correctly.
Re-consume DLQ events — publish them back to causet.projection-events.v1 for reprocessing.

Re-consuming DLQ events (proposed CLI):

causet dlq retry --fork-id production --projection artist_followers
causet dlq retry --fork-id production --all

Projection mutations are idempotent (upsert/delete by primary key), so re-processing DLQ events that were previously partially applied is safe.

Poison Events

A poison event is one that will always fail a specific projection handler — regardless of retries or timing. Examples:

An event with a payload that violates a database constraint that cannot be relaxed.
An event targeting an entity that no longer exists in a foreign-key-constrained table.

For poison events, the options are:

Fix the projection definition to handle the case gracefully.
Deploy the fix, then re-consume the DLQ.
If the event is genuinely unprocessable (data integrity issue), mark it as permanently skipped and document the decision.

Isolation Guarantee

One failing event in one projection does not block processing of other events or other projections. The consumer offset advances past the DLQ’d event, and all subsequent events continue to be processed normally.

If a tenant’s events are concentrated on a few partitions, a burst of DLQ events for that tenant may cause visible projection lag for that tenant’s projections. Monitor DLQ depth as an operational health signal.

DLQ Retention

Configure causet.projection-dlq.v1 with longer retention than the primary topic (30 days minimum). DLQ events represent data that has not been applied to projections — you need time to diagnose and recover.

Retries — retry policy before DLQ
Projection Pipeline — how events reach the DLQ
Replay — rebuilding projections without relying on DLQ re-consumption
Message Brokers — DLQ topic configuration

What is Causet?