Retries

Causet has retry behavior at multiple layers: the runtime write path, the projection worker, and intent re-submission by clients.

Write Path Retries (causet-runtime)

Transient failures on the write path — network blips to PostgreSQL or brief Kafka unavailability — are handled at the infrastructure level:

R2DBC connection pool: Connections are retried on transient failure. A failed connection is evicted from the pool and replaced.
Kafka producer retries: The Kafka producer is configured with retries > 0 and exponential backoff. Transient broker unavailability is retried automatically.
Application-level retries: causet-runtime does not retry a full intent processing cycle. If the ledger write fails after exhausting infrastructure retries, the intent returns a 5xx to the caller. The caller is responsible for re-submitting.

The write path is safe to retry end-to-end: if the ledger write did not succeed (5xx response), no events were written. Re-submitting the intent is safe.

Warning: If the caller receives a 200 success response, the events are in the ledger. Do not re-submit intents that returned 200 — duplicate events may result if the downstream action is not idempotent.

Projection Worker Retries

When a projection handler fails (throws an exception during SQL execution or IR evaluation), the worker retries the event:

Retry policy

Parameter	Default	Configuration
Max retry attempts	5	`PROJECTION_WORKER_MAX_RETRIES`
Initial backoff	1 second	`PROJECTION_WORKER_RETRY_INITIAL_MS`
Backoff multiplier	2x	`PROJECTION_WORKER_RETRY_MULTIPLIER`
Max backoff	30 seconds	`PROJECTION_WORKER_RETRY_MAX_MS`

Example retry schedule for a failing event:

Attempt 1:  immediate
Attempt 2:  1s
Attempt 3:  2s
Attempt 4:  4s
Attempt 5:  8s
→ DLQ after attempt 5 fails

What triggers a retry

SQL execution error (e.g. schema mismatch, constraint violation)
IR evaluation error (e.g. divide by zero in a derive expression)
Database connection failure

What does not retry

Events successfully processed (no retry needed)
Events with no matching projection (source_events does not include this event type) — these are skipped silently

Dead Letter Queue After Max Retries

After exhausting all retry attempts, the projection worker publishes the event to causet.projection-dlq.v1 and commits the Kafka offset. The worker does not block on the failed event — it continues processing subsequent events.

See Dead Letter Queues for DLQ message structure and recovery procedures.

Intent Retries (Client-Side)

Clients can safely re-submit intents that returned a 5xx error:

5xx response: The ledger write did not succeed. Re-submission is safe.
200 SUCCESS: Events are in the ledger. Do not re-submit.
200 REJECTED: The intent was rejected by preflight. Re-submitting the same intent will produce the same rejection (rules are deterministic).
Network timeout (no response received): The outcome is unknown. Check intent_status by intent ID before re-submitting.

For idempotent operations, use deterministic entity IDs and guard against duplicates in preflight rules.

Retry Visibility

The retry count is tracked in the DLQ message and in the projection worker’s structured logs:

{
  "event": "projection_retry",
  "projectionName": "artist_followers",
  "eventType": "ARTIST_FOLLOWED",
  "entityId": "user-123",
  "forkId": "production",
  "retryAttempt": 3,
  "error": "column \"followed_at\" does not exist",
  "nextRetryIn": "4s"
}

Configure log aggregation alerts on projection_retry events with high retry counts to detect systemic projection failures early.

Operational Retries (DLQ Consumer)

DLQ events can be re-consumed after the underlying issue is resolved (e.g. a projection bug is fixed and a new IR version is deployed). This is a manual operational step — see Dead Letter Queues.

Dead Letter Queues — handling events that exhaust retries
Projection Pipeline — how the worker processes events
Event Flow — failure scenarios

What is Causet?