Message Brokers

Causet uses Redpanda (Kafka-compatible) as the async fan-out bus between the write path and the projection pipeline. The message broker decouples durable ledger writes from eventually-consistent projection materialization.

Why Async Fan-Out

The write path (intent → rules → ledger commit) is synchronous and returns to the caller after the ledger write is durable. Projection materialization is intentionally asynchronous:

• Projection fan-out does not add to write latency.
• Multiple projections can consume the same event independently.
• Slow or failing projections do not block writes.
• The projection worker can be scaled independently.
• Kafka’s consumer group model enables parallel partition processing.

The trade-off is that projection data is eventually consistent — there is a lag between ledger commit and projection availability.

Topics

Topic Details

causet.projection-events.v1

This is the primary fan-out topic. causet-runtime publishes to this topic after every successful ledger commit (fire-and-forget).

Message structure:

{
  "eventType": "ARTIST_FOLLOWED",
  "entityId": "user-123",
  "forkId": "prod",
  "tenantSchema": "platform1_concertapp_prod",
  "irVersion": "v42",
  "sequenceNumber": 1891,
  "ts": 1719331200000,
  "payload": {
    "user_id": "user-123",
    "artist_id": "artist-456"
  }
}

Partitioning: By entityId. This ensures all events for a given entity arrive in order at the same partition.

Retention: Configurable. Long retention enables projection rebuilds from Kafka rather than requiring a separate replay mechanism.

causet.projection-dlq.v1

Events that fail all projection-worker retries land here with failure context:

{
  "originalEvent": { ... },
  "projectionName": "artist_followers",
  "tenantSchema": "platform1_concertapp_prod",
  "error": "column \"followed_at\" does not exist",
  "stackTrace": "...",
  "retryCount": 5,
  "failedAt": 1719331800000
}

causet.patches.v1

The projection worker publishes fine-grained patch messages to this topic for subscribed entities. ws-gateway consumes these patches and delivers them to connected WebSocket clients in realtime.

Patch messages describe a delta — the change to a specific entity’s projection row — rather than the full row.

Partitioning

causet.projection-events.v1 is partitioned by entityId. This guarantees:

• All events for the same entity arrive at the same partition.
• The projection worker processes events for a given entity in sequence.
• No reordering of events for the same entity across partitions.

The number of partitions determines the maximum parallelism for the projection worker consumer group. A consumer group with more instances than partitions will have idle instances.

Consumer Groups

Multiple instances of projection-worker in the same consumer group split the partitions between them. Kafka’s rebalancing protocol handles instance additions and removals.

Fire-and-Forget Semantics

causet-runtime publishes to Kafka after the ledger transaction commits. The Kafka publish is not part of the database transaction. If the publish fails:

• The ledger write is already durable.
• The caller has already received a success response.
• The event will not be in causet.projection-events.v1.

Recovery options:

1. Rely on causet.ledger-events.v1 as a catch-up stream (if the event was also published there).
2. Trigger a projection rebuild from the beginning of the Kafka topic (if retention allows).
3. Use the entity browser (gRPC) to fetch the raw ledger event and manually publish to the projection topic.

Note: For most deployments, transient Kafka publish failures are rare. The ledger-first guarantee means data is never lost, only temporarily out of sync.

Redpanda vs Kafka Compatibility

Causet uses the standard Kafka protocol. Redpanda is fully Kafka-API-compatible and is the recommended broker for:

• Local development (single-binary, fast startup)
• Lower-resource environments (no JVM overhead)
• Production deployments requiring high throughput with simplified operations

AWS MSK (Kafka) or Confluent Cloud can be substituted without any code changes — only the bootstrap server configuration changes.

Topic configuration (partitions, replication factor, retention) should be set explicitly regardless of which broker is used.

Recommended Topic Configuration

Related Pages

• Projection Pipeline — how the projection worker consumes events
• Dead Letter Queues — handling DLQ events
• Event Flow — full event lifecycle
• Scaling Model — scaling the projection worker