Logging

All Causet services write structured JSON logs to stdout. Logs are the primary debugging tool for tracing individual intent processing chains, investigating projection failures, and diagnosing infrastructure issues.

Log format

Every log line is a JSON object on a single line:

{
  "timestamp": "2026-06-25T20:00:00.123Z",
  "level": "INFO",
  "logger": "com.causet.runtime.IntentProcessor",
  "thread": "reactor-http-nio-3",
  "message": "Intent processed",
  "action": "FOLLOW_ARTIST",
  "entityId": "user-1",
  "forkId": "production",
  "platformId": "my-platform",
  "applicationId": "concert-app",
  "correlationId": "corr-abc123",
  "durationMs": 12
}

Log levels

Level	When used
`ERROR`	Unrecoverable failures: DLQ writes, handler exceptions, DB unavailable
`WARN`	Recoverable issues: retry scheduled, slow query, degraded dependency
`INFO`	Normal operational events: intent processed, event appended, projection updated
`DEBUG`	Rule evaluation detail, SQL queries, Kafka offset commits. Not enabled in production by default

Set the log level via environment variable:

LOGGING_LEVEL_COM_CAUSET=DEBUG   # verbose Causet logging
LOGGING_LEVEL_ROOT=WARN          # suppress framework noise

Key log events

Intent received

{
  "level": "INFO",
  "message": "Intent received",
  "action": "FOLLOW_ARTIST",
  "entityId": "user-1",
  "correlationId": "corr-abc123",
  "forkId": "production"
}

Intent processed (success)

{
  "level": "INFO",
  "message": "Intent processed",
  "action": "FOLLOW_ARTIST",
  "entityId": "user-1",
  "correlationId": "corr-abc123",
  "eventsEmitted": ["ARTIST_FOLLOWED"],
  "durationMs": 12
}

Intent rejected by preflight

{
  "level": "INFO",
  "message": "Intent rejected",
  "action": "FOLLOW_ARTIST",
  "entityId": "user-1",
  "correlationId": "corr-abc123",
  "rejectCode": "CANNOT_FOLLOW_SELF",
  "rejectingRule": "reject_self_follow"
}

Ledger event appended

{
  "level": "INFO",
  "message": "Ledger event appended",
  "eventType": "ARTIST_FOLLOWED",
  "eventId": "evt_789xyz",
  "entityId": "user-1",
  "correlationId": "corr-abc123"
}

Projection event published to Kafka

{
  "level": "INFO",
  "message": "Projection event published",
  "topic": "causet.projection-events.v1",
  "eventType": "ARTIST_FOLLOWED",
  "eventId": "evt_789xyz",
  "partition": 2,
  "offset": 10045
}

Projection handler executed (success)

{
  "level": "INFO",
  "message": "Projection handler executed",
  "projection": "user_following",
  "eventType": "ARTIST_FOLLOWED",
  "eventId": "evt_789xyz",
  "op": "upsert",
  "durationMs": 5
}

Projection handler failed

{
  "level": "ERROR",
  "message": "Projection handler failed",
  "projection": "user_concert_memory",
  "eventType": "CONCERT_MEMORY_CREATED",
  "eventId": "evt_123",
  "handler": "updateUserConcertMemory",
  "retryCount": 3,
  "correlationId": "corr_abc",
  "error": "column venue_name does not exist",
  "stackTrace": "org.postgresql.util.PSQLException: ERROR: column \"venue_name\" of relation ..."
}

This is the most important error log. When this appears, investigate immediately:

Does the column exist in the tenant schema? (Missing DDL)
Is there a type mismatch between the IR derive expression and the column type?
Is the IR version the projection worker is running correct?

Retry scheduled

{
  "level": "WARN",
  "message": "Retry scheduled",
  "projection": "user_concert_memory",
  "eventId": "evt_123",
  "retryCount": 1,
  "nextRetryMs": 1000,
  "error": "Connection reset by peer"
}

DLQ message written

{
  "level": "ERROR",
  "message": "DLQ message written",
  "projection": "user_concert_memory",
  "eventId": "evt_123",
  "retryCount": 3,
  "reason": "Max retries exhausted",
  "dlqTopic": "causet.projection-dlq.v1"
}

Correlation IDs

Every intent processing chain has a correlationId. This ID is:

Generated when an intent is received
Attached to the ledger event payload
Carried through the Kafka message to the projection worker
Included in all projection worker log lines for that event

To trace a complete request chain, filter logs by correlationId:

# Find all logs for a specific correlation ID
docker compose logs causet-runtime-service causet-projection-worker \
  | jq 'select(.correlationId == "corr-abc123")'
 
# In Grafana Loki
{service="causet-runtime-service"} | json | correlationId="corr-abc123"

Log aggregation

In production, Grafana Alloy collects stdout from all ECS containers and ships to Grafana Cloud Loki.

Configuration: infra/grafana-alloy/config.alloy

Key Loki queries:

# All errors across services in the last 15 minutes
{namespace="causet-prod"} | json | level="ERROR"

# Projection failures for a specific projection
{service="causet-projection-worker"} | json | level="ERROR"
  | message="Projection handler failed"
  | projection="user_concert_memory"

# Track a specific event by event ID
{namespace="causet-prod"} | json | eventId="evt_789xyz"

# Intent rejection rate
sum(rate({service="causet-runtime-service"} | json
  | message="Intent rejected" [5m]))

Local logging

# Follow all service logs
docker compose logs -f
 
# Follow a specific service
docker compose logs -f causet-runtime-service
 
# Follow projection worker errors only
docker compose logs -f causet-projection-worker \
  | jq 'select(.level == "ERROR")'
 
# Search historical logs
docker compose logs causet-projection-worker --tail=1000 \
  | grep '"eventId":"evt_123"'

What not to log

PII values (email addresses, names, phone numbers) — use opaque IDs in events and logs
Secrets or credentials — these should never appear in application code paths
Full event payloads at INFO level — use DEBUG level for payload details to avoid log volume explosion

See PII Handling for guidance on keeping sensitive data out of logs.

What is Causet?