Observability Overview
Causet provides three pillars of observability: structured logs from all services, Prometheus-compatible metrics from Spring Boot actuator, and OpenTelemetry-compatible distributed traces. The default stack ships Grafana + Grafana Alloy + Prometheus + Grafana Cloud.
Three pillars
| Pillar | What it tells you | Tool |
|---|---|---|
| Logs | What happened in each service, with context | Grafana Alloy → Grafana Cloud Loki |
| Metrics | Counts, rates, latencies, saturation | Prometheus → Grafana Cloud |
| Traces | End-to-end request flows across services | OpenTelemetry → Grafana Cloud Tempo |
All three are necessary. Metrics tell you something is wrong. Logs tell you what went wrong for a specific event. Traces tell you where time was spent across the system.
What to observe in Causet
The most operationally important signals in a Causet deployment:
| Signal | Why it matters |
|---|---|
| Projection lag | Time between event emission and projection update. Growing lag means the projection worker is falling behind. |
| DLQ depth | Messages in the dead-letter queue represent failed projection handlers. Non-zero DLQ requires immediate investigation. |
| Intent latency | End-to-end time for an intent to be processed and an event written to the ledger. |
| Projection failures | Handler errors per projection. Persistent failures indicate a bug in derive expressions or a schema mismatch. |
| Kafka consumer lag | Unprocessed messages in the projection event topic. Grows when the worker is slower than the runtime produces events. |
| Event throughput | Rate of events emitted. Useful for capacity planning and detecting traffic anomalies. |
| Checkpoint age | How old the last committed Kafka checkpoint is. Stale checkpoints mean restarts will reprocess more events. |
Stack components
Configuration lives in:
infra/grafana-alloy/— Alloy configuration for log and trace collectioninfra/prometheus/prometheus.yml— Prometheus scrape configuration
Local observability
The Docker Compose stack includes a local Grafana instance on port 3001:
# Start the stack including Grafana
docker compose up -d
# Open Grafana
open http://localhost:3001Default credentials: admin / admin. Dashboards are pre-provisioned from infra/prometheus/dashboards/.
Section pages
- Logging — structured log format, key events, log aggregation
- Metrics — Prometheus metrics, key signals, alert thresholds
- Tracing — OpenTelemetry spans, trace correlation, Tempo
- Health Checks —
/actuator/health, custom health indicators, uptime monitoring - Audit Events — the ledger as an audit trail
- Dashboards — Grafana dashboard panels and setup