Dashboards

Causet ships Grafana dashboard templates in infra/prometheus/dashboards/. Import them into your Grafana instance to get immediate visibility into system health, projection lag, and event throughput.

Setup

Local Grafana (Docker Compose)

The Docker Compose stack includes a local Grafana instance. Dashboards are pre-provisioned from infra/prometheus/dashboards/:

docker compose up -d
open http://localhost:3001   # Grafana (admin / admin)

Grafana Cloud

Use Grafana Alloy to ship metrics and logs to Grafana Cloud. The Alloy configuration in infra/grafana-alloy/config.alloy handles:

• Log collection from Docker/ECS and forwarding to Loki
• Prometheus remote write to Mimir
• OTel trace forwarding to Tempo

Configure your Grafana Cloud credentials in the Alloy config:

GRAFANA_CLOUD_LOKI_URL=https://logs-prod-123.grafana.net
GRAFANA_CLOUD_PROM_URL=https://prometheus-prod-123.grafana.net
GRAFANA_CLOUD_TEMPO_URL=https://tempo-prod-123.grafana.net
GRAFANA_CLOUD_ACCESS_TOKEN=glc_...

Dashboard: Causet Overview

Import from infra/prometheus/dashboards/causet-overview.json.

This is the primary operational dashboard. Check it after every deploy and during incidents.

Panel 1: Events emitted (rate)

sum by (event_type) (
  rate(causet_events_emitted_total{fork="$fork"}[1m])
)

Time series, split by event type. Useful for seeing traffic patterns and detecting anomalies (unexpected drop to zero, sudden spike).

Panel 2: Projection lag (P50 / P95 / P99)

histogram_quantile(0.50, sum by (le, projection) (
  rate(causet_projection_lag_seconds_bucket{fork="$fork"}[5m])
))

histogram_quantile(0.95, sum by (le, projection) (
  rate(causet_projection_lag_seconds_bucket{fork="$fork"}[5m])
))

histogram_quantile(0.99, sum by (le, projection) (
  rate(causet_projection_lag_seconds_bucket{fork="$fork"}[5m])
))

Time series with threshold lines at 30s (warning) and 5m (critical). Shows lag per projection so you can identify which projection is slow.

Panel 3: Projection failures (count by projection)

sum by (projection) (
  increase(causet_projection_failures_total{fork="$fork"}[5m])
)

Bar chart. Any non-zero value requires investigation. Displayed as a count over the last 5 minutes so brief spikes are visible.

Panel 4: DLQ depth (current messages)

sum(causet_dlq_messages_total{fork="$fork"})

Stat panel with red threshold at 1. DLQ non-empty is always a problem — this panel should be green (zero) in a healthy system.

Panel 5: Intent latency (P50 / P95 / P99)

histogram_quantile(0.50, sum by (le, action) (
  rate(causet_intent_latency_seconds_bucket{fork="$fork"}[5m])
))

Time series per action. P99 above 2s warrants investigation.

Panel 6: Kafka consumer lag

sum by (partition) (
  causet_kafka_consumer_lag{
    consumer_group="causet-projection-worker",
    topic="causet.projection-events.v1"
  }
)

Time series per partition. Total lag growing over time means the projection worker is not keeping up with the runtime.

Panel 7: DB connection pool usage

hikaricp_connections_active{job=~"causet-runtime-service|causet-projection-worker"}
  /
hikaricp_connections_max{job=~"causet-runtime-service|causet-projection-worker"}

Gauge, 0–100%. Above 80% is a warning — the pool may start queuing requests.

Panel 8: Service health status

up{job=~"causet-runtime-service|causet-projection-worker|causet-query-service|causet-saas-cloud"}

Status table showing 1 (healthy, green) or 0 (down, red) per service.

Dashboard: Projection Detail

Import from infra/prometheus/dashboards/causet-projection-detail.json.

Drill-down into a single projection’s performance. Use when you’ve identified a specific projection with high lag or failures in the overview dashboard.

Key panels:

• Handler duration histogram (P50/P95/P99 for a specific projection)
• Failure rate by error type
• Retry count over time
• DLQ messages for this projection
• UPSERT throughput vs event throughput (should match)

Dashboard: Infrastructure

Import from infra/prometheus/dashboards/causet-infrastructure.json.

JVM and infrastructure metrics:

• JVM heap usage per service (warn at 80%)
• GC pause duration (warn at > 500ms)
• Thread count
• HTTP request latency per endpoint
• CPU usage

Alerts

Configure Grafana alert rules to fire on threshold breaches. Use the Alerting contact points to route:

• CRITICAL alerts → PagerDuty (wake someone up)
• WARNING alerts → Slack channel (review next business day)

Recommended alert groups:

See Metrics for the complete alert threshold reference.