Metrics

Causet services expose Prometheus-compatible metrics through Spring Boot Actuator. Scrape the /actuator/prometheus endpoint on each service.

Prometheus scrape endpoint

GET /actuator/prometheus

Add each service to your prometheus.yml scrape config:

# infra/prometheus/prometheus.yml
scrape_configs:
  - job_name: causet-runtime-service
    static_configs:
      - targets: ['causet-runtime-service:8080']
    metrics_path: /actuator/prometheus
 
  - job_name: causet-projection-worker
    static_configs:
      - targets: ['causet-projection-worker:8083']
    metrics_path: /actuator/prometheus
 
  - job_name: causet-query-service
    static_configs:
      - targets: ['causet-query-service:8082']
    metrics_path: /actuator/prometheus
 
  - job_name: causet-saas-cloud
    static_configs:
      - targets: ['causet-saas-cloud:8085']
    metrics_path: /actuator/prometheus

Causet application metrics

Event throughput

Metric	Type	Labels	Description
`causet_events_emitted_total`	Counter	`event_type`, `platform`, `application`, `fork`	Total events written to the ledger
`causet_events_processed_total`	Counter	`event_type`, `projection`, `fork`	Total events processed by the projection worker

# Event emission rate (per event type, 1m window)
rate(causet_events_emitted_total[1m])

# Projection processing throughput
rate(causet_events_processed_total[1m])

Projection lag

Metric	Type	Labels	Description
`causet_projection_lag_seconds`	Histogram	`projection`, `fork`	Time from event emission to projection UPSERT completion

# P95 projection lag per projection
histogram_quantile(0.95,
  sum by (le, projection) (
    rate(causet_projection_lag_seconds_bucket[5m])
  )
)

# Average lag across all projections
rate(causet_projection_lag_seconds_sum[5m]) /
  rate(causet_projection_lag_seconds_count[5m])

Intent latency

Metric	Type	Labels	Description
`causet_intent_latency_seconds`	Histogram	`action`, `fork`, `status`	End-to-end intent processing time (received → event committed)

# P99 intent latency per action
histogram_quantile(0.99,
  sum by (le, action) (
    rate(causet_intent_latency_seconds_bucket[5m])
  )
)

Handler duration

Metric	Type	Labels	Description
`causet_handler_duration_seconds`	Histogram	`projection`, `event_type`, `op`	Time spent in the projection handler UPSERT

Failures and retries

Metric	Type	Labels	Description
`causet_projection_failures_total`	Counter	`projection`, `event_type`, `error_type`	Projection handler failures
`causet_retry_count_total`	Counter	`projection`, `event_type`	Total retry attempts
`causet_dlq_messages_total`	Counter	`projection`, `event_type`	Messages sent to DLQ

# Failure rate per projection (5m window)
rate(causet_projection_failures_total[5m])

# DLQ rate (any non-zero value is an alert)
rate(causet_dlq_messages_total[5m])

Kafka consumer lag

Metric	Type	Labels	Description
`causet_kafka_consumer_lag`	Gauge	`topic`, `partition`, `consumer_group`	Unprocessed messages in the projection event topic
`causet_checkpoint_age_seconds`	Gauge	`consumer_group`, `topic`	Time since last committed checkpoint

# Total lag across all partitions
sum(causet_kafka_consumer_lag{consumer_group="causet-projection-worker"})

# Lag growing over 10 minutes (early warning)
deriv(sum(causet_kafka_consumer_lag{consumer_group="causet-projection-worker"})[10m:1m]) > 0

JVM and infrastructure metrics

Spring Boot Actuator also exposes standard JVM and infrastructure metrics:

Metric	Description
`jvm_memory_used_bytes`	JVM heap and non-heap memory usage
`jvm_gc_pause_seconds`	GC pause duration histogram
`jvm_threads_live_threads`	Live thread count
`hikaricp_connections_active`	Active DB connections from connection pool
`hikaricp_connections_pending`	Threads waiting for a DB connection
`hikaricp_connections_timeout_total`	Connection acquisition timeouts
`http_server_requests_seconds`	HTTP request latency histogram, per endpoint
`process_cpu_usage`	CPU usage of the JVM process

# DB connection pool saturation (approaching limit is a warning)
hikaricp_connections_active / hikaricp_connections_max

# GC overhead
rate(jvm_gc_pause_seconds_sum[5m])

# HTTP error rate
rate(http_server_requests_seconds_count{status=~"5.."}[5m])

Alert thresholds

Configure these alerts in Grafana. These are starting points — tune based on your actual load.

Alert	Condition	Severity
Projection lag high	`causet_projection_lag_seconds P95 > 30s`	WARNING
Projection lag critical	`causet_projection_lag_seconds P95 > 5m`	CRITICAL
DLQ non-empty	`rate(causet_dlq_messages_total[5m]) > 0`	WARNING
Projection failures	`rate(causet_projection_failures_total[5m]) > 0`	WARNING
Kafka consumer lag growing	`deriv(causet_kafka_consumer_lag[10m]) > 100`	WARNING
DB connection pool near limit	`hikaricp_connections_active / hikaricp_connections_max > 0.8`	WARNING
Service health down	`up{job="causet-runtime-service"} == 0`	CRITICAL
High intent latency	`causet_intent_latency_seconds P95 > 2s`	WARNING

Sample Grafana alert rule:

- alert: CausetDLQNonEmpty
  expr: increase(causet_dlq_messages_total[5m]) > 0
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: "DLQ received messages in the last 5 minutes"
    description: "Projection {{ $labels.projection }} sent {{ $value }} messages to DLQ"

Grafana dashboard

Import the dashboard template from infra/prometheus/dashboards/causet-overview.json. It includes panels for all key metrics listed above.

See Dashboards for panel descriptions and setup instructions.

What is Causet?