Health Checks

Health checks tell you whether a running service is capable of processing requests. Causet uses Spring Boot Actuator for health endpoints on all Java services, with indicators for database, Kafka, S3, and Redis connectivity.

Note: The deployment configuration for health checks (ECS task definitions, ALB target groups, Docker Compose) is covered in Deployment: Health Checks. This page focuses on what health checks reveal and how to use them for observability.

Health endpoint

All Java services expose:

GET /actuator/health

The endpoint returns HTTP 200 when the service is healthy and HTTP 503 when any indicator is DOWN.

Health indicators

Database connectivity

Tests that the service can reach its PostgreSQL database and execute a simple query. This indicator goes DOWN if:

• The database is unreachable (network, security group)
• The connection pool is exhausted
• The database is rejecting connections (out of space, max connections reached)

"db": {
  "status": "DOWN",
  "details": {
    "error": "Unable to acquire JDBC Connection; nested exception is org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved"
  }
}

Kafka connectivity

Tests that the service can reach the Kafka broker. Goes DOWN if Kafka is unreachable or the broker returns an error. Present on causet-runtime-service and causet-projection-worker.

S3 connectivity

Tests that the service can reach the S3 endpoint and list the IR bucket. Goes DOWN if S3/MinIO is unreachable or the bucket does not exist.

Redis connectivity

Tests that Redis is reachable. Present when REDIS_URL is configured. Goes DOWN on connection failure.

Custom health indicators

In addition to the standard indicators, the projection worker exposes a custom indicator for projection health — whether projections are being updated at a normal rate.

"projectionHealth": {
  "status": "WARN",
  "details": {
    "laggingProjections": ["user_concert_memory"],
    "maxLagSeconds": 45
  }
}

This indicator uses WARN (not DOWN) when lag exceeds the threshold. The service continues to process events, but the elevated lag is surfaced in the health endpoint for monitoring.

Configure the lag threshold:

PROJECTION_HEALTH_LAG_WARN_SECONDS=30
PROJECTION_HEALTH_LAG_DOWN_SECONDS=300

Uptime monitoring

Set up external uptime monitoring (Grafana Cloud, Pingdom, UptimeRobot, or similar) against the ALB-fronted health endpoints:

https://runtime.yourdomain.com/actuator/health
https://query.yourdomain.com/actuator/health
https://saas.yourdomain.com/actuator/health

Check frequency: every 30 seconds. Alert if any endpoint returns non-200 or takes more than 5 seconds to respond.

Grafana alerting on health

Configure a Grafana alert against the Prometheus up metric:

- alert: CausetServiceDown
  expr: up{job=~"causet-runtime-service|causet-projection-worker|causet-query-service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Causet service {{ $labels.job }} is down"
    description: "Prometheus cannot scrape {{ $labels.job }} — the service is unreachable or crashed"

Route this alert to PagerDuty or OpsGenie via Grafana’s alerting contact points.

Component-level alerting

Do not wait for status: DOWN to alert. Alert on component-level degradation before it causes service unavailability:

# DB connection pool pressure (component still UP but approaching failure)
- alert: CausetDBConnectionPoolNearLimit
  expr: hikaricp_connections_active / hikaricp_connections_max > 0.85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "DB connection pool usage above 85%"
 
# Kafka consumer not making progress
- alert: CausetProjectionWorkerStalledKafka
  expr: causet_checkpoint_age_seconds > 60
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Projection worker has not committed a checkpoint in >60 seconds"

Interpreting health failures