Tracing

Causet services emit OpenTelemetry (OTel) compatible traces. Traces show you the full lifecycle of an intent — from receipt through rule evaluation, ledger write, Kafka publish, and projection UPSERT — with timing for each step.

Trace topology

A single intent produces trace spans across multiple services:

The correlationId in the event payload carries trace context from the runtime into the Kafka message, allowing the projection worker to continue the same trace.

Key trace spans

Span name	Service	What it measures
`intent.process`	runtime	Total intent processing time
`intent.preflight`	runtime	Preflight rule evaluation
`intent.core`	runtime	Core rule evaluation + state patch
`ledger.write`	runtime	INSERT to `ledger_events`
`kafka.publish`	runtime	Kafka message publish
`projection.consume`	projection worker	Kafka message consume + handler dispatch
`projection.derive`	projection worker	Field derivation from event payload
`projection.upsert`	projection worker	UPSERT to projection table

Trace attributes

Each span includes contextual attributes:

Attribute	Example	Description
`causet.platform`	`my-platform`	Platform ID
`causet.application`	`concert-app`	Application ID
`causet.fork`	`production`	Fork ID
`causet.action`	`FOLLOW_ARTIST`	Action name (intent spans)
`causet.event_type`	`ARTIST_FOLLOWED`	Event type (event + projection spans)
`causet.entity_id`	`user-1`	Entity ID
`causet.event_id`	`evt_789xyz`	Ledger event ID
`causet.projection`	`user_following`	Projection name (worker spans)
`causet.correlation_id`	`corr-abc123`	Correlation ID threading through services

Configuration

Tracing is configured via OTel environment variables. Add these to each service:

# Enable OTel tracing
OTEL_SERVICE_NAME=causet-runtime-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://grafana-alloy:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_EXPORTER=otlp
 
# Sampling rate (1.0 = 100%, adjust for high-volume production)
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Grafana Alloy receives OTel traces and forwards them to Grafana Cloud Tempo. See infra/grafana-alloy/config.alloy for the Alloy pipeline configuration.

Sampling

At low traffic volumes (development, staging), use 100% sampling:

OTEL_TRACES_SAMPLER_ARG=1.0

In production with high intent throughput, sample at a lower rate to control trace storage costs:

OTEL_TRACES_SAMPLER_ARG=0.1   # 10% of traces

Use head-based sampling with parentbased_traceidratio — this ensures all spans within a sampled trace are captured, rather than cutting spans off mid-trace. Configure Grafana Alloy for tail-based sampling if you need to guarantee all errors are always captured regardless of the sampling rate.

Grafana Cloud Tempo

Traces are stored in Grafana Cloud Tempo (or a self-hosted Tempo instance). Access them in Grafana:

Go to Explore in Grafana
Select the Tempo data source
Search by:
- Trace ID — if you have the ID from a log line
- Service + span name — to find slow spans
- Tag filter — e.g., causet.event_id = "evt_789xyz" to find the trace for a specific event

Correlating traces with logs and metrics

All three signals share the correlationId / traceId:

Log line:     { "correlationId": "corr-abc123", "eventId": "evt_789xyz", ... }
Trace span:   causet.correlation_id=corr-abc123, traceId=abc123...
Metric label: (correlationId is not typically a metric label — too high cardinality)

In Grafana, use the Logs → Traces integration: a log line with a traceId field shows a link to the corresponding trace in Tempo.

Local tracing

In the Docker Compose stack, a local Tempo instance is included. Configure services with:

OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317

Access traces in local Grafana at http://localhost:3001 using the Tempo data source.

Troubleshooting with traces

Intent is slow: look at the intent.process span. If ledger.write is slow, check DB performance. If kafka.publish is slow, check Kafka broker health.

Projection is not updating: find the kafka.publish span to verify the event was published to Kafka. If it was published, find the projection.consume span on the worker. If the worker span is missing, check Kafka consumer group lag.

Projection UPSERT failed: the projection.upsert span will have an error status and the error message as a span attribute. This matches the structured log error for the same event.

What is Causet?