Release Checklist

Follow this checklist for every production release. A failed item is a deployment blocker — do not proceed until it is resolved.

Pre-deploy checklist

1. Compile succeeds

Run causet build compile and verify it exits cleanly with no errors:

causet build compile --runtime ./your-app --out build/your-app-out

Expected output:

[INFO] Compiled N states, N events, N actions, N projections, N queries
[INFO] Output: build/your-app-out/causet.runtime.json
[INFO] Output: build/your-app-out/causet.projections.json

Any [ERROR] line is a blocker. Fix all compiler errors before continuing.

2. IR artifacts uploaded to S3

Both artifacts must be present on S3 at the versioned path before creating a release:

export IR_VERSION="2026.06.25-abc1234"
export BUCKET="causet-artifacts-prod"
 
# Upload
aws s3 cp build/your-app-out/causet.runtime.json \
  s3://$BUCKET/$IR_VERSION/causet.runtime.json
 
aws s3 cp build/your-app-out/causet.projections.json \
  s3://$BUCKET/$IR_VERSION/causet.projections.json
 
# Verify
aws s3 ls s3://$BUCKET/$IR_VERSION/
# Should show both files with non-zero sizes

3. Release created in causet-saas-cloud

Create a release record pointing to the uploaded IR version:

curl -X POST "https://saas.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/releases" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "irVersion": "2026.06.25-abc1234",
    "description": "Add venue_name to concert memory"
  }'

Confirm the release ID is returned and stored.

4. Event schemas validated

All source_events referenced in projections must exist in the event definitions. The compiler validates this, but double-check the IR manually for critical releases:

# Check that all projection source_events exist
cat build/your-app-out/causet.projections.json | \
  jq '[.projections[].sourceEvents[]] | unique | sort'
 
cat build/your-app-out/causet.runtime.json | \
  jq '[.events | keys[]] | sort'

Every event in the first list must appear in the second.

5. Projection tables verified in tenant schema

Connect to the projections database and verify that the expected tables exist in the tenant schema:

psql $PROJECTIONS_DB_URL -c "
  SELECT table_name
  FROM information_schema.tables
  WHERE table_schema = 'my_platform_my_app_production'
  ORDER BY table_name;
"

If tables are missing, the previous release was not deployed or DDL failed. Re-deploy the current release first.

6. Required indexes verified

The compiler generates index definitions in the IR. Verify that declared indexes exist on the tenant schema tables:

psql $PROJECTIONS_DB_URL -c "
  SELECT indexname, tablename, indexdef
  FROM pg_indexes
  WHERE schemaname = 'my_platform_my_app_production'
  ORDER BY tablename, indexname;
"

Missing indexes affect query performance but do not cause failures. Create them manually if the DDL migration skipped them.

7. Checkpoint store reachable

The projection worker commits offsets to Kafka. Verify the consumer group exists and can commit:

rpk group describe causet-projection-worker --brokers $KAFKA_BROKERS

Look for STABLE or EMPTY state. DEAD indicates the group has no active members and offsets may be stale — this is expected before the worker starts.

8. IR-driven DDL applied

After creating the release and deploying to the fork, confirm DDL was applied by checking for new columns:

psql $PROJECTIONS_DB_URL -c "
  SELECT column_name, data_type
  FROM information_schema.columns
  WHERE table_schema = 'my_platform_my_app_production'
    AND table_name = 'user_concert_memory'
  ORDER BY ordinal_position;
"

New columns from this release should appear. If they do not, check projection worker logs for DDL errors.

9. Fixture replay passed

Before deploying to production, replay your test event fixtures through a staging fork and verify projection output:

# Submit test events
./scripts/fixture-replay.sh --fork staging --fixtures tests/fixtures/
 
# Verify expected rows
psql $PROJECTIONS_DB_URL -c "
  SELECT * FROM my_platform_my_app_staging.user_concert_memory
  WHERE user_id = 'test-user-1';
"

Expected values should match the fixture expected output. Failures here indicate a bug in your derive expressions or mutation ops — fix and recompile before deploying.

10. DLQ checked

Check the dead-letter queue for any unresolved failures from the previous deploy:

rpk topic consume causet.projection-dlq.v1 \
  --brokers $KAFKA_BROKERS \
  --offset start \
  --num 10

Non-empty DLQ from the previous release means a projection handler was failing. Resolve those failures before deploying new code — a new deploy on top of unresolved DLQ failures makes forensics harder.

11. Retry policy configured

Verify that MAX_RETRY_ATTEMPTS and RETRY_BACKOFF_MS are set appropriately for the projection worker:

  • Production minimum: MAX_RETRY_ATTEMPTS=3, RETRY_BACKOFF_MS=1000
  • Transient failures (DB hiccup, network blip) are retried
  • Permanent failures (malformed payload, missing column) exhaust retries and go to DLQ

A retry policy that is too aggressive (high max retries, short backoff) can overwhelm a partially degraded database.

12. Observability configured

Confirm before every production deploy:

  • Grafana dashboards are accessible and receiving data
  • Prometheus scrape targets show the service as UP
  • Log ingestion pipeline is active (Grafana Alloy forwarding to Loki)
  • Alerts are configured for: projection lag > 30s, DLQ > 0, projection failures > 0

See Observability Overview.

13. Health endpoints passing

All services must report healthy before deploying:

curl -s https://runtime.yourdomain.com/actuator/health | jq .status
curl -s https://query.yourdomain.com/actuator/health | jq .status
curl -s https://saas.yourdomain.com/actuator/health | jq .status

All must return "UP". Do not deploy to a partially degraded environment.

14. Rollback plan documented

Before every production deploy, document:

  • The current active IR version (the version to roll back to if this deploy fails)
  • The rollback procedure: activate previous release in causet-saas-cloud
  • Who is responsible for monitoring the deploy and executing rollback if needed
# Record current active release
curl "https://saas.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/forks/production" \
  -H "Authorization: Bearer $TOKEN" | jq .activeRelease

15. Deployment ID set

Assign a deployment ID for this release. This ID is included in failure records and logs for correlation:

export DEPLOYMENT_ID="deploy-$(date +%Y%m%d%H%M%S)-$(git rev-parse --short HEAD)"
echo $DEPLOYMENT_ID

Include this in your deployment notes and Slack/incident channel announcement.

Deploy

With all pre-deploy checks passing:

# Activate the release on the production fork
curl -X POST "https://saas.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/forks/production/deploy" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "releaseId": "release-id-here" }'

In ECS, update the service to use the new task definition revision:

aws ecs update-service \
  --cluster causet-prod \
  --service causet-runtime-service \
  --task-definition causet-runtime-service:NEW_REVISION \
  --force-new-deployment

Post-deploy verification

1. Submit a test intent

curl -X POST "https://runtime.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/intents/execute" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "forkId": "production",
    "action": "FOLLOW_ARTIST",
    "entityId": "smoke-test-user",
    "payload": {
      "user_id": "smoke-test-user",
      "artist_id": "smoke-test-artist"
    }
  }'

Expect HTTP 200 with an event ID in the response.

2. Verify event in ledger

psql $CAUSET_DB_URL -c "
  SELECT event_type, entity_id, payload, created_at
  FROM causet.ledger_events
  WHERE entity_id = 'smoke-test-user'
  ORDER BY created_at DESC
  LIMIT 5;
"

The ARTIST_FOLLOWED event should appear.

3. Verify projection updated

Wait 1–3 seconds for the projection worker to process the event, then:

psql $PROJECTIONS_DB_URL -c "
  SELECT *
  FROM my_platform_my_app_production.user_following
  WHERE user_id = 'smoke-test-user';
"

A row should exist.

4. Execute a named query

curl -X POST "https://query.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/forks/production/queries/shows_for_followed_artists" \
  -H "Content-Type: application/json" \
  -d '{ "params": { "user_id": "smoke-test-user" } }'

Response should be valid JSON (empty array is acceptable — the query ran).

5. Check projection worker logs

# ECS: fetch recent logs
aws logs tail /ecs/causet-projection-worker --since 5m
 
# Look for errors
aws logs tail /ecs/causet-projection-worker --since 5m | grep -i "error\|failed\|dlq"

No new errors is the target state.

6. Check DLQ for new messages

# Get current DLQ message count
rpk topic describe causet.projection-dlq.v1 --brokers $KAFKA_BROKERS

If new messages appeared after deploy, a projection handler is failing. Investigate immediately.

7. Health endpoints still UP

Repeat the health endpoint checks from pre-deploy:

curl -s https://runtime.yourdomain.com/actuator/health | jq .status    # "UP"
curl -s https://query.yourdomain.com/actuator/health | jq .status      # "UP"
curl -s https://saas.yourdomain.com/actuator/health | jq .status       # "UP"

8. Monitor projection lag for 5 minutes

Open the Grafana dashboard and watch causet_projection_lag_seconds for the first 5 minutes post-deploy. Normal lag is under 5 seconds. Lag growing above 30 seconds indicates the projection worker is falling behind — check Kafka consumer lag and DB performance.

Rollback procedure

If post-deploy verification fails:

  1. Activate the previous release in causet-saas-cloud:
curl -X POST "https://saas.yourdomain.com/v1/platforms/{platformId}/applications/{appId}/forks/production/deploy" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{ "releaseId": "PREVIOUS_RELEASE_ID" }'
  1. Update ECS services to the previous task definition revision:
aws ecs update-service \
  --cluster causet-prod \
  --service causet-runtime-service \
  --task-definition causet-runtime-service:PREVIOUS_REVISION \
  --force-new-deployment

  1. Verify health endpoints return UP on the rolled-back services.

  2. Repeat post-deploy smoke tests against the previous release.

IR-driven DDL is additive — rolling back to the previous IR version is safe. Columns added by the failed release still exist in the database but are not written by the old IR.

What is Causet?