ProjectionsSchema Versioning

Schema Versioning

Events are permanent records. Once an event is in the ledger, its payload schema is part of your permanent history. Changing payload fields affects not just new events but also existing events that projections and queries may reference.

Causet’s compiler validates all cross-references at build time — if a projection references event.some_field and that field is removed from the event’s payload, the build fails. This is intentional.

What is the IR

The IR (Intermediate Representation) is the compiled output of your .causet files. It has two artifacts:

  • causet.runtime.json — loaded by the runtime for rule evaluation
  • causet.projections.json — loaded by the projection worker and query service for DDL and query execution

Each release produces a unique IR version. The active IR version per fork is stored in the control plane. Deploying a new release updates the active IR version.

Additive changes are safe

The following changes are safe to make without a projection rebuild:

  • Add a new optional payload field to an event. Existing ledger events don’t have this field — projections that derive from it will get null for historical events. If that’s acceptable, add it and deploy.
  • Add a new projection. Deploy the new IR. The projection worker creates the table and populates it via Kafka. Backfill historical data by resetting the consumer offset.
  • Add a new query. Deploy the new IR. No schema changes needed — queries are compiled SQL templates, not DDL.
  • Add a column to an existing projection. Deploy the new IR. The projection worker applies the ALTER TABLE ADD COLUMN DDL. Existing rows get the column default. Backfill by rebuilding the projection.
  • Add a new optional input field to an action. Existing callers continue to work without the field.

Breaking changes

The following changes require explicit migration steps:

Removing or renaming an event payload field

# BEFORE
events:
  TICKET_PURCHASED:
    payload:
      ticket_id:   string
      show_id:     string
      price_cents: int       # ← removing this
 
# AFTER
events:
  TICKET_PURCHASED:
    payload:
      ticket_id: string
      show_id:   string
      amount:    int         # ← renamed

If any projection has derive: { price: event.price_cents }, the compiler will catch the dangling reference on the next build. You must update all projection derive: mappings before the build succeeds.

After updating projections: deploy, then rebuild affected projections (historical events don’t have amount, they have price_cents — decide how to handle the gap).

Changing a payload field type

Changing a field from int to string affects:

  • Expressions that compare or compute with the field
  • Projection columns derived from the field (column type change)
  • Queries that filter or aggregate on the field

Update the event declaration, all dependent projections (including their fields: types), and all dependent queries. Deploy the new IR. Apply DDL manually if needed. Rebuild affected projections.

Changing a projection column type

# BEFORE
fields:
  price_cents: TEXT    # ← was TEXT (bad)
 
# AFTER
fields:
  price_cents: BIGINT  # ← corrected to BIGINT

The projection worker applies column type DDL on deploy. Large tables may need a planned migration and rebuild. See Replay and Defining Projections.

Changing a projection’s primary key

This is a destructive change. The existing table’s primary key constraint must be dropped and recreated. The safest approach:

  1. Create a new projection with the new primary key (different table name).
  2. Deploy the new IR with both old and new projections active.
  3. Rebuild the new projection from history.
  4. Migrate application queries to the new projection.
  5. Remove the old projection in a subsequent release.

IR versioning workflow

  1. Modify .causet files.
  2. Compile — the compiler validates all cross-references. Fix any errors before proceeding.
  3. Upload artifacts (causet.runtime.json + causet.projections.json) to the platform.
  4. Create a release from the uploaded artifacts.
  5. Deploy the release to a fork — this updates the active IR version for that fork.
  6. Apply DDL — the projection worker applies any DDL changes declared in the new IR on startup.
  7. Rebuild projections if needed (new projection, column type change, derive expression change).

Active IR version per fork

Different forks can run different IR versions simultaneously. This enables:

  • Staging and production on different releases.
  • Canary releases — deploy to a test fork first.
  • Safe rollback — revert the active IR version without re-deploying the service.

The runtime and query service resolve the active IR version for each fork on every request (with caching). Rolling back a release is instant — no service restart required.

Rollback

Rolling back a release reverts the active IR version. Effect on DDL:

  • Columns added in the rolled-back release are not removed. DDL changes are applied additively. Rolling back does not drop columns or tables.
  • Queries revert to the previous compiled SQL templates. If the rolled-back release added a new query, that query is no longer available to callers.
  • Rules revert to the previous version. Intents are processed with the rolled-back rule set.

If a rollback is needed due to a schema issue (e.g. a column type change that caused errors), fix the DDL manually or tolerate the extra column until the issue is resolved in a forward release.

Compiler validation

The compiler validates:

  • All entity_expr references point to declared event payload fields.
  • All projection derive: references point to declared event payload fields.
  • All query from: and join: references point to declared projections.
  • All action emit references point to declared events.
  • All listener on: references point to declared events.
  • Reserved field names (type, ts, entity_id) are not used in event payloads.

If any validation fails, the build fails — no artifacts are produced. This prevents deploying a broken IR.

Production notes

Treat event payloads as public APIs. Downstream services, projections, and queries all depend on event payload structure. Additive changes are safe; removals and renames are breaking changes.

Version your event types for major changes. If you need to fundamentally change an event’s payload shape, create a new event type (e.g. TICKET_PURCHASED_V2) rather than modifying TICKET_PURCHASED. Run both in parallel during transition. This is more verbose but eliminates the historical data gap problem.

Document payload field deprecations before removal. Remove a field in two steps: mark it as deprecated in comments (Causet DSL supports # comments), then remove it in the next release after confirming no projections or queries reference it.

See also

PartitioningBest Practices