Migration Types

This guide provides detailed examples of different migration types. For the complete workflow overview, see Migrations & Planning.

Adding New Infrastructure Components

[features]olap = truestreaming_engine = trueworkflows = true

New OLAP Table or Materialized View

interface AnalyticsSchema {  id: string;  event_type: string;  timestamp: Date;  user_id: string;  value: number;} export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");

Migration Result: Creates ClickHouse table Analytics with all fields from AnalyticsSchema

[features]olap = true

New Stream

export const userEvents = new Stream<UserSchema>("UserEvents");export const systemEvents = new Stream<SystemEventSchema>("SystemEvents");

Migration Result: Creates Redpanda topics UserEvents and SystemEvents

[features]streaming_engine = true

Schema Modifications

Adding Fields

// Beforeinterface UserSchema {  id: string;  name: string;  email: string;} // Afterinterface UserSchema {  id: string;  name: string;  email: string;  age: number;  created_at: Date;  is_active: boolean;}

Migration Result: Adds age, created_at, and is_active columns to existing table

Removing Fields

// Beforeinterface UserSchema {  id: string;  name: string;  email: string;  age: number;  deprecated_field: string;  // Will be removed} // Afterinterface UserSchema {  id: string;  name: string;  email: string;  age: number;}

Migration Result: Drops deprecated_field column (data permanently lost)

Type Changes

// Beforeinterface UserSchema {  id: string;  name: string;  email: string;  score: number;  // Will change to string} // Afterinterface UserSchema {  id: string;  name: string;  email: string;  score: string;  // Changed from number}

Migration Result: Alters score column type (data converted if compatible)

Database Macros and Clustered Deployments

When defining physical infrastructure such as clustered deployments, you may need to leverage ClickHouse cluster macros. To support dynamic, macro-driven configurations in your models, Moose allows you to use {macro} syntax directly in your schema definition properties, such as a cluster name.

Normally, Moose validates cluster identifiers strictly. However, if an identifier is wrapped in braces (e.g. {cluster}), Moose treats it as a ClickHouse macro. It bypasses local static validation and defers evaluation to the ClickHouse server at execution time.

Using Cluster Macros in Migrations

Here is an example of what a cluster macro looks like in a generated migrations/plan.yaml file:

operations:  - CreateTable:      table:        name: user_events        columns:          - name: id            data_type: String            required: true        engine: ReplicatedMergeTree        cluster_name: "{cluster}"

When this migration is applied, the CLI dynamically generates DDL using ClickHouse's single-quote interpolation for macros:

CREATE TABLE `user_events` ON CLUSTER '{cluster}' (id String) ENGINE = ReplicatedMergeTree;

ClickHouse reads the single-quoted string '{cluster}' during execution, detects the {...} syntax, and substitutes it with the cluster macro value defined in its local macros.xml configuration file.

Removing Infrastructure

// Beforeexport const usersTable = new OlapTable<UserSchema>("Users");export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");export const deprecatedTable = new OlapTable<DeprecatedSchema>("Deprecated"); // After (remove deprecated table)export const usersTable = new OlapTable<UserSchema>("Users");export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");

Migration Result: Drops Deprecated table (all data lost)

Working with Local Infrastructure

There are two main ways to inspect your local infrastructure to see how your migrations are applied:

Using the CLI

Run moose ls to see the current state of your infrastructure:

# Verify object definitions
moose ls

Connecting to your local infrastructure

You can also connect directly to your local infrastructure to see the state of your infrastructure.

Connecting to ClickHouse

# Using clickhouse-client
clickhouse-client --host localhost --port 18123 --user panda --password pandapass --database local
 
# Using connection string
clickhouse-client "clickhouse://panda:pandapass@localhost:18123/local"

Connecting to Redpanda

# Using kafka-console-consumer
kafka-console-consumer --bootstrap-server localhost:19092 --topic UserEvents --from-beginning
 
# Using kafka-console-producer
kafka-console-producer --bootstrap-server localhost:19092 --topic UserEvents

Viewing Temporal Workflows

Navigate to http://localhost:8080 to view the Temporal UI and see registered workflows.

Gotchas:

Your dev server must be running to connect to your local infrastructure.

moose dev

Only the modules that you have enabled in your moose.config.toml will be included in your migrations:

[features]olap = true # Required for OLAP Tables and Materialized Viewsstreaming_engine = true # Required for Streamsworkflows = true # Required for Workflows and Tasks