Sync to table

Use the destination option on a Stream when records from that stream should be batched and written into an OLAP table automatically.

This page covers the sync behavior after you set destination. To define the Stream itself, start with Stream.

Basic usage

app/stream.ts

import { OlapTable, Stream } from "@514labs/moose-lib"; interface Record {  id: string;  name: string;  timestamp: Date;  type: "foo" | "bar";} const table = new OlapTable<Record>("table"); const stream = new Stream<Record>("stream", {  destination: table,});

Runtime behavior

This section explains what Moose does after you set destination, not additional options you need to configure on the stream.

When a stream has a destination option set to an OlapTable, Moose starts a background sync worker that reads records from the stream topic, buffers them into batches, and writes those batches to the destination table in ClickHouse.

Aspect	What happens
Batch flush trigger	Moose flushes when a batch reaches `100000` records or when the 1 second flush interval ticks.
Query visibility	Rows become queryable only after the destination table receives a successful batch insert.
Failed inserts	If a ClickHouse insert fails, the batch stays queued and Moose retries it on later flush attempts.
Delivery guarantee	Delivery is at least once. Moose stores stream offsets after a batch insert succeeds.

Destination requirements

destination must point at an OlapTable instance.
The destination table schema must match the stream schema.
If you need to define the table first, start with Model Table.

Stream for defining the source stream
Define OLAP table for defining the destination OLAP table
Transform functions for reshaping records before they land in ClickHouse
Ingest API for HTTP producers that write into streams which then sync to tables

Sync to table

Use the destination option on a Stream when records from that stream should be batched and written into an OLAP table automatically.

This page covers the sync behavior after you set destination. To define the Stream itself, start with Stream.

Basic usage

app/stream.ts

import { OlapTable, Stream } from "@514labs/moose-lib"; interface Record {  id: string;  name: string;  timestamp: Date;  type: "foo" | "bar";} const table = new OlapTable<Record>("table"); const stream = new Stream<Record>("stream", {  destination: table,});

app/stream.ts

import { OlapTable, Stream } from "@514labs/moose-lib"; interface Record {  id: string;  name: string;  timestamp: Date;  type: "foo" | "bar";} const table = new OlapTable<Record>("table"); const stream = new Stream<Record>("stream", {  destination: table,});

Runtime behavior

This section explains what Moose does after you set destination, not additional options you need to configure on the stream.

Aspect	What happens
Batch flush trigger	Moose flushes when a batch reaches `100000` records or when the 1 second flush interval ticks.
Query visibility	Rows become queryable only after the destination table receives a successful batch insert.
Failed inserts	If a ClickHouse insert fails, the batch stays queued and Moose retries it on later flush attempts.
Delivery guarantee	Delivery is at least once. Moose stores stream offsets after a batch insert succeeds.

Destination requirements

destination must point at an OlapTable instance.
The destination table schema must match the stream schema.
If you need to define the table first, start with Model Table.

Stream for defining the source stream
Define OLAP table for defining the destination OLAP table
Transform functions for reshaping records before they land in ClickHouse
Ingest API for HTTP producers that write into streams which then sync to tables