S3Queue)Use the S3Queue engine to automatically ingest data from S3 buckets as files are added:
S3Queue tables only process new files added to S3 after table creation. When used as a source for materialized views, no backfill occurs - the MV will only start populating as new files arrive. See the Materialized Views documentation for more details.
import { OlapTable, ClickHouseEngines } from '@514labs/moose-lib'; // Use direct configuration (S3Queue does not support orderByFields)export const s3Events = new OlapTable<S3Event>("s3_events", { engine: ClickHouseEngines.S3Queue, s3Path: "s3://my-bucket/data/*.json", format: "JSONEachRow", settings: { mode: "unordered", keeper_path: "/clickhouse/s3queue/events" }});import { OlapTable, ClickHouseEngines } from '@514labs/moose-lib'; // Use direct configuration (S3Queue does not support orderByFields)export const s3Events = new OlapTable<S3Event>("s3_events", { engine: ClickHouseEngines.S3Queue, s3Path: "s3://my-bucket/data/*.json", format: "JSONEachRow", settings: { mode: "unordered", keeper_path: "/clickhouse/s3queue/events" }});S3Queue is a streaming engine and does not support orderByFields or ORDER BY clauses. Configure only engine-specific parameters like s3Path, format, and settings.
S3Queue is a streaming engine and does not support orderByFields or ORDER BY clauses. Configure only engine-specific parameters like s3Path, format, and settings.
Security Risk: Hardcoding credentials in your code embeds them in Docker images and deployment artifacts, creating serious security vulnerabilities.
Solution: Use mooseRuntimeEnv for runtime credential resolution:
import { OlapTable, ClickHouseEngines, mooseRuntimeEnv } from '@514labs/moose-lib'; // ✅ RECOMMENDED: Runtime environment variable resolutionexport const secureS3Events = new OlapTable<S3Event>("s3_events", { engine: ClickHouseEngines.S3Queue, s3Path: "s3://my-bucket/data/*.json", format: "JSONEachRow", awsAccessKeyId: mooseRuntimeEnv.get("AWS_ACCESS_KEY_ID"), awsSecretAccessKey: mooseRuntimeEnv.get("AWS_SECRET_ACCESS_KEY"), settings: { mode: "unordered", keeper_path: "/clickhouse/s3queue/events" }});Then set environment variables:
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="your-secret-key"
moose prod upBenefits:
S3Queue requires ClickHouse 24.7+ and proper ZooKeeper/ClickHouse Keeper configuration for coordination between replicas. Files are processed exactly once across all replicas.