FiveonefourFiveonefour
Fiveonefour Docs
MooseStackTemplatesGuides
Release Notes
Source514
  1. MooseStack
  2. Engines
  3. S3

Direct S3 Access (S3)

Use the S3 engine for direct read/write access to S3 storage without streaming semantics:

S3Table.ts
import { OlapTable, ClickHouseEngines, mooseRuntimeEnv } from '@514labs/moose-lib'; // S3 table with credentials (recommended with mooseRuntimeEnv)export const s3Data = new OlapTable<DataRecord>("s3_data", {  engine: ClickHouseEngines.S3,  path: "s3://my-bucket/data/file.json",  format: "JSONEachRow",  awsAccessKeyId: mooseRuntimeEnv.get("AWS_ACCESS_KEY_ID"),  awsSecretAccessKey: mooseRuntimeEnv.get("AWS_SECRET_ACCESS_KEY"),  compression: "gzip"}); // Public S3 bucket (no authentication)export const publicS3 = new OlapTable<DataRecord>("public_s3", {  engine: ClickHouseEngines.S3,  path: "s3://public-bucket/data/*.parquet",  format: "Parquet",  noSign: true  // Use NOSIGN for public buckets});
S3 vs S3Queue
  • S3: Direct read/write access to S3 files. Use for batch processing or querying static data.

Both engines support the same credential management and format options.

  • S3Queue: Streaming engine that automatically processes new files as they arrive. Use for continuous data ingestion.
    • Overview
    Build a New App
    • 5 Minute Quickstart
    • Browse Templates
    • Existing ClickHouse
    Add to Existing App
    • Next.js
    • Fastify
    Fundamentals
    • Moose Runtime
    • MooseDev MCP
    • Data Modeling
    Moose Modules
    • Moose OLAP
    • Moose Streaming
    • Moose Workflows
    • Moose APIs & Web Apps
    Deployment & Lifecycle
    • Moose Migrate
    • Moose Deploy
    Reference
    • API Reference
    • Data Types
    • Table Engines
    • CLI
    • Configuration
    • Observability Metrics
    • Help
    • Release Notes
    Contribution
    • Documentation
    • Framework