Ingest API
Use IngestApi to send records over HTTP to a destination Stream, for example from another service, webhook, or client app. Use Stream.send() when the producer can import your Moose SDK and Stream definition directly.
This page covers HTTP-based ingestion. To define the destination Stream, see Stream.
When to use Ingest APIs
| Use | Typical scenarios |
|---|---|
Stream.send() | An existing app in your monorepo or Moose workflows, where you can import the stream definition and other Moose objects directly. |
IngestApi | Services that push events into your system when you do not control that codebase or cannot use Moose objects or clients there directly. |
Basic usage
Use IngestApi with a destination Stream. Optionally, add a dead letter queue when invalid records should be retained for inspection or replay.
import { DeadLetterQueue, IngestApi, Stream } from "@514labs/moose-lib"; interface UserEvent { id: string; userId: string; timestamp: Date; eventType: string;} const userEvents = new Stream<UserEvent>("user_events");const userEventDLQ = new DeadLetterQueue<UserEvent>("user_events_dlq"); export const userEventsApi = new IngestApi<UserEvent>("user-events", { destination: userEvents, deadLetterQueue: userEventDLQ,});Request behavior
| Behavior | What happens |
|---|---|
| Accepted request | Moose validates the request body and writes accepted records to the destination stream. |
| Invalid request | Moose returns a 400 response when the payload does not match the model shape. |
| Dead letter queue | If deadLetterQueue or dead_letter_queue is configured, Moose retains failed records for inspection or replay. |
| ClickHouse writes | IngestApi writes to a stream, not directly to a table. If that stream has a destination table, records become queryable after stream-to-table sync succeeds. |
Optional fields with ClickHouse defaults
If your IngestApi model marks a field as optional but annotates a ClickHouse default, Moose treats:
- API request and Stream message: field is optional (you may omit it)
- ClickHouse table storage: field is required with a DEFAULT clause
Behavior: When the API/stream inserts into ClickHouse and the field is missing, ClickHouse sets it to the configured default value. This keeps request payloads simple while avoiding Nullable columns in storage.
Example:
field?: number & ClickHouseDefault<"18"> or WithDefault<number, "18">
Accept extra fields
Use this pattern when you do not fully control the upstream payload shape and want to accept undeclared fields without returning validation errors.
- You don't control the payload shape from an upstream service
- You want to pass through extra fields and handle them in downstream stream processing
TypeScript index signatures let you accept additional properties. When used with IngestApi or Stream, Moose accepts extra fields without returning validation errors:
import { IngestApi, Stream, Key, DateTime } from "@514labs/moose-lib"; // Input type with known fields + index signature for flexibilitytype UserEventInput = { timestamp: DateTime; eventName: string; userId: Key<string>; orgId?: string; // Index signature: accept any additional properties [key: string]: any;}; const inputStream = new Stream<UserEventInput>("UserEventInput"); // IngestApi accepts payloads with extra fields without validation errorsconst ingestApi = new IngestApi<UserEventInput>("user-events", { destination: inputStream,});- Known fields (
timestamp,eventName, etc.) are validated against their declared types - Additional fields matching the index signature are accepted by the API (no validation error returned)
- All fields, including extra fields, are passed through to downstream stream functions
- Extra fields can be extracted in your streaming function and stored in a JSON column on an OlapTable
Extracting Extra Fields
In your streaming function, use destructuring to separate known fields from extra fields:
userEventInputStream.addTransform(outputStream, (input) => { const { timestamp, eventName, userId, ...extraFields } = input; return { timestamp, eventName, userId, properties: extraFields };});Configuration options
| Option | What it does | Required |
|---|---|---|
destination | Stream that accepted records are written to. | Yes |
deadLetterQueue | Dead letter queue for failed records. | No |
version | Version label Moose uses when managing the API resource. | No |
path | Custom HTTP path for the endpoint. | No |
metadata.description | Optional description metadata. | No |
Related capabilities
- Stream for defining the destination stream
- OlapTable for defining the ClickHouse table a destination stream eventually syncs into
Stream.send()when the producer can import the stream directly- Schema registry when the destination stream publishes Schema Registry payloads
- Dead letter queues when failed records should be retained for recovery
- Sync to table when accepted records should land in ClickHouse through the destination stream