Configuring Data Models

While Moose provides sensible defaults for provisioning OLAP storage and infrastructure based on your Data Models, it also supports flexible customization for configurations like ingestion formats and storage options.

import { DataModelConfig, IngestionFormat } from "@514labs/moose-lib";
 
export const ExampleDataModelConfig: DataModelConfig<ExampleDataModel> = {
  ingestion: {
    format: IngestionFormat.JSON,
  },
  storage: {
    enabled: true,
    order_by_fields: ["example_timestamp"],
  },
};
 
export interface ExampleDataModel {
  example_timestamp: Date;
  example_string: string;
  example_number: number;
  example_boolean: boolean;
  example_string_array: string[];
}

from moose_lib import DataModelConfig, IngestionConfig, IngestionFormat, StorageConfig, moose_data_model
from dataclasses import dataclass
from datetime import datetime
 
example_config = DataModelConfig(
    ingestion=IngestionConfig(format=IngestionFormat.JSON),
    storage=StorageConfig(enabled=False, order_by_fields=["example_timestamp"])
)
 
@moose_data_model(user_activity_config)
@dataclass
class ExampleDataModel:
    example_timestamp: datetime.datetime
    example_string: str
    example_int: int
    example_float: float
    example_bool: bool
    example_string_array: list[str]
 

Config Options

DataModelConfig supports the following parameters:

• ingestion (IngestionConfig): Configures the ingestion endpoint for the data model.
• storage(StorageConfig): Configures the storage layer for the data model.

ingestion

Set the ingestion parameter to define how the API endpoint will receive data. The configuration supports the following options:

• format: Specifies the format of the data that the ingestion endpoint will accept.
  • Type: IngestionFormat
  • Default: IngestionFormat.JSON
  • Options:
    • IngestionFormat.JSON: Single JSON object expected in the body of the request.
    • IngestionFormat.JSON_ARRAY: Array of JSON objects expected in the body of the request.

Examples

export const UserActivityConfig: DataModelConfig<UserActivity> = {
    ingestion: {
        format: IngestionFormat.JSON,
    },
};
 
export interface UserActivity {
  ...
}

storage

The storage field configures the persistence layer for the data model. It supports the following options:

• enabled: A boolean value that determines whether to enable a table in the OLAP storage for the data model.

  • Type:boolean
  • Default: true
  • Description: When set to true, a table will be created in the OLAP storage for the data model, allowing data to be stored and queried. Setting it to false will disable this table creation.
  • Type: bool
  • Default: True
  • Description: When set to True, a table will be created in the OLAP storage for the data model, allowing data to be stored and queried. Setting it to False will disable this table creation.

• order_by_fields: An array of strings that adds an ORDER BY clause to the table in the OLAP storage. This is useful for optimizing queries that sort the Data Model by specific fields.

  • Type: string[]
  • Type: list[str]
  • Default: []
  • Description: This is an array of field names by which the table data should be ordered. The strings must be a subset of the fields in the Data Model.

Examples

export const UserActivityConfig: DataModelConfig<UserActivity> = {
  storage: {
    enabled: false,
  },
};