Introduction to Blocks

The Blocks primitive is central to managing database operations in your Moose application. The Blocks primtive facilitates both setup and teardown operations, allowing you to define complex workflows that interact with your database. The setup and teardown parameters within the Blocks object accept arrays of SQL statements, which Moose executes sequentially to manage the state of your database.

Some key applications of Blocks are to facilitate:

• Data Transformation and Views: Transforming data and creating views or materialized views for pre-computing metrics and performing computationally intensive data analyses.
• Data Lifecycle Management: Implementing Time-To-Live (TTL) policies to automatically remove outdated data.
• Integration Testing: Setting up and tearing down test environments for easier integration testing of data pipelines.
• Post-Ingestion Data Manipulation: Manipulating data after it has been ingested into the database, such as hiding Personally Identifiable Information (PII) or transforming historical data.
• Access Control Management: Managing user access levels to different tables to ensure data security and compliance.

Core Concepts

File and Folder Conventions

In a Moose project, Blocks are stored in the /blocks folder. Each block is defined as a .ts.py file within this directory.

Blocks Definition

Within each block file, you instantiate a Blocks object, making it the default export of the file,class that serves as a container for SQL operations related to setting up and tearing down database structures. This is commonly used to manage materialized views, tables, and other database resources:

import { Blocks } from "@514labs/moose-lib";
 
export default {
  setup: [
    // Array of SQL statements for setting up database resources
    `CREATE TABLE my_table (...)`,
    `CREATE MATERIALIZED VIEW my_view AS SELECT ...`,
  ],
  teardown: [
    // Array of SQL statements for tearing down resources
    `DROP VIEW IF EXISTS my_view`,
    `DROP TABLE IF EXISTS my_table`,
  ],
} as Blocks;

from moose_lib import Blocks
 
block = Blocks(
    setup=[
        # Array of SQL statements for setting up database resources
        "CREATE TABLE my_table (...)",
        "CREATE MATERIALIZED VIEW my_view AS SELECT ...",
    ],
    teardown=[
        # Array of SQL statements for tearing down resources
        "DROP VIEW IF EXISTS my_view",
        "DROP TABLE IF EXISTS my_table",
    ],
)

Blocks Parameters

• setup: An array of SQL statements that will be executed when the setup phase is triggered. This parameter is typically used to define operations such as creating tables, indexes, and materialized views.
• teardown: An array of SQL statements that will be executed during the teardown phase. This parameter is typically used to drop or remove tables, views, or any other resources created during setup.

Orchestration

Moose orchestrates the execution of the SQL statements within the setup array during the startup of your Moose application. The SQL statements are executed sequentially, in the order they appear in the array. Similarly, the SQL statements in the teardown array are executed sequentially when you stop your Moose application, ensuring proper cleanup of the database resources.