Build an API to Fetch Most Popular Languages

Viewing typescript

switch to python

With the TopProgrammingLanguages Materialzied View created in the previous step, you can build a Consumption API to fetch data from this view and make it available for your other applications.

MooseTip:

A Consumption API enables you to create specific API endpoints that allow your applications to access and retrieve data. It serves as the final customization layer, where you can use query parameters to dynamically generate and execute SQL queries.

In this module, you will implement a Consumption API that will:

Dynamically generate and execute a query to extract analyzed data from the TopProgrammingLanguages materialized view
Return a ranked list of programming languages based on either:
- The number of repositories using the language
- The total number of kilobytes of code written in the language
- The average number of kilobytes of code written per repository using the language

The ranking criteria will depend on the Query Parameters specified in the request URL.

Generate a New File & Route for your API

Leverage the Moose CLI to Initialize a New Consumption API

Terminal

npx moose-cli consumption init rankedLanguages

This command will generate a new file named rankedLanguages.tsranked_languages.py inside the /apis folder of your project:

- rankedLanguages.ts

It will also create a corresponding API endpoint at: https://[YOUR_FORWARDING_URL]/consumption/rankedLanguageshttps://[YOUR_FORWARDING_URL]/consumption/ranked_languages. Making an HTTP GET request to this URL will execute the route handler function defined inside the file.

Consumption API Routing

Moose uses file-based routing conventions to map the files in the apis folder to corresponding API endpoints. This means that the file rankedLanguages.tsranked_languages.py will automatically map to the /consumption/rankedLanguages/consumption/ranked_languages endpoint.

Open and Inspect the File

Go to the /apis folder and open rankedLanguages.tsranked_languages.py. You should see the following code:

/app/apis/rankedLanguages.ts

import { ConsumptionUtil } from "@514labs-moose-lib";
// This file is where you can define your API templates for consuming your data
// All query_params are passed in as strings, and are used within the sql tag to parameterize your queries
export interface QueryParams {}
 
export default async function handle(
  {}: QueryParams,
  { client, sql }: ConsumptionUtil,
) {
  return client.query(sql``);
}

Consumption API Structure

Importing ConsumptionUtilMooseClient: This utility type from the Moose library gives you access to the database client (client) and SQL query tagged template literals (sql)

Default Export: The default export is an asynchronous function named handle, which takes two parameters:
- An object conforming to an empty QueryParams interface. You will define this interface in the next step to represent the different parameters to use in your dynamic query.
- The ConsumptionUtil object you imported

Define the `QueryParams` Interface

This interface should contain one key: rankBy. This key should be a string and have a value of either total_projects, total_repo_size_kb, or average_repo_size_kb. The specified value will determine the ranking criteria for the results.

app/apis/mostUsedLanguages.ts

import { ConsumptionUtil } from "@514labs/moose-lib";
 
export interface QueryParams {
  rankBy: string; // paste this into your file
}

These parameters are set in the request URL as key-value pairs, and their values will be injected into the query that you will execute against the database.

Implement the `handle()` Function

/app/apis/rankedLanguages.ts

import { ConsumptionUtil } from "@514labs/moose-lib";
 
export interface QueryParams {
  rankBy: string;
}
 
export default async function handle(
  { rankBy = "total_projects" }: QueryParams,
  { client, sql }: ConsumptionUtil
) {
  if (
    rankBy !== "total_projects" &&
    rankBy !== "total_repo_size_kb" &&
    rankBy !== "average_repo_size_kb"
  ) {
    throw new Error("Invalid rankBy value");
  }
  return client.query(
    sql`SELECT 
          language, 
          countMerge(total_projects) AS total_projects, 
          sumMerge(total_repo_size_kb) AS total_repo_size_kb, 
          avgMerge(average_repo_size_kb) AS average_repo_size_kb 
        FROM TopProgrammingLanguages 
        GROUP BY language 
        ORDER BY ${rankBy} DESC`
  );
}

/app/apis/ranked_languages.py

from moose_lib import MooseClient
from typing import Optional
 
def run(client: MooseClient, params, jwt_payload: Optional[dict] = None):
    rank_by = params.get("rank_by", ["total_projects"])[0] 
 
    if rank_by not in ["total_projects", "total_repo_size_kb", "avg_repo_size_kb"]:
        raise ValueError("Invalid rank_by value. Must be one of: total_projects, total_repo_size_kb, avg_repo_size_kb")
    
    query = f'''
    SELECT 
        language, 
        countMerge(total_projects) AS total_projects, 
        sumMerge(total_repo_size_kb) AS total_repo_size_kb, 
        avgMerge(avg_repo_size_kb) AS avg_repo_size_kb 
    FROM 
        TopProgrammingLanguages 
    GROUP BY 
        language 
    ORDER BY 
        {rank_by} DESC
    '''
    return client.query(query, { "rank_by": rank_by })

This function will return a ranked list of programming languages based on the rankByrank_by parameter. If an invalid value is passed in for rankByrank_by, the function will throw an error. If no value is passed in for rankByrank_by, the function will use the default value of total_projectstotal_projects.

Use the parameter ?rankBy=total_repo_size_kb to rank results by the total number of bytes written in each language across all the people who starred your repo.

Test Out the API Endpoint

Call the API From Your Terminal via `curl`

Terminal

curl -X GET https://[YOUR_FORWARDING_URL]/consumption/rankedLanguages

Don't Forget to Use Your Forwarding URL

Remember when you saved it earlier?

Test the Consumption API Endpoint Query Parameters via `curl`

Terminal

curl -X GET  https://[YOUR_FORWARDING_URL]/consumption/rankedLanguages\?rankBy\=total_repo_size_kb

(Optional) For a Better Experience, Use Your API Client of choice

VSCode Users

If you agreed to install the Recommended Extensions upon initially opening your Moose project in VSCode, you will have the Thunder Client extension autmatically installed in your project workspace. You can locate the extension via the lightning bolt icon.

Try changing the query parameters to see the results change. The query parameters do not need to be backslashed in the URL, for example:

https://[YOUR_FORWARDING_URL]/consumption/rankedLanguages?rankBy=total_repo_size_kb

Congrats on Completing the Tutorial!

You've just built an impressive real-time analytics service for GitHub star events using Moose! Let's recap your amazing journey:

You created a Data Model and set up a GitHub webhook to ingest star events in real-time.
You created a Streaming Function to process and enrich the raw star event data.
You implemented a Block to analyze user programming languages.
You built a Consumption API to fetch insights about the most used languages.

Aggregate Processed Events with Blocks Data Modeling