tc is a graph-based, stateless, serverless application & infrastructure composer.

tc defines, creates and manages serveless entities such as functions, mutations, events, routes, states, queues and channels. tc compiles a tree of entities defined in the filesystem as a topology. This composable, namespaced, sandboxed, recursive, versioned and isomorphic topology is called a Cloud Functor.

The word functor was popularized by Ocaml's parameterized modules. These modules, called functors, are first class. Cloud functors are similar in that they are treated as first class and are composable much like Ocaml's elegant modules.

Why tc ?

tc is a tool that empowers developers, architects and release engineers to build a serverless system that is simple to define and easy to evolve.

1. Developers should not be drowning in permissions and provider-specific (AWS, GCP etc) services. Instead, tc provides a framework for developers to focus on domain-specific functions and abstract entities.
2. Architects should not be defining systems that are hard to implement or disconnected from the system definition. Instead, tc's topology is the system definition. The definition is representative of the entire system to a large extent with most of it inferred by tc.
3. Release engineers should not be managing manifests manually. Instead, tc provides a mechanism to deploy collection of namespaced topologies as an atomic unit. Canarys, A/B testing and rollbacks are much simpler to configure using tc

Key features of functors using tc

1. Composable Entities

At it's core, tc provides 7 entities (functions, events, mutations, queues, routes, states and channels) that are agnostic to any cloud provider. These entities are core primitives to define the topology of any serverless system. For example, consider the following topology definition:

name: example

routes:
  myposts:
    path: /api/posts
    method: GET
    function: bar
    event: MyEvent

events:
  consumes:
    MyEvent:
      function: foo
      channel: room1

channels:
  room1:
    handler: default

functions:
  remote:
    foo: github.com/bar/bar
  local:
    bar: ./bar

Now, /api/posts route calls function bar and generates an event MyEvent which are handled by functions that are locally defined (subdirectories) or remote (git repos). In this example, the event finally triggers a channel notification with the event's payload. We just defined the flow without specifying anything about infrastructure, permissions or the provider. This definition is good enough to render it in the cloud as services, as architecture diagrams and release manifests.

tc compile maps these entities to the provider's serverless constructs. If the provider is AWS (default), tc maps routes to API Gateway, events to Eventbridge, functions to either Lambda or ECS Fargate, channels to Appsync Events, mutations to Appsync Graphql and queues to SQS

2. Namespacing

If we run tc compile in the directory containing the above topology (topology.yml), we see that all the entities are namespaced. This implies there is room for several foo,bar or MyEvent entities in another topology. This also encourages developers to name the entities succinctly similar to function names in a module. With namespacing comes the benefit of having a single version of the namespace and thereby avoiding the need to manage the versions of sub-components.

3. Sandboxing

You can create a sandbox of this topology in the cloud (AWS is the default provider) using

tc create -s <sandbox-name> -e <aws-profile>

and can invoke (tc invoke -s sandbox -e env -p payload.json) this topology. This sandbox is also versioned and we can update specific entities or components in it. Sandboxing is fundamental to canary-based routing and deploys. tc create also knows how to build the functions, implicitly, for various language runtimes.

tc update -s sandbox -e env -c events|routes|mutations|functions|flow

4. Inference

tc compile generates a lot of the infrastructure (permissions, default configurations etc) boilerplate needed for the configured provider. Think of infrastructure as Types in a dynamic programming language. We can override the defaults or inferred configurations separate from the topology definition. For example we can have a repository layout as follows:

services/<topology>/<function>
infrastructure/<topology>/vars/<function>.json
infrastructure/<topology>/roles/<function>.json

This encourages developers to not leak infrastructure into domain-specific code or topology definition and vice versa. A topology definition could be rendered in with different infrastructure providers.

5. Recursive Topology

Functors can be created at any level in the code repository's heirarchy. They are like fractals where we can zoom in or out. For example, consider the following retail order management topology:

order/
|-- payment
|   |-- other-payment-processor
|   |   `-- handler.py
|   |-- stripe
|   |   |-- handler
|   |   `-- topology.yml
|   `-- topology.yml
`-- topology.yml

There are two sub-topologies in the root topology. order, payment and stripe are valid topologies. tc can create and manage sandboxes at any level preserving the integrity of the overall graph.

cd order
tc create -s <sandbox> -e <env> --recursive

This feature helps evolve the system and test individual nodes in isolation.

6. Isomorphic Topology

The output of tc compile is a self-contained, templated topology (or manifest) that can be rendered in any sandbox. The template variables are specific to the provider, sandbox and configuration. When we create (tc create) the sandbox with this templated topology, it implicitly resolves it by querying the provider. We can write custom resolvers to resolve these template variables by querying the configured provider (AWS, GCP etc).

tc compile | tc resolve -s sandbox -e env | tc create

We can replace the resolver with sed or a template renderer with values from ENV variables, SSM parameter store, Vault etc. For example:

tc compile | sed -i 's/{{API_GATEWAY}}/my-gateway/g' |tc create

The resolver can also be written in any language that is easy to use and query the provider, efficiently. The output of the compiler, the resolver and the sandbox's metadata as seen above are isomorphic. They are structurally the same and can be diffed like git-diff. Diffable infrastructure without having external state is a simple yet powerful feature.

This is all too abstract you say ? It is! Let's Get Started

Installation

Download the executable for your OS

Building your own

tc is written in Rust.

If you prefer to build tc yourself, install rustc/cargo.

Install Cargo/Rust https://www.rust-lang.org/tools/install

cd tc
cargo build --release
sudo mv target/release/tc /usr/local/bin/tc

Getting started

• A basic function
• Namespacing your functions
• Create your sandbox
• Configuring infrastructure

A basic function

Namespacing your functions

Create your sandbox

Configuring infrastructure

Compiler

tc compile does the following:

1. Discovers functions recursively in the current directory.
2. Generates build instructions for the discovered functions.
3. Interns remote, shared and local functions
4. Reads the topology.yml file and validates it using input specification
5. Generates the target representations for these entities specific to a provider
6. Generates graphql output for mutations definition in topology.yml
7. Transpiles flow definitions to stepfn etc.
8. Generates checksum of all function directories
9. Detects circular flows

To generate the topology in curent directory

tc compile [--recursive]

To generate tree of all functions

cd examples/apps/retail
tc compile -c functions -f tree

retail
├╌╌ payment
┆   ├╌╌ payment_stripe_{{sandbox}}
┆   ┆   ├╌╌ python3.10
┆   ┆   ├╌╌ provided
┆   ┆   └╌╌
┆   └╌╌ payment_klarna_{{sandbox}}
┆       ├╌╌ python3.10
┆       ├╌╌ provided
┆       └╌╌
├╌╌ pricing
┆   └╌╌ pricing_resolver_{{sandbox}}
┆       ├╌╌ python3.10
┆       ├╌╌ provided
┆       └╌╌

Builder

• Inline
• Layer
• Image
  • External parent image
• Slab
• Library
• Extension
• Recursive Builds

tc has a sophisticated builder that can build different kinds of artifacts with various language runtimes (Clojure, Janet, Rust, Ruby, Python, Node)

In the simplest case, when there are no dependencies in a function, we can specify how the code is packed (zipped) as follows in function.json:

{
  "name": "simple-function",
  "runtime": {
    "lang": "python3.10",
    "package_type": "zip",
    "handler": "handler.handler",
  },
  "build": {
    "command": "zip -9 lambda.zip *.py",
    "kind": "Code"
  }
}

Example

and then tc create -s <sandbox> -e <env> builds this function using the given command and creates it in the given sandbox and env.

Inline

The above is a pretty trivial example and it gets complicated as we start adding more dependencies. If the dependencies are reasonably small (< 50MB), we can inline those in the code's artifact (lambda.zip).

{
  "name": "python-inline-example",
  "runtime": {
    "lang": "python3.12",
    "package_type": "zip",
    "handler": "handler.handler",
    "layers": []
  },
  "build": {
    "kind": "Inline",
    "command": "zip -9 -q lambda.zip *.py"
  },
  "test": {
    "fixture": "python run-fixture.py",
    "command": "poetry test"
  }

}

Example

tc create -s <sandbox> -e <env> will implicitly build the artifact with inlined deps and create the function in the given sandbox and env. The dependencies are typically in lib/ including shared objects (.so files).

Layer

If inline build is heavy, we can try to layer the dependencies:

{
  "name": "ppd",
  "description": "my python layer",
  "runtime": {
    "lang": "python3.10",
    "package_type": "zip",
    "handler": "handler.handler",
    "layers": ["ppd-layer"]
  },
  "build": {
    "pre": [
      "yum install -y git",
      "yum install -y gcc gcc-c++"
    ],
    "kind": "Layer",

  }
}

Note that we have specified the list of layers the function uses. The layer itself can be built independent of the function, unlike Inline build kind.

tc build --kind layer
tc publish --name ppd-layer

We can then create or update the function with this layer. At times, we may want to update just the layers in an existing sandboxed function

tc update -s <sandbox> -e <env> -c layers

Image

While Layer and Inline build kind should suffice to pack most dependencies, there are cases where 250MB is not good enough. Container Image kind is a good option. However, building the deps and updating just the code is challenging using pure docker as you need to know the sequence to build. tc provides a mechanism to build a tree of images. For example:

{
  "name": "python-image-tree-example",
  "runtime": {
    "lang": "python3.10",
    "package_type": "image",
    "handler": "handler.handler"
  },

  "build": {
    "kind": "Image",
    "images": {
      "base": {
		  "version": "0.1.1",
		  "commands": [
			  "yum install -y git wget unzip",
			  "yum install -y gcc gcc-c++ libXext libSM libXrender"
		  ]
      },
      "code": {
		  "parent": "base",
		  "commands": []
      }
    }
  }
}

Example

In the above example, we define the base image with dependencies and code image that packs just the code. Note that code references base as the parent. Effectively, we can build a tree of images (say base dependencies, models, assets and code). These images can be built at any point in the lifecycle of the function. To build the base image do:

tc build --image base --publish

When --publish is specified, it publishes to the configured ECR repo [See Configuration]. Alternatively, TC_ECR_REPO env variable can be specified to override the config. The value of variable is the ECR repo URI

With python functions, the image can be built either by having a 'requirements.txt' file in the function directory or a pyproject.toml. tc build works with requirements.txt and poetry. See poetry example

When all "parent" images have been built, tc create will create the code image just-in-time. The tag is the SHA1 checksum of the function directory. The code tag is typically of the format "{{repo}}/code:req-0d4043e5ae0ebc83f486ff26e8e30f3bd404b707""

We can also optionally build the code image.

tc build --image code --publish

Note that the child image uses the parent's version of the image as specified in the parent's block

External parent image

At times, we may need to use a parent image that is shared and defined in another function or build. The following function definition is an example that shows how to specify a parent URI in code image-spec.

{
  "name": "req-external-example",
  "description": "With external parent",
  "runtime": {
    "lang": "python3.10",
    "package_type": "image",
    "handler": "handler.handler"
  },

  "build": {
    "kind": "Image",
    "images": {
      "code": {
	"parent": "{{repo}}/base:req-0.1.1",
	"commands": []
      }
    }
  }
}

Example

parent in the code image-spec is an URI. This is also a way to pin the parent image.

Slab

slab is an abstraction for building depedencies, assets and serving it via a network filesystem (EFS). An example function with slab looks like:

{
  "name": "python-example-snap",
  "description": "example function",
  "runtime": {
    "lang": "python3.12",
    "package_type": "zip",
    "mount_fs": true,
    "handler": "handler.handler",
    "layers": []
  },
  "build": {
    "kind": "slab"
  }
  "test": {
    "fixture": "python run-fixture.py",
    "command": "poetry test"
  }

}

tc build --kind slab --publish

This publishes the slab to EFS as configured (See Configuration)

Library

A library is a kind of build that recursively packs a collection of directories to serve as a single library in the target runtime.

For example, let's say we have the following directory structure

lib/
|-- bar
|   `-- lib.rb
|-- baz
|   `-- lib.rb
`-- foo
    `-- lib.rb

We can pack this as a library and publish it as a layer or a node in the image-tree. By default, tc publishes it as a layer.

cd lib
tc build --kind library --name mylib --publish --lang ruby

Why can't this just be of kind layer ? Layers typically have the dependencies resolved. Library is just standalone.

Extension

Lambda extensions are like sidecars that intercept the input/output payload events and can do arbitrary processing on them.

tc build --kind extension

Recursive Builds

To traverse through the topology and build the depedencies or code in parallel, do the following:

tc build [-kind code|image|layer] --recursive --publish

Resolver

WIP

1. Resolves the environment variables from stores ssm:/, s3:/
2. Renders templates
3. Resolves versions

Deployer

Creating a Sandbox

cd services/extraction
tc create [--sandbox SANDBOX] [-e ENV]

Incremental updates

While developing, we often need to incrementally deploy certain components without recreating the entire topology. tc provides an update command that updates given component(s).

To update the code for a function (say page-mapper) in the current directory

tc update --sandbox test -e dev-af -c page-mapper

To update the IAM roles and policies

tc update --sandbox test -e dev-af -c roles

To update the eventbridge event rules:

tc update --sandbox test -e dev-af -c events

To update the environment variables or runtime parameters. Usually these are defined in infrastucture/tc//vars dir

tc update --sandbox test -e dev-af -c vars

To build and update layers

tc update --sandbox test -e dev-af -c layers

To update the Statemachine flow

tc update --sandbox test -e dev-af -c flow

To update tags across stepfns, lambdas, roles, policies, eventbridge rules etc

tc update --sandbox test -e dev-af -c tags

To update logging and tracing config

tc update --sandbox test -e dev-af -c logs

Invoker

• Specifying Payload
  • Invoking Events and Lambdas

Specifying Payload

To simply invoke a functor

tc invoke --sandbox main --env dev

By default, tc picks up a payload.json file in the current directory. You could optionally specify a payload file

tc invoke --sandbox main --env dev --payload payload.json

or via stdin

cat payload.json | tc invoke --sandbox main --env dev

or as a param

tc invoke --sandbox main --env dev --payload '{"data": "foo"}'

Invoking Events and Lambdas

By default, tc invokes a stepfn. We can also invoke a lambda or trigger an Eventbridge event

tc invoke --kind lambda -e dev --payload '{"data"...}'
tc invoke --kind event -e dev --payload '{"data"...}'

Releaser

• Versioning
• Changelog

WIP

Versioning

Changelog

Emulator

• Lambdas
• Stepfunctions

Lambdas

To emulate the Lambda Runtime environment. The following command spins up a docker container with the defined layers in function.json, sets up the paths, environment variables, AWS access, local code and runtime parameters (mem, handlers etc)

cd <function-dir>
tc emulate

To run in foreground

tc emulate

You can now invoke a payload locally with this emulator

tc invoke --local [--payload <payload.json | json-str>]

Stepfunctions

tc also provides a stepfunction emulator. In your top-level topology directory, do:

tc emulate

This spins up a container and runs the emulator on http://localhost:8083

Details to follow on creating and executing [wip]

Bootstrapper

To create or update the base roles:

tc bootstrap --roles

Updating role tc-base-lambda-role
Updating role tc-base-sfn-role
Updating role tc-base-event-role
Updating role tc-base-api-role
Updating role tc-base-appsync-role

Grokker

WIP

Inspector

tc provides a lightweight http-based app to inspect the topologies. This is still experimental.

To run the inspector, run tc inspect --trace in the root topology directory. For example:

cd examples/patterns
tc inspect --trace

Topology Specification

topology.yml

name: <namespace>
infra: <infra-path>

nodes:
  ignore: [<path>]
  dirs: [<path>]

functions:
  shared: [<rel path>]
  remote: [<git url>]
  dirs: [<dir path>]

events:
  EventName:
    producer: <String>
	doc_only: <false>
	nth: <sequence int>
	filter: <String>
	rule_name: <String>
    functions: [<String>]
    function: <String>
    mutation: <String>
    channel: <String>
    queue: <String>
    state: <String>

routes:
  Path:
    gateway: <String>
    authorizer: <String>
    method: <POST|GET|DELETE>
	path: <String>
    sync: <true>
    request_template: <String>
    response_template: <String>
    stage: <String>
    stage_variables: <String>
    function: <String>
    state: <String>
    queue: <String>

channels:
  ChannelName:
    function: <String>
    event: <String>

mutations:
  MutationName:
    function: <String>

queues:
  QueueName:
    function: <String>

states: ./states.json | <definition>  [optional]

infra is either an absolute or relative path to the infrastructure configs (vars, roles etc). This field is optional and tc tries best to discover the infrastructure path in the current git repo.

events, routes, functions, mutations, channels and flow are optional.

flow can contain a path to a step-function definition or an inline definition. tc automatically namespaces any inlined or external flow definition.

Function Specification

function.json file in the function directory is optional. tc infers the language and build instructions from the function code. However, for custom options, add a function.json that looks like the following

{
  "name": String,
  "runtime": RuntimeSpec,
  "build": BuildSpec,
  "infra": InfraSpec,
  "test": TestSpec
}

RuntimeSpec

BuildSpec

JSON Spec

{
  "name": "string",
  // Optional
  "dir": "string",
  // Optional
  "description": "string",
  // Optional
  "namespace": "string",
  // Optional
  "fqn": "string",
  // Optional
  "layer_name": "string",
  // Optional
  "version": "string",
  // Optional
  "revision": "string",
  // Optional
  "runtime": {
    "lang": "Python39" | "Python310" | "Python311" | "Python312" | "Python313" | "Ruby32" | "Java21" | "Rust" | "Node22" | "Node20",
    "handler": "string",
    "package_type": "string",
    // Optional
    "uri": "string",
    // Optional
    "mount_fs": true,
    // Optional
    "snapstart": true,
    "layers": [
      "string",
      /* ... */
    ],
    "extensions": [
      "string",
      /* ... */
    ]
  },
  // Optional
  "build": {
    "kind": "Code" | "Inline" | "Layer" | "Slab" | "Library" | "Extension" | "Runtime" | "Image",
    "pre": [
      "dnf install git -yy",
      /* ... */
    ],
    "post": [
      "string",
      /* ... */
    ],
    // Command to use when build kind is Code
    "command": "zip -9 lambda.zip *.py",
    "images": {
      "string": {
        // Optional
        "dir": "string",
        // Optional
        "parent": "string",
        // Optional
        "version": "string",
        "commands": [
          "string",
          /* ... */
        ]
      },
      /* ... */
    },
    "layers": {
      "string": {
        "commands": [
          "string",
          /* ... */
        ]
      },
      /* ... */
    }
  },
  // Optional
  "infra": {
    "dir": "string",
    // Optional
    "vars_file": "string",
    "role": {
      "name": "string",
      "path": "string"
    }
  }
}

Infrastructure Spec

Runtime Variables

Default Path: infrastructure/tc//vars/.json Override: infra.vars_file in function.json

{
  // Optional
  "memory_size": 123,
  // Optional
  "timeout": 123,
  // Optional
  "image_uri": "string",
  // Optional
  "provisioned_concurrency": 123,
  // Optional
  "reserved_concurrency": 123,
  // Optional
  "environment": {
    "string": "string",
    /* ... */
  },
  // Optional
  "network": {
    "subnets": [
      "string",
      /* ... */
    ],
    "security_groups": [
      "string",
      /* ... */
    ]
  },
  // Optional
  "filesystem": {
    "arn": "string",
    "mount_point": "string"
  },
  // Optional
  "tags": {
    "string": "string",
    /* ... */
  }
}

Roles

Config Specification

The following is a sample config file that you can place in your infrastructure root (infrastructure/tc/) or the path in TC_CONFIG_PATH. The configs have sections specific to the module and are optional with sane defaults.

compiler:
  verify: false
  graph_depth: 4
  default_infra_path: infrastructure/tc

resolver:
  incremental: false
  cache: false
  stable_sandbox: stable
  layer_promotions: true

deployer:
  guard_stable_updates: true
  rolling: false

builder:
  parallel: false
  autosplit: true
  max_package_size: 50000000
  ml_builder: true

aws:
  eventbridge:
    bus: EVENT_BUS
    rule_prefix: tc-
    default_role: tc-base-event-role
    default_region: us-west-2
    sandboxes: ["stable"]

  ecs:
    subnets: ["subnet-tag"]
    cluster: my-cluster

  stepfunction:
    default_role: tc-base-sfn-role
    default_region: us-west-2

  lambda:
    default_timeout: 180
    default_role: tc-base-lambda-role
    default_region: us-west-2
    layers_profile: LAYER_AWS_PROFILE
    fs_mountpoint: /mnt/assets

  api_gateway:
    api_name: GATEWAY_NAME
    default_region: us-west-2

Environment variables

tc uses special environment variables as feature bits and config overrides. The following is the list of TC environment variables:

TC_DIR

We don't have to always be in the topology or function directory to run a contextual tc command. TC_DIR env var sets the directory context

TC_DIR=/path/to/services/fubar tc create -s sandbox -e env

TC_USE_STABLE_LAYERS

At times we may need to use stable layers in non-stable sandboxes. This env variable allows us to use stable layers

TC_USE_STABLE_LAYERS=1 tc create -s sandbox -e env

TC_USE_SHARED_DEPS

This feature flag uses common deps (in EFS) instead of function-specific deps.

TC_USE_SHARED_DEPS=1 tc create -s sandbox -e env

TC_FORCE_BUILD

Tries various fallback strategies to build layers. One of the strategies is to build locally instead of a docker container. Another fallback is to use a specific version of python even if the transitive dependencies need specific version of Ruby or Python

TC_FORCE_BUILD=1 tc build --trace

TC_FORCE_DEPLOY

To create or update stable sandboxes (which are prohibited by default), use this var to override.

TC_FORCE_DEPLOY=1 tc create -s sandbox -e env

TC_UPDATE_METADATA

To update deploy metadata to a dynamodb table (the only stateful stuff in TC) for stable sandboxes

TC_UPDATE_METADATA=1 tc create -s staging -e env

TC_ECS_CLUSTER

Use this to override the ECS Cluster name

TC_ECS_CLUSTER=my-cluster tc create -s sandbox -e env

TC_USE_DEV_EFS

Experimental EFS with deduped deps and models

TC_USE_DEV_EFS=1 tc create ...

TC_SANDBOX

Set this to have a fixed sandbox name for all your sandboxes

TC_SANDBOX=my-branch tc create -e env

CLI Reference

This document contains the help content for the tc command-line program.

Command Overview:

• tc↴
• tc bootstrap↴
• tc build↴
• tc cache↴
• tc compile↴
• tc config↴
• tc create↴
• tc delete↴
• tc freeze↴
• tc emulate↴
• tc inspect↴
• tc invoke↴
• tc list↴
• tc publish↴
• tc resolve↴
• tc route↴
• tc scaffold↴
• tc test↴
• tc tag↴
• tc unfreeze↴
• tc update↴
• tc upgrade↴
• tc version↴
• tc doc↴

tc

Usage: tc <COMMAND>

Subcommands:

• bootstrap — Bootstrap IAM roles, extensions etc
• build — Build layers, extensions and pack function code
• cache — List or clear resolver cache
• compile — Compile a Topology
• config — Show config
• create — Create a sandboxed topology
• delete — Delete a sandboxed topology
• freeze — Freeze a sandbox and make it immutable
• emulate — Emulate Runtime environments
• inspect — Inspect via browser
• invoke — Invoke a topology synchronously or asynchronously
• list — List created entities
• publish — Publish layers
• resolve — Resolve a topology from functions, events, states description
• route — Route events to functors
• scaffold — Scaffold roles and infra vars
• test — Run unit tests for functions in the topology dir
• tag — Create semver tags scoped by a topology
• unfreeze — Unfreeze a sandbox and make it mutable
• update — Update components
• upgrade — upgrade tc version
• version — display current tc version
• doc — Generate documentation

tc bootstrap

Bootstrap IAM roles, extensions etc

Usage: tc bootstrap [OPTIONS]

Options:

• -R, --role <ROLE>
• -e, --profile <PROFILE>
• --create
• --delete
• --show
• -t, --trace

tc build

Build layers, extensions and pack function code

Usage: tc build [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -k, --kind <KIND>
• -n, --name <NAME>
• -i, --image <IMAGE>
• --clean
• -r, --recursive
• --dirty
• --merge
• --split
• --task <TASK>
• -t, --trace
• -p, --publish

tc cache

List or clear resolver cache

Usage: tc cache [OPTIONS]

Options:

• --clear
• --list
• -n, --namespace <NAMESPACE>
• -e, --env <ENV>
• -s, --sandbox <SANDBOX>
• -t, --trace

tc compile

Compile a Topology

Usage: tc compile [OPTIONS]

Options:

• --versions
• -r, --recursive
• -c, --component <COMPONENT>
• -f, --format <FORMAT>
• -t, --trace

tc config

Show config

Usage: tc config

tc create

Create a sandboxed topology

Usage: tc create [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -T, --topology <TOPOLOGY>
• --notify
• -r, --recursive
• --no-cache
• -t, --trace

tc delete

Delete a sandboxed topology

Usage: tc delete [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -c, --component <COMPONENT>
• -r, --recursive
• --no-cache
• -t, --trace

tc freeze

Freeze a sandbox and make it immutable

Usage: tc freeze [OPTIONS] --sandbox <SANDBOX>

Options:

• -d, --service <SERVICE>
• -e, --profile <PROFILE>
• -s, --sandbox <SANDBOX>
• --all
• -t, --trace

tc emulate

Emulate Runtime environments

Usage: tc emulate [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -s, --shell
• -d, --dev
• -t, --trace

tc inspect

Inspect via browser

Usage: tc inspect [OPTIONS]

Options:

• -t, --trace

tc invoke

Invoke a topology synchronously or asynchronously

Usage: tc invoke [OPTIONS]

Options:

• -p, --payload <PAYLOAD>
• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -n, --name <NAME>
• -S, --step <STEP>
• -k, --kind <KIND>
• --local
• --dumb
• -t, --trace

tc list

List created entities

Usage: tc list [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -r, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -c, --component <COMPONENT>
• -f, --format <FORMAT>
• -t, --trace

tc publish

Publish layers

Usage: tc publish [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -k, --kind <KIND>
• --name <NAME>
• --list
• --promote
• --demote
• --download
• --version <VERSION>
• --task <TASK>
• --target <TARGET>
• -t, --trace

tc resolve

Resolve a topology from functions, events, states description

Usage: tc resolve [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -c, --component <COMPONENT>
• -q, --quiet
• -r, --recursive
• --diff
• --no-cache
• -t, --trace

tc route

Route events to functors

Usage: tc route [OPTIONS] --service <SERVICE>

Options:

• -e, --profile <PROFILE>
• -E, --event <EVENT>
• -s, --sandbox <SANDBOX>
• -S, --service <SERVICE>
• -r, --rule <RULE>
• --list
• -t, --trace

tc scaffold

Scaffold roles and infra vars

Usage: tc scaffold

tc test

Run unit tests for functions in the topology dir

Usage: tc test [OPTIONS]

Options:

• -d, --dir <DIR>
• -l, --lang <LANG>
• --with-deps
• -t, --trace

tc tag

Create semver tags scoped by a topology

Usage: tc tag [OPTIONS]

Options:

• -n, --next <NEXT>
• -s, --service <SERVICE>
• --dry-run
• --push
• --unwind
• -S, --suffix <SUFFIX>
• -t, --trace

tc unfreeze

Unfreeze a sandbox and make it mutable

Usage: tc unfreeze [OPTIONS] --sandbox <SANDBOX>

Options:

• -d, --service <SERVICE>
• -e, --profile <PROFILE>
• -s, --sandbox <SANDBOX>
• --all
• -t, --trace

tc update

Update components

Usage: tc update [OPTIONS]

Options:

• -e, --profile <PROFILE>
• -R, --role <ROLE>
• -s, --sandbox <SANDBOX>
• -c, --component <COMPONENT>
• -a, --asset <ASSET>
• --notify
• -r, --recursive
• --no-cache
• -t, --trace

tc upgrade

upgrade tc version

Usage: tc upgrade [OPTIONS]

Options:

• -v, --version <VERSION>
• -t, --trace

tc version

display current tc version

Usage: tc version

tc doc

Generate documentation

Usage: tc doc [OPTIONS]

Options:

• -s, --spec <SPEC>

Library

WIP

Patterns

• 1. Request-Response
• 2. Request-async-Response
• 3. Request-Queue
• 4. Request-Event-Routing
• 5. Events Choreography
• 6. Event Filters
• 7. Request Stepfunction
• 8. Request Map

WIP

1. Request-Response

name: request-response

routes:
  get-user:
    gateway: api-test
    kind: http
    method: GET
	sync: true
    path: "/api/user"
    function: fetcher

This is a simple topology which creates a HTTP route backed by a function. In AWS, this creates the API Gateway configuration and creates the lambda function with the right permissions.

2. Request-async-Response

name: 06-request-async-response

routes:
  post-message:
    gateway: api-test
    kind: http
    timeout: 10
    method: POST
    path: "/api/message"
    function: processor
  get-messages:
    gateway: api-test
    kind: http
    method: GET
    path: "/api/messages"
    function: fetcher

events:
  consumes:
    GetMessages:
      producer: fetcher
      channel: messages

channels:
  messages:
    on_publish:
      handler: default

3. Request-Queue

4. Request-Event-Routing

5. Events Choreography

6. Event Filters

7. Request Stepfunction

8. Request Map

Example - ETL

The following is an example of developing and creating a basic topology to enhance, transform and load data.

A Basic Example: ETL

Let's call the topology etl. Our requirements are (I'm just making this up):

• Trigger the topology via a REST API and an Eventbridge event
• Create nano functions that do minimal things by decoupling them from their dependencies.
• Build a linear flow of the enhancer, transformer and loader functions.
• Write the enhancer and transformer functions in Python and loader in Ruby (huh, don't ask me why)
• Build and use a transformer ML model (oh, it's 5GB in size and has weird build steps they say)
• Deploy and test the entire thing interactively in dev sandboxes and atomically in prod sandboxes

Let's get started!

1. Create a new directory called etl and add a file called topology.yml to it.

2. Add the following to topology.yml in the etl directory

   topology.yml

   name: etl
   routes:
   	etl:
   		gateway: api-test
   		proxy: '{{namespace}}_enhancer_{{sandbox}}'
   		kind: http
   		timeout: 10
   		async: false
   		method: POST
   		path: "/api/etl"
   
   

   We now have defined a basic topology that exposes an API endpoint to a function or proxy called enhancer. However, we haven't written or built enhancer function. Let's do that in the next step.

3. Create a directory called enhancer in the etl directory. Create a file called handler.py in etl/enhancer directory

   etl/enhancer/handler.py

   	def handler(event, context):
   		return {"data": "enhanced-data"}
   

   Now this ain't doing much is it ? That's all we need for a function though with some business logic.

   Now we may need some libraries (shared etc). Let's go ahead add a pyproject.toml with our dependencies. Since we are using python, the size of the dependencies can increase thus beating the purpose of having a nano function. However, dependencies are inevitable and let's go with it.

4. Now that we added dependencies, we may need to define some additional metadata about the function. This definition is optional if we keep our functions lean with no dependencies. Anyway, let's create a file called function.json and add the following to it.

   {
   	"name": "enhancer",
   	"description": "enhance wer data",
   	"runtime": {
   		"lang": "python3.10",
   		"package_type": "zip",
   		"handler": "handler.handler",
   		"layers": ["etl-enhancer"],
   		"extensions": []
   	},
   	"tasks": {
   		"build": "zip lambda.zip handler.py",
   		"clean": "rm *.zip"
   	}
   }
   

   The above definition describes what our enhancer is, how to invoke it etc. Note that we need to specify the layer name for the dependencies. Follow along ...

5. Let's now build the dependencies. At this point, we may want to consider downloading tc (it's 5MB executable containing 15K lines of magic written in Rust). We need to login to an AWS env (say dev):

   tc login -e dev
   
   cd etl
   tc build --publish
   

   The above command builds and publishes the dependencies as a lambda layer to a centralized account (CICD). Now if our dependencies are really bloated, tc build will split the layers into 40MB (x 3) chunks. If we have nested directories (lib/ vendor/ etc), it will merge it. It will also be able to pull private repos, pull AWS assets when needed.

   To see if the dependency layer actually got published, run tc build --list

   name                                      | version | created_date
   -------------------------------------------+---------+------------------------------
   etl-enhancer                              | 1       | 2024-01-04T17:24:28.363+0000
   

   Note that the layer only contains the dependencies we added for etl-enhancer, not the enhancer code itself. That gets packed and deployed separately to our sandbox. The reason the layer build and code packing steps are decoupled is because the former is heavy and the latter is leaner.

6. Phew! building dependencies is not straightforward. It has to be built for the right CPU architecture, find shared objects, resolve shared libs, fetch private repositories, autofix incompatible transitive dependencies. That's a lot of complexity to absorb. Incidental complexity you say, eh ? Anyhow, let's create a sandbox with our "enhanced" code.

   tc create -s bob -e dev
   
   2024-01-15T19:57:03.865 Composing topology...
   2024-01-15T19:57:04.168 Initializing functor: etl@bob.dev/0.0.1
   2024-01-15T19:57:04.431 Creating function etl_enhancer_bob (214 B)
   2024-01-15T19:57:04.431 Creating route /api/etl (OK)
   

   Voila! Our enhancer function is tiny and the bloated dependencies got layered away in the previous step. Dependencies don't change much do they ? Things that move fast ought to be lean.

7. Let's say we modify our code and would like to incrementally update the sandbox.

   We can do tc diff -s bob -e dev to see what the diff is between our local edits and the code in our remote lambda function. When satisfied:

   cd etl
   tc update -s bob -e dev -c enhancer
   

8. Well, there are other infrastructure components in a topology and that is something we prefer to isolate from the code. We can scaffold roles and vars json files to an infrastructure directory

   tc scaffold --create functions
   

   The above command will create roles and vars files in infrastructure/tc/etl/{vars, roles}/enhancer.json. We can add any additional env vars, secret uris and other function-specific IAM permissions.

   We can incrementally update the vars, roles etc

   tc update -s bob -e dev -c roles
   tc update -s bob -e dev -c vars
   tc update -s bob -e dev -c routes
   

9. Now we may need to create an eventbridge event to trigger our enhancer (Remember, that is a requirement). So let's add that to the topology defintiion.

   name: etl
   routes:
   	etl:
   	gateway: api-test
   	proxy: '{{namespace}}_enhancer_{{sandbox}}'
   	kind: http
   	timeout: 10
   	async: false
   	method: POST
   	path: "/api/etl"
   
   events:
   	consumes:
   		StartETL:
   			producer: default
   			function: '{{namespace}}_enhancer_{{sandbox}}'
   
   

   Now just update the events component

   tc update -s bob -e dev -c events
   

10. One of the requirements is to build and use a ML model for the transformer.

    {
    	"name": "transformer",
    	"description": "tranform your soul",
    	"runtime": {
    		"lang": "python3.10",
    		"package_type": "zip",
    		"handler": "handler.handler",
    		"layers": [],
    		"extensions": []
    	},
    	"assets": {
    		"MODEL_PATH": "/mnt/assets/etl/transformer/1.0/artifacts",
    		"DEPS_PATH": "/mnt/assets/etl/transformer/deps"
    	},
    	"tasks": {
    		"build": "zip lambda.zip handler.py",
    		"clean": "rm *.zip"
    	}
    }
    

    Now building model (primarily using pytorch) is no child's play. Yet, tc build makes it simple

    cd transformer
    tc build --kind artifacts --publish
    

    If an assets key in present in function.json file, tc build --kind deps --publish publishes it to EFS. The models and deps are available to the function automagically.

11. Now, let's write our loader function in Ruby. Can tc build it ? Let's see.

    Add a Gemfile, a handler (handler.rb or a module) and function.json in loader directory.

    {
    	"name": "loader",
    	"description": "load your jiggle wiggle",
    	"runtime": {
    		"lang": "ruby3.2",
    		"package_type": "zip",
    		"handler": "handler.handler",
    		"layers": [],
    		"extensions": []
    	},
    	"tasks": {
    		"build": "zip lambda.zip handler.rb",
    		"clean": "rm *.zip"
    	}
    }
    

    Like we did with python dependencies, we can create a layer and publish it

    cd loader
    tc build --publish
    

    tc build --list to see if it got published

    name                                      | version | created_date
    -------------------------------------------+---------+------------------------------
    etl-enhancer                              | 1       | 2024-01-04T17:24:28.363+0000
    etl-loader                                | 1       | 2024-01-04T18:24:28.363+0000
    

12. Let's create the function:

    cd etl
    tc create -s bob -e dev
    
    2024-01-15T19:57:03.865 Composing topology...
    2024-01-15T19:57:04.168 Initializing functor: etl@bob.dev/0.0.1
    2024-01-15T19:57:04.431 Creating function etl_enhancer_bob (214 B)
    2024-01-15T19:57:04.431 Creating function etl_transformer_bob (10 KiB)
    2024-01-15T19:57:04.431 Creating function etl_loader_bob (629 B)
    2024-01-15T19:57:04.431 Creating route /api/test (OK)
    

13. Perhaps we can now create a flow of data between enhancer and transformer functions. We can define the flow using the AWS stepfunction ASL.

    name: etl
    routes:
    	etl:
       	gateway: api-test
    	proxy: '{{namespace}}_enhancer_{{sandbox}}'
    	kind: http
    	timeout: 10
    	async: false
    	method: POST
    	path: "/api/etl"
    
    events:
    	consumes:
    		StartETL:
    			producer: default
    			function: '{{namespace}}_enhancer_{{sandbox}}'
    flow:
    	Comment: ETL
    	StartAt: enhance
    	TimeoutSeconds: 1200
    	States:
    		enhance:
    			Type: Task
    			Resource: arn:aws:states:::lambda:invoke
    			OutputPath: $.Payload
    			InputPath: $
    			Parameters:
    				FunctionName: '{{namespace}}_enhancer_{{sandbox}}'
    				Payload:
    					data.$: $
    	        Next: transform
    		transform:
    			Type: Task
    			Resource: arn:aws:states:::lambda:invoke
    			OutputPath: $.Payload
    			InputPath: $
    			Parameters:
    				FunctionName: '{{namespace}}_transformer_{{sandbox}}'
    				Payload:
    					data.$: $
    	        Next: transform
    		load:
    			Type: Task
    			Resource: arn:aws:states:::lambda:invoke
    			OutputPath: $.Payload
    			InputPath: $
    			Parameters:
    				FunctionName: '{{namespace}}_loader_{{sandbox}}'
    				Payload:
    					data.$: $
    	        End: true
    

    To update the flow do:

    tc update -s bob -e dev -c flow
    

14. To invoke the stepfunction flow:

    tc invoke -s bob -e dev --payload payload.json [--sync]
    

15. Finally, lets delete our dev sandbox and deploy this to a stable sandbox in upper envs

    tc delete -s bob -e dev
    

Release and CI workflow

Well, the above steps work well if we need to interactively build, test and try in our sandbox. Wouldn't it be nice to atomically create a sandbox and attach all the infrastructure components. Oh, while we are it, can we also version the topology ?

tc deploy --sandbox stable --env qa --service etl --version 0.1.4

How do we bump the versions and release it to a QA env ? tc provides a simplified versioning scheme. The following command bumps the minor part of the semver and deploys to a QA sandbox

tc release --service etl
;=> 0.2.0

To see a meaningful changelog between releases:

cd etl
tc diff --changelog

Job Tracker

wip - a frontend app