Builds

A build checks out your repository at a pinned commit and runs the transform’s entrypoint with uv run. The library reads everything it needs from the environment the platform sets, so a transform written with the decorators never touches these variables. This page is for the cases where you do: a script that does its own IO, a debugging session, or a tool that is not Python.

How the entrypoint runs

uv run [--python <version>] [--with-requirements <file>] [--with <spec>]... <entrypoint>

uv resolves the repository’s own environment from its pyproject.toml and uv.lock, or from PEP 723 inline metadata at the top of the script, and runs the entrypoint inside it. Wheels are cached across builds. Three optional settings on the transform add to that without a repository change:

Setting	Effect
`python`	the interpreter version or path (`--python`)
`requirements`	a requirements file in the repository (`--with-requirements`)
`packages`	extra package specs (`--with`, one per entry)

A module that holds several transforms should end with run(client, ...) so a build runs only the one it targets: the platform names it in CATALOG_FUNCTION.

Identity

Variable	Meaning
`GWENLAKE_BASE_URL`	the platform’s own address for the API, inside the cluster
`GWENLAKE_USER_TOKEN`	the build’s identity: the user who triggered it, narrowed to what their credential allowed

The client picks both up, which is why a script needs no API key in a build. Calls to the inference models go through the same address and run as that identity, with its rate limits and attribution.

Inputs and outputs

Variable	Meaning
`CATALOG_INPUTS`	`{slug: location}` for every input: a Parquet glob for a file dataset, the files directory of a mediaset, the vector store of a vector dataset
`CATALOG_INPUT_IDS`	`{slug: id}` for the same inputs
`CATALOG_OUTPUT`	the storage location of the output dataset (`CATALOG_OUTPUT_S3` is its former name)
`CATALOG_OUTPUT_ID`, `CATALOG_OUTPUT_TYPE`	the output’s id and kind (`files`, `iceberg`, `vector`)
`CATALOG_STORAGE_TYPE`	`s3` or `smb`, the organization’s storage
`CATALOG_BRANCH`	the data branch the build writes to (`main` on the trunk)
`CATALOG_FUNCTION`	the decorated function this build runs
`CATALOG_CONNECTIONS`	`{slug: id}` for the connections the transform declares
`CATALOG_MODELS`	`{slug: {model_id, path, parameters, version}}` for the models it loads; `path` is a directory in the checkout
`TRANSFORM_PARAMETERS`	the transform’s `parameters` setting, as JSON

Tabular inputs are read through the API by the library, not from their files. A @train has no CATALOG_OUTPUT: its result is the model directory, committed after the run. TRANSFORM_PARAMETERS is how one transform serves several datasets with different settings: a chunk size, a model name, a threshold set on the transform rather than in the code.

import json, os

settings = json.loads(os.environ.get("TRANSFORM_PARAMETERS", "{}"))
chunk_size = settings.get("chunk_size", 512)

Incremental runs

Variable	Meaning
`CATALOG_BUILD_ID`	the build’s id, what the part files of an incremental output are named after
`CATALOG_INCREMENTAL`	`{output id: {semantic_version}}`, the version each output was last written with
`CATALOG_INPUT_SNAPSHOTS`	`{slug: {id, from, to}}`, where each tabular input stood before the run and after the last successful one

See Incremental transforms. The library reads these; a script has no reason to.

Storage credentials

A build receives the credentials of the organization’s storage, so that code reading CATALOG_INPUTS or writing CATALOG_OUTPUT directly can: AWS_* on S3, SMB_* (host, share, path, domain, port, username, password) on an SMB share. They are the organization’s, shared by every build of the organization. Prefer the library’s filesystem() and write_dataframe(), which go through the API with the build’s identity and need none of this.

Everything else

The process environment is built from a list, never copied from the worker: PATH, HOME, the locale and time zone, the CA bundles and proxies, UV_*, and the variables the platform is configured to pass through. Nothing of the platform’s own configuration reaches your code.

Logs, time and cancellation

Standard output and error are streamed to the build’s log as they are printed and kept for seven days, then archived on the organization’s storage.
Cancelling a build sends the process a termination signal and, if it does not exit, kills it. A build polls for cancellation every ten seconds.
A build that stops reporting for ten minutes is marked failed and its waiting siblings cancelled.

Get started

Concepts

Workspace

Data

Code

Build

Models

Inference

Agents

Client

Changelog

How the entrypoint runs

Identity

Inputs and outputs

Incremental runs

Storage credentials

Everything else

Logs, time and cancellation

​How the entrypoint runs

​Identity

​Inputs and outputs

​Incremental runs

​Storage credentials

​Everything else

​Logs, time and cancellation

How the entrypoint runs

Identity

Inputs and outputs

Incremental runs

Storage credentials

Everything else

Logs, time and cancellation