gwenlake 0.23.1
Builds from the command line.
gwenlake builds run --wait prints each step as it settles, with its duration, and on a failure its error and the end of its log; --follow prints the logs as they come, and so does builds logs --follow. run takes several datasets and --dry-run; retry builds a finished build’s target again on its data branch, propagate builds again what is downstream of a dataset, cancel --batch stops a whole batch. builds list filters by --project, --trigger and --since, and names targets <project>.<dataset>. See Command line.Build plans in the library. client.builds.plan answers what a build would run without running it, and client.builds.logs takes since= to read only the new lines. See Workflows.Platform
Build plans.
POST /orchestration/builds/plan answers the steps a build would run, in order, with their outputs and dependencies, and starts nothing.Build listings. GET /orchestration/builds?limit= counts only the builds the caller can see: a small limit no longer answers an empty list.gwenlake 0.22.0
Exports to stdout.
gwenlake sql and gwenlake datasets export write the whole result to stdout with -o - --format parquet (or csv, json, jsonl), so it can travel through a pipe or ssh; the row count goes to stderr. statements.to_file takes an open binary stream with format=. See Queries.Global options anywhere. --profile, --api-key, --base-url and --json are accepted after the command too: gwenlake datasets list --profile fleetsight.gwenlake 0.20.0
The whole catalog in the library. Every route of the Atlas API now has a method:
schedules, transforms (the producers, with set_parameters), builds (run, wait, run_and_wait, logs), lineage, repositories (with scan_lineage and the git routes), queries, ml_models, mediasets, syncs, flows, organizations, users, api_keys, filesystem, agents, skills, tools, ontologies, applications; datasets gains update, summary, delete_rows, branches, maintenance, search; connections gains create, update, query, test. client.resolve("<project>.<dataset>") gives a dataset id. A test compares the library to the API’s OpenAPI description, so a new route cannot go unnoticed. See Client configuration.Read a result whole, export it. statements.iter_arrow, read_all, to_dataframe page a statement past the 1000-row cap; statements.to_file writes it to Parquet, CSV, JSON or JSON Lines, streaming. See Queries.A real command line. gwenlake (alias gwl) covers the resources in noun-verb form, prints tables or JSON, runs and exports SQL, follows builds, manages schedules, scans repositories, and manages the credentials file (gwenlake config set-profile). A profile or --base-url names the endpoint of a tenant deployed on its own host. See Command line.Lighter. The library no longer depends on pydantic: its request and response objects are plain dataclasses.gwenlake 0.14.0
Incremental inputs. Inside a build, a tabular input of an
@incremental transform answers dataframe("added"), the rows appended since the last successful run, read from the table’s own snapshot history up to the point pinned before the run, and dataframe("previous"), the table as the last run saw it. The platform records what each run consumed and exposes a snapshot span in SQL ('p.ds@since..until'). See Incremental transforms.gwenlake 0.13.0
Incremental transforms.
@incremental, stacked above @transform or @transform_df, gives outputs a "previous" reading mode and makes writes add to the dataset; semantic_version starts a transform over. The platform records the version each successful build wrote an output with and reports a failed run’s partial writes.File calls on branches. files.list, files.download and files.delete take branch=, and datasets.empty is new. Inside a branch build the transforms’ filesystem() addresses the branch for reads and deletes as it did for writes.Platform
One build for a function with several outputs. A
@transform_df declaring several Outputs is built once for all of them: one build carries every output, a second build of the function is refused while one runs, the outcome lands on every output, and a propagation starts from each.Slugs. Resources are named by a slug, unique in their scope; alias remains as its former name in responses and requests.Iceberg by default. A new dataset is a tabular dataset wherever the organization supports it.