Connections

A connection is a source a project declares once: an SMB share, a bucket, a SharePoint library, a database. The platform keeps its credentials and reads on your behalf, so they never reach your code. Two ways to use one:

a sync copies what the connection holds into a dataset or a mediaset, on demand or on a schedule;
a transform declares the connection as an input and reads it itself, picking the files it wants.

List and get

for c in client.connections.list(project_id="res.project.…"):
    print(c["slug"], c["type"], c["id"])

connection = client.connections.get("res.connection.…")

Files

Listings are recursive and paged. Each entry carries path (relative to the connection’s root), filename, size, last_modified and is_dir.

cid = "res.connection.…"

# one page
page = client.connections.list_files(cid, path="2026", glob="**/*.pdf", limit=500)
page["data"], page["next_start_after"]

# every file, following the pages
for f in client.connections.iter_files(cid, glob="**/*.pdf"):
    print(f["path"], f["size"])

# the first level only
client.connections.list_files(cid, max_depth=1)

glob follows the same rule as a sync’s path_glob: * stays within a directory, ** descends.

Download

data = client.connections.download(cid, "2026/q3/report.pdf")

# a byte range, where the connection's type can seek
head = client.connections.download(cid, "2026/q3/big.bin", offset=0, length=1024)

In a transform

Bind the connection with Connection("<project>.<slug>"). The function receives a read-only filesystem over the same relay:

import shutil
from gwenlake.transforms import transform, Connection, Output

@transform(
    share=Connection("ops.drop-folder"),
    out=Output("ops.collected"),
)
def collect(share, out):
    fs_in, fs_out = share.filesystem(), out.filesystem()
    for f in fs_in.ls(glob="**/*.pdf"):
        with fs_in.open(f["path"], "rb") as src, fs_out.open(f["path"], "wb") as dst:
            shutil.copyfileobj(src, dst)

The lineage shows the connection feeding the output. To process only new files on each run, see Incremental transforms.

Connections are read-only from code: a connection is where data comes from, and a transform writes into its output dataset.

Get started

Concepts

Workspace

Data

Code

Build

Models

Inference

Agents

Client

Changelog

List and get

Files

Download

In a transform

​List and get

​Files

​Download

​In a transform

List and get

Files

Download

In a transform