Datasets

A dataset holds data. It is identified by res.dataset.… and addressed in SQL and transforms by its handle "<project_slug>.<dataset_slug>".

Kinds

`type`	Holds	Read with
`iceberg`	a table, with a history of snapshots and data branches	SQL, `Input(...)` in a transform
`files`	arbitrary files: Parquet parts, CSV, Excel exports, model outputs	the `files` resource, `input.filesystem()`
`vector`	embeddings, one row per chunk with its text and metadata	semantic search, from the Atlas interface and agents

iceberg is the default when the organization supports it, files otherwise. A transform’s output must be a tabular dataset where the organization supports them; the kind cannot change to or from vector after creation. Documents, images, audio and video belong in a mediaset, not in a file dataset.

List and get

for d in client.datasets.list():
    print(d["slug"], d["id"], d.get("type"))

# datasets of one project or one organization
client.datasets.list(project_id="res.project.…")
client.datasets.list(organization_id="res.organization.…")

dataset = client.datasets.get("res.dataset.…")

A dataset carries id, name, slug (also returned as alias, its former name), description, type, project_id, organization_id, created_by, created_at, updated_at, and for a tabular dataset its declared columns and keys.

Create

dataset = client.datasets.create({
    "slug": "delayed",
    "name": "Delayed flights",
    "project_id": "res.project.…",
})

A tabular dataset may declare its schema and key columns up front; otherwise the schema is inferred from the first upload. Keys are the columns an UPDATE upserts on, and they must never be null.

dataset = client.datasets.create({
    "slug": "orders",
    "name": "Orders",
    "project_id": "res.project.…",
    "type": "iceberg",
    "columns": [
        {"name": "order_id", "type": "long", "required": True},
        {"name": "placed_at", "type": "timestamp"},
        {"name": "amount", "type": "double"},
    ],
    "keys": ["order_id"],
})

A dataset is created empty. Write to it with a transform, a file upload (below) or a sync from a connection.

Reading a dataset

A tabular dataset is read with SQL. The handle goes in the FROM, the result comes back as JSON rows, CSV, or Arrow for anything large:

rows = client.statements.create(
    statement="SELECT * FROM 'flights.flight-data' WHERE carrier = 'AF' LIMIT 100",
    format="json",
)["data"]

import pyarrow as pa
raw = client.statements.create(statement="SELECT * FROM 'flights.flight-data'", format="pyarrow", limit=1_000_000)
df = pa.ipc.open_file(pa.BufferReader(raw)).read_all().to_pandas()

A result without a LIMIT is capped at 1000 rows unless limit= raises the cap; page with LIMIT/OFFSET beyond that. See Queries. A file dataset is read file by file:

for entry in client.files.list(dataset_id, path="exports"):
    data = client.files.download(dataset_id, f"exports/{entry['filename']}")

In a transform, the binding does the reading: Input("flights.flight-data") arrives as a DataFrame in a transform_df, or as a handle in a transform with dataframe(), iter_dataframes(), to_parquet() and filesystem(). columns= on the binding is the cheapest way to read less. See Transforms and Large datasets. A vector dataset is searched, not read: from an agent’s search tool or the Atlas interface.

Files

The files resource works on both kinds of dataset. Every call takes the dataset id.

dataset_id = "res.dataset.…"

for f in client.files.list(dataset_id):
    print(f["filename"], f["file_size"], f["last_modified"])

client.files.list(dataset_id, path="exports")                        # one subdirectory

On a tabular dataset the listing shows the table’s physical files, an implementation detail; read the rows with SQL.

Upload

upload accepts a local path or raw bytes. path= places the file in a subdirectory; filename= names bytes that have no name of their own.

client.files.upload(dataset_id, "rows.parquet")
client.files.upload(dataset_id, "export.xlsx", path="exports/2026")
client.files.upload(dataset_id, b"a,b\n1,2\n", filename="rows.csv")

On a file dataset the file lands as it is. On a tabular dataset the upload is parsed and written into the table. Accepted: Parquet, ORC, Arrow IPC (.feather, .arrow), Avro, CSV, TSV, JSON (an array or one object per line), .jsonl/.ndjson, Excel (first sheet), and any of the text formats gzipped. The columns are matched to the table’s by name and cast; a column the table does not have yet is added. mode says how the rows land:

`mode`	Effect
`"APPEND"` (default)	adds the rows
`"SNAPSHOT"`	replaces every row with the upload
`"UPDATE"`	upserts on the dataset’s key columns (`?keys=` over HTTP to name others)
`"DELETE"`	removes the rows matching `?delete_filter=` (over HTTP, no file)

client.files.upload(dataset_id, "rows.parquet")                      # append
client.files.upload(dataset_id, "rows.parquet", mode="SNAPSHOT")     # replace all
client.files.upload(dataset_id, "rows.parquet", mode="UPDATE")       # upsert on the dataset's keys

Every write is a snapshot of the table, which is what data branches and incremental transforms build on. A vector dataset takes a tabular file carrying a vector column, with the same modes.

Download, presign, delete

content = client.files.download(dataset_id, "exports/2026/export.xlsx")            # bytes
url = client.files.presigned_url(dataset_id, "rows.parquet", expires_in=600)["url"]  # one hour by default
client.files.delete(dataset_id, "rows.parquet")

A presigned URL lets a third party download a file without a Gwenlake API key, for a limited time. Organizations on an SMB share have no presigned URLs; download through the API instead. On a tabular dataset, rows are removed with mode="DELETE" or the dataset is emptied; its physical files are not deleted one by one. Every file call takes branch= to work on a data branch instead of the trunk.

Inside a transform, prefer the .filesystem() of an input or output: it carries the build’s branch for you and is what the lineage records. See Working with files.

Vector datasets and search

A vector dataset holds embeddings: one row per chunk of text, with the chunk’s text, its vector, and any scalar column as metadata (path, page, …). The platform’s document pipeline produces one from a mediaset, and the embedding model is set on the dataset, so the table and the search always agree. An upload into a vector dataset is a tabular file carrying a vector column, with the same modes as a table. Searching is POST /datasets/{id}/search: a query, embedded with the dataset’s model as the caller, or a ready-made vector; limit (5 by default, 100 at most) and where, a SQL filter on the metadata columns.

curl -X POST https://api.gwenlake.com/v1/datasets/res.dataset.…/search \
  -H "Authorization: Bearer $GWENLAKE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the refund policy?", "limit": 5, "where": "path LIKE '"'"'contracts/%'"'"'"}'

Each hit carries id, text, score (the cosine distance: lower is closer) and metadata, every other column of the row. A dataset holding no vectors yet answers 400. The same search is an agent’s search tool when the dataset is bound to it, and the natural first stage before a rerank. A vector dataset has no SQL surface and no data branches; POST /datasets/{id}/empty drops its table.

Update

PATCH /datasets/{id} changes the name, slug, description, declared columns or keys. Renaming a dataset without giving a slug regenerates the slug from the name; giving the slug alone changes it alone. The slug is what every SQL statement and binding uses, so change it knowingly.

client.datasets.update("res.dataset.…", {"name": "AISDK positions", "slug": "aisdk_positions"})
client.datasets.delete_rows("res.dataset.…", "source_file = 'aisdk-2026-10-25.csv'")   # a DELETE write

gwenlake datasets update fleetsight.ais_positions --name "AISDK positions" slug=aisdk_positions
gwenlake datasets delete-rows fleetsight.aisdk_positions_raw "source_file = 'aisdk-2026-10-25.csv'"

Empty and delete

empty removes every row or file but keeps the dataset and, for a tabular dataset, its history; delete removes the dataset. Deleting a dataset also removes the transform or sync that produced it: a producer cannot outlive its output.

client.datasets.empty("res.dataset.…")
client.datasets.empty("res.dataset.…", branch="experiment")

client.datasets.delete("res.dataset.…")

Summary

The summary endpoint answers what the dataset holds, by kind: num_rows, size_bytes and last_modified for a tabular dataset; num_files, size_bytes and last_modified for a file dataset, plus num_rows when it holds Parquet; num_rows, dimensions and the embedding model for a vector dataset. A field is null when nothing has been written yet. client.datasets.summary(id) and gwenlake datasets summary <project>.<dataset> read it.

Maintenance

Every write to a tabular dataset leaves a snapshot behind, and branches pin the ones they point at. Expiring old snapshots reclaims their storage; it is done from the Atlas interface, on demand or on a schedule, or with client.datasets.maintenance(id, older_than_days=7). An incremental transform reads a table’s history, so keep its maintenance window longer than the interval between two of its runs.

Get started

Concepts

Workspace

Data

Code

Build

Models

Inference

Agents

Client

Changelog

Kinds

List and get

Create

Reading a dataset

Files

Upload

Download, presign, delete

Vector datasets and search

Update

Empty and delete

Summary

Maintenance

​Kinds

​List and get

​Create

​Reading a dataset

​Files

​Upload

​Download, presign, delete

​Vector datasets and search

​Update

​Empty and delete

​Summary

​Maintenance

Kinds

List and get

Create

Reading a dataset

Files

Upload

Download, presign, delete

Vector datasets and search

Update

Empty and delete

Summary

Maintenance