Organizations and projects
An organization is your company or team: it owns the members, the API keys and the storage. A project groups the resources of a use case inside an organization. Each project has an id (res.project.…) and a short human-readable slug, which prefixes the handle of every resource it holds.
Datasets
A dataset holds data. It comes in three kinds:- Tabular datasets are Apache Iceberg tables. They are what SQL queries and
transform_dfread, and they keep a history of snapshots. - File datasets hold arbitrary files: documents, images, Parquet parts, model exports.
- Vector datasets hold embeddings, produced from documents by the platform’s own transforms and searched by agents.
res.dataset.… and addressed in SQL and transforms by its handle, "<project_slug>.<dataset_slug>". See Datasets and Handles and identifiers.
Files
Files live inside a dataset, optionally under a path. Upload, download, presign and delete them with thefiles resource, or through the .filesystem() of a transform. See Files.
Mediasets
A mediaset is a store of media files of one kind — documents, images, audio, video — fed by a sync from a connection. It is where a document pipeline starts: the platform’s ready-made transforms turn it into pages, chunks and vectors. See Mediasets.Connections
A connection is an external source a project declares once: an S3 bucket, an SMB share, a SharePoint library, a PostgreSQL database. The platform holds its credentials and reads on your behalf. A sync copies what a connection holds into a dataset; a transform can also read a connection directly. See Connections.Repositories
A repository is a git remote declared in a project: the code of your transforms, trainings, SQL files and agents, scanned by the platform and checked out at a pinned commit for each build. See Repositories.Transforms and builds
A transform produces a dataset from others. Three kinds: a Python transform is a function you write with thegwenlake.transforms decorators in a git repository the platform scans; a SQL transform is a query stored on the platform whose FROM references are its inputs; a ready-made transform is one the platform supplies (OCR, chunking, embedding), configured with parameters rather than code. A sync is the fourth producer: it copies what a connection holds (see Syncs). All of them run in a build, on demand or on a schedule, and the platform records what each read and wrote. See Transforms and Lineage, builds and branches.
Models
A model is a catalog resource whose artifacts live in the git repository of your code. A@train function produces one; a transform binds it with Model(...) to load it. The model card keeps parameters, metrics and the commit that pins its version. See Running.