Training

A training is a transform whose output is a model rather than a dataset. It reads datasets, fits something, and writes the artifacts under the model’s directory in the repository; the platform commits them and pins that commit as the model’s version. The model is then loaded by a transform, see Running; what the run records is on MLflow. A training is a transform whose output is a model rather than a dataset. It reads datasets, fits something, and writes the artifacts under the model’s directory; the platform commits them and pins that commit as the model’s version. Two ways to write one, for two sizes of job.

A short fit

For a training that holds its data in memory and lasts minutes, train from gwenlake.transforms is a transform like any other: inputs arrive as DataFrames, the function writes under output.path, and the dict it returns becomes the model’s metrics.

import joblib
from gwenlake.transforms import train, Input, Output

@train(
    training_set=Input("Project_A.churn_training"),
    output=Output("Project_A.churn"),          # the Output of a train is a model
)
def fit(training_set, output):
    clf = fit_classifier(training_set)         # training_set is a DataFrame
    joblib.dump(clf, output.file("model.pkl")) # written under the model's directory
    return {"auc": 0.91}                       # the model's metrics

Run it with a client like a transform, or let a build run it. A Model(...) binding alongside the Output expresses fine-tuning: the lineage then reads model -> train -> model.

A long training on the GPU pool

A training on a shared GPU is different: the data does not fit in memory, the run lasts hours, and it is evicted whenever inference needs the card. train from gwenlake.training keeps the same bindings and adds one object, the Run, which owns the four things that are easy to get wrong:

Eviction. The termination signal is caught and turned into an ordinary end of loop, between two steps, never inside one.
Resumption. Checkpoints live in a directory keyed to the run, and steps() fast-forwards to the step they carry.
Tracking. The tracking run id is stored next to the checkpoints, so a restart reports into the same run and the curve stays whole.
Durability. Checkpoints are uploaded as they are written, not at the end.

from gwenlake.training import train, Input, Output

@train(
    training_set=Input("crm.churn-train"),
    output=Output("crm.churn"),
    steps=30_000, eval_every=1_000,
)
def fit(training_set, output, run):
    model, opt = build()
    if run.resumed:                      # a previous life left state here
        load(model, opt, run.resume_path)

    for step, batch in run.steps(training_set.batches(32)):
        run.log({"loss": train_step(model, opt, batch)})
        if run.at_eval:
            with run.checkpoint() as ck:
                save(model, opt, ck.path("state.safetensors"))
                ck.metrics = evaluate(model)

    return run.summary()

The function receives its bindings like a transform, plus run. A dict it returns becomes the model’s metrics; returning nothing stores run.summary(). Inputs are streamed in batches (training_set.batches(32)) rather than loaded whole, see Large datasets.

Member of `run`	Meaning
`run.steps(batches)`	the training loop: fast-forwards on a resume, stops cleanly on eviction
`run.resumed`, `run.resume_path`, `run.start_step`	whether and where to load state from
`run.at_eval`	true on the steps where `eval_every` says to evaluate, and on the last
`run.log({...})`	metrics for the current step
`run.checkpoint(keep=)`	a context that writes a checkpoint and ships it
`run.summary()`	the metrics so far
`run.steps_for(rows, batch_size)`	the steps one epoch takes

Write checkpoints inside the loop, at run.at_eval: an evicted run never reaches the end, and the disk it writes to belongs to one node. On the pool nothing is launched by hand. The platform runs gwenlake-train, which finds the decorated function:

gwenlake-train --list          # fit
gwenlake-train --steps 30000

Locally, the same command runs the training with no GPU, no cluster and no tracking server: it degrades to a plain loop. Interrupt it and start it again, and it resumes from its last checkpoint. steps, epochs and eval_every on the decorator are defaults; the command line overrides them. model="name" names the run, so two models trained on the same data stay apart. tracking=False runs without touching the tracking server. What the run records, and how to browse it, is on MLflow.

The repository has a complete runnable example in examples/training.py. It trains on noise so it runs anywhere.

Which one

	`gwenlake.transforms.train`	`gwenlake.training.train`
Data	in memory, as DataFrames	streamed in batches
Duration	minutes	hours
Where	wherever a transform runs	the GPU pool, or your machine
Interruptions	start over	resume from the last checkpoint
Output	a model, committed and versioned	the same

Versioning

In a build, the engine sets path, commits whatever the training wrote there, and pins that commit as the model’s version. A version is a commit, and a commit is something you can go back to. This is why a model and the code that trains or predicts with it live in the same repository. Training on your own machine has no engine behind it, so the upload is explicit:

fit(client)
model.save("artifacts/", message="churn v3, auc 0.91")

Get started

Concepts

Workspace

Data

Code

Build

Models

Inference

Agents

Client

Changelog

A short fit

A long training on the GPU pool

Which one

Versioning

​A short fit

​A long training on the GPU pool

​Which one

​Versioning

A short fit

A long training on the GPU pool

Which one

Versioning