Pumpkin runs agents as workflows, each run in sandboxes that reach only what the workflow grants, and each leaving a record that no sandbox can alter.
Workflows · Language · Running · Records · Providers · Models · Architecture
A workflow is a short program, in a subset of Starlark, that says what a run is made of: its
roles, each a sandbox image with the resources it needs; what each role may reach, such as a model
through a route with a budget, and which functions it may call; and how the run's result is
produced from what the agent submits. pumpkin checks a workflow without running anything, builds
its images, and runs it on a provider: the machine's own Docker engine, EC2, or Modal, from the
machine or through a controller. Every run leaves a record, which pumpkin wait, pumpkin ls and
a web page read.
The agent's code is untrusted: it may be buggy, and it uses whatever it can reach to finish its task or raise its score. So:
- A sandbox's only way out is its session with the run's driver, the process that runs the run. It has no network of its own and no credentials, and the driver authorizes each call it makes against the workflow.
- Every bound is the workflow's: the agent's deadline, each function's calls and seconds, each model entry's budget. The run's hard end follows from them, and every sandbox ends by it.
- A model key stays on the host. The route adds it to each request it admits, and the record keeps every exchange as it passed.
- The same sources and arguments give the same workflow. Evaluation reads no clock, no environment, and no network, and a recipe must pin what Docker fetches for its image.
A workflow is the one workflow() call of a task's entry file: the workflow's name, its roles,
and the final call that gives a run its result. examples/hello needs no model, so it runs
wherever Docker does. Its agent is a shell script: it has a job function count the lines and bytes
of its greeting, then submits the greeting, and the final call scores the greeting by its size.
examples/hello/task.star, the entry file:
"""A workflow that needs no model: the agent counts its greeting's lines and bytes by a call of its
job function `count`, then submits the greeting, and the final call scores its size and counts its
lines."""
load("//hello/count.star", "count")
load("//hello/final.star", "final_call")
load("//submit.star", "submit")
IMAGE = image("Dockerfile")
SANDBOXCTL = "/.pumpkin/bin/sandboxctl"
# Calls `count`, prints the job's status once it has ended, and submits the greeting.
AGENT = """
echo hello > greeting
SANDBOXCTL call count greeting
while SANDBOXCTL status 0 | grep -q -E '"state":"(queued|running)"'; do sleep 1; done
SANDBOXCTL status 0
SANDBOXCTL call final greeting --arg finalize=true
""".replace("SANDBOXCTL", SANDBOXCTL)
workflow(
name = "examples/hello",
roles = [
{
"name": "agent",
"image": IMAGE,
"user": {"uid": 1000, "gid": 1000},
"needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
"runtime": {"agent": {
"command": {
"argv": ["sh", "-c", "set -e" + AGENT],
"cwd": "/workspace",
},
"deadline_seconds": 300,
"finalize_grace_seconds": 30,
"grants": [
{"name": "count", "job": {
"target": "counter",
"body": count,
"payload_max_bytes": 1 * KiB,
"call_timeout_seconds": 60,
}},
{"name": "final", "now": {"body": submit, "payload_max_bytes": 1 * KiB}},
],
}},
},
{
"name": "counter",
"image": IMAGE,
"user": {"uid": 65532, "gid": 65532},
"needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
"runtime": {"target": {"cwd": "/workspace"}},
},
{
"name": "judge",
"image": IMAGE,
"user": {"uid": 65532, "gid": 65532},
"needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
"runtime": {"target": {"cwd": "/pumpkin/home"}},
},
],
final = {"target": "judge", "body": final_call, "payload_max_bytes": 1 * KiB, "timeout_seconds": 60},
)The bodies it loads, each a def with typed parameters, in
examples/hello/count.star:
"""The body of the agent's job function `count`."""
def count(target: Target, payload: Payload) -> dict[str, int]:
"""Counts the payload's lines and bytes with `wc`, in a fresh sandbox of the counter.
Args:
target: the counter's sandbox.
payload: the greeting.
Returns:
The greeting's number of lines and of bytes.
"""
target.put("/workspace/greeting", payload)
target.run(["sh", "-c", "printf '{\"lines\": %s, \"bytes\": %s}' $(wc -l < greeting) $(wc -c < greeting) > counts.json"])
return json.decode(target.get("/workspace/counts.json"))examples/submit.star, shared by the examples:
"""The agent's submissions."""
def submit(run: Run, payload: Payload, finalize: bool = False) -> dict[str, str]:
"""Selects `payload` for the final call, replacing the last.
Args:
run: the run's state.
payload: the agent's submission.
finalize: whether the call also ends the agent's work.
Returns:
The selection's state: `stored`, or `accepted` once the work has ended.
"""
run.select(payload)
if finalize:
run.end_work()
return {"state": "accepted"}
return {"state": "stored"}and examples/hello/final.star:
"""The hello example's final call: it counts the greeting's bytes and lines in its target."""
def final_call(target: Target, payload: Payload) -> dict[str, int]:
"""Shows the greeting's size as its score, and its lines.
Args:
target: the call's target.
payload: the greeting the agent submitted.
Returns:
The score and the lines, in the order the run shows them.
"""
target.put("/workspace/greeting", payload)
target.run(["sh", "-c", "printf '{\"score\": %s, \"lines\": %s}' $(wc -c < /workspace/greeting) $(wc -l < /workspace/greeting) > /pumpkin/out/result.json"])
result = json.decode(target.get("/pumpkin/out/result.json"))
return {"score": result["score"], "lines": result["lines"]}Roles. A role is a template of sandboxes: an image, the user its commands run as, never root,
and what each of its sandboxes needs, from which the run picks the class of sandbox, among those the
providers offer, that each is placed in. Its runtime says its kind. The one agent works on the
task: its command runs from the run's start until it exits, its work ends, or deadline_seconds
pass. A target runs nothing of its own: each call of the function that names it gets a fresh
sandbox of it, which the function's body drives and which ends with the call. A runner is a
sandbox that a role holding a session leases and drives. What a role may reach and call is written
inside its kind: an agent holds egress and grants, a target egress and sessions, and a
runner neither.
Grants. A grant is a name and one function, under its kind's key: {"name": "count", "job": {…}}. The agent calls it by name, sandboxctl call count greeting, and the file's bytes are the
call's payload, at most the grant's payload_max_bytes. The driver checks every call against the
grant before anything runs and refuses, with a reason, one that breaks it: a job function runs one
call at a time, and its calls and cumulative_seconds, left out here, bound how many it runs and
for how long.
A job function's body, here count, is a def of a file the entry loads, called once for each
call with a Target and the payload. The Target is the handle on a fresh sandbox of the grant's
target: target.put writes the payload as a file, target.run runs a command in the target's
cwd and waits for it, and target.get reads back a file of at most 1 MiB as text. The dict
the body returns is the job's result. The call answers at once with the job's number, and
sandboxctl status 0 follows job 0 to its end. The agent never reaches the target: of a call it
learns only the job's state, what the grant has left, and the dict.
A now function's body, here submit, answers in the call's reply, with no sandbox. The agent
calls it as final, the name of its grant, which is not the final call. Its handle is a Run, the
run's state: run.select(payload) makes the payload the selection, the one the final call gets,
and run.end_work() ends the agent's work, after which the agent has finalize_grace_seconds to
exit. The run applies what the body asked once it has returned, all of it or none. A parameter
after the payload is the caller's: annotated str, int or bool, it is given as
--arg finalize=true and read as its type before the body runs. sandboxctl ops lists each grant
with its bounds and its caller's parameters.
Sessions are the third kind of function, for work that one call of a body can't hold, such as
training on a GPU. A session grant names a runner and bounds its leases: how many are open at once,
how long each and all of them last, how many commands they run, and how many bytes each copy moves.
The role opens a lease with sandboxctl call <session>.open, runs commands in its runner and
copies directory trees to and from it with .run, .put and .get, follows it with .status,
and ends it with .close.
The final call runs once the agent's work has ended and everything it started has ended too:
its body runs as a job's does, on a fresh sandbox of its target, which no job function may name,
with the last payload selected. The dict it returns is the run's result, and its entries, in their
order, are the values the run shows: score=6 lines=1 in pumpkin ls. A run with nothing
selected, or whose final call fails, finishes without a result.
Egress is what a role reaches beyond its session. This agent has none; one that uses a model
has a model route, whose entries each name an API, the models it allows and, if it is bounded, a
budget: "egress": {"model": [{"api": "chat", "models": ["model-a"], "budget": {"microusd": 5 * USD}}]}. The route counts each entry's spend at the prices of the machine's configuration, and
the model's key stays on the host.
What pumpkin check proves before anything is built: that every file evaluates, every
image's recipe pins what it fetches, every key and bound of the workflow is one the IR takes, every
role a grant or the final call names is of the kind it must be, and every body takes the handle and
the payload that its kind passes, with caller's parameters of a type --arg gives. It says each
problem at its file, line and column. A body's own statements run only when it is called.
docs/workflows.md holds every key a workflow writes and what sandboxctl
answers; docs/language.md holds the language.
The operator, who runs pumpkin and holds the machine's configuration and keys, installs it in a
repository of tasks as ./pumpkin, a copy of deploy/pumpkin beside a
.pumpkinversion that names the tag and the commit to build, on Linux x86-64. A machine whose
runs stay on its own Docker engine needs only this ~/.config/pumpkin/pumpkin.toml:
[docker]
context = "default"
[providers.local.docker]With Pumpkin's examples/ directory copied into the repository of tasks, examples/hello, which
needs no model, goes through the five commands:
$ ./pumpkin check examples/hello/task.star # evaluates and checks; silent if no problem
$ ./pumpkin run examples/hello/task.star # builds the images, starts the run, prints its id
20261116-195228-234cd166
$ ./pumpkin wait 234c # follows the run, named by a prefix of its digits
pumpkin: run 20261116-195228-234cd166 finished with a result (score=6 lines=1), ...
$ ./pumpkin ls # lists the runs live or ended in the last day
20261116-195228-234cd166 finished 2s - - examples/hello: score=6 lines=1
$ ./pumpkin view # serves the runs to this machine's browserdocs/running.md goes through installing, configuring and these commands in order, with the full output of each.
pumpkin evaluates a task's entry file, and the files it loads, to the workflow IR, which is JSON,
and to the images its roles run, which it builds on the machine's Docker engine, each tagged by the
digest of its inputs. The runtime reads only the IR, so it depends on no language. A function's
body stays code: the IR names it by its file and its def, and the driver keeps the text of every
file the evaluation read, in the run's record, and evaluates the body from that text at each call,
so a call runs the body that was checked as the run started, whatever has changed on disk since.
Evaluation is hermetic and always ends: it reads files only below the task's root and Pumpkin's
library, and its arguments only from --arg.
The driver runs a run from start to end, on the machine that started it or on a controller. In it, the decision core takes each event that happened, such as a call that arrived or a sandbox that ended, and returns what to do next; it does no I/O and reads no clock, so the same events give the same actions. The runtime carries those actions out on the providers, the gateway holds one end of each sandbox's session, and the model route answers the roles' model requests.
Every sandbox runs sandboxd, which meets the gateway and serves one session. Over it the driver
puts and collects files and starts commands, and code in the sandbox makes its calls, with
sandboxctl, and its model requests, at the endpoint sandboxd serves. Nothing else of the
network is reachable: a Docker sandbox has loopback alone, an EC2 sandbox's instance drops all but
the gateway's address and port, and a Modal sandbox opens no connection but DNS.
ARCHITECTURE.md's security model says what each guard holds against.
The driver writes each run's record as the run goes, in a directory of its own: the IR and the
text of the files it came from, the run log of every event and what was done about it, each
sandbox's output, each model exchange, and how the run ended. pumpkin wait, pumpkin ls and the
page reads only the record, so each says the same about a run, wherever it ran.
Details: workflows · language · running · records · providers · models · architecture
A provider makes a run's sandboxes. Each role is placed in the smallest class of sandbox, of every provider that offers what it needs, such as its CPUs, its memory or its GPU.
| Provider | A sandbox is | What the machine needs |
|---|---|---|
docker |
a container on the machine's own Docker engine, on cores of its run's own | the engine |
ec2 |
a container on an EC2 instance of its own | the envelope that deploy/aws makes |
modal |
a sandbox in a Modal workspace, under gVisor | a Modal profile, and the envelope's images |
The controller is the same pumpkin, on an EC2 instance of the operator's own, which
deploy/aws makes. A machine whose configuration names [controller]
sends its runs there over ssh through SSM, and follows, lists, stops and views them there with the
same commands; the controller builds the images and runs the drivers, so the operator's machine can
go to sleep. It can also publish its runs to a read-only page for a team.
docs/providers.md says how the pieces fit.
- Writing workflows: workflows, the language.
- Running workflows: running, records and the page.
- Deploying: providers and the controller,
deploy/aws,pumpkin help config. - Models: routes, budgets and prices.
- Contributing:
ARCHITECTURE.md, the map of the code, anddocs/dev/, the rules for changing it.