Skip to content
FrontierCSPublic

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

1 watching

Forks

Repository files navigation

Pumpkin's logo: an open ring with an orange tile, turned 45 degrees, in the break at its top.

Pumpkin

Pumpkin runs agents as workflows, each run in sandboxes that reach only what the workflow grants, and each leaving a record that no sandbox can alter.

Workflows · Language · Running · Records · Providers · Models · Architecture

What Pumpkin is

A workflow is a short program, in a subset of Starlark, that says what a run is made of: its roles, each a sandbox image with the resources it needs; what each role may reach, such as a model through a route with a budget, and which functions it may call; and how the run's result is produced from what the agent submits. pumpkin checks a workflow without running anything, builds its images, and runs it on a provider: the machine's own Docker engine, EC2, or Modal, from the machine or through a controller. Every run leaves a record, which pumpkin wait, pumpkin ls and a web page read.

The agent's code is untrusted: it may be buggy, and it uses whatever it can reach to finish its task or raise its score. So:

  • A sandbox's only way out is its session with the run's driver, the process that runs the run. It has no network of its own and no credentials, and the driver authorizes each call it makes against the workflow.
  • Every bound is the workflow's: the agent's deadline, each function's calls and seconds, each model entry's budget. The run's hard end follows from them, and every sandbox ends by it.
  • A model key stays on the host. The route adds it to each request it admits, and the record keeps every exchange as it passed.
  • The same sources and arguments give the same workflow. Evaluation reads no clock, no environment, and no network, and a recipe must pin what Docker fetches for its image.

A workflow

A workflow is the one workflow() call of a task's entry file: the workflow's name, its roles, and the final call that gives a run its result. examples/hello needs no model, so it runs wherever Docker does. Its agent is a shell script: it has a job function count the lines and bytes of its greeting, then submits the greeting, and the final call scores the greeting by its size.

examples/hello/task.star, the entry file:

"""A workflow that needs no model: the agent counts its greeting's lines and bytes by a call of its
job function `count`, then submits the greeting, and the final call scores its size and counts its
lines."""

load("//hello/count.star", "count")
load("//hello/final.star", "final_call")
load("//submit.star", "submit")

IMAGE = image("Dockerfile")
SANDBOXCTL = "/.pumpkin/bin/sandboxctl"

# Calls `count`, prints the job's status once it has ended, and submits the greeting.
AGENT = """
echo hello > greeting
SANDBOXCTL call count greeting
while SANDBOXCTL status 0 | grep -q -E '"state":"(queued|running)"'; do sleep 1; done
SANDBOXCTL status 0
SANDBOXCTL call final greeting --arg finalize=true
""".replace("SANDBOXCTL", SANDBOXCTL)

workflow(
    name = "examples/hello",
    roles = [
        {
            "name": "agent",
            "image": IMAGE,
            "user": {"uid": 1000, "gid": 1000},
            "needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
            "runtime": {"agent": {
                "command": {
                    "argv": ["sh", "-c", "set -e" + AGENT],
                    "cwd": "/workspace",
                },
                "deadline_seconds": 300,
                "finalize_grace_seconds": 30,
                "grants": [
                    {"name": "count", "job": {
                        "target": "counter",
                        "body": count,
                        "payload_max_bytes": 1 * KiB,
                        "call_timeout_seconds": 60,
                    }},
                    {"name": "final", "now": {"body": submit, "payload_max_bytes": 1 * KiB}},
                ],
            }},
        },
        {
            "name": "counter",
            "image": IMAGE,
            "user": {"uid": 65532, "gid": 65532},
            "needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
            "runtime": {"target": {"cwd": "/workspace"}},
        },
        {
            "name": "judge",
            "image": IMAGE,
            "user": {"uid": 65532, "gid": 65532},
            "needs": {"cpu_millis": 500, "memory_bytes": 256 * MiB},
            "runtime": {"target": {"cwd": "/pumpkin/home"}},
        },
    ],
    final = {"target": "judge", "body": final_call, "payload_max_bytes": 1 * KiB, "timeout_seconds": 60},
)

The bodies it loads, each a def with typed parameters, in examples/hello/count.star:

"""The body of the agent's job function `count`."""

def count(target: Target, payload: Payload) -> dict[str, int]:
    """Counts the payload's lines and bytes with `wc`, in a fresh sandbox of the counter.

    Args:
      target: the counter's sandbox.
      payload: the greeting.

    Returns:
      The greeting's number of lines and of bytes.
    """
    target.put("/workspace/greeting", payload)
    target.run(["sh", "-c", "printf '{\"lines\": %s, \"bytes\": %s}' $(wc -l < greeting) $(wc -c < greeting) > counts.json"])
    return json.decode(target.get("/workspace/counts.json"))

examples/submit.star, shared by the examples:

"""The agent's submissions."""

def submit(run: Run, payload: Payload, finalize: bool = False) -> dict[str, str]:
    """Selects `payload` for the final call, replacing the last.

    Args:
      run: the run's state.
      payload: the agent's submission.
      finalize: whether the call also ends the agent's work.

    Returns:
      The selection's state: `stored`, or `accepted` once the work has ended.
    """
    run.select(payload)
    if finalize:
        run.end_work()
        return {"state": "accepted"}
    return {"state": "stored"}

and examples/hello/final.star:

"""The hello example's final call: it counts the greeting's bytes and lines in its target."""

def final_call(target: Target, payload: Payload) -> dict[str, int]:
    """Shows the greeting's size as its score, and its lines.

    Args:
      target: the call's target.
      payload: the greeting the agent submitted.

    Returns:
      The score and the lines, in the order the run shows them.
    """
    target.put("/workspace/greeting", payload)
    target.run(["sh", "-c", "printf '{\"score\": %s, \"lines\": %s}' $(wc -c < /workspace/greeting) $(wc -l < /workspace/greeting) > /pumpkin/out/result.json"])
    result = json.decode(target.get("/pumpkin/out/result.json"))
    return {"score": result["score"], "lines": result["lines"]}

Roles. A role is a template of sandboxes: an image, the user its commands run as, never root, and what each of its sandboxes needs, from which the run picks the class of sandbox, among those the providers offer, that each is placed in. Its runtime says its kind. The one agent works on the task: its command runs from the run's start until it exits, its work ends, or deadline_seconds pass. A target runs nothing of its own: each call of the function that names it gets a fresh sandbox of it, which the function's body drives and which ends with the call. A runner is a sandbox that a role holding a session leases and drives. What a role may reach and call is written inside its kind: an agent holds egress and grants, a target egress and sessions, and a runner neither.

Grants. A grant is a name and one function, under its kind's key: {"name": "count", "job": {…}}. The agent calls it by name, sandboxctl call count greeting, and the file's bytes are the call's payload, at most the grant's payload_max_bytes. The driver checks every call against the grant before anything runs and refuses, with a reason, one that breaks it: a job function runs one call at a time, and its calls and cumulative_seconds, left out here, bound how many it runs and for how long.

A job function's body, here count, is a def of a file the entry loads, called once for each call with a Target and the payload. The Target is the handle on a fresh sandbox of the grant's target: target.put writes the payload as a file, target.run runs a command in the target's cwd and waits for it, and target.get reads back a file of at most 1 MiB as text. The dict the body returns is the job's result. The call answers at once with the job's number, and sandboxctl status 0 follows job 0 to its end. The agent never reaches the target: of a call it learns only the job's state, what the grant has left, and the dict.

A now function's body, here submit, answers in the call's reply, with no sandbox. The agent calls it as final, the name of its grant, which is not the final call. Its handle is a Run, the run's state: run.select(payload) makes the payload the selection, the one the final call gets, and run.end_work() ends the agent's work, after which the agent has finalize_grace_seconds to exit. The run applies what the body asked once it has returned, all of it or none. A parameter after the payload is the caller's: annotated str, int or bool, it is given as --arg finalize=true and read as its type before the body runs. sandboxctl ops lists each grant with its bounds and its caller's parameters.

Sessions are the third kind of function, for work that one call of a body can't hold, such as training on a GPU. A session grant names a runner and bounds its leases: how many are open at once, how long each and all of them last, how many commands they run, and how many bytes each copy moves. The role opens a lease with sandboxctl call <session>.open, runs commands in its runner and copies directory trees to and from it with .run, .put and .get, follows it with .status, and ends it with .close.

The final call runs once the agent's work has ended and everything it started has ended too: its body runs as a job's does, on a fresh sandbox of its target, which no job function may name, with the last payload selected. The dict it returns is the run's result, and its entries, in their order, are the values the run shows: score=6 lines=1 in pumpkin ls. A run with nothing selected, or whose final call fails, finishes without a result.

Egress is what a role reaches beyond its session. This agent has none; one that uses a model has a model route, whose entries each name an API, the models it allows and, if it is bounded, a budget: "egress": {"model": [{"api": "chat", "models": ["model-a"], "budget": {"microusd": 5 * USD}}]}. The route counts each entry's spend at the prices of the machine's configuration, and the model's key stays on the host.

What pumpkin check proves before anything is built: that every file evaluates, every image's recipe pins what it fetches, every key and bound of the workflow is one the IR takes, every role a grant or the final call names is of the kind it must be, and every body takes the handle and the payload that its kind passes, with caller's parameters of a type --arg gives. It says each problem at its file, line and column. A body's own statements run only when it is called.

docs/workflows.md holds every key a workflow writes and what sandboxctl answers; docs/language.md holds the language.

Run it

The operator, who runs pumpkin and holds the machine's configuration and keys, installs it in a repository of tasks as ./pumpkin, a copy of deploy/pumpkin beside a .pumpkinversion that names the tag and the commit to build, on Linux x86-64. A machine whose runs stay on its own Docker engine needs only this ~/.config/pumpkin/pumpkin.toml:

[docker]
context = "default"

[providers.local.docker]

With Pumpkin's examples/ directory copied into the repository of tasks, examples/hello, which needs no model, goes through the five commands:

$ ./pumpkin check examples/hello/task.star   # evaluates and checks; silent if no problem
$ ./pumpkin run examples/hello/task.star     # builds the images, starts the run, prints its id
20261116-195228-234cd166
$ ./pumpkin wait 234c                        # follows the run, named by a prefix of its digits
pumpkin: run 20261116-195228-234cd166 finished with a result (score=6 lines=1), ...
$ ./pumpkin ls                               # lists the runs live or ended in the last day
20261116-195228-234cd166 finished         2s  -               -  examples/hello: score=6 lines=1
$ ./pumpkin view                             # serves the runs to this machine's browser

docs/running.md goes through installing, configuring and these commands in order, with the full output of each.

Architecture

A task's sources go to pumpkin, which evaluates them to the workflow IR and builds the images; the driver, one process for each run, reads the IR, calls the functions' bodies from the sources the evaluation read, and has a provider create a sandbox for each instance of a role; it writes the record, which pumpkin wait, pumpkin ls and the page read.

A workflow becomes data

pumpkin evaluates a task's entry file, and the files it loads, to the workflow IR, which is JSON, and to the images its roles run, which it builds on the machine's Docker engine, each tagged by the digest of its inputs. The runtime reads only the IR, so it depends on no language. A function's body stays code: the IR names it by its file and its def, and the driver keeps the text of every file the evaluation read, in the run's record, and evaluates the body from that text at each call, so a call runs the body that was checked as the run started, whatever has changed on disk since. Evaluation is hermetic and always ends: it reads files only below the task's root and Pumpkin's library, and its arguments only from --arg.

A run is one process

The driver runs a run from start to end, on the machine that started it or on a controller. In it, the decision core takes each event that happened, such as a call that arrived or a sandbox that ended, and returns what to do next; it does no I/O and reads no clock, so the same events give the same actions. The runtime carries those actions out on the providers, the gateway holds one end of each sandbox's session, and the model route answers the roles' model requests.

The programs and machines of a run: the operator's pumpkin starts the driver on the operator's machine or on the controller; the driver's parts, the sandboxes that the providers create and the model provider that the route reaches.

A sandbox's only way out is its session

Every sandbox runs sandboxd, which meets the gateway and serves one session. Over it the driver puts and collects files and starts commands, and code in the sandbox makes its calls, with sandboxctl, and its model requests, at the endpoint sandboxd serves. Nothing else of the network is reachable: a Docker sandbox has loopback alone, an EC2 sandbox's instance drops all but the gateway's address and port, and a Modal sandbox opens no connection but DNS. ARCHITECTURE.md's security model says what each guard holds against.

The record

The driver writes each run's record as the run goes, in a directory of its own: the IR and the text of the files it came from, the run log of every event and what was done about it, each sandbox's output, each model exchange, and how the run ended. pumpkin wait, pumpkin ls and the page reads only the record, so each says the same about a run, wherever it ran.

Details: workflows · language · running · records · providers · models · architecture

Providers and the controller

A provider makes a run's sandboxes. Each role is placed in the smallest class of sandbox, of every provider that offers what it needs, such as its CPUs, its memory or its GPU.

Provider A sandbox is What the machine needs
docker a container on the machine's own Docker engine, on cores of its run's own the engine
ec2 a container on an EC2 instance of its own the envelope that deploy/aws makes
modal a sandbox in a Modal workspace, under gVisor a Modal profile, and the envelope's images

The controller is the same pumpkin, on an EC2 instance of the operator's own, which deploy/aws makes. A machine whose configuration names [controller] sends its runs there over ssh through SSM, and follows, lists, stops and views them there with the same commands; the controller builds the images and runs the drivers, so the operator's machine can go to sleep. It can also publish its runs to a read-only page for a team. docs/providers.md says how the pieces fit.

Documentation

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages