feat(computers): an egress proxy in the ubuntu-24.04 computer, so live sites load at any universe date (FD-5478)

.github/dependabot.yml

@@ -70,6 +70,16 @@ updates:
     groups:
       infra-base-images:
         patterns: ["*"]
+  # The base computer's egress proxy, a Go program its Dockerfile builds: its modules' fixes, uTLS's and
+  # golang.org/x/crypto's among them. A bump changes the base's sources, so the next release rebuilds and publishes it.
+  - package-ecosystem: "gomod"
+    directory: "/src/agent_env/computers/artifacts/ubuntu-24.04/egress"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+    groups:
+      egress-go-modules:
+        patterns: ["*"]
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:

.github/workflows/local-backends.yml

@@ -25,6 +25,21 @@ env:
   PIP_DISABLE_PIP_VERSION_CHECK: "1"
 
 jobs:
+  # The base computer's egress proxy (src/agent_env/computers/artifacts/ubuntu-24.04/egress/): gofmt, go vet and its
+  # tests under the race detector, in the Go image its Dockerfile builds with, so a failing Go test fails here rather
+  # than as a broken base image in the slow tier.
+  egress:
+    name: egress
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+      # ECR Public's anonymous pulls fail here for hours at a time.
+      - name: Pull the Go image from mirror.gcr.io
+        run: python3 .github/scripts/mirror_base_images.py src/agent_env/computers/artifacts/ubuntu-24.04/Dockerfile
+      - name: gofmt, go vet and go test -race
+        run: make egress-test
+
   # The unit tier on two runners, AGENTENV_TEST_SHARD picking each one's share by the times in
   # tst/integration/slow_durations.json (tst/util/shards.py); unit passes when both shards do. The
   # lockfile and lint checks run on the first shard only.

AGENTS.md

@@ -45,6 +45,7 @@ or one of them only where it is used.
 | `make int-test-slow` | the `int_test_slow` tests, serially (they build the same Docker tags from module fixtures and race under xdist) |
 | `make installer-test` | `tst/installer`: `plugin add` / `remove` through the real pip, uv and pipx, offline against wheels built from the checkout, and the container user journey (`python:3.12-slim`, `--network none`) with the PEP 668 refusal; needs uv, pipx and Docker |
 | `make clock-test` | `tst/clock`: agent-env's libfaketime (`src/agent_env/computers/artifacts/ubuntu-24.04/clock`) and stock 0.9.12 built in an Ubuntu 24.04 image for this machine's Docker, and its gates run there: every wait API in C, Python, Java and Node at several rates, jumps, cancelled waits, timerfd duplicates, condvar and semaphore stress, the waits again with `futex_waitv` refused (as before Linux 5.16 and under gVisor), a stalled writer, a dead one, torn reads, the marker, the environment race, forks while another thread closes a timerfd, the TLS shim's certificate checks on real time (curl, Python and `openssl s_client` at 182 days back: live sites verify, expired and untrusted certificates don't; and two copies of OpenSSL in one process, the system's 3.0 and a 1.1, each checking with its own), and Chromium launched 40 times with it and 10 with stock, side by side, lateness held to `tst/clock/bounds.py`; needs Docker, and fetches Chrome's archive, upstream's tarball and OpenSSL 1.1.1w's once into `AGENTENV_TEST_CLOCK_CACHE` |
+| `make egress-test` | the base computer's egress proxy (`src/agent_env/computers/artifacts/ubuntu-24.04/egress`, Go): `gofmt`, `go vet` and `go test -race` in the Go image its Dockerfile builds with; needs Docker, not Go |
 | `make trl-test` | `tst/trl`: one `GRPOTrainer` step on CPU through `agent_env.training.trl`, with a tiny model from the Hub; needs trl and CPU torch, which the `dev` extra leaves out |
 | `make nemo-gym-test` | `tst/nemo_gym`: rollouts through `agent_env.training.nemo_gym`'s resources server in-process; needs the `nemo-gym` extra, on Python 3.13.14 or later, which the `dev` extra leaves out (`uv sync --extra dev --extra nemo-gym`) |
 | `make verl-test` | `tst/verl`: rows played through `agent_env.training.verl`'s agent loop on verl itself, with Qwen3's tokenizer from the Hub and a stand-in rollout server; needs verl and CPU torch, which the `dev` extra leaves out |
@@ -138,7 +139,7 @@ and deserialized through a registry.
 | `explorer/` | Optional local web UI: `agent-env up`, binds loopback `:8234`. |
 | `cli/` | Click CLI with the groups `a2a-agent`, `artifact`, `bundle`, `config`, `env`, `eval`, `plugin`, `run`, `task`, `up`. `task run`, `task run-batch` and `eval run` hand their runs to the configured `[runner]` and follow them. |
 | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. |
-| `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, settings as `AGENTENV_SETTING_<NAME>`; it starts as root, its container form's `start_as`, whose supervisord, `root.conf`, runs the clock follower as `agentenv-clock` and the session's supervisord as the session user, which runs every session program), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed.  `artifacts/ubuntu-24.04/clock/` is the computer's clock, in the base's folder since an image builds only from its own: agent-env's build of libfaketime 0.9.12 (`libfaketime.patch` and the two `agentenv_clock_*.c` files it includes, GPL-2.0-only as libfaketime is, built by `build.sh` from upstream's sha256-pinned tarball, in a stage of the base's Dockerfile of its own) and the TLS shim, which `/etc/ld.so.preload` loads into every process and which fake only the session user's: clock reads and waits for a set clock time follow the page the follower (`clock-follower`) writes (`clockpage.py`), monotonic and relative waits staying real, with its default settings (`clock.conf`); `tst/clock` gates it. The follower long-polls the gateway's `/clock/watch` under the config core writes through the root channel when a multi env deploys the computer and after a reset (`env/envs/computer/utils/clock.py`), each time waiting on the computer's `sync_time` until it follows, a warning naming why when it doesn't (arming the clock refuses such a computer), and the driver serves `clock/v1` `sync_time` where the follower is (`agentenv_protocol.computer.clock`), not on a disk guest, whose follower core can't reach. |
+| `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, settings as `AGENTENV_SETTING_<NAME>`; it starts as root, its container form's `start_as`, whose supervisord, `root.conf`, runs the clock follower as `agentenv-clock`, the egress proxy as `agentenv-egress` (`egress/`, a Go program its Dockerfile builds and tests, on real time, through which every session program goes out, so a live site loads at any universe date: it passes a site's TLS through when the universe is inside the site's certificate, else re-signs it with the computer's own CA, minted at boot and valid from 1950 to 9999, handshaking with the site as the client did, and refuses any destination that isn't global) and the session's supervisord as the session user, which runs every session program), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed.  `artifacts/ubuntu-24.04/clock/` is the computer's clock, in the base's folder since an image builds only from its own: agent-env's build of libfaketime 0.9.12 (`libfaketime.patch` and the two `agentenv_clock_*.c` files it includes, GPL-2.0-only as libfaketime is, built by `build.sh` from upstream's sha256-pinned tarball, in a stage of the base's Dockerfile of its own) and the TLS shim, which `/etc/ld.so.preload` loads into every process and which fake only the session user's: clock reads and waits for a set clock time follow the page the follower (`clock-follower`) writes (`clockpage.py`), monotonic and relative waits staying real, with its default settings (`clock.conf`); `tst/clock` gates it. The follower long-polls the gateway's `/clock/watch` under the config core writes through the root channel when a multi env deploys the computer and after a reset (`env/envs/computer/utils/clock.py`), each time waiting on the computer's `sync_time` until it follows, a warning naming why when it doesn't (arming the clock refuses such a computer), and the driver serves `clock/v1` `sync_time` where the follower is (`agentenv_protocol.computer.clock`), not on a disk guest, whose follower core can't reach. |
 | `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. |
 | `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default; each call an entry in the browser's log, begun and ended with `browser/v1` `calls/begin` and `calls/end`, that its CDP commands name as `for`; `scripts/upgrade_playwright_mcp.py` pins its version), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. |
 | `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, `<name> = "<package>"` shipping `src/<package as a path>/<name>/`. |
@@ -419,8 +420,8 @@ A merge into `main` needs an approving review from a code owner (`CODEOWNERS`),
 resolved, and the checks the `main` ruleset requires: `unit`, `integration-local` and
 `integration-local-slow`. The other checks below run on every pull request too and are expected to
 pass, but a failing one does not block the merge.
-`.github/workflows/local-backends.yml` runs `unit`, `ui`, `integration-local`, `integration-local-slow`,
-`integration-local-gateway` (the slow tier's `gateway_test.py`, run beside the rest of it) and
+`.github/workflows/local-backends.yml` runs `unit`, `ui`, `egress` (`make egress-test`), `integration-local`,
+`integration-local-slow`, `integration-local-gateway` (the slow tier's `gateway_test.py`, run beside the rest of it) and
 `installer`. The slow tier runs on twelve runners (`integration-local-slow (1/12)` to `(12/12)`), the
 gateway tests on five, and the unit and installer tiers on two each; `unit`, `integration-local-slow`,
 `integration-local-gateway` and `installer` each pass when all of their shards do. The Python jobs install from `uv.lock` (public PyPI only), with the uv that
@@ -489,9 +490,10 @@ image's digest. Run by hand on `main`, the workflow publishes the images alone.
 Its jobs that run agent-env (`.github/scripts/infra_images.py`) hold no token that writes; the jobs that push
 images or publish to PyPI install nothing and check what they're handed (`.github/scripts/infra_build.py`,
 standard library and the docker CLI only).
-Actions are pinned to commit SHAs; Dependabot updates them, the pinned uv, `uv.lock`, the UI's npm packages and
-the infra images' base images, each pinned by digest so that a bump rebuilds and republishes the image; Dependabot
-proposes only a base's digest and patch updates, and a major or minor upgrade is made by hand.
+Actions are pinned to commit SHAs; Dependabot updates them, the pinned uv, `uv.lock`, the UI's npm packages, the
+base computer's egress proxy's Go modules (`go.sum`), and the infra images' base images, each pinned by digest so
+that a bump rebuilds and republishes the image; Dependabot proposes only a base's digest and patch updates, and a
+major or minor upgrade is made by hand.
 
 ## Conventions
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install lint unit-test int-test int-test-fast int-test-slow installer-test clock-test trl-test nemo-gym-test verl-test skyrl-test miles-test slime-test clean-install-test test clean
+.PHONY: help install lint unit-test int-test int-test-fast int-test-slow installer-test clock-test egress-test trl-test nemo-gym-test verl-test skyrl-test miles-test slime-test clean-install-test test clean
 
 VENV := .venv
 VENV_BIN := $(abspath $(VENV))/bin
@@ -72,6 +72,14 @@ installer-test: ## Run plugin add/remove through the real pip, uv and pipx, and
 clock-test: ## Build agent-env's libfaketime and run its gates in Docker (tst/clock)
 	$(PYTHON) -m pytest tst/clock --timeout=2400 $(PYTEST_ARGS)
 
+# The base computer's egress proxy, in the Go image its Dockerfile builds it with, so it needs Docker, not Go.
+EGRESS := src/agent_env/computers/artifacts/ubuntu-24.04/egress
+EGRESS_GO_IMAGE = $(shell awk '/^FROM .*golang/ {print $$2; exit}' $(EGRESS)/../Dockerfile)
+
+egress-test: ## gofmt, go vet and go test -race for the base computer's egress proxy, in Docker
+	docker run --rm -v "$(CURDIR)/$(EGRESS):/src" -w /src $(EGRESS_GO_IMAGE) \
+		sh -c 'test -z "$$(gofmt -l .)" || { gofmt -l .; exit 1; }; go vet ./... && go test -race -count=1 ./...'
+
 trl-test: ## Run one GRPOTrainer step on CPU through the TRL helper (needs trl and CPU torch, and the Hub)
 	$(PYTHON) -m pytest tst/trl --timeout=900 $(PYTEST_ARGS)
 

THIRD_PARTY_NOTICES.md

@@ -2556,6 +2556,22 @@ The folder's other files, like the rest of the repository, are licensed under th
 - Source: <https://github.com/wolfcw/libfaketime/tree/v0.9.12>
 - License: GPL-2.0-only
 
+## The egress proxy's Go modules
+
+`src/agent_env/computers/artifacts/ubuntu-24.04/egress/` is agent-env's egress proxy, a Go program the base
+computer's Dockerfile builds, statically linked with the Go standard library and the modules below. Their source is
+not in the repository: the folder's `go.sum` pins each, the build downloads them, and the program runs inside
+computers.
+
+| Module | Version | License |
+| --- | --- | --- |
+| Go standard library | 1.26.11 | BSD-3-Clause |
+| github.com/refraction-networking/utls | v1.8.2 | BSD-3-Clause |
+| golang.org/x/crypto | v0.36.0 | BSD-3-Clause |
+| golang.org/x/sys | v0.31.0 | BSD-3-Clause |
+| github.com/andybalholm/brotli | v1.0.6 | MIT |
+| github.com/klauspost/compress | v1.17.4 | BSD-3-Clause (a few packages the proxy doesn't link are Apache-2.0 or MIT) |
+
 ## License texts
 
 License and notice files as shipped by each package, with copyright lines removed: they are listed with each package above. Packages whose files differ only in those lines share a text.

packages/agentenv-protocol/src/agentenv_protocol/computer/dataplane.py

@@ -15,7 +15,8 @@
 from pydantic import TypeAdapter, ValidationError
 
 from ..agent_env_environment import add_data, get_data
-from ..transfers import NamespaceUploader, ReadObject, RelativePath, TransferError, WriteNamespaceGrant, download
+from ..transfers import (NamespaceUploader, ReadObject, RelativePath, TransferError, WriteNamespaceGrant, client_options,
+                         download)
 from ..types import DATA_OBJECTS_EXTENSION_URI, FilePart, FileWithBytes, FileWithUri, uploaded_file_part, uploaded_object_path
 
 if TYPE_CHECKING:
@@ -115,7 +116,7 @@ async def _get_url(file: FileWithUri) -> bytes:
     if not file.uri.startswith("https://"):
         raise ValueError("a file part's uri is an https URL")
     data = bytearray()
-    async with httpx.AsyncClient(follow_redirects=True, timeout=60) as client:
+    async with httpx.AsyncClient(follow_redirects=True, timeout=60, **client_options()) as client:
         async with client.stream("GET", file.uri) as response:
             response.raise_for_status()
             async for chunk in response.aiter_bytes():