Skip to content

dstack-ingress: observability surface within the TEE trust boundary - #174

Draft
Leechael wants to merge 1 commit into
mainfrom
feat/ingress-visibility
Draft

Leechael wants to merge 1 commit into
mainfrom
feat/ingress-visibility

Conversation

@Leechael

Copy link
Copy Markdown
Contributor

Motivation

Debugging a long-lived-session incident through the stack (client → host HAProxy → dstack-gateway → WG → CVM) exposed how thin the ingress observability surface is:

  • sessions only log at end, with the default tcplog fields — no SNI, no TLS version, no correlation id, so a terminated session can't even be matched to a domain after the fact;
  • there is no runtime API to ask "is this flow still alive here, and how much is buffered" while it happens;
  • container logs vanish with the container, and a CVM restart loses them entirely;
  • the HAProxy build flags (e.g. +LINUX_SPLICE) have to be reverse-engineered from the image digest.

What this adds

All of it defaults to staying inside the CVM trust boundary — in the TEE threat model the host operator is untrusted, and on the dstack WireGuard network other tenants' CVMs are neighbors:

Addition Access
stats socket /var/run/haproxy/admin.sock, operator level (show sess/fd/info work; destructive admin commands disabled) whoever may exec into the container = the tenant
Extended TCP log-format: id=%ID sni=%[ssl_fc_sni] sslv=%sslv sslc=%sslc on top of the default fields (timers, %B, termination state %ts), plus unique-id-format stdout, unchanged
haproxy -vv printed at startup stdout
Explicit timeout tunnel (default = TIMEOUT_CLIENT, behavior unchanged) — in TCP mode every accepted connection is a tunnel and the fallback was implicit —
LOG_ASAP opt-in: log at accept, so a session that never ends still leaves a line stdout
SYSLOG_SERVER opt-in: off-box UDP syslog copy, with a startup warning — plaintext UDP only, must be a tenant-controlled collector tenant's explicit choice
METRICS_PORT opt-in: Prometheus exporter pinned to 127.0.0.1 (the built-in service has no auth; a wider bind would expose metadata to the whole WG network) loopback only

The config path becomes overridable (HAPROXY_CFG) so generation is testable off-container.

Tests

  • scripts/tests/test_observability_config.sh (new, needs only bash + a haproxy binary): 27 checks — socket level, log-format fields, tunnel-timeout default, opt-in emission, sanitizer rejection cases, and haproxy -c on both the default and the fully-opted-in config.
  • Runtime smoke (haproxy 3.4.7): TLS session with SNI through the generated config produces
    ... tls_in~ be_upstream/app1 6/3/8 5 CD ... id=7F000001:FAA0_... sni=rep.example.com sslv=TLSv1.3 sslc=TLS_AES_128_GCM_SHA256, and show info over the operator-level socket works.
  • Existing test_sanitizers.sh passes; shellcheck -S warning clean on all touched files.
  • Not run here: the docker-based tier-1 tests (test_proxy_protocol.sh, e2e) — no docker on this machine. The emit-trio call pattern they use is unchanged (TIMEOUT_TUNNEL has an in-lib default for exactly that reason), but they should run in CI / on a docker host before undrafting.

Deliberately out of scope

  • No admin-level socket, no expose-fd listeners, no routable metrics bind, no TLS-syslog (HAProxy doesn't speak it — a tenant-owned logging sidecar is the right shape, and needs no change here.
  • No unique-id forwarding to backends: PROXY v2 TLVs would be the channel, but the backend must understand them — left out until a consumer exists.
    EOF
    )

…dary

Incident response on long-lived proxied sessions (WebSocket and friends)
showed the current surface wanting: sessions only log at end with the
default tcplog fields (no SNI, no TLS version, no correlation id), there
is no runtime API to ask whether a flow is still alive, and container
logs vanish with the container.

Add, defaulting to stay inside the CVM trust boundary:

- stats socket at /var/run/haproxy/admin.sock, operator level: read-only
  show sess/fd/info work, destructive admin commands stay disabled.
- Extended TCP log-format: per-connection id, SNI, TLS version/cipher on
  top of the default tcplog fields (timers, bytes, termination state).
- haproxy -vv at startup so the exact build flags are in the log.
- Explicit 'timeout tunnel' (default = TIMEOUT_CLIENT, unchanged
  behavior): in TCP mode every accepted connection is a tunnel and the
  fallback was implicit.
- Opt-ins, each documented with its trust consequence: LOG_ASAP (log at
  accept for sessions that never end), SYSLOG_SERVER (off-box UDP syslog
  copy; plaintext, tenant-controlled collectors only), METRICS_PORT
  (Prometheus exporter pinned to loopback; the built-in service has no
  authentication and the WireGuard network is shared with other tenants).

The config path is overridable (HAPROXY_CFG) so generation is testable
off-container; scripts/tests/test_observability_config.sh exercises the
surface with only bash + a haproxy binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant