Repository navigation
Conversation
…dary Incident response on long-lived proxied sessions (WebSocket and friends) showed the current surface wanting: sessions only log at end with the default tcplog fields (no SNI, no TLS version, no correlation id), there is no runtime API to ask whether a flow is still alive, and container logs vanish with the container. Add, defaulting to stay inside the CVM trust boundary: - stats socket at /var/run/haproxy/admin.sock, operator level: read-only show sess/fd/info work, destructive admin commands stay disabled. - Extended TCP log-format: per-connection id, SNI, TLS version/cipher on top of the default tcplog fields (timers, bytes, termination state). - haproxy -vv at startup so the exact build flags are in the log. - Explicit 'timeout tunnel' (default = TIMEOUT_CLIENT, unchanged behavior): in TCP mode every accepted connection is a tunnel and the fallback was implicit. - Opt-ins, each documented with its trust consequence: LOG_ASAP (log at accept for sessions that never end), SYSLOG_SERVER (off-box UDP syslog copy; plaintext, tenant-controlled collectors only), METRICS_PORT (Prometheus exporter pinned to loopback; the built-in service has no authentication and the WireGuard network is shared with other tenants). The config path is overridable (HAPROXY_CFG) so generation is testable off-container; scripts/tests/test_observability_config.sh exercises the surface with only bash + a haproxy binary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Debugging a long-lived-session incident through the stack (client → host HAProxy → dstack-gateway → WG → CVM) exposed how thin the ingress observability surface is:
+LINUX_SPLICE) have to be reverse-engineered from the image digest.What this adds
All of it defaults to staying inside the CVM trust boundary — in the TEE threat model the host operator is untrusted, and on the dstack WireGuard network other tenants' CVMs are neighbors:
stats socket /var/run/haproxy/admin.sock, operator level (show sess/fd/infowork; destructiveadmincommands disabled)log-format:id=%ID sni=%[ssl_fc_sni] sslv=%sslv sslc=%sslcon top of the default fields (timers,%B, termination state%ts), plusunique-id-formathaproxy -vvprinted at startuptimeout tunnel(default =TIMEOUT_CLIENT, behavior unchanged) — in TCP mode every accepted connection is a tunnel and the fallback was implicitLOG_ASAPopt-in: log at accept, so a session that never ends still leaves a lineSYSLOG_SERVERopt-in: off-box UDP syslog copy, with a startup warning — plaintext UDP only, must be a tenant-controlled collectorMETRICS_PORTopt-in: Prometheus exporter pinned to127.0.0.1(the built-in service has no auth; a wider bind would expose metadata to the whole WG network)The config path becomes overridable (
HAPROXY_CFG) so generation is testable off-container.Tests
scripts/tests/test_observability_config.sh(new, needs only bash + ahaproxybinary): 27 checks — socket level, log-format fields, tunnel-timeout default, opt-in emission, sanitizer rejection cases, andhaproxy -con both the default and the fully-opted-in config.... tls_in~ be_upstream/app1 6/3/8 5 CD ... id=7F000001:FAA0_... sni=rep.example.com sslv=TLSv1.3 sslc=TLS_AES_128_GCM_SHA256, andshow infoover the operator-level socket works.test_sanitizers.shpasses;shellcheck -S warningclean on all touched files.test_proxy_protocol.sh, e2e) — no docker on this machine. The emit-trio call pattern they use is unchanged (TIMEOUT_TUNNELhas an in-lib default for exactly that reason), but they should run in CI / on a docker host before undrafting.Deliberately out of scope
admin-level socket, noexpose-fd listeners, no routable metrics bind, no TLS-syslog (HAProxy doesn't speak it — a tenant-owned logging sidecar is the right shape, and needs no change here.EOF
)