Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions custom-domain/dstack-ingress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,10 @@ otherwise arbitrary — the gateway answers every name under its domain.
| `TIMEOUT_CONNECT` | `10s` | Backend connect timeout |
| `TIMEOUT_CLIENT` | `86400s` | Client-side timeout (24h for long-lived connections) |
| `TIMEOUT_SERVER` | `86400s` | Server-side timeout |
| `TIMEOUT_TUNNEL` | value of `TIMEOUT_CLIENT` | Tunnel timeout. In TCP mode every accepted connection becomes a tunnel, so this supersedes the client/server timeouts for established connections; the default keeps the previous behavior |
| `LOG_ASAP` | `false` | Log at connection accept instead of at session end. A session that hangs past the container's lifetime still leaves a line, but its timers and byte counts are incomplete — the end-of-session line is the one with the termination state |
| `SYSLOG_SERVER` | | Off-box copy of the traffic log, `host:port` (UDP syslog). See [Observability](#observability) before enabling: the log carries client addresses, SNI and timing metadata, and HAProxy speaks plaintext UDP syslog only — point it at a collector you control |
| `METRICS_PORT` | | Prometheus exporter on `127.0.0.1:<port>`. Loopback only, deliberately: the built-in service has no authentication. See [Observability](#observability) |
| `EVIDENCE_SERVER` | `true` | Serve evidence files at `/evidences/` on the TLS port. Set `false` for backends whose server speaks first — see [Evidence & Attestation](#evidence--attestation) |
| `EVIDENCE_PORT` | `80` | Internal port for evidence HTTP server |
| `ALPN` | | TLS ALPN protocols (e.g. `h2,http/1.1`). Only set if backends support h2c |
Expand All @@ -211,6 +215,54 @@ otherwise arbitrary — the gateway answers every name under its domain.

Settings are checked at startup, and a bad value stops the container with an error naming the variable: booleans must be exactly `true` or `false`, ports 1-65535 without leading zeros, counts and intervals whole numbers, and every `DOMAINS` entry a valid hostname or `*.` wildcard.

## Observability

What a session did is reconstructable from the traffic log; what a session is
doing right now is answerable through the runtime API. Both are designed
around the TEE trust boundary: everything stays inside the CVM unless you
explicitly ship it out, and only you (whoever may exec into the container)
can read it.

**Traffic log** (stdout): the extended TCP log format records, per session,
the client and backend, the three timers (`Tw/Tc/Tt`), byte count, the
two-character termination state, a per-connection id, and the TLS SNI,
version and cipher:

```
127.0.0.1:64160 [14/Nov/2026:05:06:15.026] tls_in~ be_upstream/app1 6/3/8 5 CD 1/1/0/0/0 0/0 id=7F000001:FAA0_... sni=rep.example.com sslv=TLSv1.3 sslc=TLS_AES_128_GCM_SHA256
```

The termination state is the field that matters when sessions die young:
its first letter names which side ended the session — uppercase `C`/`S` =
the client/server closed or aborted, lowercase `c`/`s` = HAProxy's own
client/server timeout expired — and the second letter the phase (`D` =
data). The full table is section 8.5 of the HAProxy configuration manual.

**Runtime API**: a stats socket at `/var/run/haproxy/admin.sock`, operator
level — `show sess`, `show fd`, `show info` and the other read-only commands
work; the destructive admin commands (shutdown session, set server, clear
counters) are disabled. Example: `docker exec <container> sh -c 'echo show
sess | nc -U /var/run/haproxy/admin.sock'`.

**Metrics**: `METRICS_PORT=8405` serves the built-in Prometheus exporter on
`127.0.0.1:8405`. The exporter has no authentication, so the bind is pinned
to loopback — on the dstack WireGuard network a wider bind would be
reachable by other tenants' CVMs. Scrape it from a sidecar in your own
compose file if you want it collected.

**Off-box logs**: `SYSLOG_SERVER=host:port` adds a UDP syslog copy. Container
logs die with the container, and a CVM restart loses them entirely, so an
off-box copy is the only way to keep the last minutes of a crashing
deployment — but the log carries your clients' metadata, HAProxy speaks
plaintext UDP syslog only, and in the TEE threat model the host operator is
not trusted with it. Ship to a collector you control; for anything further,
run a logging sidecar (vector, fluent-bit) in your own compose file.

**Build identification**: startup prints the image's source revision (from
its OCI labels) and `haproxy -vv`, so the exact HAProxy version and its
build flags (e.g. splice support) are in the log before the first
connection.

For DNS provider credentials, see [DNS_PROVIDERS.md](DNS_PROVIDERS.md).

### ACME challenge delegation (least-privilege DNS token)
Expand Down
41 changes: 40 additions & 1 deletion custom-domain/dstack-ingress/scripts/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ if [ -r /etc/dstack-ingress/build-info ]; then
/etc/dstack-ingress/build-info
fi

# Build features of the haproxy binary (splice support, TLS library, exact
# version). Build flags like +LINUX_SPLICE change forwarding behavior, and
# incident response should not have to guess them from an image digest.
haproxy -vv

source /scripts/functions.sh
source /scripts/haproxy-lib.sh

Expand All @@ -28,6 +33,19 @@ MAXCONN=${MAXCONN:-4096}
TIMEOUT_CONNECT=${TIMEOUT_CONNECT:-10s}
TIMEOUT_CLIENT=${TIMEOUT_CLIENT:-86400s}
TIMEOUT_SERVER=${TIMEOUT_SERVER:-86400s}
# Unset, a tunnel (every accepted TCP connection) falls back to the
# client/server timeouts; pinning the default to TIMEOUT_CLIENT keeps that
# behavior while making the knob explicit.
TIMEOUT_TUNNEL=${TIMEOUT_TUNNEL:-$TIMEOUT_CLIENT}
# Log at connection accept instead of at session end: a session that hangs
# or outlives the container still leaves a line, but its timers/bytes are
# incomplete. Off by default because the end-of-session line is the one
# with the termination state.
LOG_ASAP=${LOG_ASAP:-false}
# Off-box log copy; see the warning below before setting it.
SYSLOG_SERVER=${SYSLOG_SERVER:-}
# Loopback-only Prometheus exporter; off when empty.
METRICS_PORT=${METRICS_PORT:-}
EVIDENCE_SERVER=${EVIDENCE_SERVER:-true}
EVIDENCE_PORT=${EVIDENCE_PORT:-80}
ALPN=${ALPN:-}
Expand Down Expand Up @@ -116,6 +134,26 @@ fi
if ! TIMEOUT_SERVER=$(sanitize_haproxy_timeout "$TIMEOUT_SERVER" "TIMEOUT_SERVER"); then
exit 1
fi
if ! TIMEOUT_TUNNEL=$(sanitize_haproxy_timeout "$TIMEOUT_TUNNEL" "TIMEOUT_TUNNEL"); then
exit 1
fi
if ! LOG_ASAP=$(sanitize_boolean "$LOG_ASAP" "LOG_ASAP"); then
exit 1
fi
if [ -n "$SYSLOG_SERVER" ]; then
if ! SYSLOG_SERVER=$(sanitize_syslog_server "$SYSLOG_SERVER"); then
exit 1
fi
# The traffic log carries client addresses, SNI and timing metadata. In
# the TEE threat model the host operator is an untrusted party, so this
# copy must go to a collector the *tenant* controls -- HAProxy speaks
# UDP syslog only, and a plaintext hop to somewhere inside the tenant's
# own network is the tenant's call to make.
echo "Warning: SYSLOG_SERVER ships traffic metadata (client IPs, SNI, timings) off this CVM; make sure $SYSLOG_SERVER is a collector you control" >&2
fi
if [ -n "$METRICS_PORT" ] && ! METRICS_PORT=$(sanitize_port "$METRICS_PORT" "METRICS_PORT"); then
exit 1
fi
if ! EVIDENCE_SERVER=$(sanitize_boolean "$EVIDENCE_SERVER" "EVIDENCE_SERVER"); then
exit 1
fi
Expand Down Expand Up @@ -171,7 +209,8 @@ fi
# Everything from here on belongs to one mode. Exported so the mode script and
# the helpers it invokes see the sanitized values.
export PORT DOMAIN DOMAINS GATEWAY_DOMAIN TARGET_ENDPOINT ROUTING_MAP TXT_PREFIX MAXCONN
export TIMEOUT_CONNECT TIMEOUT_CLIENT TIMEOUT_SERVER EVIDENCE_SERVER EVIDENCE_PORT ALPN
export TIMEOUT_CONNECT TIMEOUT_CLIENT TIMEOUT_SERVER TIMEOUT_TUNNEL LOG_ASAP SYSLOG_SERVER METRICS_PORT
export EVIDENCE_SERVER EVIDENCE_PORT ALPN
export ACCEPT_PROXY_PROTOCOL SEND_PROXY_PROTOCOL
export ACME_EMAIL ACME_STAGING DETERMINISTIC_TLS_KEY RENEW_INTERVAL RENEW_DAYS_BEFORE
export DNS_SETUP_TIMEOUT DNS_SETUP_INTERVAL DNS_SETUP_MODE DNS_WEBHOOK_TIMEOUT
Expand Down
23 changes: 23 additions & 0 deletions custom-domain/dstack-ingress/scripts/functions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,29 @@ sanitize_target_endpoint() {
fi
}

# A UDP syslog target, host:port. Reuses the host and port rules so a value
# that would corrupt the generated haproxy config (whitespace, quotes, extra
# fields) cannot reach it. Bracketed IPv6 ([::1]:514) works; bare IPv6 does
# not, matching how HAProxy writes log targets.
sanitize_syslog_server() {
local candidate="$1"
local host port
if [[ "$candidate" == \[*\]:* ]]; then
host="${candidate%]:*}]"
port="${candidate##*:}"
else
host="${candidate%:*}"
port="${candidate##*:}"
fi
if [ "$host" = "$candidate" ] || [ -z "$port" ] || [ "$port" = "$candidate" ]; then
echo "Error: Invalid SYSLOG_SERVER value: $candidate (must be host:port)" >&2
return 1
fi
host=$(sanitize_host "$host" SYSLOG_SERVER) || return 1
port=$(sanitize_port "$port" SYSLOG_SERVER) || return 1
echo "${host}:${port}"
}

sanitize_dns_label() {
local candidate="$1"
if [ -z "$candidate" ]; then
Expand Down
92 changes: 81 additions & 11 deletions custom-domain/dstack-ingress/scripts/haproxy-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@
# reload are the same either way, so they live here and each mode composes them.
# Nothing in this file branches on the mode; callers pass what differs.

# Where the rendered configuration goes. Overridable so the generation logic
# can be exercised off-container (scripts/tests/test_observability_config.sh);
# production
# never sets this.
HAPROXY_CFG="${HAPROXY_CFG:-/etc/haproxy/haproxy.cfg}"

# entrypoint.sh pins this to TIMEOUT_CLIENT; default it here too so the emit
# functions stay callable without the entrypoint (as the tests call them).
TIMEOUT_TUNNEL="${TIMEOUT_TUNNEL:-${TIMEOUT_CLIENT:-}}"

# Parse TARGET_ENDPOINT into host:port for haproxy backend
parse_target_endpoint() {
local endpoint="$1"
Expand All @@ -23,11 +33,36 @@ parse_target_endpoint() {

# global + defaults. Always the first thing written to haproxy.cfg.
haproxy_emit_global() {
cat <<EOF >/etc/haproxy/haproxy.cfg
# Optional second log destination, off the box. Off by default on
# purpose: the log carries client addresses, SNI and timing metadata,
# and in the TEE threat model the host operator is not trusted with
# tenant traffic metadata. SYSLOG_SERVER must point at a collector the
# tenant controls (HAProxy speaks UDP syslog only; TLS needs a sidecar).
local syslog_line=""
if [ -n "${SYSLOG_SERVER:-}" ]; then
syslog_line="
log ${SYSLOG_SERVER} format rfc5424 local0"
fi
local logasap_line=""
if [ "${LOG_ASAP:-false}" = "true" ]; then
logasap_line="
option logasap"
fi
cat <<EOF >"$HAPROXY_CFG"
global
log stdout format raw local0
log stdout format raw local0${syslog_line}
log-tag dstack-ingress
maxconn ${MAXCONN}
pidfile /var/run/haproxy/haproxy.pid
# Runtime API for incident response: show sess / show fd / show info
# answer "is this flow still alive here, and how much is buffered" while
# it is happening. The operator level covers every read-only command;
# the admin-level commands (shutdown session, set server, clear table)
# stay disabled. Unix socket, root-only, inside the container: access
# equals whoever may exec into the container, i.e. the tenant. Never
# volume-mount this out of the CVM.
stats socket /var/run/haproxy/admin.sock mode 660 level operator
stats timeout 30s
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
Expand All @@ -36,11 +71,45 @@ global
defaults
log global
mode tcp
option tcplog
# Default tcplog fields plus a per-connection id and the TLS parameters,
# so a terminated long-lived session can be matched to a domain and a
# TLS stack after the fact. The custom log-format replaces option tcplog.
# %ts is the two-character termination state; its first letter names
# which side's timer or close ended the session (lowercase = a timeout:
# c client, s server).
unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
log-format "%ci:%cp [%tr] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq id=%ID sni=%[ssl_fc_sni] sslv=%sslv sslc=%sslc"${logasap_line}
timeout connect ${TIMEOUT_CONNECT}
timeout client ${TIMEOUT_CLIENT}
timeout server ${TIMEOUT_SERVER}
# Tunnels: in TCP mode a connection becomes a tunnel as soon as no
# analyzer remains attached, and the tunnel timeout then supersedes the
# client/server timeouts. Unset, the fallback is implicit; pinning it to
# TIMEOUT_CLIENT keeps that behavior but makes the effective value
# visible in the rendered config.
timeout tunnel ${TIMEOUT_TUNNEL}

EOF
}

# Optional Prometheus exporter, loopback only. The built-in service has no
# authentication: binding it beyond loopback would serve per-connection
# metadata to everything that can route to it, which on the dstack WireGuard
# network includes other tenants' CVMs. Scrape it from a tenant-controlled
# sidecar inside this CVM, or not at all. Scrapes are logged (httplog): an
# unlogged HTTP listener would draw a config warning on every reload, and a
# steady one-line-per-scrape trail doubles as proof the scrape path works.
haproxy_emit_metrics() {
if [ -z "${METRICS_PORT:-}" ]; then
return 0
fi
cat <<EOF >>"$HAPROXY_CFG"
frontend metrics_in
bind 127.0.0.1:${METRICS_PORT}
mode http
option httplog
option dontlognull
http-request use-service prometheus-exporter
EOF
}

Expand All @@ -59,13 +128,13 @@ haproxy_emit_tls_frontend() {
local bind_spec="$1"
# "crt <dir>" loads all PEM files from the directory.
# ALPN is appended conditionally via ${ALPN:+ alpn ${ALPN}}.
cat <<EOF >>/etc/haproxy/haproxy.cfg
cat <<EOF >>"$HAPROXY_CFG"
frontend tls_in
bind ${bind_spec} ssl crt /etc/haproxy/certs/${ALPN:+ alpn ${ALPN}}
EOF

if [ "$EVIDENCE_SERVER" = "true" ]; then
cat <<'EVIDENCE_BLOCK' >>/etc/haproxy/haproxy.cfg
cat <<'EVIDENCE_BLOCK' >>"$HAPROXY_CFG"

# Route /evidences and /evidences/... to the local evidence HTTP
# server. The byte after the path has to be checked too, or an app
Expand Down Expand Up @@ -93,7 +162,7 @@ EVIDENCE_BLOCK
# Append the evidence backend block to haproxy.cfg
haproxy_emit_evidence_backend() {
if [ "$EVIDENCE_SERVER" = "true" ]; then
cat <<EOF >>/etc/haproxy/haproxy.cfg
cat <<EOF >>"$HAPROXY_CFG"

backend be_evidence
mode http
Expand All @@ -108,7 +177,7 @@ haproxy_emit_backends_single() {
local target_hostport
target_hostport=$(parse_target_endpoint "$TARGET_ENDPOINT")

cat <<EOF >>/etc/haproxy/haproxy.cfg
cat <<EOF >>"$HAPROXY_CFG"

default_backend be_upstream

Expand Down Expand Up @@ -156,15 +225,15 @@ backend ${be_name}
server s1 ${target}$(haproxy_send_proxy)"
done <<< "$routing_map_normalized"

echo "${exact_rules}${wildcard_rules}" >> /etc/haproxy/haproxy.cfg
echo "${exact_rules}${wildcard_rules}" >> "$HAPROXY_CFG"

# Default to first backend in ROUTING_MAP
if [ -n "$first_be_name" ]; then
echo "" >> /etc/haproxy/haproxy.cfg
echo " default_backend ${first_be_name}" >> /etc/haproxy/haproxy.cfg
echo "" >> "$HAPROXY_CFG"
echo " default_backend ${first_be_name}" >> "$HAPROXY_CFG"
fi

echo "$backend_sections" >> /etc/haproxy/haproxy.cfg
echo "$backend_sections" >> "$HAPROXY_CFG"
}

# Parse and validate ROUTING_MAP (newline- or comma-separated domain=target
Expand Down Expand Up @@ -214,6 +283,7 @@ haproxy_emit_backends() {
haproxy_emit_backends_single
fi
haproxy_emit_evidence_backend
haproxy_emit_metrics
}

haproxy_reload() {
Expand Down
Loading
Loading