Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ Implementation details and infrastructure patterns.
| Example | Description |
|---------|-------------|
| [launcher](./launcher) | Generic launcher pattern for Docker Compose apps |
| [zero-downtime-upgrade](./zero-downtime-upgrade) | Upgrade a stateful app by handing over to a second CVM |
| [prelaunch-script](./prelaunch-script) | Pre-launch script patterns (Phala Cloud) |
| [private-docker-image-deployment](./private-docker-image-deployment) | Using private Docker registries |
| [attestation/rtmr3-based](./attestation/rtmr3-based) | RTMR3-based attestation (legacy) |
Expand Down
71 changes: 71 additions & 0 deletions zero-downtime-upgrade/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Zero-downtime upgrade

Updating a CVM's compose reboots it, and a stateful server is down until it boots again
(about 65 seconds for this example on Phala Cloud). This example upgrades without downtime:
it starts a second CVM of the same app on the new compose, hands the state and the writer
role to it, then deletes the old one.

The server is a small key-value store, inline Node in `docker-compose.yaml`. Its handover
code is what you would copy into your own server.

## How it works

Both CVMs get traffic from the gateway, so exactly one is the **writer** and the other
forwards to it. A **follower** forwards to its predecessor and takes over at boot; a
**retired** CVM forwards to its successor.

The handover is two calls from the new CVM B to the old CVM A. On `begin`, A holds requests
and returns its state. On `commit`, A retires and forwards to B, and B starts serving. With
no `commit` within 15 seconds, A resumes.

The calls are authenticated, and the state encrypted, with a key from `GetKey`. dstack derives
it from the app id, so A and B get the same key on different composes, and only a compose
the app allows can derive it.

`upgrade.sh` runs the upgrade:

1. `phala cvms replicate` A with `DORMANT=1`, so the replica only forwards.
2. `phala deploy --cvm-id B` with the new compose. Only B reboots; the gateway skips it while
its port is closed.
3. B boots on the new compose and takes over.
4. After 20 seconds for requests in flight through A, delete A.

## Run it

```bash
phala deploy -n kv -c docker-compose.yaml -t tdx.small --wait -e PREDECESSOR_URL= -e DORMANT=
node client.mjs https://<app_id>-8080.<gateway base> # in another terminal
sed 's/APP_VERSION=1/APP_VERSION=2/' docker-compose.yaml > v2.yaml
./upgrade.sh kv v2.yaml
```

`client.mjs` writes and reads a counter with no retries and prints every failure.
`upgrade.sh` needs `jq` and takes about two minutes. `node test-local.mjs` runs the same
upgrade on one machine.

The empty `-e` flags matter. dstack passes only env vars named in `allowed_envs`, which a
replica inherits. Without them the replica boots as a second writer with empty state; we saw
8 wrong reads in 20 seconds. `upgrade.sh` checks for them.

Change `APP_VERSION` with every change to the server code. Docker Compose does not recreate a
container when only an inline config's content changes, and the container survives the
reboot, so without a changed env var the CVM keeps running the old code.

The role is saved with the state and wins over the env at boot, because after an upgrade
`PREDECESSOR_URL` still names the deleted CVM.

## Results

Phala Cloud prod7, `tdx.small`, `dstack-dev-0.5.12`, with `client.mjs` running throughout:

| Upgrade | Failed requests | Longest request |
|---------|-----------------|-----------------|
| `phala deploy --cvm-id A` (in place) | 13, over 65 s | 5.4 s |
| `upgrade.sh` v1 → v2 → v3 | 0 of 334 | 1.2 s |
| `upgrade.sh` v1.1 → v2, this version of the server | 0 of 169 | 3.8 s |

While A is retired, a request through A costs about 10 ms more than one sent to B directly
(median of 12 each, after the TLS handshake). After A is deleted there is no forwarding.

State goes in one request, so a large database would hand over a pointer to encrypted
storage instead.
37 changes: 37 additions & 0 deletions zero-downtime-upgrade/client.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
// Writes then reads a counter through the app URL in a loop, with no retries and a fresh
// connection per request, and prints every failure. Ctrl-C prints the summary.
// Usage: node client.mjs https://<app_id>-8080.<gateway base>
import http from 'node:http'
import https from 'node:https'

const base = process.argv[2]
const req = (method, path, body) => new Promise((resolve, reject) => {
const r = (base.startsWith('https') ? https : http).request(base + path, { method, agent: false, timeout: 10000 }, res => {
let s = ''
res.on('data', d => { s += d })
res.on('end', () => resolve({ status: res.statusCode, body: s }))
})
r.on('timeout', () => r.destroy(new Error('timeout')))
r.on('error', reject)
r.end(body)
})

const stats = { ok: 0, fail: 0, regressions: 0, maxMs: 0, lastVersion: 0 }
process.on('SIGINT', () => { console.log(JSON.stringify(stats)); process.exit(0) })
for (let i = Date.now(); ; i++) {
const t = Date.now()
try {
const w = await req('PUT', '/kv/counter', String(i))
const r = await req('GET', '/kv/counter')
const got = JSON.parse(r.body)
if (w.status !== 200 || got.value !== String(i)) throw new Error(w.status + ' ' + w.body + ' / ' + r.status + ' ' + r.body)
if (got.version < stats.lastVersion) stats.regressions++
stats.lastVersion = got.version
if (++stats.ok % 50 === 0) console.log(new Date().toISOString(), JSON.stringify(stats))
} catch (e) {
stats.fail++
console.log(new Date(t).toISOString(), 'FAIL', e.message)
}
stats.maxMs = Math.max(stats.maxMs, Date.now() - t)
await new Promise(r => setTimeout(r, 100))
}
155 changes: 155 additions & 0 deletions zero-downtime-upgrade/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
services:
kv:
image: node:22-slim@sha256:e9f565d32859f0cbfb52125eafe8be39390f24dac302dc3cdf566e8ba3ab21ee
restart: unless-stopped
command: ["node", "/app/server.mjs"]
ports:
- "8080:8080"
environment:
# Change with every code change: compose doesn't recreate the container for config content alone.
- APP_VERSION=1
- PREDECESSOR_URL=${PREDECESSOR_URL:-}
- DORMANT=${DORMANT:-}
volumes:
- /var/run/dstack.sock:/var/run/dstack.sock
- kv:/data
configs:
- source: server
target: /app/server.mjs

volumes:
kv:

configs:
server:
content: |
// A key-value server that hands its state and writer role to a successor CVM of the same app.
// writer serves from disk; follower forwards to PREDECESSOR_URL and takes over at boot (unless
// DORMANT); retired forwards to its successor. Whichever instance the gateway picks, the writer answers.
import http from 'node:http'
import fs from 'node:fs'
import crypto from 'node:crypto'

const { PORT = 8080, DATA_FILE = '/data/kv.json', DSTACK_SOCK = '/var/run/dstack.sock',
PREDECESSOR_URL = '', DORMANT, SELF_URL, APP_VERSION } = process.env
let s = { kv: {}, version: 0, role: PREDECESSOR_URL ? 'follower' : 'writer', target: PREDECESSOR_URL }
// The role on disk wins over env: after an upgrade the env still names a predecessor that is gone.
if (fs.existsSync(DATA_FILE)) s = JSON.parse(fs.readFileSync(DATA_FILE, 'utf8'))
let hold = null, pending = null, committed = ''
const save = () => { fs.writeFileSync(DATA_FILE + '.tmp', JSON.stringify(s)); fs.renameSync(DATA_FILE + '.tmp', DATA_FILE) }
const sleep = ms => new Promise(r => setTimeout(r, ms))

const dstack = (method, body) => new Promise((resolve, reject) => {
const req = http.request({ socketPath: DSTACK_SOCK, path: '/' + method, method: 'POST',
headers: { 'content-type': 'application/json' } }, res => {
let out = ''
res.on('data', d => { out += d })
res.on('end', () => res.statusCode === 200 ? resolve(JSON.parse(out)) : reject(new Error(method + ' ' + res.statusCode + ' ' + out)))
})
req.on('error', reject)
req.end(JSON.stringify(body))
})

// Same app_id, same key: only CVMs running a compose hash this app allows can derive it.
const key = Buffer.from((await dstack('GetKey', { path: 'zero-downtime/handover' })).key.replace(/^0x/, ''), 'hex').subarray(0, 32)
const mac = body => crypto.createHmac('sha256', key).update(body).digest('hex')
function seal(obj, nonce) {
const iv = crypto.randomBytes(12), c = crypto.createCipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(nonce))
const ct = Buffer.concat([c.update(JSON.stringify(obj)), c.final()])
return { iv: iv.toString('hex'), ct: ct.toString('hex'), tag: c.getAuthTag().toString('hex') }
}
function open(box, nonce) {
const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(box.iv, 'hex')).setAAD(Buffer.from(nonce))
d.setAuthTag(Buffer.from(box.tag, 'hex'))
return JSON.parse(Buffer.concat([d.update(Buffer.from(box.ct, 'hex')), d.final()]).toString())
}
async function post(url, obj) {
const body = JSON.stringify(obj)
const r = await fetch(url, { method: 'POST', body, headers: { 'x-mac': mac(body) } })
if (r.status !== 200) throw Object.assign(new Error(url + ' ' + r.status), { status: r.status })
return r.json()
}

// Writer side. begin holds requests and returns the sealed state; commit retires this instance.
// Without a commit within 15 s it goes back to serving.
function begin({ nonce }) {
if (s.role !== 'writer' || pending) return [409, {}]
let release
hold = new Promise(r => { release = r })
pending = { nonce, release, timer: setTimeout(() => { pending = hold = null; release() }, 15000) }
return [200, seal({ kv: s.kv, version: s.version }, nonce)]
}
function commit({ nonce, successor }) {
if (committed === nonce) return [200, {}]
if (pending?.nonce !== nonce) return [409, {}]
clearTimeout(pending.timer)
committed = nonce
s.role = 'retired'
s.target = successor
save()
pending.release()
pending = hold = null
return [200, {}]
}

// Successor side: hold our own requests, take the state, commit, then serve.
async function takeOver() {
const { instance_id } = await dstack('Info', {})
const me = SELF_URL || PREDECESSOR_URL.replace(/\/\/[0-9a-f]+-/, '//' + instance_id + '-')
for (;;) {
const nonce = crypto.randomBytes(16).toString('hex')
let release
hold = new Promise(r => { release = r })
try {
const got = open(await post(s.target + '/_handover/begin', { nonce }), nonce)
for (let i = 0; ; i++) {
try { await post(s.target + '/_handover/commit', { nonce, successor: me }); break }
catch (e) { if (e.status === 409 || i === 4) throw e; await sleep(1000) }
}
s = { ...got, role: 'writer', target: '' }
save()
return console.log('took over at version', s.version)
} catch (e) {
console.log('handover failed, retrying:', e.message)
} finally {
hold = null
release()
}
await sleep(3000)
}
}

const send = (res, status, obj) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(obj)) }
const server = http.createServer(async (req, res) => {
let body = ''
for await (const d of req) body += d
try {
if (req.url === '/_status') return send(res, 200, { app: APP_VERSION, role: s.role, version: s.version })
if (req.url.startsWith('/_handover/')) {
if (req.headers['x-mac'] !== mac(body)) return send(res, 401, {})
return send(res, ...(req.url === '/_handover/begin' ? begin : commit)(JSON.parse(body)))
}
if (hold) await hold
if (s.role !== 'writer') {
const hops = Number(req.headers['x-zdt-hops'] || 0)
if (hops > 3) return send(res, 508, { error: 'forwarding loop' })
const r = await fetch(s.target + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body, headers: { 'x-zdt-hops': String(hops + 1) } })

Check warning on line 136 in zero-downtime-upgrade/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / Basic Checks (dev.sh)

136:161 [line-length] line too long (169 > 160 characters)

Check warning on line 136 in zero-downtime-upgrade/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / check-all

136:161 [line-length] line too long (169 > 160 characters)
res.writeHead(r.status, { 'content-type': 'application/json' })
return res.end(Buffer.from(await r.arrayBuffer()))
}
if (!req.url.startsWith('/kv/')) return send(res, 404, { error: 'use /kv/<key>' })
const k = decodeURIComponent(req.url.slice(4))
if (req.method === 'PUT') {
s.kv[k] = body
s.version++
save()
return send(res, 200, { version: s.version })
}
send(res, k in s.kv ? 200 : 404, { value: s.kv[k], version: s.version })
} catch (e) {
send(res, 502, { error: e.message })
}
}).listen(PORT, () => console.log('listening on', PORT, 'as', s.role))
process.on('SIGTERM', () => server.close(() => process.exit(0)))

if (s.role === 'follower' && !DORMANT) takeOver()
91 changes: 91 additions & 0 deletions zero-downtime-upgrade/test-local.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
// The upgrade with no TEE: a fake dstack socket, the server from docker-compose.yaml, a stand-in
// gateway, and client.mjs running throughout. Ends by restarting the new writer.
import http from 'node:http'
import fs from 'node:fs'
import os from 'node:os'
import path from 'node:path'
import { spawn } from 'node:child_process'

const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'zdt-'))
const compose = fs.readFileSync(new URL('./docker-compose.yaml', import.meta.url), 'utf8').split('\n')
const server = path.join(dir, 'server.mjs')
fs.writeFileSync(server, compose.slice(compose.indexOf(' content: |') + 1).map(l => l.slice(6)).join('\n'))
const sock = path.join(dir, 'dstack.sock')
// The real guest agent decodes a body without this content type as protobuf and fails.
http.createServer((req, res) => req.headers['content-type'] !== 'application/json' ? res.writeHead(400).end()
: res.end(JSON.stringify(req.url === '/GetKey' ? { key: '11'.repeat(32) } : { instance_id: 'local' }))).listen(sock)

const sleep = ms => new Promise(r => setTimeout(r, ms))
const status = port => fetch('http://127.0.0.1:' + port + '/_status').then(r => r.json())
const procs = {}
function run(name, port, env) {
fs.mkdirSync(path.join(dir, name), { recursive: true })
procs[name] = spawn('node', [server], { env: { ...process.env, PORT: String(port), DSTACK_SOCK: sock, DATA_FILE: path.join(dir, name, 'kv.json'), ...env } })
procs[name].stdout.on('data', d => process.stdout.write(name + ': ' + d))
procs[name].stderr.on('data', d => process.stdout.write(name + '! ' + d))
}

// Like dstack-gateway, the stand-in rotates over every instance of the app, writer or not, and
// skips one whose port refuses the connection (a CVM that is rebooting).
const backends = [8101]
let rr = 0
http.createServer(async (req, res) => {
let body = ''
for await (const d of req) body += d
const first = rr++
for (let i = 0; i < backends.length; i++) {
try {
const r = await fetch('http://127.0.0.1:' + backends.at((first + i) % backends.length) + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body })
res.writeHead(r.status)
return res.end(Buffer.from(await r.arrayBuffer()))
} catch (e) {
if (e.cause?.code !== 'ECONNREFUSED') { res.writeHead(599); return res.end(e.message) }
}
}
res.writeHead(599)
res.end('no instance accepted the connection')
}).listen(8100)

const b = { PREDECESSOR_URL: 'http://127.0.0.1:8101', SELF_URL: 'http://127.0.0.1:8102' }
run('a', 8101, { APP_VERSION: '1' })
await sleep(500)
const client = spawn('node', [new URL('./client.mjs', import.meta.url).pathname, 'http://127.0.0.1:8100'])
let out = ''
client.stdout.on('data', d => { out += d; process.stdout.write('client: ' + d) })
await sleep(2000)

console.log('--- B boots dormant, like a fresh replica, and the gateway starts sending it traffic')
run('b', 8102, { ...b, DORMANT: '1' })
await sleep(300)
backends.push(8102)
await sleep(1500)
const dormant = await status(8102)
console.log('--- B restarts on the new compose and takes over')
procs.b.kill()
await sleep(300)
run('b', 8102, { ...b, APP_VERSION: '2' })
await sleep(3000)
const sa = await status(8101), sb = await status(8102)
console.log('A', sa, 'B', sb)
console.log('--- A is removed')
backends.shift()
await sleep(1000)
procs.a.kill()
await sleep(2000)
client.kill('SIGINT')
await sleep(300)
const stats = JSON.parse(out.trim().split('\n').at(-1))

console.log('--- B restarts; its env still names the stopped A as its predecessor')
procs.b.kill()
await sleep(300)
run('b', 8102, { ...b, APP_VERSION: '2' })
await sleep(500)
const restarted = await status(8102)
console.log('restarted B', restarted)
Object.values(procs).forEach(p => p.kill())

const pass = stats.ok > 30 && stats.fail === 0 && stats.regressions === 0 && dormant.role === 'follower' && sa.role === 'retired'
&& sb.role === 'writer' && sb.app === '2' && restarted.role === 'writer' && restarted.version === stats.lastVersion
console.log(pass ? 'PASS' : 'FAIL')
process.exit(pass ? 0 : 1)
31 changes: 31 additions & 0 deletions zero-downtime-upgrade/upgrade.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/sh
# Upgrades a running CVM to a new compose with no downtime.
# Usage: ./upgrade.sh <current CVM name or uuid> <new docker-compose.yaml>
set -eu
A=$1
NEW=$2

info=$(phala cvms get "$A" --json)
a_url=https://$(printf "%s\n" "$info" | jq -r '.instance_id + "-8080." + .gateway.base_domain')

# dstack drops env vars missing from allowed_envs, and the replica inherits A's list. A replica
# without DORMANT and PREDECESSOR_URL would boot as a second writer with empty state.
printf "%s\n" "$info" | jq -e '.compose_file | if type == "string" then fromjson else . end | .allowed_envs | index("DORMANT") and index("PREDECESSOR_URL")' > /dev/null \
|| { echo "$A was not deployed with -e PREDECESSOR_URL= -e DORMANT=" >&2; exit 1; }

# 1. A replica of A with the same compose. DORMANT keeps it forwarding to A instead of taking over.
env=$(mktemp)
printf 'DORMANT=1\nPREDECESSOR_URL=%s\n' "$a_url" > "$env"
B=$(phala cvms replicate "$A" -e "$env" --json | jq -r .vm_uuid)
echo "replica $B"
until phala cvms get "$B" --json | jq -e '.status == "running"' > /dev/null; do sleep 10; done

# 2. B alone reboots into the new compose and takes the writer role and state from A.
phala deploy --cvm-id "$B" -c "$NEW" -e PREDECESSOR_URL="$a_url" -e DORMANT= --wait
b_url=https://$(phala cvms get "$B" --json | jq -r '.instance_id + "-8080." + .gateway.base_domain')
until curl -sf "$b_url/_status" | jq -e '.role == "writer"' > /dev/null; do sleep 3; done

# 3. A now forwards to B. Let requests already routed to A finish, then remove it.
sleep 20
phala cvms delete "$A" --force
echo "upgraded: $B is the writer"
Loading