diff --git a/README.md b/README.md index da7300e..319fa76 100644 --- a/README.md +++ b/README.md @@ -233,6 +233,7 @@ Implementation details and infrastructure patterns. | Example | Description | |---------|-------------| | [launcher](./launcher) | Generic launcher pattern for Docker Compose apps | +| [zero-downtime-upgrade](./zero-downtime-upgrade) | Upgrade a stateful app by handing over to a second CVM | | [prelaunch-script](./prelaunch-script) | Pre-launch script patterns (Phala Cloud) | | [private-docker-image-deployment](./private-docker-image-deployment) | Using private Docker registries | | [attestation/rtmr3-based](./attestation/rtmr3-based) | RTMR3-based attestation (legacy) | diff --git a/zero-downtime-upgrade/README.md b/zero-downtime-upgrade/README.md new file mode 100644 index 0000000..9941240 --- /dev/null +++ b/zero-downtime-upgrade/README.md @@ -0,0 +1,71 @@ +# Zero-downtime upgrade + +Updating a CVM's compose reboots it, and a stateful server is down until it boots again +(about 65 seconds for this example on Phala Cloud). This example upgrades without downtime: +it starts a second CVM of the same app on the new compose, hands the state and the writer +role to it, then deletes the old one. + +The server is a small key-value store, inline Node in `docker-compose.yaml`. Its handover +code is what you would copy into your own server. + +## How it works + +Both CVMs get traffic from the gateway, so exactly one is the **writer** and the other +forwards to it. A **follower** forwards to its predecessor and takes over at boot; a +**retired** CVM forwards to its successor. + +The handover is two calls from the new CVM B to the old CVM A. On `begin`, A holds requests +and returns its state. On `commit`, A retires and forwards to B, and B starts serving. With +no `commit` within 15 seconds, A resumes. + +The calls are authenticated, and the state encrypted, with a key from `GetKey`. dstack derives +it from the app id, so A and B get the same key on different composes, and only a compose +the app allows can derive it. + +`upgrade.sh` runs the upgrade: + +1. `phala cvms replicate` A with `DORMANT=1`, so the replica only forwards. +2. `phala deploy --cvm-id B` with the new compose. Only B reboots; the gateway skips it while + its port is closed. +3. B boots on the new compose and takes over. +4. After 20 seconds for requests in flight through A, delete A. + +## Run it + +```bash +phala deploy -n kv -c docker-compose.yaml -t tdx.small --wait -e PREDECESSOR_URL= -e DORMANT= +node client.mjs https://-8080. # in another terminal +sed 's/APP_VERSION=1/APP_VERSION=2/' docker-compose.yaml > v2.yaml +./upgrade.sh kv v2.yaml +``` + +`client.mjs` writes and reads a counter with no retries and prints every failure. +`upgrade.sh` needs `jq` and takes about two minutes. `node test-local.mjs` runs the same +upgrade on one machine. + +The empty `-e` flags matter. dstack passes only env vars named in `allowed_envs`, which a +replica inherits. Without them the replica boots as a second writer with empty state; we saw +8 wrong reads in 20 seconds. `upgrade.sh` checks for them. + +Change `APP_VERSION` with every change to the server code. Docker Compose does not recreate a +container when only an inline config's content changes, and the container survives the +reboot, so without a changed env var the CVM keeps running the old code. + +The role is saved with the state and wins over the env at boot, because after an upgrade +`PREDECESSOR_URL` still names the deleted CVM. + +## Results + +Phala Cloud prod7, `tdx.small`, `dstack-dev-0.5.12`, with `client.mjs` running throughout: + +| Upgrade | Failed requests | Longest request | +|---------|-----------------|-----------------| +| `phala deploy --cvm-id A` (in place) | 13, over 65 s | 5.4 s | +| `upgrade.sh` v1 → v2 → v3 | 0 of 334 | 1.2 s | +| `upgrade.sh` v1.1 → v2, this version of the server | 0 of 169 | 3.8 s | + +While A is retired, a request through A costs about 10 ms more than one sent to B directly +(median of 12 each, after the TLS handshake). After A is deleted there is no forwarding. + +State goes in one request, so a large database would hand over a pointer to encrypted +storage instead. diff --git a/zero-downtime-upgrade/client.mjs b/zero-downtime-upgrade/client.mjs new file mode 100644 index 0000000..c5767d9 --- /dev/null +++ b/zero-downtime-upgrade/client.mjs @@ -0,0 +1,37 @@ +// Writes then reads a counter through the app URL in a loop, with no retries and a fresh +// connection per request, and prints every failure. Ctrl-C prints the summary. +// Usage: node client.mjs https://-8080. +import http from 'node:http' +import https from 'node:https' + +const base = process.argv[2] +const req = (method, path, body) => new Promise((resolve, reject) => { + const r = (base.startsWith('https') ? https : http).request(base + path, { method, agent: false, timeout: 10000 }, res => { + let s = '' + res.on('data', d => { s += d }) + res.on('end', () => resolve({ status: res.statusCode, body: s })) + }) + r.on('timeout', () => r.destroy(new Error('timeout'))) + r.on('error', reject) + r.end(body) +}) + +const stats = { ok: 0, fail: 0, regressions: 0, maxMs: 0, lastVersion: 0 } +process.on('SIGINT', () => { console.log(JSON.stringify(stats)); process.exit(0) }) +for (let i = Date.now(); ; i++) { + const t = Date.now() + try { + const w = await req('PUT', '/kv/counter', String(i)) + const r = await req('GET', '/kv/counter') + const got = JSON.parse(r.body) + if (w.status !== 200 || got.value !== String(i)) throw new Error(w.status + ' ' + w.body + ' / ' + r.status + ' ' + r.body) + if (got.version < stats.lastVersion) stats.regressions++ + stats.lastVersion = got.version + if (++stats.ok % 50 === 0) console.log(new Date().toISOString(), JSON.stringify(stats)) + } catch (e) { + stats.fail++ + console.log(new Date(t).toISOString(), 'FAIL', e.message) + } + stats.maxMs = Math.max(stats.maxMs, Date.now() - t) + await new Promise(r => setTimeout(r, 100)) +} diff --git a/zero-downtime-upgrade/docker-compose.yaml b/zero-downtime-upgrade/docker-compose.yaml new file mode 100644 index 0000000..3908f2f --- /dev/null +++ b/zero-downtime-upgrade/docker-compose.yaml @@ -0,0 +1,155 @@ +services: + kv: + image: node:22-slim@sha256:e9f565d32859f0cbfb52125eafe8be39390f24dac302dc3cdf566e8ba3ab21ee + restart: unless-stopped + command: ["node", "/app/server.mjs"] + ports: + - "8080:8080" + environment: + # Change with every code change: compose doesn't recreate the container for config content alone. + - APP_VERSION=1 + - PREDECESSOR_URL=${PREDECESSOR_URL:-} + - DORMANT=${DORMANT:-} + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - kv:/data + configs: + - source: server + target: /app/server.mjs + +volumes: + kv: + +configs: + server: + content: | + // A key-value server that hands its state and writer role to a successor CVM of the same app. + // writer serves from disk; follower forwards to PREDECESSOR_URL and takes over at boot (unless + // DORMANT); retired forwards to its successor. Whichever instance the gateway picks, the writer answers. + import http from 'node:http' + import fs from 'node:fs' + import crypto from 'node:crypto' + + const { PORT = 8080, DATA_FILE = '/data/kv.json', DSTACK_SOCK = '/var/run/dstack.sock', + PREDECESSOR_URL = '', DORMANT, SELF_URL, APP_VERSION } = process.env + let s = { kv: {}, version: 0, role: PREDECESSOR_URL ? 'follower' : 'writer', target: PREDECESSOR_URL } + // The role on disk wins over env: after an upgrade the env still names a predecessor that is gone. + if (fs.existsSync(DATA_FILE)) s = JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')) + let hold = null, pending = null, committed = '' + const save = () => { fs.writeFileSync(DATA_FILE + '.tmp', JSON.stringify(s)); fs.renameSync(DATA_FILE + '.tmp', DATA_FILE) } + const sleep = ms => new Promise(r => setTimeout(r, ms)) + + const dstack = (method, body) => new Promise((resolve, reject) => { + const req = http.request({ socketPath: DSTACK_SOCK, path: '/' + method, method: 'POST', + headers: { 'content-type': 'application/json' } }, res => { + let out = '' + res.on('data', d => { out += d }) + res.on('end', () => res.statusCode === 200 ? resolve(JSON.parse(out)) : reject(new Error(method + ' ' + res.statusCode + ' ' + out))) + }) + req.on('error', reject) + req.end(JSON.stringify(body)) + }) + + // Same app_id, same key: only CVMs running a compose hash this app allows can derive it. + const key = Buffer.from((await dstack('GetKey', { path: 'zero-downtime/handover' })).key.replace(/^0x/, ''), 'hex').subarray(0, 32) + const mac = body => crypto.createHmac('sha256', key).update(body).digest('hex') + function seal(obj, nonce) { + const iv = crypto.randomBytes(12), c = crypto.createCipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(nonce)) + const ct = Buffer.concat([c.update(JSON.stringify(obj)), c.final()]) + return { iv: iv.toString('hex'), ct: ct.toString('hex'), tag: c.getAuthTag().toString('hex') } + } + function open(box, nonce) { + const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(box.iv, 'hex')).setAAD(Buffer.from(nonce)) + d.setAuthTag(Buffer.from(box.tag, 'hex')) + return JSON.parse(Buffer.concat([d.update(Buffer.from(box.ct, 'hex')), d.final()]).toString()) + } + async function post(url, obj) { + const body = JSON.stringify(obj) + const r = await fetch(url, { method: 'POST', body, headers: { 'x-mac': mac(body) } }) + if (r.status !== 200) throw Object.assign(new Error(url + ' ' + r.status), { status: r.status }) + return r.json() + } + + // Writer side. begin holds requests and returns the sealed state; commit retires this instance. + // Without a commit within 15 s it goes back to serving. + function begin({ nonce }) { + if (s.role !== 'writer' || pending) return [409, {}] + let release + hold = new Promise(r => { release = r }) + pending = { nonce, release, timer: setTimeout(() => { pending = hold = null; release() }, 15000) } + return [200, seal({ kv: s.kv, version: s.version }, nonce)] + } + function commit({ nonce, successor }) { + if (committed === nonce) return [200, {}] + if (pending?.nonce !== nonce) return [409, {}] + clearTimeout(pending.timer) + committed = nonce + s.role = 'retired' + s.target = successor + save() + pending.release() + pending = hold = null + return [200, {}] + } + + // Successor side: hold our own requests, take the state, commit, then serve. + async function takeOver() { + const { instance_id } = await dstack('Info', {}) + const me = SELF_URL || PREDECESSOR_URL.replace(/\/\/[0-9a-f]+-/, '//' + instance_id + '-') + for (;;) { + const nonce = crypto.randomBytes(16).toString('hex') + let release + hold = new Promise(r => { release = r }) + try { + const got = open(await post(s.target + '/_handover/begin', { nonce }), nonce) + for (let i = 0; ; i++) { + try { await post(s.target + '/_handover/commit', { nonce, successor: me }); break } + catch (e) { if (e.status === 409 || i === 4) throw e; await sleep(1000) } + } + s = { ...got, role: 'writer', target: '' } + save() + return console.log('took over at version', s.version) + } catch (e) { + console.log('handover failed, retrying:', e.message) + } finally { + hold = null + release() + } + await sleep(3000) + } + } + + const send = (res, status, obj) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(obj)) } + const server = http.createServer(async (req, res) => { + let body = '' + for await (const d of req) body += d + try { + if (req.url === '/_status') return send(res, 200, { app: APP_VERSION, role: s.role, version: s.version }) + if (req.url.startsWith('/_handover/')) { + if (req.headers['x-mac'] !== mac(body)) return send(res, 401, {}) + return send(res, ...(req.url === '/_handover/begin' ? begin : commit)(JSON.parse(body))) + } + if (hold) await hold + if (s.role !== 'writer') { + const hops = Number(req.headers['x-zdt-hops'] || 0) + if (hops > 3) return send(res, 508, { error: 'forwarding loop' }) + const r = await fetch(s.target + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body, headers: { 'x-zdt-hops': String(hops + 1) } }) + res.writeHead(r.status, { 'content-type': 'application/json' }) + return res.end(Buffer.from(await r.arrayBuffer())) + } + if (!req.url.startsWith('/kv/')) return send(res, 404, { error: 'use /kv/' }) + const k = decodeURIComponent(req.url.slice(4)) + if (req.method === 'PUT') { + s.kv[k] = body + s.version++ + save() + return send(res, 200, { version: s.version }) + } + send(res, k in s.kv ? 200 : 404, { value: s.kv[k], version: s.version }) + } catch (e) { + send(res, 502, { error: e.message }) + } + }).listen(PORT, () => console.log('listening on', PORT, 'as', s.role)) + process.on('SIGTERM', () => server.close(() => process.exit(0))) + + if (s.role === 'follower' && !DORMANT) takeOver() diff --git a/zero-downtime-upgrade/test-local.mjs b/zero-downtime-upgrade/test-local.mjs new file mode 100644 index 0000000..7edafbd --- /dev/null +++ b/zero-downtime-upgrade/test-local.mjs @@ -0,0 +1,91 @@ +// The upgrade with no TEE: a fake dstack socket, the server from docker-compose.yaml, a stand-in +// gateway, and client.mjs running throughout. Ends by restarting the new writer. +import http from 'node:http' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { spawn } from 'node:child_process' + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'zdt-')) +const compose = fs.readFileSync(new URL('./docker-compose.yaml', import.meta.url), 'utf8').split('\n') +const server = path.join(dir, 'server.mjs') +fs.writeFileSync(server, compose.slice(compose.indexOf(' content: |') + 1).map(l => l.slice(6)).join('\n')) +const sock = path.join(dir, 'dstack.sock') +// The real guest agent decodes a body without this content type as protobuf and fails. +http.createServer((req, res) => req.headers['content-type'] !== 'application/json' ? res.writeHead(400).end() + : res.end(JSON.stringify(req.url === '/GetKey' ? { key: '11'.repeat(32) } : { instance_id: 'local' }))).listen(sock) + +const sleep = ms => new Promise(r => setTimeout(r, ms)) +const status = port => fetch('http://127.0.0.1:' + port + '/_status').then(r => r.json()) +const procs = {} +function run(name, port, env) { + fs.mkdirSync(path.join(dir, name), { recursive: true }) + procs[name] = spawn('node', [server], { env: { ...process.env, PORT: String(port), DSTACK_SOCK: sock, DATA_FILE: path.join(dir, name, 'kv.json'), ...env } }) + procs[name].stdout.on('data', d => process.stdout.write(name + ': ' + d)) + procs[name].stderr.on('data', d => process.stdout.write(name + '! ' + d)) +} + +// Like dstack-gateway, the stand-in rotates over every instance of the app, writer or not, and +// skips one whose port refuses the connection (a CVM that is rebooting). +const backends = [8101] +let rr = 0 +http.createServer(async (req, res) => { + let body = '' + for await (const d of req) body += d + const first = rr++ + for (let i = 0; i < backends.length; i++) { + try { + const r = await fetch('http://127.0.0.1:' + backends.at((first + i) % backends.length) + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body }) + res.writeHead(r.status) + return res.end(Buffer.from(await r.arrayBuffer())) + } catch (e) { + if (e.cause?.code !== 'ECONNREFUSED') { res.writeHead(599); return res.end(e.message) } + } + } + res.writeHead(599) + res.end('no instance accepted the connection') +}).listen(8100) + +const b = { PREDECESSOR_URL: 'http://127.0.0.1:8101', SELF_URL: 'http://127.0.0.1:8102' } +run('a', 8101, { APP_VERSION: '1' }) +await sleep(500) +const client = spawn('node', [new URL('./client.mjs', import.meta.url).pathname, 'http://127.0.0.1:8100']) +let out = '' +client.stdout.on('data', d => { out += d; process.stdout.write('client: ' + d) }) +await sleep(2000) + +console.log('--- B boots dormant, like a fresh replica, and the gateway starts sending it traffic') +run('b', 8102, { ...b, DORMANT: '1' }) +await sleep(300) +backends.push(8102) +await sleep(1500) +const dormant = await status(8102) +console.log('--- B restarts on the new compose and takes over') +procs.b.kill() +await sleep(300) +run('b', 8102, { ...b, APP_VERSION: '2' }) +await sleep(3000) +const sa = await status(8101), sb = await status(8102) +console.log('A', sa, 'B', sb) +console.log('--- A is removed') +backends.shift() +await sleep(1000) +procs.a.kill() +await sleep(2000) +client.kill('SIGINT') +await sleep(300) +const stats = JSON.parse(out.trim().split('\n').at(-1)) + +console.log('--- B restarts; its env still names the stopped A as its predecessor') +procs.b.kill() +await sleep(300) +run('b', 8102, { ...b, APP_VERSION: '2' }) +await sleep(500) +const restarted = await status(8102) +console.log('restarted B', restarted) +Object.values(procs).forEach(p => p.kill()) + +const pass = stats.ok > 30 && stats.fail === 0 && stats.regressions === 0 && dormant.role === 'follower' && sa.role === 'retired' + && sb.role === 'writer' && sb.app === '2' && restarted.role === 'writer' && restarted.version === stats.lastVersion +console.log(pass ? 'PASS' : 'FAIL') +process.exit(pass ? 0 : 1) diff --git a/zero-downtime-upgrade/upgrade.sh b/zero-downtime-upgrade/upgrade.sh new file mode 100755 index 0000000..dc573fb --- /dev/null +++ b/zero-downtime-upgrade/upgrade.sh @@ -0,0 +1,31 @@ +#!/bin/sh +# Upgrades a running CVM to a new compose with no downtime. +# Usage: ./upgrade.sh +set -eu +A=$1 +NEW=$2 + +info=$(phala cvms get "$A" --json) +a_url=https://$(printf "%s\n" "$info" | jq -r '.instance_id + "-8080." + .gateway.base_domain') + +# dstack drops env vars missing from allowed_envs, and the replica inherits A's list. A replica +# without DORMANT and PREDECESSOR_URL would boot as a second writer with empty state. +printf "%s\n" "$info" | jq -e '.compose_file | if type == "string" then fromjson else . end | .allowed_envs | index("DORMANT") and index("PREDECESSOR_URL")' > /dev/null \ + || { echo "$A was not deployed with -e PREDECESSOR_URL= -e DORMANT=" >&2; exit 1; } + +# 1. A replica of A with the same compose. DORMANT keeps it forwarding to A instead of taking over. +env=$(mktemp) +printf 'DORMANT=1\nPREDECESSOR_URL=%s\n' "$a_url" > "$env" +B=$(phala cvms replicate "$A" -e "$env" --json | jq -r .vm_uuid) +echo "replica $B" +until phala cvms get "$B" --json | jq -e '.status == "running"' > /dev/null; do sleep 10; done + +# 2. B alone reboots into the new compose and takes the writer role and state from A. +phala deploy --cvm-id "$B" -c "$NEW" -e PREDECESSOR_URL="$a_url" -e DORMANT= --wait +b_url=https://$(phala cvms get "$B" --json | jq -r '.instance_id + "-8080." + .gateway.base_domain') +until curl -sf "$b_url/_status" | jq -e '.role == "writer"' > /dev/null; do sleep 3; done + +# 3. A now forwards to B. Let requests already routed to A finish, then remove it. +sleep 20 +phala cvms delete "$A" --force +echo "upgraded: $B is the writer"