From fa0aba6d56f6cec6b3ff404cf4814e2c0399a012 Mon Sep 17 00:00:00 2001 From: Andrew Miller Date: Sat, 14 Nov 2026 23:00:11 -0500 Subject: [PATCH 1/3] Add zero-downtime-upgrade example: hand state to a second CVM of the same app --- README.md | 1 + zero-downtime-upgrade/README.md | 95 ++++++++++ zero-downtime-upgrade/client.mjs | 40 +++++ zero-downtime-upgrade/docker-compose.yaml | 200 ++++++++++++++++++++++ zero-downtime-upgrade/test-local.mjs | 118 +++++++++++++ zero-downtime-upgrade/upgrade.sh | 33 ++++ 6 files changed, 487 insertions(+) create mode 100644 zero-downtime-upgrade/README.md create mode 100644 zero-downtime-upgrade/client.mjs create mode 100644 zero-downtime-upgrade/docker-compose.yaml create mode 100644 zero-downtime-upgrade/test-local.mjs create mode 100755 zero-downtime-upgrade/upgrade.sh diff --git a/README.md b/README.md index da7300e..319fa76 100644 --- a/README.md +++ b/README.md @@ -233,6 +233,7 @@ Implementation details and infrastructure patterns. | Example | Description | |---------|-------------| | [launcher](./launcher) | Generic launcher pattern for Docker Compose apps | +| [zero-downtime-upgrade](./zero-downtime-upgrade) | Upgrade a stateful app by handing over to a second CVM | | [prelaunch-script](./prelaunch-script) | Pre-launch script patterns (Phala Cloud) | | [private-docker-image-deployment](./private-docker-image-deployment) | Using private Docker registries | | [attestation/rtmr3-based](./attestation/rtmr3-based) | RTMR3-based attestation (legacy) | diff --git a/zero-downtime-upgrade/README.md b/zero-downtime-upgrade/README.md new file mode 100644 index 0000000..522349e --- /dev/null +++ b/zero-downtime-upgrade/README.md @@ -0,0 +1,95 @@ +# Zero-downtime upgrade + +Updating a CVM's compose reboots it. A stateful server on that CVM is unreachable until the +new image boots, which on Phala Cloud took about 70 seconds for a `tdx.small` running this +example. This example upgrades the server without downtime instead: it starts a second +CVM of the same app on the new compose, hands the state and the writer role to it, and only +then deletes the old one. + +The server is a small key-value store (`docker-compose.yaml`, inline Node, no build step). +The handover code in it is what you would copy into your own server. + +## How it works + +Two CVMs of the same app can both receive traffic through the gateway, so at any moment +exactly one of them is the **writer** and the other forwards every request to it: + +- **writer** serves requests from its own state, persisted on its disk. +- **follower** forwards to its predecessor. At boot it takes over (below). +- **retired** forwards to its successor, until it is deleted. + +The handover is two calls from the new CVM B to the old CVM A: + +1. `begin`: A stops serving writes (requests wait) and returns its state, encrypted. +2. `commit`: A becomes retired and forwards everything to B. B loads the state and becomes + the writer. Requests that waited on A are forwarded to B. + +If A gets no `commit` within 15 seconds it goes back to serving, so a B that crashes +mid-handover cannot strand the app. + +Both calls go over the public gateway. They are authenticated with a key from `GetKey`, and the +state is encrypted under that key. dstack derives the key from the app id, so A and B get +the same key even though they run different composes, and nothing outside an approved +CVM of this app can derive it. + +The upgrade itself is `upgrade.sh`: + +1. `phala cvms replicate` A with `DORMANT=1`. The replica boots on A's compose and only + forwards. Without `DORMANT` it would take over before running the new code. +2. `phala deploy --cvm-id B` with the new compose. Only B reboots. Meanwhile A keeps serving, + and the gateway skips B because its port is closed. +3. B boots on the new compose and takes over from A. +4. After 20 seconds for in-flight requests through A, delete A. + +## Run it + +```bash +phala deploy -n kv -c docker-compose.yaml -t tdx.small --wait \ + -e PREDECESSOR_URL= -e DORMANT= -e GATEWAY_BASE= +node client.mjs https://-8080. # in another terminal + +sed 's/APP_VERSION=1/APP_VERSION=2/' docker-compose.yaml > v2.yaml +./upgrade.sh kv v2.yaml +``` + +`client.mjs` writes and reads a counter in a loop with no retries and a fresh connection +per request, and prints every failure. `/_status` on the app URL shows which version and +role answered. `upgrade.sh` needs `jq`. Each upgrade takes about two minutes, most of it +the replica booting and then rebooting into the new compose. + +The three `-e` flags on the first deploy are required even though they are empty. dstack only +passes env vars named in the CVM's `allowed_envs`, which the CLI sets from the `-e` names, +and a replica inherits its source's list. Without them the replica silently gets no +`DORMANT` and no `PREDECESSOR_URL`, boots as a second writer with empty state, and the +gateway splits traffic between two databases. When we tried that, the client saw 8 failed +reads and 4 version regressions in 20 seconds. `upgrade.sh` now refuses to start unless the +three names are allowed. + +The role is saved on the data volume with the state, and at boot the saved role wins over +the env. A finished upgrade leaves `PREDECESSOR_URL` pointing at the deleted CVM, so +without this a restart of the new writer would make it a follower of nothing. + +`node test-local.mjs` runs the same upgrade on one machine with a fake dstack socket and a +stand-in gateway. + +## Results + +On Phala Cloud (prod7, dstack 0.5.12, `tdx.small`), with `client.mjs` running throughout: + +| Upgrade | Failed requests | Longest request | +|---------|-----------------|-----------------| +| `phala deploy --cvm-id A` (in place) | 13, from 03:51:56 to 03:53:01 | 5.4 s | +| `upgrade.sh` v1 → v2, then `upgrade.sh` v2 → v3 | 0 of 334 | 1.2 s | + +The in-place failures were `socket hang up`. After the two upgrades, `phala cvms restart` of +the final CVM brought it back as the writer with the last value and version. + +## Limits + +- Requests wait during the handover, for the length of the two calls through the gateway. + State is sent in one request, so a large database would hand over a pointer to + encrypted storage instead. +- dstack-gateway can skip non-writers using `requirements.health_check` in the app compose. The + server already writes a health file if `HEALTH_FILE` is set. The Phala CLI cannot set + this field yet, so this example relies on forwarding instead. +- The port is fixed at 8080 in `upgrade.sh`. diff --git a/zero-downtime-upgrade/client.mjs b/zero-downtime-upgrade/client.mjs new file mode 100644 index 0000000..9dc42cd --- /dev/null +++ b/zero-downtime-upgrade/client.mjs @@ -0,0 +1,40 @@ +// Writes then reads a counter through the app URL forever, with no retries and a fresh +// connection per request, and prints every failure. Ctrl-C prints the summary. +// Usage: node client.mjs https://-8080. +import https from 'node:https' + +const base = process.argv[2] +const req = (method, path, body) => new Promise((resolve, reject) => { + const r = https.request(base + path, { method, agent: false, timeout: 10000 }, res => { + let s = '' + res.on('data', d => { s += d }) + res.on('end', () => resolve({ status: res.statusCode, body: s })) + }) + r.on('timeout', () => r.destroy(new Error('timeout'))) + r.on('error', reject) + r.end(body) +}) + +const stats = { ok: 0, fail: 0, regressions: 0, maxMs: 0, lastVersion: 0, firstFail: null, lastFail: null } +process.on('SIGINT', () => { console.log(JSON.stringify(stats)); process.exit(0) }) +for (let i = Date.now(); ; i++) { + const t = Date.now() + try { + const w = await req('PUT', '/kv/counter', String(i)) + const r = await req('GET', '/kv/counter') + if (w.status !== 200 || r.status !== 200) throw new Error(w.status + ' ' + w.body + ' / ' + r.status + ' ' + r.body) + const got = JSON.parse(r.body) + if (got.value !== String(i)) throw new Error('read ' + got.value + ' after writing ' + i) + if (got.version < stats.lastVersion) stats.regressions++ + stats.lastVersion = got.version + if (++stats.ok % 50 === 0) console.log(new Date().toISOString(), JSON.stringify(stats)) + } catch (e) { + stats.fail++ + stats.firstFail ??= new Date(t).toISOString() + stats.lastFail = new Date(t).toISOString() + console.log(new Date(t).toISOString(), 'FAIL', e.message) + } + const ms = Date.now() - t + stats.maxMs = Math.max(stats.maxMs, ms) + await new Promise(r => setTimeout(r, 100)) +} diff --git a/zero-downtime-upgrade/docker-compose.yaml b/zero-downtime-upgrade/docker-compose.yaml new file mode 100644 index 0000000..5b0527a --- /dev/null +++ b/zero-downtime-upgrade/docker-compose.yaml @@ -0,0 +1,200 @@ +services: + kv: + image: node:22-slim@sha256:e9f565d32859f0cbfb52125eafe8be39390f24dac302dc3cdf566e8ba3ab21ee + restart: unless-stopped + command: ["node", "/app/server.mjs"] + ports: + - "8080:8080" + environment: + - APP_VERSION=1 + - PREDECESSOR_URL=${PREDECESSOR_URL:-} + - DORMANT=${DORMANT:-} + - GATEWAY_BASE=${GATEWAY_BASE:-} + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - kv:/data + configs: + - source: server + target: /app/server.mjs + +volumes: + kv: + +configs: + server: + content: | + // A key-value server that can hand its state and its writer role to a successor CVM. + // Roles: writer serves locally; follower forwards to its predecessor until it takes over; + // retired forwards to its successor. Whichever instance the gateway picks, the writer answers. + // DORMANT makes a follower forward without taking over, for a replica that waits for its new compose. + import http from 'node:http' + import fs from 'node:fs' + import crypto from 'node:crypto' + + const env = process.env + const PORT = Number(env.PORT || 8080) + const DATA = env.DATA_FILE || '/data/kv.json' + const SOCK = env.DSTACK_SOCK || '/var/run/dstack.sock' + const HEALTH = env.HEALTH_FILE || '' + const ABORT_MS = 15000 + + let kv = {} + let version = 0 + let role = env.PREDECESSOR_URL ? 'follower' : 'writer' + let target = env.PREDECESSOR_URL || '' + let hold = null + let pending = null + let committed = '' + + // The role on disk wins over env: after an upgrade the env still names a predecessor that is gone. + if (fs.existsSync(DATA)) ({ kv, version, role, target } = JSON.parse(fs.readFileSync(DATA, 'utf8'))) + + function dstack(method, body) { + return new Promise((resolve, reject) => { + const req = http.request({ socketPath: SOCK, path: '/' + method, method: 'POST', + headers: { 'content-type': 'application/json' } }, res => { + let s = '' + res.on('data', d => { s += d }) + res.on('end', () => res.statusCode === 200 ? resolve(JSON.parse(s)) : reject(new Error(method + ' ' + res.statusCode + ' ' + s))) + }) + req.on('error', reject) + req.end(JSON.stringify(body)) + }) + } + + // Same app_id, same key: only CVMs running an allowed compose of this app can derive it. + const key = Buffer.from((await dstack('GetKey', { path: 'zero-downtime/handover' })).key.replace(/^0x/, ''), 'hex').subarray(0, 32) + const mac = s => crypto.createHmac('sha256', key).update(s).digest('hex') + function seal(obj, aad) { + const iv = crypto.randomBytes(12) + const c = crypto.createCipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(aad)) + const ct = Buffer.concat([c.update(JSON.stringify(obj)), c.final()]) + return { iv: iv.toString('hex'), ct: ct.toString('hex'), tag: c.getAuthTag().toString('hex') } + } + function open(box, aad) { + const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(box.iv, 'hex')).setAAD(Buffer.from(aad)) + d.setAuthTag(Buffer.from(box.tag, 'hex')) + return JSON.parse(Buffer.concat([d.update(Buffer.from(box.ct, 'hex')), d.final()]).toString()) + } + async function signedPost(url, obj) { + const body = JSON.stringify(obj) + const r = await fetch(url, { method: 'POST', body, headers: { 'x-mac': mac(body) } }) + if (r.status !== 200) throw Object.assign(new Error(url + ' ' + r.status + ' ' + await r.text()), { status: r.status }) + return r.json() + } + + function save() { + fs.writeFileSync(DATA + '.tmp', JSON.stringify({ kv, version, role, target })) + fs.renameSync(DATA + '.tmp', DATA) + } + function writeHealth() { + if (HEALTH) fs.writeFileSync(HEALTH, (role === 'writer' ? 'healthy' : 'unhealthy') + '\n' + Math.floor(Date.now() / 1000) + '\n') + } + setInterval(writeHealth, 5000) + writeHealth() + + async function selfUrl() { + if (env.SELF_URL) return env.SELF_URL + const info = await dstack('Info', {}) + return 'https://' + info.instance_id + '-' + PORT + '.' + env.GATEWAY_BASE + } + + // Writer side of the handover. begin freezes writes and returns sealed state; commit + // makes this instance forward to the successor; no commit within ABORT_MS resumes writing. + function begin(req) { + if (role !== 'writer' || pending) return [409, { error: 'not the writer, or a handover is running' }] + let release + hold = new Promise(r => { release = r }) + const timer = setTimeout(() => { pending = null; hold = null; release() }, ABORT_MS) + pending = { nonce: req.nonce, timer, release } + return [200, { box: seal({ kv, version }, req.nonce) }] + } + function commit(req) { + if (role === 'retired' && committed === req.nonce) return [200, { ok: true }] + if (!pending || pending.nonce !== req.nonce) return [409, { error: 'no matching handover' }] + committed = req.nonce + clearTimeout(pending.timer) + role = 'retired' + target = req.successor + save() + pending.release() + pending = null + hold = null + writeHealth() + return [200, { ok: true }] + } + + // Successor side: runs once at boot when PREDECESSOR_URL is set, retried until it succeeds. + async function takeOver() { + const me = await selfUrl() + for (;;) { + const nonce = crypto.randomBytes(16).toString('hex') + let release + hold = new Promise(r => { release = r }) + try { + const { box } = await signedPost(target + '/_handover/begin', { nonce }) + const state = open(box, nonce) + for (let i = 0; ; i++) { + try { await signedPost(target + '/_handover/commit', { nonce, successor: me }); break } + catch (e) { if (e.status === 409 || i === 4) throw e; await new Promise(r => setTimeout(r, 1000)) } + } + ;({ kv, version } = state) + role = 'writer' + target = '' + save() + hold = null + release() + writeHealth() + console.log('took over as writer at version', version) + return + } catch (e) { + hold = null + release() + console.log('handover failed, retrying:', e.message) + await new Promise(r => setTimeout(r, 3000)) + } + } + } + + async function forward(req, res, body) { + const hops = Number(req.headers['x-zdt-hops'] || 0) + if (hops > 3) return send(res, 508, { error: 'forwarding loop' }) + const r = await fetch(target + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body, + headers: { 'x-zdt-hops': String(hops + 1) } }) + res.writeHead(r.status, { 'content-type': r.headers.get('content-type') || 'application/json' }) + res.end(Buffer.from(await r.arrayBuffer())) + } + function send(res, status, obj) { + res.writeHead(status, { 'content-type': 'application/json' }) + res.end(JSON.stringify(obj)) + } + + const server = http.createServer(async (req, res) => { + let body = '' + for await (const d of req) body += d + try { + if (req.url === '/_status') return send(res, 200, { app: env.APP_VERSION, role, version, keys: Object.keys(kv).length }) + if (req.url.startsWith('/_handover/')) { + if (req.headers['x-mac'] !== mac(body)) return send(res, 401, { error: 'bad mac' }) + const [status, out] = (req.url === '/_handover/begin' ? begin : commit)(JSON.parse(body)) + return send(res, status, out) + } + if (hold) await hold + if (role !== 'writer') return await forward(req, res, body) + const k = decodeURIComponent(req.url.replace(/^\/kv\//, '')) + if (!req.url.startsWith('/kv/')) return send(res, 404, { error: 'use /kv/' }) + if (req.method === 'PUT') { + kv[k] = body + version++ + save() + return send(res, 200, { version }) + } + if (req.method === 'GET') return k in kv ? send(res, 200, { value: kv[k], version }) : send(res, 404, { version }) + return send(res, 405, { error: 'GET or PUT' }) + } catch (e) { + send(res, 502, { error: e.message }) + } + }).listen(PORT, () => console.log('listening on', PORT, 'as', role)) + process.on('SIGTERM', () => server.close(() => process.exit(0))) + + if (role === 'follower' && !env.DORMANT) takeOver() diff --git a/zero-downtime-upgrade/test-local.mjs b/zero-downtime-upgrade/test-local.mjs new file mode 100644 index 0000000..0c5f89c --- /dev/null +++ b/zero-downtime-upgrade/test-local.mjs @@ -0,0 +1,118 @@ +// Runs the handover with no TEE: a fake dstack socket, two copies of the server from +// docker-compose.yaml, and a round-robin proxy standing in for the gateway. A client loop +// writes and reads through the proxy with no retries for the whole upgrade. +import http from 'node:http' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { spawn } from 'node:child_process' + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'zdt-')) +const compose = fs.readFileSync(new URL('./docker-compose.yaml', import.meta.url), 'utf8').split('\n') +const start = compose.indexOf(' content: |') + 1 +fs.writeFileSync(path.join(dir, 'server.mjs'), compose.slice(start).map(l => l.slice(6)).join('\n')) + +const sock = path.join(dir, 'dstack.sock') +http.createServer((req, res) => { + res.setHeader('content-type', 'application/json') + if (req.url === '/GetKey') return res.end(JSON.stringify({ key: '11'.repeat(32) })) + res.end(JSON.stringify({ instance_id: 'local' })) +}).listen(sock) + +const procs = [] +function run(name, port, extra) { + fs.mkdirSync(path.join(dir, name), { recursive: true }) + const p = spawn('node', [path.join(dir, 'server.mjs')], { env: { ...process.env, PORT: String(port), + DSTACK_SOCK: sock, DATA_FILE: path.join(dir, name, 'kv.json'), ...extra } }) + p.stdout.on('data', d => process.stdout.write(name + ': ' + d)) + p.stderr.on('data', d => process.stdout.write(name + '! ' + d)) + procs.push(p) +} + +// The gateway stand-in rotates over every registered instance, healthy or not, so an instance +// that has not taken over yet still gets traffic. Like dstack-gateway's connect_multiple_hosts, it +// skips an instance whose port refuses the connection (a CVM that is rebooting). +const backends = [8101] +let rr = 0 +http.createServer(async (req, res) => { + let body = '' + for await (const d of req) body += d + const start = rr++ + for (let i = 0; i < backends.length; i++) { + const port = backends[(start + i) % backends.length] + try { + const r = await fetch('http://127.0.0.1:' + port + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body }) + res.writeHead(r.status) + return res.end(Buffer.from(await r.arrayBuffer())) + } catch (e) { + if (e.cause?.code !== 'ECONNREFUSED') { res.writeHead(599); return res.end(e.message) } + } + } + res.writeHead(599) + res.end('no instance accepted the connection') +}).listen(8100) + +const sleep = ms => new Promise(r => setTimeout(r, ms)) +const stats = { ok: 0, fail: 0, maxMs: 0, lastVersion: 0, regressions: 0 } +let running = true +async function client() { + for (let i = 1; running; i++) { + const t = Date.now() + try { + const w = await fetch('http://127.0.0.1:8100/kv/counter', { method: 'PUT', body: String(i) }) + const r = await fetch('http://127.0.0.1:8100/kv/counter') + const got = await r.json() + if (w.status !== 200 || r.status !== 200 || got.value !== String(i)) throw new Error(w.status + ' ' + r.status + ' ' + JSON.stringify(got)) + if (got.version < stats.lastVersion) stats.regressions++ + stats.lastVersion = got.version + stats.ok++ + } catch (e) { + stats.fail++ + console.log('client failure:', e.message) + } + stats.maxMs = Math.max(stats.maxMs, Date.now() - t) + await sleep(20) + } +} + +run('a', 8101, { APP_VERSION: '1' }) +await sleep(500) +const loop = client() +await sleep(2000) +console.log('--- starting B dormant, as a fresh replica boots; the proxy sends half the traffic to it') +const b = { PREDECESSOR_URL: 'http://127.0.0.1:8101', SELF_URL: 'http://127.0.0.1:8102' } +run('b-dormant', 8102, { ...b, DORMANT: '1' }) +await sleep(300) +backends.push(8102) +await sleep(1500) +const dormant = await (await fetch('http://127.0.0.1:8102/_status')).json() +console.log('dormant B', dormant) +console.log('--- restarting B with the new compose; the proxy keeps it in rotation while it is down') +procs[1].kill() +await sleep(300) +run('b', 8102, { ...b, APP_VERSION: '2' }) +await sleep(3000) +const sa = await (await fetch('http://127.0.0.1:8101/_status')).json() +const sb = await (await fetch('http://127.0.0.1:8102/_status')).json() +console.log('A', sa, 'B', sb) +console.log('--- removing A from the proxy, then stopping it') +backends.shift() +await sleep(1000) +procs[0].kill() +await sleep(2000) +running = false +await loop +console.log('client', stats) +console.log('--- restarting B, whose env still names the stopped A as its predecessor') +procs[2].kill() +await sleep(300) +run('b', 8102, { ...b, APP_VERSION: '2' }) +await sleep(500) +const restarted = await (await fetch('http://127.0.0.1:8102/_status')).json() +const value = await (await fetch('http://127.0.0.1:8100/kv/counter')).json() +console.log('restarted B', restarted, value) +procs.forEach(p => p.kill()) +const pass = stats.fail === 0 && stats.regressions === 0 && dormant.role === 'follower' && sa.role === 'retired' && sb.role === 'writer' && sb.app === '2' && stats.ok > 50 + && restarted.role === 'writer' && value.version === stats.lastVersion +console.log(pass ? 'PASS' : 'FAIL') +process.exit(pass ? 0 : 1) diff --git a/zero-downtime-upgrade/upgrade.sh b/zero-downtime-upgrade/upgrade.sh new file mode 100755 index 0000000..ba32982 --- /dev/null +++ b/zero-downtime-upgrade/upgrade.sh @@ -0,0 +1,33 @@ +#!/bin/sh +# Upgrades a running CVM to a new compose with no downtime. +# Usage: ./upgrade.sh +set -eu +A=$1 +NEW=$2 + +info=$(phala cvms get "$A" --json) +base=$(printf "%s\n" "$info" | jq -r .gateway.base_domain) +a_url=https://$(printf "%s\n" "$info" | jq -r .instance_id)-8080.$base + +# dstack drops env vars missing from allowed_envs, and the replica inherits A's list. A replica +# without DORMANT and PREDECESSOR_URL would boot as a second writer with empty state. +printf "%s\n" "$info" | jq -e '.compose_file | if type == "string" then fromjson else . end | .allowed_envs + | index("DORMANT") and index("PREDECESSOR_URL") and index("GATEWAY_BASE")' > /dev/null \ + || { echo "$A was not deployed with -e PREDECESSOR_URL= -e DORMANT= -e GATEWAY_BASE=" >&2; exit 1; } + +# 1. A replica of A with the same compose. DORMANT keeps it forwarding to A instead of taking over. +env=$(mktemp) +printf 'DORMANT=1\nPREDECESSOR_URL=%s\nGATEWAY_BASE=%s\n' "$a_url" "$base" > "$env" +B=$(phala cvms replicate "$A" -e "$env" --json | jq -r .vm_uuid) +echo "replica $B" +until phala cvms get "$B" --json | jq -e '.status == "running"' > /dev/null; do sleep 10; done + +# 2. B alone reboots into the new compose and takes the writer role and state from A. +phala deploy --cvm-id "$B" -c "$NEW" -e PREDECESSOR_URL="$a_url" -e DORMANT= -e GATEWAY_BASE="$base" --wait +b_url=https://$(phala cvms get "$B" --json | jq -r .instance_id)-8080.$base +until curl -sf "$b_url/_status" | jq -e '.role == "writer"' > /dev/null; do sleep 3; done + +# 3. A now forwards to B. Let requests already routed to A finish, then remove it. +sleep 20 +phala cvms delete "$A" --force +echo "upgraded: $B is the writer" From 23486260cabee63a38424e60b6fa848d473e6238 Mon Sep 17 00:00:00 2001 From: Andrew Miller Date: Sat, 14 Nov 2026 23:14:57 -0500 Subject: [PATCH 2/3] Name the OS image the results were measured on --- zero-downtime-upgrade/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/zero-downtime-upgrade/README.md b/zero-downtime-upgrade/README.md index 522349e..438469d 100644 --- a/zero-downtime-upgrade/README.md +++ b/zero-downtime-upgrade/README.md @@ -74,7 +74,7 @@ stand-in gateway. ## Results -On Phala Cloud (prod7, dstack 0.5.12, `tdx.small`), with `client.mjs` running throughout: +On Phala Cloud (prod7, `tdx.small`, the `dstack-dev-0.5.12` image Phala picked by default), with `client.mjs` running throughout: | Upgrade | Failed requests | Longest request | |---------|-----------------|-----------------| From 37a262e099b6b99ab204484649124c408ba2b98c Mon Sep 17 00:00:00 2001 From: Andrew Miller Date: Sat, 14 Nov 2026 23:49:42 -0500 Subject: [PATCH 3/3] Simplify the zero-downtime example; bump APP_VERSION on code changes --- zero-downtime-upgrade/README.md | 106 +++++------- zero-downtime-upgrade/client.mjs | 15 +- zero-downtime-upgrade/docker-compose.yaml | 187 ++++++++-------------- zero-downtime-upgrade/test-local.mjs | 113 +++++-------- zero-downtime-upgrade/upgrade.sh | 14 +- 5 files changed, 167 insertions(+), 268 deletions(-) diff --git a/zero-downtime-upgrade/README.md b/zero-downtime-upgrade/README.md index 438469d..9941240 100644 --- a/zero-downtime-upgrade/README.md +++ b/zero-downtime-upgrade/README.md @@ -1,95 +1,71 @@ # Zero-downtime upgrade -Updating a CVM's compose reboots it. A stateful server on that CVM is unreachable until the -new image boots, which on Phala Cloud took about 70 seconds for a `tdx.small` running this -example. This example upgrades the server without downtime instead: it starts a second -CVM of the same app on the new compose, hands the state and the writer role to it, and only -then deletes the old one. +Updating a CVM's compose reboots it, and a stateful server is down until it boots again +(about 65 seconds for this example on Phala Cloud). This example upgrades without downtime: +it starts a second CVM of the same app on the new compose, hands the state and the writer +role to it, then deletes the old one. -The server is a small key-value store (`docker-compose.yaml`, inline Node, no build step). -The handover code in it is what you would copy into your own server. +The server is a small key-value store, inline Node in `docker-compose.yaml`. Its handover +code is what you would copy into your own server. ## How it works -Two CVMs of the same app can both receive traffic through the gateway, so at any moment -exactly one of them is the **writer** and the other forwards every request to it: +Both CVMs get traffic from the gateway, so exactly one is the **writer** and the other +forwards to it. A **follower** forwards to its predecessor and takes over at boot; a +**retired** CVM forwards to its successor. -- **writer** serves requests from its own state, persisted on its disk. -- **follower** forwards to its predecessor. At boot it takes over (below). -- **retired** forwards to its successor, until it is deleted. +The handover is two calls from the new CVM B to the old CVM A. On `begin`, A holds requests +and returns its state. On `commit`, A retires and forwards to B, and B starts serving. With +no `commit` within 15 seconds, A resumes. -The handover is two calls from the new CVM B to the old CVM A: +The calls are authenticated, and the state encrypted, with a key from `GetKey`. dstack derives +it from the app id, so A and B get the same key on different composes, and only a compose +the app allows can derive it. -1. `begin`: A stops serving writes (requests wait) and returns its state, encrypted. -2. `commit`: A becomes retired and forwards everything to B. B loads the state and becomes - the writer. Requests that waited on A are forwarded to B. +`upgrade.sh` runs the upgrade: -If A gets no `commit` within 15 seconds it goes back to serving, so a B that crashes -mid-handover cannot strand the app. - -Both calls go over the public gateway. They are authenticated with a key from `GetKey`, and the -state is encrypted under that key. dstack derives the key from the app id, so A and B get -the same key even though they run different composes, and nothing outside an approved -CVM of this app can derive it. - -The upgrade itself is `upgrade.sh`: - -1. `phala cvms replicate` A with `DORMANT=1`. The replica boots on A's compose and only - forwards. Without `DORMANT` it would take over before running the new code. -2. `phala deploy --cvm-id B` with the new compose. Only B reboots. Meanwhile A keeps serving, - and the gateway skips B because its port is closed. -3. B boots on the new compose and takes over from A. -4. After 20 seconds for in-flight requests through A, delete A. +1. `phala cvms replicate` A with `DORMANT=1`, so the replica only forwards. +2. `phala deploy --cvm-id B` with the new compose. Only B reboots; the gateway skips it while + its port is closed. +3. B boots on the new compose and takes over. +4. After 20 seconds for requests in flight through A, delete A. ## Run it ```bash -phala deploy -n kv -c docker-compose.yaml -t tdx.small --wait \ - -e PREDECESSOR_URL= -e DORMANT= -e GATEWAY_BASE= +phala deploy -n kv -c docker-compose.yaml -t tdx.small --wait -e PREDECESSOR_URL= -e DORMANT= node client.mjs https://-8080. # in another terminal - sed 's/APP_VERSION=1/APP_VERSION=2/' docker-compose.yaml > v2.yaml ./upgrade.sh kv v2.yaml ``` -`client.mjs` writes and reads a counter in a loop with no retries and a fresh connection -per request, and prints every failure. `/_status` on the app URL shows which version and -role answered. `upgrade.sh` needs `jq`. Each upgrade takes about two minutes, most of it -the replica booting and then rebooting into the new compose. +`client.mjs` writes and reads a counter with no retries and prints every failure. +`upgrade.sh` needs `jq` and takes about two minutes. `node test-local.mjs` runs the same +upgrade on one machine. -The three `-e` flags on the first deploy are required even though they are empty. dstack only -passes env vars named in the CVM's `allowed_envs`, which the CLI sets from the `-e` names, -and a replica inherits its source's list. Without them the replica silently gets no -`DORMANT` and no `PREDECESSOR_URL`, boots as a second writer with empty state, and the -gateway splits traffic between two databases. When we tried that, the client saw 8 failed -reads and 4 version regressions in 20 seconds. `upgrade.sh` now refuses to start unless the -three names are allowed. +The empty `-e` flags matter. dstack passes only env vars named in `allowed_envs`, which a +replica inherits. Without them the replica boots as a second writer with empty state; we saw +8 wrong reads in 20 seconds. `upgrade.sh` checks for them. -The role is saved on the data volume with the state, and at boot the saved role wins over -the env. A finished upgrade leaves `PREDECESSOR_URL` pointing at the deleted CVM, so -without this a restart of the new writer would make it a follower of nothing. +Change `APP_VERSION` with every change to the server code. Docker Compose does not recreate a +container when only an inline config's content changes, and the container survives the +reboot, so without a changed env var the CVM keeps running the old code. -`node test-local.mjs` runs the same upgrade on one machine with a fake dstack socket and a -stand-in gateway. +The role is saved with the state and wins over the env at boot, because after an upgrade +`PREDECESSOR_URL` still names the deleted CVM. ## Results -On Phala Cloud (prod7, `tdx.small`, the `dstack-dev-0.5.12` image Phala picked by default), with `client.mjs` running throughout: +Phala Cloud prod7, `tdx.small`, `dstack-dev-0.5.12`, with `client.mjs` running throughout: | Upgrade | Failed requests | Longest request | |---------|-----------------|-----------------| -| `phala deploy --cvm-id A` (in place) | 13, from 03:51:56 to 03:53:01 | 5.4 s | -| `upgrade.sh` v1 → v2, then `upgrade.sh` v2 → v3 | 0 of 334 | 1.2 s | - -The in-place failures were `socket hang up`. After the two upgrades, `phala cvms restart` of -the final CVM brought it back as the writer with the last value and version. +| `phala deploy --cvm-id A` (in place) | 13, over 65 s | 5.4 s | +| `upgrade.sh` v1 → v2 → v3 | 0 of 334 | 1.2 s | +| `upgrade.sh` v1.1 → v2, this version of the server | 0 of 169 | 3.8 s | -## Limits +While A is retired, a request through A costs about 10 ms more than one sent to B directly +(median of 12 each, after the TLS handshake). After A is deleted there is no forwarding. -- Requests wait during the handover, for the length of the two calls through the gateway. - State is sent in one request, so a large database would hand over a pointer to - encrypted storage instead. -- dstack-gateway can skip non-writers using `requirements.health_check` in the app compose. The - server already writes a health file if `HEALTH_FILE` is set. The Phala CLI cannot set - this field yet, so this example relies on forwarding instead. -- The port is fixed at 8080 in `upgrade.sh`. +State goes in one request, so a large database would hand over a pointer to encrypted +storage instead. diff --git a/zero-downtime-upgrade/client.mjs b/zero-downtime-upgrade/client.mjs index 9dc42cd..c5767d9 100644 --- a/zero-downtime-upgrade/client.mjs +++ b/zero-downtime-upgrade/client.mjs @@ -1,11 +1,12 @@ -// Writes then reads a counter through the app URL forever, with no retries and a fresh +// Writes then reads a counter through the app URL in a loop, with no retries and a fresh // connection per request, and prints every failure. Ctrl-C prints the summary. // Usage: node client.mjs https://-8080. +import http from 'node:http' import https from 'node:https' const base = process.argv[2] const req = (method, path, body) => new Promise((resolve, reject) => { - const r = https.request(base + path, { method, agent: false, timeout: 10000 }, res => { + const r = (base.startsWith('https') ? https : http).request(base + path, { method, agent: false, timeout: 10000 }, res => { let s = '' res.on('data', d => { s += d }) res.on('end', () => resolve({ status: res.statusCode, body: s })) @@ -15,26 +16,22 @@ const req = (method, path, body) => new Promise((resolve, reject) => { r.end(body) }) -const stats = { ok: 0, fail: 0, regressions: 0, maxMs: 0, lastVersion: 0, firstFail: null, lastFail: null } +const stats = { ok: 0, fail: 0, regressions: 0, maxMs: 0, lastVersion: 0 } process.on('SIGINT', () => { console.log(JSON.stringify(stats)); process.exit(0) }) for (let i = Date.now(); ; i++) { const t = Date.now() try { const w = await req('PUT', '/kv/counter', String(i)) const r = await req('GET', '/kv/counter') - if (w.status !== 200 || r.status !== 200) throw new Error(w.status + ' ' + w.body + ' / ' + r.status + ' ' + r.body) const got = JSON.parse(r.body) - if (got.value !== String(i)) throw new Error('read ' + got.value + ' after writing ' + i) + if (w.status !== 200 || got.value !== String(i)) throw new Error(w.status + ' ' + w.body + ' / ' + r.status + ' ' + r.body) if (got.version < stats.lastVersion) stats.regressions++ stats.lastVersion = got.version if (++stats.ok % 50 === 0) console.log(new Date().toISOString(), JSON.stringify(stats)) } catch (e) { stats.fail++ - stats.firstFail ??= new Date(t).toISOString() - stats.lastFail = new Date(t).toISOString() console.log(new Date(t).toISOString(), 'FAIL', e.message) } - const ms = Date.now() - t - stats.maxMs = Math.max(stats.maxMs, ms) + stats.maxMs = Math.max(stats.maxMs, Date.now() - t) await new Promise(r => setTimeout(r, 100)) } diff --git a/zero-downtime-upgrade/docker-compose.yaml b/zero-downtime-upgrade/docker-compose.yaml index 5b0527a..3908f2f 100644 --- a/zero-downtime-upgrade/docker-compose.yaml +++ b/zero-downtime-upgrade/docker-compose.yaml @@ -6,10 +6,10 @@ services: ports: - "8080:8080" environment: + # Change with every code change: compose doesn't recreate the container for config content alone. - APP_VERSION=1 - PREDECESSOR_URL=${PREDECESSOR_URL:-} - DORMANT=${DORMANT:-} - - GATEWAY_BASE=${GATEWAY_BASE:-} volumes: - /var/run/dstack.sock:/var/run/dstack.sock - kv:/data @@ -23,178 +23,133 @@ volumes: configs: server: content: | - // A key-value server that can hand its state and its writer role to a successor CVM. - // Roles: writer serves locally; follower forwards to its predecessor until it takes over; - // retired forwards to its successor. Whichever instance the gateway picks, the writer answers. - // DORMANT makes a follower forward without taking over, for a replica that waits for its new compose. + // A key-value server that hands its state and writer role to a successor CVM of the same app. + // writer serves from disk; follower forwards to PREDECESSOR_URL and takes over at boot (unless + // DORMANT); retired forwards to its successor. Whichever instance the gateway picks, the writer answers. import http from 'node:http' import fs from 'node:fs' import crypto from 'node:crypto' - const env = process.env - const PORT = Number(env.PORT || 8080) - const DATA = env.DATA_FILE || '/data/kv.json' - const SOCK = env.DSTACK_SOCK || '/var/run/dstack.sock' - const HEALTH = env.HEALTH_FILE || '' - const ABORT_MS = 15000 - - let kv = {} - let version = 0 - let role = env.PREDECESSOR_URL ? 'follower' : 'writer' - let target = env.PREDECESSOR_URL || '' - let hold = null - let pending = null - let committed = '' - + const { PORT = 8080, DATA_FILE = '/data/kv.json', DSTACK_SOCK = '/var/run/dstack.sock', + PREDECESSOR_URL = '', DORMANT, SELF_URL, APP_VERSION } = process.env + let s = { kv: {}, version: 0, role: PREDECESSOR_URL ? 'follower' : 'writer', target: PREDECESSOR_URL } // The role on disk wins over env: after an upgrade the env still names a predecessor that is gone. - if (fs.existsSync(DATA)) ({ kv, version, role, target } = JSON.parse(fs.readFileSync(DATA, 'utf8'))) + if (fs.existsSync(DATA_FILE)) s = JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')) + let hold = null, pending = null, committed = '' + const save = () => { fs.writeFileSync(DATA_FILE + '.tmp', JSON.stringify(s)); fs.renameSync(DATA_FILE + '.tmp', DATA_FILE) } + const sleep = ms => new Promise(r => setTimeout(r, ms)) - function dstack(method, body) { - return new Promise((resolve, reject) => { - const req = http.request({ socketPath: SOCK, path: '/' + method, method: 'POST', - headers: { 'content-type': 'application/json' } }, res => { - let s = '' - res.on('data', d => { s += d }) - res.on('end', () => res.statusCode === 200 ? resolve(JSON.parse(s)) : reject(new Error(method + ' ' + res.statusCode + ' ' + s))) - }) - req.on('error', reject) - req.end(JSON.stringify(body)) + const dstack = (method, body) => new Promise((resolve, reject) => { + const req = http.request({ socketPath: DSTACK_SOCK, path: '/' + method, method: 'POST', + headers: { 'content-type': 'application/json' } }, res => { + let out = '' + res.on('data', d => { out += d }) + res.on('end', () => res.statusCode === 200 ? resolve(JSON.parse(out)) : reject(new Error(method + ' ' + res.statusCode + ' ' + out))) }) - } + req.on('error', reject) + req.end(JSON.stringify(body)) + }) - // Same app_id, same key: only CVMs running an allowed compose of this app can derive it. + // Same app_id, same key: only CVMs running a compose hash this app allows can derive it. const key = Buffer.from((await dstack('GetKey', { path: 'zero-downtime/handover' })).key.replace(/^0x/, ''), 'hex').subarray(0, 32) - const mac = s => crypto.createHmac('sha256', key).update(s).digest('hex') - function seal(obj, aad) { - const iv = crypto.randomBytes(12) - const c = crypto.createCipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(aad)) + const mac = body => crypto.createHmac('sha256', key).update(body).digest('hex') + function seal(obj, nonce) { + const iv = crypto.randomBytes(12), c = crypto.createCipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(nonce)) const ct = Buffer.concat([c.update(JSON.stringify(obj)), c.final()]) return { iv: iv.toString('hex'), ct: ct.toString('hex'), tag: c.getAuthTag().toString('hex') } } - function open(box, aad) { - const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(box.iv, 'hex')).setAAD(Buffer.from(aad)) + function open(box, nonce) { + const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(box.iv, 'hex')).setAAD(Buffer.from(nonce)) d.setAuthTag(Buffer.from(box.tag, 'hex')) return JSON.parse(Buffer.concat([d.update(Buffer.from(box.ct, 'hex')), d.final()]).toString()) } - async function signedPost(url, obj) { + async function post(url, obj) { const body = JSON.stringify(obj) const r = await fetch(url, { method: 'POST', body, headers: { 'x-mac': mac(body) } }) - if (r.status !== 200) throw Object.assign(new Error(url + ' ' + r.status + ' ' + await r.text()), { status: r.status }) + if (r.status !== 200) throw Object.assign(new Error(url + ' ' + r.status), { status: r.status }) return r.json() } - function save() { - fs.writeFileSync(DATA + '.tmp', JSON.stringify({ kv, version, role, target })) - fs.renameSync(DATA + '.tmp', DATA) - } - function writeHealth() { - if (HEALTH) fs.writeFileSync(HEALTH, (role === 'writer' ? 'healthy' : 'unhealthy') + '\n' + Math.floor(Date.now() / 1000) + '\n') - } - setInterval(writeHealth, 5000) - writeHealth() - - async function selfUrl() { - if (env.SELF_URL) return env.SELF_URL - const info = await dstack('Info', {}) - return 'https://' + info.instance_id + '-' + PORT + '.' + env.GATEWAY_BASE - } - - // Writer side of the handover. begin freezes writes and returns sealed state; commit - // makes this instance forward to the successor; no commit within ABORT_MS resumes writing. - function begin(req) { - if (role !== 'writer' || pending) return [409, { error: 'not the writer, or a handover is running' }] + // Writer side. begin holds requests and returns the sealed state; commit retires this instance. + // Without a commit within 15 s it goes back to serving. + function begin({ nonce }) { + if (s.role !== 'writer' || pending) return [409, {}] let release hold = new Promise(r => { release = r }) - const timer = setTimeout(() => { pending = null; hold = null; release() }, ABORT_MS) - pending = { nonce: req.nonce, timer, release } - return [200, { box: seal({ kv, version }, req.nonce) }] + pending = { nonce, release, timer: setTimeout(() => { pending = hold = null; release() }, 15000) } + return [200, seal({ kv: s.kv, version: s.version }, nonce)] } - function commit(req) { - if (role === 'retired' && committed === req.nonce) return [200, { ok: true }] - if (!pending || pending.nonce !== req.nonce) return [409, { error: 'no matching handover' }] - committed = req.nonce + function commit({ nonce, successor }) { + if (committed === nonce) return [200, {}] + if (pending?.nonce !== nonce) return [409, {}] clearTimeout(pending.timer) - role = 'retired' - target = req.successor + committed = nonce + s.role = 'retired' + s.target = successor save() pending.release() - pending = null - hold = null - writeHealth() - return [200, { ok: true }] + pending = hold = null + return [200, {}] } - // Successor side: runs once at boot when PREDECESSOR_URL is set, retried until it succeeds. + // Successor side: hold our own requests, take the state, commit, then serve. async function takeOver() { - const me = await selfUrl() + const { instance_id } = await dstack('Info', {}) + const me = SELF_URL || PREDECESSOR_URL.replace(/\/\/[0-9a-f]+-/, '//' + instance_id + '-') for (;;) { const nonce = crypto.randomBytes(16).toString('hex') let release hold = new Promise(r => { release = r }) try { - const { box } = await signedPost(target + '/_handover/begin', { nonce }) - const state = open(box, nonce) + const got = open(await post(s.target + '/_handover/begin', { nonce }), nonce) for (let i = 0; ; i++) { - try { await signedPost(target + '/_handover/commit', { nonce, successor: me }); break } - catch (e) { if (e.status === 409 || i === 4) throw e; await new Promise(r => setTimeout(r, 1000)) } + try { await post(s.target + '/_handover/commit', { nonce, successor: me }); break } + catch (e) { if (e.status === 409 || i === 4) throw e; await sleep(1000) } } - ;({ kv, version } = state) - role = 'writer' - target = '' + s = { ...got, role: 'writer', target: '' } save() - hold = null - release() - writeHealth() - console.log('took over as writer at version', version) - return + return console.log('took over at version', s.version) } catch (e) { + console.log('handover failed, retrying:', e.message) + } finally { hold = null release() - console.log('handover failed, retrying:', e.message) - await new Promise(r => setTimeout(r, 3000)) } + await sleep(3000) } } - async function forward(req, res, body) { - const hops = Number(req.headers['x-zdt-hops'] || 0) - if (hops > 3) return send(res, 508, { error: 'forwarding loop' }) - const r = await fetch(target + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body, - headers: { 'x-zdt-hops': String(hops + 1) } }) - res.writeHead(r.status, { 'content-type': r.headers.get('content-type') || 'application/json' }) - res.end(Buffer.from(await r.arrayBuffer())) - } - function send(res, status, obj) { - res.writeHead(status, { 'content-type': 'application/json' }) - res.end(JSON.stringify(obj)) - } - + const send = (res, status, obj) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(obj)) } const server = http.createServer(async (req, res) => { let body = '' for await (const d of req) body += d try { - if (req.url === '/_status') return send(res, 200, { app: env.APP_VERSION, role, version, keys: Object.keys(kv).length }) + if (req.url === '/_status') return send(res, 200, { app: APP_VERSION, role: s.role, version: s.version }) if (req.url.startsWith('/_handover/')) { - if (req.headers['x-mac'] !== mac(body)) return send(res, 401, { error: 'bad mac' }) - const [status, out] = (req.url === '/_handover/begin' ? begin : commit)(JSON.parse(body)) - return send(res, status, out) + if (req.headers['x-mac'] !== mac(body)) return send(res, 401, {}) + return send(res, ...(req.url === '/_handover/begin' ? begin : commit)(JSON.parse(body))) } if (hold) await hold - if (role !== 'writer') return await forward(req, res, body) - const k = decodeURIComponent(req.url.replace(/^\/kv\//, '')) + if (s.role !== 'writer') { + const hops = Number(req.headers['x-zdt-hops'] || 0) + if (hops > 3) return send(res, 508, { error: 'forwarding loop' }) + const r = await fetch(s.target + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body, headers: { 'x-zdt-hops': String(hops + 1) } }) + res.writeHead(r.status, { 'content-type': 'application/json' }) + return res.end(Buffer.from(await r.arrayBuffer())) + } if (!req.url.startsWith('/kv/')) return send(res, 404, { error: 'use /kv/' }) + const k = decodeURIComponent(req.url.slice(4)) if (req.method === 'PUT') { - kv[k] = body - version++ + s.kv[k] = body + s.version++ save() - return send(res, 200, { version }) + return send(res, 200, { version: s.version }) } - if (req.method === 'GET') return k in kv ? send(res, 200, { value: kv[k], version }) : send(res, 404, { version }) - return send(res, 405, { error: 'GET or PUT' }) + send(res, k in s.kv ? 200 : 404, { value: s.kv[k], version: s.version }) } catch (e) { send(res, 502, { error: e.message }) } - }).listen(PORT, () => console.log('listening on', PORT, 'as', role)) + }).listen(PORT, () => console.log('listening on', PORT, 'as', s.role)) process.on('SIGTERM', () => server.close(() => process.exit(0))) - if (role === 'follower' && !env.DORMANT) takeOver() + if (s.role === 'follower' && !DORMANT) takeOver() diff --git a/zero-downtime-upgrade/test-local.mjs b/zero-downtime-upgrade/test-local.mjs index 0c5f89c..7edafbd 100644 --- a/zero-downtime-upgrade/test-local.mjs +++ b/zero-downtime-upgrade/test-local.mjs @@ -1,6 +1,5 @@ -// Runs the handover with no TEE: a fake dstack socket, two copies of the server from -// docker-compose.yaml, and a round-robin proxy standing in for the gateway. A client loop -// writes and reads through the proxy with no retries for the whole upgrade. +// The upgrade with no TEE: a fake dstack socket, the server from docker-compose.yaml, a stand-in +// gateway, and client.mjs running throughout. Ends by restarting the new writer. import http from 'node:http' import fs from 'node:fs' import os from 'node:os' @@ -9,39 +8,34 @@ import { spawn } from 'node:child_process' const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'zdt-')) const compose = fs.readFileSync(new URL('./docker-compose.yaml', import.meta.url), 'utf8').split('\n') -const start = compose.indexOf(' content: |') + 1 -fs.writeFileSync(path.join(dir, 'server.mjs'), compose.slice(start).map(l => l.slice(6)).join('\n')) - +const server = path.join(dir, 'server.mjs') +fs.writeFileSync(server, compose.slice(compose.indexOf(' content: |') + 1).map(l => l.slice(6)).join('\n')) const sock = path.join(dir, 'dstack.sock') -http.createServer((req, res) => { - res.setHeader('content-type', 'application/json') - if (req.url === '/GetKey') return res.end(JSON.stringify({ key: '11'.repeat(32) })) - res.end(JSON.stringify({ instance_id: 'local' })) -}).listen(sock) +// The real guest agent decodes a body without this content type as protobuf and fails. +http.createServer((req, res) => req.headers['content-type'] !== 'application/json' ? res.writeHead(400).end() + : res.end(JSON.stringify(req.url === '/GetKey' ? { key: '11'.repeat(32) } : { instance_id: 'local' }))).listen(sock) -const procs = [] -function run(name, port, extra) { +const sleep = ms => new Promise(r => setTimeout(r, ms)) +const status = port => fetch('http://127.0.0.1:' + port + '/_status').then(r => r.json()) +const procs = {} +function run(name, port, env) { fs.mkdirSync(path.join(dir, name), { recursive: true }) - const p = spawn('node', [path.join(dir, 'server.mjs')], { env: { ...process.env, PORT: String(port), - DSTACK_SOCK: sock, DATA_FILE: path.join(dir, name, 'kv.json'), ...extra } }) - p.stdout.on('data', d => process.stdout.write(name + ': ' + d)) - p.stderr.on('data', d => process.stdout.write(name + '! ' + d)) - procs.push(p) + procs[name] = spawn('node', [server], { env: { ...process.env, PORT: String(port), DSTACK_SOCK: sock, DATA_FILE: path.join(dir, name, 'kv.json'), ...env } }) + procs[name].stdout.on('data', d => process.stdout.write(name + ': ' + d)) + procs[name].stderr.on('data', d => process.stdout.write(name + '! ' + d)) } -// The gateway stand-in rotates over every registered instance, healthy or not, so an instance -// that has not taken over yet still gets traffic. Like dstack-gateway's connect_multiple_hosts, it -// skips an instance whose port refuses the connection (a CVM that is rebooting). +// Like dstack-gateway, the stand-in rotates over every instance of the app, writer or not, and +// skips one whose port refuses the connection (a CVM that is rebooting). const backends = [8101] let rr = 0 http.createServer(async (req, res) => { let body = '' for await (const d of req) body += d - const start = rr++ + const first = rr++ for (let i = 0; i < backends.length; i++) { - const port = backends[(start + i) % backends.length] try { - const r = await fetch('http://127.0.0.1:' + port + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body }) + const r = await fetch('http://127.0.0.1:' + backends.at((first + i) % backends.length) + req.url, { method: req.method, body: req.method === 'GET' ? undefined : body }) res.writeHead(r.status) return res.end(Buffer.from(await r.arrayBuffer())) } catch (e) { @@ -52,67 +46,46 @@ http.createServer(async (req, res) => { res.end('no instance accepted the connection') }).listen(8100) -const sleep = ms => new Promise(r => setTimeout(r, ms)) -const stats = { ok: 0, fail: 0, maxMs: 0, lastVersion: 0, regressions: 0 } -let running = true -async function client() { - for (let i = 1; running; i++) { - const t = Date.now() - try { - const w = await fetch('http://127.0.0.1:8100/kv/counter', { method: 'PUT', body: String(i) }) - const r = await fetch('http://127.0.0.1:8100/kv/counter') - const got = await r.json() - if (w.status !== 200 || r.status !== 200 || got.value !== String(i)) throw new Error(w.status + ' ' + r.status + ' ' + JSON.stringify(got)) - if (got.version < stats.lastVersion) stats.regressions++ - stats.lastVersion = got.version - stats.ok++ - } catch (e) { - stats.fail++ - console.log('client failure:', e.message) - } - stats.maxMs = Math.max(stats.maxMs, Date.now() - t) - await sleep(20) - } -} - +const b = { PREDECESSOR_URL: 'http://127.0.0.1:8101', SELF_URL: 'http://127.0.0.1:8102' } run('a', 8101, { APP_VERSION: '1' }) await sleep(500) -const loop = client() +const client = spawn('node', [new URL('./client.mjs', import.meta.url).pathname, 'http://127.0.0.1:8100']) +let out = '' +client.stdout.on('data', d => { out += d; process.stdout.write('client: ' + d) }) await sleep(2000) -console.log('--- starting B dormant, as a fresh replica boots; the proxy sends half the traffic to it') -const b = { PREDECESSOR_URL: 'http://127.0.0.1:8101', SELF_URL: 'http://127.0.0.1:8102' } -run('b-dormant', 8102, { ...b, DORMANT: '1' }) + +console.log('--- B boots dormant, like a fresh replica, and the gateway starts sending it traffic') +run('b', 8102, { ...b, DORMANT: '1' }) await sleep(300) backends.push(8102) await sleep(1500) -const dormant = await (await fetch('http://127.0.0.1:8102/_status')).json() -console.log('dormant B', dormant) -console.log('--- restarting B with the new compose; the proxy keeps it in rotation while it is down') -procs[1].kill() +const dormant = await status(8102) +console.log('--- B restarts on the new compose and takes over') +procs.b.kill() await sleep(300) run('b', 8102, { ...b, APP_VERSION: '2' }) await sleep(3000) -const sa = await (await fetch('http://127.0.0.1:8101/_status')).json() -const sb = await (await fetch('http://127.0.0.1:8102/_status')).json() +const sa = await status(8101), sb = await status(8102) console.log('A', sa, 'B', sb) -console.log('--- removing A from the proxy, then stopping it') +console.log('--- A is removed') backends.shift() await sleep(1000) -procs[0].kill() +procs.a.kill() await sleep(2000) -running = false -await loop -console.log('client', stats) -console.log('--- restarting B, whose env still names the stopped A as its predecessor') -procs[2].kill() +client.kill('SIGINT') +await sleep(300) +const stats = JSON.parse(out.trim().split('\n').at(-1)) + +console.log('--- B restarts; its env still names the stopped A as its predecessor') +procs.b.kill() await sleep(300) run('b', 8102, { ...b, APP_VERSION: '2' }) await sleep(500) -const restarted = await (await fetch('http://127.0.0.1:8102/_status')).json() -const value = await (await fetch('http://127.0.0.1:8100/kv/counter')).json() -console.log('restarted B', restarted, value) -procs.forEach(p => p.kill()) -const pass = stats.fail === 0 && stats.regressions === 0 && dormant.role === 'follower' && sa.role === 'retired' && sb.role === 'writer' && sb.app === '2' && stats.ok > 50 - && restarted.role === 'writer' && value.version === stats.lastVersion +const restarted = await status(8102) +console.log('restarted B', restarted) +Object.values(procs).forEach(p => p.kill()) + +const pass = stats.ok > 30 && stats.fail === 0 && stats.regressions === 0 && dormant.role === 'follower' && sa.role === 'retired' + && sb.role === 'writer' && sb.app === '2' && restarted.role === 'writer' && restarted.version === stats.lastVersion console.log(pass ? 'PASS' : 'FAIL') process.exit(pass ? 0 : 1) diff --git a/zero-downtime-upgrade/upgrade.sh b/zero-downtime-upgrade/upgrade.sh index ba32982..dc573fb 100755 --- a/zero-downtime-upgrade/upgrade.sh +++ b/zero-downtime-upgrade/upgrade.sh @@ -6,25 +6,23 @@ A=$1 NEW=$2 info=$(phala cvms get "$A" --json) -base=$(printf "%s\n" "$info" | jq -r .gateway.base_domain) -a_url=https://$(printf "%s\n" "$info" | jq -r .instance_id)-8080.$base +a_url=https://$(printf "%s\n" "$info" | jq -r '.instance_id + "-8080." + .gateway.base_domain') # dstack drops env vars missing from allowed_envs, and the replica inherits A's list. A replica # without DORMANT and PREDECESSOR_URL would boot as a second writer with empty state. -printf "%s\n" "$info" | jq -e '.compose_file | if type == "string" then fromjson else . end | .allowed_envs - | index("DORMANT") and index("PREDECESSOR_URL") and index("GATEWAY_BASE")' > /dev/null \ - || { echo "$A was not deployed with -e PREDECESSOR_URL= -e DORMANT= -e GATEWAY_BASE=" >&2; exit 1; } +printf "%s\n" "$info" | jq -e '.compose_file | if type == "string" then fromjson else . end | .allowed_envs | index("DORMANT") and index("PREDECESSOR_URL")' > /dev/null \ + || { echo "$A was not deployed with -e PREDECESSOR_URL= -e DORMANT=" >&2; exit 1; } # 1. A replica of A with the same compose. DORMANT keeps it forwarding to A instead of taking over. env=$(mktemp) -printf 'DORMANT=1\nPREDECESSOR_URL=%s\nGATEWAY_BASE=%s\n' "$a_url" "$base" > "$env" +printf 'DORMANT=1\nPREDECESSOR_URL=%s\n' "$a_url" > "$env" B=$(phala cvms replicate "$A" -e "$env" --json | jq -r .vm_uuid) echo "replica $B" until phala cvms get "$B" --json | jq -e '.status == "running"' > /dev/null; do sleep 10; done # 2. B alone reboots into the new compose and takes the writer role and state from A. -phala deploy --cvm-id "$B" -c "$NEW" -e PREDECESSOR_URL="$a_url" -e DORMANT= -e GATEWAY_BASE="$base" --wait -b_url=https://$(phala cvms get "$B" --json | jq -r .instance_id)-8080.$base +phala deploy --cvm-id "$B" -c "$NEW" -e PREDECESSOR_URL="$a_url" -e DORMANT= --wait +b_url=https://$(phala cvms get "$B" --json | jq -r '.instance_id + "-8080." + .gateway.base_domain') until curl -sf "$b_url/_status" | jq -e '.role == "writer"' > /dev/null; do sleep 3; done # 3. A now forwards to B. Let requests already routed to A finish, then remove it.