Repository navigation
Conversation
j0sh
changed the title
025f316 to
eb1f8fd
Compare
de98232 to
0f1c07b
Compare
Instrument payment phases and authoritative RPC work. Consolidate shared chain paths and RPC fixtures, retain CI regressions, and remove the local payment benchmarks and historical comparison harness.
0f1c07b to
6539bca
Compare
eb1f8fd to
15c6057
Compare
|
Completed the requested end-to-end payment-path comparison against the combined stack, using builder issue #89 as the reference. Performance-test implementation and reproduction harness: livepeer/e2e-runner#3. Revision scope: these measurements and pass/fail outcomes are for combined commit At 110 ms additional latency per JSON-RPC HTTP exchange, the combined stack reduced median serial dispatch from 3,248.753 ms to 2.619 ms (99.9194%). Its warm latency was slightly higher than PR #1, and go-livepeer remained faster at 32 clients. The combined stack passed the shared cache-correctness cases that failed on PR #1.
The six go-livepeer warm failures were paid-request HTTP 409 The test ran matched signer/orchestrator pairs on one GCP Pinned comparison inputs:
There were 48 final cells and 27,648 attempted warm jobs: three repetitions at 0/110 ms added RPC delay and 1/32 clients, with 128/1,024 jobs per repetition respectively. Each cell used fresh worker processes, payer, and databases, sharing one already-active recipient. Eight warmup jobs preceded measured load; explicit readiness retries were retained as warmup records. Timing targeted approximately 10,000,000 wei ticket EV and one ticket per job. Actual ticket parameters were decoded and retained; go-livepeer selected a different face value/winning probability with the same approximate EV. JSON-RPC batches retained their original bodies and incurred one delay per HTTP exchange. All latency distributions and observed goodput
Quantiles are pooled across the three repetitions. Serial p99 is omitted because only 384 successes are available per condition; concurrent conditions have at least 3,067. These are finite closed-loop bursts. The cached paths finish quickly, so the goodput figures do not establish sustained saturation capacity. Ratios are observed differences, without statistical confidence bounds. Immediate first-job probes are separate from warm results. The combined stack succeeded on 10/12 (83.3%), with two signer HTTP 400 At 110 ms, median successful first-job dispatch for the serial condition was 455.163 ms combined, 230.778 ms PR #1, 228.656 ms go-livepeer, and 3,255.484 ms baseline Node. The controlled payment-read sequence used four cold exchanges combined versus two batched exchanges for PR #1; both used zero warm exchanges, versus 29 warm exchanges for baseline Node. The cold failures were retained and were not substituted with successful warm requests. The real-chain and shared controlled correctness outcomes were:
The frozen-head age requirement is asserted for cached implementations. The direct baseline performs fresh RPC reads and has no equivalent cache-age promise. PR #1 failed the same additional, test-only scenarios against the shared provider fixture:
The combined stack rejected the stale/orphaned observations and canceled provider work. Other shared cases covered funding/current-round/registry changes, payer failure isolation, pre-cancellation, concurrent cold reads, and payer populations of 0/16/128/1,024. In the 32-distinct-key cold burst, combined provider concurrency was bounded at four, with 28/32 requests rejected while warm cached reads remained available. This is an admission-control tradeoff, separate from the all-successful warm load. Recovery covered expired parameters, actual redemption and duplicate rejection, persisted pending tickets across restart, RPC interruption, delayed finality, and reorg recovery. The fixtures used genuine held finalized headers, canonical EIP-1898 reads, and actual snapshot/revert. Preparation allowed a bounded 20-second wait for funded-payer visibility before scenario faults; strict post-expiry rejection remained in place. Canonical replay was distinguished by exact transaction hash from the original already-public transaction: its original receipt still had to follow expiry, every settlement receipt had to match the canonical block hash, and canonical escrow was debited once. Superseded setup/assertion failures and subsequent reruns were retained. A separate race-detector probe against the unchanged go-livepeer manifest-ID generator, in an isolated pinned source export with 32 callers, found a data race in Quiet-head refresh costs from the controlled provider test show the scaling tradeoff. Each entry is HTTP exchanges / contained JSON-RPC calls per refresh:
These quiet-history costs do not imply constant work when every payer changes on-chain. Whole-worker idle traffic also remains nonzero: under this harness at 110 ms after known-payer load, combined emitted approximately 19.18 contained RPC calls/s, versus 17.40 PR #1, 17.93 baseline Node, and 1.57 go-livepeer, counting both workers. Load-window RPC totals include background polling/redemption traffic and are not exact foreground counts. Worker resource statistics at 110 ms and 32 clients
CPU comes from worker process counters and excludes setup, the load generator, and Anvil. RSS was sampled every 100 ms; the table takes the maximum sampled peak across repetitions. Interpretation limits: the final reviewed combined revision was measured after the frozen baseline matrix on the same VM, so small differences between cached implementations can include ordering effects. The local chain mined at one-second intervals; production execution-block cadence, large history gaps, event-heavy histories, and Internet latency can alter results. This tests application behavior rather than production consensus safety. Test-only overlays did not alter the product checkouts. The raw JSONL requests/RPC logs, per-cell metadata, source/binary hashes, fixture overlays, and reproduction harness are retained with the run artifacts. The temporary VM and boot disk were deleted and independently verified absent. |
Depends on #42. Merge after that PR.
Adds payment-phase duration histograms, fixed-allowlist contract-call counts, and foreground/background RPC accounting. Existing public metric names and label sets remain compatible. Observer health, observation age, head/L1 progress and partial payer failures come from memory; scrapes make no RPC calls.
Removes redundant chain/address adapters and duplicate observer flags, and reuses shared startup and paid-request fixtures. The concrete payment chain implements the existing interfaces directly. Cancellation remains serialized with cold-read admission before shutdown joins readers.
Focused regression tests run in normal CI. They enforce four foreground RPCs for a cold cached payment and zero for warm requests, alongside the direct-read budget, RPC failures, stale observations, canonical history, reorg, cancellation and settlement checks.
Production metrics support diagnosis and external measurement. End-to-end performance, deployment/provider comparisons and load scaling are measured through livepeer/e2e-runner.
Validation: affected package race suites (
eth,internal/observability,orchestrator,signer) and diff/format checks passed. Source attribution is retained.Closes #36.