Skip to content

About

OpenEverest generic plugin to benchmark databases

Topics

Resources

Security policy

Stars

4 stars

Watchers

0 watching

Forks

Repository files navigation

OpenEverest Plugin Bench

OpenEverest Plugin Bench runs PostgreSQL (pgbench), MySQL (Sysbench), and MongoDB (go-ycsb) benchmarks as temporary Kubernetes Jobs from the database cluster tab or backend API. All three drivers are enabled by default in the Helm chart.

MVP complete: The PostgreSQL pgbench workflow supports configuring, launching, and inspecting benchmark runs.

How it works

The plugin backend coordinates a benchmark run:

  1. Resolve authorized connection details for the selected instance through the OpenEverest API.
  2. Pick the database: the one entered in the form, otherwise the instance's default database (the provider's database connection key, or the database in its connection URI). The go-ycsb driver supports only the ycsb database.
  3. Select the requested driver, validate its options and connection details, and create a temporary Secret containing its password or connection URI.
  4. Create a Kubernetes Job using the configured driver image and prepared settings.
  5. Wait for the Job and collect the runner logs and optional structured metrics.
  6. Delete the Job and temporary Secret.

The benchmark tab is built with MUI and themed from the host through @openeverest/plugin-theme. The backend uses a shared driver interface with pgbench, Sysbench, and go-ycsb implementations. Additional benchmark tools can supply their own option validation and workload preparation while sharing the Kubernetes execution lifecycle.

Backend API and results

New clients submit driver and a driver-specific options object to POST /api/runs. Target, database, resources, and node affinity remain shared request fields. The driver applies defaults, validates options, and records the effective configuration and configured runner image for later inspection.

GET /api/runs/{id} returns status, effective settings, bounded log output, and optional numeric metrics. Access requires authorization for the stored target. Missing or invalid metrics do not turn a successful benchmark into a failed run. The frontend submits the explicit driver/options envelope and displays these effective settings and metrics.

GET /api/runs?k8sCluster=…&namespace=…&instance=… lists one instance's runs, newest first. limit is optional (default 20, maximum 100). The caller's access to the instance is verified through OpenEverest before history is read. Entries contain status, timestamps, effective settings, metrics, and errors but omit runner output; fetch a single run for its logs. Both run responses include the target's databaseType as reported by OpenEverest.

GET /api/status returns generic backend readiness with status, runnerReady, storage (ephemeral for in-memory history, postgresql when persistent), message, and a sorted drivers array of enabled driver IDs, such as ["go-ycsb", "pgbench", "sysbench"]. Disabled drivers are excluded. The former PostgreSQL-only database field has been removed. This endpoint does not test database connectivity or runner-image availability.

See the request and runner result contracts for payloads, defaults, compatibility, and metrics transport details.

MySQL benchmarks through the API

Sysbench is enabled by default in Helm. Submit an explicit driver: "sysbench" with the OpenEverest target and Sysbench options to POST /api/runs:

{
  "target": {"k8sCluster": "main", "namespace": "default", "instance": "my-mysql"},
  "database": "sbtest",
  "driver": "sysbench",
  "options": {"durationSeconds": 5, "threads": 2, "tables": 1, "tableSize": 1000, "initialize": true}
}

The target must report database type mysql; its database and account must already exist. Poll GET /api/runs/{id} with the accepted ID and caller authorization. See the request contract for defaults and connection rules, and the runner README for standalone usage and table cleanup.

MongoDB benchmarks through the API

go-ycsb is enabled by default in Helm and uses the runner image from the matching release. Tilt supplies a locally built image for development. Submit this body to POST /api/runs with caller authorization:

{
  "target": {"k8sCluster": "main", "namespace": "default", "instance": "my-mongodb"},
  "database": "ycsb",
  "driver": "go-ycsb",
  "options": {"collection": "benchmark_records", "recordCount": 1000, "operationCount": 2000, "threads": 2, "initialize": true}
}

The target must report type mongodb and supply a mongodb:// URI. Only the ycsb database is supported by the pinned upstream tool: omit database or set it to ycsb; other names are rejected. The backend preserves authentication, including provider URIs using /admin, while targeting ycsb. The supplied account needs read/write access to ycsb. For TLS instances, a provider CA published as ca.crt in the connection details is passed to the runner to verify the server certificate.

Workload A uses equal reads and updates. Set initialize=true to load a fresh, disposable collection; reloading existing keys fails. To reuse it, set initialize=false and keep the collection and record count unchanged. Job cleanup leaves the data in MongoDB; drop the collection separately when finished. Poll GET /api/runs/{id} for logs and optional ops_per_second (ops/s) and latency_avg (ms). See the runner reference for defaults.

Using the benchmark tab

Open Performance Benchmark in the sidebar to list accessible instances in the host's current Kubernetes cluster. Only providers supported by an enabled driver appear. Select an instance or Benchmark to open its benchmark tab; use Retry if discovery fails. Discovery uses your OpenEverest permissions and reads instance metadata without requesting database credentials.

Open a supported instance's Performance Benchmark tab. Its provider determines which driver is offered:

Instance provider Driver
provider-cloudnative-pg, provider-percona-postgresql pgbench
percona-server-mysql, percona-xtradb-cluster Sysbench
percona-server-mongodb go-ycsb

The tab checks /api/status and only offers drivers enabled on the backend. If none are enabled for the provider, run lookup remains available.

Database, Job resources, and Job placement are shared settings. For SQL drivers, a blank database uses the instance default when available; otherwise enter an existing database name. Use a disposable benchmark database. For both percona-server-mysql and PXC (percona-xtradb-cluster), a database name is required because neither provider supplies a default. Create the database before starting the benchmark; table preparation does not create it.

pgbench starts with 30 seconds, one client/thread, scale 1, and initialization enabled. Initialization drops and recreates pgbench tables. When reusing tables, scale is disabled and displayed as From existing tables in the run result.

Sysbench starts with 30 seconds, one thread, one table, 10,000 rows per table, and preparation on. It runs oltp_read_write. Prepare Sysbench tables creates a fresh dataset; existing table names cause preparation to fail. To reuse tables, turn preparation off and match Tables and Rows per table to the existing dataset. Tables remain after Job cleanup. Sysbench must be enabled with an accessible runner image; see Driver configuration.

go-ycsb fixes Database to the read-only value ycsb. It starts with collection usertable, 10,000 records, 20,000 operations, one thread, and Load benchmark data checked. Workload A performs equal reads and updates. Use a fresh disposable collection for loading; to reuse it, turn loading off and keep Collection and Records matched to the dataset. Operations controls work performed, subject to the overall execution timeout. The account needs read/write access to ycsb; MongoDB can create the database/collection during loading, and Job cleanup leaves the data intact.

Each accepted run has its own card and polls for status every two seconds until completion. Completed cards show the driver, configured runner image, effective options, and a metrics table. Raw logs expands the captured output; a Truncated badge identifies output that reached the backend limit. Missing or invalid metrics show Structured metrics unavailable while preserving the run's status and logs. Failed runs show their error alongside any available results.

History lists the instance's 20 most recent runs, newest first, with the tool, status, duration, and metrics (or the error of a failed run). It reloads when you start a run and when that run finishes; Refresh reloads it on demand. View opens a run as a card with its full results and logs; runs already shown are marked Shown. When history is kept only in memory, the section says so. See run history storage.

Use Look up a benchmark run to retrieve a stored run by ID. Looking up an already displayed ID does not add a duplicate card. Records remain subject to the configured run history storage.

Runner placement

Administrators can pin every runner Job to specific nodes, such as a tainted, dedicated benchmark node pool, with runner.nodeSelector and runner.tolerations in Helm:

runner:
  nodeSelector:
    node-role/benchmark: "true"
  tolerations:
    - key: dedicated
      operator: Equal
      value: benchmark
      effect: NoSchedule

Invalid values stop the backend at startup. Users can additionally require node labels for a single run under Job placement in the UI. Those rules are sent as nodeAffinity and apply on top of the deployment's node selector, so a node must satisfy both.

Scheduling failures

While waiting for a Job, the coordinator checks the runner Pod's scheduling condition. If the same Pod is continuously observed as PodScheduled=False with reason Unschedulable for runner.schedulingGracePeriod (default 2m, or BENCHMARK_SCHEDULING_GRACE_PERIOD), the run fails and its Job and temporary credential Secret are cleaned up. Users see a generic "could not be scheduled" message; the scheduler's explanation, which reveals node counts and taints, is written to the backend log.

The grace period starts at the first observation of scheduler rejection, not at submission. Scheduling recovery, Pod replacement, or a gap in scheduling observations resets it. It does not limit benchmark runtime or image downloading. Choose a period that allows enough time for autoscalers to provision nodes, or set "0" to disable early failure and skip the extra Pod lookup on each Job poll.

BENCHMARK_EXECUTION_TIMEOUT (default 5m) bounds the whole run, including scheduling, image startup, connection setup, and table preparation. The duration option limits only the measured benchmark phase.

Requirements

For local development, install:

  • Docker
  • kubectl
  • Helm
  • k3d
  • Tilt
  • Go
  • Node.js and npm

An OpenEverest installation and a provider for the selected database type are required when running the plugin in a cluster. Sysbench matches database type mysql rather than provider name. A live benchmark against a PXC (percona-xtradb-cluster) instance has been verified. The API also accepts provider-mariadb targets reporting mysql, but MariaDB compatibility has not been verified. The Sysbench runner has been verified against MySQL 8.4 on Linux/ARM64.

Local development

Create the development cluster and start the Tilt environment:

make dev-up

If the cluster already exists, run:

tilt up -f dev/Tiltfile

Tilt builds the frontend, backend image, and pgbench, Sysbench, and go-ycsb runner images. It injects each runner's repository and tag through drivers.<id>.image.repository and drivers.<id>.image.tag. Runner images are used when the coordinator starts benchmark Jobs; no published runner image is needed for this flow.

Set BENCH_STORAGE=postgresql in dev/.env to deploy a disposable PostgreSQL and store run history there, which lets you exercise persistence and restart recovery locally. Its data is lost when that Pod is replaced.

Run history storage

By default, run history lives in backend memory: it is lost on restart and completed runs expire after one hour. To keep history, point the plugin at a PostgreSQL database, for example one provisioned with OpenEverest:

  1. Create a database for the plugin, such as benchmarks, and an account that can create tables in it.

  2. Create a Secret in the plugin's release namespace that holds the connection URI. Kubernetes reads Secret references only from the Pod's own namespace, so a connection Secret in another namespace must be copied:

    kubectl -n everest-system create secret generic plugin-bench-history \
      --from-literal=uri='postgresql://bench:<password>@<host>:5432/benchmarks?sslmode=require'
  3. Enable PostgreSQL storage:

    storage:
      type: postgresql
      postgresql:
        existingSecret: plugin-bench-history
        uriKey: uri

On startup, the backend applies its own schema migrations (the benchmark_runs, benchmark_run_metrics, plugin_bench_installation, and plugin_bench_schema_migrations tables) under an advisory lock and refuses to start against a schema created by a newer release. A database belongs to one plugin installation, identified by its Helm release name: a second installation pointed at the same database refuses to start before changing its schema, so give each installation its own database or schema (for example, with search_path=<schema> in the URI). An unreachable database or an invalid URI stops startup rather than falling back to memory. Each run records its target, database type (for example postgresql or mysql), driver, effective options, resources, status, timestamps, and redacted logs; it never contains target database credentials. Metrics are stored one row per name and unit in benchmark_run_metrics, so any benchmark tool can report its own metrics and results can be compared across runs with SQL. PostgreSQL history is kept until you delete it; there is no automatic retention yet.

Avoid benchmarking the database that stores history: the benchmark load skews its own results and competes with history writes.

Restart recovery

A run cannot be resumed after the backend restarts, because redacting its logs requires target credentials that only existed in the previous process. Before accepting requests, the backend therefore marks runs left running as failed ("interrupted because the benchmark service restarted") and deletes the runner Jobs and credential Secrets in the workload namespace labeled app.kubernetes.io/managed-by=plugin-bench and app.kubernetes.io/instance=<release name>. Other installations' resources are untouched. In memory mode, the records are already gone, but leftover resources are still removed. If cleanup fails, for example because of a transient Kubernetes API error, startup continues and cleanup is retried in the background with backoff until it succeeds. Each backend process also labels its runner resources with benchmark.openeverest.io/backend-process, and cleanup skips the current process's resources, so retries never touch runs started after the restart.

This cleanup assumes one backend per installation. The chart uses the Recreate Deployment strategy, so the old backend stops before the new one starts, and rejects replicaCount values other than 1.

Driver configuration

Configure enabled benchmark drivers and their runner images through Helm:

drivers:
  pgbench:
    enabled: true
    image:
      repository: ""
      tag: ""
      pullPolicy: ""
  sysbench:
    enabled: true
    image:
      repository: ""
      tag: ""
      pullPolicy: ""
  go-ycsb:
    enabled: true
    image:
      repository: ""
      tag: ""
      pullPolicy: ""

Empty image fields use the defaults below after any legacy pgbench overrides. With no overrides, pgbench uses ghcr.io/openeverest/plugin-bench-pgbench:<appVersion> and IfNotPresent. appVersion comes from the chart being installed, including metadata set by helm package --app-version during a release.

All three drivers are enabled by default and use their matching release images. To use a custom Sysbench image, override its repository or tag:

drivers:
  sysbench:
    enabled: true
    image:
      repository: registry.example.com/team/plugin-bench-sysbench
      tag: tested-version
      pullPolicy: IfNotPresent

The example registry and tag are placeholders for an image you supply. Empty Sysbench image fields resolve to ghcr.io/openeverest/plugin-bench-sysbench:<appVersion> and IfNotPresent; the release workflow publishes the matching image alongside the chart.

Image field First choice Deprecated pgbench fallback Default
Repository drivers.<id>.image.repository runner.image.repository ghcr.io/openeverest/plugin-bench-<id>
Tag drivers.<id>.image.tag runner.image.tag Chart appVersion
Pull policy drivers.<id>.image.pullPolicy runner.image.pullPolicy IfNotPresent

Precedence applies independently to each field. For example, this pins just the pgbench runner version while inheriting its repository and pull policy:

drivers:
  pgbench:
    image:
      tag: "0.1.0"

Legacy runner.image overrides apply only to pgbench. Migrate them to drivers.pgbench.image when updating your values. Shared resources, timeouts, node selectors, and tolerations remain under runner; the backend Deployment's own image remains under the top-level image settings.

Helm serializes the resolved settings into one backend environment variable. For a chart with appVersion: 0.2.0:

{"go-ycsb":{"enabled":true,"image":"ghcr.io/openeverest/plugin-bench-go-ycsb:0.2.0","imagePullPolicy":"IfNotPresent"},"pgbench":{"enabled":true,"image":"ghcr.io/openeverest/plugin-bench-pgbench:0.2.0","imagePullPolicy":"IfNotPresent"},"sysbench":{"enabled":true,"image":"ghcr.io/openeverest/plugin-bench-sysbench:0.2.0","imagePullPolicy":"IfNotPresent"}}

The variable is named BENCHMARK_DRIVERS. Each entry requires an explicit enabled boolean. Enabled entries require a nonempty image reference, and pull policy must be Always, IfNotPresent, or Never (omitted/empty defaults to IfNotPresent). Only drivers compiled into the backend can be enabled; adding a Helm entry does not install an implementation. The compiled drivers are pgbench, Sysbench, and go-ycsb. At least one must remain enabled with a usable image.

For deployments setting environment variables directly, RUNNER_IMAGE and RUNNER_IMAGE_PULL_POLICY remain deprecated pgbench fallbacks only when BENCHMARK_DRIVERS is absent. In that legacy mode, RUNNER_IMAGE is required and pull policy defaults to IfNotPresent. A present payload is authoritative: empty/malformed JSON, unknown fields, unknown enabled drivers, or no enabled drivers stop startup instead of falling back to legacy variables. Explicit disablement is never reversed by a legacy image setting.

Runner settings

The pgbench runner receives its PostgreSQL connection through these variables:

PGHOST
PGDATABASE
PGUSER
PGPASSWORD or PGPASSFILE

Benchmark options are configured with:

BENCH_INITIALIZE   default false
BENCH_DURATION     default 60 seconds
BENCH_CLIENTS      default 1
BENCH_THREADS      default 1
BENCH_SCALE        default 1

BENCH_INITIALIZE=true drops and recreates the pgbench tables. Use it only with a disposable database. See runners/pgbench/README.md for detailed runner instructions.

The Sysbench runner receives MYSQL_HOST, MYSQL_PORT, MYSQL_DATABASE, MYSQL_USER, and MYSQL_PASSWORD, plus BENCH_DURATION, BENCH_THREADS, BENCH_TABLES, BENCH_TABLE_SIZE, and BENCH_INITIALIZE. The backend supplies the password through a Kubernetes Secret reference; ordinary connection and benchmark settings use environment values. See the Sysbench configuration reference for defaults and credential-file encoding rules.

The go-ycsb runner receives MONGODB_URI and, when the provider publishes one, MONGODB_TLS_CA through Secret references and ordinary BENCH_* options through environment values. See its configuration reference.

Useful commands

npm ci
npm run test:form
npm run test:components
npm run test:submission
npm run test:status
npm run test:polling
npm run test:result
npm run build

cd backend && go test ./...
cd backend && go vet ./...
# PostgreSQL storage integration tests (skipped unless the URI is set):
cd backend && PLUGIN_BENCH_TEST_POSTGRES_URI='postgres://user:pass@localhost:5432/db?sslmode=disable' go test ./internal/pgstore/ .

helm lint charts/plugin-bench/
docker build -t plugin-bench-pgbench:dev runners/pgbench
docker build -t plugin-bench-sysbench:dev runners/sysbench
docker build -t plugin-bench-go-ycsb:dev runners/go-ycsb

Both PR CI and release builds run these frontend test suites before building the bundle. Component tests include the complete submission/polling/results and run lookup workflows with mocked API responses, including concurrent cards and cancellation on unmount or target changes.

For a live check after starting Tilt, use a disposable supported database:

  1. Start a short run and confirm the POST contains driver and nested options.
  2. Confirm polling reaches a terminal status and stops. Check effective options, metrics, and expandable logs; look up the same ID and confirm no duplicate card.
  3. Start two runs and confirm both cards retain their own status and results.
  4. Leave the tab during a run and confirm status requests stop. A missing run ID should pause status checks with a clear error rather than resubmit a benchmark.

For Sysbench, prepare a small fresh dataset, then repeat with preparation off and matching table settings. Preparing the same tables again should fail clearly. For go-ycsb, load a fresh collection, then repeat with loading off and the same collection and record count.

Releasing

Push a vX.Y.Z tag, or vX.Y.Z-<pre-release> for a pre-release, to run the release workflow. It publishes:

  • ghcr.io/openeverest/plugin-bench, ghcr.io/openeverest/plugin-bench-pgbench, ghcr.io/openeverest/plugin-bench-sysbench, ghcr.io/openeverest/plugin-bench-go-ycsb, and ghcr.io/openeverest/plugin-bench-hammerdb images for linux/amd64 and linux/arm64
  • the oci://ghcr.io/openeverest/charts/plugin-bench Helm chart
  • a GitHub release with the chart archive and release notes

The workflow tests all four runners from their built AMD64 archives before publishing. HammerDB is currently a standalone runner; API/UI integration is a separate task. See its runner reference for direct usage. Release notes include each image reference and digest.

GHCR creates new packages as private, so the first release fails at the "Verify public access" step. Make each new package, including plugin-bench-sysbench, plugin-bench-go-ycsb and plugin-bench-hammerdb, public in its GHCR package settings, then re-run the failed job. Publishing is idempotent, so re-runs are safe.

Project structure

src/                         React frontend
backend/                     Go backend
backend/internal/everest/    OpenEverest API client
backend/internal/driver/     Driver contracts, registry, workload preparation
backend/internal/coordinator Kubernetes Job and Secret orchestration
backend/internal/runnerresult Generic runner metrics decoder
backend/internal/api/storetest Shared run store conformance tests
backend/internal/pgstore/    PostgreSQL run history storage and migrations
runners/pgbench/             pgbench runner image
runners/sysbench/            MySQL Sysbench runner image
runners/go-ycsb/             MongoDB go-ycsb runner image
runners/hammerdb/            Standalone PostgreSQL/MySQL HammerDB runner image
charts/plugin-bench/         Helm chart and Plugin resource
dev/                         Tilt and k3d configuration
spec.md                      MVP requirements

Current limitations

The UI supports the providers listed above; MySQL requires an enabled Sysbench driver and a usable image. MongoDB is limited to the ycsb database and Workload A; additional workloads remain future work. With the default in-memory storage, run records, including output and metrics, are lost on restart, and completed records become unavailable one hour after completion. Configure PostgreSQL storage to keep history. Runs in progress during a restart are marked failed rather than resumed. One backend replica is supported. The UI lists recent runs per instance; paging beyond them, charts, and comparisons remain future work.

License

Licensed under the Apache License 2.0. See LICENSE.

About

OpenEverest generic plugin to benchmark databases

Topics

Resources

Security policy

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages