Skip to main content

bench

@webda/bench

@webda/bench

Benchmark framework for the Webda abstractions. One suite per abstraction, shared by every backend:

SuiteFunctionAbstractionScenarios
store.*benchStoreStoreget, exists, update, patch, setAttribute, incrementAttribute, upsert, queries (eq, range, IN, ORDER BY), create, delete, contention, paginate, iterate, bulk ops
queue.*benchQueueQueuesend, raw drain, consume() per maxConsumers × workerParallelism, open-loop end-to-end latency, redelivery
pubsub.*benchPubSubPubSubServicepublish, open-loop end-to-end latency, end-to-end throughput, fan-out, payload sizes

Stores are benchmarked through the Repository API, the path applications use.

Run​

pnpm run bench                                   # every package, sequentially (repository root)
pnpm --filter @webda/fs run bench                # one package
WEBDA_BENCH_PROFILE=full pnpm run bench          # smoke | quick (default) | full
WEBDA_BENCH_FILTER="store.get,queue.*" pnpm run bench
pnpm run bench:report                            # merge every bench-results folder into one table

Results are written to <package>/bench-results/<suite>-<backend>.json (WEBDA_BENCH_OUTPUT to change the folder).

ProfileUseDatasetConcurrencyOps per scenario
smokeunit tests of the suites501, 420
quickpull requests (default)1,0001, 101,000
fullnightly / manual10,0001, 10, 5010,000

Add a backend​

Benchmarks are regular vitest files named *.bench.ts, so they reuse the Webda test harness (WebdaApplicationTest, addService). The package only provides a harness: how to create its service.

// packages/<backend>/src/<service>.bench.ts
import { suite, test } from "@webda/test";
import { WebdaApplicationTest } from "@webda/core/lib/test/application.js";
import { benchStore, useServiceType } from "@webda/bench";
import type { MyStore } from "./mystore.service.js";

@suite
class MyStoreBench extends WebdaApplicationTest {
  getTestConfiguration() {
    return { parameters: { ignoreBeans: true }, services: {} };
  }

  @test
  async store() {
    await benchStore({
      backend: "MyStore",
      // The class registered by the application: it has the configuration factory (defaults, unit conversions)
      createStore: (name, models) =>
        this.addService(useServiceType<MyStore>("Webda/MyStore"), { models /* connection... */ } as any, name),
      clean: store => store.__clean(), // optional, faster than deleting the documents one by one
      capabilities: { orderBy: true, bulk: true }, // optional, skip unsupported scenarios
      emulated: false // true for localstack / GCP emulators
    });
  }
}

Then in the package:

  • vitest.bench.config.ts: 
    import { defineConfig } from "vite";
    import { benchVitestOptions } from "@webda/bench/lib/vitest.js";
    export default defineConfig({ test: benchVitestOptions() });
  • package.json: "bench": "vitest run --config vitest.bench.config.ts" and "@webda/bench": "workspace:*" in devDependencies
  • tsconfig.json: add "src/**/*.bench.ts" to exclude, so bench files are not compiled into lib

Harness options:

HarnessOptionPurpose
StoreBenchHarnesscleanempty the store before/after the suite
capabilitiesorderBy, bulk, atomic: skip unsupported scenarios
QueueBenchHarnesspurgeempty the queue between scenarios (defaults to __clean)
inconsistentSizesize() is approximate (SQS, GCP): size checks skipped
redeliveryDelayMsdelay before an unacknowledged message is delivered again: enables redelivery
rawReceivefalse for queues delivering only through consume() (GCPQueue consumer mode)
PubSubBenchHarnessmaxPayloadBytesskip bigger payloads (e.g. 7900 for Postgres NOTIFY)
maxSubscribersskip bigger fan-outs
allemulatedresults come from an emulator: regression tracking only

Application benchmark: blog-system​

sample-apps/blog-system/bench/ measures a whole Webda application: requests go through the Router (routing, operation resolution, validation, canAct / getPermissionQuery, audit, store), without the network.

ScenarioWhat
blog.domain.<op>each operation alone: feed, readPost, tagPage, profile, comment, draft, publish, follow
blog.domain.mixweighted mix (35% feed, 30% post pages, 10% tags, 5% profiles, 10% comments, 5% drafts, 3% publish, 2% follow), with and without the AuditService
blog.loginpassword check of User.login
blog.http.<transport>.<read>get a post and the feed over REST and GraphQL, through a real HTTP server (keep-alive)
blog.coldboot.<phase>boot in fresh Node processes like webda serve / serverless entrypoints: node start, framework import, application load, Core init, first and warm request
blog.bootload the application, create and initialize the Core (warm, in process)

The blog is seeded deterministically (smoke: 30 posts, quick: 1,000 posts, full: 10,000 posts) in the store selected by WEBDA_BENCH_BLOG_STORE (memory, fs, postgres, mongodb); the store modules not under test are not loaded, so the application boots as it would be shipped.

blog.coldboot.init is also split in blog.coldboot.resolve (create and resolve the services) and blog.coldboot.servicesInit (their init(), the slowest services are in its checks). Every phase is measured twice: plain node, and with compileCache=true, the Node compile cache the webda launcher enables. To profile one boot: node bench/coldboot-child.mjs prints the phases and the init time of each service.

pnpm --filter @webda/sample-blog-system run bench

Measurement​

  • Closed loop: N workers each waiting for its operation. Measures throughput.
  • Open loop: operations start at a fixed rate whatever the previous ones did, latency is measured from the scheduled start: a slow backend cannot hide its queueing delay (no coordinated omission). Used for end-to-end latency.
  • Custom: the scenario measures itself (drains, fan-out deliveries).

Latencies are recorded in native histograms (perf_hooks.createHistogram); each result also holds CPU, memory, event loop delay, the delta of the Webda metrics of the service under test, and correctness checks (lost / duplicated messages, final counter of the contention scenario...). A result with more than 1% errors or a failed check is flagged unstable and excluded from regression comparisons.

Compare runs​

webda-bench report <paths...> [--markdown file] [--summary] [--github-benchmark prefix]
webda-bench compare <baseline> <current> [--threshold 1.5] [--fail]

--github-benchmark writes the customBiggerIsBetter (throughput) and customSmallerIsBetter (p99) files of github-action-benchmark.

Classes​

Interfaces​

Type Aliases​

Variables​

Functions​