Benchmarks

Single source of truth for every benchmark in this repo: what each one measures, when to use it, and the one-line command that runs it.

Everything goes through scripts/bench. Run scripts/bench help for the full command list; scripts/bench list queries both the interpreter and compiler benchmark projects and enumerates every available @Benchmark method without duplicates.

One command per case

Default JMH config is -wi 3 -i 5 -f 1 (~25 s per bench). Override with BENCH_WI, BENCH_MI, BENCH_F env vars when you want tighter A/B:

BENCH_F=2 scripts/bench interp recursionFib   # two forks for a stable A/B
BENCH_WI=1 BENCH_MI=1 scripts/bench interp    # quick smoke across all benches

What lives where

JMH (the sbt-driven microbenchmarks)

If you are micro-optimising the interpreter, almost everything you need is in InterpreterBench. The cross-backend benches are for periodic "how far are we from native?" checkpoints.

Wall-clock (subprocess-based)

Use the wall-clock benches when you need cold-JVM, fresh-process numbers (JMH only measures the warmed-up steady state).

Anti-fold strategy (keeping the corpus honest across backends)

A constant-folding compiler can replace a whole benchmark loop with a single constant load, reporting a dishonest ~0 ms. The corpus defends against this uniformly but minimally:

• Primary (all backends): each folded workload carries a non-linear 64-bit

LCG seed and consumes every result, so no backend can derive a closed form (see docs/bench/corpus-antifold.md).

• Rust only — one extra barrier: LLVM's scalar evolution is far more

aggressive than HotSpot/V8 (it solves affine/polynomial recurrences symbolically — even opaque inputs don't stop it). So bench/run.sc adds a single std::hint::black_box(...) on the first loop-carried reassignment of each emitted pub fn. One barrier is necessary and sufficient: measured on sumTco(100000,0) at -O3, 0 barriers folds to 0.000001 ms, one barrier on the carried accumulator gives an honest 0.10 ms, and the loop's time scales linearly with the trip count. The earlier harness wrapped every assignment (3–4 barriers/iter), which inflated rust loop cells 3–4× and made the column look slower than codegen-equal jvm; that redundant tax is gone (recursion-tco 0.34 → 0.025 ms, now ≈ jvm). A single irreducible barrier still taxes rust on loops jvm folds for free — that asymmetry is real, not a harness defect.

Adding a new workload to the dashboard

The wall-clock dashboard (bench/BASELINE.md) is fed by bench/corpus/*.ssc, auto-discovered by bench/run.sc — there is no registry to edit. To add one:

1. Create bench/corpus/<name>.ssc. It is a normal .ssc markdown doc: a

title, a short paragraph saying what the workload measures and on which backends it is supported, then one scalascript ` fence defining the entry point. The harness (ssc bench --machine) calls one of:

• def workload(seed: Long): Long — preferred. The harness feeds an opaque

seed; carry it through a generator so nothing constant-folds (see below).

• def workload(): Long — only when the body is already non-foldable.

Return a Long and make the loop consume every result into it, or a backend may delete the loop. Put any fixtures (a top-level val, a case class) above the def, inside the same fence.

2. Keep it honest (anti-fold). Carry a per-iteration generator and feed its

output into the work:

• Default: the 64-bit LCG s = s * 2862933555777941757L + 3037000493L

(relies on 64-bit wrap; fine for interp/JVM/Rust).

• If the value feeds an array index (or anything where overflow must not

escape a range): use MINSTD s = (s * 48271L) % 2147483647L (start (seed % 2147483646L) + 1L). Its product stays below 2⁵³, so it is exact in JS's f64 Number; a 64-bit-wrapping LCG overflows to NaN on JS and an arr(NaN) read returns undefined → crash → a misleading n/a. See vector-index.ssc / array-update.ssc for the pattern.

3. Verify on one backend (fast): ./bench.sh --backend ssc <name>. A backend

that can't run the workload (e.g. Rust has no Vector/Array/LazyList, JS has no LazyList.from) reports n/a — that is the honest support signal, not a failure. Run the other backends with --backend jvm|js|rust.

4. Publish numbers. ./bench.sh --baseline regenerates the whole corpus

table into bench/BASELINE.md (needs bin/ssc staged via sbt installBin, plus node / scala-cli / a Rust toolchain for those columns). To add just your rows without a full multi-backend sweep, run the single-backend commands above and paste a focused table into bench/BASELINE.md (as the Collection-type microbenchmarks section does).

The matching JMH micro (InterpreterBench/RuntimeBench) is optional and separate — add a method there only if you want a warmed-up steady-state number; note the scale caveat below.

⚠️ JMH and the corpus measure different scales under the same name

Several JMH methods in InterpreterBench share a name with a bench/corpus/*.ssc workload but run a different amount of work, so their absolute numbers are not directly comparable:

So a reader comparing JMH typeclassFold ≈ 0.009 ms to the cross-backend table's typeclass-fold ≈ 1.8 ms is comparing one call to three thousand — both are honest, they just measure different scales. When a JMH method is a deliberate micro-call, prefer a …Macro sibling that mirrors the corpus loop for A/B work (e.g. typeclassFoldMacro was added for exactly this reason). If you add a JMH method that diverges from its corpus namesake, either give it a Macro/Micro suffix or note the divergence in its source comment.

What each interpreter bench is for

Listed in InterpreterBench.scala order. The name pattern is what you pass to scripts/bench interp <pattern> (regex; matched against the full method name).

Type-level lambdas — capability tracker (not a perf bench)

Type lambdas are surface-only (types are erased at runtime in this interpreter-first language), so there is nothing to micro-benchmark. The equivalent "see what works / where the progress is" artifact is lang/core/.../typer/TypeLambdaProgressTest.scala: [now] tests pin the current parser/SType behaviour; [target] tests are pending and flip to passing as type-lambda-p2 lands. The passing-vs-pending split is the progress dashboard (run sbt "core/testOnly scalascript.typer.TypeLambdaProgressTest"). A real parse/typecheck-throughput bench is only worth adding once the surface parses.

JIT backend selector

The bytecode JIT has two implementations of the JitBackend SPI:

To A/B the two backends:

scripts/bench interp recursionFib   # Javac (default)
scripts/bench asm    recursionFib   # ASM
scripts/bench asm                   # all benches with ASM backend

Expected: numbers within ±5% at steady state. Cold-start (first iter, low warmup) should show ASM 30–100 ms faster per function.

Off-mode A/B (proving fall-backs work)

The interpreter has two opt-out flags:

scripts/bench off recursionFib    # both BYTECODE + FASTTIER off

Expected result today: recursionFib 1.2 ms (on) ↔ ~28 ms (off). A roughly 24× gap; anything closer means the bytecode JIT isn't actually firing.

For pure-tree-walk numbers (no SscVm.exec either):

SSC_JIT=off scripts/bench off recursionFib

Profiling

scripts/bench profile recursionFib
scripts/bench compile-profile parseActors

Both routes add -prof gc (deterministic alloc rate / norm) and -prof jfr:configName=profile (sampled allocation events + CPU) to the selected JMH project. JFR output lands in that project's generated <Class>.<method>-<Mode>/profile.jfr; open it with jfr view, jfr print, or JDK Mission Control.

For interpreting alloc samples: cross-check jdk.ObjectAllocationSample counts against gc.alloc.rate.norm — a sampler can over-attribute to a hot leaf, so deterministic numbers are the tie-breaker.

Bytecode size — the perf defect no profiler shows you

scripts/bytecode-size-census <classes-dir|jar> [threshold]   # default threshold 8000
tests/e2e/v2-jit-size.sh --self-test                         # the gate, plus both its verdicts

HotSpot ships -XX:+DontCompileHugeMethods on by default: a method whose bytecode exceeds -XX:HugeMethodLimit (8000) is never compiled by C1 or C2 and runs in the bytecode interpreter for the life of the process. There is no warning, no log line, and no behavioural difference — the method is simply 10-100× slower than the identical logic split across smaller methods, forever.

A JFR/JMH profile does not point at this. It shows the huge method as hot, which is what you already expected of a dispatch point; nothing in the sampled data says "and this one is interpreted". The census is what tells you, and it takes seconds.

Measured 2026-07-28: ssc.Prims's __method__ arm — the single dispatch point for every non-arithmetic operation in every ScalaScript program — was 49,384 bytecodes. Splitting it into sequential sub-8000 parts made the v2 runtime 2.4-10.8× faster across the bench corpus while the pure-arithmetic workloads (which never reach it) did not move. See BUGS.md v2-method-dispatch-never-jits and specs/v2-runtime-perf-vs-v1.md.

Rule of thumb: run the census whenever a big match grows, and treat anything ≥6000 as a method that needs splitting soon rather than a number to watch.

Comparing v1 against v2 in one table

./bench.sh --backends ssc,v2,v2-bytecode --reps 30 <workloads…>

--backends exists because the canned --v2-backends / --v2-bytecode modes cannot express a mixed v1+v2 column set, and cross-tier claims are only meaningful when every column was measured under one machine state. Two things to keep straight when reading the result:

• v2-bytecode is the product lane (JvmByteGen emits real JVM bytecode); plain v2 is the

--interpret reference lane and is slower by design. A "v2 is slow" claim built on the v2 column alone is measuring the reference lane.

• Keep ssc in the column set as the noise control. It is unaffected by v2 work, so its

drift between two runs bounds what counts as signal (typically ≤1.3×).

Measuring the v2 JIT — warm-up must outlast tier-up

SSC_V2_JIT=on compiles hot sites while the program runs (specs/v2-wide-jit.md). The bench times per-iteration work, so a short warm-up leaves compilation happening inside the measured window and the JIT reads slower than it is. The AOT lane has no such phase — it enters measurement already compiled — which makes a naive JIT-vs-AOT table an unfair comparison in the JIT's disfavour.

Use this for any steady-state JIT number:

SSC_V2_JIT=on SSC_V2_JIT_SYNC=1 \
  ssc-tools --backend v2 bench --machine --warmup-time 3000 --reps 8 <workload>

SSC_V2_JIT_SYNC=1 makes tier-up synchronous, so it finishes deterministically during warm-up instead of racing the measurement. Measured on float-loop, three runs each:

The recipe is worth it for the variance, not just the mean: it is the only configuration here that reproduces, on a host where even the AOT lane swings 2.3×.

What this invalidates, stated so it is not repeated: a float-loop gap of 1.94× measured at --warmup-time 400 looked like a JIT code deficit and was not — in steady state the two lanes are indistinguishable (1.29 vs 1.16, inside AOT's own spread), and a direct 600 M-iteration probe puts them at parity with both spending ~60 % of samples in the same generated method.

Smoke + manifest

• scripts/bench smoke runs one iteration of one bench

(InterpreterBench.arithLoop) and writes raw JSON to bench/jmh-smoke.json. The point is "did the JMH plumbing break?", not any perf claim.

• ssc bench --smoke runs bench/corpus/hello-world.ssc through the

interpreter — same intent, but exercises the CLI's bench subcommand.

• bench/perf-manifest.yaml is the machine-readable manifest the CI/gate

policy reads from. Update it if you add a smoke target.

Default gate policy: informational. Numbers only become a non-zero exit code if the caller explicitly passes --require-target --target-ms N.

Adding a new benchmark

1. Add a private val mod<Name>: Module = src("""…""") block at the top

of InterpreterBench.scala with a one-line comment explaining WHAT the workload is meant to stress (the WHY, not the WHAT of the code).

2. Add a @Benchmark def <name>(): Unit = Interpreter(devNull).runSections(mod<Name>).
3. Smoke-run it with BENCH_WI=1 BENCH_MI=1 scripts/bench interp <name>

and confirm the result looks sensible.

4. Add a one-line row to the table above.
5. If the new bench is going to appear in the smoke path, update

bench/perf-manifest.yaml.

Avoid:

• Modules that depend on BuiltinsRuntime.initBuiltins (e.g. raw Set(...)

constructors) — the bench harness skips that init, so use .toSet instead. Map(...) is fine because it routes through intrinsics/Core.

• Workloads under ~100 µs/op — too small to separate from JMH noise.
• Workloads over ~10 s/op — pushes total run time past the patience

budget for routine A/B work.

Gotcha: stale incremental state → NoClassDefFoundError at bench init

If every interp bench suddenly fails at _jmh_tryInit with java.lang.NoClassDefFoundError: org/commonmark/ext/gfm/tables/TableCell (or a similar transitive class), it is stale incremental build state, not a missing dependency — lang/core correctly declares commonmark-ext-gfm-tables (the parser registers the GFM-tables extension on every parse). It shows up after heavy parallel-branch churn / interleaved cli/assembly builds leave the interpreterBench JMH-fork classpath inconsistent. Fix:

sbt "interpreterBench/clean" "interpreterBench/Jmh/compile"

then re-run. No source/dependency change is needed.

Related docs

• bench/README.md — gate policy and full-run recipes (durable summaries:

bench/BASELINE.md, bench/BUNDLE_SIZES.md).

• bench/perf-manifest.yaml — manifest of smoke and full-run commands.
• specs/vm-jit-spec.md — Phase-B SscVm spec (historical VmJitBench

numbers; the bench was removed 2026-06-02 as it was superseded by InterpreterBench + the off-mode flags).

• specs/vm-jit-next.md — current perf roadmap (Phase C BytecodeJit +

Phase D FastTier).