Benchmark history¶
Loading the chart…
The CI benchmark's figures over master's commits, oldest to newest. The chart above
draws every figure at once, each line on its own axis; below it, each chart is one
figure, each point a commit (hover or focus a chart for its commit, pull request, harness
version and runs; each chart's table lists every point, linked to its commit, its pull
request and the Actions runs that measured it). The benchmark itself is
described in bench/README.md, and each harness version's changes in
bench/CHANGELOG.md.
- Lines by harness version. A figure means the same only within a major version of
the harness, so each major version is its own line, and a new one starts with a
hairline (b8, b9, …). Its first point is usually the commit before its first record,
measured again by the new harness, so a model's correction shows as a jump at one
commit, not as a speed-up. Where a harness also computes an earlier model's figure on
the same runs (
eta_10,pipeline_eta_9,finalise_eta_8, …), that figure is dashed, in the earlier model's colour. - Bands are the 95% interval of the mean over the runs of that commit.
- Runner speed, and the default view: GitHub's runners differ by up to about 1.8× on one commit (AMD EPYC 7763, 9V45, 9V74, and a few Intel Xeons; the 9V45 runs the release ETA in about 0.64 of the 7763's time), so a commit that drew faster or slower runners moves by itself. The page opens on "One runner CPU", the CPU with the most runs (the EPYC 7763, which measured every commit from bench 2 on): its points compare like with like. "All runners (mixed)" averages every run of a commit whatever its CPU (its hover lists each CPU's mean). "Paired change" shows each merge's change against the commit it ran beside on the same runners, which the runner mix cannot move.
- Cumulative change chains master's merges' own paired changes (the ledger's records, each against the commit before it; never the backfill's) from a chosen commit, #133 by default: the ratios multiplied, the 95% band each step's half-width added in quadrature on the log scale, the steps' runs taken as independent. Commits the bench skipped (docs-only pushes, the same code) count as no change. The table above the charts gives the headline figures' totals; a wide interval there usually comes from one or two noisy records, which "Paired change" shows.
- Shift-left: a change that moves work from one stage to another (#134 and #148 moved work from the pipeline into the split, by design) raises the stage it moved to while the total falls. In "CPU by stage" and the ETAs, read a stage's rise beside the total's row before calling it a regression.
- Backfill: the campaign's milestone commits from #84 to #132 (marked on the x axes), measured by bench 11.0.0 from the Actions tab, each paired with #132 on the same runners, join bench 11's line, so its figures (the measured end-to-end total among them) reach back past its own records. Their paired changes are each older commit against #132, not a step from its predecessor (a positive one means that commit was slower than #132), so "Paired change" leaves them out unless its backfill box is ticked, and "Cumulative change" never chains them. Bench 12.0.0's line starts at #136 (a docs-only commit after #132, so #132's code), measured again by it as the commit before its first record.
- The bench's own duration (bench 17.15.0 on): how long master's bench took, a round job's mean, split into its setup, its timed measurements and its supplementary checks (the full tier's sections). It is not a figure of the pipeline, so it is left out of the chart at the top and has no paired change. Pull requests run the shorter PR tier and are not recorded (bench/README.md).
What the recent major versions changed (each a break in the figures it names):
| version | pull request | what changed meaning |
|---|---|---|
| 12.0.0 | #133 | the pipeline's input: its chunks are the split's own output (5 chunks on the quick tier, not 4; 42 on the full tier, not 66), so every pipeline and finalise figure. The ETA model is 11.0.0's |
| 11.0.0 | #130 | the pipeline ETA (its worker starts a fixed cost), and the split's slice written as the dumps write it (the split's figures and ETA); the measured end-to-end total added |
| 10.0.0 | #128 | the pipeline and finalise ETAs: the claims scaled by their own rows and bytes, not by entities |
| 9.0.0 | #119 | the finalise ETA: worker start-up a fixed cost; finalise's peak RSS at a release added |
| 8.0.0 | #111 | finalise on three times the rows: the claims key from a release's file layout; finalise's peak RSS added |
The data is scripts/bench_history.py's, from the bench-results branch (master's
records) and docs/benchmarks/backfill/, regenerated by the site's build on every push to
master and after every Benchmark run on master.
Loading the benchmark history…