Benchmarks¶

What it costs to build a model and hand it to a solver, in specsolve and in four other ways of writing the same model. No number includes solve time. Every rival model is hand-written in bench/models/, and CI solves each against ours, so nothing here is fast because it built a different model.

dispatch

Meets a demand from a fleet of units: bounds and one balance. The shape an array library is built for, so the comparison least in specsolve's favour.

nodal

Dispatch at nodes, where a technology exists only at the nodes it is installed at: a quarter of the node and technology pairs have a variable. The chart counts the variables that exist, not the pairs.

network

A small PyPSA dispatch model: buses joined by lines, a store at every bus that carries its charge from each snapshot to the next, a technology portfolio installed at some buses only, and solar and wind availability per snapshot. Each plant draws from one of fifty pools, such as a fuel contract, and a pool caps what its plants generate together.

sector

nodal's portfolio, where each technology delivers one of five carriers, such as heat or hydrogen, at its own efficiency. A node has a balance only for the carriers its technologies deliver.

Reading the charts¶

Each panel is one model, and each line is one library. Every panel has the same axes, so a height or a size reads the same in each. Solver chooses the solver that every panel and the table run through.

• specsolve is the thick violet line.
• The grey line is the matrix floor: the time to hand a finished matrix to the solver in one bulk call, passModel for HiGHS and loadModel for Gurobi. The floor builds its arrays before the clock starts, so it reads no parquet and builds nothing. Its distance from specsolve is everything specsolve does before the solver holds the model. Its peak RSS includes the arrays as well as the solver's copy of them.
• The line is the median of nine rounds, or of five where a round takes over a second, and the whisker is the middle half. Overlapping whiskers are numbers this run cannot tell apart (why the median).
• A shaded funnel is a projection past the time or memory budget. It opens from the last measured point. One edge carries the last step's cost per variable forward, and the other keeps the last step's growth rate. A wide funnel is a guess the measurements do not narrow.
• ÷ specsolve divides each point by specsolve's at the same size, so specsolve is the line at 1× and a line above it is that many times slower or larger. It applies to wall time and to peak RSS. The window is 0× to 5× on linear axes and 0.1× to 50× on log axes, and a point outside it is named with its ratio at the edge it crosses.
• Drag a box on any chart to zoom every panel to it: its width sets the range of model sizes, and its height the range of values. A drag along one side zooms only that axis. The table keeps only the sizes inside the box. Reset zoom above the charts, or a double-click on one, shows everything again. Switching between wall time and peak RSS, or to and from ÷ specsolve, zooms out too.
• The table divides by specsolve. Purple is where specsolve is faster, orange where the other library is, grey within ten per cent. The specsolve-in-memory and matrix floor columns are not shaded: they are references, not rivals. >30 s and >16 GB are sizes the harness refused.

A script that solves one model¶

The charts leave out what a script pays to start: the interpreter, the import and the first build. This table times a new process from launch to exit, on HiGHS.

A script that solves one small model with specsolve finishes in about half a second, about half of linopy's time (#2031).

Not measured yet¶

• Sparsity other than a quarter. nodal, network and sector are sparse at one density, a quarter of the node and technology pairs. pixi run density sweeps nodal from dense to one pair in twelve, and no published run has taken that sweep.
• Solve time. Every number stops when the model reaches the solver.

Method¶

Each measurement runs in its own process, with import excluded and teardown included, on an idle machine. Peak memory is ru_maxrss. uv run --locked bench/reproduce.py runs it again with every version pinned, and the harness has every flag and caveat, such as the allocator cost on polars' peak (#896).