grid-bench
Open-source power system tools timed on the same problems, where every result is graded by an oracle that no tool under test takes part in: no speed number without a correctness number. Choose a problem; how it is set up and graded is shown with its results.
Scoreboard
Each cell counts its column's cases: ✓ / ✗ / FAILED. ✓: the oracle accepted the result. ✗: the tool returned a result the oracle rejected. FAILED: the tool raised or did not converge (see Failures). ·: the tool reads none of the column's cases.
Thread scaling
One panel per case, same y-axis. Dashed grey: ideal scaling (speedup equal to the thread count). Filled point: the oracle verified that thread count's solution; hollow: it did not. Hover a point for the time.
As a table
Accuracy
Failures
Every run that raised, with the tool's own exception.
How every problem is measured
- Solve is timed warm: one persistent model, one untimed warm-up solve, then repeated solves (median shown). Import is the median of 3 cold loads after one warm-up load.
- Oracle. Tier 1 (MATPOWER cases): the residual
V·conj(Ybus·V) − Swith Ybus built from the.mfile by the benchmark itself. Tier 2 (CGMES fixtures): deviation from the case's published SV solution. Each problem's own check builds on these; it is described with the problem, above. - Settings. Every setting of every tool is justified in its adapter's docstring, including what it deliberately does not do.
- Memory is the peak RSS of a fresh process loading and solving the case, minus the peak after importing the tool (values below 1 MB are drawn at 1 MB on the log axis). Only the benchmark process counts: cgmes2pgm's Fuseki server is not included.
AC power flow, the same problem for every tool: flat start on every solve, one slack, no reactive limits, no tap/phase-shifter/shunt control, generator voltage regulation as the case defines it, convergence tolerance 1e-8 p.u. MATPOWER cases are also converted to CGMES, and a tool reading a conversion is graded against the original .m. CGMES conformity fixtures are graded against their published SV solution.
State estimation: weighted least squares on each case with a measurement set generated from its own power flow. exact: |V| and P/Q injections at every bus without noise, so the optimum is the true state. noisy: |V| at generator buses, P/Q injections at every bus, P/Q flows at every branch's from end, with Gaussian noise. Same measurements and sigmas for every tool, flat start, no bad-data handling. The oracle accepts an estimate when one Gauss-Newton step from it moves it by at most 1e-6 and its J is no larger than J at the true state.
Optimal power flow: MATPOWER's AC-OPF on PGLib-OPF v23.07 cases (polynomial cost; power-flow equations; voltage, generator, branch MVA and angle-difference limits), flat start, tolerance 1e-6. The oracle checks the solution's feasibility against the .m and recomputes its cost, accepted at most 0.01% above PGLib's published reference (a local optimum given to five digits).
Batch power flow: each case with 100 operating points (every bus's demand along a daily curve between 60% and 100% of the case, 5% noise per bus, generators redispatched in proportion), all solved in one timed call with the base case: the base case from a flat start, then each scenario the AC power-flow problem of the Power flow tab, started from the tool's own base-case solution, as a sweep from a solved base case is run (power-grid-model takes no start voltages and starts flat). Tools with a batch API run it at 1, 2, 4, … threads, up to every core available; a tool without one runs a loop of its single solve, on one thread. Times are per scenario. The oracle grades every scenario of every thread count. The GPU machine adds 1,000 and 10,000 operating points on the larger cases: 100 leave a GPU mostly idle. The CPU baselines run these, and its full N-1 cases, at one thread and at every core only.
N-1 contingencies: each transmission case with 200 single-branch outages that keep the grid connected (all 19 of case14), solved in one timed call with the base case, each outage started from the tool's own base-case solution (power-grid-model takes no start voltages and starts flat). Contingency APIs that take a thread count run at 1, 2, 4, … threads. Times are per outage. The oracle grades every outage of every thread count against the case with that branch out. The GPU machine adds every outage that keeps the grid connected and converges (input "all outages", up to 14,384 on case9241pegase).
DC power flow and DC N-1: MATPOWER's DC model (branch admittance 1/(x·tap), resistance and charging left out, phase shifts as injections, shunt conductance as load, one reference bus) on the transmission cases, and on the N-1 cases' own outages ("all outages": every outage that keeps the grid connected). A tool reports bus angles and from-end branch flows; the oracle checks the nodal balance |Bθ − P| with the tool's angles (whatever its reference angle) and the flows against its own solution of the model, both to 1e-3 MW. An outage may be solved anew or from the base case's sensitivities (LODF, PTDF): both are exact in DC. A tool that computes flows without angles (PTDF-based) is graded on its flows alone, marked "(flows)".
CIM import/export: CIM libraries on the Svedala and RealGrid CGMES 3.0 conformity models, every profile (SV included) read from the published XML. Import: files to the library's model. Export: that model back to CGMES RDF/XML, warm, into an empty directory. Validate: files to violation report in one call, parsing included (cimoxide validates files only); cimoxide runs its own rules, triplets and OpenCGMES (with Jena SHACL) the ENTSO-E CGMES 3.0 SHACL shapes. These are timed, not graded, as in cim-bench: what each tool read and found is shown, but the tools read CGMES into different models and validate against different rules. The Java libraries run in-process through JPype, JVM included in memory.
GPU machine: GPU solvers on the same problems, cases, settings and oracle, against CPU baselines run on the same machine. That machine is not the one behind the CPU tools' results, so its tools are compared only with each other. A GPU tool's timed call ends with a device synchronize; memory is host RSS only, not device memory.