grid-bench
GitHub repositoryOpen-source power system tools timed on the same problems, where every result is graded by an oracle that no tool under test takes part in: no speed number without a correctness number. Choose a problem; how it is set up and graded is shown with its results.
GPU tools have no thread count, so this view leaves them out.
One panel per case, same y-axis. Dashed grey: ideal scaling (speedup equal to the thread count). Filled point: the oracle verified that thread count's solution; hollow: it did not. Hover a point for the time.
As a table
Each cell counts its column's cases: ✓ / ✗ / FAILED. ✓: the oracle accepted the result. ✗: the tool returned a result the oracle rejected. FAILED: the tool raised or did not converge (see Failures). ·: the tool reads none of the column's cases, or was not asked to run them (the large cases are opt-in).
Every run that raised, with the tool's own exception.
As each tool's package reports them. A build from an untagged commit reports the last version it was given, so where a commit is listed, the commit is what ran. ·: a pinned release, or a build whose commit its results did not record.
How every problem is measured
- Solve is timed warm: one persistent model, one untimed warm-up solve, then repeated solves (median shown). Import is the median of 3 cold loads after one warm-up load.
- Oracle. Tier 1 (MATPOWER cases): the residual
V·conj(Ybus·V) − Swith Ybus built from the case's MATPOWER file by the benchmark itself. Tier 2 (CGMES fixtures): deviation from the case's published SV solution. Each problem's own check builds on these; it is described with the problem, above. - Settings. Every setting of every tool is justified in its adapter's docstring, including what it deliberately does not do.
- Memory is the peak RSS of a fresh process loading and solving the case, minus the peak after importing the tool (values below 1 MB are drawn at 1 MB on the log axis). Only the benchmark process counts: cgmes2pgm's Fuseki server is not included.
- GPU tools solve the same problems, cases and settings under the same oracle, on the machines named at the top of the page: each solver runs on the hardware it would be run on, so a comparison is of solver and hardware together. Its timed call ends with a device synchronize; its memory is host RSS only, not device memory.
AC power flow, the same problem for every tool: one slack, no reactive limits, no tap, phase-shifter or shunt control.
- Start
- Flat, on every solve.
- Tolerance
- 1e-8 p.u.
- Graded
- The residual against the original MATPOWER file; CGMES fixtures against their published SV solution.
Full method
AC power flow, the same problem for every tool: flat start on every solve, one slack, no reactive limits, no tap/phase-shifter/shunt control, generator voltage regulation as the case defines it, convergence tolerance 1e-8 p.u. MATPOWER cases are also converted to CGMES, and a tool reading a conversion is graded against the original MATPOWER file. CGMES conformity fixtures are graded against their published SV solution.
AC state estimation: weighted least squares, with the same measurements and sigmas for every tool and no bad-data handling.
- Start
- Flat.
- Tolerance
- 1e-8 on the state update.
- Graded
- One Gauss-Newton step moves the estimate by at most 1e-6, and its J is no larger than at the true state.
Full method
AC state estimation: weighted least squares on each case with a measurement set generated from its own AC power flow. exact: |V| and P/Q injections at every bus without noise, so the optimum is the true state. noisy: |V| at generator buses, P/Q injections at every bus, P/Q flows at every branch's from end, with Gaussian noise. Same measurements and sigmas for every tool, flat start, no bad-data handling. The oracle accepts an estimate when one Gauss-Newton step from it moves it by at most 1e-6 and its J is no larger than J at the true state.
AC optimal power flow: MATPOWER's AC-OPF on PGLib-OPF v23.07 cases.
- Start
- Flat, where the tool lets one be set.
- Tolerance
- 1e-6.
- Graded
- Feasible against the MATPOWER file, with a recomputed cost at most 0.01% above PGLib's reference.
Full method
AC optimal power flow: MATPOWER's AC-OPF on PGLib-OPF v23.07 cases (polynomial cost; power-flow equations; voltage, generator, branch MVA and angle-difference limits), flat start, tolerance 1e-6. The oracle checks the solution's feasibility against the MATPOWER file and recomputes its cost, accepted at most 0.01% above PGLib's published reference (a local optimum given to five digits).
AC batch power flow: each case with 100 operating points, solved in one timed call with the base case. Times are per scenario.
- Start
- Base case flat, then each scenario from the tool's base-case solution (power-grid-model: flat).
- Tolerance
- 1e-8 p.u.
- Threads
- 1, 2, 4, … up to every core with a batch API; a loop on one thread without one.
- Graded
- Every scenario of every thread count, by the residual with that scenario's demand.
Full method
AC batch power flow: each case with 100 operating points (every bus's demand along a daily curve between 60% and 100% of the case, 5% noise per bus, generators redispatched in proportion), all solved in one timed call with the base case: the base case from a flat start, then each scenario the problem of the AC power flow tab, started from the tool's own base-case solution, as a sweep from a solved base case is run (power-grid-model takes no start voltages and starts flat). Tools with a batch API run it at 1, 2, 4, … threads, up to every core available; a tool without one runs a loop of its single solve, on one thread. Times are per scenario. The oracle grades every scenario of every thread count. The GPU tools, and the CPU tools chosen to compare with them, also run 1,000 and 10,000 operating points on the larger cases, since 100 leave a GPU mostly idle; CPU tools run these, and the full N-1 cases, at one thread and at every core only.
AC N-1 contingencies: each transmission case with 200 single-branch outages that keep the grid connected, solved in one timed call with the base case. Times are per outage.
- Start
- Base case flat, then each outage from the tool's base-case solution (power-grid-model: flat).
- Tolerance
- 1e-8 p.u.
- Threads
- 1, 2, 4, … where the contingency API takes a thread count.
- Graded
- Every outage of every thread count, by the residual with that branch out.
Full method
AC N-1 contingencies: each transmission case with 200 single-branch outages that keep the grid connected (all 19 of case14), solved in one timed call with the base case, each outage started from the tool's own base-case solution (power-grid-model takes no start voltages and starts flat). Contingency APIs that take a thread count run at 1, 2, 4, … threads. Times are per outage. The oracle grades every outage of every thread count against the case with that branch out. The GPU tools, and the CPU tools chosen to compare with them, also run every outage that keeps the grid connected and converges (input "all outages", up to 14,384 on case9241pegase).
DC power flow and DC N-1: MATPOWER's DC model on the transmission cases, and on the N-1 cases' own outages.
- Start
- None: the model is linear.
- Outages
- Solved anew or from the base case's sensitivities: both are exact in DC.
- Graded
- The balance |Bθ − P| with the tool's angles and the flows against the oracle's solution, to 1e-3 MW.
Full method
DC power flow and DC N-1: MATPOWER's DC model (branch admittance 1/(x·tap), resistance and charging left out, phase shifts as injections, shunt conductance as load, one reference bus) on the transmission cases, and on the N-1 cases' own outages ("all outages": every outage that keeps the grid connected). A tool reports bus angles and from-end branch flows; the oracle checks the nodal balance |Bθ − P| with the tool's angles (whatever its reference angle) and the flows against its own solution of the model, both to 1e-3 MW. An outage may be solved anew or from the base case's sensitivities (LODF, PTDF): both are exact in DC. A tool that computes flows without angles (PTDF-based) is graded on its flows alone, marked "(flows)".
CIM import/export: CIM libraries on the Svedala and RealGrid CGMES 3.0 conformity models, every profile read from the published XML.
- Import
- Files to the library's model.
- Export
- That model back to CGMES RDF/XML, warm, into an empty directory.
- Validate
- Files to violation report in one call, parsing included.
- Graded
- No, only timed: the tools read CGMES into different models and apply different rules.
Full method
CIM import/export: CIM libraries on the Svedala and RealGrid CGMES 3.0 conformity models, every profile (SV included) read from the published XML. Import: files to the library's model. Export: that model back to CGMES RDF/XML, warm, into an empty directory. Validate: files to violation report in one call, parsing included (cimoxide validates files only); cimoxide runs its own rules, triplets and OpenCGMES (with Jena SHACL) the ENTSO-E CGMES 3.0 SHACL shapes. These are timed, not graded, as in cim-bench: what each tool read and found is shown, but the tools read CGMES into different models and validate against different rules. The Java libraries run in-process through JPype, JVM included in memory.