Observability and Bad Data

A converged estimate is not a correct one. Weighted least squares will absorb a broken sensor by bending the state toward it, and will happily return an answer for a network where half the quantities were never determined at all. Both failures look exactly like success. These two analyses are what distinguish them.

Observability: what the measurements determine

An unobservable system is not a numerical accident, and treating it as one produces the least useful diagnostic available — a factorization that failed, with no indication why. The useful answer names the unknowns nobody is watching.

There are two distinct failures.

Structural. A column of \(H\) that is identically zero: no measurement function mentions that unknown at all. The estimator has to find these anyway, since they make \(G\) singular on their own. In gridoxide they are usually the virtual slack buses synthesized per source — a network power-grid-model considers fully observable can still leave gridoxide with surplus unknowns, because PGM has no such bus in its state space.

Numerical. A column that is present but linearly dependent on the others. Two branch-flow measurements on a radial feeder with no injection measurement between them constrain the same combination of angles twice and leave another combination free. Nothing about the sparsity pattern gives this away; it takes a rank computation.

\(G = H^{T} W H\) is symmetric positive semidefinite by construction, and the standard rank-revealing factorization for that class is a Cholesky with symmetric diagonal pivoting: take the largest remaining diagonal at each step, stop when it falls below tolerance, and the number of steps is the rank. gridoxide uses faer's blocked implementation for the factorization but makes the rank decision itself — faer stops at \(\varepsilon n\), around \(10^{-16}\) relative, which is the threshold for "not numerically positive definite". Observability wants a much looser one: a direction determined only at the \(10^{-12}\) level is not usefully determined, and calling it observable would be the more misleading answer.

The cost is that \(G\) is densified, making this \(O(n^3)\) time and \(O(n^2)\) memory. The analysis refuses above DENSE_LIMIT and says so, rather than quietly allocating gigabytes on a transmission grid; a sparse rank-revealing method is the natural follow-up.

Unknowns pinned this way are held at their starting value rather than estimated, and reported. That is deliberate: an unknown nothing observes cannot be recovered, and moving it would be inventing information. It also improves the failure mode — an under-measured system now solves its observable part and names the rest, instead of failing outright.

Bad data: whether the measurements agree

Two questions, answered separately.

Is there bad data at all?

Under the assumption that each error is independent and normal with the declared \(\sigma\), the objective is chi-squared distributed:

\[ J = r^{T} W r \sim \chi^2(m - n + k) \]

with \(m\) measurements, \(n\) estimated state variables and \(k\) equality constraints — each constraint gives a degree of freedom back, and a pinned unobservable variable never consumed one. A \(J\) far out in the tail says the residuals are too large to be noise.

Rejection says something is wrong, not which measurement. A test that does not reject is also not a clean bill of health: a single moderate error, or several that partly cancel, can sit inside the threshold.

Which measurement?

The largest normalized residual. Raw residuals are not comparable — 0.1 is enormous against a \(\sigma\) of 0.001 and negligible against a \(\sigma\) of 1 — and dividing by \(\sigma\) alone is still not enough, because a redundantly measured quantity spreads its error across its neighbours and so under-shows in its own residual. The right scale is the residual's own standard deviation:

\[ r_i^{N} = \frac{\vert r_i \vert}{\sqrt{\Omega_{ii}}}, \qquad \Omega = R - H G^{-1} H^{T} \]

conventionally compared against 3. \(\Omega\) is computed through the augmented system, so the zero-injection constraints count: a constrained estimate has less freedom to move, which changes how much of an error surfaces in the residual rather than in the state.

\(\Omega_{ii}\) costs one linear solve per measurement, so the full diagonal costs more than the estimate itself. gridoxide shortlists candidates by the cheap proxy \(\vert r_i \vert / \sigma_i\) and computes \(\Omega_{ii}\) exactly for the worst 20, configurable. This is an approximation with a real failure mode, not a free shortcut: because \(\Omega_{ii}\) varies per measurement, the true worst can in principle sit outside the shortlist. Raising the limit to \(m\) removes the doubt at the corresponding cost.

A measurement whose residual has no variance at all is critical in the usual terminology — the estimate is forced to reproduce it exactly, so an error in it cannot be detected by any amount of analysis. Those are skipped rather than assigned a meaningless normalized residual.

What the fixtures show

Run over power-grid-model's own state-estimation fixtures:

The last one's worst suspect is its injection sensor at a normalized residual of exactly 100.00 — the 100-sigma conflict that fixture is built around, recovered as a number.

One caveat when reading those figures: the consistent fixtures produce \(\chi^2 \approx 0\) rather than \(\chi^2 \approx \text{dof}\), because power-grid-model generated their readings from the true state without adding noise. There is nothing for a correct estimate to disagree with. Real telemetry would sit near its degrees of freedom, and a near-zero statistic on real data would itself be suspicious — it would suggest the declared sigmas are far too large.