Garbage in, garbage out: healing sensor data before you forecast

There is a dirty secret under every impressive forecast: the data was clean. In a demo, the meter never drops out, the sensor never sticks, the array never reports a negative kilowatt at midnight. In production, all of that happens — and a model trained or run on gappy, spiky telemetry doesn’t fail loudly. It quietly gets worse, and you blame the model.

So before Vastvic forecasts anything, it heals the data. Not with a blanket fillna(median) that flattens the very shape the forecast depends on, but with repair that respects what the number means.

You can’t out-model bad data. You can only repair it honestly — and then prove the repair.

Repair that knows it’s solar

The healing pipeline runs in tiers, each matched to a kind of damage. Physically impossible values go first — negatives clamped to zero, spikes above plausible capacity removed. Night is masked using solar geometry, because a missing reading at midnight isn’t a gap to interpolate; it’s a zero. Then gaps are filled by length: short gaps by interpolation, medium gaps by a seasonal-naive carry-forward that preserves the daily rhythm, and long gaps by a clear-sky physics model rather than a flat line. A final pass sweeps up anything left.

Reconstruction, and a conscience

For heavier damage, a more rigorous reconstruction stage resamples, drops sensors that are simply dead, and fills what remains with cubic-spline interpolation, seasonal medians and a round-robin ridge-regression imputation — a from-scratch MICE, using neighbouring signals to reconstruct a missing one. Crucially, it doesn’t just fill and hope. A self-validation step hides known-good values, reconstructs them, and measures the error — so the pipeline reports how trustworthy its own repair is, and flags when a gap was too big to fill honestly.

The same idea, across the grid

On the grid side the same philosophy runs over multi-node feature tensors: gaps repaired per node in parallel, data-quality assessed against schema, topology and capacity, with thresholds that warn on light damage and refuse on structural corruption. And all of it is dependency-light — numpy and pandas, no heavyweight scientific stack — so it deploys where the data actually lives.

---

Data healing isn’t the glamorous part of forecasting, which is exactly why it’s where accuracy is won or lost. Repair the signal in a way that respects its physics, then prove the repair against ground truth — and the model finally gets the clean data everyone else only had in the demo.