Distributed Systems
The Water That Flowed Backwards
One line of Go made our water meters report negative consumption. The fix we added afterwards was the same bug in disguise.
You open the dashboard for a flagged flat. The consumption graph dips below zero, as if the resident had un-drunk fifty litres of water, then spikes to double the dip and carries on as if nothing happened.
A flaky meter, you think. Junk data. That is what we told ourselves, more than once, because the alternative (our own pipeline quietly corrupting good data) seemed far less likely.
What kept the hardware theory alive was the pattern. Only some flats were affected. Their neighbours sat on the same gateway, ran the same firmware and went through the same Go binary into the same table, and their graphs were perfect. That asymmetry felt like proof. It was actually the clue.
In this post
- 01Meters don't report consumption
- 02The hypothesis that survived too long
- 03Following the timestamp
- 04Careful code, wrong clock
- 05Why it worked for years
- 06Three changes, one fix
- 07The guard that brought the bug back
- 08What transfers
- Distributed Systems
- IoT
- MQTT
- Golang
- Debugging