stummel

Click counts stopped for nine hours

Redirects worked, but no click was counted between 22:40 and 07:51. A migration had locked the clicks table.

Status
Resolved
Severity
Minor

Every redirect worked, but no click was counted for nine hours. A migration I started at 22:40 waited for a lock on clicks and held up every insert behind it. The counter queue filled, and stummel dropped clicks it could not write.

  • 2026-09-29 22:40 CEST Migration adds a column to clicks and waits for a lock.
  • 2026-09-30 07:40 Investigating: Clicks for the day stand at zero.
  • 07:46 Identified: ALTER TABLE is waiting behind a long SELECT from the stats export.
  • 07:49 The export is cancelled; the migration finishes in 2 seconds.
  • 07:51 Resolved: Clicks are counted again. The nine hours are lost.

Migrations now run with SET lock_timeout = '5s' and fail instead of waiting. Rerunning a failed migration costs a minute; losing a night of clicks costs a night of clicks.

Comments

Buy Logbuch