Skip to main content

Incident Report: Low balance detected for top-up wallet

Date: 2023-09-27
Time: 16:38 (GMT+5:30)
Duration: 4 hours

Description​

Unexpected low balance detected for top-up wallet '0xA1a3530a161A8F0e83D1090e1d9f45d96eB46C11' on chain arbitrum (42161) alert despite being funded enough.

Root Cause​

It seems Aaron's Airseeker sponsor wallet got funded around less than 2 hours ago. Depository contract balance seems to be below the critical low level but has 2.656396199413010301 ETH left which should be enough for a few more round of fundings. This alert is from the operations-database gas worker instead of the wallet watcher.

Impact​

Unexpected and unintended low balance alert.

Timeline​

  • 16:44 - First noticed the issue.
  • 16:50 - Initial diagnosis.
  • 17:00 - Started the fix.
  • 20:00 - Issue resolved.

Lessons Learned​

Those alerts work by comparing gas consumption of the target wallet over the last 7 days against the current balance. The balance is auto-topped up by the Merkle Funder, so probably shouldn't care about that kind of alert anymore.

Actions Taken​

Aaron added a rule to auto-close them for time being and created a task to disable that alert.

Incident Reviewer(s)​

  • Andrew
  • Aaron
  • Santiago Aceñolaza
  • Vekil
Last updated on by Warren Anderson