Incident Report: Manta Low Balance Alert RPC Failure
Date: 2024-08-21
Time: 09:34 (GMT+3)
Duration: 1.5 Minutes in repeated interval
Description
Critical wallet balance alert appears but description of the alert changes into "Closable" after few seconds. It keeps re-opening after closing it.
Root Cause
One of our RPC's for Manta was having an unanticipated issue. That RPC was switching inbetween other chains from time to time, therefore it was answering with wrong information from a different chain which triggers an alert.
Impact
Didn't have a major impact.
Timeline
- 09:34 - First noticed the issue.
- 09:45 - Initial diagnosis.
- 09:59 - Started the fix.
- Next day - Issue resolved.
Lessons Learned
There is a bug with some of the RPCs which cause them switching inbetween chains.
Actions Taken
From now on, for each call from the RPC there will be an indicator of blocknumber and chainid of that call.
Related Images/Logs
Incident Reviewer(s)
- Abdel
- Arda
- Andrew
- Ali
- Warren