Incident Report: Multiple 'Unable to Get Balance' Alerts on Moonbeam, Moonriver, and Mantle
Date: 2023-12-26
Time: 22:12 (GMT+3)
Duration: 10 minutes
Description
Multiple 'Unable to get balance' alerts were reported across chains including Moonbeam, Moonriver, and Mantle. The alerts persisted for about 10 minutes. Additionally, it was noted that GRT-KSM-RSR/USD feeds on Moonbeam were not updating, although they were not part of the monitoring spreadsheet.
Root Cause
The root cause was identified as a 10-minute outage. However, it was confirmed that the GRT-KSM-RSR/USD feeds not updating was due to insufficient funding and unrelated to the outage.
Impact
The temporary outage resulted in multiple alerts across different chains, potentially affecting monitoring and alerting systems. There was no lasting impact on the chain due to utilizing other providers that did not experience the same issue.
Timeline
- 22:12 - Arda reported the issue with multiple 'Unable to get balance' alerts.
- 22:22 - Bedirhan confirmed a 10-minute outage had occurred and was resolved, and that alarms could be closed.
Lessons Learned
Maintaining multiple providers for chain data can mitigate the impact of outages. Quick identification and communication of issues are vital in managing and resolving incidents promptly. Regular funding checks for data feeds are necessary to ensure they are updating as expected.
Actions Taken
- Monitoring and reporting of the 'Unable to get balance' alerts across chains.
- Investigation into the GRT-KSM-RSR/USD feeds on Moonbeam.
- Resolution of the outage and confirmation of continued data provision through other providers.
- Communication and confirmation that the alarms could be closed post-outage.
Related Images/Logs
- Escalation link.
Incident Reviewer(s)
- Arda (Reported the initial issue and followed up on resolution)
- Bedirhan (Investigated and clarified the causes, confirming resolution)