Incident Report: Stuck Collector and Unable to Get Block Alerts on Mantle
Date: 2024-03-15
Time: 17:09 (UTC+5:30)
Duration: 1 hours 40 minutes
Description
Persisted Detected stuck collector and Unable to get block alerts on Mantle.
Root Cause
The alerts were primarily caused because the Mantle chain went down for an upgrade and the gas price changes. The RPC fluctuates between synced and not-synced nodes.
Impact
We got stuck collectors on affected chains and the RPC provider is not working properly. Around 16k+ blocks behind. The market also did not show live self-funded feeds.
Timeline
- 17:09 - First noticed the issue.
- 18:14 - Initial diagnosis.
- 18:16 - Started the fix.
- 18:36 - Issue resolved.
Lessons Learned
Whenever a chain undergoes an upgrade or downtime, it's important to have a close look at that chain once it gets back online. Making relevant updates like forwarding the collectors, checking with RPC providers, and fixing the timeouts will prevent us from potential risks.
Actions Taken
- Vekil reduced the block range for the Mantle to solve the timeouts.
- A task is created to address the gas price changes issue.
Related Images/Logs
Incident Reviewer(s)
- Abdel
- Bedirhan
- Vekil
- Ugur
- Aaron