Incident Report: Fantom RPC issue (Downtime)
Date: 2024-08-19
Time: 03:05 (UTC+3)
Duration: 7 hours
Description
Unexpected downtime on Fantom RPC
Root Cause
Fantom was having RPC issues. RPC urls were going up and down.
Impact
Alerts were triggering consistently.
Timeline
- 03:05 - First noticed the issue.
- 03:08 - Initial diagnosis.
- 03:59 - Started the fix.
- 10:08 - Issue resolved.
Lessons Learned
Fantom should have announced of this issue on their end but they did not and so we have encountered this issue for sometime instead of being prepared.
Actions Taken
Warren disabled Fantom. It was extremely problematic, and then enable it back once the issue resolved. also Ali fixed the RPC issues.
Related Images/Logs
Incident Reviewer(s)
- Abdel
- Ali
- Warren