Skip to main content

Incident Report: Linea Public RPC Issue

Date: 2024-01-15
Time: 15:51 (GMT+3)
Duration: 27 minutes

Description​

An unexpected deviation in CAD/USD exchange rates was observed on the Centurion platform. The issue was specific to this platform and did not affect other platforms.

Root Cause​

The likely root cause is the use of a public RPC URL for Linea, which was down for approximately 30 minutes. This was initially diagnosed and confirmed by the team.

Impact​

The impact was limited to the Centurion platform, where the CAD/USD exchange rate data was inaccurate for the duration of the incident.

Timeline​

  • 15:51: Deviation in CAD/USD exchange rate on Centurion noticed by Abdel.
  • 15:53: Bedirhan identified possible issue with the Linea public RPC URL.
  • 15:55: Alternative RPC providers discussed and tested.
  • 16:18: Successfully switched to a working RPC URL.

Lessons Learned​

The incident highlighted the need for diversification in RPC providers to prevent single points of failure. The reliance on a single public RPC can lead to data inaccuracies.

Actions Taken​

  • Switched to a stable RPC provider (1rpc.io) after testing.
  • Discussed the potential use of diverse RPC sources to reduce dependency on a single provider.
  • Decided against using reblok.io in monitoring to maintain diversity in RPC sources.

Escalation link.

Incident Reviewer(s)​

  • Abdel
  • Bedirhan
  • Warren
Last updated on by Warren Anderson