Incident Report: Polygon RPC Data Inconsistency

Date: 2024-03-08
Time: Detected Mar 7th at 4:52 PM
Duration: 5 hours

Description

Multiple Polygon RPCs, specifically from polygon-rpc.com and rpc.ankr.com, were discovered to be returning outdated data while presenting a current block number. This discrepancy led to false positives and incorrect data being displayed on Centurion.

Root Cause

Polygon RPC providers were intermittently returning stale historical data instead of the most recent blockchain information. The critical issue stemmed from an anti-pattern in RPC provider infrastructure: aggressive caching mechanisms designed to improve response times at the expense of data accuracy. These providers were essentially caching block numbers and data, then serving cached responses to improve performance. This approach creates a critical reliability problem where:

• Cached data becomes outdated
• Block numbers are artificially updated to appear current
• The system presents stale information as fresh and accurate

The caching strategy, while potentially reducing computational load, introduces a severe data integrity risk. By prioritizing speed over real-time accuracy, these RPC providers create a dangerous scenario where applications can inadvertently process and act upon obsolete blockchain information.

Impact

• False positive alerts generated
• Incorrect data displayed on Centurion
• Risk of decision-making based on outdated information

Timeline

• Mar 7th, 4:52 PM - Initial observation of data inconsistency
• Mar 8th, 12:37 AM - Confirmed issue with specific RPC providers
• Mar 8th - Investigation and proposed solution developed

Proposed Solution

Implement a data sorting mechanism using dapiTimestamp from Api3MarketV2 and AirseekerRegistry contract calls for:

• pricesWorker
• walletsWorker

This approach will allow for:

• Verifying data recency
• Filtering out outdated blockchain information
• Ensuring only the most up-to-date data is processed

Lessons Learned

1. RPC providers can return inconsistent data
2. Block numbers alone are not sufficient to validate data freshness
3. Additional timestamp verification is crucial for data integrity

Actions Taken

• Identified problematic RPC providers
• Developed solution to re-sort RPCs using timestamp data
• Proposed implementation of timestamp-based data validation

Incident Reviewers

• Ali
• Warren
• Arda
• Umut