Incident Report: Excessive Updates Due to Ankr RPC Caching on Polygon

Date: 2024-12-13
Time: 03:22 AM UTC
Duration: Momentarily

Description

Excessive update transactions were observed for USDC/USD and other price feeds on Polygon network due to Ankr RPC provider caching eth_call responses or returning data from out-of-sync nodes.

Root Cause

Ankr RPC provider appears to be:

• Caching eth_call responses, or
• Returning data from out-of-sync node clients
• While eth_blockNumber and eth_sendRawTransaction continue to work correctly

Impact

• Increased number of unnecessary transactions from sponsor wallet
• Higher gas costs due to redundant updates
• Affected feeds include USDC/USD and other feeds on Polygon network
• Sponsor wallet address: 0x3BEfF42b258A2C3E87dE4fb1a1Efb49Acc928dFC

Timeline

• 03:22 AM - Issue reported with excessive updates observed
• 04:38 AM - Initial investigation and clarification of issue
• 10:32 AM - Patch development initiated
• 03:41 PM - Team discussion scheduled for solution implementation

Lessons Learned

1. Need for better monitoring of update frequencies
2. Importance of RPC provider reliability monitoring
3. Need for automated detection of abnormal update patterns

Actions Taken

• Immediate:

• Excluded Ankr provider from the Airseeker batch for Polygon

• Initiated patch development for @api3/chains

• Planned:

• Develop worker in Centurion to detect excessive updates

• Implement monitoring system for abnormal update frequencies

• Review and update RPC provider selection criteria

Related Images/Logs

• Escalation link.
• Polygonscan transaction history from sponsor wallet.
• Airseeker logs showing excessive updates.

Incident Reviewer(s)

• Bedirhan
• Warren
• Ali
• Hayrettin