Incident Report: DAI/USD Beacon Fluctuation

Date: 2025-03-20
Time: Detected Mar 19th at 11:00 AM (GMT +03.00)
Duration: ~1 day

Description

Multiple over-updates were detected on the DAI/USD wallet for the Fraxtal chain, triggering a Nonces Worker alert in OpsGenie. Investigation by Bedirhan revealed anomalous behavior from one of the providers, Blocksize API, causing repeated updates with fluctuating values between $0.87 and $0.99. Similar irregularities were seen across other chains but with lower frequency.

Root Cause

The root cause of the anomaly was traced back to a temporary issue with Blocksize's API, particularly linked to a UniswapV3 pool on Arbitrum. This pool produced volatile price swings (lasting under one minute) resulting in unnecessary updated on DAI/USD wallet in Fraxtal. The issue was further complicated by a bug in the alerting logic, which prevented all relevant Nonces Worker alerts regarding DAI/USD wallets. Problematic beacon ID: B692F7FEC82E2CD3B32DDE3818845DAE1FAC42F9B35C99A7F3FDAD86187D9E2B (blocksize)

Impact

• Over-updates on wallets across multiple chains (Fraxtal, BOB, Blast, Base, Arbitrum, Linea, Manta, Mode, Polygon zkEVM, X Layer, OEV Network)
• Reduced trust in data integrity for affected providers. (Blocksize, dxFeed)
• Alert logic bug caused an obstacle for other feed/chain pair alerts.

Timeline

• Mar 19th, 11:00 AM - Nonces Worker alert for DAI/USD feed on Fraxtal triggered
• Mar 19th, 11:47 AM - Update counts shared for each wallet; anomaly confirmed
• Mar 19th, 11:57 AM - Provider identified as Blocksize; issue resembles past dxfeed problems
• Mar 19th, 12:08 PM - Identified data spike due to a UniswapV3 pool on Arbitrum (Explained by Blocksize)
• Mar 19th, 22:23 PM - Bug identified in alerting logic
• Mar 20th, 13:22 PM - Blocksize confirms root cause; affected pool blacklisted
• Mar 20th, 13:46 PM - Alerts closed manually

Proposed Solution

• Blacklist volatile pools that cause price anomalies
• Introduce short-term fluctuation detection logic to flag and ignore prices that deviate for less than 1-2 minutes
• Improve alerting logic to ensure all significant anomalies are caught and surfaced
• Add real-time provider-level monitoring to detect spikes across providers.

Lessons Learned

• Brief price spikes can introduce false updates that bypass monitoring
• Provider-level anomalies can significantly affect multiple chains
• Real-time analysis and redundancy in alert logic are critical
• Blacklisting problematic sources is a valid short-term mitigation

Actions Taken

• Identified and blacklisted the volatile UniswapV3 pool on Arbitrum
• Logged all provider update counts for anomaly tracking
• Fixed the bug in alerting logic
• Closed stale Nonces Worker alerts manually
• Discussed future monitoring improvements and faster anomaly response workflows

Incident Reviewers

Ali
Bedirhan
Mertcan
Umut