Skip to main content

Incident Report: Datafeed ID and DAPI Name Mismatch Alerts

Date: 2023-12-09
Time: 18:10 (GMT+3)
Duration: 16 hours 20 minutes

Description​

Unexpected alerts for Datafeed ID mismatch and DAPI name mismatch were observed, affecting multiple dapis across different chains.

Root Cause​

The root cause seems to be a misalignment between the database dataFeedId and the on-chain IDs due to updates in contracts and a delay in event collectors syncing with the blockchain.

Impact​

The mismatch caused repeated and numerous alerts, creating confusion and requiring constant monitoring and acknowledgment of these alerts.

Timeline​

  • 18:10 - Multiple alerts for DApi name mismatch and updated datafeedId noticed.
  • 18:43 - Ongoing discussion and diagnosis by team members Burak, Vekil, Mertcan Karik to identify and resolve the issue.
  • 02:20 - Andrew noticed that alerts are reopening for Mantle chain.
  • 08:41 - Ongoing efforts by Vekil to synchronize event collectors with blockchain updates.
  • 09:15 - Issues are resolved.

Lessons Learned​

The incident highlighted the importance of timely updates and synchronization between the database and blockchain data. The need for a robust monitoring system that can distinguish between actual issues and expected discrepancies post-updates was recognized.

Actions Taken​

  • Acknowledgment of alerts to reduce immediate noise.
  • Investigation into the cause of the mismatch by checking the database and on-chain contract IDs.
  • Monitoring of dataFeedId updates and alignments in the database.
  • Adjustment of event collectors to catch up with blockchain updates.

Incident Reviewer(s)​

  • Burak, Vekil, Mertcan Karik, Andrew Prasaath.
Last updated on by Warren Anderson