Skip to main content

Incident Report: No Off-chain Prices in Sentinels Dashboard for Beacon Sets

Date: 2023-09-26
Time: 18:00 (GMT+5:30)
Duration: 03:30 hours:minutes

Description​

Monitoring for Beacon Sets and Monitoring for data sources dashboards does not show offchain prices for 3 hours in Sentinels dashboard with unusual deviation values.

Root Cause​

Off-chain data stopped flowing for sentinels dashboard across all feeds because of /partners_latest failure. Airseeker generates alerts based on on-chain vs off-chain data (deviation between on-chain and signed data from gateways). It also generates "shadow alerts" by comparing on-chain values to /partners_latest values. Sentinels dashboard uses that data in the SQL queries for the graphs.

Impact​

In opsgenie there are multiple shadow alerts, they were due to the API failure.

Timeline​

  • 20:13 - First noticed the issue.
  • 20:13 - Initial diagnosis.
  • 20:30 - Started the fix.
  • 20:45 - Issue resolved.

Lessons Learned​

This is because of the /partners_latest endpoint failure which Mertcan was handling and resolved it later.

Actions Taken​

Mertcan fixed the issue which mitigated the deviation alerts and backed up the dashboard.

  • Slack conversation link

Incident Reviewer(s)​

  • Andrew
  • Mertcan Karik
  • Aaron
  • Bedirhan
  • Warren Anderson
  • Santiago
  • Jiri Herzan
Last updated on by Warren Anderson