Skip to main content

Incident Report: Missing Nodary Value Alerts on All Chains

Date: 2023-11-28
Time: 18:52 (GMT+5:30)
Duration: 3 hours

Description​

There were multiple Missing Nodary Value on all chains (Kava, Gnosis, Polygon-zkevm, Arbitrum, Polygon Mainnet, Moobeam, Linea, Mantle)

Root Cause​

This is due to a naming mismatch. FYI: Telemetry Airseeker gets extra reference values from Nodary for all feeds - it does this by matching up the feeds based on their names. Nodary uses a slightly different naming to what the ops-db uses. The result is that Airseeker Telemetry can't find the values. The impact is that some graphs in the Sentinels system won't be populated with reference data

Impact​

Unexpected and unnecessary panic as this endpoint was falsely named as Nodary.

Timeline​

  • 18:52 - First noticed the issue.
  • 18:53 - Initial diagnosis.
  • 19:15 - Started the fix.
  • 20:30 - Issue resolved.

Lessons Learned​

Naming of the endpoints and alerting messages on Opsgenie should be athentic and reliable to avoid unnecessary excitments.

Actions Taken​

Aaron created the issue to update Opsgenie message and Mertcan Karik returned the APIs to their correct names.

Incident Reviewer(s)​

  • Aaron
  • Mertcan Karik
  • Bedirhan Uzun
Last updated on by Warren Anderson