Skip to main content

Incident Report: Multiple shadow alerts deviation on KAVA

Date: 2024-03-18
Time: 02:02 (UTC+2)
Duration: 13 minutes

Description​

Multiple shadow deviation alerts on Kava affects managed feed ATOM/USD, BTC/USD, ETH/USD

Root Cause​

The issue was on Mertcan side and they updated it on their end and the alerts were auto-closed/fixed.

Impact​

Feeds were out of deviation for 13 minutes.

Timeline​

  • 02:02 - Aaron checked and informed that this might be because of market volatility.
  • 02:14 - Mertcan was pinged and he informed the team that the issue on his end.
  • 02:15 - Mertcan informed that they were working on updating the faliure on their end.
  • 02:16 - Issue solved and the update has done.

Lessons Learned​

As Aaron mentioned it's a bit unfortunate for us because these shadow alerts could potentially be serious, but we get so many alerts lately that it's difficult to say what's serious and what isn't. I guess it depends on the chain, so we should check and verify each alert on-chain and report immediately.

Actions Taken​

After investigating from Aaron, Mertcan was tagged to check and the issue confirmed to be on his end, then he checked and solved it.

Incident Reviewer(s)​

  • Abdel
  • Andrew
  • Aaron
  • Mertcan
Last updated on by Warren Anderson