Incident Report: Missing Value Alerts

Date: 2024-01-22
Time: 16:58 (GMT+3)
Duration: 1 hours 30 minutes

Description

An incident was reported regarding missing value alerts on several chains. Some alerts were closed, but a significant number remained open for more than 20 minutes. The issue appears to be recurring, with additional missing value alerts occurring on different chains.

Root Cause

The root cause of the missing value alerts and confirmation issues appears to be related to a server problem and potential rate limiting issues with OpsGenie.

Impact

There have been no impacts.

Timeline

• 16:58 - Missing Value Alerts on Linea have been noticed by Abdel.
• 17:17 - Arda has observed various missing value alerts in large numbers.
• 17:21 - Aaron and Mertcan have started to investigate and resolve the issue.
• 17:28 - Issue has been fixed by Mertcan.

Lessons Learned

• Missing value alerts are not chain specific.
• OpsGenie rate limiting may impact the closure of alerts.

Actions Taken

• Mertcan moved the collector to another server.
• Aaron manually closed alerts suspected to be affected by OpsGenie rate limiting.

Related Images/Logs

Escalation link.

Incident Reviewer(s)

• Abdel, Arda, Aaron, Mertcan

Similar Incident(s)

• Many Missing value alerts on Mantle, Kava, Polygon, and Blast. (Escalation link)