Skip to main content

Incident Report: Delay in Mantle Pricing Updates Despite Sufficient Funds

Date: 2024-01-03
Time: 12:39 (GMT+3)
Duration: Since 30th December

Description​

Alerts were received regarding Mantle pricing not being updated since the 30th of December despite there being enough funds. The on-chain value was updating correctly, but these updates were not reflected in the market GUI or the dashboard.

Root Cause​

The root cause was identified as a lag in the event collectors syncing with the chain, causing a delay in the updates being reflected on the dashboard. The chain was moving rapidly, and the system couldn't keep up with the pace, leading to a backlog of 5 million blocks.

Impact​

The delay in pricing updates led to confusion and potentially impacted decisions based on Mantle pricing data. Although the on-chain value was accurate, the discrepancy in the dashboard representation caused concern.

Timeline​

  • 12:39 - Abdel reported the issue with delayed updates for Mantle pricing.
  • 12:52 - Team members, including Abdel, Ali, Aaron, and Vekil, contributed to identifying and resolving the issue.
  • 13:37 - Aaron performed a resync and increased the RPC max blocks per cycle to 40k to catch up with the backlog.

Lessons Learned​

Continuous monitoring and rapid response to discrepancies in data feeds are crucial. Regularly adjusting system parameters and capabilities to handle rapid chain movements can prevent backlogs and ensure data is up-to-date. Utilizing multiple data sources for verification helps in identifying discrepancies early.

Actions Taken​

  1. Identification and reporting of the delayed updates in Mantle pricing.
  2. Confirmation of on-chain value accuracy and discrepancy in dashboard updates.
  3. Resync of event collectors and investigation into the cause of the delay.
  4. Increase of RPC max blocks per cycle to 40k to expedite catching up with the chain.

Incident Reviewer(s)​

  • Abdel (Reported and followed up on the issue)
  • Ali (Assisted in clarifying the on-chain values and coordination)
  • Aaron (Investigated and implemented the solution)
  • Vekil (Provided insights on the current state of block collection)
Last updated on by Warren Anderson