Black Swan Event of August 5th, 2024

Overview

The black swan event occurred early in the morning for most of the technical team members, around 4:30 AM UTC +3. The rapid price changes led to substantial stress on various blockchain networks and caused delays in block production, increased gas fees, and increased price updates, which in turn led to rapid depletion of sponsor wallet balances across multiple chains.

Timeline and Key Events

• T + 0 min (4:29 AM): The market began showing signs of extreme volatility with critical updates failing to appear on some networks. This was the first indication that the market was undergoing a significant stress test.
• T + 1-6 min: Some networks, including Taiko and others, were not producing new blocks, signaling potential issues with block propagation or network congestion. Balances in several sponsor wallets on various chains started deviating from expected levels.
• T + 10-18 min: As networks began to recover, the impact of the event on wallet balances became more apparent. Several sponsor wallets were identified with significantly lower balances than expected, particularly on chains like Arbitrum, Polygon, and Fraxtal. Immediate action was required to assess the situation and respond accordingly. The situation escalated to Bedirhan for funding the sponsor wallets.
• T + 28-31 min: Taiko resumed block production, but concerns remained on Arbitrum due to the rapid decrease in wallet balances. Bedirhan took control of the situation.
• T + 35-38 min: Although the feeds across networks began to stabilize, the heightened gas fees, especially on Arbitrum, led to accelerated depletion of sponsor wallets.
• T + 81-135 min: Funding actions were taken to restore wallet balances to acceptable levels. By this time, most networks had returned to normal operations, though continued monitoring was necessary to ensure the stability of feeds and wallet balances.
• T + 240 min: Some sponsor wallets on Polygon experienced critically low balances for the second time. The situation escalated to Burak, and the wallets were funded within a minute.

Response and Recovery

Our monitoring systems effectively detected and reported critical issues in real-time, allowing our team to take swift action. Alerts regarding low wallet balances and chain downtimes were promptly addressed to minimize potential disruptions.

Despite the external challenges posed by chain downtimes and gas price surges, our infrastructure demonstrated robust performance. Our systems were able to maintain accurate price updates across all chains where the conditions allowed.

The team was able to replenish all impacted wallets beyond their expected balances to ensure that our operations could continue without further interruption.

Conclusion

The August 5th, 2024 event served as a successful stress test for our infrastructure. The primary issues encountered were external and stemmed from either chain downtimes or sudden increases in gas prices. Our infrastructure performed as expected, with our systems maintaining functionality and accuracy as long as the chain was operational and the necessary funds were available.