One of the leading real-time crypto on-chain data providers, Pyth Network, minor hiccup in the code logic created a -90% crash in BTC/USD price within a 2-min interval then bounced back thereafter. This temporary crash occurred yesterday (September 20, 2021 – 2:21 and 12:23 UTC) Pyth Network also provided pricing for Jane Street (one of the world’s largest market makers) and other banking institutionals.
The on-chain Pyth code base uses a fixed-point number representation, where each number is represented as the combination of an integer plus a number of decimal points called the exponent. For example, $52.21 might be represented as 52210 and an exponent of 10^-3. All prices submitted to BTC/USD should use the exponent 10^8.
BTC/USD price was quoted as low as $4,500 (drop of 80-90%) b/c of how the code base is designed to read integer and exponents. In essence, a price publisher submitted an integer (for the part of the code that’s suppose to read exponents) Instead of throwing an exception, the code automatically converted the floating-point number to the integer 0 and published it. The second publisher encountered a race condition between two programs that resulted in them reading an exponent of 10^0 (zero exponent = 1) for BTC/USD instead of 10^-8 for the 2-minute interval of the incident.
Similar situations has happened before in other protocols and off-chain and on-chain data providers in which the logic of the code base actually created *sometimes* irreversible situations. So it’s worth modeling out all the possible scenarios out there and run experiments against your program/code base…and sometimes, there are things that will likely only be discovered through actual market occurrence, so prepared the team to respond in a timely and transparent manner will help restore confidence in the program/code base.
The event and the likes emphasize, once again, the importance to stress-test your code base internally and externally, constantly.
Intersection notepad 