When Do Flow Transactions Fail? (Part II)
This dashboard is an in depth analysis on the addresses and contracts that are causing the majority of transaction failures on Flow.
Conventional Blockchain application development used to be done on Ethereum, but technological progress and the increase in users has brought with it scalability issues. Flow was born as a new generation blockchain to solve these problems.
It has a unique architecture that allows it to scale without fragmentation in a secure, fast and simple way. In other words, its speed and performance are superior, and it also maintains a friendly environment for developers.
Flow operates with the Proof of Stake (PoS) algorithm. Its network protocol is designed to scale thanks to a multifunctional architecture in which each node fulfills its own function. And with the novelty that, instead of implementing a horizontal flow, the Flow platform uses the so-called vertical flow. Its advantage is that it separates the objective (deterministic) processes from the subjective (non-deterministic) processes.
The required data to obtain the transactions success rate was obtained from the flow.core.fact_transactions table. First, a table was created with the daily amount of successful transactions by filtering by tx_succeeded = TRUE. Then, the same process was followed for failed transactions by using tx_succeeded = FALSE. Those tables were used to get the following:
-
Transaction volume = successful transactions + failed transactions
-
Success Rate = successful transactions / transaction volume
In this dashboard we will answer the following questions:
-
Transaction success rate by transaction volume?
-
Most common error message types?
-
How frequent is each error code?
-
What addresses and contracts are causing the majority of failures?
-
What types of activities and events are these addresses doing that are spamming the network?
-
What is their success ratio?
-
How profitable are these accounts?
-
What has happened with the botting issue that Solana has been facing causing outages?
-
How do they compare and how could Flow attempt to mitigate these issues?
First, let's display the transaction success rate and the transaction volume since May 9th.
There is no clear relationship between transaction volume and success rate, as the day were the success rate was the lowest (May 19th), transaction volume was not close to its highest value. Nevertheless, when the success rate is high, the volume doesn't tend to be at its peak either. This might be because an excessive amount of transactions usually come from bot arbitraging, which have many more failed transactions than sucessful transactions, resulting in a lower success rate. An example of the latter happened on May 30th.
Data can also be interpreted based on what error code might be causing these failed transactions. Thus, another approach is to check what error messages were being obtained from failed transactions since May 9th. After doing some research, the following error codes were found:
These error codes can be interpreted as:
-
1006: Invalid proposal key. This means that the public key used on a certain account does not have a valid signature. This error also generates another error code (1008), which is due to an invalid payload signature. It won't be counted, as it is generated within the same failed transaction as the error code 1006
-
1007: As the previous one, it is also an invalid proposal key error. The difference is that this time it's because the account is using a public key that is not the one that has been given to it
-
1055: The authorization failed for the account. This happens when the payer account does not have sufficient signatures (1 < 1000)
-
1101: Cadence runtime error execution failed. It is usually due to underflow, but it can happen in other scenarios like when there is a failed precondition (e.g. when no token matches an ID for sale)
-
1103: The account is using more bytes of storage than its capacity (100000 bytes). Capacity can be increased by adding FLOW tokens to the account
-
1109: Failed to deduct transaction fees. This generates the error code 1105, which happens when the total event byte size exceeds the limit (256000). It won't be counted, as it is generated within the same failed transaction as the error code 1109
-
1110: Computation exceeds limit
-
1205: Storage used is not initialized or not initialized correctly
Now that we know what error codes are there and what does each one mean, let's see if any has occurred more times than the others.
We can clearly see that when the success rate decreased in May 25th, most transactions failed due to the "Error Code 1101". The following chart shows that since May 9th, out of all the failed transactions, 91.3% of them happened because of this error.
We know the error that is being reported in most failed transactions, but are there any specific addresses causing this?
The previous chart shows the top 5 wallets that have the most amount of failed transactions with the "Error Code 1101" since May 25th:
-
'0a386460e7fe7e27': with 1886426 failed transactions
-
'9829c926abe95d7c': with 860494 failed transactions
-
'7bb1e28c69407925': with 552032 failed transactions
-
'18eb4ee6b3c026d2': with 116125 failed transactions
-
'6a587be304c1224c': with 35662 failed transactions
We can confirm that the top 3 wallets are the reason behind the recent increase in the number of failed transactions.
If we display the number of daily failed transactions for the top 3 wallets we can observe that they followed a similar pattern. They increased in number of failed transactions per day from May 25th to May 30th, and decreased since then.
We confirm that these 3 accounts have been responsible for the recent transaction success rate drop, as they account for 93.3% of all the failed transactions that occurred on the FLOW blockchain between May 25th and May 30th.
Arbitrage transaction success ratio can be measured as the amount of successful transactions divided by the total amount of transactions an account performs every day. We will use this metric to see how well each account has done.
The third account (7bb1e28c69407925) also uses the same kind of script, meaning that it is also an arbitrage bot that uses Increment Finance and BloctoSwap as dex. As most of the liquidity is in the FLOW/USDC pool, the more arbitrage performed in this pool, the more similar the price will be in Increment Finance and BloctoSwap, and the fewer opportunities there will be to obtain benefits. Even though transaction fees in the FLOW blockchain are very low, the cost of increasing the number of daily transactions adds up when sending thousands of them. That is why when the 3 accounts started doing the same thing, it started to get harder to cover the costs. On June 1st, the first two accounts decided that carrying out such a high number of transactions was not worth it anymore. That drastically improved the success ratio of the the third account, which increased by 6 times.
Cadence code is the smart contract programming language of the Flow blockchain. By analyzing a script of one of this account's (0a386460e7fe7e27) transactions, we can see that it has been doing arbitrage in FLOW, taking advantage of the low fees of this blockchain. It performs the arbitrage with a bot using Increment Finance and BloctoSwap as dex, and the FLOW/USDC pool (as it is the pool with the highest liquidity). Nevertheless, it gets so many failed transactions a day that the overall transaction success rate in the FLOW blockchain has dropped significantly.
Analyzing also a cadence script from the '9829c926abe95d7c' address, we observe that it is very similar to the one used by the previous account. The success ratio was better for this account during the first days after being created. That means that it probably used the code from the previous account as a base script and improved it.
This account was created in April 26th 2022, and it is still active. As the account is older than the other two, let's analyze if it has always been an arbitrage account or it started doing this activity recently.
This account was created in May 20th 2022, and it has been innactive since June 1st (surprisingly when its success ratio was at its highest).
This account was created in May 25th 2022, and it has been innactive since June 1st, just like the other one. As days went by, the arbitrage bot that it was using started to loose efficiency, so the account is not being used anymore. We see that it went innactive around 6 hours before the first account, so that is the reason why the latter had a spike in success ratio on June 1st.
Solana's blockchain recently received some criticism when the network went down on June 1st as it has happened multiple times so far this year. To fix this issue, Solana developers and validators agreed to a hard reset, to allow node validators to come back online. However, this is a measure that has already been implemented several times. In fact, it has been run every time the network has failed due to massive transaction volume and high demand.
According to their Twitter account, what happened was that a bug in the durable nonce transactions feature led to nondeterminism when nodes generated different results for the same block, which prevented the network from advancing.
While developers say that funds on the network were never at risk in the most recent outage, analysts criticize the creators' response. The outcome is that now more than ever people are asking Solana for a definitive solution to its network problems.
The short answer to the question of whether an outage can happen in the Flow blockchain is yes.
Solana processes an average of 2,700 transactions per second (TPS), so when millions of transactions per second flood the network, surpassing 100 Gbps of traffic at individual nodes, validators ran out of memory while trying to clear up transactions in other, abandoned blocks. If we compare that to Flow, which has a throughput of 100 TPS and around 305 nodes, it seems plausible to think that Flow is not invulnerable to outages. Nevertheless, let's explore this topic in detail.
The second in the series of the FLOW white papers formalizes the block formation process and the consensus process based on Proof of Stake in Flow. Flow adapts a variant of the HotStuff consensus algorithm for Consensus Nodes to agree on which blocks to privilege at each block height.
The specialization of nodes also allows for each component of the system to prioritize a certain segment of the scalability, security, and decentralization trilemma. Currently, Consensus, Verification, and Collection nodes are more decentralized relative to Execution nodes. Because Execution nodes are deterministic, they don’t require the same level of decentralization. In the same vein, Execution nodes require professional hardware to run, thus favoring scalability over decentralization. Also, in order to become a validator on the Flow network, a minimum amount of the native FLOW token must be staked. Nowadays, the cheapest amount to be staked is 135,000 FLOW for a Verification Node. On the other hand, Execution Nodes require 1,250,000 FLOW staked.
The team behind FLOW knows about the outages issue in Solana, and they are aware of it. Network performance and availability issues are common to arise due to Execution Node complications. Although the failure of these nodes does not threaten the network’s security, the potential of all eight Execution nodes experiencing problems could halt transaction processing. For example, Flow experienced a network outage after its most recent network upgrade when Execution nodes introduced a new execution identifier. Execution nodes are purely deterministic, but the newly implemented identifier was not deterministic which led to blocks not being sealed. The nodes required additional memory to maintain network operations. The Flow blockchain is designed to tradeoff liveness in favor of safety, so the network was halted temporarily to prevent any incorrect block sealing.
The incident response teams, however, responded diligently to the issue and provided clarity on what occurred during the network instability. The team also took steps to solve and mitigate the problem from happening in the future.
Even if Flow is not invulnerable to outages, the Flow foundation has proven to have plans to mitigate them and get things back up and running if something goes wrong. They also have one of the biggest grant programs and bug bounty programs in the blockchain world, and they encourage everyone to dig into their code to help them make it as safe as possible.
The consequences of the other two accounts going inactive on June 1st were the following:
-
The third account had no competitors to perform arbitrage, so the amount of failed transactions decreased from 134k to 10k in one day. As mentioned above, the account success ratio also got better.
-
With the two accounts with more failed transactions being inactive and the third one lowering the amount of daily failed transactions, the blockchain's success rate went back up to over 0.9 as it was before these accounts started to do arbitrage.
The above chart shows that when comparing the number of failed and successful transactions since the account was created, this account has always been doing arbitrage. Moreover, it was very sucessful doing so, until the other two accounts were created. This makes us think that given the success that this account was having, the other accounts' purpose was to profit in the same way. When they saw that there was no room for the three accounts arbitraging at the same time, they stopped and only the original account has remained active. Nevertheless, the more arbitrage performed in a pool, the more similar the price is in the different DEXs, and the fewer opportunities there are to obtain benefits. That is the reason why even though the other accounts are not active anymore, this account's performance (in terms of successful to failed transaction ratio) has diminished over time.
-
Flow Tecnical Papers: https://www.onflow.org/technical-paper
-
Enter the Flowverse (Messari Article): https://messari.io/article/enter-the-flowverse?referrer=asset:flow
-
Solana's Twitter Account: https://twitter.com/solana
Transaction success ratio is a great indicator of how well an account is performing arbitrage. Nevertheless, with transaction fees being so low in Flow, it might happen that even when the success ratio is not high, if the successful transactions are good enough, arbitraging can still be profitable. Thus, let's obtain the inflow and outflow of each account, to see if they made any money doing arbitrage since May 20th (when the account 1 was created, becoming the first competitor of account 3).
The chart above shows the profit that each account made each day. We can see that the day after account 1 was created, account 3 (that had been doing arbitrage for a long time) went from 76 FLOW in profit to -518 FLOW in one day. Account 1 did good for a couple of days, until account 2 was created. The latter managed to get 1,059 FLOW in profit on its first day. We can observe that after May 25th, account 3 stayed close to even in profits, while the other two accounts had losses that reached over 500 FLOW/day on June 1st. For that reason, accounts 1 and 2 decided to go inactive. Since then, the profits of account 3 are similar to where they were before the other two accounts were created.
The daily profit chart previously analyzed might be misleading in a daily basis, as we are counting the initial transfer to pay the fees as an inflow, and the withdrawal of the benefits as an outflow. Thus, the table above shows the total profit for each account since May 20th (total outflow - total inflow). We can see that the first two accounts have ended up with a small amount of FLOW in profit, while the third one is in one of the following scenarios:
-
It has lost a lot of FLOW doing arbitrage
-
It has witdrawn most of its FLOW to send some profits to its personal account