SRLabs research suggests that security vulnerabilities remain unpatched for many Ethereum blockchain participants for extended periods of time, putting the blockchain ecosystem at risk.
Crypto currencies provide a popular alternative to centralized payment systems, and promise transactions between mutually anonymous parties, often called “trustless” transactions. More specifically, blockchain participants rely on a majority of participants taking rational actions, rather than having to rely on a single banking institution or government. However, the required rational actions seem to extend beyond what many blockchain users are willing to do. In particular, we found early evidence that blockchain participants do not sufficiently patch and hence carry known vulnerabilities.
Ethereum is a cryptocurrency realized through a peer-to-peer network. With a market capitalization in excess of USD 19 billion, Ethereum is a highly attractive target for hackers. Each participant needs a software client to access the Ethereum network: The most common choices are Parity-Ethereum (Parity) and Go-Ethereum (Geth).
Ethereum relies on high availability to prevent double spending. A hacker who controls more than 51% of the computational power in the network can double spend coins, enriching the hacker and undermining the trust in the ecosystem. If a hacker can crash a large number of nodes, controlling 51% of the network becomes easier. Hence, software crashes are a serious security concern for blockchain nodes (unlike in other pieces of software where the hacker does not usually benefit from a crash).
For that reason, denial of service vulnerabilities have a particularly high severity in cryptocurrency networks; they can be used to massively reduce the amount of computational power needed to perform a 51% attack and double-spend. Ethereum has to rely on the node software to be very hard to crash remotely. However, creating perfect software is near impossible and bugs that allow for remote crashes appear from time to time in blockchain clients.
Unpatched Parity Ethereum nodes can be remotely crashed. In February 2019, we reported a vulnerability in the Parity Ethereum client that could be used to remotely crash any Parity Ethereum node prior to version 2.2.10. The crash is caused by an integer overflow during chain synchronization between two nodes, which can be remotely triggered. Since every node accepts such connection requests to stay synchronized with the main network, the vulnerability allows an attacker to crash any unpatched Parity node active in the Ethereum network.
According to our collected data, only two thirds of nodes have been patched so far. Shortly after we reported this vulnerability, Parity released a security alert, urging participants to update their nodes.
One month after this alert, we used data from Ethernodes.org to assess the security of the Ethereum node landscape and found that around 40% of all scanned Parity Ethereum nodes, making up 15% of all scanned nodes [1], remained unpatched and thus vulnerable to the mentioned attack [2]. Another patch that was released on Mar 2, 2019 reached around 70% of Parity Ethereum nodes, still leaving another 30% outdated.
Figure 1 illustrates that the percentage of vulnerable nodes shrinks slowly over time. For Parity-Ethereum, we measured how many nodes are below version 2.2.10 (released Feb 13, 2019, security-critical), and how many are below version 2.3.9 (released Apr 2, 2019, not security critical). For comparison, we track Geth-based nodes that did not receive a consensus critical patch (below patch 1.8.21, released Dec 11, 2018). The data emphasizes that patching happens relatively slowly.
7% of active Parity Ethereum nodes have not been patched for nine months. Using the same data, we also found that 7% of the Parity Ethereum nodes announce a version, which is vulnerable to a critical consensus vulnerability that was patched on July 5, 2018.
Breaking the backbone of the Ethereum network requires crashing only a handful of nodes. Unfortunately, the data from ethernodes.org does not include whether a node is a miner. However, we know that currently the vast amount of hashing power is concentrated in a few mining pools. Mining pools often share one node to communicate with the Ethereum network, and we can safely assume that those mining pools are very security aware and keep their nodes up-to-date. While this indeed means that the threat of a collapse due to a patch gap is currently low, this centralization carries a risk as well: Breaking the backbone of the Ethereum network requires crashing only a handful of nodes, which is easily doable given a denial of service vulnerability like the one we found. Attacks that do not scale with more nodes would be prevented from breaking the network if it was more decentralized.
To resolve this situation, more reliable update mechanisms are needed. It is therefore desirable (and in line with Ethereum core beliefs) to decentralize the hashing power – this decentralization however would only increase security if the new mining nodes would still be security aware. Our research indicates that this might be too much to ask and draws a pessimistic picture for a more decentralized future. Blockchain ecosystems should look for ways to decentralize the power in their network, without running the risk of relying on vulnerable nodes to maintain the infrastructure. One way to achieve this is an automated updating mechanism, which is indeed the way Parity chose for their Ethereum client.
Even if the miner nodes are secure for now, failure to close known vulnerabilities may lead to a collapse of the blockchain ecosystem if and when the hashing power becomes more decentralized. This failure to update could leave the blockchain ecosystem in a more vulnerable state by lowering the barrier for performing a 51% attack.
The Parity Ethereum has an automated update process – but it suffers from high complexity and some updates are left out. Parity Ethereum does have an automated update mechanism, suggesting that the patch gap should be much smaller. However, the Parity Ethereum auto-update mechanism relies on multiple smart contracts on the blockchain. This blockchain reliance creates two possible problems:
First, not all users are synchronized to the blockchain and hence will not receive information about the latest updates. Before Parity can auto-update for the first time, the client needs to download large parts of the blockchain and stay synchronized to receive future updates. As a result, outdated nodes are vulnerable to attacks while they are synchronizing and might never catch up if they are used only for short periods of time. Relying on an on-chain smart contract also means that only clients connected to the canonical Ethereum chain get notifications about new updates. Clients connected to smaller, alternative chains that do not have said contracts on-chain are left vulnerable.
Second, the update infrastructure is complex: Not only must the data in the contracts be kept up-to-date, giving rise to additional overhead during a release process. But the Parity maintainers must also ensure that the data those smart contracts point to – downloadable binaries – are always available from all nodes. This makes the automatic update process error-prone.
Complicating matters further, the Parity Ethereum client only downloads updates that are considered critical when using the default settings. The threshold to be marked a critical patch appear rather high: The mentioned denial of service bug did not make it to the critical release track, although the Parity security alert urges users to install the patch.
Geth lacks an update process, by design. The other popular choice of Ethereum client, Geth, does not include an auto-update feature, which appears to be an intentional design decision. Consequently, the patch gap for Geth clients is larger than for Parity clients. According to their announced headers, around 44% of the Geth nodes visible at ethernodes.org were below version v.1.8.20, a security-critical update, released two-month before our measurement.
Lack of basic patch hygiene undermines the security of the Ethereum ecosystem. The lack of patch hygiene among Ethereum users suggests that more serious vulnerabilities might also survive for days, weeks, or months among a significant number of Ethereum users, putting their own security and the integrity of the Ethereum ecosystem at risk. The consequences of the patch gap would be most severe if a remote code execution were found in a popular client software.
Ethereum, as a peer-to-peer network, relies on rationally acting participants: The current patch gap violates this core assumption since leaving yourself and others exposed to hacking is not strictly rational.
Global trustless systems rely on software, and software has bugs. It’s our responsibility as blockchain users to immunize these large decentralized ecosystems from against infections by installing security updates as soon as they become available. This requires both more automated patching options and more patching hygiene from blockchain users.
(Whether moving from a world in which a few large banks control financial systems to one in which a few patch-signing keys wield similar power is progress, that is a topic for another day.)
For all of the technical details on the DoS-vulnerability and how we found it, check out our parallel blogpost here.
Footnotes
[1] Not all Parity nodes contribute the same computing power to the Ethereum network. In fact, the computing power appears highly concentrated among a small number of nodes. Supposing that these major nodes are well maintained, the urgency of patching minor nodes is somewhat diminished to prevent a 51% attack. However, other hacking risks grow due to the concentrated risk.
[2] The data was acquired from Ethernodes.org on Mar 21, 2019. The version of a node is identified based on its announced header. Ethernodes.org leverages the Ethereum network’s discovery protocol to scan the network and discovered almost 11.000 nodes at that time. Please note that Ethernodes.org data might miss nodes that are not constantly online. We assume that the missed nodes are less well-patched on average. Hence, the patch gap estimates are a lower bound. The real patch gap is likely larger.
