The scaling issue facing innumerable blockchain projects is persisting, and current debate is evaluating the application of a technique called “sharding” to the malaise.
Particularly with the number two digital currency, Ethereum, some promising solutions have already been in application for a while now. With Ethereum, the project places an emphasis on both security and decentralization, and sharding seems a good fit to enable scaling without compromise.
What Is Sharding?
Sharding is in fact a legacy protocol, and one fairly simple in construct, although it becomes somewhat more complicated when applied to blockchains. Because of the constant and global nature of a public blockchain like Ethereum, sharding applied within blockchain tech is a sound application, yet more complex than simply sharding a database, as in legacy applications.
The process of sharding has been around for decades as an approach to scaling and managing data. The typical benefits of enterprise database optimization that sharding provides have even been prepackaged in Google’s Spanner database management offering.
Technically, sharding involves horizontally partitioning the data within a particular database. A database is thus segmented into various shards, all of which maintain the speed and accuracy of the data management required, without succumbing to the overload it would experience without sharding of incoming information. Aggregated shards together reform the complete, original database.
How Will Sharding Help Ethereum Scale?
On the Ethereum decentralized blockchain network, nodes function in a P2P environment without any central authority. Every node holds all states of the chain and processes all network transactions in most current blockchain models.
Although this generates watertight security, with the Bitcoin and Ethereum Proof of Work (PoW) protocols at play, scaling becomes a problematic issue. Ethereum is adopting a Proof of Stake consensus and slowly phasing out the PoW protocol, but scaling issues currently remain.
Proof of Stake is not a final solution to scaling issues either. On Ethereum, for example, the whole state of the blockchain’ account balances, contract code and all other data are stored in a full node. Problems are arising now that the network is increasing in size, as consensus is only increasing in a linear way. The limitation comes in because of the comms required between nodes in order to reach consensus.
No nodes in the Ethereum network have particular privileges, with each node storing and processing any and every transaction. Scaling problems typically present in the form of rising gas costs and extended timelines to confirm a transaction. The network remains only as fast as individual nodes can be, despite the increase in incoming data, and there is no resolution of scaling issues by generating more nodes.
Sharding has the potential to alleviate scaling problems as it represents a tested and innovative solution. Sharding the Ethereum blockchain would involve collating subsets of nodes to form shards. These shards in turn will only process transactions that are specific to the shard. This allows the network to deal with transactions in parallel, thereby significantly raising throughput.
The fruits of sharding are analogous to the states of America. Each state might have its own particular rules, geopolitical boundaries and specific population dynamics, but it remains a component of the United States, sharing a language and overarching set of rules and principles.
Vitalik Buterin depicted the sharding of Ethereum as imagining that the network had been “split into thousands of islands.” In his depiction, Buterin noted that every island would be able to do its own thing. Every one of the islands has “its own unique features, and everyone belonging on that island [like the accounts] can interact with each other AND they can freely indulge in all its features.”
Nodes or shards that want to get into contact with other islands, will “have to use some sort of protocol.” In a nutshell, sharding into highly efficient “pieces” enables the Ethereum network to perform as the sum of all its parts, as opposed to being tied to the speed of every individual node.
Sharding And Blockchain Technology
Ethereum developers are currently implementing something called “state sharding.” The extant state of the Ethereum blockchain is called the “global state” and it is what any user sees if they look at the network. The trick to sharding Ethereum will be to allow subsets to process nominated sets of transactions while also simultaneously updating the global network state, also making sure that validity is maintained.
The project has mooted rolling out the sharding of Ethereum in two phases, tied into the ditching of the PoW for the PoS consensus.
The first phase is the establishment of the data layer that contains the consensus around which data is contained by the shards. The second phase consists of the “state” layer. This sounds exact, yet it will all be a very fluid arena. An architecture snapshot would look like this:
Ethereum divides their network into specific shards. Every shard is then assigned their specific transaction group. These groups of transactions are derived of bundling certain accounts to form a shard. Every “transaction group” will have a header and body, as follows:
Ø This is the transaction group’s shard ID
Ø The assignment of needed validators by random sampling that will verify the shard’s transactions
Ø State Root (the state of the shard’s Merkle root before and after validating a transaction)
Ø This is all the transactions belonging to a transaction group that is a part of a specific shard.
Once a blockchain is sharded, transactions become shard-specific and happen between accounts that are native to that particular shard. As transactions get verified, the network state changes and its account balances and storage become updated.
So that the “transaction group” verifies validly, any pre-state transaction group root has to match the shard root in the global state. If there’s a match, then the transaction group is validated. The global state is updated via the specific shard ID state root.
Whereas before each block on the Ethereum chain contained only the state root, applying sharding sees each block now carrying both a state root as well as a transaction group root.
Also, a transaction group root forms the merkle root of the collected transaction groups from all of the specific shards for that particular block of transactions. Put another way, there exists a merkle root of all the various shards containing verified and updated transaction groups.
This root is also stored on the blockchain as with the now-updated state root. The use of merkle trees in this process is critical to ensure the validity of the blockchain.
Blockchain Sharding Specifics
The consensus reached in a shard is obtained employing Proof of Stake, sampling random nodes that are employed in a specific consensus round. This generates finality to consensus, something necessary within shards, and also avoids susceptibility to attack that a PoW consensus would enable.
By applying random sampling of a selection of validators within a shard, Ethereum can eliminate this kind of attack, as crooks won't know which shard they are positioned in before they are so positioned.
Cross-shard comms is achieved by applying the notion of transaction receipts. Receipts for transactions are stored in a merkle root. This can be verified easily enough, yet doesn’t form part of the state root.
Any shard that receives transactions from another shard will check the merkle root in order to make sure that the receipt has not actually been spent. Basically, receipts are stored in shared memory that can always be validated by other shards, although they can’t alter entries. The distributed storage of receipts is what allows for validating comms between shards.
Moving Forward With Sharding
Sharding as a policy is anticipated implemented after the current Casper PoS upgrade. It would appear that, as things currently stand, Casper is slated for 2019, while wholesale sharding is to follow in 2020 or even 2021.
There have been other prominent sharding interventions, most especially with Zilliqa. That project, however, doesn’t currently implement state sharding, instead focusing on enabling a high throughput blockchain, employing transaction and computational sharding.
Sharding does present some promising solutions to the persistent S-word that just won’t go away. The Bitcoin project is sampling Lightning Network’s solution to scaling that network, to good effect. Unfortunately, scaling Ethereum with sharding is a lot more complex, as Ethereum remains, in essence, a Turing complete world computer, and not simply a cryptocurrency.
Sharding can thus only work at protocol level, hence dapp developers and other users will continue without it impacting their activities as important intel. The Ethereum attempt to institute state sharding is ambitious and complex, and will be an impressive accomplishment if the project can successfully implement it.