• A slight 25% improvements in chain tip performance

• Slightly smaller disk footprint

• A 50% improvement to Caplin’s chain tip performance

• A lot of bug fixing

And, as we mentioned before, we developed a new blazingly fast syncing algorithm to spin up archive nodes in mere hours.

Introducing OtterSync: The fastest sync algorithm for Ethereum nodes ever created.

OtterSync is a new sync protocol/algorithm designed to drastically reduce the time required to sync Ethereum nodes (both full and archive nodes). Leveraging Erigon’s 3.0 static files model, it allows 98% of the computation to be shifted from the CPU to network bandwidth for initial sync.

In Erigon 3.0, most of the state is designed to be downloadable through a secondary protocol (such as BitTorrent) rather than being retrieved through execution. However, the existing method of serving BitTorrent data was not quickly adaptable and required a release each time to allow for a new set of data to be available for download. This was primarily due to checksum checks and the need to maintain some degree of decentralization.

1. Dynamic Data Availability: By refining the way data is distributed and checked, OtterSync eliminates the need for frequent releases, enabling faster and more flexible data availability.

2. Mantained Decentralization: Despite the improvements in data distribution, OtterSync maintains an high degree of decentralization, ensuring data integrity and security.

3. Integrity: OtterSync does integrity checks on the data downloaded to assure that users do not end up on neither a malicious nor wrong chain.

In other words, OtterSync is an extension and major improvement over the previous Snapshot downloader that Erigon 3.0 and Erigon 2.0 used to rely on.
But how does it actually perform? How good is it?

For reference, on these same machines Geth and Reth takes respectively ~12 hours and ~3.5 days to sync a Full Node on Ethereum Mainnet.

From the new alpha release onwards, OtterSync will be the default sync on Erigon 3.0, and you can start playing with it with Alpha 2.

Additionally, this is only the First iteration of OtterSync as we are striving (and also have a path forward) to push the boundary of what is possible even more.

Alpha 2: Secondary improvements - Chain Tip performance, Disk footprint reductions

There were minor improvements in Chain Tip performance across the board, mostly in preparation for a significant optimization expected in the next alpha release. Users can expect an improvement of about 22% in block importing times compared to alpha 1.0. Consequently, this also allowed us to consistently stay in sync with Polygon on Erigon 3.0 for the first time. Moreover, we managed to halve Caplin’s block importing time with this release, earning us significant time savings on our internal Consensus Layer as well.

Regarding disk footprint, we managed to reduce it by roughly 2% by fine-tuning our underlying database. Similarly to chain tip sync times, we are also expecting a significant improvement in the near future in terms of disk footprint.

Aside from all of the above, we have been fixing many bugs that emerged with the first alpha release, and we look forward to fixing even more with the upcoming second alpha release.

At the current state, we encourage users to run Erigon 3.0 alpha nodes on powerful machines as chain tip performance is not yet optimal.

Issue with historical data on the Beacon chain.

Modern historical data management on the Ethereum beacon chain has encountered several challenges, making operations cumbersome and inefficient for archival nodes. These challenges primarily revolve around high input/output (I/O) operations, latency, and the sheer volume and complexity of the data, with disk sizes for archival nodes often exceeding 4 terabytes (TB). Let's delve into these issues to understand their impact:

1. High I/O Operations: Archival nodes store a complete history of all states and blocks on the Ethereum network. This requires frequent reading and writing operations to the disk, leading to high I/O throughput. The intensive I/O demands can slow down the performance of archival nodes, especially when processing or validating new transactions and blocks. This is because the system's ability to read from and write to the disk efficiently is a critical factor in overall performance. When these operations become a bottleneck, it directly impacts the speed and efficiency of the network.

2. Latency: The latency in accessing data from disk storage is significantly higher than accessing the same data from memory (RAM). As archival nodes deal with vast amounts of data, they rely heavily on disk storage. This reliance results in increased latency when accessing historical data, which can slow down query responses and data processing.

3. Disk Sizes Exceeding 4 TB: The Ethereum blockchain continuously grows, adding new blocks approximately every 12 seconds. This constant growth leads to an exponential increase in the data that archival nodes must store. Modern archive nodes often see their disk sizes exceed 4 TB, which increases hardware costs and complicates data management and retrieval. Large disk sizes mean that more data must be scanned for each query, which can further increase latency and I/O operations, exacerbating the existing challenges.

Right now, the two main (that we know of) contenders are trying to fix this problem:

• Lighthouse’s experimental tree states data model.

• Caplin’s data model.

Therefore, to give a better picture of it. We will present some benchmarks by putting them against one another, comparing the memory consumption of the two nodes under stress, latency of retrieved data, and disk size.

Disk Footprint: Lighthouse Tree-States VS Caplin.

First of all, Lighthouse Tree-states and Caplin have similar archival sizes (up to 8.4 Million slots)

.

Caplin is a little smaller, with 145 GB of additional disk footprint required, while Lighthouse stands at 180 GB. They are roughly the same in terms of sync speed and database population speed. The main difference between the two, when it comes to the data model, is that Lighthouse tree-states keep track of what essentially is a diff-tree of each instance of the state throughout history using xdelta3 (a delta-compression algorithm), while Caplin uses internal indexing and pre-loading to keep track of the changes as opposed to the tree-states approach. This leads to a slightly smaller disk footprint. However, this difference in approach also leads to faster data processing and extraction times. So, how do they differ in application latency?

Latency: Lighthouse Tree-States VS Caplin.

To measure latency, we decided to compare the timings on the Restfull API from both Beacon nodes. We selected three endpoints to test, which we believe to touch all the spots regarding data processing. The first and most important for reorg handling is how fast each client can read a previous state. We benchmarked the /eth/v2/debug/beacon/states/{state_id} in octet-stream to do this. Below are the results:

We can see that caplin sits all the time below the one-second mark while lighthouse-tree states go as far as to exceed 2 seconds at times. Making Caplin approximately 2x times faster at reading historical states.

We also ran two other benchmarks on how fast historical individual (or even multiple) validator data and rewards are retrieved. Below are the results for retrieving individual validator data.

We can see that Caplin is six times faster at retrieving this kind of data by showing a slow growth curve over time, while Lighthouse seems to pick up latency as we approach the most recent slot data. Lastly, a measurement for validators deltas tracking is shown:

We also picked this endpoint to test computation and processing on pre-existing beacon data. The results show that Caplin is ten times faster than a lighthouse regarding historical data processing.

Network improvements

To add a final note: in the past two weeks, the Caplin team finally fixed the recurring issue of Caplin occasionally missing a few gossip blocks. The issue was that we completely messed up the size of a P2P Message. Luckily, that does not happen anymore, and other functionalities, such as being a Validator, are in reach.

What’s next?

From the next release, Caplin’s archive node will be able to be enabled using the flags —-internalcl and —-caplin.archive, these two flags will enable backfilling and reconstruction of states. The next step is to implement the full-validator functionality and work on bug fixing and improving current things. The Caplin team also starts researching efficient means of blob retrieval and storage for Layer 2s.

Motivation

Caplin was born out of the need to simplify the complex technical details that arose from "The Merge." Dealing with various client setups had become a headache, causing many issues. Additionally, changing the Engine API is a tough and time-consuming process. When new protocol concepts come in that require API changes, there's a long and strict validation process. This whole situation highlights the requirement for a smoother approach, which is what Caplin tries to achieve. As we explore this further, it's clear that Caplin's development journey aimed to make things more efficient and compatible in the evolving Ethereum environment.

Achievable Benefits

Caplin brings about advantages that can be divided into three main groups: Soft, Hard, and Protocol benefits. Soft benefits are the subtle perks that might not catch users' attention immediately. They include clearer code, easier codebase maintenance, and faster block import speeds. On the flip side, Hard benefits are more concrete and noticeable. They directly improve user experience, simplify infrastructure, make better use of data, and even have the potential to enhance Execution Layer (EL) pruning capabilities. These benefits are all quantifiable and enhance the overall usability.

Now, the Protocol benefits are the big-picture gains. These benefits extend beyond immediate user interactions and have a measurable impact on the Ethereum protocol as a whole. Caplin's design allows for introducing features that aren't available through the Engine API. This means protocol improvements can be made without needing all the other Consensus and Execution clients to agree. This approach accelerates protocol enhancements without getting stuck in prolonged consensus and approval processes. As Protocol benefits stack up, they enhance client performance, make the Ethereum ecosystem more adaptable and innovative, and introduce valuable optimizations.

Furthermore, an additional advantage lies in the potential to deliver an optimized Ethereum archive node (150GB< estimated) and beacon API (faster retrieval of data), which only Lighthouse and Nimbus are currently making an attempt at. However, it's worth noting that while this capability exists, it currently isn't a high priority due to limited demand for the beacon API, given its current structure. Designing a new API presents challenges and risks regarding effort and time investment. This pragmatic approach underscores the importance of aligning development efforts with tangible user needs and optimizing resource allocation for maximum impact.

Caplin Roadmap

Regarding the roadmap, Caplin's evolution is divided into four distinct phases, each signifying a critical milestone in its development and integration:

Phase 1: Engine API Integration In the initial phase, Caplin interfaces with the Engine API. This step enables Caplin to harness the comprehensive capabilities and functions provided by the Engine API. Notably, this integration also extends its utility to other Ethereum clients such as Geth, Besu, and Nethermind.

Phase 2: Internal Erigon API Integration Transitioning to the second phase, Caplin utilizes an internal Erigon API. This API is also crafted to integrate other chains other than Ethereum 2.0 (such as Polygon). Notably, this capability is a central point of the Caplin project.

Phase 3: Distributed Validator and Validation Client Progressing to the third phase, Caplin integrates the validator REST API capable of fulfilling the role of a proposing client.

Phase 4: Archive Node Capabilities Culminating in the fourth phase, Caplin extends its functionalities to include archive node capabilities.

Currently positioned in Phase 2 of development, our progress aligns with the integration milestone that involves utilizing the internal Erigon API. As we march forward, we anticipate making Caplin accessible to the public by the end of the year. This progressive step reflects our commitment to streamlining the operation of Erigon through Caplin's capabilities, serving as the default mode unless specifically indicated otherwise. Our strategic approach ensures the alignment of Erigon's functionality with Caplin's core principles and advancements, ultimately contributing to a more seamless and efficient Ethereum ecosystem.

Benchmarks

Below, we outline some Technical benchmarks which are subject to change as they are not that optimal yet but below are our results in block import time and Disk Footprint.

Disk Footprint

So, the disk footprint comprises 222 MB of indices and 16 GB of beacon blocks stored to serve the P2P RPC. There is still some room for improvement here in this regard, so this is not definite (it is quite a dumb approach so far). There is to note that the 16 GB worth of beacon blocks are stored as blobs, so one can choose to dump them on an HDD, thus practically reducing their cost to 0.

Block time importing

Below is the block importing time on a machine with an AMD Ryzen 9 5950X 16-Core Processor, which is a quite powerful but still affordable machine compared with Prysm:

Note: This is for block importing for Block processing in addition to ForkChoice computation. Additionally, Prysm and Caplin do different things; although they fully implement the consensus specs and fork choice rules, Prysm is a far bigger and more complete software. Above are just the rough numbers for raw optimistic (BLS verification only) block processing at the chain tip (so we exclude EL overhead).

Memory Consumption

Below are memory consumption benchmarks; I think that for non-validating nodes, this metric can very well be halved (also, this is out of a very dumb and unoptimized implementation of the forkchoice rule, just a more decent copy-cat of the consensus specification). I do not want to give definite answers here. It pretty much caps at 3.8 GB and never goes up, giving it a requirement of 4 GB of RAM to run Caplin.

Below is Caplin's Memory usage compared to Prysm:

Prysm (approximately 11 GB):

Caplin (Approximately 4 GB):

What is Caplin specifically?

Caplin is the codename for the Consensus Layer client, which will be embedded into Erigon. Caplin will be a full-fledged validating Consensus Client like Prysm, Lighthouse, Teku, Nimbus and Lodestar. Our goal is to make it able to: provide better stability, validation of the chain, stay in sync, keep the execution of blocks on chain tip, provide Beacon API, compact data model, and low CPU and memory usage. The main reason why we are developing a new Consensus Layer is to experiment with the possible benefits that could come with it. We can already envision some significant benefits that could be derived, but we have yet to discover all of them. Some of the most important benefits are that Erigon was not designed around Engine API, the beacon chain archive node is 3 TB, and there is less protocol complexity for Ethereum Mainnet. For example, The Engine API does not work well with Erigon. The Engine API sends data one block at a time, which does not suit how Erigon works. Erigon is designed to handle many blocks simultaneously and needs to sort and process data efficiently. Therefore, it would be better for Erigon to handle the blocks independently instead of relying on the Engine API.

What is Caplin Roadmap?

As of now, we envisioned 4 phases for Caplin development:

• Phase 1: Embedded communication through Engine API (Finished)

• Phase 2: Embedded communication with Erigon custom protocol

• Phase 3: Functioning Beacon API (and support for running validators)

• Phase 4: Compact data model using Erigon 4 compaction.

As of now, Phase 1 is finished and will be released in the next release and will replace the Lightclient and will need to be enabled through the --internalcl flag in Erigon CLI (it is not default anymore due to the alpha state of Caplin during this first iteration). We will try to stabilise it as much as possible. Only after we believe it is stable enough, we will move on to Phase 2, where the Protocol simplifications (and most of the benefits) will be able to come to fruition (Noting again that we do not even know the full extent and effects of the benefit that can result from this experiment). Phase 3 and Phase 4 are extras that may or may not be implemented.

What is the current state of the art of Caplin?

Currently, the development of Caplin is at the point where we have a working validating client which implements Ethereum 2.0 fork choice and that can communicate with the Erigon Execution Layer both in separate and embedded modes. Currently, there is no persistent storage in Caplin, and only the last 128 blocks are kept in memory for the sake of chain reorgs, so at the time of writing, the hardware requirement for Caplin are as follows:

It can sync the chain and stay on chain tip through checkpoint sync, nothing more, nothing less; it will behave as the embedded consensus Lightclient which will soon be removed. You can use it right away from the next release with --internalcl if you still want to avoid using a Consensus Layer and want to try it out and help us test it. 

More on sunsetting on Consensus Lightclient

Although the Consensus Layer Lightclient embedded into Erigon has proven helpful, it is time to sunset him in favour of Caplin. Despite having better memory requirements than Caplin, the light client network in Ethereum 2.0 is supported by insufficient LightClient servers, which has become a minority in the network as Erigon’s consensus’s Lightclient has started to be used. In other words, finding peers took a lot of time and went from being very helpful to a maintainability nightmare. Additionally, since Caplin and the Erigon embedded lightclient are maintained by only one person, we determined that it was best to focus all our efforts on one thing instead of maintaining both simultaneously.

Our initial hope was that most of the code maintenance would be done by external contributors, which was partially the case.

However, the demand to our time from the support of these chains, has increased considerably. There seem to have been regular regressions that were difficult to diagnose because we do not ourselves maintain fully-synced Erigon nodes for BNB Smart Chain and Bor. Still, when time allowed it, I tried personally to debug the issues (mostly about sync being stuck in headers and bodies stage), with some success, but at as cost of somewhat “frantic” release schedule. At the same time, continuing developments in the go-ethereum-derived versions (which are so far dominant and “officially” supported) seem to diverge from our roadmap. We expect this divergence to further increase our burden, unless properly integrated into our roadmap. Despite issues, so far we had an impression that there is still a strong external interest in Erigon to continue to support these large chains.

We do definitely expect Erigon 3 (as specified in our roadmap) to bring significant benefits to our Ethereum and Gnosis Chain implementations (both in terms of reduction of sync time and disk footprint), but benefits would be even greater for BNB Smart Chain and Bor.

We feel it is unfair for us to spend ever increasing resources on continuing support of these platforms while only being funded by Ethereum Foundation and Gnosis DAO. Over the past few months, we had conversations with some representatives of BNB Smart Chain and Polygon about the need for continuing funding of our efforts, if the support is to continue. However, as it stands, our feeling also is that the expectation is (as it was traditionally the case) that our team needs to “apply” for funding, providing some extra commitments beyond what we already do (or trying to do), and then navigate the internal processes of the respective organisations to get the funding. As a result, we would also over-commit in the process of negotiating the level of funding. As unusual as it may sound, we would like to reverse this “process”.

We would like BNB Smart Chain and Polygon representatives to “apply” to Erigon for continuing the support of the respective chain, which we will agree to, if sufficient ongoing funding is provided - as simple as that. If this does not happen, we aim at removing the respective code from Erigon shortly after 1st of April 2023, and therefore discontinue support. It is, of course, possible, for anyone to create a fork of Erigon at that point and continue support “on the side”, but without our team being held responsible for the results.

Important note: representatives mentioned above have been given advance view of this post, and also our terms of funding. So far, BNB Smart Chain replied that unfortunately, they are not able to provide funding on these terms.

We feel that current situation is untenable for us, so we need a proper resolution, and we are ready for either outcome. We either drop support and concentrate our efforts on the platforms where our funding comes from, or, having received appropriate funding, incorporate support into our roadmap, without over-committing in the attempt to gain funding using the currently expected process.

Finally, we would like to clarify, that this proposal to “apply” only extends to BNB Smart Chain and Polygon, because Erigon already has support for these platforms and there are apparently a lot of users who would not want this to be discontinued. In other words, we are not planning to add support for any more platforms. Instead, our plan is to consolidate development and ensure we have sustainable runway for accomplishing our complete roadmap. Short version of the roadmap for quick reference:

Here are the draft release notes:

In particular, the logging output should now be less confusing.

In the previous release there was an unfortunate flaw that meant that the block snapshot generation was no longer going on, and therefore the blocks would accumulate in MDBX without being converted into snapshots files and cleaned up from the MBDX. The newest release fixes it.

For some users who previously ran one of the flawed version in devel branch (that flaw never got to the released versions), there would be automatic database migration performed, to re-alight the transaction enumeration. When this migration is performed, the block headers and block bodies are truncated up the most recent snapshot, and re-downloaded again, to be inserted with properly aligned numbering.

In the next release, we will update the BitTorrent info hashes for the new block snapshots (doing it now may cause problems with downloading them, as most of the node would not have them generated and not able to seed).

Renaming

To simplify the communication about the planned incompatible upgrades of Erigon, described in the previous post, we decided to rename these upgrades.

From now on, we will call the currently released version Erigon 2 (instead of Erigon 2.1). Since it required full re-sync to go from Erigon 1, it makes sense.

What we previously called Erigon 2.2, will now be called Erigon 3. This upgrade introduces new way of storing and distributing history, as well as increased granularity of history. As mentioned before, it is currently being integrated into the code base. Upgrade from Erigon 2 to Erigon 3 will require full resync, as database layouts are incompatible.

What we previously called Erigon 2.3, will now be called Erigon 4. It introduces new way of storing and distributing state and state commitment. As mentioned before, it is currently being prototyped. Upgrade from Erigon 3 to Erigon 4 will require full resync, though we expect this to be much quicker than before (that is one of the objectives of Erigon 4 after all, to make full resync quicker).

Release versioning

We are also planning to move from calendar-based versioning of our releases to semantic versioning, using 2, 3, and 4 as major versions. This will enable us to ensure smother transition from Erigon 2 to Erigon 3 supporting both versions for some time, allowing users to migrate. However, this will be done a bit later, so not to do all the changes at once.

Merge

On the 15th of September 2022, Ethereum transitioned to Proof Of Stake (POS) by merging two of its networks/chains, and from then on, their blocks are synchronised. What was known as “beacon chain” or “Ethereum 2” (and now is called “Consensus Layer”, or CL) is allocating block slots, which are filled by the blocks produced by what was known “Ethereum 1” (and now is called “Execution Layer” or EL). You can see this now by inspecting any POS block in a block explorer:

While previously, blocks had 1 numerical identifier, called “Block number”, or “Block height”, now each block will have three: block height, slot number, and epoch number. Epochs are relevant when we talk about finalisation, as Ethereum POS finalises blocks in whole epochs.

Although the merge event has been seen as very successful, we see quite a few user issues at the moment, which is somewhat understandable, because the complexity of the system running a node did go up: two pieces of software are required to run a node instead of one, there are multiple combinations of these pieces, and these pieces have their own “sync” lifecycles.

Given the above, some of us think that there will be a need for embedded CL at some point. Of course, as far as Erigon architecture is concerned, such embedding would also be separable, working via GRPC. Perhaps a standard GRPC specification will emerge for EL←→ CL communication.

Alpha and beta designations

From the release which will be published shortly after this article (2022.09.02 - this is not a date but second release of September 2022), we are dropping alpha and beta designations from our releases. It did make sense in the past, as we were experimenting with a few approaches to managing the code. But recently we decided that designating our releases as alpha and beta does not serve us very well. Firstly, to the best of our knowledge, no other Ethereum implementations use such designations, and the users assume that all other implementations are therefore “production ready”. What “production ready” actually means is relatively subjective. But there were also instances where our alpha designation was used to dissuade users from using our software. We would like node operators to use Erigon if they find it useful, if it provides them with the functionality or cost efficiency that other implementations are not providing. As will be described in the section below, there will be big changes in Erigon for a while, as we are gradually rolling out Erigon 2, which we believe will make Erigon competitive with other implementations according to most criteria. For now, it is up to the users to make their choices.

Erigon 2 progress

As described in one of the previous posts, the plan was for Erigon 2 to include three upgrades, which we now call Erigon 2.1, Erigon 2.2, and Erigon 2.3. It is still the plan. We can say that Erigon 2.1 has been shipped, it is now part of the all releases. There are still issues once in a while, but for most users, the functionality works. To recap, Erigon 2.1 has brought:

• Infrastructure for using BitTorrent to download and seed block snapshots.

• Block snapshots contain information about block headers, block bodies (including transaction payloads), and pre-computed transaction senders.

• Compression used inside snapshot files (mostly for transaction payloads) reduces the total disk footprint of a node.

• Indexing based on minimal perfect hash tables for looking up transactions by their hashes further reduces the total disk footprint of a node.

Erigon 2.2 is currently out of the prototype phase and is being integrated into the code base. It is not developed in a separate branch, but instead is "hiding” behind an experimental command line flag. Here is what Erigon 2.2 will be bringing:

• In addition to block snapshots, all historical data will be provided as snapshots, delivered and seeded via BitTorrent.

• State (accounts, contract storage, byte code of contract) will still be only in the MDBX and will not be delivered via BitTorrent (this should be happening in Erigon 2.3).

• History of state will become more granular. Instead of being able to query the state of any account, or contract storage item, or byte code “as of” certain block height, it will be possible to query these things “as of” before any transaction in any block.

• Transaction-level granularity of history will improve performance of queries like debug_traceTransaction, trace_transaction, but more importantly, of trace_filter, which currently has to re-execute a lot of irrelevant transactions to advance the state from beginning of a block.

• Improved performance of trace_filter may lead to block explorers like Otterscan being able to offer smoother user experience when displaying list of transactions for certain account.

• Slightly experimental but promising feature - state reconstitution. Given downloaded history in the form of snapshot files, it is possible to reconstitute recent state by re-executing only part of the transactions occurred since Genesis. Such reconstitution may also exploit multiple CPU cores. So far experiments on relatively powerful machine (16 cores, 128 Gb RAM) showed reconstitution for Ethereum main net up to 15m block within 11,5 hours. Experiment is now prepared for BSC (but it takes a long time to generate history snapshot files).

There is still a lot of work to be done for Erigon 2.2, including, but not limited to proper parallelism of snapshot creating, and rewriting most of the RPC methods to support new state history layout.

Erigon 2.3 is still in the prototype phase (and will likely stay in the prototype until Erigon 2.2 is rolled out), though a lot of progress has been made here too. According to the current vision, Erigon 2.3 will bring:

• State of accounts, contract storage items, and contract byte code will be split into parts stored as snapshot files and MDBX (that part will contain recent updates).

• The separation into snapshot files and MDBX is similar to the idea used in LSM (Log Structured Merge) databases, such as LevelDB, RocksDB etc. Recently updated entries live in “Level 0” (which is usually in-memory skip list, analogous to MDBX in Erigon 2.3). As they get “more mature”, they migrate to “Level 1” (bunch of files), and then to “Level 2” (larger files), etc.

• The main difference between LSM organisation and Erigon 2.3 is that snapshots files are produced in deterministic way, and therefore can be shared around the network in the form of BitTorrent swarms.

• There will be further decoupling of “commitment” from the “state”, with the code designed to easily support multiple commitments simultaneously. This will be quite useful for easier support of Verkle tree commitments, and also for alternative commitment schemes, used in StarkNet, for instance.

After Erigon 2.3 upgrade, we see the Erigon archive node being mostly bootstrapped via snapshot files, and then relatively quickly catch up to the more recent state. Also, the size of MDBX database file will be quite limited, and perhaps fitting entirely in RAM on relatively powerful hardware.

Firstly, the architecture diagram:

In the diagram above, and also in the one below, a somewhat traditional colour coding is used - red for things that have not been started yet, amber (orange) - for things that are in progress but not finished yet, and green - for things that are done (but will be improved over time, of course).

The amber box with “ETH core” corresponds to the bulk of the silkworm project in this repository. Another amber box, “ETH RPC api(s)” corresponds to the silkrpc project in this repository. SilkRPC project uses Silkworm as a library, in order to run EVM inside such RPC methods as eth_call.

Broadly, silkworm, as being developed, is currently compatible with Erigon’s ETH Sentry component, and silkRPC is compatible with Erigon’s ETH core and Erigon’s TxPool components. Also, silkworm maintains database-level compatibility with Erigon beta (the version which will only work on Ethereum main net until Ethereum POS transition, and then superseded by Erigon alpha).

SilkRPC has a lot of functionality that Erigon’s RPC daemon has, but in comparison has many more tests, including performance benchmarks.

Below is the second diagram which zooms specifically into silkworm:

Initially, the development of silkworm was focused on creating and testing individual stages in separation. Erigon was used to prepare the database state for these stages. Later, the work started on implementing two stages that depend on interaction with peers (via Sentry component provided by Erigon), as well as on assembly of the Silkworm StageLoop and converting disparate binaries (one per stage) into a useable product with all stages integrated into one binary.

As shown above, work is currently being done on the stages responsible for downloading block bodies and maintaining the state root hash.

The two most challenging stages

It has become a somewhat surprising, but now recognised experience that the two stages from the diagram above, which turned out to be most challenging in Erigon, then Silkworm (and also Akula, Rust implementation) are “Headers” stage (downloading header chain) and “Trie” stage (computing state root hash).

However, it appears that we are gaining better conceptual understanding of why these stages are challenging, and, as a result, simpler constructions appear iteratively. In Erigon, the simplification of “Headers” stage is ongoing. For Erigon 2 Upgrade 3, a new algorithm of calculating and updating state root hash has been created, and this algorithm promised to be much more tractable.

Performance

Silkworm uses evmone as its EVM interpreter, which is already known to be the fastest fully compatible EVM implementation. In addition to that, Silkworm uses MDBX, which is the fastest embedded key-value store with fully ACID transactions and MVCC (Multi-Version Concurrency Control). And on top of that, Silkworm has been written with performance being one of the highest priorities. In all stages it currently implements, silkworm beats Erigon in processing speed.

As mentioned above, silkRPC includes performance benchmarks, comparing it to Erigon’s RPC Daemon. On these benchmarks, it shows improvements over Erigon’s RPC Daemon especially under highly concurrent load. SilkRPC uses C++ co-routines, a relatively new feature introduced in C++20, to orchestrate such highly concurrent operations efficiently.

Software licence

Silkworm and silkRPC are both licences under Apache 2.0. Not sure how important this will turn out to be, but this may be a benefit for some users.

In this post we will be looking closer into the internals of the “ETH core” component from the architecture diagram in one of the previous posts. Inside is something we call “Stage Loop”. Here is the diagram:

There are two main control flows here - one shown by yellow arrows, and another - by blue arrows. Stage loop is driven by a single go-routine (thread) that repeatedly goes through so-called stages. Below are short descriptions of what each stage does:

• Headers. Requests block headers from ETH sentry, receives them, verifies and persists in 3 tables in the database (one can find full list of tables in the file tables.go in the package kv of the erigon-lib repository):

  • Headers: height|hash => RLP encoding of header

  • HeaderTD: height|hash => RLP encoding of total_difficulty

  • HeaderCanonical: height => hash

• Block Hashes. Computes the inverted mapping (and persists it into a database table): HeaderNumber: hash => height. It is more efficient to perform this separately from the Headers stage, because hashes are not monotonous, so their insertion into a database table is most efficient when hashes are first pre-sorted.

• Bodies. Requests block bodies corresponding to canonical block hashes, receives them, verifies and persists in 2 tables in the database:

  • BlockBody: height => (start_tx_id; tx_count)

  • EthTx: tx_id => RLP encoding of tx

• Senders. Processes digital signatures (ECDSA) of transactions and “recovers” corresponding public keys, and therefore, “From” addresses for each transaction, persists them into a table in the database:

  • Senders: height => sender_0|sender_1|…|sender_{tx_count-1}

• Execution. Replays all transactions and computes so-called “Plain State”, as well as 2 types of change logs, and receipts and event logs, persisting all these into tables in the database. It also create another, temporary table, which is later used one the Call Trace Index Stage.

  • PlainState: account_address => (balance; nonce; code_hash) or account_address|incarnation|location => storage_value

  • PlainContractCode: account_address|incarnation => code_hash

  • Code: code_hash => contract_bytecode

  • AccountChangeSet: height => account_address => (prev_balance; prev_nonce; prev_code_hash)

  • StorageChangeSet: height => account_address|location => prev_storage_value

  • Receipts: height => CBOR encoding of receipts

  • Log: height|tx_index => CBOR encoding of event logs

  • CallTraceSet: height => account_address => (bit_from; bit_to)

• Hashed State. Exists only to provide input data for the next stage. Reads new records in the tables AccountChangeSet and StorageChangeSet, to determine new account addresses and storage locations added to the state, and adds entries to two tables, which have similar content to PlainState, except the mapping is from “hashed keys”, and accounts and storage items are in separate tables:

  • HashesAccounts: keccak256(account_address) => (balance; nonce; code_hash)

  • HashedStorage: keccak256(account_address)|incarnation|keccak256(location) => storage_value

• Trie. Calculates state root hash, as well as maintains two tables (TrieOfAccounts and TrieOfStorage) in the database allowing incrementation computation of state root hash to be done more efficiently.

• Call Trace Index. Processes data from temporary table CallTraceSet and creates two inverted indices (represented by roaring bitmaps):

  • CallFromIndex: account_address => bitmap of heights where account has “from” traces

  • CallToIndex: account_address => bitmap of heights where accounts has “to” traces

• History Index. Processes data from AccountChangeSet and StorageChangeSet tables and creates two inverted indices (as roaring bitmaps):

  • AccountHistory: account_address => bitmap of heights where account was modified

  • StorageHistory: account_address|location => bitmap of heights where storage item was modified

• Log Index. Processes data from Log table and creates two inverted indices (as roaring bitmaps):

  • LogAddressIndex: account_address => bitmap of heights where account is mentioned in any event logs

  • LogTopicIndex: topic => bitmap of heights where topic is mentioned in any event logs

• Tx Lookup. Processes data from BlockBody and EthTx tables and persists mapping that allows finding transactions by their tx hash:

  • TxLookup: tx_hash => height

Whenever control flow reaches any stage, it attempts to process all data available for this stage at the moment. This means that during initial launch, Headers stage will attempt to download all existing block headers, Bodies stage will attempt to download all corresponding block bodies, and so on. It can take many hours before control returns to the Headers stage again, by which time there are more headers available, so the process repeats, but with much fewer number of headers, and then blocks. Eventually, these repetitions converge to processing 1 (or sometimes more) blocks at a time, as they are being produced by the network.

Another control flow, shown in blue, is driven by data arriving from other peers, as managed by sentry. Receiving newly produced blocks, or response to requested headers and block bodies, may happen at the time when the Stage Loop (yellow control flow) is away from the “Headers” or “Bodies” stages, those that are able to “ingest” new headers and block bodies correspondingly. In order not to block the blue control flow, so-called “Exchange” data structures are introduced. They allow for the blue control to deposit data and for the yellow control flow to pick the data up. Of course, these exchanges need to be designed such that they do not consume unlimited memory, but at the same time minimise amount of repeated requests, and latency of delivery of the latest headers and blocks. In order words, it is not trivial, and there are still bugs in the implementation, unfortunately.

The Merge

Design of “The Merge” (a.k.a. POS transition) introduced another, third control flow in the diagram above. It is control flow communicating with the Consensus Layer (CL). Details of the interaction between Staged Sync, Sentry flow, and CL flow are still being worked out, and hopefully will be described in another post. As one may suspect, things are getting more complicated…

./build/bin/mdbx_copy -c <source_db> <destination_db>

Now if we remove that “dead weight table”, the charts would look as follows (this is theoretical, because it will take couple of days to actually produce such database now):

Note that the actual sizes will be bigger because of the inevitable slack in the database after some period of use.

Multiple things were fixed and improved.

Most important problem with alpha release 2022.04.03 was the slow download speed of block snapshots via BitTorrent. Even though there were enough seeders around, the download speed was usually limited by 2Mb/s. Although the exact root cause has not yet been determined, we have observed that:

• Downloading via 3rd party BitTorrent clients does not have this issue, so it is likely that the root cause is due to the library Erigon uses or the manner in which it is used.

• If files are downloaded one at a time, instead of all at the same time, the speed improves.

In version 2022.04.04, we therefore introduce a temporary workaround, to download files one by one, which leads to much more reasonable download times (otherwise it takes around 32 hours to download Ethereum main net block snapshots of 225 Gb at the speed of 2Mb/s).

There were also issues with the last stage of initial sync, called “TxLookup”. This stage manages the mapping that allows locating transaction payload by their hashes. Since in alpha release, most of this mapping was moved from the MDBX database to the indices based on perfect hashing, the issues may arise on the “edge”. If there are any issues, please update to the new version, and if they persist, use integration tool to reset the “TxLookup” stage:

make integration

./build/bin/integration stage_tx_lookup —reset —datadir <your_datadir>

The main topic of this post is the comparison of disk footprint between beta (current stable versions of Erigon) and alpha (with improvements coming from Erigon 2 upgrade 1). Here are the charts:

First thing to note is that in beta version, all the data are kept in the single MDBX database file, which at the time of writing should come at around 1858 Gb after a fresh sync for Ethereum main net. Within that database file, there are individual tables shown as segments in the charts. In alpha version, data related to block headers, block bodies, transaction payloads, senders (originators of transactions), and mapping from transaction hashes to transaction payload, are mostly moved from the MDBX database file into separate static files which are shared and downloaded via BitTorrent. Total size of these shared files is around 225 Gb.

All corresponding segments are coloured in the same colour on both charts, which makes it easy to see that two segments that shrank the most are Transactions (light green, 477 Gb => 17.5 Gb), and Transaction Lookup (grey, 101.3 Gb => 1.3 Gb).

Another thing to note is the biggest tables in the ALPHA footprint breakdown: Event Logs, Account History, Storage History, Call Trace Set. These tables will be the focus of Erigon 2 Upgrade 2.

And lastly, similar to go-ethereum freezer (a.k.a ancient storage), the static files can now be mounted on different drives and do not need to be co-located with the rest of the data. However, we have not yet tested the performance of the configuration where these files are moved to a cheaper but slower storage, like HDD.

This is the TAG.

Main things to know about this release:

1. You CANNOT upgrade your existing Erigon database that was built by one of the previous beta releases. You need to sync from scratch.

2. Currently, to build this alpha code, go 1.18 is required. We have a workaround for 1.17, but it is not in this version.

3. If you are a “casual user”, you do not need to run RPC daemon in a separate process. Instead, you can use the embedded RPC daemon, which is on by default (can be turned off by http false command line flag).

4. BitTorrent protocol gets involved to download most of the headers and blocks (turned on by default for Ethereum main net, Görli testnet and BSC main net). By default, download rate limit is 8mb/s, and upload rate limit is 4mb/s. These limits can be changed by command line flags torrent.download.rate and torrent.upload.rate. In the current code, BitTorrent DHT is not used. Some number of seeders have been bootstrapped for Ethereum mainnet (in progress for BSC mainnet), but the more nodes join the network, the quicker the download will be for everyone else.

5. For BitTorrent download to work, port 42069 needs to be open for both TCP and UDP.

6. BSC mainnet works on this release, but not yet Bor (Polygon).

Plan for further releases

Further releases may become more confusing, at least for some time. Here is an attempt to explain:

Up to now, we were using stable branch to publish beta releases. Out of all PRs made to the devel branch (where most of the current development happens), only some are cherry-picked into stable. These are bug fixes and missing features. This allowed us to keep developing the support for “The merge” as well as Erigon 2 without affecting Erigon users who required relative stability of the beta.

Now we are creating alpha branch, which is going to be used to publish alpha releases. Our intent, similarly to the stable branch, to only cherry-pick into alpha branch bug fixes, missing features (mostly for the POS transition) and substantial optimisations (for example, we know that current version required 6Gb in the heap for Huffman decoding tables, a side effect of an optimisation, and we may make a change to move these 6Gb out of the heap into the virtual memory). At some point before the POS transition we will discontinue what currently is called “Beta” and the current “Alpha” will become de-facto beta.

What will happen after that with the various branches and releases is not clear yet, but the plans will be published as soon as they solidify. Our current goal is to help Erigon users to:

• Experience the POS transition functionality and the first part of Erigon 2 (blockchain snapshots).

• Gradually migrate their infrastructure before the POS transition, flagging up any issues so they can be fixed.

Components are shown as rectangles. In the current version of Erigon code, all components shown as rectangles are “separable”, meaning that they can be run as separate processes. Where two overlapping rectangles are shown, it means that more than one component of this type can be run at the same time, and on multiple computers. All components are separable, and all of them can currently run within the same process (embeddable). However, downloader/seeder components need need to be run on the same computer even if they are separated, to be able to share downloaded and seeded files with Erigon.

The “lollipop” notation is explained below:

All interactions in the diagram above are numbered with numbers in blue background. Below is the description of what each interaction is for:

1. ETH sentry connects to the Ethereum p2p network. It performs these main functions: peer discovery (via Kademlia DHT, via DNS lookup, via configured static peers, via node info saved in the database, or via boot nodes pre-configured in the source code), peer management (handshakes, holding p2p connection even if Erigon is restarted).

2. ETH core interacts with the Ethereum p2p network via Sentry component. Sentry presents a simple interface to the core, with functions to download data, receive notifications about gossip messages, upload data upon request from peers, and broadcast gossip messages to either specially selected set of peers, or to all peers. It is possible to have multiple Sentries to increase the connectivity to the network, or obscure the position of the Core computer.

3. ETH core instructs the ETH downloader/seeder component to download (and then seed) specific files from BitTorrent network. Files are specified as their “info hashes”, which is a form of content addressing. The files ETH core instructs to download are block headers, block bodies, and in the near future also parts of the Ethereum state and various indices.

4. ETH downloader/seeder interacts with the BitTorrent network in order to retrieve files required by ETH core.

5. ETH TxPool uses ETH sentry to download initial set of transactions currently unconfirmed, as well as to receive gossip about new transactions, re-broadcast them, and inject transactions created by users and submitted via ETH RPC api.

6. ETH TxPool connects to ETH core to be able to fetch the up-to-date information about account balances and account nonces for the transactions in the pool. It also subscribes to the stream of state updates to be able to maintain a partial cache of the state, to reduce the interaction in the long run.

7. ETH RPC api connects to ETH core to be able to access any data in the ETH core’s database or other sources (downloaded files), as well as other information (for example, number of currently connected peers via ETH sentries). Through this connection RPC api also subscribes to the stream of state updates to be able to maintain a partial cache of the state, so it can respond to some of the queries using fewer interactions with the ETH core.

8. ETH RPC api connects to ETH TxPool to be able to query transaction pool as well as inject new transactions to be broadcasted to the Ethereum network.

9. User connects to ETH PRC api to query data about current or historical state of Ethereum, subscribe to notifications, and send new transactions to the Ethereum network.

Rollup architecture

Though not yet realised, the modularity of Erigon architecture makes it very suitable platform for implementing rollups based on Ethereum. Here is a theoretical rollup node architecture:

One can observe on the diagram above that the “rollup” part is mostly a mirror image of the “ETH” part, with one extra connection (number 10) from Rollup core to ETH core. This connection is required for the Rollup Core to be notified of events happening in ETH code (usually interacting with the dedicated contract deployed on Ethereum). In the version of the architecture for the block-producing rollup nodes, where will also be a connection (not shown on the diagram above) between Rollup Core and ETH TxPool(s), so that transactions that “roll up” data into Ethereum blocks can be added directly without having to go via JSON RPC and pay the extra cost of data transformations and having less interactive interface (connections between components support duplex streams, for example).

Since all the components are embeddable, it should be possible to realise the entire rollup node (including ETH part) in one executable for convenience. Of course, for more scalable installations, any parts can be taken out and many of them - multiplied.

Architecture after the transition to Proof Of Stake

Current design of “The Merge” does disrupt the architectural elegance presented above. There is no diagram for it yet, but it is easy to imagine a CL (Consensus Layer) component plugged in to the ETH RPC api component. Also, for some time, CL will not be embeddable, and will not utilise the richness of the communication protocol used between other components (duplex streams, for example).

This is, of course, the area where more thinking and design needs to happen. Perhaps the way to go is to create a version of CL implementation that can be embeddable. This is especially interesting for rollup implementations that will themselves use CL implementation for their rollup consensus.

On the picture above, it is assumed that Ethereum implementation processes the blocks received from the network, as a series of pipelines. On the picture, there are 7 stages in such pipeline, shown by 7 colours (in reality, it does not have to be 7). Intuitively, one would want to maximise the utilisation of multi-core or multi-CPU computers and so run many pipelines concurrently. Staged sync goes against this intuition, and proposes the processing model where all available blocks first go through stage 1 of the pipeline, then all of them go through stage 2 of the pipeline and so on. Within each stage, concurrency is allowed, but more often than not, it is actually parallelism more than concurrency that matters. Certain stages in the pipeline, for example, verification of signatures are so-called “embarrassingly parallel” tasks. Others are mostly I/O bound and do not benefit from parallelism. In fact, concurrency may even degrade performance due to content switching and lost locality of data access.

The reason why Staged Sync looks counter-intuitive is that simple model shown in the picture above predicts that it will be slower than concurrent processing. In reality, it is not slower. It is not much faster either. As it was determined later, the main reason Erigon’s Staged Sync ends up being faster than alternatives, is the decoupling of state and Patricia Merkle tree in the data model. Using this decoupling, it is possible to defer the update of the Patricia Merkle tree, and computation of the state root until after all the EVM transaction processing is done, which speeds up the process significantly. Staged Sync makes this approach very natural and almost inevitable.

But other very interesting consequences of adopting the Staged Sync are to do with the development process of Erigon (and later Silkworm and Akula), so-called force multipliers:

1. Profiling of individual stage gives better insights about performance of that stage and results in quicker optimisations (force multiplier 1).

2. Well-defined inputs and outputs of each stage makes it possible to assemble and maintain them in isolation (force multiplier 2).

3. Significant speed up of the full sync (cut from couple of weeks to couple of days) enables more testing and empowers bigger changes (force multiplier 3).

The longest stage was (and still is) the replay of all transaction in EVM, and computation of the resulting state of Ethereum. This stage is customary called “Execution” stage. Due to the force multiplier mentioned above, a lot of profiling and optimisations have been performed in the beginning on 2020, and eventually we arrived at this rough split of wall time:

• Roughly 1/3 was taken by the EVM interpreter

• Roughly 1/3 was taken by the Golang’s garbage collector (after optimisations targeting allocations)

• Roughly 1/3 was taken by I/O (mostly reads from LMDB - this was prior to MDBX)

At the same time, we saw experiments of evmone’s team (evmone is EVM interpreter written in C/C++) to integrate into go-ethereum. We integrated evmone (via EVMC interface) into Erigon (then Turbo-Geth). Even though the sheer performance of evmone was better than built-in interpreter inherited from go-ethereum, there was a lot of overhead introduced by the EVMC interface coupled with CGo (Golang’s interface for interacting with the code written in other languages). The main problem was that EVMC is designed around not just calling of the “host” into EVMC, but also calling out of EVMC into the “host”. And the latter part was introducing the overhead. In order to fix this, we needed to effectively “enlarge” EVMC to envelope it with some extra C/C++ code to remove the interfacing overheads.

We also determined that squeezing more out of Golang in terms of minimising allocation would lead to poorer readability of code and deviation from the idiomatic use of the language. We thought: instead of writing in Go as if it was C++, we might as well just write in C++.

And this is how the Silkworm project (C++ implementation of Ethereum based on Erigon component model, data model, and Staged sync architecture) started. This happened around May 2020.

The current status of Silkworm project will be described in one of the future posts.