The Core Issue: Outrunning Entropy, Why Bitcoin Can’t Stand Still
Bitcoin Magazine The Core Issue: Outrunning Entropy, Why Bitcoin Can’t Stand Still The IBD Process Synchronizing a new node to the network tip involves several distinct stages: Peer discovery and chain selection where the node connects to random peers and determines the most-work chain. Header download when block headers are fetched and connected to form the full header chain. Block download when the node requests blocks belonging to that chain from multiple peers simultaneously. Block and transaction validation where each block’s transactions are verified before the next one is processed. While block validation itself is inherently sequential, each block depends on the state produced by the previous one, much of the surrounding work runs in parallel. Header synchronization, block downloads and script verification can all occur concurrently on different threads. An ideal IBD saturates all subsystems maximally: network threads fetching data, validation threads verifying signatures, and database threads writing the resulting state. Without continuous performance improvement, cheap nodes might not be able to join the network in the future. Intro Bitcoin’s “don’t trust, verify” culture requires that the ledger can be rebuilt by anyone from scratch. After processing all historical transactions every user should arrive at the exact same local state of everyone’s funds as the rest of the network. This reproducibility is at the heart of Bitcoin’s trust-minimized design, but it comes at a significant cost: after almost 17 years, this ever-growing database forces newcomers to do more work than ever before they can join the Bitcoin network. When bootstrapping a new node it has to download, verify, and persist every block from genesis to the current chain tip – a resource-intensive synchronization process called Initial Block Download (IBD). While consumer hardware continues to improve, keeping IBD requirements low remains critical for maintaining decentralization by keeping validation accessible to everyone – from lower-powered devices like Raspberry Pis to high-powered servers. Benchmarking process Performance optimization begins with understanding how software components, data patterns, hardware, and network conditions interact to create bottlenecks in performance. This requires extensive experimentation, most of which gets discarded. Beyond the usual balancing act between speed, memory usage, and maintainability, Bitcoin Core developers must choose the lowest-risk/highest-return changes. Valid-but-minor optimizations are often rejected as too risky relative to their benefit. We have a significant suite of micro-benchmarks to ensure existing functionality doesn’t degrade in performance. These are useful for catching regressions, i.e. performance backslides in individual pieces of code, but aren’t necessarily representative of overall IBD performance. Contributors proposing optimizations provide reproducers and measurements across different environments: operating systems, compilers, storage types (SSD vs HDD), network speeds, dbcache sizes, node configurations (pruned vs archival), and index combinations. We write single-use benchmarks and use compiler explorers for validating which setup would perform better in that specific scenario (e.g. intra-block duplicate transaction checking with Hash Set vs Sorted Set vs Sorted vector). We’re also regularly benchmarking the IBD process. This can be done by reindexing the chainstate and optionally the block index from local block files, or doing a full IBD either from local peers (to avoid slow peers affecting timings) or from the wider p2p network itself. IBD benchmarks often show smaller improvements than micro-benchmarks since network bandwidth or other I/O is often the bottleneck; downloading the blockchain alone takes ~16 hours with average global internet speeds. For maximum reproducibility -reindex-chainstate is often favored, creating memory and CPU profiles before and after the optimization and validating how the change affects other functionality. Historical and ongoing improvements Early Bitcoin Core versions were designed for a much smaller blockchain. The original Satoshi prototype laid the foundations, but without constant innovation from Bitcoin Core developers it would not have been able to handle the network’s unprecedented growth. Originally the block index stored every historic transaction and whether they were spent, but in 2012, “Ultraprune” (PR #1677) created a dedicated database for tracking unspent transaction outputs, forming the UTXO set, which pre-caches the latest state of all spendable coins, providing a unified view for validation. Combined with a database migration from Berkeley DB to LevelDB validation speeds were significantly improved. However, this database migration caused the BIP50[1] chain fork when a block with many transaction inputs was accepted by upgraded nodes but rejected by older versions as being too complicated. This highlights how Bitcoin Core development differs from typical software engineering: even pure performance optimizations have the potential to result in unintended chain splits. The following year (PR #2060) enabled multithreaded signature validation. Around the same time, the specialized cryptographic library libsecp256k1 was created, and was integrated into Bitcoin Core in 2014. Over the following decade, through continuous optimizations, it became more than 8x faster than the same functionality in the general-purpose OpenSSL library. Headers-first sync (PR #4468, 2014) restructured the IBD process to first download the block header chain with the most accumulated work, then fetch blocks from multiple peers simultaneously. Besides accelerating IBD it also eliminated wasted bandwidth on blocks that would be orphaned as they were not in the main chain. In 2016 PR #9049 removed what appeared to be a redundant duplicate-input check, introducing a consensus bug that could have allowed supply inflation. Fortunately, it was discovered and patched before exploitation. This incident drove major testing resource investments. Today, with differential fuzzing, broad coverage, and stricter review discipline, Bitcoin Core surfaces and resolves issues far more quickly, with no comparable consensus hazards reported since.[2]. In 2017 -assumevalid (PR #9484) separated general block validity checks from the expensive signature verification, making the latter optional for most of IBD, cutting its time roughly in half. Block structure, proof-of-work, and spending rules remain fully verified: -assumevalid skips signature checks entirely for all blocks up to a certain block height. In 2022 PR #25325 replaced Bitcoin Core’s ordinary memory allocator with a custom pool-based allocator optimized for the coins cache. By designing specifically for Bitcoin’s allocation patterns, it reduced memory waste and improved cache efficiency, delivering ~21% faster IBD while fitting more coins in the same memory footprint. While code itself doesn’t rot, the system it operates within constantly evolves. Every 10 minutes Bitcoin’s state changes – usage patterns shift, bottlenecks migrate. Maintenance and optimization aren’t optional; without constant adaptation, Bitcoin would accumulate vulnerabilities faster than a static codebase could defend against, and IBD performance would steadily regress despite advances in hardware. The increasing size of the UTXO set and growth in average block weight exemplify this evolution. Tasks that were once CPU-bound (like signature verification) are now often Input/Output (IO)-bound due to heavier chainstate access (having to check the UTXO set on disk). This shift has driven new priorities: improving memory caching, reducing LevelDB flush frequency, and parallelizing disk reads to keep modern multi-core CPUs busy. A look at IBD times for different Bitcoin Core releases. Recent optimizations The software designs are based on predicted usage patterns, which inevitably diverge from reality as the network evolves. Bitcoin’s deterministic workload allows us to measure actual behavior and course correct later, ensuring performance keeps pace with the network’s growth. We’re constantly adjusting defaults to better fit real-world usage patterns. A few examples: PR #30039 increased LevelDB’s max file size – a single parameter change that delivered ~30% IBD speedup by better matching how the chainstate database (UTXO set) is actually accessed. PR #31645 doubled the flush batch size, reducing fragmented disk writes during IBD’s most write-intensive phase and speeding up progress saves when IBD is interrupted. PR #32279 adjusted the internal prevector storage size (used mainly for in-memory script storage). The old pre-segwit threshold prioritized older script templates at the expense of newer ones. By adjusting the capacity to cover modern script sizes, heap allocations are avoided, memory fragmentation is reduced, and script execution benefits from better cache locality. All small, surgical changes with measurable validation impacts. Beyond parameter tuning, some changes required rethinking existing designs: PR #28280 improved how pruned nodes (which discard old blocks to save disk space) handle frequent memory cache flushes. The original design either dumped the entire cache or scanned it to find modified entries. Selectively tracking modified entries enabled over 30% speedup for pruned nodes with maximum dbcache and ~9% improvement with default settings. PR #31551 introduced read/write batching for block files, reducing the overhead of many small filesystem operations. The 4x-8x speedup in block file access improved not just IBD but other RPCs as well. PR #31144 optimized the existing optional block file obfuscation (used to make sure data isn’t stored in cleartext on disk) by processing 64-bit chunks instead of byte-by-byte operations, delivering another IBD speedup. With obfuscation being essentially free users no longer need to choose between safe storage and performance. Other minor caching optimizations (such as PR #32487) enabled adding additional safety checks that were deemed too expensive before (PR #32638). Similarly, we can now flush the cache more frequently to disk (PR #30611), ensuring nodes never lose more than one hour of validation work in case of crashes. The modest overhead was acceptable because earlier optimizations had already made IBD significantly faster. PR #32043 currently serves as a tracker for IBD-related performance improvements. It groups a dozen ongoing efforts, from disk and cache tuning to concurrency enhancements, and provides a framework for measuring how each change affects real-world performance. This approach encourages contributors to present not only code but also reproducible benchmarks, profiling data, and cross-hardware comparisons. Future optimization suggestions PR #31132 parallelizes transaction input fetching during block validation. Currently, each input is fetched from the UTXO set sequentially – cache misses require disk round trips, creating an IO bottleneck. The PR introduces parallel fetching across multiple worker threads, achieving up to ~30% faster -reindex-chainstate (~10 hours on a Raspberry Pi 5 with 450MB dbcache). As a side effect, this narrows the performance gap between small and large -dbcache values, potentially allowing nodes with modest memory to sync nearly as fast as high-memory configurations. Besides IBD, PR #26966 parallelizes block filter and transaction index construction using configurable worker threads. Keeping the persisted UTXO set compact is critical for node accessibility. PR #33817 experiments with reducing it slightly by removing an optional LevelDB feature that might not be needed for Bitcoin’s specific use case. SwiftSync[3] is an experimental approach leveraging our hindsight about historical blocks. Knowing the actual outcome, we can categorize every encountered coin by its final state at the target height: those still unspent (which we store) and those spent by that height (which we can ignore, merely verifying they appear in matching create/spend pairs anywhere). Pre-generated hints encode this classification, allowing nodes to skip UTXO operations for short-lived coins entirely. Bitcoin Is Open To Anyone Beyond synthetic benchmarks, a recent experiment[4] ran the SwiftSync prototype on an underclocked Raspberry Pi 5 powered by a battery pack over WiFi, completing -reindex-chainstate of 888,888 blocks in 3h 14m. Measurements with equivalent configurations show a 250% full validation speedup[5] across recent Bitcoin Core versions. Years of accumulated work translate to genuine impact: fully validating nearly a million blocks can now be done in less than a day on cheap hardware, maintaining accessibility despite continuous blockchain growth. Self-sovereignty is more accessible than ever. Get your copy of The Core Issue today! Don’t miss your chance to own The Core Issue — featuring articles written by many Core Developers explaining the projects they work on themselves! This piece is the Letter from the Editor featured in the latest Print edition of Bitcoin Magazine, The Core Issue. We’re sharing it here as an early look at the ideas explored throughout the full issue. [1] https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki [2] https://en.bitcoin.it/wiki/Common_Vulnerabilities_and_Exposures [3] https://delvingbitcoin.org/t/swiftsync-speeding-up-ibd-with-pre-generated-hints-poc/1562 [4] https://x.com/L0RINC/status/1972062557835088347 [5] https://x.com/L0RINC/status/1970918510248575358 All Pull Requests (PR) listed in this article can be looked up by number here: https://github.com/bitcoin/bitcoin/pulls This post The Core Issue: Outrunning Entropy, Why Bitcoin Can’t Stand Still first appeared on Bitcoin Magazine and is written by willcl-ark, l0rinc and hodlinator.
