Red Herring
In a recent post on bitcointalk, I stated that focusing solely on the forking risk created by alternative implementations is a red herring because, while there is certainly risk involved, it is merely a symptom of a more fundamental issue that all implementations, including Bitcoin Core itself, suffer. This statement seems to have confused a few people, so, in this blog post, I’d like to delve a little deeper into why this is the case, identify what the real underlying issue is, and offer a potential solution.
First, let’s establish a few baseline facts that, to my knowledge, everyone agrees with:
- Every fully-validating node on the network must follow the exact same consensus rules or a fork will occur
- The consensus rules are complex and contain various non-intuitive corner cases, some of which probably still have not been identified
- Chain forks can be abused to create double spends and generally wreak havoc
- Every new version of Bitcoin Core carries some level of forking risk (if there is any doubt about this, see the March 2013 fork that already happened)
- Alternative implementations carry some level of forking risk
The Red Herring and the Real Underlying Issue
There is something suspiciously similar about those last two baseline facts. Whenever this occurs, there is a very high probability that there is an underlying cause and the observed behavior is actually a red herring. In this case, the red herring is only focusing on forking risk introduced by alternative implementations. It is very much true that alternative implementations carry some forking risk, but it is equally true that every new version of Bitcoin Core carries some forking risk too. You can certainly debate which one has more or less forking risk, but the underlying fact that both carry forking risk is indisputable. That fact leads to the real underlying issue:
There is currently no way to guarantee that any two versions of Bitcoin software, whether they are two different versions of Bitcoin Core, two different versions of alternative implementations, a version of Bitcoin Core versus a version of an alternative implementation, or even two copies of the same version of Bitcoin Core built with different compiler versions are in exact consensus agreement. Doing so is incredibly difficult and borders on impossible. The issue is implementation independent.
The Bitcoin Core Solution?
First, I want to make it clear that my intention here is not an attack on Bitcoin Core or its developers, who I think are talented individuals doing a fine job overall. My goal here is to point out an issue that is largely being ignored with the desire of having discussions/working towards providing real solutions for it.
The current solution as proposed by Bitcoin Core is, when you get right down to it, “let’s just be really careful with code changes and hope for the best”. There is no real disaster recovery or prevention plan. Yes, there is ongoing work to split the consensus critical bits out into a library and I think this is a good idea and am glad to see it happening. However, it still does nothing to address the real fundamental issue stated above nor does it provide disaster recovery or prevention.
Anyone who has been involved with software development for even a modicum of time knows that mistakes do and will happen regardless of how talented the people working on it are. Believing there will never again be an inadvertent mistake in Bitcoin Core, or any of the dependent technology, which causes a chain fork is not a realistic expectation and certainly not a solution.
In my opinion, the current state just isn’t good enough for a financial system that we all hope will one day be a major force in the global economy. It is akin to running a data center with critical customer data without disaster recovery mechanisms, such as backups, and hoping for the best.
Alternative Solution
Now that we’ve covered why the currently proposed methodology is problematic, what can we do? There are various approaches that can be taken, but, in order to illustrate that it is possible to solve the fundamental issue, I’d like to propose just one solution which possesses this property along with a few other benefits.
Consider for a moment if miners consulted multiple community-blessed implementations about whether they consider the block valid (with the exception of the proof of work check). This would allow a miner to construct a block and obtain consensus from multiple implementations before they ever even start solving the block. Keep in mind here when I say multiple implementations, I’m speaking in very generic terms. For example, that includes multiple versions of Bitcoin Core as well as alternative implementations like btcd, Obelisk, etc.
With an approach such as this, a miner could detect that, say Bitcoin Core 0.9.5 and btcd 0.9.0 consider the block valid, but Bitcoin Core 0.10.0 does not, or Bitcoin Core 0.9.5 and Bitcoin Core 0.10.0 consider the block valid, but btcd 0.9.0 does not. In any case, there is an extremely high probability that the 2 of the 3 which agree are right while the other one has a bug. As the number of community-blessed implementations you check against increases, the probability that a debilitating chain fork can occur decreases. Unlike the current scheme, this would allow the Bitcoin network as a whole to continue functioning properly even if a mistake were made in a new version of Bitcoin Core, and as a natural extension, alternative implementations.
The only people that would be affected then are the ones running the specific version/implementation which does not agree. Compare this to the current approach where it could potentially bring down the entire network. Upon updating/reverting their version, they would sync back up to the main chain, which the rest of the implementations agreed upon, and continue business as normal. This approach provides real disaster prevention for the network and disaster recovery for the affected nodes.
The good news is this capability already exists in the form of BIP0023 block proposals. The icing on the cake is that both btcd and Bitcoin Core already support them!
We, as a community, would have to do some work to make it easier for miners to plug in to the capability and educate them why it is in their best interest to do so, but these do not seem to be insurmountable obstacles.
Wait, Doesn’t That Take More Resources?
Yes. Disaster recovery and, more importantly, prevention, invariably require more resources. Keeping with the data backup analogy, more resources are required in the form of disk space, processor cycles, etc to backup data. However, anyone mining should understand why they’d not want to be the odd man out.
Additionally, there are a myriad of techniques to reduce the burden such as the use of distributed super nodes of various blessed implementations which allow miners to check against without having to incur a lot of extra resource hits themselves.
Conclusion
We have been accused on various occasions of not understanding the issues in play, but I hope this blog post shows that is not the case. We just simply disagree that the current Bitcoin Core approach is healthy for the longevity of Bitcoin. I would hope those who think the current approach Bitcoin Core is adopting is a real solution would seriously consider the content of this post with an open mind. The first step to recovery is admitting there is a problem.
What should be clear is that if you gracefully handle Bitcoin Core forking against itself, you also gracefully handle forking of alternative implementations. That very fact exposes why solely focusing on forking risk created by alternative implementations is a red herring which is a mere distraction from the real issue of the ecosystem not having any infrastructure to gracefully prevent and/or recover from mistakes in Bitcoin Core, regardless of the existence of alternative implementations.
We’re aware there are other potential issues with the alternative solution proposed which haven’t been covered here, but this post is already too long and the goal of it is not to 100% nail down the perfect solution, rather it is to elucidate why the current approach is ill-fated, demonstrate it’s not an unsolvable problem, and hopefully start serious discussion on approaches that address the real fundamental issue that exists.
Consensus is even harder than you describe: old versions of bitcoind could theoretically “self-fork” due to the March 2013 bug — identical software running on identical hardware could disagree about whether a block was valid or not (because the bug was tickled by the exact layout of data on disk, which differs between nodes because they receive transactions and blocks at different times).
I agree that miners should be encouraged to run software that pre-checks blocks against multiple implementations; that would be an improvement from today.
Have you had any luck getting pool operators or big miners or developers of pool software to support block proposals?
And have you talked with them about what should happen if a block proposal fails? (shutdown and setoff a pager alarm? Try to figure out if one or more transactions made it fail and rebuild the block without those transactions?)
As paralyzing as it might be, mining empty blocks would be better than forking.
Thanks for your response, Gavin. I agree with you that consensus difficulty goes even deeper than the article covers. I can tell based on your response that you understand the point I was trying to make.
To answer your questions:
I’m not sure if Luke has had a chance to get everything setup, but he seemed to be amenable to the idea of BIP0023 block proposals. Given he was the one that submitted the BIP, this is not too surprising. Outside of that, I have not yet been in contact with other large mining pools.
One of the challenges to overcome is that there are a couple of very vocal individuals with notoriety who are actively campaigning against even bothering to address the fundamental issue and instead saying the consensus library is the answer. It definitely helps and is a good thing, but unfortunately it also has the negative consequence of lulling people into a false sense of security.
I think once we can get the community, as a whole, to understand why that’s the case, it will be more amenable to developing infrastructure to better deal with it.
Fantastic post.
A rather beautiful thing to consider is that the existing design of Bitcoin provides an incentive for miners to adopt your proposed solution.
A blockchain fork driven by consensus-failure, poses risks to miners:
The miner may end up loosing 1 or more block rewards as a result of backing the wrong side of 2 competing chains.
Even if the fork is resolved quickly, minimising the damage, news articles covering this consensus failure driven fork, would create uncertainty about the reliability of Bitcoin which would drive some to dump their Bitcoins in fear. This would create a price drop. Not good for the miner who sells their Bitcoins as soon as they are mined to cover their expenses in their low-margin, mining venture!
A miner can reduce these risks by choosing to use mining software that checks their computed blocks are compatible with multiple Bitcoin implementations and multiple versions of those implementations, as you have described.
Thanks for your response. It’ll certainly be a challenge. I just hope that, regardless of whether or not this particular approach is used, we can raise awareness about the fundamental issue so it’s leads a real solution.
And yet irresponsible parties like Gavin here feel perfectly at ease pushing the bugaboo of imaginary blockchain size related “uncertainties” that supposedly chill adoption.
All the while the quite present and quite undisputed actual uncertainties such as … well… the fact – the actual historical verifiable fact – that the Power Rangers have yet to release a single version that doesn’t break something, these aren’t chilling anything and don’t need to be addressed.
It is the behaviour of the vain idiot to resolve imaginary problems he knows how to approach instead of actual problems above his pay grade. That Gavin has a paygrade at all in these circumstances is outrageous enough, but the matter is in the process of being sorted out, what with the wringing of the last semblance of life out of Vessenes’ scam “Foundation”.
Meanwhile, some effort to salvage what can still be salvaged out of the Satoshi codebase is underway.
As one of the developers of the biggest Haskell implementation of the Bitcoin protocol, I have been nodding throughout this entire post. I’m glad you took the time to write this post. These same ideas have been swimming in my mind too. An ecosystem of Bitcoin implementations against which miners can check their blocks does a lot more for Bitcoin than a single implementation. I will elaborate.
Consensus algorithms in Bitcoin Core have bugs, as any complex piece of software does. Some bugs are obvious, like the extra element that the CHECKMULTISIG operator takes from the stack. These obvious bugs are documented and added to test suites, like BlueMatt has done. These bugs are part of the Bitcoin consensus, and are expected to show up in all versions of Bitcoin full node software. If the implementation passes the tests, it is reasonable to expect that its consensus is “correct”. This is the easy part.
Now there are hidden non-obvious yet-to-be-discovered bugs. These can be catastrophic for Bitcoin. These bugs tend to be implementation-specific, unless they are rare logical bugs. By making it standard practice to test blocks against as many Bitcoin implementations of the consensus protocol as practical, miners can make these hidden implementation bugs obvious, and steer away from potential systemic failure of the whole Bitcoin network.
Gavin’s question of what to do once it is found that a block would not pass consensus rules is legitimate. I can attempt to suggest courses of action for miners when this situation is encountered, although I believe that more important than what to do with them is to have visibility. Knowing that a block will not be accepted by certain versions of a particular client before mining it allows the miner to take evasive action, like discarding whatever transaction is affecting consensus, and notifying Bitcoin developers that something odd is happening. By as much as miners choose not to mine blocks that break any client, they will be benefiting themselves as well as the rest of the Bitcoin ecosystem by avoiding forks. Also, programmers would be allowed to be human and make mistakes. We would not need to unreasonably expect core developers to have superhuman abilities to detect bugs before they cause problems.