As btcd nears completion we decided to have at least one round of deslugging because we were much slower then bitcoind during chain download. Let me clarify that term for the ones that don’t know what that means. Deslugging is the art of measuring (or profiling) run times of individual functions, determining which ones are slugs and which ones could be optimized. Armed with that data one attacks said functions to see if there are some less then optimal algorithms. Surprisingly enough, this is often the case. An industry truism is “measuring is knowing” and attacking functions based on gut feeling often does not yield satisfying results. As always with these exercises, some things work and some don’t. Throughout this blog I’d like to walk our readers through some optimizations that succeeded, and one, objectively, failed attempt.
Category Archives: Bitcoin
btcwallet and btcgui: Wallet handling for btcd
Today Conformal is announcing alpha releases of btcwallet and btcgui, the wallet components of btcd, written in Go. We announced in a recent blog post that btcd, our full node Bitcoin implementation, was ready for public testing. We also announced that wallet functionality was being implemented separately and would be coming in the near future. Although our wallet daemon and GUI interface are not yet ready for production use, we feel they have progressed to the point where early adopters can begin testing their functionality on the Bitcoin Testnet network.
Our btcd blog post briefly discussed why wallet functionality is not a part of btcd. It highlighted various reasons why we believe separating wallet handling from blockchain handling improves on the integrated wallet design used by bitcoind and bitcoin-qt. This post will continue on that topic, further exploring the details of why a multiprocess wallet design was chosen, how such a design is beneficial to the Bitcoin community as a whole, and the implementation details this design.
btcd: Not your mom’s Bitcoin daemon
We are pleased to announce that btcd, our full-node bitcoind alternative written in Go, is finally ready for public testing!
The installation instructions and source code can be found on github at:
A Brief History
Back in May, we first announced our plans to release btcd. A week later we released our first core package from btcd, btcwire, and announced our plans to continue releasing the component packages of btcd in a staggered fashion.
Over the next month, we released btcjson, btcdb, and btcscript. Then in mid-July we released btcchain at which time we announced btcd was next. At that point, btcd had most of the core bits and we figured we’d be releasing it within a few weeks. Well, as you have no doubt noticed, it is now 10 weeks later…
btcchain: The bitcoin chain package from btcd
As all of you following our blog are aware, we have previously released several bitcoin-related packages (btcwire, btcjson, btcutil, btcdb, btcec, and btcscript) on our way towards the full release of btcd.
We are happy to announce our next package from btcd. The package is named btcchain and it implements the bitcoin block handling and chain selection rules. The code can be reviewed on github here:
Overall Package Design
The bitcoin block handling and chain selection rules are an integral, and quite likely the most important, part of bitcoin. Unfortunately, at the time of this writing, these rules are also largely undocumented and had to be ascertained from the bitcoind source code. At its core, bitcoin is a distributed consensus of which blocks are valid and which ones will comprise the main block chain (public ledger) that ultimately determines accepted transactions, so it is extremely important that fully validating nodes agree on all rules.
gotk3: GTK3 the Go way!
Today marks the first release of gotk3, Conformal’s own GTK3 bindings for Go (available on Github here). These bindings were developed out of a frustration with other GTK bindings for Go either using ancient versions of GTK, not handling memory in a way a Go developer would expect, or simply not working at all on our developers’ OpenBSD and Bitrig machines. gotk3 is Conformal’s response to these issues and attempts to be the best solution for developing new GTK applications with Go.
One of the goals for developing gotk3 was to perform memory management in a very Go-like manner. Like many libraries which must handle memory management manually due to the language they are implemented with, GLib (and GTK which uses it) uses reference counting to determine when an object will never again be used and is ready to be freed, releasing the memory resources it required back to the operating system. However, GLib chooses not to use traditional reference counting, but instead uses a quirky variant called “floating references” to achieve this goal. The rest of this post will cover what floating references do, why they exist, and how gotk3 works around this design to handle memory the way a Go developer expects.
btcscript – the script package from btcd
Following up on our previous releases (btcwire, btcjson, btcutil, and btcdb) we are happy to announce btcscript, the script package from btcd:
btcscript provides code and data structures to parse and execute bitcoin scripts. The scripting system bitcoin uses is a stack-based, FORTH-like language. It is not turing complete by design (as adding a full turing complete language opens up all sorts of potential complications and problems), but it still provides a fair amount of power.
btcdb: The bitcoin database package from btcd
Previously, we released btcwire and btcjson and we are now releasing btcdb, the database package from btcd:
As of late May 2013, there are over 238,000 blocks in the bitcoin blockchain and over 18 million transactions. This works out to be slightly under 11 GB of data when including transaction metadata. btcdb provides an efficient and relatively simple way to store and access this dataset from Go. Although it was built with btcd in mind, there are certainly other potential uses for it and we have designed the interface with that flexibility in mind. Unlike the other packages we have released, it is likely that there will be some minor interface changes as we continue to develop btcdb.
btcjson: The bitcoin JSON-RPC package from btcd
We recently announced btcd, an alternative full-node implementation of the bitcoin wire protocol and block validation written in Go that is under active development.
btcwire was the first package released, and now we would like to announce the public preview of our second package, btcjson, the JSON-RPC package from btcd:
btcwire: The bitcoin wire protocol package from btcd
As was recently announced in a previous blog (here), btcd is an alternative full-node implementation of the bitcoin wire protocol and block validation written in Go that is currently under active development.
We’d now like to announce a public preview of one of the core packages from btcd. The package is named btcwire and it implements the bitcoin wire protocol. The code can be reviewed on github here:
Overall Package Design
At a high level, this package provides support for marshalling and unmarshalling supported bitcoin messages to and from the wire. This package does not deal with the specifics of message handling such as what to do when a message is received. This provides the caller with a high level of flexibility.
btcd: a bitcoind alternative written in Go
btcd is an alternative full-node implementation of the bitcoin protocol written in Go and is currently under active development. btcd has been under development for 10 weeks and the initial code is nearly ready for public release. We feel that by providing an alternative to bitcoind we can substantially improve the diversity and resilience of the bitcoin ecosystem and infrastructure.
A number of us at Conformal Systems had been keeping an eye on bitcoin as passive observers for the past couple years since bitcoin combines technologies that are already of interest to us: practical use of cryptography, distributed systems, and electronic payments. In January 2013 I had one of our developers, David Hill, attempt to port bitcoind and its GUI to Bitrig, an OS that several of our developers forked from OpenBSD. David encountered several problems with porting to Bitrig and in the process found issues with unit tests, non-portable functions and seeding of a PRNG. While pushing to get the port complete, it was clear that it would take a lot more effort than usual to complete this port. After seeing these issues with the porting, I felt that the bitcoin ecosystem could use an alternative to bitcoind.