btcdb: The bitcoin database package from btcd

Previously, we released btcwire and btcjson and we are now releasing btcdb, the database package from btcd:

https://github.com/conformal/btcdb

As of late May 2013, there are over 238,000 blocks in the bitcoin blockchain and over 18 million transactions. This works out to be slightly under 11 GB of data when including transaction metadata. btcdb provides an efficient and relatively simple way to store and access this dataset from Go. Although it was built with btcd in mind, there are certainly other potential uses for it and we have designed the interface with that flexibility in mind.

Unlike the other packages we have released, it is likely that there will be some minor interface changes as we continue to develop btcdb.

Database Choice and Portability

btcdb presents a slight departure from our previous packages. btcwire and btcjson are both written in pure Go with few external dependencies, and only external dependencies that are also pure Go. This means that they should run on any platform where Go is available – we even have reports of btcwire working on Plan9.

For btcdb, we had to use a database. Currently, it uses sqlite for its database storage and there are a few reasons for this. First, sqlite is well known and well understood. We have enough previous experience with it to use it effectively and to minimize surprises. Second, since it is file based, it requires no separate database setup for users. We have experimented with postgresql, but that would require a user to setup a database server and database to use this package. We may support postgresql at some point in the future if we decide there is enough interest or utility.

The other side of using sqlite is that the Go bindings are not pure Go and depend on linking to the C library using cgo. This means that this package will not be usable on platforms that do not have a functioning cgo. That should not be a major portability problem as we currently have developers using OpenBSD, Bitrig, Linux, and Windows. We are currently using it on a number of platforms already, but it will leave platforms such as Plan9 out for now.

We did look at other database options, particularly ones written entirely in Go. We had some interest in LevelDB, but at the time we selected a database, the native Go version was not complete. Go is still a young language, so there is significant development going on, especially in the packages. We will certainly keep an eye out for newer or updated database packages to try, while keeping the functionality of btcdb the same or improved.

Testing and Performance

Per the usual we are aiming for 100% test coverage for all the btcd related packages. There is a notable caveat with test coverage for database-specific code in Go: database testing in go-testdb does not include support for prepared statements and transactions, which we use heavily. btcdb is currently at 82.23% coverage and we hope to add more tests in the coming months, however much of the remaining coverage would require a sql database framework which could introduce errors in transactions and prepared statements.

For btcdb, there has also been a large amount of testing related to performance. Although we do not currently have any benchmark code included, we have been testing btcdb on a variety of systems and configurations to allow us to experiment and try to get the best performance possible.

btcutil

Along with btcdb, we are also releasing the package btcutil:

https://github.com/conformal/btcutil

btcutil contains several convenience function and types for bitcoin (including the datatype to represent a bitcoin block). As with the other packages we have released, this was built for btcd, but should be usable in other situations as well. We expect to add additional types to btcutil as we continue the development along with increasing the test coverage.

Next Up

To help keep the quality, reusability, and test coverage up to our standards, we are continuing to release btcd one reusable component at a time. We hope to announce the next component here soon.