All posts by Dave Collins

The Bitcoin Consensus Red Herring

Red Herring

Red Herring

In a recent post on bitcointalk, I stated that focusing solely on the forking risk created by alternative implementations is a red herring because, while there is certainly risk involved, it is merely a symptom of a more fundamental issue that all implementations, including Bitcoin Core itself, suffer. This statement seems to have confused a few people, so, in this blog post, I’d like to delve a little deeper into why this is the case, identify what the real underlying issue is, and offer a potential solution.

First, let’s establish a few baseline facts that, to my knowledge, everyone agrees with:

  • Every fully-validating node on the network must follow the exact same consensus rules or a fork will occur
  • The consensus rules are complex and contain various non-intuitive corner cases, some of which probably still have not been identified
  • Chain forks can be abused to create double spends and generally wreak havoc
  • Every new version of Bitcoin Core carries some level of forking risk (if there is any doubt about this, see the March 2013 fork that already happened)
  • Alternative implementations carry some level of forking risk

Continue reading

BIP0064 – Not yet.

The Bitcoin Core team recently committed BIP0064, which adds two new commands to the protocol, getutxos and utxos. The getutxos command is used to request unspent transaction information based on the given outpoints, while the utxos command is the response.

The BIP was authored by Mike Hearn who also provided the implementation for Bitcoin Core.

Now, since this affects the Bitcoin wire protocol, btcwire needs to add support for this change, even though btcd will not implement these new commands. btcwire is the bitcoin wire protocol package that btcd uses. It has 100% test coverage, meaning it can find bugs before implementing the new API into btcd.

Why you do dat?

I wrote some code to test against Bitcoin Core. This code issues a single getutxos command for outpoint bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f:1 on TestNet3. However, I made the command contain 50,000 requests for this outpoint, since I prefer test-to-fail instead of test-to-pass type development.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
package main

import (
        "fmt"
        "net"
        "sync"

        "github.com/conformal/btcwire"
        "github.com/davecgh/go-spew/spew"
)

const (
        pver   = btcwire.ProtocolVersion
        btcnet = btcwire.TestNet3
)

var (
        conn net.Conn
        wg   sync.WaitGroup
)

func readMessage() {
        defer wg.Done()
        for {
                msg, _, err := btcwire.ReadMessage(conn, pver, btcnet)
                if err != nil {
                        fmt.Printf("ReadMessage error: %v\n", err)
                        return
                }

                switch m := msg.(type) {
                case *btcwire.MsgPing:
                        pong := btcwire.NewMsgPong(m.Nonce)
                        err := btcwire.WriteMessage(conn, pong, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                case *btcwire.MsgVerAck:
                // Ignore
                case *btcwire.MsgVersion:
                        verack := btcwire.NewMsgVerAck()
                        err = btcwire.WriteMessage(conn, verack, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                default:
                        spew.Dump(m)
                }
        }
}

func main() {
        var err error

        host := "127.0.0.1:18333"
        fmt.Printf("Attempting connection to %s\n", host)

        conn, err = net.Dial("tcp", host)
        if err != nil {
                fmt.Printf("dial error: %v\n", err)
                return
        }
        defer conn.Close()

        wg.Add(1)
        go readMessage()

        nonce, err := btcwire.RandomUint64()
        if err != nil {
                fmt.Printf("RandomUint64 error: %v\n", err)
                return
        }

        // Build version message
        version, err := btcwire.NewMsgVersionFromConn(conn, nonce, 0)
        if err != nil {
                fmt.Printf("NewMsgVersionFromConn error: %v\n", err)
                return
        }
        err = version.AddUserAgent("test", "0.0.1", "")
        if err != nil {
                fmt.Printf("AddUserAgent error: %v\n", err)
                return
        }

        // Send version message
        err = btcwire.WriteMessage(conn, version, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }

        // Build getutxos command
        get := btcwire.NewMsgGetUtxos()
        sha, err := btcwire.NewShaHashFromStr("bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f")
        if err != nil {
                fmt.Printf("NewShaHashFromStr error: %v\n", err)
                return
        }

        // Add outpoint index1 50000 times.
        for i := 0; i < btcwire.MaxOutPointsPerMsg; i++ {
                op := btcwire.NewOutPoint(sha, 1)
                err = get.AddOutPoint(op)
                if err != nil {
                        fmt.Printf("AddOutPoint error: %v\n", err)
                        return
                }
        }

        // Send getutxos message
        err = btcwire.WriteMessage(conn, get, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }
        wg.Wait()
}

When run, btcwire rejects the response due to the payload being larger than the bitcoin specification, which is 32MB.

ReadMessage error: ReadMessage: message payload is too large - header indicates 202306292 bytes, but max message payload is 33554432 bytes.

Add a few loops and extra connections in the code, and you can crash Bitcoin Core with out of memory errors:

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:48 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=271
2014-08-26 21:02:49 

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:49 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=275

So, this shows that Bitcoin Core is trying to send us a little over 202MB. The bug is that Bitcoin Core does not follow its own specification of sending messages that do not exceed the maximum message size of 32MB. Secondly, I also wonder if Bitcore Core should skip duplicate outpoints in the getutxos request.

Another issue this brings up is the amount of testing, or the lack thereof, that goes into protocol changes. It is worrisome that protocol changes get merged without accompanying extensive test coverage. This is why the Conformal team puts such a strong emphasis on complete test coverage to help catch such issues that easily go unnoticed by developers.

btcd won’t be supporting the command because it is completely unauthenticated and insecure as pointed out numerous times on the initial pull request. In response to these concerns, a section entitled “Authentication” was added to the BIP which attempts to address the concerns by simply calling them out along with some potential modifications that could happen in the future. However, we prefer to wait until the things necessary to implement this functionality in a secure fashion are already in place, which Bitcoin Core should have done as well.

Introducing btcrpcclient — Bitcoin RPC Made Easy

Bitcoin JSON-RPC

Bitcoin JSON-RPC

Have you been looking for a robust and easy to use way to interface with Bitcoin through the JSON-RPC API? We’ve got you covered!

We’re excited to announce btcrpcclient, a new Websocket-enabled Bitcoin JSON-RPC client package written in Go. This package allows you to quickly create robust Bitcoin RPC clients in just a few minutes.

Major Features of the btcrpcclient Package

  • Supports Websockets (btcd and btcwallet) and HTTP POST mode (Bitcoin Core)
  • Provides callback and registration functions for btcd and btcwallet notifications
  • Supports btcd and btcwallet extensions
  • Translates to and from high-level statically-typed Go types
  • Offers a synchronous (blocking) and asynchronous (non-blocking) API
  • When running in Websockets mode (the default):
    • Provides automatic reconnect handling (can be disabled if desired)
    • Outstanding commands are automatically reissued on reconnect
    • Registered notifications are automatically re-registered on reconnect
    • Back-off support on reconnect attempts

Continue reading

Btcd + getwork + cgminer = profit

Bitcoins

Bitcoins

We are pleased to announce that btcd, our full-node bitcoind (bitcoin core) alternative written in Go, now has support for the getwork RPC which allows it to function with cgminer!

We have extensively tested the code on testnet. The following is a sampling of the latest blocks we’ve generated on testnet using btcd + cgminer: 226713 226830 227228 227390 227393.

If you are simply interested in learning how to configure btcd to work with cgminer, you can view the setup instructions on the btcd wiki, however if you’re interested to learn about some of the nuts and bolts that make it all work, read on.

Isn’t getwork deprecated?

Before I dig into some of the details, I would like to touch on a topic that I’m sure some astute readers will undoubtedly be asking, which is “Why implement getwork when it has been deprecated in favor of getblocktemplate (BIP0022 and BIP0023)?”

There are a few reasons why we chose to implement it first:

  • Supporting getwork was a lot less time consuming than getblocktemplate proposals and software such as cgminer still functions properly with getwork. This means we were able to release support for mining more quickly by supporting getwork first.
  • The bulk of the work involved was the code to create and manipulate block templates which is used by both getwork and getblocktemplate. Thus it helps pave the way to supporting getblocktemplate proposals while allowing getwork to function in the mean time.
  • The current reference implementation (bitcoin core) still supports getwork. Even though it will likely be removed in the next version, one of the goals of btcd is to be a drop-in alternative that can be used with existing infrastructures.

Continue reading

btcd: Not your mom’s Bitcoin daemon

We are pleased to announce that btcd, our full-node bitcoind alternative written in Go, is finally ready for public testing!

The installation instructions and source code can be found on github at:

https://github.com/conformal/btcd

A Brief History

Back in May, we first announced our plans to release btcd. A week later we released our first core package from btcd, btcwire, and announced our plans to continue releasing the component packages of btcd in a staggered fashion.

Over the next month, we released btcjson, btcdb, and btcscript. Then in mid-July we released btcchain at which time we announced btcd was next. At that point, btcd had most of the core bits and we figured we’d be releasing it within a few weeks. Well, as you have no doubt noticed, it is now 10 weeks later…
Continue reading

btcchain: The bitcoin chain package from btcd

As all of you following our blog are aware, we have previously released several bitcoin-related packages (btcwire, btcjson, btcutil, btcdb, btcec, and btcscript) on our way towards the full release of btcd.

We are happy to announce our next package from btcd. The package is named btcchain and it implements the bitcoin block handling and chain selection rules. The code can be reviewed on github here:

https://github.com/conformal/btcchain

Overall Package Design

The bitcoin block handling and chain selection rules are an integral, and quite likely the most important, part of bitcoin. Unfortunately, at the time of this writing, these rules are also largely undocumented and had to be ascertained from the bitcoind source code. At its core, bitcoin is a distributed consensus of which blocks are valid and which ones will comprise the main block chain (public ledger) that ultimately determines accepted transactions, so it is extremely important that fully validating nodes agree on all rules.
Continue reading

btcwire: The bitcoin wire protocol package from btcd

As was recently announced in a previous blog (here), btcd is an alternative full-node implementation of the bitcoin wire protocol and block validation written in Go that is currently under active development.

We’d now like to announce a public preview of one of the core packages from btcd.  The package is named btcwire and it implements the bitcoin wire protocol.  The code can be reviewed on github here:

https://github.com/conformal/btcwire

Overall Package Design

At a high level, this package provides support for marshalling and unmarshalling supported bitcoin messages to and from the wire. This package does not deal with the specifics of message handling such as what to do when a message is received. This provides the caller with a high level of flexibility.
Continue reading