Category Archives: btcd

The Bitcoin Consensus Red Herring

Red Herring

Red Herring

In a recent post on bitcointalk, I stated that focusing solely on the forking risk created by alternative implementations is a red herring because, while there is certainly risk involved, it is merely a symptom of a more fundamental issue that all implementations, including Bitcoin Core itself, suffer. This statement seems to have confused a few people, so, in this blog post, I’d like to delve a little deeper into why this is the case, identify what the real underlying issue is, and offer a potential solution.

First, let’s establish a few baseline facts that, to my knowledge, everyone agrees with:

  • Every fully-validating node on the network must follow the exact same consensus rules or a fork will occur
  • The consensus rules are complex and contain various non-intuitive corner cases, some of which probably still have not been identified
  • Chain forks can be abused to create double spends and generally wreak havoc
  • Every new version of Bitcoin Core carries some level of forking risk (if there is any doubt about this, see the March 2013 fork that already happened)
  • Alternative implementations carry some level of forking risk

Continue reading

Monetas brings colored coins to btcd

This article is syndicated from the Monetas blog with the approval of the author, Justus Ranvier.

As a company with current and planned blockchain-synergistic products, Monetas is strongly invested in Bitcoin wallet development. After choosing btcwallet from Conformal Systems as our technical foundation for blockchain integration, Monetas is currently devoting four full-time software engineers to adding features and improvements to btcwallet.

One of those features is colored coin support. This article will explain colored coins, and how we are implementing them in btcwallet, and what capabilities these changes entail for both Monetas and Conformal.

colored-1

Continue reading

btcsim: simulating the rise of Bitcoin

btcsim: to the moon!
Over the past 4 months we have had 2 interns, Javed Khan and Michalis Kargakis, work on creating a system for simulating high transaction volumes with Bitcoin called btcsim. Since there has been and is currently a great deal of attention being paid to the issues surrounding mining incentives and block propagation, we figured that it would be interesting to investigate the less-discussed topic of high transaction volumes. There are a variety of claims about how the Bitcoin network will behave at transaction volumes approaching those of major credit card companies, e.g. 3,000 transactions per second (“tps”), so we used btcsim to put btcd and btcwallet to the test. After simulating the creation of blocks up to 32 MB in size, we have arrived at some interesting conclusions:

  • a 32 MB block, when filled with simple P2PKH transactions, can hold approximately 167,000 transactions, which, assuming a block is mined every 10 minutes, translates to approximately 270 tps
  • a single machine acting as a full node takes approximately 10 minutes to verify and process a 32 MB block, meaning that a 32 MB block size is near the maximum one could expect to handle with 1 machine acting as a full node
  • a CPU profile of the time spent processing a 32 MB block by a full node is dominated by ECDSA signature verification, meaning that with the current infrastructure and computer hardware, scaling above 300 tps would require a clustered full node where ECDSA signature checking is load balanced across multiple machines.

While the current mainnet rate of transactions (0.9 tps) are much lower than those tested using btcsim, this paints a relatively rosy picture for Bitcoin should the block size be allowed to increase.

Continue reading

BIP0064 – Not yet.

The Bitcoin Core team recently committed BIP0064, which adds two new commands to the protocol, getutxos and utxos. The getutxos command is used to request unspent transaction information based on the given outpoints, while the utxos command is the response.

The BIP was authored by Mike Hearn who also provided the implementation for Bitcoin Core.

Now, since this affects the Bitcoin wire protocol, btcwire needs to add support for this change, even though btcd will not implement these new commands. btcwire is the bitcoin wire protocol package that btcd uses. It has 100% test coverage, meaning it can find bugs before implementing the new API into btcd.

Why you do dat?

I wrote some code to test against Bitcoin Core. This code issues a single getutxos command for outpoint bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f:1 on TestNet3. However, I made the command contain 50,000 requests for this outpoint, since I prefer test-to-fail instead of test-to-pass type development.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
package main

import (
        "fmt"
        "net"
        "sync"

        "github.com/conformal/btcwire"
        "github.com/davecgh/go-spew/spew"
)

const (
        pver   = btcwire.ProtocolVersion
        btcnet = btcwire.TestNet3
)

var (
        conn net.Conn
        wg   sync.WaitGroup
)

func readMessage() {
        defer wg.Done()
        for {
                msg, _, err := btcwire.ReadMessage(conn, pver, btcnet)
                if err != nil {
                        fmt.Printf("ReadMessage error: %v\n", err)
                        return
                }

                switch m := msg.(type) {
                case *btcwire.MsgPing:
                        pong := btcwire.NewMsgPong(m.Nonce)
                        err := btcwire.WriteMessage(conn, pong, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                case *btcwire.MsgVerAck:
                // Ignore
                case *btcwire.MsgVersion:
                        verack := btcwire.NewMsgVerAck()
                        err = btcwire.WriteMessage(conn, verack, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                default:
                        spew.Dump(m)
                }
        }
}

func main() {
        var err error

        host := "127.0.0.1:18333"
        fmt.Printf("Attempting connection to %s\n", host)

        conn, err = net.Dial("tcp", host)
        if err != nil {
                fmt.Printf("dial error: %v\n", err)
                return
        }
        defer conn.Close()

        wg.Add(1)
        go readMessage()

        nonce, err := btcwire.RandomUint64()
        if err != nil {
                fmt.Printf("RandomUint64 error: %v\n", err)
                return
        }

        // Build version message
        version, err := btcwire.NewMsgVersionFromConn(conn, nonce, 0)
        if err != nil {
                fmt.Printf("NewMsgVersionFromConn error: %v\n", err)
                return
        }
        err = version.AddUserAgent("test", "0.0.1", "")
        if err != nil {
                fmt.Printf("AddUserAgent error: %v\n", err)
                return
        }

        // Send version message
        err = btcwire.WriteMessage(conn, version, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }

        // Build getutxos command
        get := btcwire.NewMsgGetUtxos()
        sha, err := btcwire.NewShaHashFromStr("bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f")
        if err != nil {
                fmt.Printf("NewShaHashFromStr error: %v\n", err)
                return
        }

        // Add outpoint index1 50000 times.
        for i := 0; i < btcwire.MaxOutPointsPerMsg; i++ {
                op := btcwire.NewOutPoint(sha, 1)
                err = get.AddOutPoint(op)
                if err != nil {
                        fmt.Printf("AddOutPoint error: %v\n", err)
                        return
                }
        }

        // Send getutxos message
        err = btcwire.WriteMessage(conn, get, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }
        wg.Wait()
}

When run, btcwire rejects the response due to the payload being larger than the bitcoin specification, which is 32MB.

ReadMessage error: ReadMessage: message payload is too large - header indicates 202306292 bytes, but max message payload is 33554432 bytes.

Add a few loops and extra connections in the code, and you can crash Bitcoin Core with out of memory errors:

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:48 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=271
2014-08-26 21:02:49 

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:49 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=275

So, this shows that Bitcoin Core is trying to send us a little over 202MB. The bug is that Bitcoin Core does not follow its own specification of sending messages that do not exceed the maximum message size of 32MB. Secondly, I also wonder if Bitcore Core should skip duplicate outpoints in the getutxos request.

Another issue this brings up is the amount of testing, or the lack thereof, that goes into protocol changes. It is worrisome that protocol changes get merged without accompanying extensive test coverage. This is why the Conformal team puts such a strong emphasis on complete test coverage to help catch such issues that easily go unnoticed by developers.

btcd won’t be supporting the command because it is completely unauthenticated and insecure as pointed out numerous times on the initial pull request. In response to these concerns, a section entitled “Authentication” was added to the BIP which attempts to address the concerns by simply calling them out along with some potential modifications that could happen in the future. However, we prefer to wait until the things necessary to implement this functionality in a secure fashion are already in place, which Bitcoin Core should have done as well.

Introducing btcrpcclient — Bitcoin RPC Made Easy

Bitcoin JSON-RPC

Bitcoin JSON-RPC

Have you been looking for a robust and easy to use way to interface with Bitcoin through the JSON-RPC API? We’ve got you covered!

We’re excited to announce btcrpcclient, a new Websocket-enabled Bitcoin JSON-RPC client package written in Go. This package allows you to quickly create robust Bitcoin RPC clients in just a few minutes.

Major Features of the btcrpcclient Package

  • Supports Websockets (btcd and btcwallet) and HTTP POST mode (Bitcoin Core)
  • Provides callback and registration functions for btcd and btcwallet notifications
  • Supports btcd and btcwallet extensions
  • Translates to and from high-level statically-typed Go types
  • Offers a synchronous (blocking) and asynchronous (non-blocking) API
  • When running in Websockets mode (the default):
    • Provides automatic reconnect handling (can be disabled if desired)
    • Outstanding commands are automatically reissued on reconnect
    • Registered notifications are automatically re-registered on reconnect
    • Back-off support on reconnect attempts

Continue reading

Btcd + getwork + cgminer = profit

Bitcoins

Bitcoins

We are pleased to announce that btcd, our full-node bitcoind (bitcoin core) alternative written in Go, now has support for the getwork RPC which allows it to function with cgminer!

We have extensively tested the code on testnet. The following is a sampling of the latest blocks we’ve generated on testnet using btcd + cgminer: 226713 226830 227228 227390 227393.

If you are simply interested in learning how to configure btcd to work with cgminer, you can view the setup instructions on the btcd wiki, however if you’re interested to learn about some of the nuts and bolts that make it all work, read on.

Isn’t getwork deprecated?

Before I dig into some of the details, I would like to touch on a topic that I’m sure some astute readers will undoubtedly be asking, which is “Why implement getwork when it has been deprecated in favor of getblocktemplate (BIP0022 and BIP0023)?”

There are a few reasons why we chose to implement it first:

  • Supporting getwork was a lot less time consuming than getblocktemplate proposals and software such as cgminer still functions properly with getwork. This means we were able to release support for mining more quickly by supporting getwork first.
  • The bulk of the work involved was the code to create and manipulate block templates which is used by both getwork and getblocktemplate. Thus it helps pave the way to supporting getblocktemplate proposals while allowing getwork to function in the mean time.
  • The current reference implementation (bitcoin core) still supports getwork. Even though it will likely be removed in the next version, one of the goals of btcd is to be a drop-in alternative that can be used with existing infrastructures.

Continue reading

2014 Summer Internships: hacking on Bitcoin with Go

We are pleased to announce our 2014 summer internship program at Conformal Systems. A short summary of the program is as follows:
hacking Bitcoin with Go

  • 2 full-time intern positions are available for the summer of 2014.
  • Interns will be expected to work on remote.
  • Interns can be from any country or timezone, but timezone may be a factor in intern selection due to synchronization issues.
  • Interns will be expected to speak and write in English at a conversant but not fluent level.
  • Interns will work on open source Bitcoin-related projects the entire summer.
  • Interns will be expected to develop exclusively in the Go programming language.
  • Interns are expected to have the “Minimum Recommended Skillset” from the code contribution guidelines for the btcd suite.
  • Interns can either (A) work on projects assigned by Conformal Systems staff or (B) “bring your own project” (BYOP) which we approve as part of the internship.
  • The internships are open to anyone currently attending undergraduate or graduate school, or anyone of similar age. Broadly speaking, we consider anyone age 15 and up to be eligible.
  • Pay will be roughly a “graduate student” level, approximately USD 2,000 (pre-tax) per month.

Continue reading

Redecentralization: building a robust cryptocurrency developer network

As it was originally proposed by Satoshi Nakamoto, Bitcoin was conceived of as a peer-to-peer system that was fundamentally decentralized.  Many past and current discussions about the future of Bitcoin acknowledge that too much centralization is a threat to the Bitcoin network.  To some extent this process of avoiding centralization has been successful, but in several key areas it has not been very successful and it is in need of redecentralization.  The most pressing case for redecentralization in Bitcoin (and cryptocurrencies more generally) is the current development community:

  • the number of developers who have experience developing cryptocurrencies is very low, I would estimate there are less than 100 such developers
  • the majority of the developers are located in the US and other “Western” nations
  • the developer incentive structure with successful cryptocurrencies creates conflicts of interest

In what follows, I propose solutions to these issues cited above.  Since we develop our own full-node Bitcoin implementation, btcd, the criticism of the developer community status quo that follows applies not only to other developers, but also our developers who work on btcd.

Continue reading

Deslugging in Go with pprof: btcd

As btcd nears completion we decided to have at least one round of deslugging because we were much slower then bitcoind during chain download. Let me clarify that term for the ones that don’t know what that means. Deslugging is the art of measuring (or profiling) run times of individual functions, determining which ones are slugs and which ones could be optimized. Armed with that data one attacks said functions to see if there are some less then optimal algorithms. Surprisingly enough, this is often the case. An industry truism is “measuring is knowing” and attacking functions based on gut feeling often does not yield satisfying results. As always with these exercises, some things work and some don’t. Throughout this blog I’d like to walk our readers through some optimizations that succeeded, and one, objectively, failed attempt.

Continue reading