Blockchains and Hierarchy in the Forest
Dissecting the benefits of different kinds of decentralization
The last posts have been taking stock of the technical ideas that are most important for spanning a future world of cooperating and competing AI agents. I spent a few posts on formal verification, a personal specialty, but also highlighted cryptography. The public at large today most associates cryptography with blockchains, so it may have seemed strange that I didn’t cover that topic. The reason was a sort of multilayered complexity of technology and motivations, complex enough to deserve its own post… and here it is.
Am I betting on an important role for blockchains in the future we’re planning? It depends on how we resolve the terminology. There’s a lot of potential for better organization through cryptography and distributed systems, but the details matter. Let me give my summary of defensible meanings of “blockchains,” draw attention to an underappreciated sister technology, and then pull in some evolutionary psychology to highlight a systematic bias in how many people resolve the engineering tradeoffs of the domain.
What is a Blockchain?
Sometimes terminology with a clear meaning for some group of specialists gets coopted into another meaning, by the sheer force of adoption by a much larger audience. The word “hacker” originally refer to someone skilled at problem-solving creatively with computers, but it was coopted to refer to someone who specializes in finding security vulnerabilities. The academic and practical field of cryptography has a history of many decades, but, shortened to “crypto,” it was recently coopted to refer to a particular kind of distributed system. The word “blockchain” doesn’t work too much better, since different commentators assign it such a wide range of meanings.
But let me take a crack at characterizing the true scope. It seems roughly to come out to “a system sufficiently like bitcoin or Ethereum.” Now we’ve passed the buck to deciphering “sufficiently like,” so I’ll get more specific, at the cost of using jargon that I’ll explain right afterward. “Blockchain” and “crypto” seem to refer to a distributed ledger implemented with Byzantine consensus.
Let’s start with what a ledger is: a log of transactions that only grows (at its end) and never shrinks (i.e. never has old entries removed). A ledger is relatively easy to set up on the Internet.
We see many parties interacting with the ledger by sending in transaction descriptions. The system (visualized as a clerk here) validates transactions (e.g. checks that someone sending money has enough funds available) and adds the valid ones to the ledger. It’s a pretty easy job as such things go, since the clerk is centralized: only one exists for the whole financial network. Requests are conveyed to the one clerk, and his responses are conveyed back out, through the Internet or a similar network.
It helps ground our discussion to pause here and quantify how easy the job of the centralized clerk really is (if we ignore, for the moment, additional requirements that are coming in the next paragraph). Worldwide, there are about 25,000 credit card transactions per second. Each transaction is pretty simple: its description includes buyer, seller, amount, perhaps cryptographic credentials, and probably not much other essential information, so 1,000 bytes is probably more than enough to represent it. That means total traffic is about 25 megabytes per second, which should easily work over coffee shop WiFi. An unexceptional current-generation laptop sitting in that coffee shop could also keep up with that full demand, consulting a local ledger.
Now, a real financial network has other requirements. Some of them, like antifraud protections, are also not tackled by canonical blockchains, so let’s set those aside. We want to protect against failures of the servers, including in the face of natural disasters, so we’ll need a few servers in each of several geographic regions. We’ll want to be sure to store ledger information reliably and redundantly, and we’ll want to make sure to protect cryptographic keys. It should still suffice to run on the order of 10 middle-of-the-road servers, distributed geographically. With standard cloud services, the cost is easily under $1 million per year.
Imagine that $1 million distributed evenly among all credit-card transactions, about 800 billion annually. A fee in the neighborhood of a ten-thousandth of a cent per transaction is enough to cover the technical infrastructure. The fact that we see much higher fees charged to merchants by credit-card networks today is one motivation for adopting alternative systems. There are also concerns that centralized infrastructure providers will exercise judgment poorly, say about which customers deserve credit cards or about which businesses are legitimate and worthy of the ability to make charges.
These complaints are major sources of motivation for blockchain systems, which implement distributed ledgers. That is, there is no longer any centralized ledger or clerk, and instead relevant information is distributed across many participants, who are somehow forced to carry out the right protocol to maintain what is, logically speaking, one shared ledger.
It’s easier said than done to simulate one shared ledger in the face of simultaneous action by many parties who don’t all trust each other. Making it work for bitcoin and friends has required major technical breakthroughs, of, I should add, a kind clearly deserving of further research. One common technique that gives blockchain its name has to do with how to maintain a truly append-only ledger, where any participant can detect if someone tries to “rewrite history” by modifying old entries. The ledger entries form a chain of blocks, and every sequence of blocks includes a cryptographic hash of the full ledger content up to that point (really some cryptographically secure summary of it), allowing easy verification that only true contents are represented in the ledger. (Typical approaches are just a little more complicated at the level of this diagram, using an idea called Merkle trees.)
We still need to figure out which new blocks are added when. That’s where we come to Byzantine consensus protocols: ways that mutually distrusting parties can come to common decisions. Common tricks underlying those protocols include proof of work, which allows decisions to be made by parties who are first to solve hard computational problems; and proof of stake, which supports a pattern like for shareholders in companies getting votes in proportion to their holdings. I won’t dwell on the technical details, except to note that they’re quite computationally expensive. We can consult a comparison of major blockchains and find that bitcoin and Ethereum are in the neighborhood of 10 transactions per second (1000X below the global credit-card network), while the top performers like Solana are in the neighborhood of 1000 transactions per second (only about 10X below credit cards). Blockchains do charge explicit transaction fees, in the neighborhood of a few dollars for bitcoin, a variable range around one cent to $10 and up for Ethereum, and a bit under a thousandth of a cent for lean champion Solana. The options are still an order of magnitude (often multiple) off from credit-card-network-ready centralized ledgers in both dimensions.
The traditional financial system charges fees that can be in-between bitcoin/Ethereum and Solana. What accounts for the differences? Additional services are provided, like automatic fraud screening and manual investigation of claims of fraud, which can end in refunds being issued. “All sales are final” with a blockchain: the ledger is designed to be append-only. Traditional finance also implements government-mandated regimes like know your customer and anti-money laundering, which are designed to block certain criminal enterprises from the financial system (see Bits about Money for much more detail).
The crux of disagreements around the value of blockchain alternatives often comes down to disagreement about the value of these additional services, and we’ll come back to one unifying theory of where these preferences come from, via evolutionary psychology. First, though, let me mention an alternative technical approach that doesn’t seem to be as widely known as it deserves to be.
Certificate Transparency
Another approach developed at about the same time as bitcoin is certificate transparency, which arguably achieves most of the goals that motivate adoption of blockchains. It deploys a number of classic ideas from distributed systems developed significantly earlier. It doesn’t require a Byzantine consensus protocol and is, in general, much simpler and cheaper to maintain. The true motivation of certificate transparency is keeping certificate authorities from getting away with issuing inaccurate security certificates, which allow bad actors to pretend to be name-brand companies. We want to force all certificate issuances to appear in an append-only public ledger, so misbehaving issuers can’t hide from the evidence of their malfeasance. However, for simplicity in aligning with the prior section, let’s imagine a variant approach for multiple banks providing financial ledger services.
Certificate transparency works via servers publishing ledgers publicly as computer files. However, to make the explanation more accessible, I’ll start describing it in real-world physical terms.
Imagine that banks routinely post their ledgers on their front doors. (There are certainly privacy concerns, though many blockchains indeed retain these concerns by keeping their ledgers truly public, and others use techniques like zero-knowledge proofs to maintain anonymity.) A population of skeptical auditors is constantly circulating around, making copies of what banks have posted. Different auditors can bring their findings together into global ledger information, which can then be shown to others and analyzed to detect inconsistencies – like if a bank has tried to get away with removing or modifying an old entry. Note that, unlike in blockchains where all parties are exchanging information in all directions, certificate transparency has a clear hierarchical flow: a relatively small set of parties make the primary decisions, and many other parties can read and analyze the logs of those decisions.
Only small tweaks are needed for this approach to meet the requirements of certificate transparency. Assume each bank has a widely known public key, and it uses the same basic trick from blockchain ledgers, cryptographically signing all new entries in a way that guarantees outsiders can detect any after-the-fact modification of published entries. Ledgers are now posted on public web sites instead of bank front doors, and automated bots run by all sorts of organizations can pull ledgers repeatedly, checking for any attempted changes to old entries. Through this less tightly coupled flow, we arrive at the same level of guarantee from blockchains that transactions can’t be undone, assuming that failures from misbehavior and otherwise are relatively rare. (We may require a variation that parties don’t consider a transaction complete until multiple trustworthy auditors have acknowledged its receipt, and we can even consider taking a small performance hit, roughly the time for network round trips to all of those auditors, so that a bank itself doesn’t consider a transaction finalized until that point.)
That’s almost all there is to it to replicate the key benefits of blockchains as the term is usually used. Pretty much “for free,” we get the following benefits:
Flexibility: The whole protocol is new and can benefit from the latest information technology, without being tied to legacy systems (e.g., goodbye slow and expensive bank wire transfers).
Append-only integrity: The banks can’t edit their own ledgers without being detected.
Auditability: Everyone can audit ledgers to detect violations of rules and norms.
Redundancy: When a bank “goes down,” others are still operating and could even take over operating the same ledger, assuming they’ve been maintaining backup copies throughout.
Advanced features: Smart contracts are even easier to support with more centralization: both the main servers and the auditors run the contracts, the latter to confirm that they were run correctly to form the primary ledgers.
A few refinements of the idea can reintroduce familiar benefits of blockchains, with additional technical twists.
Leader election: The tricky part of assigning new responsibility for a failed bank is what’s called a leader-election protocol in distributed systems, to figure out who takes over the job upon failure, which requires basically the kind of complex protocol behind blockchains as popularly understood – it’s just that it only needs to take place at the moments of relatively rare failures. Such failures may even be rare enough to rely on succession orders curated in advance by committees of users.
Cross-bank transfers: We do usually want to have a way to transfer funds across banks, which gets tricky without one mega-bank that they all do business with, and the mega-bank could go down, too, though the mitigations of the last point still apply.
Misbehaving banks: Banks may start misbehaving, say refusing to accept certain transactions while publicly denying that they ever refused. If they selectively fail to add relevant records to their ledgers, what should users do? This scenario is also fairly easy to deal with, by letting users send their transactions to many intermediaries, which maintain their own ledgers of requested transactions that are passed on to the relevant banks. Now if a bank refuses to log a transaction that is clearly represented in many intermediary ledgers, it becomes possible (automatically or based on user action) to declare a bank misbehaving, triggering the above kind of handling to take over its ledger.
It’s worth noting at this point that this solution is more similar to permissioned blockchains than default bitcoin or Ethereum. The approach is still more centralized, with most participants auditing passively until finding rare rule violations.
In short, a centralized system fairly easily replicates the advantages of blockchains, so long as there is proper backing-up of its ledger, and so long as there are ready substitutes waiting in the wings. A single not-so-expensive server in this system can easily handle all requests worldwide (at least at current volumes), and it doesn’t take many geodistributed servers to be ready for the important failure cases. There are definitely real-world scenarios that justify the additional features of highly decentralized blockchains, but those scenarios arise relatively infrequently for, say, the population of people likely to read this post. So then why has the idea of highly decentralized blockchains been so popular? I think we find a good answer in evolutionary psychology.
Hierarchy in the Forest
In practice, the technically grounded argument for adopting a highly decentralized blockchain seems to come down to ability to resist an extremely well-resourced adversary who wants to tamper with the system. It really does take quite a lot of resources to subvert, say, ten different servers in different countries, chosen well to be independent of each other in the ways that matter. The canonical adversary of that style is a government, which leads to a reframing of the distinctive advantage as ability to resist oversight. The case for the benefits can be strong in, for instance, countries with authoritarian governments and therefore highly corrupt traditional finance systems. However, the rigorous case for most users in the developed world seems to be weak. Though it is not hard to find examples of centralized systems abusing power in any country, most users don’t seem to be genuinely worried about government imposing rules on financial institutions, and indeed they often appreciate affordances like banks working to reverse fraudulent transactions or catch money-launderers. Still, there is a persistent phenomenon of many people wanting to say that they are worried about that kind of interference.
We can find an explanation in the book Hierarchy in the Forest, which set out a framework for thinking about the conflict in human societies between top-down and egalitarian organization. The author Christopher Boehm argued that our brains have evolved specific mechanisms to resist domination, cases where our fellow humans use physical force or intimidation to compel our behavior. Our societies often do contain dominance hierarchies, but we also tend to construct what he called reverse dominance hierarchies, collaborations to exert pressure on would-be dominators to knock it off. Mechanisms like gossip and public ridicule are the weapons of a reverse dominance hierarchy. We can imagine the evolutionary pressures leading to development of parallel modes (dominance and reverse dominance), since the benefits of competent-if-forceful leaders are clear, but also individuals want to defend their own genetic destinies and not be too cowed into following orders.
The upshot is that human brains are always on-the-lookout for dominators that should be resisted. The idea of an important centralized system pattern-matches against that description. Whoever controls the one primary server of a relatively centralized ledger system has power over the users, and our primate brains get nervous about that power – even if there are relatively sophisticated technical approaches to dealing with malfeasance by the supposed dominators. It’s an unfortunate case of hyperactive misfiring of our mental anti-dominance circuitry, similar to some puzzles around the question of consciousness that I wrote about before.
The problem is exacerbated by our tendency to signal to each other through displays of competence, and one way to perform to impress others is to do an especially good job calling out would-be dominators. As a result, individuals can gain status in many communities by resisting technical approaches with centralization of compute, because those approaches pattern-match as dominance sources. Again, there can be sound technical explanations of how misbehavior by server maintainers can be handled quickly and effectively, but these explanations aren’t enough to get around the initial firing of our mental anti-dominance circuits.
Conclusion
Circling back to imagining future networks of AI agents cooperating and competing with each other, I don’t expect they will want to use bitcoin-style blockchains, though they will want to use several key ideas best-known from that style. This post’s running example of a simple financial ledger stands in for a whole set of possible applications that can actually be handled quite well by relatively centralized solutions. Ideas familiar from blockchains, around cryptography and distributed systems, can be used for relatively cheap detection of when central servers misbehave. Economic incentives should make that misbehavior relatively rare. However, when it does occur, failover to other servers can be straightforward, automatic, and quick. With the right tuning of the incentives side, such failovers should even be infrequent enough that handing off responsibility can follow user-curated succession lists, without requiring expensive Byzantine consensus.
We’ve almost finished touring through the most important technical ingredients in the approach I’m suggesting for future intelligence. Let me wrap it up with the next sequence of posts, covering one other important technology, deep learning, a valuable partner for the others, with interesting relative strengths and weaknesses.