by James Surowiecki
How can a crowd correctly guess the number of beans in a jar, time after time, far more accurately than the average of the best guesser? How can all of us be smarter than the smartest of us? Especially when “all of us” includes some really dumb people?
That’s the question that James Surowiecki attacks in The Wisdom of Crowds, a deceptively short book packed with examples of collective wisdom.
The crowd collectively is smarter than any of its individuals – even the smartest. While there will always be someone smarter than the crowd in any given run, those individuals will vary from run to run, not able to consistently repeat their success.
In order to harness this collective knowledge, three conditions must obtain: 1) there must be a means of aggregating the results, 2) individual decisions must be made independently, and 3) the decisions must be unbiased, uninfluenced by an outside bias pushing the crowd in one direction. The first condition makes the results useful, the second keeps the crowd from turning into a mob, and the third prevents a dictator from manipulating that mob.
Surowiecki also identifies three different kinds of problems: cognition, coordination, and cooperation. Crowds are best at solving cognitive problems on their own. Coordination problems require a feedback mechanism, and cooperation problems, by far the most interesting, can require an entire social structure to enforce certain norms and incentives.
The first half of the book lays out the definitions and examples, the second provides increasingly complex applications.
Surowiecki discusses electronic markets, like Tradesports and the Iowa Electronic Market, supplementing data with anecdotal evidence. For instance, the very day of the Challenger Shuttle loss, Morton Thiokol stock dropped like a rock, even though dozens of contractors were involved in the shuttle program. Somehow, the market correctly aggregated the available information to arrive at the culprit.
I’d point out that the problem with anecdotal evidence is that it tends to be … selective. Surowiecki does deal with bubbles at length later in the book, but this is a different matter. The shuttle incident is supposed to provide evidence of the market’s ability to accurately aggregate information. But certainly there are times when the market is surprised by new information. While a comprehensive statistical analysis is well outside the scope of the book, one would expect counterexamples to be numerous and mentioned.
He also makes a spirited defense PAM, the abortive attempt by the US intelligence and defense community to harness such markets for intelligence purposes. One sympathizes with him here. No system in perfect, and one as high-profile as PAM would have attracted both sharpies and enemies attempting to game it. Still, with little money at stake, the outrage and horror expressed by certain US politicians may well have cost us a good source of information. One wonders if those morally offended by such markets are really just morally offended by markets.
Another interesting argument Surowiecki makes is in favor of shorting stocks – despite the regulatory, financial, and societal restraints on doing so -, in order to filter out the upward bias in stock prices (p. 227):
We know that the crowds that make the best collective judgments are crowds where there’s a wide range of opinions and diverse sources of information, where people’s biases can cancel themselves out, rather than reinforcing each other. If a company’s stock price, as we’ve seen, represents a weighted average of investors’ judgments, it’s more likely to be accurate if those investors aren’t all cut from the same cloth.
The case is compelling, but he ignores at least a couple of other reasons why short-selling isn’t more pervasive. First, it can be very hard to find someone to loan you the shares to sell. Second, if you’re an analyst, and you work hard, and do your do diligence, and go to meet a company’s management and find out about what they’re about, and still decide to short the stock, you make it harder for that company to raise capital, much harder than if you simply issued a “hold” recommendation. And maybe that management will never, ever talk to you again. There are good societal reasons working the other way, too.
Surowiecki also makes a logical error, arguing that rising stock prices aren’t necessarily a good thing, after all, look at Enron, and wouldn’t we have been better off if its price hadn’t gone up? Well, rising stock prices reflect a rising national economy, as any cursory comparison of 2005 and 1905 living standards will tell you. When people talk about rising stock prices, they’re generally speaking of aggregates over long times, not about accounting and financial abdication of responsibility.
Interestingly, companies, who gain the most from markets, are among the quickest to abandon collective decision-making in their own strategies and operations. Only a few companies, with very strong corporate cultures, such as IDEO, have managed to make it work. In IDEO’s case, it begins with unconstrained, non-judgmental brainstorming to solve problems, and pervades the entire company.
And this comes to social construct and social restraints. None of this works in a society without trust. Without going into great detail, the argument here goes that societal norms can be enforced, and form the background against which we operate. Since cooperation can’t be fairly coerced most of the time, society needs to subtly encourage it. It also suggests that the biggest cost of lawsuits isn’t the fees or even the fines, but the loss of trust that makes it possible to do deals in the first place, and the extra expense of proactively defending against every conceivable after-the-fact objection.
Like most interesting books, this one does overstate its case from time to time. There’s a fine line among collective wisdom, trusting your subordinates, and a society that works, and sometimes Surowiecki seems to cross it. Take Saturn’s much-discussed production line emergency rip-cord. Any worker on the line can pull the cord and stop production, if necessary. Is it really the Wisdom of Crowds that makes that work? The US military has always relied on platoon-level initiative. But it would hardly do to have Lt. Colonels voting on strategy.
He closes with an appeal that liberal democracy is collective wisdom applied to politics. It’s not a bad extension, especially when one considers that democracies do make mistakes, such as being fully unprepared for war. In the past, some have used this as an excuse to attack capitalism, although the real culprit was pluralism. In fact, democracies work quite well during peactime, just as markets work quite well in the absence of shocks. Perhaps the opening stages of wars are analagous to bubble or crashes, when the assumptions about how crowds function suddenly and catastrophically fail to obtain. This is most emphatically not an argument either against democracy or for martial law. It is a suggestion that Surowiecki’s analysis supports the conclusion that there are very limited times when normal political structures don’t work, and some level of command is necessary.
Still, experience bears out the overall principle, and an appealing principle it is. That within certain social constraints, we can all contribute simply by doing our best. Even if sometimes that’s not very good.