The Wisdom of Crowds

On my flight to Baltimore about two months ago I read The Wisdom of Crowds by James Surowiecki. I actually didn’t plan to buy the book–I just saw an autographed copy at Kepler’s and picked it up on impulse.

The first page of the introduction sucked me into wild intellectual romp from which I’m still recovering.

Since that flight I’ve recommended it to dozens of people and purchased it for two (to whom I owed a book). I’ve been meaning to write about it ever since, but I kept getting distracted. Plus I saw that two of the bloggers I read commented on it: Jordon Cooper and Todd Hunter (who commented not once, not twice, but thrice), so I knew the book was getting the buzz it deserved. 

So what’s the big deal? What’s the idea that is still rocking my world? Simply this: given the right conditions, diverse groups of people collectively solve certain types of problems better than experts.

This isn’t a bolt from the blue: the basic idea has been kicking around for a long time, but the book is magnificent nonetheless. The anecdotes are precise and illuminating, the data is detailed, documented, and convincing, and the writing sparkles.

What Problems Do Groups Solve Better?
There are some problems you need experts to handle (problems of skill are the most important kind: landing a plane or operating on the brain are good examples), but there are several broad types of problems that groups tend to outperform experts on:

1. Cognition Problems: questions with factual answers
   How many jelly beans are in a jar?
   Where is a sunken submarine?
2. Coordination Problems: how do we all work together when it’s in our best interest to do so?
   How can we drive safely in heavy traffic?
   How should we deliver this product to market?
3. Cooperation Problems: how do we work together when we have divergent goals and values?
   How can we control pollution while promoting industry?
   How can borrowers get money from lenders at the best rate for each?

Under What Conditions Do Groups Solve These Problems Better?

There are four key qualities that make a crowd smart. It needs to be diverse, so that people are bringing different pieces of information to the table. It needs to be decentralized, so that no one at the top is dictating the crowd’s answer. It needs a way of summarizing people’s opinions into one collective verdict. And the people in the crowd need to be independent, so that they pay attention mostly to their own information, and not worrying about what everyone around them thinks. 
from The Wisdom of Crowds Q &A, emphasis added 

Note that these criteria (diversity, decentralization, aggregation, and independence) often tend to move us towards a solution that not everyone is happy with. In Surowiecki’s own words:

The wisdom of crowds isn’t about consensus. It really emerges from disagreement and even conflict. It’s what you might call the average opinion of the group, but it’s not an opinion that every one in the group can agree on. So that means you can’t find collective wisdom via compromise.
from The Wisdom of Crowds Q &A

What Can Go Wrong?
When any of the above criteria are not met, groups often perform abysmally worse than experts or even isolated idiots. Some specific challenges:

1. Cascades (p 40f and throughout the book): people imitate each other without understanding and everybody jumps off a cliff because all their friends did. Think about the stock market in the late 90s.
2. Groupthink (p 36): people don’t feel free to disagree and groups reach suboptimal decisions that almost everyone can see a problem with but no one is willing to comment on. This is one of the cardinal sins of the Assemblies of God, by the way.
3. Polarization (p184-190): people egg one another on until the entire group adopts a more radical view than any of the members would have advocated going in.

Summary Thoughts
Surowiecki’s real contribution, in my estimation, is detailing the criteria under which groups outperform experts and the conditions under which groups fail catastropically. 

Also, his endnotes rocked–they’re as good as the footnotes in Gordon Fee’s commentary on 1st Corinthians. If you read this book and didn’t read the notes, go back and read them right now!

The most stimulating idea in the entire book for me was using of artificial markets to predict future events (pages 17, 79, 103, 220–221, especially 278–280, and 285). I have no idea how it applies to my context, but it was a fascinating concept.

Learn More
You can read an excerpt from the book, read an article by the author or hear him discuss the book on NPR.