Crowdsourcing and its limits

Books and articles about crowdsourcing often begin with Francis Galton and his visit to the West of England Fat Stock and Poultry Exhibition. Think of it as a state county fair. There is typically a competition to guess the weight of a cow, bull or ox. Apparently 800 people competed, including butchers and farmers who might be knowledgeable about the weight of a large ox; but there were plenty of non-experts who might guess wildly at best.

This opening story is used by James Suriowiecki in The Wisdom of Crowds, subtitled “Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, and Nations”. But Suriowiecki doesn’t just trumpet the benefits of crowdsourcing. He also discusses its limits, including where a crowd can turn single-mindedly pathological. The Wild Horde with the Mob Mentality.

But let’s get back to Galton’s story. The average guess of the crowd turned out to be 1,197 pounds, just one pound shy of the ox’s actual weight. Pooling its results together, the crowd does shockingly well. Galton was shocked. His hypothesis was that the “average” voter was an ignoramus and that the “average” guess would be wildly off. Suriowiecki writes: “After all, mix a few very smart people with some mediocre people and a lot of dumb people, and it seems likely you’d end up with a dumb answer.” Are experts over-rated?

Who cares about the weight of an ox? Or the number of jellybeans in a jar? Maybe you would if there was prize money involved. Although sometimes dangling the reward of prize money makes the average guess worse, not to mention the individual guesses. But what about betting on horse-racing or football games? How are those betting odds or point spreads calculated? You might be surprised to know that in today’s world wide web of electronic wagering, that the wisdom of crowds is built into the system. Bookies crowdsourced before there was an internet, and you can bet they leveraged lightning speeds and crowd access as it became available. You’ll have to read The Wisdom of Crowds to learn more. As a bonus, you’ll also learn about how crowdsourcing impacts traffic patterns, intelligence spooks, Linux, movie ticket prices, Google search results, financial bubbles, and tipping practices.

To tackle complex problems with no easy or obvious solutions, Suriowiecki argues for three essential ingredients in wise crowdsourcing: diversity, independence, decentralization. Diversity promotes the generating of a wide range of ideas and solutions, some of which might be out-of-the-box, yet applicable. Independence avoids the pitfall of groupthink. To get the best out of Diversity, there needs to be Independent generating of ideas. Suriowiecki provocatively suggests that “Diversity and Independence are important because the best collective decisions are the product of disagreement and contest, not consensus or compromise.” We’ll revisit this idea momentarily.

The third ingredient, Decentralization, makes its power felt when it comes to the actual process of problem-solving. It has pros and cons. According to Suriowiecki: “Decentralization’s great strength is that it encourages independence and specialization on the one hand while still allowing people to coordinate their activities and solve difficult problems on the other. Decentralization’s great weakness is that there’s no guarantee that valuable information never gets disseminated, making it less useful than it otherwise would be. What you want is individuals to specialize and to acquire local knowledge – which increases the total amount of information available in the system – while also being able to aggregate that local knowledge and information into a collective whole…”

Interestingly, or oddly, The Wisdom of Crowds does not address higher education. There is a chapter on science, that illustrates the power of the three essential ingredients in combating the SARS viral outbreak. You might think that universities provide an ideal mix of diversity, independence and decentralization, particularly in the U.S. (Higher education can be much more hierarchical in many other countries.) The university consists of a diverse group of people, at least in terms of field-of-expertise, and also includes many novices (students). Some universities are the microcosm of a small town: there are campus police, electricians, folks in food service, gardeners, athletics coaches, among many other roles.

Independence is less obviously true across the university, although faculty members are largely fiercely independent by training. As a faculty member, part of my role is to train students to think independently, critically, and hopefully with nuance and wisdom. Independence of thought, and the freedom to study whatever one chooses, is or should be the hallmark of an institute of higher education. The ethos of independence is present, although the reality may differ across the institution. Due to this ethos, coupled with the diversity in the university, decentralization was a feature. Different departments operated in their own idiosyncratic way. I use the past tense because decentralization is dying in higher education, with the rise of the all-administrative university.

Perhaps there is a dis-alignment of purposes. The primary goal of the university (from my point of view as an academic) is the dissemination of the best knowledge available, both old and new. For some universities, generation of new knowledge is an equally important goal. For other institutions, preservation of knowledge may take great importance. But the signals I receive from an ever-increasing centralized administration seem to be more about how to survive (let alone thrive) in the ever-increasingly-competitive industry of higher education. There is also an aversion to disagreement and contest as providing the best outcomes for collective decision-making. I hear much more the language of consensus and compromise. Not that these are bad per se, but it’s unclear to me that university leadership tries its best to leverage the wisdom of its crowd, when it tries to avoid argumentative faculty members. Those many surveys feel like lip service information gathering, rather than truly aggregating local expertise into a collective whole.

Departments in a university can be considered as small groups, particularly if you’re at a smaller college. So for the most part, our day-to-day life revolves around small group decision-making. Suriowiecki discusses the pros and cons of small groups, providing examples of good and bad decision-making. While group wisdom can be enormously beneficial, “many groups struggle to make even mediocre decisions, while others wreak havoc with their bad judgment”. I’m very thankful to be a very-well functioning department. I’ve read about many dysfunctional ones. “Groups benefit from members talking to and learning from each other, but too much communication, paradoxically, can actually make the group as a whole less intelligent.” There are limits, and one of the large downsides can be meandering, inefficient, and ultimately poor decision-making. There’s also the danger that “aggregating individual decisions produces a collective decision that’s utterly irrational.” The ugly single-minded mob is as scary as it sounds.

The Wisdom of Crowds covers some of the same ground as The Starfish and the Spider. The latter is a breezier, lighter read. Suriowiecki’s examples have more detail and analysis, and he includes a number of counter-examples to help the reader see the ever-lurking danger of mob mentality. While the stories might now be a bit dated given the exponential increasing in internet crowdsourcing (the book was published in 2004), the principles haven’t changed be it the advantages or the dangers. I recommend it. (And yes, we did crowdsource a porchetta recipe for Christmas a few years ago!)