Weapons of Math Destruction lifts the veil on the darker side of Big Data. In ten chapters
of vignettes, the author Cathy O’Neil appropriates the acronym WMD bringing it
very close to home to anyone touched by the new economy – a globalized
technocentric world order. O’Neil has had a varied career. With a Ph.D. in
Mathematics from Harvard, she started out as an academic at Barnard. But then
the siren song of investment banking and hedge funds called out to her, and she
joined D.E. Shaw as a quant. (Having a computational background, I was
once tempted to apply to D.E. Shaw earlier in my academic career, but I think
I’m much happier teaching college students.) Disillusioned by the financial
crisis, she “was especially disappointed in the part that mathematics had
played [and] was forced to confront the ugly truth: people had deliberately
wielded formulas to impress rather than clarify.” Leaving the hedge fund, she
has worked on illuminating WMDs, and also started the Lede Program in Data
Journalism in Columbia.
The chapters include (1) a peek at predictive crime models
that can create pernicious feedback loops, (2) personality “tests” on job
applicants that lead to new automated forms of discrimination, (3) corporate efficiency
schedulers that disenfranchise employees, (4) using credit reports as proxies
for all manner of future predicted risk behavior, (5) large-scale social media
manipulation, and more. If any of these topics interest you, I highly recommend
reading her book in full. The two topics I will highlight in today’s post are
Chapter 3 (“Arms Race”) and Chapter 4 (“Propaganda Machine”).
As a faculty member in a private liberal arts college in the
U.S., I have grown increasingly concerned with the unsustainable arms race that
has accelerated in earnest over the last 5-10 years. Holding down tuition is
increasingly difficult as institutions fight for the small swath of students
that will increase their prominence so as to keep themselves afloat.
Administrative staff must be hired to support the ever-increasing facilities
and amenities in an effort to be a full-service college. Increasing mental
health concerns among students are putting strains on universities struggling
to provide adequate support. The current strategy of discounting to draw more
desirable students is financially unsustainable, and yet the arms race
continues.
O’Neil begins this story in 1983 with U.S. News and World
Report’s first set of college rankings. It says something that in 2017, the “rankings”
have expanded to cover more types of institutions and programs, while its
original news-magazine roots have died with the times. Top Ten lists have
always been popular, but Big Data, with its whiff of being scientifically
based, has turned these lists into a gargantuan beast of its own. The siren
call of Big Data is strong, perhaps overwhelmingly so. With more data,
we should be able to make better decisions. It is a strategy that runs well
when making a pitch to administrators for more money; I’ve used it myself
multiple times. The larger and more complex a system becomes, the more it needs
to feed on Big Data to optimize efficiency. We humans might say to ourselves
that we are the overlords of the system, but more and more are becoming enslaved.
How exactly? Thanks to the correlation between increasing complexity and
obtuseness, things are becoming less and less clear. That’s why O’Neil’s book
is important in highlighting these WMDs.
Everything starts with a model. Ideally, and this is most true
in the “hard” sciences, the feedback loop in developing a model is much more
rigorous. You test the model with experiments, generate more data, and then
refine the model using the new data. Rinse. Repeat. The models that work best
have narrow and clearly defined outcomes. But many complex problems are
“squishier” (O’Neil’s term), and the journalists tasked with ranking colleges
were trying to measure something as nebulous as “education excellence”. O’Neil
writes: “They had no direct way to quantify how a four-year process affected
one single student, much less tens of millions of them.”
So what do most people do in this case? You pick a proxy.
Now hopefully you choose a proxy that correlates well with what you’re trying
to measure, and you provide appropriate feedback into your model. In the
college rankings game the proxies that “seemed to correlate with success”: SAT
scores, student-teacher ratios, acceptance rates, retention rates, alumni
giving, and peer evaluations. With fame or infamy come problems. O’Neil
explains: “As the ranking grew into a national standard, a vicious feedback
loop materialized. The trouble was that the rankings were self-reinforcing. If
a college fared badly in U.S. News, its reputation would suffer, and conditions
would deteriorate. Top students would avoid it, as would top professors. Alumni
would howl and cut back on conditions. The ranking would tumble further. The
ranking, in short, was destiny.” In recent years we’ve heard a number of
stories exposing institutions that attempted to game the system. What a sad
state of affairs.
It gets worse. O’Neil explains that to establish initial
credibility, the early lists needed to have the known “elites” on top. What
made them special? High SAT scores, great graduation rates, excellent
retention, strong giving from rich alumni, and name-recognition among peers.
Sound familiar? Glaringly, cost of attendance was not included in the formula
of the model. Perniciously, this led to gaming the system without needing to
keep tuition down. In O’Neil’s words, “in fact, if they raised prices, they’d
have more resources for addressing the areas where they were being measured.”
Tuition has skyrocketed in the last twenty years – a correlation, at least, if
not one of the causes.
As the arms race increases the gap between the haves and
have-nots, the next WMD comes into play: the propaganda machine of for-profit
higher education advertising. Big data has sharpened this model with deadliness
in targeting the most needy with the taglines most likely to succeed in prying
cash they don’t have from their wallets. Uncle Sam steps in with federal loans
to make up the difference, up to a whopping ninety percent! O’Neil writes:
“Anywhere you find the combination of great need and ignorance, you’ll likely see
predatory ads… they zero in on the most desperate among us at enormous scale.”
That last phrase is what defines a WMD, the enormous scale. Tuition is set to
maximize borrowing at the limits of federal loans. But, that’s not so important
if you’re selling the dream of upward mobility. Society’s veneration of the
entrepreneur, coupled with well-chosen anecdotal stories, provide what may
sound like the only hope of those in difficult circumstances. No risk, no
payoff.
The details of this sad story are told in Lower Ed: The Troubling Rise of For-Profit
Colleges in the New Economy by Tressie McMillan Cottom. Before she
became a sociologist and professor, Cottom worked as a recruiter and enrollment
specialist in the for-profit world. She eschews the simple tropes used to
explain what is happening at the other end of the spectrum, far away from the
noses of the “elites”. Her analysis is penetrating, and I strongly recommend
her book if you want to know how and why Lower
Ed began its thriving ascent. As they add a slew of graduate programs and
post-baccalaureate certifications to their programs, their nimbleness is
outpacing traditional public and private institutions of higher education.
There is no simple way to assign blame, because this complex issue is part of a
much larger ecosystem
Here’s a quote from the Epilogue that summarizes one of the
main issues. “In the absence of social policy, public subsidies to Lower Ed
become a negative social insurance program. A negative social insurance program
is a market-based response to collective social conditions. Negative social
insurance, unlike actual social insurance programs (e.g. Social Security),
doesn’t actually make us more secure. It only makes our collective insecurity profitable.”
Given the limited information I’ve seen in the current (U.S.) Administration’s higher
education stance, the gap between the haves and have-nots will widen further.
For-profits aren’t going to bridge this gap but instead will “perpetuate
long-standing inequalities”. I’m not doing justice to Cottom’s careful
argument; I recommend reading her book in full.
O’Neil closes her book by reminding us that Big Data is here
to stay. “Predictive models are, increasingly, the tools we will be relying on
to run our institutions, deploy our resources, and manage our lives. [But]
these models are constructed not just from data but from the choices we make
about which data to pay attention to – and which to leave out. Those choices
are not just about logistics, profits, and efficiency. They are fundamentally
moral. If we back away from them and treat mathematical models as a neutral and
inevitable force, like the weather or the tides, we abdicate our
responsibility. And the result, as we’ve seen, is WMDs that treat us like
machine parts in the workplace, that blackball employees and feast on
inequities… Math deserves much better than WMDs and democracy does too.”