Studying Bull

I recently watched the movie Green Book. There’s a scene where Tony Lip is proud of his ability to BS his way through different situations. It reminded me of an article published last month by the Institute of Labor Economics. The three authors are academics: John Jerrim, Phil Parker and Nikki Shure. The title and abstract are shown below.

Yes, people study BS. Per the typical academic article, it provides a history of previous BS studies. There are many ‘over’-statement definitions in the field. Over-confidence. Over-claiming. Over-estimation. Over-placement. Over-precision. You find what you might expect: “participants are more likely to bullshit when they believe they can get away with it, and are less likely to bullshit when they know they will be held accountable for the responses they provide”. There’s even a Bullshit Receptivity scale.

Compared to previous studies, the present one has a much larger sample size (40,000+ respondents from nine Anglophone countries) because it uses data from PISA 2012. That’s how I stumbled on this report, since I follow educational developments and I’ve looked at the latest PISA results with respect to scientific inquiry. PISA is aimed at 15-year-olds and measures skills in mathematics, reading and science. Since PISA 2012 had an additional focus on math, there were extra questionnaire items that allowed researchers to crunch data for the present study.

Classifying students’ propensity for BS was based on a list of 16 items where they rated whether they had ‘never heard of it’ up to ‘know it well, understand the concept’ on a five-point scale. Some items on this list were nonsense. Students also answered Likert-scale questions that provided measures of (1) self-efficacy, (2) self-belief in problem-solving ability, (3) self-reported popularity at school, and (4) self-reported measures of perseverance. The self-efficacy questions were aimed at gauging how comfortable students were with a math-related task, such as ‘calculating how many square metres of tiles you need to cover a floor’ or ‘finding the actual distance between two places on a map with a 1:10,000 scale’ or ‘solving an equation like 2(x+3) = (x+3)(x-3). And of course, there are the actual PISA results in mathematics so we know how well students actually performed.

The results are interesting. Teenagers in the U.S. and Canada are more likely to be BS-ers than in Britain (England, Ireland, N. Ireland, Scotland, Wales) or Australia and New Zealand. Along gender lines, boys tend to be BSers more than girls; effect sizes were quite large in this area for Britain and smaller for North America. Effect sizes were also significant comparing socio-economic status (a standard PISA measure) – teenagers from higher socio-economic status showed a larger preponderance for BS.

Not surprisingly, BSers show higher self-efficacy scores and have higher self-views of their problem-solving abilities relative to how they actually performed on PISA. They also viewed themselves as more likely to persevere when confronted with a difficult task (unclear if they would actually do so) and tend to believe they are more popular at school. The effect sizes are smaller in the latter case compared to the former. When given ‘approach to problem solving’ tasks, they tend to pick conventional ‘safe’ answers rather than try to seem less knowledgeable (again, unclear if this is what they would actually do).

The authors acknowledge limitations of their study. BS in one area (math) may not overlap in a simple way with other areas. The data is a one-time snapshot; BS behavior likely changes over time within an individual. Also, the “social and labor market outcomes” and implications of BSers remains unclear. Does it help? Does it hurt? Don’t know.

Do BSers reflect on their self-efficacy on any related traits? Do they recognize their own BS? Or does repetition inure one to that fact such that you start believing in your own imagined abilities? I don’t know. There’s certainly self-help advice out there in the ‘fake it until you make it’ variety. One thing I try to get across to students is how to avoid self-deception in thinking you know more than you actually do when preparing for exams. As a result, I’ve included more metacognitive exercises and introduced annotated self-grading. Perhaps I am in an area where it’s much more difficult to BS on an exam. You know it, or you don’t. And it shows up clearly.

One useful reminder to myself: Domain knowledge and critical thinking in one area doesn’t necessarily extend to other areas, particularly if they are less proximate. I might know a lot about chemistry, or maybe my particular subfield of chemistry, but I should be wary of the temptation to pontificate in other areas. When you’re in a profession where you’re constantly viewed (by students) as an authoritative source, it’s easy to slip into professing BS, and not realize it.