Asking the Right Questions

In the 2004 movie I, Robot; detective Del Spooner investigates the death of Dr. Alfred Lanning by speaking to an A.I. incarnation of the deceased. You might think that the A.I. is some archaic early version that doesn’t know very much because when queried, Lanning’s hologram often replies: “My responses are limited. You must ask the right questions.” This is frustrating if you are the questioner.

I wonder how students would respond if I imitated Lanning’s hologram. Most of my students seem nervous about asking questions; there is often a self-effacing preface to the question. I’ve always encouraged students to ask questions as much as possible since it helps them to learn, and it helps me gauge where they might be confused. Sometimes my response is a clarification; sometimes I ask a question in return to sharpen a particular point. I’ve never said “That’s the wrong question.”

While there might not be any ‘wrong’ questions per se, quality may vary – some questions are better than others. I’ve been pondering this issue in relationship to what it means to learn a particular subject area. As an academic adviser, I regularly chat with my students about how their classes are going. One point I try to emphasize: Learning a subject area is about how to ask good questions about that field. Instead, students are more often concerned with finding the right answers. However, what’s more important at the tertiary level is asking the right questions.

So perhaps Dr. Lanning’s seeming unhelpfulness is a way to focus on what’s important: learning how to ask the right questions! A good question should lead to another good question as one deepens down the rabbit hole. What might such a series of questions be in my field?

What is chemistry?

A fundamental question. What makes chemistry distinct from biology or physics? My students have tackled the issue of definitions – and oh, what a tangle it is! While chemistry broadly deals with the interaction of matter, the conceptual crux of chemistry is relating these interactions down to the tiny scale of molecules. We’re talking nanometers here. It doesn’t help that we cannot see anything so small; we have to use models, analogies, and mental pictures, to help us understand what’s going on. There are small molecules, there are big molecules, but fundamentally chemistry is about molecules interacting with each other. That leads to the next question.

What is a molecule?

A molecule is two or more atoms ‘bonded’ together in a specific way. These connections between atoms lead to a three-dimensional structure that has specific properties. Chemistry takes place when molecules exchange atomic partners by breaking existing chemical bonds and making new ones. Bond-making and bond-breaking are at the heart of chemistry. The type of exchange that takes place is dependent on the overall structure of the molecule and also its specific bond connections. But this begs the question of what causes the exchange of atoms to occur.

What controls bond-making and bond-breaking?

Fundamentally this has to do with energy. But it’s tricky because energy is a slippery concept: We don’t really know what it is, but we can count it! I’ve rearranged my introductory chemistry course to center on the bond energy curve. We start with the generic picture of a chemical bond and then burrow our way down to what makes a chemical bond weaker or stronger. Later we talk about collisions between molecules and the sources of energy needed to break chemical bonds and start a chemical reaction.

That about sums up the first year of college chemistry. We build upon this foundation in subsequent courses. Or at least I hope we do so systematically. Folks in my department talk to each other regularly, and I think I have a good sense of what goes on in other classes (including ones I don’t teach) so that I can make forward-looking connections in my general chemistry course. In my physical chemistry courses, I can make backward-looking connections: “Remember when you saw X in G-Chem?” (Many students don’t remember, so reminding them often is very necessary.) “Now we can delve into why X…” then segues into the topic of the day.

I’ve started asking my physical chemistry students to write mock exam questions in an effort to help them prepare for the exam. I think there was a payoff to students who put in the effort to write good questions, but I haven’t assessed this systematically. I need to consider how to get students to pose those important fundamental conceptual questions. Maybe one test is that if the question leads to a very limited response – yes, no, maybe so – then perhaps it’s not a very good question, at least for fundamental learning rather than simple clarification. In that sense, Lanning’s canned statement is a little deeper than at first glance. “My responses are limited.” The sign of a less useful question perhaps? It requires more work to ask the right questions.

First Instinct Fallacy

You’re taking a multiple-choice exam. Looking at one of your answers, you begin to second-guess yourself. But you’re not still not sure. It seems like a 50-50 toss-up. Should you switch? Or should you stick with your first guess?

Most people prefer to stick rather than switch. Even though this is often detrimental to your exam score. Several studies have shown that switching from a wrong answer to a right answer is significantly more probable than switching from a right to wrong answer. Hence, those who switch score better on the exam, on average.

Yet the belief persists that it’s better not to switch. A Barron’s GRE Prep guide (from 2000) even reads: “Exercise great caution if you decide to change an answer. Experience indicates that many students who change answers change to the wrong answer.” This is known as the First Instinct Fallacy.

Why does the first instinct fallacy persist? That’s the subject of a 2005 paper by Kruger, Wirtz and Miller (Journal of Personality and Social Psychology, vol. 88, no. 5, pp. 725-735). The title and abstract are shown below.

The authors present experimental data arguing that this fallacy persists because when you switch from a right to a wrong answer, it is both more regrettable and more memorable, compared to when you didn’t switch (even though your answer was wrong and your second guess was the right answer although you didn’t ultimately choose it). This is one of those explanations that seems obvious after-the-fact, but in reality the causal link is not easy to tease out – hence the need for clever and sneaky experiments. I recommend reading the paper in full if you are interested in all the details.

I have not taken a high-stakes exam in a while that ‘counted’. (I do take the exams I have written to ensure they are of the appropriate length and difficulty.) But I do grade plenty of exams every semester. Very few of my questions are multiple choice or true-false, however I have observed many times the regret of students who seemingly believe the first instinct fallacy. When the graded exams are returned, there is plenty of groaning and ‘if only’ self-recriminations. I have very rarely observed a student bemoaning not changing their answer. Part of this has to do with attending only to what is easily observable in front of you, although there are occasions when a student writes out two solutions and then scratches out the first one.

Reflecting on what I do as a grader, I might be exacerbating the first instinct fallacy by explicitly writing comments about the student’s crossed-out first answer being correct. Perhaps I should stop doing so. I’m surprised I had not heard of the first instinct fallacy until alerted to it from a recent blog post by Tim Harford. I suspect that if I had taken an exam, I would respond with similar groaning as my students. More importantly, when taking an exam, I would likely also resist making a change in a second-guess situation – probably to my detriment.

I think I need to alert my students to this fallacy and add it to the list of learning tips and strategies in my syllabus. Since I am trying to improve student metacognition, I could couple it with the importance of looking over your exam after you think you’ve finished it. Would such an alert be of any use? The authors question this in a sobering final paragraph:

“The most obvious implication of this research, however, is that test takers should be warned that sticking with one’s first instinct is an ill-advised strategy. Even with such exhortations, however, people may be reluctant to switch as often as they should. Students who have been explicitly instructed to switch as often as they should. Students who have been explicitly instructed as to the invalidity of the first instinct theory are no more likely to change their answers – nor receive higher test scores – than test takers who do not receive instruction [many references included]. The (misleading) personal evidence garnered from a lifetime of test taking may be difficult for test takers to overcome.”

Human beings are not robots. Things that affect us more significantly, and lodge in our memory, can be blind spots that we’ve learned only too well.

Remaking the News

When I was young, our family subscribed to two newspapers. Not that there was much choice locally or nationally. I didn’t listen to the radio and we didn’t have a TV until I was in my mid-teens. TV news was read by newscasters with little supporting video; I found it boring and preferred the newspaper – mainly for the comics and the sports section.

Nowadays, I consume my news through the Internet. My main topic of interest is world affairs, and I typically read a variety of international sources – Reuters, Associated Press, BBC, Channel News Asia. I occasionally read articles from The New York Times (my spouse has a subscription), sometimes CNN, and I nibble on NPR and a smorgasbord of online articles that catch my eye. I rarely stream video as I much prefer reading text at my own pace. I avoided Buzzfeed. My social media usage is low, and I almost never click on any news links in that stream.

The death of traditional news media has been making the news regularly over the last decade or two. Fake news and alternative facts have muddied the waters, as a plethora of news ‘sources’ vie for your eyeballs on the Internet. Are there guardians of the truth any longer? Were there ever? What is truth anyway?

In her new book Merchants of Truth, Jill Abramson juxtaposes the stories of four news outfits: The New York Times, The Washington Post, Buzzfeed and Vice Media. Four different philosophies guided the rise of each, but as you would find in any good story, they face various travails and challenges, and each has evolved with the times to stay fresh, to stay afloat, and to keep you coming back to them. This is where things get interesting.

Before the heated competition of the Internet, the venerable New York Times kept its business arm separate from news content. It was considered a conflict of interest for the ad and revenue folks to strategize with the content folks – the journalists and editors, the marquee defenders and guardians of the truth. Buzzfeed’s strategy was to first steer eyeballs to its web content – by hiring quick-hack editors to speedily repackage original content from sources such as the New York Times, but with catchier headlines and ledes. Costs are low because you don’t need to pay journalists to find original news-breaking content. Once you have the eyeballs, you can steer ad revenue to your site.

The effectiveness of these upstart strategies, particularly in cooperation with Facebook and gaming Google page-rank, was devastating to traditional newspapers – many of which have closed shop in the last two decades. Vice Media’s early focus on video content, and the blending of content and advertising from auteurs such as Spike Jonze, further hastened the economic precipice facing traditional media. Where the eyeballs are concentrated is where the advertisers flock. Traditional media, having built its empire on the willingness of advertisers to cough up huge sums of money for ad placement, was thrown into crisis as revenue dried up. It was simply too expensive to maintain a stable of journalists and editors to keep up the delivery of new content. But then new media realized that to thrive, they needed to generate news-worthy content, and they began to add old-school type journalists to their ranks.

The death of traditional colleges and universities has also been making the news regularly over the last decade or two. Is it getting too expensive to maintain a stable of professors and administrative staff to keep up the delivery of content? Worse, isn’t most of it just old recycled content? Users are supposedly paying to ‘learn’ the content. So how does one learn anyway? Reading complex material is hard work. The more background you have, the easier it becomes, but the early stages can be tough-going. Perhaps if the material is presented in an ‘interesting’ way, it becomes ‘easier’ to learn; at least that’s the premise of new media’s approach: repackage and jazz things up with video.

I dream of the magic bullet that will make my students understand chemistry. They dream of it too. The content of a standard college chemistry curriculum has been known for many, many years. You’d think that our various leaps in media technology would have ushered in a golden age of learning osmotically – efficiently imbibing new content into our consciousness. But new content doesn’t stick unless conceptually mapped to old content, made and remade in the hard work of learning. Complex content is even more challenging with layers upon layers of knowledge, built hierarchically and dynamically, in our three-pound brains in a process that remains mysterious.

If as a society we are no longer willing to do the hard work of complex learning, or to take time to understand the complex issues of today, we are in trouble. Subsisting on listicles, personality quizzes, and celebrity voyeurism – all excellent eyeball-attention-grabbing moneymakers for ad revenue – will lead us down the road to idiocracy. Perhaps that is one takeaway message from Merchants of Truth. The merchants may end up defining truth. Or perhaps ill-defining it. Remaking the news remakes more than just the news.

Downward Causation

What is Life? As a chemist studying the origin of life, I’ve posed the question multiple times in this blog. There’s a temptation to describe the chemicals that make up life – if only we could put them together in an appropriate container in the optimum proportions, we would observe life-like characteristics. Needless to say this experiment doesn’t quite work, at least not in the simple naïve picture I have painted.

What is Living? This might be a more useful question, shifting the focus away from solely concentrating on what molecules are present, to how the molecules are interactively functioning within a system. That last phrase, functioning within a system, significantly complicates the matter. Function is context-dependent. You can only define function within a particular environmental milieu, and this makes homing in on a simple universal definition of life very challenging.

This week I’ve been reading and thinking about Downward Causation. What is Downward Causation? It is easier to think about this by first considering its opposite, bottom-up causation: elementary particles in physics explain molecules in chemistry, which explain molecular and cell biology, which explain living organisms. This approach is known as reductionism. It has served very well in advancing progress in the natural sciences. Downward causation is top-down causation: A ‘higher’ level of organization influences the direction taken by its ‘lower’ level components in a hierarchy. This is emergence, the counterpart to reductionism.

How does such emergence, um… emerge? “By informational selection and control.” At least, that’s the answer provided in the book chapter titled “Living through Downward Causation” by Farnsworth, Ellis, and Jaeger, in From Matter to Life. The authors bring together a number of concepts I have previously mentioned: coarse-graining, robust behavior, digital versus analog in biology. They provide several examples ranging from cybernetic systems to ecology as to how all this works out.

You’ll note, wise reader, that I haven’t explained the phrase “informational selection and control”. That’s because I’m still wrapping my mind around the idea – I don’t have a pithy description to trot out as a clear example. (Writing about it, I hope, will help bring some clarity to my muddled ideas.) For a clear exposition of the issues regarding how to think about information in biology and life’s origin, I recommend “The Algorithmic Origins of Life” by Walker and Davies. As to how such information is controlled, I’m up to my eyeballs in a mash-up of cybernetics-meets-statistical-mechanics, which I still don’t quite fathom. Defining macrostates and counting microstates seems to be important. And somehow a situation emerges whereby top-down and bottom-up causation are simultaneously in play.

Some take-away bullet points I’ve come up with:

·      Systems can be both dynamic and persistent.

·      The environment matters, and must be coupled with the system.

·      Different levels in an organizational hierarchy are partly insulated from each other, but pass information and influence back-and-forth.

·      The functional arrangement of the molecules is important, perhaps more important, than which molecules are particularly involved.

·      Modularity, complexity, and energy transduction are tied together in some crucial way.

That’s all I’ve got for now. Enlightenment Later.

Dimensions

What would it be like to enter a world of different dimensionality? While not the main point of the story, this subject is explored in the classic novel Flatland, perhaps with shades of Plato’s Allegory of the Cave in The Republic. I’m sure there have been many attempts since Flatland to explore this experience, but I haven’t read any of them. Perhaps this is why I was enjoyably tickled by imagining entering different dimensions through the eyes of novelist Cixin Liu in Death’s End, the conclusion to his trilogy that began with The Three-Body Problem.

Dimensionality isn’t the key storyline of Death’s End, which mainly explores further consequences of the Dark Forest answer to Fermi’s Paradox on a galactic scale. But there are parts that explore what it is like for a three-dimensional being to enter four-dimensional space. It is difficult to imagine the experience but Liu does a nice job, in my opinion, by richly describing the visuals as best he can. He also follows the well-worn path of making analogies to what a two-dimensional being would experience going into three-dimensional space.

We humans are so dependent on our eyes as a sense organ, and our lives are caught up interacting with objects of roughly similar size to us. It can be challenging to imagine something much larger in size. In our day-to-day experience, we treat the Earth as (mostly) flat. It is also challenging to imagine something much smaller in size. As a chemist, I’m used to imagining the structures of molecules, predicated by a theoretical model, and using analogies to familiar objects such as balls, sticks and springs. I even picture atoms in unique colors: hydrogens are white, oxygens are red, nitrogens are blue, because I have been socialized into the chemistry community to think that way.

A two-dimensional denizen in Flatland approaching a circle would have to go around it. The three-dimensional analogy: If I was walking and came up to a large pillar, I would have to go around it. The pillar looks like a solid wall, as far I can tell. To the two-D ant, the circle would look act as a similar barrier. However, if the two-D ant could access the third dimension as it approached the boundary, it could keep going straight ahead ‘through’ the circle by being infinitesimally above or below it. At least that’s what it would like to an observer ant. The observer ant sees the other ant disappearing into the barrier and emerging on the other side. Seems ghostly! 

If a three-dimensional bubble appeared in Flatland where the circle was located, the ant that enters the bubble and experiences 3D can now move above or below the plane and look out at its surroundings. It would then be able to see the front, rear, outline and the internal structure of the other ant simultaneously. In fact, it would be able to see all of Flatland. Now it might not be used to seeing all that additional data in one go, not to mention other objects in the third dimension. And perhaps if it rotates, it can now see a cloud hanging above Flatland in a different plane. The data coming in would be overwhelming. Rotating a little bit more would reveal even more that was previously unseen!

By analogy, if you entered a four-dimensional bubble, all of your previous 3D world would be laid out simultaneously. These objects would look ‘flat’ to you, and if you encountered a ‘solid’ looking object, it could be four-dimensional or possibly higher. I suspect it is possible to simulate this through virtual reality although it would likely make one nauseous to see a different landscape so-to-speak every time you tilt your head slightly. Not to mention, you should be able to see all your internal organs simultaneously if you were to look at any part of your own body.

In The Matrix movies, the experienced operator simply gets used to looking at streams of code that describe what is going on in the Matrix. If you’re encountering it for the first time, it just looks like a rain of gobbledygook symbols. That’s one way to process an incoming data stream. We are, in fact already getting used to overlaying additional data streams on top of the 3D world we experience. Witness the phenomenon that is Pokemon Go, making use of augmented reality. Layers upon layers of augmented reality can be geotagged to a particular spot. Like radio, you tune into your desired station. Except, your device can probably overlay more multiple data streams simultaneously. That would be trippy. (I recommend Vernor Vinge's Rainbow's End.)

What’s even more trippy – we’ve only briefly discussed expanding spatial dimensions – could one encounter an additional time dimension? I’m not sure how my brain would deal with that. But the language of mathematics can describe behavior in multiple dimensions, and I’m guessing as you familiarize yourself with the language, you’re becoming like an operator looking at Matrix code. I suppose I could go learn more math. Or maybe I’ll just enjoy reading and watching science-fiction.

The Veneer of Science

Death is the end. All those life-prolonging activities you’re engaged in are meaningless, especially if they are faddish. This sounds like an old story from a Solomon-style Teacher who wrote a book called Ecclesiastes. It’s also the main story of Natural Causes, a new book by Barbara Ehrenreich (of Nickled and Dimed fame). It’s a very quick read and full of Ehrenreich’s razor-sharp humor and polemics. The stronger and more interesting parts of the book relate to medicine and biology; Ehrenreich has a strong background and is insightful in her analysis. The book is weaker as she opines outside her field; I started to skim when her prose started to sound like ranting.

One part of the book that resonated strongly with me was her discussion of changes in the field of medicine with the advance of technology. Ehrenreich’s main point: The advancement of medical technology and biochemical analysis is one cause of additional medical testing. Let’s do a CT scan on an MRI to see what’s going on inside. Perhaps a colonoscopy. Let’s check the bloodwork. The patient is poked, prodded and probed. There is an analogy to the scientific method. How do we learn about the natural world? Why, we do experiments of course! Nature is poked, prodded and probed, and our advanced scientific instruments will get us to the heart of the Matter. (Pun intended!)

Ironically, what she described is my personal experience these past few months. A cyst was discovered during a routine x-ray of something else. (I had no symptoms.) This led to CT scans. Surgery was scheduled but then cancelled during pre-op because my breathing apparently sounded abnormal. (I was feeling fine.) Then I was required to undergo a battery of tests, that consistently showed that things were ‘normal’. The upshot is that I seem to be healthy otherwise. The downside is that I had additional exposure to scanning radiation, among other things, and I was in and out of hospitals with multiple appointments. And besides being probed, I was poked and prodded multiple times. I was relatively cheerful throughout the process, and I’m glad I only read Ehrenreich’s book after all this was done – otherwise I would have been very irritated at the whole business.

Ehrenreich calls this ‘the Veneer of Science’, it’s one of the early chapter titles in her book. She briefly discusses the history of medicine, focusing on its codification and professionalization, and how the rise of science influenced this trajectory. (Why do you think doctors have white lab coats?) She couples this with an anthropological analysis of the role of ritual – our encounters with medical professionals have become increasingly ritualized. She also has some interesting comments about the latest trends in what is known as ‘evidence-based’ medicine.

It made me think about the veneer of science in my own area – education. Educators have increasingly turned to proxies to investigate whether a student is ‘learning’. We can’t see what is going through the student’s mind, but maybe if we track their behavior and their performance on multiple small ‘tests’, we might be able to nudge them in the right direction. The parallel in medicine might be doing many little tests to pinpoint what might be an issue and then providing some sort of palliative to help the body heal itself in the right direction. Doctors don’t cure you. Your body heals itself; medications and related instructions are given to the patient to aid the process. Similarly, the teacher may provide additional explanation and suggest additional exercises for you to learn something. But the learning must still take place in the mind of the student.

I am glad that I live in an era where medical technology is advanced; I certainly wouldn’t want to have lived a hundred years ago when quackery was rampant. But I agree with Ehrenreich that there needs to be a judicious use of scanning technology or medical treatments. When your body receives additional radiation, or a chemical goes into you, that’s not a neutral effect. (I discovered I was mildly allergic to the CT contrast agent for a chest scan that I thought was unnecessary when the initial x-ray suggested no problems.) There are a number of reasons why doctors may order additional tests. Ehrenreich discusses these so if you’re interested, you can read her book. Instead, I’d like to turn my attention to educational technologies.

First, let me say that in general I’m in favor of data-driven decisions. In my oxymoronic opinion, this beats sheer inexperienced opining. But expertise isn’t always reducible to data streams, nor is it always easily quantifiable. Data analytics in educational technology presents teachers with their ubiquitous dashboards. Supposedly these offer the minute data chunks for you to understand each of your students personally – their online study habits and their level of knowledge (based on tests and assignments). You can learn how much time they spent on a problem, what hints they needed, how many incorrect attempts they made, whether they clicked a link to the online text, and what time they did their homework or took a quiz.

Adaptive learning systems claim even more. Based on all that information I mentioned above, the system can generate the next question for the student that will solidify their understanding or further their learning in a particular area. The system gets better, it is claimed, as more students use it because the artificial intelligence running the software adapts as it is shaped by the data gathered.

If you’re in education, you are likely experiencing living in an education culture of Assessment. The cry for more data rings ever louder and more stringent, purportedly so we can advance the notion of ‘evidence-based’ education. No longer do we trust individual instructors to foster the learning of their students, they must provide evidence that their pedagogies ‘work’ and these must be standardized for cross-comparison. It doesn’t matter if you’re trying to knock a square peg into a round hole. Just shave off the edges here and there and it will sorta fit. We need standardized rubrics! We need inter-rater compatibility on these rubrics! In fact, what we ideally need is a machine that when fed our rubric, will consistently grade the tests and spit out the numbers we need!

Will personalized education or personalized medication be increasingly robotic? That sounds terribly oxymoronic, but maybe that’s what technology does for you when fed with data streams. It categorizes. It provides averages. It helps you ‘fit’ things more ‘cleanly’. And to ensure privacy we’ll assign patients and students special ID numbers. I seriously question the data being collected and used. Teaching and learning, in my opinion as someone with expertise, is complex and not easily reducible to data streams. Even the best rubrics are simply a proxy. It’s not that the data isn’t important. It’s just not as important as the dashboard bells and whistles might suggest. The data is peripheral. Not the core. Educators should use what they find useful but not be overly enamored or distracted by it. Data analytics gives us the veneer of science, but our primary focus should be on the core business of teaching and learning where human mind meets human mind. The A.I. mind in the long run, if you rely on it exclusively, will only teach you how to be a better robot.

Studying Bull

I recently watched the movie Green Book. There’s a scene where Tony Lip is proud of his ability to BS his way through different situations. It reminded me of an article published last month by the Institute of Labor Economics. The three authors are academics: John Jerrim, Phil Parker and Nikki Shure. The title and abstract are shown below.

Yes, people study BS. Per the typical academic article, it provides a history of previous BS studies. There are many ‘over’-statement definitions in the field. Over-confidence. Over-claiming. Over-estimation. Over-placement. Over-precision. You find what you might expect: “participants are more likely to bullshit when they believe they can get away with it, and are less likely to bullshit when they know they will be held accountable for the responses they provide”. There’s even a Bullshit Receptivity scale.

Compared to previous studies, the present one has a much larger sample size (40,000+ respondents from nine Anglophone countries) because it uses data from PISA 2012. That’s how I stumbled on this report, since I follow educational developments and I’ve looked at the latest PISA results with respect to scientific inquiry. PISA is aimed at 15-year-olds and measures skills in mathematics, reading and science. Since PISA 2012 had an additional focus on math, there were extra questionnaire items that allowed researchers to crunch data for the present study.

Classifying students’ propensity for BS was based on a list of 16 items where they rated whether they had ‘never heard of it’ up to ‘know it well, understand the concept’ on a five-point scale. Some items on this list were nonsense. Students also answered Likert-scale questions that provided measures of (1) self-efficacy, (2) self-belief in problem-solving ability, (3) self-reported popularity at school, and (4) self-reported measures of perseverance. The self-efficacy questions were aimed at gauging how comfortable students were with a math-related task, such as ‘calculating how many square metres of tiles you need to cover a floor’ or ‘finding the actual distance between two places on a map with a 1:10,000 scale’ or ‘solving an equation like 2(x+3) = (x+3)(x-3). And of course, there are the actual PISA results in mathematics so we know how well students actually performed.

The results are interesting. Teenagers in the U.S. and Canada are more likely to be BS-ers than in Britain (England, Ireland, N. Ireland, Scotland, Wales) or Australia and New Zealand. Along gender lines, boys tend to be BSers more than girls; effect sizes were quite large in this area for Britain and smaller for North America. Effect sizes were also significant comparing socio-economic status (a standard PISA measure) – teenagers from higher socio-economic status showed a larger preponderance for BS.

Not surprisingly, BSers show higher self-efficacy scores and have higher self-views of their problem-solving abilities relative to how they actually performed on PISA. They also viewed themselves as more likely to persevere when confronted with a difficult task (unclear if they would actually do so) and tend to believe they are more popular at school. The effect sizes are smaller in the latter case compared to the former. When given ‘approach to problem solving’ tasks, they tend to pick conventional ‘safe’ answers rather than try to seem less knowledgeable (again, unclear if this is what they would actually do).

The authors acknowledge limitations of their study. BS in one area (math) may not overlap in a simple way with other areas. The data is a one-time snapshot; BS behavior likely changes over time within an individual. Also, the “social and labor market outcomes” and implications of BSers remains unclear. Does it help? Does it hurt? Don’t know.

Do BSers reflect on their self-efficacy on any related traits? Do they recognize their own BS? Or does repetition inure one to that fact such that you start believing in your own imagined abilities? I don’t know. There’s certainly self-help advice out there in the ‘fake it until you make it’ variety. One thing I try to get across to students is how to avoid self-deception in thinking you know more than you actually do when preparing for exams. As a result, I’ve included more metacognitive exercises and introduced annotated self-grading. Perhaps I am in an area where it’s much more difficult to BS on an exam. You know it, or you don’t. And it shows up clearly.

One useful reminder to myself: Domain knowledge and critical thinking in one area doesn’t necessarily extend to other areas, particularly if they are less proximate. I might know a lot about chemistry, or maybe my particular subfield of chemistry, but I should be wary of the temptation to pontificate in other areas. When you’re in a profession where you’re constantly viewed (by students) as an authoritative source, it’s easy to slip into professing BS, and not realize it.