Cybernetics Informing Learning?

I stumbled across an interesting blog post connecting Ross Ashby’s principles of cybernetics to how one designs questions to probe student learning. I have some familiarity with the cybernetics principles for thermostat design; several years ago I was reading papers using this to analyze a complex prebiotic chemistry problem adjacent to one of my research projects. I had not, however, considered how this affects instructional design. Given that A.I. methods are heavily encroaching on education, I think the article highlights some of the potential pitfalls of a computerized system that supposedly personalizes learning.

 

The word “system” is important here. The blogger, Carl Hendricks, has this to say: “An instructional system can only regulate what it can detect and many learning environments rely on a channel of extremely low capacity: correct or incorrect [which] carries almost no information about process. It does not distinguish decoding from guessing, understanding from memorisation, reasoning from elimination.” Prior to the present LLM burst, the computerized learning systems relied on multiple choice questions (MCQs) or True/False questions. A subject matter expert designed these questions as a proxy to probe certain learning goals, usually atomized Taylorian-style. In the last decade, this morphed into “adaptive” systems that mixed-and-matched questions depending on whether a student got this right or wrong.

 

I think that expert-designed questions and answers for the computer-distance-online-learner can be effective to some extent. Writing good questions and answers is time-consuming and challenging. It’s also why lazy me doesn’t use exam MCQs. It’s faster for me to write a short-answer question and then evaluate the student answers, i.e., the time it takes me to grade the student answers is less than the time it would take to design really good MCQs. I’ve tried getting the LLM to help generate good answer-question pairs but right now the results are low quality. I expect they will improve with time; I might even be able to train one on a limited chemistry corpus.

 

But while expertly-designed individual questions may be quite good, stringing them together in an A.I. “adaptive” system degrades that goodness. This is why after trying out some pre-LLM systems, I never selected the “adaptive” option. Hendricks mentions the drawbacks of not coming up with good questions and answers that really get at what you want the student to learn, and the additional problem of having a regulating control system that supposedly personalizes the learning. He writes that measuring such performance is “fragile” because “the system ensured that answers were right often enough, but it never ensured that the right thinking had occurred. [It is] informationally impoverished; and no amount of pedagogical enthusiasm can compensate…”

 

LLMs continue to push the personal tutor aspect; I should say A.I. tech companies are pushing heavily because they need revenue streams. Last month I found that the current LLMs do a better job generating chemistry questions and answers compared to previous ones. I see more nuance and better accuracy overall. And the “voice” of the LLM tutor leans heavily on trying to sound helpful, offering follow-up information and more. One thing I learned is that I can make an effort to sound more helpful when students ask me questions, so the LLM had at least that aspect to help me improve. But the LLM doesn’t always (and maybe not often) offer what might be best in actually improving learning. It’s good at helping the student feel good. But that’s because it was designed to do so. It wasn’t designed to be an expert tutor.

 

A thermostat does just one thing – regulate the temperature by measuring the ambient value and then turning on (or off) the heating/cooling device. Even with its narrow purpose, the mechanics of designing a good thermostat is trickier than it looks at first glance. The business of teaching and learning is more nebulously defined in purpose and certainly much more complex. Cybernetics may be a starting point to think about adaptive tutors but there is far to go before it will replace an actual human expert in terms of quality. Pessimistically, I predict that overhyped adaptive tutors will degrade the desired quality to a low common denominator. Hendricks writes: “Learner ingenuity will always exceed designer foresight; there will always be shortcuts that were not anticipated, strategies that were not mapped, paths that were left open by accident. Requisite variety is an asymptote, not a destination.”

 

I’m reminded how amazing it is to learn something as a human being. I don’t pretend to know exactly how it happens especially in glorious moments of gestalt “aha!” understanding. Present neural networks underlying LLMs are not like our brain or our mind or our sense of self. As a computational scientist, I have some vague and wild ideas of how to improve on this. I’m sure others like me have such thoughts and hence I expect over time that LLMs will continue to improve. Whether they will eventually achieve the quality of the hype remains an open question.