Discovery Learning: ADOM Edition

This past weekend, I notched my fifth ADOM win with a Drakeling Elementalist who is now #2 on the high-score list. In preparation for today’s blog post, I also replayed a Tutorial game on a new installation to remind myself what tips the system provides to brand new players.

 

What I’ve been musing about is “Discovery Learning”, a buzz-phrase that leverages (in this case simple-minded) “common sense” thinking. In the extreme version, there is no formal schooling for kids. Let them explore and discover the world and learn “naturally” from nature. Natural – good! Artificial – Bad!

 

I don’t disagree that much can be improved about the seemingly artificial settings of today’s classrooms especially for kids who have lots of energy and are bouncing off the walls. But I don’t think Discovery Learning and doing away with formal schooling is the answer. It could work well for some people after they’ve had a decent foundation (acquirable in diverse ways). The media likes highlighting the college dropout who went on to found a tech company and become ridiculously wealthy. They don’t tell you about the tens of thousands of other dropouts who did not become billionaires or even millionaires.

 

In ADOM, the world of Ancardia has rules. The basics are provided in the manual, and I found the tutorial much clearer now that I have 70-80 games under my belt compared to the very first time when I was floundering around. Because I had experience with old-school CRPGs, ADOM wasn’t impenetrable, but many of the rules are “hidden”. I actually did okay getting to my first mid-game character within a dozen games purely through Discovery Learning. Characters die early and often in ADOM. I could sink hundreds or thousands of hours into the game and learn more nuances about staying alive and making further progress towards the end goal, or I could learn from experts who have already traversed the path. I chose the latter, and my enjoyment of the game increased by more efficiently getting over many of the otherwise frustrating barriers that would have killed dozens of characters.

 

The “natural” environment of ADOM is brutal. You might even say they are hostile to learning efficiently. While the tutorial gives you a “warning” when you first enter the Small Cave, you have no idea what that really means. And until you notice or understand how the hostile monsters are generated, you’ll bang your head against the wall trying to get through. You have no sense of how the difficulty level scales. You encounter monsters you know nothing about. You might get cursed or doomed and not realize why it happened and what it means for your character. You don’t know what talents or attributes are helpful and how they might be trained naturally. There is plenty that is hidden unless you know exactly what to look for. I’m not sure how many games it would have taken me to figure out that dropping a potion of water on a co-aligned altar blesses it, and that when you dip a scroll of identify into holy water, you can then read it (if your literacy is high enough) to identify all your items in a single swoop.

 

As a chemistry professor, the natural sciences and math are the areas I am most familiar with. Learning math or chemistry efficiently is very unnatural. If you had to figure it out from scratch, it might take you several lifetimes. (Also, failure is not always productive.) The accumulation of human knowledge has taken lifetimes – small bits of info passed down from teacher to student. The apprenticeship model has been true for a long, long time. It’s far better than having to discover everything from scratch through trial and error, but this one-on-one learning is inefficient and very expensive on a larger scale. I don’t like having forty students in G-Chem; I think I do a better job when I have five or ten or twenty. (At least it’s not four hundred.) But I recognize the efficiency of teaching a group of students. They can also help and encourage each other, which is a plus in my opinion.

 

Becoming an expert requires depth of knowledge and acquiring abstract schemas in long-term memory. Without books and teachers and some very effortful thinking on my part, I would not have the expertise that I now have in chemistry. I can’t imagine getting there through pure discovery. Of course, here I’m caricaturing Discovery Learning, and an advocate would say that no one is promoting pure “throw you into the deep end of the pool and you sink or swim”. They’d say the learning has to be guided. I don’t disagree. But the same advocates caricature current classroom practices, especially what is known as “explicit teaching” as inferior to discovery approaches, or “lecturing” as an artifice and therefore worse than a more “natural” approach. In reality, one balances multiple aspects when considering pedagogical strategies.

 

My current ADOM character is a level 13 gnome druid. I just made it to the High Mountain Village although I was not able to retrieve the waterproof blanket on the way because I understand how the Small Cave works. The fun in ADOM is that the dungeon layouts (and the game is a dungeon-crawler) are randomly procedurally generated, so each game feels quite different. Your character’s inherent skill set provides even more variation. I think this is my third druid (the previous two did not make it past level 10 before succumbing), and I’ve learned how to balance spellcasting with traditional weapons. I also now know that most animals are generated friendly, and switching my alignment to Lawful means that I have a reasonably good chance of completing the Rolf Quest and getting the ring of the master cat, provided I don’t die in the Pyramid or somewhere else. The balance of some discovery and some guide-reading, in my case, has led to maximum enjoyment. I still do bits of both when I encounter something rare (statues and artifacts) or exploring a different aspect of the game, and I wouldn’t do this any other way.

Square Integrable

I am reading about the extraordinary math and science contributions of John von Neumann in Ananyo Bhattacharya’s book The Man from the Future. I definitely get the feeling that von Neumann was indeed a rare genius. I also got the feeling that maybe I should have persevered in learning more math when I was younger. If so, not only would I have a better appreciation of von Neumann’s achievements, I would also be able to tackle some interesting problems in my research that require mathematically modeling beyond my current abilities. Feynman’s quote notwithstanding, I would like to better understand quantum mechanics since I use it heavily in my research.

 

Today’s blog post is about Chapter 3 of Bhattacharya’s engaging book. The chapter is titled “The Quantum Evangelist” and leverages the author’s physics background. While I know a number of facts about the history of the development of quantum mechanics, I learned a lot more about von Neumann’s contributions and the context surrounding his work. Reading this chapter gave me a better idea of the conceptual differences between Heisenberg’s matrix mechanics and Schrodinger’s wave mechanics. The connections to set theory in mathematics (and Hilbert’s program of systematization) helped clarify the context. Quoting the author: “An atom has an infinite number of orbits… so Heisenberg’s matrices must also be of infinite size to represent all possible transitions between them. The members of such a matrix can… be lined up with a list of the counting numbers – they are ‘countably’ infinite. Schrodinger’s formulation, on the other hand, yielded wave functions describing… an uncountably infinite number of possibilities. An electron that is not bound to an atom… could be literally anywhere.”

 

I now have a better appreciation of Dirac’s “ingenious trick to merge the ‘discrete’ space of Heisenberg’s matrices and the ‘continuous’ space of Schrodinger’s waves” with the delta function. Bhattacharya describes it as a “salami slicer, cutting up the wavefunction into ultra-thin slivers in space”. While Hilbert space still feels fuzzy to me and I don’t quite comprehend it, I can dimly see where square-integrable functions come from. When I teach quantum chemistry, I tell students about this important property and its practical uses along with Born’s probability postulate, I had never talked about their mathematical basis (because I didn’t understand it myself).

 

Where does von Neumann come into the story? Given his mathematical talents, he realized that square integrable functions “can be represented by an infinite series of orthogonal functions, sets of mathematical independent functions that can be added together to make any other… How much of each function is required is indicated by their coefficients... [which] were exactly the elements that appear in the state matrix.” In my class, I invoke orthogonality from a consequence of Hermitian operators. I discuss the importance of having linearly independent functions and spaces (e.g. Cartesian space or polar coordinates) conceptually but my students still struggle to think about it. Linear algebra is not a pre-requisite for my class and most students haven’t taken it. Neither have I for that matter. Until reading this chapter, I had not realized the connection between square integrable wavefunctions and orthogonality. In my class, when we get to multi-electron multi-atom systems, I introduce students to manipulating linear combinations of functions that sum up (invoking the principle of superposition) to get better results when solving the Schrodinger equation. They learn that the sum of the squares of the coefficients must add up to one, but I hadn’t made the connection to square-integrability.

 

There is plenty more in the chapter about the weirder aspects of quantum mechanics, wavefunction collapse, hidden variable theory, pilot waves, Bell inequalities, and Many Worlds. But what really stood out to me was where square integrable functions come from (as part of Hilbert space) and how they connected to orthogonal component wavefunctions. All these connections were a revelation to me, and I’d been teaching for a quarter of a century! How little I know. How much more to learn. This reminds me that I should get back to Beyond Weird by Philip Ball.

Cybernetics Informing Learning?

I stumbled across an interesting blog post connecting Ross Ashby’s principles of cybernetics to how one designs questions to probe student learning. I have some familiarity with the cybernetics principles for thermostat design; several years ago I was reading papers using this to analyze a complex prebiotic chemistry problem adjacent to one of my research projects. I had not, however, considered how this affects instructional design. Given that A.I. methods are heavily encroaching on education, I think the article highlights some of the potential pitfalls of a computerized system that supposedly personalizes learning.

 

The word “system” is important here. The blogger, Carl Hendricks, has this to say: “An instructional system can only regulate what it can detect and many learning environments rely on a channel of extremely low capacity: correct or incorrect [which] carries almost no information about process. It does not distinguish decoding from guessing, understanding from memorisation, reasoning from elimination.” Prior to the present LLM burst, the computerized learning systems relied on multiple choice questions (MCQs) or True/False questions. A subject matter expert designed these questions as a proxy to probe certain learning goals, usually atomized Taylorian-style. In the last decade, this morphed into “adaptive” systems that mixed-and-matched questions depending on whether a student got this right or wrong.

 

I think that expert-designed questions and answers for the computer-distance-online-learner can be effective to some extent. Writing good questions and answers is time-consuming and challenging. It’s also why lazy me doesn’t use exam MCQs. It’s faster for me to write a short-answer question and then evaluate the student answers, i.e., the time it takes me to grade the student answers is less than the time it would take to design really good MCQs. I’ve tried getting the LLM to help generate good answer-question pairs but right now the results are low quality. I expect they will improve with time; I might even be able to train one on a limited chemistry corpus.

 

But while expertly-designed individual questions may be quite good, stringing them together in an A.I. “adaptive” system degrades that goodness. This is why after trying out some pre-LLM systems, I never selected the “adaptive” option. Hendricks mentions the drawbacks of not coming up with good questions and answers that really get at what you want the student to learn, and the additional problem of having a regulating control system that supposedly personalizes the learning. He writes that measuring such performance is “fragile” because “the system ensured that answers were right often enough, but it never ensured that the right thinking had occurred. [It is] informationally impoverished; and no amount of pedagogical enthusiasm can compensate…”

 

LLMs continue to push the personal tutor aspect; I should say A.I. tech companies are pushing heavily because they need revenue streams. Last month I found that the current LLMs do a better job generating chemistry questions and answers compared to previous ones. I see more nuance and better accuracy overall. And the “voice” of the LLM tutor leans heavily on trying to sound helpful, offering follow-up information and more. One thing I learned is that I can make an effort to sound more helpful when students ask me questions, so the LLM had at least that aspect to help me improve. But the LLM doesn’t always (and maybe not often) offer what might be best in actually improving learning. It’s good at helping the student feel good. But that’s because it was designed to do so. It wasn’t designed to be an expert tutor.

 

A thermostat does just one thing – regulate the temperature by measuring the ambient value and then turning on (or off) the heating/cooling device. Even with its narrow purpose, the mechanics of designing a good thermostat is trickier than it looks at first glance. The business of teaching and learning is more nebulously defined in purpose and certainly much more complex. Cybernetics may be a starting point to think about adaptive tutors but there is far to go before it will replace an actual human expert in terms of quality. Pessimistically, I predict that overhyped adaptive tutors will degrade the desired quality to a low common denominator. Hendricks writes: “Learner ingenuity will always exceed designer foresight; there will always be shortcuts that were not anticipated, strategies that were not mapped, paths that were left open by accident. Requisite variety is an asymptote, not a destination.”

 

I’m reminded how amazing it is to learn something as a human being. I don’t pretend to know exactly how it happens especially in glorious moments of gestalt “aha!” understanding. Present neural networks underlying LLMs are not like our brain or our mind or our sense of self. As a computational scientist, I have some vague and wild ideas of how to improve on this. I’m sure others like me have such thoughts and hence I expect over time that LLMs will continue to improve. Whether they will eventually achieve the quality of the hype remains an open question.