Is the CRAAP Test crappy?

Students enjoy being introduced to the CRAAP Test. It has an amusing name, and the double A lends itself to lengthening its pronunciation for emphasis. I’ve introduced the CRAAP Test the last two times I’ve taught Research Methods, as part of a unit in searching the world wide web for information. It is also a checklist, and my students like checklists. Checklists are good things when packing your bags before a trip, or following the safety protocol before running an experiment; but are they good for learning?

What got me thinking about whether the CRAAP Test is crappy was John Warner’s post at InsideHigherEd titled Teaching Without Learning: The Limits of Checklists. Warner has taught writing for many years, notes that “checklists, often in the form of rubrics, and also often tied to high stakes assessments are ubiquitous in the teaching of writing.” I don’t have too many writing assignments in my chemistry classes and don’t use rubrics for those. My higher stakes assessments are exams and sometimes a final project. However for oral presentations, I do provide my students with a rubric of how I will be grading them.

Rubrics are the current darling of assessment strategies. A couple of summers ago, I participated in an exercise assessing student learning in general chemistry by applying a rubric to student answers on common final exam questions. The exercise itself was useful to get semi-quantitative data, and more importantly got me thinking about how students present their work when problem-solving. This led me to make modifications to some of my class activities and assessments, so in this case the exercise (while tedious) was of value to me as an instructor.

At the end of his blog post, Warner poses a number of interesting questions related to the use of checklists/rubrics in teaching. Yes, the students might be using your wonderful rubric merely as a checklist. “What if checklists are only appropriate for things we’ve already learned, but are prone to forget[?]… What is the impact of checklists on people who have not yet developed disciplinary instincts[?]” The concern arises because “tools like the CRAAP test [for literacy] are heavily domain dependent, not based on skills, but on a body of knowledge that comes from mindful immersion in context.”

Warner’s post also led me to the synopsis of a study carried out by a group at Stanford examining web literacy among grade school and college students – how to tell fake news from reliable information. Many of the suggestions of how to improve such literacy (also found on the web) are in the form of checklists, such as the CRAAP test. The author explains the problem: “The checklist approach falls short because it underestimates just how sophisticated the web has become. Worse, the approach trains students’ attention on the website itself, thus cutting them off from the most efficient route to learning more about a site: finding out what the rest of the web has to say (after all, that’s why we call it a web). In other words, students need to harness the power of the web to evaluate a single node in it. This was the biggest lesson we learned by watching expert fact checkers as they evaluated unfamiliar web content.”

What struck me most was the information gleaned by interviewing fact-checkers. Here’s an excerpt with two great quotes. “The senior fact checker at a national publication told us what she tells her staff: ‘The greatest enemy of fact checking is hubris’ – that is, having excessive trust in one’s ability to accurately pass judgment on an unfamiliar website.’ Even on seemingly innocuous topics, the fact checker says to herself, ‘This seems official; it may be or may not be. I’d better check.’ ” And I thought hubris was mainly confined to the characters of the Iliad. That’s my one word summary of the Iliad if you needed one. Hubris might also be a key theme in Quenta Silmarillion but I need to ponder that idea a little more.

The Stanford groups recommend strategies to “fend off hubris”. I recommend reading their synopsis in full for the details, but here are the three key strategies mentioned. “(1) Teach students to read laterally. (2) Help students make smarter selections from search results. (3) Teach students to use Wikipedia wisely.”

Many moons ago when I was first introduced to the Apple II, part of my time was spent finding programs that would do a better job copying other programs so I could make my own copy without shelling out cash. In the early days Disk Muncher was effective, until increased protections required more sophisticated programs such as Locksmith and their ilk. I learned early on from a DOS book that there is a constant battle between the code-protectors and the code-breakers – and that the latter will always win eventually. The checklists that we use now are likely to become obsolete quickly, at least where web literacy is concerned. The next time I teach Research Methods, I need to give a bit more thought on how I am teaching my students and whether I should toss the CRAAP Test. It might not be crappy now, but at some point it will no longer help distinguish the crap from the good stuff.