Benchmarking Teaching

My department had its external academic program review this past year. (It went very well – the joys of being in a superb department!) Reports and letters have been written and circulated. The final step of the process is to submit our Five Year Plan to the administration. As part of the report, we discussed benchmarks for research and scholarship in our department faculty meeting last week. We have actually been collecting benchmarking data annually for a number of years since we received a generous external award/grant to “bring our department to the next level” (according to the funders) which was generously matched by our institution. (The good thing about getting a prestigious award that requires an institutional match, is that no administration would want to turn it down.)

After looking over our annualized data for the past five years, we started to discuss what our benchmarks should be for 2020. All these were research or scholarship related, and could be assessed quantitatively. (This data was submitted annually to the funding body as part of our annual review during the grant period.) Examples include number of peer-reviewed publications per year, number of faculty presentations at national conferences per year, number of grant proposals submitted, dollar amounts of external funding received, number of undergraduates presenting their research at conferences, and number of undergraduates involved in research during the year. Our benchmarks are aggregate numbers across the department, recognizing that output varies from year-to-year and from individual-to-individual. Sometimes we arrive at the benchmark by adding up target numbers individuals have set for themselves. Other times we propose different aggregate values and debate their relative merits. We may not always agree, but being in a cohesive and highly functional department means that we are often able to reach some sort of consensus both amicably and efficiently.

We did not discuss teaching benchmarks as a department, nor have we ever in recent memory. I did not think about this, but a couple of days after the meeting, one of my colleagues e-mailed the group asking us to consider how we will discuss our teaching goals in the Five Year Plan – given we had discussed the research and scholarship benchmarks in great detail. Individuals have target goals perhaps, but our department does not have benchmarks. We do have a (possibly vague) statement about our aim to be known for our excellent teaching, and we certainly have the reputation in the college for having great teachers and providing an excellent learning environment for students (who flock to our classes and our major). Have we improved over the past five to ten years? I don’t know. How do we measure it? Can it even be measured in a way that is both fair and useful? (I can think of many ways to measure this poorly.) I’ve been pondering this question the last couple of days. Here are some thoughts, with no strong conclusion as of yet.

Let’s start with examples of some poor benchmarks. “We, as a department, would like to see the average numerical score on questions X and Y in the student teaching evaluations rise by N points over the next five years.” I hope our administration will never suggest any department do such a thing. (They never have yet, but who’s to know in the fad of “accountability”.) “We would like to see an increase in the overall final grades of students in our classes by X indicating that they are learning more and thus we are teaching more effectively.” I can see many pitfalls with this approach, and if anything we are trying to fight grade inflation. Interestingly, our students take the standardized American Chemical Society major field test before they graduate – but students vary from year to year so we don’t see an increase necessarily in how well they do. Some of our students also “blow off the test” since it has no effect on their GPA. On average our students do better than “comparative institutions” but those lists also change annually. Some individuals do exceedingly well, and others do rather poorly.

Here are some examples that don’t sound so bad, but might still be dubious. “We would like our faculty to attend an average of X workshops run by the teaching center and give Y presentations at conferences on teaching/pedagogy.” It turns out that we can track X and Y (and we keep track of Y at the department level) and we know about X when individual annual review comes around. However, we have never set these as benchmarks. We do bring in “non-research” grant funding (by that I mean not directly related to our research projects in lab) that’s related to education or student scholarships, but while we track the numbers, we do not set a benchmark. “We will substantially improve X courses on an N-year cycle.” I’m being vague about what this entails since I’m sure it varies greatly. My colleagues and I constantly update and improve our classes. I personally try to overhaul each class I regularly teach every 3-4 years – this means that I’m usually overhauling one class each year. But that’s a personal goal – and I wouldn’t foist it on any of my colleagues.

How about this one that I suspect an administration might actually try and do in the future (although I would never suggest it): “In the rubrics of our assessment plan, we would like to increase the number of students in the Developing category by X, the Accomplished category by Y, and the Mastery category by Z.” If you’re an academic in the U.S. and have not yet been inundated by the assessment wave sweeping the nation, count yourself lucky for the moment.  Or: “We would like to increase our average scores by X on the Peer Teaching Observation Inventory (PTOI).” Use of a PTOI (an acronym I just made up) with a numerical score is not widely used, but there are pilot suggestions. Here’s a version I discussed in a previous post. I have since read the actual papers and looked carefully at the tool. What troubles me is that certain pedagogical techniques are privileged over others, and this is a dangerous road. I can see an administration tempted by the claim of some PTOI being research-based and therefore appropriate. I don’t doubt that some of the guidelines suggested by education and cognitive research are useful (and I have learned much as an instructor), but I would be strongly against mandating/privileging particular teaching techniques or using certain types of teaching technology.

Maybe benchmarking as a department is unnecessary. As individuals we have our personal goals in how to improve as teachers and how we would like to restructure our courses to maximize students learning at a deep (rather than superficial) level. My colleagues frequently talk about our classes, our students, our teaching ideas, new things we are trying, things that are working, things that are not working, etc. Most of our personal goals are qualitative rather than quantitative. Perhaps a narrative that discusses what we are currently doing, and lists some individual goals (perhaps not associated with names) would be suitable to include in the Five Year Plan. In a sense, teaching goals are like service goals. Many individuals in our department have substantial service responsibilities and involvement both within the institution and to the profession at-large. I do think that it behooves us to represent both service and teaching (especially teaching since we are college professors!) in a Five Year Plan that may not include things that are “countable”. A dictum I’ve often repeated is “Not everything that counts can be counted. Not everything that can be counted counts.” We should do our part in reminding the administration that when 2020 comes around, success isn’t just measured by the benchmarking data laid out by our research and scholarship.