Can Teaching be Measured?

4175299981_614e7d9dc5 (1)


By Justine Rogers

Last week UNSW had its second ‘Great Debate’, introduced last year as a fun, accessible way for the UNSW community to explore a serious and stirring topic. (For a post on last year’s, click here)

Each team: professor-manager, non-prof academic, and student.

The topic: Of Course Teaching Can be Measured (it’s a 5.3!).

I was on the affirmative (which I knew going in would be tough).

Given it was a private event for staff and students, I’ve written this assuming some version of the Chatham House Rule applies.

The affirmative’s arguments were:

  1. Teaching can be measured, albeit imperfectly, and certainly better and more reliably than it is now.
  2. Teaching needs to be measured to enhance the quality, rewards and status of teaching.

The negative’s arguments were:

  1. Teaching cannot be measured, only learning experiences and learning outcomes can. 
  2. Teaching measures are flawed and unreliable.

The negative committed to the empirical questions, whereas I tried (unsuccessfully in the 4 or so mins we had) to engage both sides in the wider empirical and normative argument suggested in affirmative point 2: whether there is some positive correlation between measurement, and motivation, quality and status, and therefore whether a more robust measurement of teaching is worthwhile.

I wish we’d had the format and time to examine this: whether this is true, or whether, using research measures as example, such measures have too many biases, perverse incentives, and inefficient and/or demoralising effects to be of real value (even if it entails superficial value). 

I will share my main arguments here, some of which I am fairly convinced, many posed as part of my role on the affirmative side, and some raised in the spirit of fun and provocation. Above all, I think the topic raised several questions left that need to be contemplated, many of which I’ve posted below – so please share your thoughts!

Continue reading

A not so subtle metaphor


Imagine if a faculty of science in Australia focused its undergraduate/foundational teaching on the science of the local flora and fauna.  Students would learn about kangaroos and eucalyptus tress, about achidnas and banksias, about the funnel-web spider and the Murray cod.  Of course, they would also learn about palm trees and dingos, about the cane toad and about invasive weeds.  While there would occasionally be upper level electives on comparative anatomy or botany or on species not found locally, those courses would be few and far between, perhaps largely left for specialised postgraduate or later studies.

Further, imagine if the science faculties around the world followed the same model and also focused their undergraduate/foundational degree programs on their own local flora and fauna. Such a situation of university science education might arise around the world if the primary jobs for science graduates were in managing and developing the local flora and fauna, with only occasional work involving foreign or trans-regional species (despite an increasingly interconnected world of flora and fauna).  It might be perpetuated by the senior people in the field, who set the employment regulations, and who believe such an intense grounding in local science is necessary for the successful work of the scientist and the country.

Such an approach to science education around the world would mean that it would be difficult for science graduates to move to other countries for work in their field – as they may lack the local knowledge demanded of them by employers and by the regulators of their field.  Further (local) studies would be required to then secure employment in that foreign country. If the flora and fauna were too radically different it might even be the case that the newly arrived science graduate would have to start over again, enrolling in the local foundational science degree.

In such a situation we should expect there to be little science student international mobility, especially into the undergraduate/foundational level. After all, why go to the trouble and expense to study the flora and fauna of another country – especially for three to four years when that knowledge will be unlikely to help secure a job back home.  Though, if the home has similar flora and fauna it may be that the undergraduate/foundational degree would be acceptable, providing enough substantive knowledge to permit successful work back home. Thus, science studies in California may be suitable for Mexico, or studies in Vietnam would be applicable in Cambodia.  But a degree in such a localised science course in Australia would not be of much use in Norway.

Perhaps another approach to science teaching might be to push for an undergraduate/foundational degree in world/comparative flora and fauna, focusing on holistic concepts and transferable skills, perhaps with a few upper level electives or specialized post graduate courses in the local flora and fauna that would then provide the detailed knowledge to work locally (though the basic courses should be sufficient to provide the ability to assimilate and understand the science of the local flora and fauna).  Such an approach might reflect an understanding that science is best taught by starting at an holistic and conceptual level, and then moving to the specific if necessary.  Such an approach accepts and understands that science  is more similar than different across the world.  Imagine if that is how science faculties actually taught.

By Colin Picker

The Bluebook’s Secret History!

Even in Australia one can feel the dread reach of the Bluebook.  For those interested in its history, a new article entitled “The Secret History of the Bluebook” reveals all!  Written by two members of Yale Law School, Fred R. Shapiro & Julie Graves Krishnaswami, it is forthcoming at Minnesota Law Review, Vol. 100, No. 4, 2016.  It can also be found here at SSRN.

The SSRN abstract (noting  HLS’ early monopolization of the royalties (nothing about HLS would surprise me!) as well as noting the succinct early versions of one to fifteen pages compared to today’s grotesque length):

“The Bluebook, or Uniform System of Citation as it was formerly titled, has long been a significant component of American legal culture. The standard account of the origins of the Bluebook, deriving directly from statements made by longtime Harvard Law School Dean and later Solicitor General of the United States Erwin N. Griswold, maintains that the citation manual originated at the Harvard Law Review in the 1920s and was created or adapted by Dean Griswold himself. This account is wildly erroneous, as proven by intensive research we conducted in the archives of Harvard and Yale. In fact, the Bluebook grew out of precursor manuals at Yale Law School, apparently inspired by a legal scholar even more important than Griswold, namely Karl N. Llewellyn. The “uniform citations” movement that began at Yale was actually at first opposed by Harvard.

In his most extreme misstatement, Griswold asserted that a collaborative decision was made in the 1920s by Harvard Law Review, Yale Law Journal, Columbia Law Review, and University of Pennsylvania Law Review to share the revenues from publishing the Bluebook (eventually amounting to millions of dollars) among the four journals. There is indeed now four-way revenue-sharing, but it did not commence until the 1970s, and then only after a revolt of the three “junior partners” against Harvard Law Review’s complete monopolization of Bluebook income for half a century, a revolt initiated by Joan Wexler of the Yale Law Journal.

Some readers may question whether originating the hyper-complicated Bluebook should be a source of pride for Yale. Our response is that, although the Bluebook version that subsequently developed under the leadership of Harvard Law Review currently consists of 582 pages, the two earliest Yale precursors of the Bluebook were, respectively, one page and fifteen pages long.”

Colin Picker

Do students know best about teaching?

In the Conversation today ( 30th November 2015) there is an article entitled ‘Students don’t know what’s best for them’. — It argues that students often rate teachers best who make it easier for them, and that students don’t understand when they are learning and when they are not. So they think they have learned if the task they are doing is easy.

I am not entirely certain that a thoughtful student can’t go beyond this, but I think another thing the article suggests is very important. That is that the amount of learning being done is very often significantly connected to the amount of effort being put in.  More importantly, other educational evidence suggests that students continue to try hard and cheat and plagiarise less when they are praised for effort rather than achievement.

So for teaching purposes this suggests that finding ways to motivate students to try hard may at the very least make them learn better. Whether they recognise that they have done so may be another matter.!



Academic greetings and what they may signify

As I come to the end of a busy few weeks meeting with American law academics I have been struck by many things I never noticed when I was a US law academic.  Perhaps the first thing which I have noticed again and again on this trip is that when meeting American law academics, whether at an elite or local US law school, their first question of a professional nature has so far always been “what do you teach?”  Not what are you working on at the moment (as in research or writing).  Indeed, the whole conversation may never raise research or writing.  In contrast, my experience when meeting European or Australian law academics is that they ask about research and writing first and may not even ask about teaching.

No doubt there are many explanations for the above (assuming it is an accurate reflection of the legal academic cultures).  Though, I tend to think that it is is less about enthusiasm (or not) for teaching or research but rather reflects the focus of law school missions.  In the US law schools exist primarily to train lawyers. Outside the US it is certainly the case that a law faculty must teach students about the law, even perhaps with the idea that some (perhaps most) may practice.  But, the role of the non-American academic may be much more orientated towards research and writing than may be the case in the US where the academic may be more oriented towards their students’ development. (an orientation reinforced by performance metrics).

Of course, the above is a generalization.  Many law schools outside the US, such as UNSW’s law school, focus heavily on legal training.  And in some American law schools one may be forgiven for thinking that the students are just an annoyance in the way of their professors’ careers (I certainly felt that at times as a law student).  Also, we may be seeing non-American schools slowly moving towards the student-focused US approach.  Nonetheless, I think the difference in how we greet each other reflects some real and deep-seated differences.  This is not to say one approach is better than the other, just that they are different and may reflect different contexts.  Accordingly, transnational legal academic projects and relationships need to keep these sorts of differences in mind.

Colin Picker

One Day Symposium: Teaching Legal Analysis and Writing Dec 10th : Melbourne Law School

Melbourne Law School, Melbourne University, is hosting a one day symposium on Teaching Legal Analysis and Writing Skills on Thursday, December 10, 2015, from 8:30 AM – 5:15 PM, at 185 Pelham Street, Carlton, VIC. Faculty from New Zealand, the United States of America, and 8 law schools in Australia are attending.

The symposium will be an opportunity for professional development in a stimulating and interactive series of sessions.  Topics include:

  • Using games to stimulate student learning
  • How to develop online teaching and learning materials
  • Creative methods of demonstrating legal concepts and reasoning in the classroom
  • The variety of legal writing exercises that might be used to teach and assess analysis and communication skills

This is a great opportunity to share promising practices in teaching legal analysis and writing.

If you have any questions about the symposium, please contact Dr Chantal Morton at chantal.morton  @





Paolo Freire and Testing Assumptions

Recently I saw an article in a newspaper raging about how Paolo Freire had taught South Americans to take ‘left wing’ views. The fact that the South Americans Paolo Freire was working with were amongst the most downtrodden and poor of those in South America seems to have passed the author by. What Paolo Freire taught those people was to ask questions and not to simply accept answers without testing them. This is a fundamental aspect of education. It may or may not lead to revolution; but it will lead to change of some kind. The outrage at the ‘political’ nature of this sort of education is based on the idea that things should stay the same. The people who want things to stay the same are usually those who are benefiting from it.

In first year teaching we often have very bright students who have been well trained and who can answer questions within a given set of parameters very quickly. But some of them have never really used the excellent brain they were given. We can take the opportunity to let them see what it is like to really use your brain. The secret is simple – it is to ask questions, and to keep on asking questions, and when the answers come, it is to test those answers against material which you know to be true – because you asked questions about it and tested the answers.

In my first year law classes I try to have one two hour class where we only have questions. As usual they are required to do their reading. I choose something from the reading, eg a case, and tell them that they need to think of a question which would illuminate this reading. WE go round the class, each asking a question. There are no answers. I do not answer and I do not permit them to answer. They have to keep asking questions. I write the questions on the whiteboard. Normally they think once everyone has asked a question that there are no more, but I don’t let them stop. As we keep going and they struggle to come up with questions their questions change quality and become larger and often more about evaluation; they find themselves asking profound questions about the item, going into philosophical questions, historical questions and beyond.

After this we spend some time looking at all the questions on the whiteboard and I ask them questions about the questions –

  • Which questions have a yes/no answer?
  • Which questions can be answered with certainty?
  • Which questions are historical?
  • Which questions evaluate something?
  • How else could you think about these questions?
  • What assumptions underlie these questions?

I find that this class has a significant effect on my students. They seem to suddenly understand the process of analysis that is required in university work when many of them did not before. And they have learned new types of questions and I hear these questioned repeated later on in the semester as they take them on as their own.

This is the kind of education that Paolo Freire wanted to begin with. Neil Postman and Charles Weingartner wrote a book about it called Teaching as a Subversive Activity. I wish I were brave enough to run more of my classes like this instead of assuming my students may need more content explained to them.