Big Data Analytics on student surveys

It’s the new “thing” – analytics applied to student responses to courses. And it is really quite scary.

To give an example, I will share my own results from a recently taught course of 22 students of which 10 filled out the survey. This is “small data”. It takes about 5-10 minutes (generously) to read and reflect upon the student feedback. Since I am sharing, they generally liked the course including guest lectures and excursions, but felt that one topic didn’t need as much time and that my Moodle page wasn’t well organised. All very helpful for the next time I run the course (note to self to start my Moodle page earlier and tweak the class schedule).

The problem is no longer the feedback, it is the “analytics” which now accompany it. The worst is the “word clouds”. I look at the word cloud for my course and see big words (these generally reflect the feedback, subject to an exception discussed below) and then smaller words and phrases. Now the smaller ones in a word cloud are obviously meant to be “less” important but these are really quite concerning, so much so that I initially panicked. They include “disrespectful/rude”, “unapproachable”, “not worthwhile”, “superficial” and “unpleasant”. Bear in mind the word cloud precedes the actual comments in my report. None of these terms (nor their synonyms) were used by ANY of the students (unless an organised Moodle page could count as “unapproachable”). And they are really horrible things to say about someone, especially when there is no basis for these kinds of assertions in the actual feedback received.

The problem here is applying a “big data” tool to (very) small data. It doesn’t work, and it can be actively misleading. One of the word clouds (there are different ones for different “topics”) had the word “organised”. That came up because students were telling me my Moodle page was NOT well organised, but it would be easy to think at a quick glance that this was praise.

So what is the point of this exercise? One imagines it might be useful if you have a course with hundreds of students (so that reading the comments would take an hour, say). But the fact that the comments can be actively misleading (as in “organised” above) demonstrates, you still need to read the comments to understand the context. Further, students often make subtle observations in comments (like the fact that too much time was spent on a particular topic) that are difficult to interpret in a word cloud where the phrases are aggregated and sprinkled around the place. So, it doesn’t really save time. The comments still need to be read and reflected on.

Big Data tools always sound very exciting. So much buzz! Imagine if we could predict flu epidemics from Google searches (that no longer works, by the way) or predict crime before it happens (lots of jurisdictions are trying this, particularly in the US). But the truth is more like the word cloud on student feedback – inappropriately applied, error prone, poorly understood by those deploying the tool, and thus often unhelpful. Data analytics CAN be good tool – but it is a bit like a hammer in the hands of those who don’t understand its function and limitations, everything looks like a nail.

Lyria Bennett Moses

Critical thinking in legal education: What? Why? How? By Lucas Lixinski

by Lucas Lixinski

An article in today’s The Conversation asks whether universities really do a good job (or any job at all) of teaching critical thinking. While acknowledging that defining critical thinking is incredibly difficult, and that most definitions out there are vague at best, the article then moves to discussing whether universities actually teach critical thinking in the way they promise they do. In what seems like a job market that increasingly pays a premium for applicants who can demonstrate having learned critical thinking skills, there is a clear financial incentive (beyond the obvious intellectual one) to be more self-aware of what critical thinking is in our discipline, and how we actually go about teaching it.


What is critical thinking in law?

I will not by any means attempt to give an all-encompassing definition of critical thinking more broadly, nor critical thinking in the law. Instead, let me just say where I come from, and try and make sense of the landscape from there. The intention here is to start conversations and provoke reactions, rather than lay down the law (pardon the pun) on the matter.


In my opinion, critical thinking has to do with challenging assumed wisdom, and showing students how to do that themselves. In the law, as far as I can see, there are two ways in which I can do that. The first one is to focus on the contingencies of the law, whether they are economic, historical, or political. Things like the old adage that “the law is made largely by, and for the benefit of, white, male[, heterosexual, able-bodied] property owners” tends to be a great starting point to unravel those contingencies. As is the broader historical context of critical moments in the formation of the legal system (like the influence of Protestant ethics in the shaping of the Common Law and its approaches to labor and property, which is different from the way the mostly Catholic Civil Law jurisdictions behaved in Europe at around the same time).


Secondly, another way of critically thinking about the law, in my view, is to look into the background. More specifically, when we think about, say, a contract for the purchase of milk, the foreground body of rules operating is contract law. However, in the background there are a number of other bodies of law that influence what is possible for a contract (even though on paper contract law is still the quintessential guardian of private liberty), such as food security rules, (international) trade law if milk is considered to be a strategic product the production of which is incentivized, the corresponding tax arrangements, etc. Admittedly, it makes teaching a simple case daunting, but I always tell my students that I don’t need to have all the answers to those all the time, nor do they. But they need to be mindful of those knock-on effects of the simplest legal rule (sort of a “butterfly effect”, but in the law, and hopefully not creating any hurricanes anywhere).


How can we “teach” it?

If you haven’t caught on to it yet, let me out myself here. The way I think about critical thinking, and consequently teach it, is influenced by the way I think and write about the law more generally. Which is to say, I have a hard time dissociating critical thinking as an abstract and transferrable skill from critical legal studies, which is a specific way of theorizing and understanding the law. In other words, the way I conceptualize and “do” critical thinking is deeply influenced by my own bias as a critical scholar (well, much of the time anyway), which is framed by my politics, rather than my raw analytical ability. Assuming this neutrality is desirable (and the article on The Conversation referred to above suggests as much), how do I counter my own biases?


Maybe the assumption is that teaching a lefty orthodoxy induces critical thinking, in that it challenges status quo and conventional wisdom students come to the table with. So, maybe the way to teach critical thinking is to constantly challenge student’s assumptions. Except that those assumptions vary radically within a cohort, and change a lot throughout the degree. Which is to say, it may be safe to assume that a first-year undergraduate class at an elite university is made up of students whom you can assume espouse certain center- to right-leaning assumptions about the world, inherited from their parents and their upbringing. But, after spending a year being challenged on those assumptions, it may be that an upper level class needs to be re-presented with the Liberal version of the world. That is, of course, if critical thinking is to be conveyed through “thick descriptions” of reality as a means to understand and apply the law.


Which is to say, maybe the way to teach critical thinking is to make the teaching less about what I think, and more about playing devil’s advocate all the time to what students think. And that is a fair enough proposition in a student-centric model of education, but, if teaching is also meant to be (at least to some extent) research-driven (not to mention students’ insistence on “answers”), isn’t it my job to convey what I think after all? I constantly try to strike a balance between what I think and other opinions out there, and present them all, but I’m not sure I’m always successful.


This discussion brings to mind an old and still current debate about the purpose of legal education. Is legal education about teaching substantive knowledge of the law, or just skills (“thinking like a lawyer”)? I tend to think the latter, but, in considering the legal profession is subject to an increasingly strict regulatory environment, content is also incredibly important. It is also easier to measure and assess. Problem questions have a way of assessing critical thinking, but often enough (as people marking exams everywhere may attest to), answers to problem questions can too easily devolve into knowledge-spewing for significant segments of the student population.


So, what to do?

I honestly don’t know, and invite other people’s views on the matter. As far as I can see, I will keep on trying to challenge students at every turn (and have they challenge me), but being mindful that my opinion counts, while certainly not the only one that does.


In one of my classes (an Introduction to the Legal System-type class, called “Introducing Law and Justice”), I have the privilege of talking to students in one of their early contacts with the legal discipline. And in doing that I present students with a list of questions that they should be asking of materials they read (cases, statutes, scholarly texts) as a means to stimulate critical thinking:

–        Why is the law this way?

–        Who stands to gain?

–        Who loses?

–        What does the law as is miss? What are its blind spots?

–        What do other people do faced with similar legal problems, and why? Can we learn lessons there?

–        When was this case decided? What was the broader context around this case?

–        What was the court / law-maker trying to say between the lines?

–        Who is the court / law-maker (white, male, property owner)?

–        What is this legal statement / assertion / rule a reaction to?

–        How does the private affect the public (and vice-versa)?


That strikes me as a fairly useful checklist to spark critical thinking, on the models above. But are there other ways of doing that in law teaching? Let me hear your thoughts!



Can Teaching be Measured?

4175299981_614e7d9dc5 (1)


By Justine Rogers

Last week UNSW had its second ‘Great Debate’, introduced last year as a fun, accessible way for the UNSW community to explore a serious and stirring topic. (For a post on last year’s, click here)

Each team: professor-manager, non-prof academic, and student.

The topic: Of Course Teaching Can be Measured (it’s a 5.3!).

I was on the affirmative (which I knew going in would be tough).

Given it was a private event for staff and students, I’ve written this assuming some version of the Chatham House Rule applies.

The affirmative’s arguments were:

  1. Teaching can be measured, albeit imperfectly, and certainly better and more reliably than it is now.
  2. Teaching needs to be measured to enhance the quality, rewards and status of teaching.

The negative’s arguments were:

  1. Teaching cannot be measured, only learning experiences and learning outcomes can. 
  2. Teaching measures are flawed and unreliable.

The negative committed to the empirical questions, whereas I tried (unsuccessfully in the 4 or so mins we had) to engage both sides in the wider empirical and normative argument suggested in affirmative point 2: whether there is some positive correlation between measurement, and motivation, quality and status, and therefore whether a more robust measurement of teaching is worthwhile.

I wish we’d had the format and time to examine this: whether this is true, or whether, using research measures as example, such measures have too many biases, perverse incentives, and inefficient and/or demoralising effects to be of real value (even if it entails superficial value). 

I will share my main arguments here, some of which I am fairly convinced, many posed as part of my role on the affirmative side, and some raised in the spirit of fun and provocation. Above all, I think the topic raised several questions left that need to be contemplated, many of which I’ve posted below – so please share your thoughts!

Continue reading

What lawyers actually do in practice (at least in the US)

SSRN has recently posted a great ethnographic study of young US lawyers in terms of what they actually do in the office.

Sinsheimer, Ann and Herring, David J., Lawyers at Work: A Study of the Reading, Writing, and Communication Practices of Legal Professionals (March 14, 2016). Legal Writing Journal, Vol. 21, Forthcoming; U. of Pittsburgh Legal Studies Research Paper No. 2016-11.

It includes great evidence of lawyers dealing with the following (SSRN pinpoints):

  • Using close reading and skimming strategies (pp13ff)
  • Strategic reading (p23, 30ff)
  • Reading from computer screens (p26) but using printed materials by preference (p24)
  • Huge use of email for written communication (p45ff)
  • Use of precedents (p48)
  • Reviewing and revising constantly (p49ff), being meticulous (p50)
  • Research/writing nexus (p51ff)
  • Interpersonal skills and stress in the office (p58ff)
  • Time-management (p60ff)
  • Cross cultural communication (p64)
  • Developing professional identities (p66ff)
  • Suggestions for curriculum change (p24, 71)

It’s a wonderful collection of vignettes and data that help to flesh out what we are often  trying to impress on students are the real skills they need in preparing for legal practice environments.



Love lectures?

A recent New York Times opinion piece goes slightly against the learning and teaching orthodoxy but arguing in FAVOUR of lectures – It looks at some of the advantages of a lecture format, particularly in the humanities, including teaching students how to listen critically, take notes that summarise ideas and arguments, and learn to understand before commenting/opining.

In law, there is some information that is best communicated through lecture elements, but my classes run in a more “Socratic” or questioning style (having been partially trained in the US), with problem-solving, group discussions and class debate. So I don’t do a lot of pure “lecture” although there is some content that I do present in this way.

But it got me thinking. I would probably be frustrated if I wanted to learn something in an area with which I wasn’t familiar (say at a conference or seminar) and the speaker adopted an “active learning” approach. Sometimes all you want to do is hear someone knowledgeable about something deliver an engaging, interesting and informative “lecture”. And when listening to such, I am rarely “passive” but usually constantly questioning them (initially in my head and eventually by raising my hand in question time). Of course, one difference is that I already know how to “do” legal reasoning, so that is not what I am learning. But the same could be said of later year students too.

So, my question is this: When are lectures the best way to teach?


Lyria Bennett Moses

Postcard from Mexico

Anna Cody

mexico flag


Experiencing another law school, and particularly another legal clinic using clinical education methodology is always eye opening.   For the last 9 months I have been collaborating with the National Autonomous University of Mexico, Human Rights program legal clinic. It has given me some fascinating insights into both the Mexican legal and political system, as well as what it’s like to be a law student in the biggest public university in Latin America.

The clinic here focuses on access to information in migration matters and disability rights cases. They have made over 100 access to information claims with some success but which demonstrate how un-transparent the immigration detention system is.  And as most clients are from Central America and are on their way to the USA, the clients don’t want to stay in Mexico to pursue their legal rights when they are mistreated, tortured or detained illegally. The disability rights cases challenge the automatic use of ‘guardianship’ for many people with disability. They have also supported various people seeking de-institutionalisation. Students are an integral part of these cases, researching them, interviewing and advising clients and representing clients in court. Cases in Mexico are mostly run on the papers without oral evidence, submissions or argument.

Three insights

1. Students, the world over, thrive when given responsibility for their legal work, and are supported in their learning by thorough and effective supervision. The students in Mexico are given a lot of responsibility and in order to succeed in this, they need intensive support from their supervisor who meets with them, drafts documents and discusses some of the pitfalls of the civil system. Writing in plain language is even more of a challenge in Mexico than in Australia where students receive little training in how to communicate effectively.

2.  Despite the various criticisms of Australian ethical frameworks and regulatory system, having some ethical rulings and requirements are definitely preferable to not having any at all. In Mexico there is no ethical code to guide and regulate lawyers. It is a real discussion between supervisor and students about how to influence a judge and whether or not paying bribes is appropriate. The fact that this can be discussed openly is a tribute to both teacher and students. And yet for an Australian lawyer this explicit form of “influencing” by bribe is shocking. It strikes me how hard it is to continue to act in good faith as a lawyer and to believe in a legal system which relies on bribery. This has meant some fascinating discussions with students about what our role is within a legal system as lawyers.

3. When we are working with clients with disability and teaching students about their and our own ableism, it’s essential to design a clinical course with people with disability at the centre. This means integrating speakers, readings, class content which has people with disability giving their perspectives. One of the most powerful classes which the Mexican clinic teaches its clinical students provides perspectives of people with disability, who are not clients. The clinic invites a member of the Colectivo Chuhcan, a collective formed by and for people with psycho-social disability to discuss their experiences of the health and legal system in order to give students an insight into these issues.

These are just 3 of the insights of my time in Mexico, on sabbatical from UNSW.



photo credit

By Carolyn Penfold

Self-plagiarism is a thorny issue. On the one hand I sympathise with students who say ‘but it’s not dishonest, it’s my own work, why can’t I hand it in again?’ This view seems borne out by the University’s website where in answer to the question ‘what is plagiarism?’ the following response is given:

Plagiarism … is using the words or ideas of others and passing them off as your own. Plagiarism is a type of intellectual theft.

It is not until we follow the link to ‘common forms of plagiarism’ that we come across the following (as the very last example):

‘Self-plagiarism’ occurs where an author republishes their own previously written work and presents it as new without referencing the earlier work, either in its entirety or partially.   Self-plagiarism is also referred to as ‘recycling’, ‘duplication’, or ‘multiple submissions of research findings’ without disclosure.   In the student context, self-plagiarism includes re-using parts of, or all of, a body of work that has already been submitted for assessment without proper citation.

At my university students attach a cover sheet to their submissions acknowledging ‘that this work has not been submitted for credit elsewhere.’  We may think that as law students they should be aware of the need to READ declarations before signing them! On the other hand, it is understandable that students might attach and sign the cover sheet without taking too much notice, treating it as a record of their name, student number, word count and date of submission.

Nonetheless, I would have thought it obvious that when we’re assessing students we want to know what they’ve learned and how they’ve developed their understanding during the particular course in which they’re being assessed. However, I have to wonder how obvious this really is in relation to self plagiarism. My brother, a teacher, says ‘it’s their own work, it’s not dishonest to re-submit it.’ My son, a university student, says ‘but that’s what academics do all the time…’

Perhaps we would do better not to call this self plagiarism at all. Let’s separate it from the copying / theft concept and simply call it what it is: recycling or resubmitting. Students know why ‘plagiarism’ is prohibited, now let’s teach them why ‘recycling / resubmitting’ are prohibited, and see if that gets the message across.

Any thoughts?