(Re)introducing closed book exams at law school – by Cathy Sherry

(Re)introducing closed book exams at law school – by Cathy Sherry

Over the past three decades, law schools, along with others, have consistently moved away from closed book exams. Closed book exams have been associated with cramming, rote learning and superficial understanding. In contrast, open book exams, take-home exams and research assignments have been shown to promote deeper learning and genuine understanding.

From its inception in the 1970s, UNSW Law only ever used open book exams, and so the decision to use a closed book exam in the compulsory course, Land Law in 2016, was not a return to past practices, but rather the introduction of a hitherto untested form of assessment.

The decision to change to a closed book exam was motivated by increasing concern among staff, in a number of subjects, about the way in which students are now doing open book exams. For example, I have been teaching law for more than twenty years, and for most of that period would never have considered using anything other than an open book exam. However, in the past five to ten years I have been increasingly frustrated by students who copy out their notes into their exam booklet, rather than answering the question they have been asked. One of our colleagues calls this the ‘tip-truck method’ of answering exams. You dump everything you think could possibly be relevant into the exam paper in the hope that it will earn you marks.

The ‘tip truck method’ works. While it will never earn a student a high distinction, all students who have scored below this and still passed, have passed without getting everything correct. As a result, if a student dumps down a lot of information, some of which is irrelevant (but correct, having been copied from notes), and some of which is on point (even if by chance), it is not possible to fail them. I spent the best part of a decade desperately trying to craft exam questions which minimised ‘information dump’, accompanied by the strict instruction, “Do NOT copy out your notes; answer the question asked”, but to no avail. A sizeable proportion of students would still tell me everything they possibly could about the law of easements, even though 80% of it was irrelevant. If they consistently did this in practice, charging clients for irrelevant advice, it would arguably be professional misconduct.

Excessive use of notes raises particular concerns in the internet era. Pre-internet, it could be safely assumed that most students would be using their own notes in an open book exam, (within the same cohort, sharing notes required time-consuming and expensive photocopying, as well as a measure of organisation most of us lacked). Post-internet, notes can be shared between large numbers of students with the click of a button. Students share notes through google.dox, Drop Box, Facebook, Messenger, WeChat and no doubt other platforms I am too old to know about.

In New South Wales, and possibly in other jurisdictions, the nature of final school exams has also encouraged excessive ‘pre-preparation’, as opposed to simple preparation for exams. By ‘pre-preparation’ I mean the academic equivalent of ‘here’s a cake I made earlier’, as opposed to ‘here are the ingredients I have assembled in preparation for making a cake now’. In the compulsory Higher School Certificate English courses, detailed modules and rubric-based marking mean that it is possible to pre-write and memorise all English essays, including creative writing pieces. Many schools require students to practise this ‘skill’ throughout high school, telling them the essay question well in advance of the exam. Good students will have written their own essays, while others have cobbled together essays from multiple unacknowledged sources and at the extreme, some have bought or entirely replicated other people’s work. The result is that for many of our students, pre-preparing answers or chunks of answers is par for the course and they see no problem in copying out sections of prepared material from their notes into their answers booklets during an exam. This fundamentally changes the nature of an exam answer from work that a marker could be confident was solely the product of a student’s personal understanding at the time the answer was written to an answer that may be their own understanding or it may not.

As a result of these concerns, in 2016 my colleague Leon Terrill and I decided to use a closed book exam for Land Law. There was some consternation from sections of the student body and from some staff. A particular focus of student concern was the stress a closed book exam would cause, and the unreasonableness of expecting students to do a ‘new’ form of assessment in the fourth year of their combined law degrees. Despite this resistance, we pressed on, with the promise to conduct some research on our experience and share this with colleagues.

The exam was conducted without incident. When we received our papers, most of us (other than sessional staff who had marked closed book law exams before at other universities) were surprised by how little difference there was between the length and detail in closed book exam scripts. We had considered doing some research closely comparing open and closed book exam scripts but quickly realised that it was unlikely to be fruitful.

We had applied for a Faculty Learning and Teaching grant which we used to conduct a three-pronged study. First, our colleague and co-researcher, Julian Laurens, conducted a literature review. While this revealed a number of studies that attest to the benefits of open book exams, there was no study specific to our circumstance, that is, a law school changing from open book to closed book exams in the post-internet period.

Second, we surveyed the six teachers who had taught and marked Land Law that semester. There were four sessional staff and two permanent staff, varying from a few semesters’ teaching experience to over twenty years. Staff answered a number of open-ended questions. There were a range of responses, including as noted above, that the scripts in closed book exams were almost as long and as detailed as those in open book exams; that students were more likely to get to the point and not take the ‘kitchen sink’ approach; that it was easier to award high distinctions because it was clearer when a student had attained that level of understanding, and that it was easier to identify students who had not mastered even the basics of the subject.

Finally, we surveyed the students. We administered an anonymous, online/hard copy survey in class in the compulsory course that followed Land Law for most students. We received 174 responses, mainly from our undergraduate rather than JD cohort. There were 39 questions on general study motivations and techniques, the experience of open book exams and the experience of closed book exams.

These are some of the key findings:

  • 75% of students said they had never purchased notes or study guides
  • 70% of students said they had worked in groups to produce shared notes or answers for assessment or exams
  • Almost 70% of students had used another student’s notes in an open book exam
  • 60% of students had copied directly from their notes, articles or books into an exam booklet and 50% had included material they did not entirely understand
  • 90% of students said they always try to understand the material
  • 90% of students said they feel pressure to do well in their law studies and 75% had experienced ‘significant’ anxiety during their studies
  • Almost 80% of students thought that open book exams were a good form of assessment
  • 50% of students said they went into the closed book exam knowing more than they would know for an open book exam
  • 30% of students said they included some material they did not entirely understand
  • 35% of students said that not having notes encouraged them to think more in the exam
  • Just over 40% of students said they thought it was worthwhile having a mix of open and closed book exams at law school.

Written comments included multiple comments strongly opposed to the closed book exam and strongly in favour of it. Many students thought the closed book exam was a storm in a tea cup and that the form of the exam was not particularly significant.

My conclusion on the change to a closed book format is that unsurprisingly, no form of assessment is perfect; they all have strengths and weaknesses. While I believe that open book exams are generally preferable to closed book, our research confirmed my concerns – many students are not using their own notes in exams and many are copying directly from notes, articles or books into exam booklets. This fundamentally alters the assumption that a formal exam script is all a student’s own work.

Perhaps the most significant finding for me was that 25% of our students were violently opposed to closed book exams, and many because they believed it was unfair to expect them to do a different form of assessment 4/5 of the way through their degree. As we are training lawyers who will go into an ever-changing profession (statutes and case law change, entire areas of clients’ businesses disappear, technology may eradicate a sizeable proportion of the profession), this is extremely concerning. My hunch is that some of our students thrived in a system of rigid, predictable assessment at high school, and may not have the skills or desire to work in less predictable environments. This suggests that the best thing we could do for our student body is mix things up as much as we can, altering assessment from semester to semester, course to course, so that they develop the skills they need to rise to whatever challenges the workplace inevitably throws at them.

Associate Professor Cathy Sherry, Scientia Education Academy Fellow




NewLaw NewLegalEducation

By Justine Rogers

I was lucky to be part of an invigorating panel discussion hosted by The Australian recently, as part of their Legal Week initiative. I joined two colleagues, our Dean, Professor George Williams, and Associate Professor Michael Legg, as well as Gilbert+Tobin lawyer, Sam Nickless.

It was a wide, free-flowing discussion of the current and future changes to the profession, what Williams called ‘a once-in-a-century process of disruption in the legal market’, and the meanings these have for practice and for legal education. I thought I’d share here some of the main changes discussed and their significance for legal education and the law school.

The Changes:

1. Automation of legal services and its integration with services delivered by people.
2. The possibility for technology to increase access to justice.
2. Unbundling of legal services and flexible, ad hoc and assorted teams brought together for certain projects.
3. Professional ethics centred less and less in the profession and professional association and more in organisations and even the mode of legal service delivery itself.
4. Ethics of introducing new technology to clients or helping them with their artificial intelligence.
5. The globalisation of law – where clients, for instance, are from other jurisdictions.
6. The changes to law firms and their business arrangements, specialisations and recruitment practices, including firms coming from overseas to recruit UNSW Law (or Australian) students.
7. Cross-border disputes and the extent to which Australian courts can remain forums for litigation.

What They Mean for Legal Education and the Law School:

1. Urgent need to answer questions about the value that a person adds as a lawyer.
2. Law schools may need to focus on the ‘professional’ things computers can’t do (or can’t do as well): certain forms of problem solving and analysis, integrity, ethics, professional relationships, creativity and imagination.
3. Law combined with computer science, maths or engineering will add to classic combinations.
4. All law students must understand technology (coding and programming, for instance) regardless of their other degree.
5. Law students must develop capacities in team work and project management.
6. Law students need to be able to identify and address ethics and accountability issues in a range of contexts, when working with external lawyers, non-lawyer professionals and, crucially, with technology.
7. Law students need to understand not just other, non-Australian legal systems but also the cultures in which law operates.
8. Law schools need to help their students appreciate the range of different firms in Australia and the region in making their career decisions.

Without wanting to sound too home-team-y, we’re already doing some pretty fabulous stuff at UNSW Law to support each of these and, through a mini curriculum review, we’re about to do a whole lot more.

(The full video and an edited transcript of the discussion is here.)

Research in Legal Education conference update

We are pleased to announce a second keynote speaker for the December conference at UNSW:

Professor Fiona Cownie, Pro Vice Chancellor (Education and Student Experience), Keele University

Fiona Cownie’s research interests centre on legal education and the legal system. Her work covers all areas of legal education. It includes analysis of the ways in which law is taught, especially arguments about the importance of educational theory and philosophy to legal education. It also includes work on the purpose of the law school and on relationships between the academic study of law and vocational training as required by the legal professions.

Professor Cownie has also made a major contribution to debates about the academic staff who work in law schools; she has carried out extensive empirical research in this area, both in the U.K. and in Canada, and has written a monograph and a series of articles exploring different aspects of legal academic life, including work on women in the law school and values in law teaching, as well as on the everyday dress of law teachers. Recently, she has become interested in the history of law schools, and in particular of legal academics. She is the author of Legal Academics: culture and identities (Hart, 2004) and co-author of A Great and Noble Occupation: the history of the Society of Legal Scholars (Hart, 2009). Her latest research project is an extended study of the biographies of early women law professors in the U.K.

Full details of the call for papers can be found here: https://lawschoolvibe.wordpress.com/2017/06/06/call-for-papers-research-in-legal-education-state-of-the-art/ 

Student evaluations and innovative teaching


Julian Laurens

Recent research on student evaluations of teacher performance suggest, I argue, that assessing teacher performance via narrowly constructed student evaluation surveys designed to produce a quantifiable indicator of ‘good teaching’ may in fact have the indirect consequence of hindering innovation in ‘good’ teaching approaches. The May 2017 study [1] findings are consistent with a growing body of research [2] that shows university students are simply unable to recognise ‘good teaching’ or ‘what is best for their own learning’. The research identifies that students do not reward ‘good’ or ‘innovative’ teaching in the sense of an improved student evaluation mark for that teacher. Indeed, the evidence suggests the biggest factor informing a positive evaluation from a student is the grade the student is given.

On the other hand, as the studies show, student ignorance of ‘good teaching’ exists alongside evidence demonstrating that quality teaching has a positive impact on a student’s grades and learning outcomes. Moreover, when exposed to quality teaching the improved learning outcomes are transferred to subsequent subjects. This effect is consistent with findings from educational psychology: the work of Albert Bandura and other social-cognitive theorists on the development of student self-efficacy is particularly relevant. Worryingly, the research suggests that students who rated their teachers based on marks they received actually did worse on subsequent courses.

The research thus far raises implications for legal education. A specific and immediate issue raised by the findings is that given student evaluations of teaching are used by University and Faculty management when considering an academic’s career progress, there is a real risk that teachers may choose to ‘play it safe’. What incentive is there for a teacher to actually try something new in their teaching given that they will potentially not receive improved evaluation scores, and in fact may be penalised by students for being ‘innovative’?

A limitation is that much of the research so far is from non-law disciplines. There is yet to be a systematic look at how this particular problem with student evaluations does (if at all) apply to an Australian law school. This should not preclude us from taking note though.  Issues surrounding student evaluations generally have long been recognised in law schools. As Roth said (back in 1984), “[e]veryone agrees that evaluations ought to be done, but few are satisfied that it is now being done properly, or meaningfully” [3]. This remains the wider challenge. Elsewhere on this blog, colleagues Justine Rogers and Carolyn Penfold have also begun examining issues surrounding student’s evaluations.

In conclusion, for present purposes, the research may support the argument that over reliance on current narrow neo-liberal/managerialist inspired approaches to student evaluations of teachers at law schools in Australia may actually hinder innovative ‘good’ teaching practice. Current iterations of such practices can indeed appear as mechanisms of academic control, rather than tools that promote mutually collaborative learning environments. The research calls into question more broadly claims by universities to be dedicated to ‘good teaching’ and ‘innovation in learning’. There is a need to explore this issue further in the context of Australian legal education, situating it alongside continuing conversations around what actual good teaching looks like in law for example. I would like to make three brief practical observations at this stage derived from analysis of the research that may assist us to address some of the negative challengeS posed by the findings:

  • Firstly, we should always strive to improve our teaching, and we should be uncompromising in that. We should communicate this commitment to our students;
  • Secondly, we need to do a better job of explaining to students our teaching approach and rationales and how something relates to the learning experience;
  • Thirdly, we need to give students more opportunities to practice and reflect on what they have achieved along the way. Students need to see how they are progressing. They need to be regularly reminded of the value of learning. Royce Sadler’s work on the importance of feedback is worth reflecting on here.

[1] Brian A Jacob et al, ‘Measuring Up: Assessing instructor effectiveness in higher education’ (2017) 17(3) Education Next 68.

[2] See e.g. Michela Bragga et al, ‘Evaluating students’ evaluations of professors’ (2014) 41 Economics of Education Review 71; Arthur Poropat, ‘Students don’t know what’s best for their own learning’ (The Conversation, November 19, 2014) (online: http://theconversation.com/students-dont-know-whats-best-for-their-own-learning-33835 )

[3] William Roth, ‘Student Evaluation of Law Teaching’ (1984) 17(4) Akron Law Review 609, 610.


Call for Papers
What is it? Are we really doing it? What can it achieve? Where is it going?
3rd to 5th December 2017
Hosted by UNSW Law Legal Education Research Group

Legal education has developed enormously in recent years. An increase in law schools and law students, more education oriented regulation and an increased understanding of the professional nature of teaching have led to a growth in legal education publications.

This conference seeks to interrogate the purpose and the power of that body of scholarship. Is scholarship in legal education a subset of legal scholarship, or of educational scholarship, or is it sui generis? To whom is it speaking and what is it seeking to achieve – to convince legal academics to think differently about law, or about their practice as teachers of law, or simply to help them become more effective law teachers? Is it rigorous enough? Does it have practical value, and clear applicability in the legal classroom? Can anyone do it? Does being a law teacher give you instant access and insight into legal education scholarship? How do we know what goes on in law classrooms let alone the minds of law students as they make that slow transition to ‘thinking like a lawyer’?

We welcome a broad range of responses to the conference theme – from classroom experiences to theoretical and historical interrogations of what it means to be engaged in legal education scholarship. Papers can deal with a range of issues including:

– The history, present and future directions of legal education scholarship
– The theory and practice of legal education scholarship
– The politics of legal education scholarship
– Legal education scholarship and the nexus between teaching and research
– The relation between legal education scholarship and doctrinal, empirical and theoretical scholarship in law
– Methodological debates in legal education scholarship
– The status of the classroom – face-to-face, clinical and online – in legal education scholarship
– Legal education scholarship and the contemporary university (in Australia and internationally)
– The practicalities and ethics of student involvement in legal education scholarship – ‘students as partners’ or students as data-providers (or a mixture of both)?
– Scholarship on the use of technology in the law classroom
– Legal education, legal education scholarship and social justice in the 21st century
– The globalisation of legal knowledge, legal practice and legal education
– Legal education scholarship on international students

More broadly, we welcome any paper or poster proposal in the genre of legal education on any substantive area of law that makes an effort to think through the status of legal education scholarship. We particularly welcome doctoral, early career and junior scholars.
The call for papers for this Conference closes on 1 September 2017. There are two types of participation in the conference: conference papers and poster presentations. Please send conference paper abstracts of no more than 500 words to law-education@unsw.edu.au by 1 September 2017. If accepted, participants should plan to present papers of 20 minutes’ duration. Participants wishing to present a poster should send a 300 words description of their poster presentation by the same date. We plan to publish a selection of the conference papers in a special edition of a journal.

We are delighted to announce three keynote speakers of the conference:

Professor Carrie Menkel-Meadow, Chancellor’s Professor of Law at the University of California, Irvine, School of Law, USA..

A founder of the dispute resolution field, Professor Menkel-Meadow came to UC Irvine School of Law from Georgetown University Law Center, where she is the A.B. Chettle, Jr. Professor of Dispute Resolution and Civil Procedure and Director of the Georgetown-Hewlett Program in Conflict Resolution and Legal Problem Solving. She has been the Faculty Director of Georgetown’s innovative partnership with 20 law schools from around the world, the Center for Transnational Legal Studies in London, in which faculty and students from participating programs study international and comparative law in a multi-national setting.

Professor Menkel-Meadow was a professor of law at UCLA for nearly 20 years, also serving as a professor in the Women’s Studies program, Acting Director of the Center for the Study of Women, and Co-Director of UCLA’s Center on Conflict Resolution. She has taught as a Distinguished Visiting Professor of Legal Theory at the University of Toronto, a visiting professor at Harvard Law School, Stanford Law School, and as a clinical professor at the University of Pennsylvania.

Professor Fiona Cownie, Pro Vice Chancellor (Education and Student Experience), Keele University, England.

Fiona Cownie’s research interests centre on legal education and the legal system. Her work covers all areas of legal education. It includes analysis of the ways in which law is taught, especially arguments about the importance of educational theory and philosophy to legal education. It also includes work on the purpose of the law school and on relationships between the academic study of law and vocational training as required by the legal professions.

Professor Cownie has also made a major contribution to debates about the academic staff who work in law schools; she has carried out extensive empirical research in this area, both in the U.K. and in Canada, and has written a monograph and a series of articles exploring different aspects of legal academic life, including work on women in the law school and values in law teaching, as well as on the everyday dress of law teachers. Recently, she has become interested in the history of law schools, and in particular of legal academics. She is the author of Legal Academics: culture and identities (Hart, 2004) and co-author of A Great and Noble Occupation: the history of the Society of Legal Scholars (Hart, 2009). Her latest research project is an extended study of the biographies of early women law professors in the U.K.

Professor Paul Marharg, Distinguished Professor of Practice, Osgoode Hall Law School,  York University, Canada.

Paul Maharg is a leading scholar in legal education whose work is focused on interdisciplinary educational innovation, the design of regulation in legal education, and the use of technology-enhanced learning. He joined Osgoode as Distinguished Professor of Practice in May 2017.  Since 2013, he has served as Professor of Law and Director of the Centre for Profession, Education and Regulation in Law (PEARL) at Australian National University College of Law as well as Professor of Law (part-time) at Nottingham Law School. He also currently holds Visiting Professorships at Hong Kong University Faculty of Law and Chinese University of Hong Kong Faculty of Law.

He has produced four books and two book series on legal education, and digital games & learning; and influential reports into the regulation of legal services education in England and Wales and other jurisdictions. A Fellow of the Royal Society of Arts, a UK Higher Education Academy (HEA) National Teaching Fellow and a Principal Fellow of the HEA, he has a PhD in Literature, Aesthetics and Philosophy (Edinburgh University); an LLB, Dip Ed and MA in English Literature and Language (Glasgow University); and a PGCE (Jordanhill College of Education).

The conference will be held at UNSW Law school. It is free but registration is required as numbers will be limited. Any inquiries can be directed to: law-education@unsw.edu.au

The importance of regulatory context: some questions for legal educators

By Justine Rogers

ANU College of Law hosted its annual legal ethics roundtable last week. The theme was ‘reimagining lawyer regulation’.

The regulatory ideas presented raised many worthwhile questions for legal education. I’ve selected two here.

From the talk given by keynote speaker, Professor Leslie Levin, expert in the legal profession, ethical decision-making and lawyer discipline, University of Connecticut:

1) How do we teach law students to be professional when the primary influence over their ethical attitudes, decision-making and compliance will be their particular, divergent work contexts?

Building on other research, Levin’s study of some 1300 lawyers revealed that of all the determinants of future ethical behaviour, most decisive is practice context (workplace, type of client, court etc) and the behaviour of those who inhabit it. Far less so or less significantly are the things that students need to disclose for admission (such as mental health).

Context shapes the importance given to the professional bodies outside the workplace when deciding what values and rules are worth following, such as, the associations, the regulators (or the disciplinary architecture), the court, and the insurers. For instance, big firms look within their own firms and otherwise interact with insurers, prosecutors are less concerned with criminal liability (when does that happen?), in-house counsel don’t worry about discipline, whereas sole practitioners do. In other words, different things matter to different practices.

Levin asked: How do we create professional training, sanctions, and incentives in order to motivate lawyers to behave the way we want them to and to teach them what positive norms there are in the profession? How do we regulate lawyers if context (what matters in each context) is the key variable?

This also means we need to think about how to develop professional integrity and core ethical skills among students for contexts that will introduce, emphasise and enforce professional values in very different ways.

From the talk given by Dr Stephen Tang, Lecturer, ANU College of Law:

2) What is the proper role of behavioural ethics (or any applied psychology) in legal ethics courses?

Behavioural (Legal) Ethics is ‘trending’ in legal ethics education. Popularised by books like Thinking Fast and Slow, and Nudge, this scholarship argues that people are fundamentally irrational and use cognitive short cuts that can lead to suboptimal decision-making. At UNSW Law, we use behavioural legal ethics material in our core course to allow students to better identify, prepare for and discuss ethics issues. Our material includes this leading Robbennholt and Sternlight piece and this wonderful series of ‘Ethics Unwrapped’ videos from UT Austin, both of which were commended by Tang.

This material is useful, he argued, at least as a set of cautionary tales of how irrationality can contribute to immorality and discrimination in routine, subconscious ways. But his concern with behavioural ethics, or the behavioural economics or applied psychological approaches from which it derives, is how the information can be used  – and is used – to manipulate behaviour, even if in a benignly paternalistic or ‘nudging’ way. Equally concerning, its use is usually guided by simplistic, narrow, short-term and consequentialist (eco-based) ideas of people’s motivations and values.

When regulating lawyers, he argued, we need to contemplate and include professional narratives, and organisational climates, and cultures – or more complexity. We will have, then, a better chance of success in fostering certain positive behaviour because we will understand bigger psychological dimensions and developments over time, not just quick, aesthetic behavioural changes, of the same sort as the etched image of a fly in a urinal used to lead to cleaner bathrooms (yes, that’s a real example of applied psych). “If regulation is inescapable, then we must understand people in context. We need to have a sense of our own limits as regulators when deciding what other people decide.”

Similarly, I would say, when teaching students behavioural ethics to more effectively engage in ethical discussions, it must involve a commitment to use the information transparently and inclusively as a way of better understanding and discussing mistakes and fallibilities, of broadening the range of harms considered, and not ruling out or underestimating other motivations, explanations and solutions.

Working the nexus and drawing pictures

Two new articles from the Legal Education Review may be of interest.

Working the Nexus: Teaching Students to Think, Read and Problem-solve Like a Lawyer
(by Kate Galloway, Mary Heath, Alex Steel, Anne Hewitt and Natalie Skead) draws on our Smart Casual experiences and considers the way in which students need to learn to read, think and write simultaneously.  The paper explores how each is a both a separate and linked skill and the way teachers move between explicit and implicit reference to these skills in teaching. It argues that no one skill alone amounts to “thinking like a lawyer”, and that although the skills are often presented as lineal processes, for the student it is deeply iterative.

Where are the Graphics? Communicating Legal Ideas Effectively Using Images and Symbols (by Tania Leiman) looks at the history and current use of visual images to describe the law.  Tania provides an rich and fascinating set of examples of how lawyers use visual aids and visual thinking internationally and in Australia.  She finishes with a set of issues for legal education to ponder.  In a similar vein to the increasing concerns about coding and law school, Tania suggests we can’t teach students to be designers, but we can teach them to think in ways that enable them to both create and critique visual summaries of the law.  As she points out, her article contains no graphics, underlining the difficulty of converting complex arguments to accurate visual guides.

Big Data Analytics on student surveys

It’s the new “thing” – analytics applied to student responses to courses. And it is really quite scary.

To give an example, I will share my own results from a recently taught course of 22 students of which 10 filled out the survey. This is “small data”. It takes about 5-10 minutes (generously) to read and reflect upon the student feedback. Since I am sharing, they generally liked the course including guest lectures and excursions, but felt that one topic didn’t need as much time and that my Moodle page wasn’t well organised. All very helpful for the next time I run the course (note to self to start my Moodle page earlier and tweak the class schedule).

The problem is no longer the feedback, it is the “analytics” which now accompany it. The worst is the “word clouds”. I look at the word cloud for my course and see big words (these generally reflect the feedback, subject to an exception discussed below) and then smaller words and phrases. Now the smaller ones in a word cloud are obviously meant to be “less” important but these are really quite concerning, so much so that I initially panicked. They include “disrespectful/rude”, “unapproachable”, “not worthwhile”, “superficial” and “unpleasant”. Bear in mind the word cloud precedes the actual comments in my report. None of these terms (nor their synonyms) were used by ANY of the students (unless an organised Moodle page could count as “unapproachable”). And they are really horrible things to say about someone, especially when there is no basis for these kinds of assertions in the actual feedback received.

The problem here is applying a “big data” tool to (very) small data. It doesn’t work, and it can be actively misleading. One of the word clouds (there are different ones for different “topics”) had the word “organised”. That came up because students were telling me my Moodle page was NOT well organised, but it would be easy to think at a quick glance that this was praise.

So what is the point of this exercise? One imagines it might be useful if you have a course with hundreds of students (so that reading the comments would take an hour, say). But the fact that the comments can be actively misleading (as in “organised” above) demonstrates, you still need to read the comments to understand the context. Further, students often make subtle observations in comments (like the fact that too much time was spent on a particular topic) that are difficult to interpret in a word cloud where the phrases are aggregated and sprinkled around the place. So, it doesn’t really save time. The comments still need to be read and reflected on.

Big Data tools always sound very exciting. So much buzz! Imagine if we could predict flu epidemics from Google searches (that no longer works, by the way) or predict crime before it happens (lots of jurisdictions are trying this, particularly in the US). But the truth is more like the word cloud on student feedback – inappropriately applied, error prone, poorly understood by those deploying the tool, and thus often unhelpful. Data analytics CAN be good tool – but it is a bit like a hammer in the hands of those who don’t understand its function and limitations, everything looks like a nail.

Lyria Bennett Moses

Legal Writing Resources

Chantal Morton from the University of Melbourne has put together this very helpful list of resources for legal writing.  Many thanks Chantal!

  1. Professor Ellen Zweibel and Virginia McRae at the University of Ottawa Faculty of Law (Canada) have led the design of a great website with modules on how to write legal memos and edit your own work: http://pointfirstwriting.com/.  They are in the process of designing two more modules.  The resources are fantastic – and why reinvent the wheel?  I have been pointing my students to the resource ever since I heard about it at the Legal Writing Institute conference in Portland.
  2. I am in the midst of turning the 2015 Legal Writing symposium website into a blog with broader reach with respect to skills for law (https://skillsforlaw.wordpress.com/ ).  If you have any resources, articles, or websites you would like to see featured, please send me an email.  At the moment, we are storing some teaching modules from Charles Calleros.  I love using his exercise Rules for Lina (slightly modified for an Australian audience – feel free to contact me if you’d like the modified script and tips for having students role play Lina and her parent).  The videos will be posted in the new year.
  3. I just received my ‘holiday reading’ and thought I’d mention two Australian resources: a second edition of Nichola Corbett-Jarvis and Brendan Grigg, Effective Legal Writing: A Practical Guide (LexisNexis, 2nd ed, 2017) and the new book by Kenneth Yin and Anibeth Desierto, Legal Problem Solving and Syllogistic Analysis (LexisNexis, 2016).
  4. I am just back from a two-week stint at Stetson University College of Law (Florida).  Professor Kirsten Davis has put together a new project called Teaching Legal Writing: Out of the Box Ideas (http://www.stetson.edu/law/academics/lrw/teaching-legal-writing.php?ad=20160702).  You can email her directly for your free ‘box’ (kkdavis@law.stetson.edu) – I have already developed one of the activities for a workshop I am running for my JD and masters students in 2017.
  5. The Legal Writing Institute is primarily USA-based but the (free!) membership includes legal research and writing faculty from Australia, Canada and the UK (etc).  They publish some really interesting (and helpful) articles through the Legal Writing Journal and The Second Draft (I just happen to be one of the editors of the latter).  They also have something called The Idea Bank where legal research and writing teachers will post modules for others to adapt and use.  You can get access to it for free – just post one of your own modules (although if you are not ready to share, they’ll let you poke around without a contribution): http://www.lwionline.org/
  6. The LWI conference takes place every other year – so the next big one is in 2018.  In 2017, there are two related conferences you might want to consider: a) the Association of Legal Writing Directors has a few opportunities for professional development listed here http://www.alwd.org/events/ and b) the next Global Legal Skills Conference (easily the best conferences I have ever been to) takes place in March, 2017, in Mexico.  Details here: http://glsc.jmls.edu/2017/ (although keep in mind I am hoping to get them to come to Australia in 2018 so if you can’t make the trip to Mexico, keep a watch on the announcements for 2018!)

Create a free website or blog at WordPress.com.

Up ↑