As we come to the end of semester the following two studies might be of interest:
As many of us struggle through a pile of marking and wonder if we’ve really assessed every student to exactly the same standard, the conclusion to a detailed study of experienced markers in the UK might be food for thought.
… we need fresh thinking about reliability, fairness and standards in higher education assessment, and that our current reliance on criteria, rubrics, moderation and standardising grade distributions is unlikely to tackle the proven lack of grading consensus. One way forward worth considerably more investigation is the use of community processes aimed at developing shared understanding of assessment standards. …
The real challenge emerging from this paper is that, even with more effective community processes, assessment decisions are so complex, intuitive and tacit that variability is inevitable. Short of turning our assessment methods into standardised tests, we have to live with a large element of unreliability and a recognition that grading is judgement and not measurement. Such a future is likely to continue the frustration and dissatisfaction for students which is reflected in satisfaction surveys. Universities need to be more honest with themselves and with students, and help them to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning. Indeed, there is some evidence that students who have developed a more complex view of knowledge see criteria as guidance rather than prescription and are less dissatisfied.
Accepting the inevitability of grading variation means that we should review whether current efforts to moderate are addressing the sources of variation. This study does add some support to the comparison of grade distributions across markers to tackle differences in the range of marks awarded. However, the real issue is not about artificial manipulation of marks without reference to evidence. It is more that we should recognise the impossibility of a ‘right’ mark in the case of complex assignments, and avoid overextensive, detailed, internal or external moderation. Perhaps, a better approach is to recognise that a profile made up of multiple assessors’ judgements is a more accurate, and therefore fairer, way to determine the final degree outcome for an individual. Such a profile can identify the consistent patterns in students’ work and provide a fair representation of their performance, without disingenuously claiming that every single mark is ‘right’. It would significantly reduce the staff resource devoted to internal and external moderation, reserving detailed, dialogic moderation for the borderline cases where it has the power to make a difference. This is not to gainsay the importance of moderation which is aimed at developing shared disciplinary norms, as opposed to superficial procedures or the mechanical resolution of marks. (references omitted)
Sue Bloxham, Birgit den-Outer, Jane Hudson & Margaret Price (2016) Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria, Assessment & Evaluation in Higher Education, 41:3, 466-481, https://doi.org/10.1080/02602938.2015.1024607
“I think too many kids do law,” …”They could spend those years at university doing something more useful and more valuable to whatever career they ultimately took on.” … “A lot of kids do law as though it is a sort of interesting background qualification and it is not,” …”If you want to have a general humanities degree that is an intellectual endowment that isn’t particularly specific in a vocational sense then you would be better off doing languages, history, literature, philosophy. Frankly you would be better off doing economics.”. …”I did law because I wanted to be a lawyer and I practised as a lawyer for a decade,” …”Why would you do dentistry if you don’t want to be a dentist, or medicine if you don’t want to be a doctor?”
Teaching in broader contexts is everything, and much of what Turnbull implies can be learnt from the humanities applies equally to a broad legal education. The point is strongly made by our colleague Ros Dixon in her piece: Studying law is about much more than becoming a lawyer, Malcolm Turnbull
What makes Australian graduates internationally sought after is their breadth of critical thinking skills – which come from immersion in a full three years of reasoned critique and learning the perspectives of others For most this is balanced with a second undergraduate degree that provides a broadening of their knowledge base (often qualifying for an alternative profession).
Of course not all students embrace that approach to education. Turnbull himself is famous for paying other students to attend lectures and take notes for him. Although clearly a prodigious intellect, one suspects he missed out on the immersive educational environment that many modern law schools now provide. The phrase “user experience may vary” seems apt.
Turnbull is right that students shouldn’t do law just because they have the marks. But law underlies so much of society’s successes and failures, and is inescapable in its impact on individual lives. No citizen is lesser for knowing more about individual, corporate and government rights and responsibilities. If ever there was a degree that applies to any possible career it is law.
When we survey our students we find that many don’t see themselves as being solicitors or barristers in the long-term. They have many other interesting career aspirations. But for almost all, law remains an important underlying skill they think they will use.
And internationally law school is looking cool again. After drops in enrolment and dire predictions of mass closures of law schools, enrolment applications in the US have risen this year. One suspects that despite Turnbull’s plea, law will remain wildly popular.
We are excited to let you know that the full program for the Research in Legal Education: State of the Art? conference from 3-5 December is now online http://www.legaledconference.law.unsw.edu.au/
The conference kicks off on Sunday 3 Dec at 6pm with welcome drinks and a book launch. On Monday 4 and Tuesday 5 Dec we have over 50 speakers, and three international plenary speakers.
Best of all, the conference is free.
So make sure you register at https://www.eventbrite.com.au/e/legal-education-conference-2017-tickets-36994327028
by Lyria Bennett Moses
On Monday 18 September, UNSW Law ran a workshop on What Law Students Need to Know about Technology, in which a range of presenters combined with UNSW Law faculty to ask themselves what law graduates will need to know about technology. The workshop aimed to a flavour of some of the technologies that may be important to current and future lawyers and pose the question – what will legal graduates need to know (and thus should be taught in the core) and what should legal graduates have the opportunity to learn (in electives or via extra-curricular activities). The goal of the workshop was not to teach the faculty everything about the technologies, but rather to give members of faculty enough information to be able to make a decision about what to include in their core course and what they may want to learn more about.
Kathy Laster, Director of the Sir Zelman Cowen Centre at Victoria University, began by asking how law graduates should be trained and should market themselves in a world where lawyerly caution is often seen as a barrier to innovation and entrepreneurship. Michael Legg’s summary of the flip committee recommendations reverberated throughout the day, particularly the importance of the skills of project management, working across disciplines and the ability to comprehend the outputs of technology and subject them to challenge and critique. Ross Buckley and David Rountree (Allens) explained core elements of blockchain and its potential impact on transactions as well as smart contracts (code which seeks to facilitate, enforce or automate the performance of a contractual obligation or term and will automatically execute in accordance with its coded terms) and how these relate to broader trends (FinTech, RegTech, monetization of data etc). After morning tea, Peter Leonard (Data Synergies) and Lyria Bennett Moses discussed the uses and limitations of data-driven reasoning.
In the afternoon, Judge Riethmuller of the Federal Circuit Court brought us back to more basic technologies (such as being able to use document templates correctly, understanding strengths and limits of PDF documents, how to capture email and social media as evidence) and gave the excellent suggestion that students should take notes using databases such as Zotero rather than word documents so that it could become a platform for growing their legal knowledge over time. He also discussed the limits of automated means of enhancing access to justice and the challenges the Internet poses to the increase of slightly mad legal claims. Caryn Sandler and Matthew Golab gave an insight into how technology is changing legal practice at Gilbert + Tobin through e-discovery, automated document review, automated searching, collaboration tools and so forth, and introduced a new kind of employee “technolegals” – junior lawyers working on process innovation.
We ended with a panel and a wide ranging discussion with insights from private practice, our own National Children and Youth Law Centre, UTS (and its new Legal Tech major), and Google’s legal team. Broad messages were:
- Don’t get caught in the hype – the curriculum doesn’t need to include every new tech fad, but it does need to teach students to think critically about how they use technology and how technology affects their clients.
- On a related point, students don’t need to know how to code but they need to be able to communicate with someone who writes code and tease out the legal and policy issues related to its use. Students should be exposed to facts (and have to grasp those) as well as law.
- We can help students do this by using modern contexts as example facts in class and in assessment. For example – what remedy would you need to reverse a transaction that had already been processed on the Blockchain?
- We don’t need to reinvent the wheel in articulating problems such as “technological fix” – lots of helpful insights from other disciplines such as Science and Technology Studies.
- The importance of teaching future lawyers to write clear, brief, simple advice that identifies next steps and best solutions (in addition to essays and other longer pieces) – recognising these skills in brevity are useful beyond the legal profession itself.
- Help students develop a well-rounded CV, including through experience at CLCs, Hackathons and other student competitions that encourage development of problem-solving skills.
- Teach students how legal risk fits in with a bigger picture (including the broader risk picture)
by Lucas Lixinski
A recent feature on the Times Higher Education supplement questions the power and effectiveness of learning analytics as a tool in higher education. These tools, which go all out in “flipping”, “blending”, and otherwise “personalizing the student experience”, are all the hype in universities these days. I wonder, though, what they can actually do for legal education.
Don’t get me wrong: I want to like them. While I am not exactly the most tech-savvy, early-adopter type, I think there is some potential in the promises of learning analytics. I would love to be able to know how students interact with the reading materials posted online beyond anecdotal evidence (how much time they spend on the readings; what readings they consider to be priority; even whether they print or read on-line). But I am also mindful that, as far as I can see, legal education is about creating a certain analytical habit of critical thinking. And that habit is best developed, in my opinion, in the back-and-forth in the classroom.
So, maybe what learning analytics can do is to help free up time in the classroom for more of the back-and-forth, and help me as a teacher know what to focus on in that ever-shrinking time.
This strategy, of course, assumes that I take the time to stay on top of the data constantly, while simultaneously not deferring to it to much. There is a risk, as the THE story linked above indicates in all-too-real terms, that learning analytics will displace responsibility away from teachers, that “the machine” will keep track of student progress for me, and I will become numb to signs in the classroom.
Also, and relatedly, we need to be very mindful of who analytics are used for (or, in jargon I am still uncomfortable with in a university context, who the “customer” of data analytics is). If analytics are all about the student experience, does that mean that the data is geared towards catering to student preferences? If that is the case, aren’t we missing the point of education as challenging, rather than settling into, path dependency?
As comfy as it would be for me to have data tell me what courses to explore based on what materials I focus on the most in my readings if I were a student, as an educator I need to know that I need to challenge, rather than cater to, what students think they know about reality. That is how critical thinking develops (or that is my opinion, about which I wrote elsewhere on this blog).
So, learning analytics: yay or nay? I remain hopeful, but I think it all boils down to how the university treats the data, and who it sees as being primarily in control of it: the students whose fees the university needs, or the professionals who fulfill the ethical mandate of education.
We are delighted to announce our third keynote speaker:
Professor Paul Marharg, Distinguished Professor of Practice, Osgoode Hall Law School, York University, Canada.
Paul Maharg is a leading scholar in legal education whose work is focused on interdisciplinary educational innovation, the design of regulation in legal education, and the use of technology-enhanced learning. He joined Osgoode as Distinguished Professor of Practice in May 2017. Since 2013, he has served as Professor of Law and Director of the Centre for Profession, Education and Regulation in Law (PEARL) at Australian National University College of Law as well as Professor of Law (part-time) at Nottingham Law School. He also currently holds Visiting Professorships at Hong Kong University Faculty of Law and Chinese University of Hong Kong Faculty of Law.
He has produced four books and two book series on legal education, and digital games & learning; and influential reports into the regulation of legal services education in England and Wales and other jurisdictions. A Fellow of the Royal Society of Arts, a UK Higher Education Academy (HEA) National Teaching Fellow and a Principal Fellow of the HEA, he has a PhD in Literature, Aesthetics and Philosophy (Edinburgh University); an LLB, Dip Ed and MA in English Literature and Language (Glasgow University); and a PGCE (Jordanhill College of Education).
The updated Call for Papers can be found here: https://wordpress.com/post/lawschoolvibe.wordpress.com/156095
(Re)introducing closed book exams at law school – by Cathy Sherry
Over the past three decades, law schools, along with others, have consistently moved away from closed book exams. Closed book exams have been associated with cramming, rote learning and superficial understanding. In contrast, open book exams, take-home exams and research assignments have been shown to promote deeper learning and genuine understanding.
From its inception in the 1970s, UNSW Law only ever used open book exams, and so the decision to use a closed book exam in the compulsory course, Land Law in 2016, was not a return to past practices, but rather the introduction of a hitherto untested form of assessment.
The decision to change to a closed book exam was motivated by increasing concern among staff, in a number of subjects, about the way in which students are now doing open book exams. For example, I have been teaching law for more than twenty years, and for most of that period would never have considered using anything other than an open book exam. However, in the past five to ten years I have been increasingly frustrated by students who copy out their notes into their exam booklet, rather than answering the question they have been asked. One of our colleagues calls this the ‘tip-truck method’ of answering exams. You dump everything you think could possibly be relevant into the exam paper in the hope that it will earn you marks.
The ‘tip truck method’ works. While it will never earn a student a high distinction, all students who have scored below this and still passed, have passed without getting everything correct. As a result, if a student dumps down a lot of information, some of which is irrelevant (but correct, having been copied from notes), and some of which is on point (even if by chance), it is not possible to fail them. I spent the best part of a decade desperately trying to craft exam questions which minimised ‘information dump’, accompanied by the strict instruction, “Do NOT copy out your notes; answer the question asked”, but to no avail. A sizeable proportion of students would still tell me everything they possibly could about the law of easements, even though 80% of it was irrelevant. If they consistently did this in practice, charging clients for irrelevant advice, it would arguably be professional misconduct.
Excessive use of notes raises particular concerns in the internet era. Pre-internet, it could be safely assumed that most students would be using their own notes in an open book exam, (within the same cohort, sharing notes required time-consuming and expensive photocopying, as well as a measure of organisation most of us lacked). Post-internet, notes can be shared between large numbers of students with the click of a button. Students share notes through google.dox, Drop Box, Facebook, Messenger, WeChat and no doubt other platforms I am too old to know about.
In New South Wales, and possibly in other jurisdictions, the nature of final school exams has also encouraged excessive ‘pre-preparation’, as opposed to simple preparation for exams. By ‘pre-preparation’ I mean the academic equivalent of ‘here’s a cake I made earlier’, as opposed to ‘here are the ingredients I have assembled in preparation for making a cake now’. In the compulsory Higher School Certificate English courses, detailed modules and rubric-based marking mean that it is possible to pre-write and memorise all English essays, including creative writing pieces. Many schools require students to practise this ‘skill’ throughout high school, telling them the essay question well in advance of the exam. Good students will have written their own essays, while others have cobbled together essays from multiple unacknowledged sources and at the extreme, some have bought or entirely replicated other people’s work. The result is that for many of our students, pre-preparing answers or chunks of answers is par for the course and they see no problem in copying out sections of prepared material from their notes into their answers booklets during an exam. This fundamentally changes the nature of an exam answer from work that a marker could be confident was solely the product of a student’s personal understanding at the time the answer was written to an answer that may be their own understanding or it may not.
As a result of these concerns, in 2016 my colleague Leon Terrill and I decided to use a closed book exam for Land Law. There was some consternation from sections of the student body and from some staff. A particular focus of student concern was the stress a closed book exam would cause, and the unreasonableness of expecting students to do a ‘new’ form of assessment in the fourth year of their combined law degrees. Despite this resistance, we pressed on, with the promise to conduct some research on our experience and share this with colleagues.
The exam was conducted without incident. When we received our papers, most of us (other than sessional staff who had marked closed book law exams before at other universities) were surprised by how little difference there was between the length and detail in closed book exam scripts. We had considered doing some research closely comparing open and closed book exam scripts but quickly realised that it was unlikely to be fruitful.
We had applied for a Faculty Learning and Teaching grant which we used to conduct a three-pronged study. First, our colleague and co-researcher, Julian Laurens, conducted a literature review. While this revealed a number of studies that attest to the benefits of open book exams, there was no study specific to our circumstance, that is, a law school changing from open book to closed book exams in the post-internet period.
Second, we surveyed the six teachers who had taught and marked Land Law that semester. There were four sessional staff and two permanent staff, varying from a few semesters’ teaching experience to over twenty years. Staff answered a number of open-ended questions. There were a range of responses, including as noted above, that the scripts in closed book exams were almost as long and as detailed as those in open book exams; that students were more likely to get to the point and not take the ‘kitchen sink’ approach; that it was easier to award high distinctions because it was clearer when a student had attained that level of understanding, and that it was easier to identify students who had not mastered even the basics of the subject.
Finally, we surveyed the students. We administered an anonymous, online/hard copy survey in class in the compulsory course that followed Land Law for most students. We received 174 responses, mainly from our undergraduate rather than JD cohort. There were 39 questions on general study motivations and techniques, the experience of open book exams and the experience of closed book exams.
These are some of the key findings:
- 75% of students said they had never purchased notes or study guides
- 70% of students said they had worked in groups to produce shared notes or answers for assessment or exams
- Almost 70% of students had used another student’s notes in an open book exam
- 60% of students had copied directly from their notes, articles or books into an exam booklet and 50% had included material they did not entirely understand
- 90% of students said they always try to understand the material
- 90% of students said they feel pressure to do well in their law studies and 75% had experienced ‘significant’ anxiety during their studies
- Almost 80% of students thought that open book exams were a good form of assessment
- 50% of students said they went into the closed book exam knowing more than they would know for an open book exam
- 30% of students said they included some material they did not entirely understand
- 35% of students said that not having notes encouraged them to think more in the exam
- Just over 40% of students said they thought it was worthwhile having a mix of open and closed book exams at law school.
Written comments included multiple comments strongly opposed to the closed book exam and strongly in favour of it. Many students thought the closed book exam was a storm in a tea cup and that the form of the exam was not particularly significant.
My conclusion on the change to a closed book format is that unsurprisingly, no form of assessment is perfect; they all have strengths and weaknesses. While I believe that open book exams are generally preferable to closed book, our research confirmed my concerns – many students are not using their own notes in exams and many are copying directly from notes, articles or books into exam booklets. This fundamentally alters the assumption that a formal exam script is all a student’s own work.
Perhaps the most significant finding for me was that 25% of our students were violently opposed to closed book exams, and many because they believed it was unfair to expect them to do a different form of assessment 4/5 of the way through their degree. As we are training lawyers who will go into an ever-changing profession (statutes and case law change, entire areas of clients’ businesses disappear, technology may eradicate a sizeable proportion of the profession), this is extremely concerning. My hunch is that some of our students thrived in a system of rigid, predictable assessment at high school, and may not have the skills or desire to work in less predictable environments. This suggests that the best thing we could do for our student body is mix things up as much as we can, altering assessment from semester to semester, course to course, so that they develop the skills they need to rise to whatever challenges the workplace inevitably throws at them.
Associate Professor Cathy Sherry, Scientia Education Academy Fellow
I was lucky to be part of an invigorating panel discussion hosted by The Australian recently, as part of their Legal Week initiative. I joined two colleagues, our Dean, Professor George Williams, and Associate Professor Michael Legg, as well as Gilbert+Tobin lawyer, Sam Nickless.
It was a wide, free-flowing discussion of the current and future changes to the profession, what Williams called ‘a once-in-a-century process of disruption in the legal market’, and the meanings these have for practice and for legal education. I thought I’d share here some of the main changes discussed and their significance for legal education and the law school.
1. Automation of legal services and its integration with services delivered by people.
2. The possibility for technology to increase access to justice.
2. Unbundling of legal services and flexible, ad hoc and assorted teams brought together for certain projects.
3. Professional ethics centred less and less in the profession and professional association and more in organisations and even the mode of legal service delivery itself.
4. Ethics of introducing new technology to clients or helping them with their artificial intelligence.
5. The globalisation of law – where clients, for instance, are from other jurisdictions.
6. The changes to law firms and their business arrangements, specialisations and recruitment practices, including firms coming from overseas to recruit UNSW Law (or Australian) students.
7. Cross-border disputes and the extent to which Australian courts can remain forums for litigation.
What They Mean for Legal Education and the Law School:
1. Urgent need to answer questions about the value that a person adds as a lawyer.
2. Law schools may need to focus on the ‘professional’ things computers can’t do (or can’t do as well): certain forms of problem solving and analysis, integrity, ethics, professional relationships, creativity and imagination.
3. Law combined with computer science, maths or engineering will add to classic combinations.
4. All law students must understand technology (coding and programming, for instance) regardless of their other degree.
5. Law students must develop capacities in team work and project management.
6. Law students need to be able to identify and address ethics and accountability issues in a range of contexts, when working with external lawyers, non-lawyer professionals and, crucially, with technology.
7. Law students need to understand not just other, non-Australian legal systems but also the cultures in which law operates.
8. Law schools need to help their students appreciate the range of different firms in Australia and the region in making their career decisions.
Without wanting to sound too home-team-y, we’re already doing some pretty fabulous stuff at UNSW Law to support each of these and, through a mini curriculum review, we’re about to do a whole lot more.
(The full video and an edited transcript of the discussion is here.)