As we come to the end of semester the following two studies might be of interest:
As many of us struggle through a pile of marking and wonder if we’ve really assessed every student to exactly the same standard, the conclusion to a detailed study of experienced markers in the UK might be food for thought.
… we need fresh thinking about reliability, fairness and standards in higher education assessment, and that our current reliance on criteria, rubrics, moderation and standardising grade distributions is unlikely to tackle the proven lack of grading consensus. One way forward worth considerably more investigation is the use of community processes aimed at developing shared understanding of assessment standards. …
The real challenge emerging from this paper is that, even with more effective community processes, assessment decisions are so complex, intuitive and tacit that variability is inevitable. Short of turning our assessment methods into standardised tests, we have to live with a large element of unreliability and a recognition that grading is judgement and not measurement. Such a future is likely to continue the frustration and dissatisfaction for students which is reflected in satisfaction surveys. Universities need to be more honest with themselves and with students, and help them to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning. Indeed, there is some evidence that students who have developed a more complex view of knowledge see criteria as guidance rather than prescription and are less dissatisfied.
Accepting the inevitability of grading variation means that we should review whether current efforts to moderate are addressing the sources of variation. This study does add some support to the comparison of grade distributions across markers to tackle differences in the range of marks awarded. However, the real issue is not about artificial manipulation of marks without reference to evidence. It is more that we should recognise the impossibility of a ‘right’ mark in the case of complex assignments, and avoid overextensive, detailed, internal or external moderation. Perhaps, a better approach is to recognise that a profile made up of multiple assessors’ judgements is a more accurate, and therefore fairer, way to determine the final degree outcome for an individual. Such a profile can identify the consistent patterns in students’ work and provide a fair representation of their performance, without disingenuously claiming that every single mark is ‘right’. It would significantly reduce the staff resource devoted to internal and external moderation, reserving detailed, dialogic moderation for the borderline cases where it has the power to make a difference. This is not to gainsay the importance of moderation which is aimed at developing shared disciplinary norms, as opposed to superficial procedures or the mechanical resolution of marks. (references omitted)
Sue Bloxham, Birgit den-Outer, Jane Hudson & Margaret Price (2016) Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria, Assessment & Evaluation in Higher Education, 41:3, 466-481, https://doi.org/10.1080/02602938.2015.1024607
Our colleague Alex Steel has recently given a public lecture as a Fellow of the UNSW Scientia Education Academy. A video of his lecture is now online (a bibliography is on the right of the page)
Our colleague Cathy Sherry has recently given a public lecture as a Fellow of the UNSW Scientia Education Academy. A video of her lecture is now online
“I think too many kids do law,” …”They could spend those years at university doing something more useful and more valuable to whatever career they ultimately took on.” … “A lot of kids do law as though it is a sort of interesting background qualification and it is not,” …”If you want to have a general humanities degree that is an intellectual endowment that isn’t particularly specific in a vocational sense then you would be better off doing languages, history, literature, philosophy. Frankly you would be better off doing economics.”. …”I did law because I wanted to be a lawyer and I practised as a lawyer for a decade,” …”Why would you do dentistry if you don’t want to be a dentist, or medicine if you don’t want to be a doctor?”
Teaching in broader contexts is everything, and much of what Turnbull implies can be learnt from the humanities applies equally to a broad legal education. The point is strongly made by our colleague Ros Dixon in her piece: Studying law is about much more than becoming a lawyer, Malcolm Turnbull
What makes Australian graduates internationally sought after is their breadth of critical thinking skills – which come from immersion in a full three years of reasoned critique and learning the perspectives of others For most this is balanced with a second undergraduate degree that provides a broadening of their knowledge base (often qualifying for an alternative profession).
Of course not all students embrace that approach to education. Turnbull himself is famous for paying other students to attend lectures and take notes for him. Although clearly a prodigious intellect, one suspects he missed out on the immersive educational environment that many modern law schools now provide. The phrase “user experience may vary” seems apt.
Turnbull is right that students shouldn’t do law just because they have the marks. But law underlies so much of society’s successes and failures, and is inescapable in its impact on individual lives. No citizen is lesser for knowing more about individual, corporate and government rights and responsibilities. If ever there was a degree that applies to any possible career it is law.
When we survey our students we find that many don’t see themselves as being solicitors or barristers in the long-term. They have many other interesting career aspirations. But for almost all, law remains an important underlying skill they think they will use.
And internationally law school is looking cool again. After drops in enrolment and dire predictions of mass closures of law schools, enrolment applications in the US have risen this year. One suspects that despite Turnbull’s plea, law will remain wildly popular.
We are excited to let you know that the full program for the Research in Legal Education: State of the Art? conference from 3-5 December is now online http://www.legaledconference.law.unsw.edu.au/
The conference kicks off on Sunday 3 Dec at 6pm with welcome drinks and a book launch. On Monday 4 and Tuesday 5 Dec we have over 50 speakers, and three international plenary speakers.
Best of all, the conference is free.
So make sure you register at https://www.eventbrite.com.au/e/legal-education-conference-2017-tickets-36994327028
There is ongoing interest in what is known as contract cheating – where students pay others to write assessment for them.
The Australian higher education regulatory agency TEQSA has just released a Good Practice Note on the subject. A team of researchers is currently in the middle of a large scale project to develop strategies to design assessment to counter the phenomenon: ‘Contract Cheating and Assessment Design: Exploring the Connection’
Contract cheating is conceptually different to traditional notions of plagiarism in that it is a predominantly commercial, market driven enterprise. Rather than a student surreptitiously borrowing another student’s work, contract cheating involves arms length payments to strangers who provide assessment answers for profit. Research has uncovered that such organisations can be multinational in scope, sourcing paper writers from Africa, Asia and Europe. In such circumstances it becomes relevant to ask whether such behaviour should be seen as criminal, and what offence may have been committed. While students are easy targets here, finding ways to prosecute the ‘essay mills’ is more complex.
Two articles address this issue. My recent article ( Contract cheating: Will students pay for serious criminal consequences?) looks at criminal liability in Australia, with a focus on NSW, suggesting that offences such as fraud, forgery and conspiracy to defraud may be committed. It also considers New Zealand’s specific anti-cheating legislation. The existence of an arguable basis for criminal prosecution of such entities also permits authorities to apply to freeze bank accounts under proceeds of crime legislation. This may be seen as a controversial approach, but it has already been adopted in New Zealand: Commissioner of Police v Li  NZHC 479
In a UK context, an article earlier this year by Michael Draper, Victoria Ibezim and Philip Newton ( Are Essay Mills committing fraud? ) explored whether essay mills can be prosecuted under the English Fraud Act 2006. They suggest it would be difficult and argue for the creation of a strict liability offence. Although their article does not discuss it, conspiracy to defraud and proceeds of crime offences are likely to be also available.
The conundrum that underlies the analysis in both articles is the fact that the criminality is at base that of a group crime – joint criminal enterprise or conspiracy – but on policy grounds it seems best to have the students dealt with in an educative and restorative manner outside of the criminal courts while throwing the book at the essay mills. Specific legislation banning the operations of the essay mills is the simple fix, but raises further questions of scope and standards of proof. It will be interesting to see if governments in Australia and the UK decide to take that path.
by Lyria Bennett Moses
On Monday 18 September, UNSW Law ran a workshop on What Law Students Need to Know about Technology, in which a range of presenters combined with UNSW Law faculty to ask themselves what law graduates will need to know about technology. The workshop aimed to a flavour of some of the technologies that may be important to current and future lawyers and pose the question – what will legal graduates need to know (and thus should be taught in the core) and what should legal graduates have the opportunity to learn (in electives or via extra-curricular activities). The goal of the workshop was not to teach the faculty everything about the technologies, but rather to give members of faculty enough information to be able to make a decision about what to include in their core course and what they may want to learn more about.
Kathy Laster, Director of the Sir Zelman Cowen Centre at Victoria University, began by asking how law graduates should be trained and should market themselves in a world where lawyerly caution is often seen as a barrier to innovation and entrepreneurship. Michael Legg’s summary of the flip committee recommendations reverberated throughout the day, particularly the importance of the skills of project management, working across disciplines and the ability to comprehend the outputs of technology and subject them to challenge and critique. Ross Buckley and David Rountree (Allens) explained core elements of blockchain and its potential impact on transactions as well as smart contracts (code which seeks to facilitate, enforce or automate the performance of a contractual obligation or term and will automatically execute in accordance with its coded terms) and how these relate to broader trends (FinTech, RegTech, monetization of data etc). After morning tea, Peter Leonard (Data Synergies) and Lyria Bennett Moses discussed the uses and limitations of data-driven reasoning.
In the afternoon, Judge Riethmuller of the Federal Circuit Court brought us back to more basic technologies (such as being able to use document templates correctly, understanding strengths and limits of PDF documents, how to capture email and social media as evidence) and gave the excellent suggestion that students should take notes using databases such as Zotero rather than word documents so that it could become a platform for growing their legal knowledge over time. He also discussed the limits of automated means of enhancing access to justice and the challenges the Internet poses to the increase of slightly mad legal claims. Caryn Sandler and Matthew Golab gave an insight into how technology is changing legal practice at Gilbert + Tobin through e-discovery, automated document review, automated searching, collaboration tools and so forth, and introduced a new kind of employee “technolegals” – junior lawyers working on process innovation.
We ended with a panel and a wide ranging discussion with insights from private practice, our own National Children and Youth Law Centre, UTS (and its new Legal Tech major), and Google’s legal team. Broad messages were:
- Don’t get caught in the hype – the curriculum doesn’t need to include every new tech fad, but it does need to teach students to think critically about how they use technology and how technology affects their clients.
- On a related point, students don’t need to know how to code but they need to be able to communicate with someone who writes code and tease out the legal and policy issues related to its use. Students should be exposed to facts (and have to grasp those) as well as law.
- We can help students do this by using modern contexts as example facts in class and in assessment. For example – what remedy would you need to reverse a transaction that had already been processed on the Blockchain?
- We don’t need to reinvent the wheel in articulating problems such as “technological fix” – lots of helpful insights from other disciplines such as Science and Technology Studies.
- The importance of teaching future lawyers to write clear, brief, simple advice that identifies next steps and best solutions (in addition to essays and other longer pieces) – recognising these skills in brevity are useful beyond the legal profession itself.
- Help students develop a well-rounded CV, including through experience at CLCs, Hackathons and other student competitions that encourage development of problem-solving skills.
- Teach students how legal risk fits in with a bigger picture (including the broader risk picture)
by Lucas Lixinski
A recent feature on the Times Higher Education supplement questions the power and effectiveness of learning analytics as a tool in higher education. These tools, which go all out in “flipping”, “blending”, and otherwise “personalizing the student experience”, are all the hype in universities these days. I wonder, though, what they can actually do for legal education.
Don’t get me wrong: I want to like them. While I am not exactly the most tech-savvy, early-adopter type, I think there is some potential in the promises of learning analytics. I would love to be able to know how students interact with the reading materials posted online beyond anecdotal evidence (how much time they spend on the readings; what readings they consider to be priority; even whether they print or read on-line). But I am also mindful that, as far as I can see, legal education is about creating a certain analytical habit of critical thinking. And that habit is best developed, in my opinion, in the back-and-forth in the classroom.
So, maybe what learning analytics can do is to help free up time in the classroom for more of the back-and-forth, and help me as a teacher know what to focus on in that ever-shrinking time.
This strategy, of course, assumes that I take the time to stay on top of the data constantly, while simultaneously not deferring to it to much. There is a risk, as the THE story linked above indicates in all-too-real terms, that learning analytics will displace responsibility away from teachers, that “the machine” will keep track of student progress for me, and I will become numb to signs in the classroom.
Also, and relatedly, we need to be very mindful of who analytics are used for (or, in jargon I am still uncomfortable with in a university context, who the “customer” of data analytics is). If analytics are all about the student experience, does that mean that the data is geared towards catering to student preferences? If that is the case, aren’t we missing the point of education as challenging, rather than settling into, path dependency?
As comfy as it would be for me to have data tell me what courses to explore based on what materials I focus on the most in my readings if I were a student, as an educator I need to know that I need to challenge, rather than cater to, what students think they know about reality. That is how critical thinking develops (or that is my opinion, about which I wrote elsewhere on this blog).
So, learning analytics: yay or nay? I remain hopeful, but I think it all boils down to how the university treats the data, and who it sees as being primarily in control of it: the students whose fees the university needs, or the professionals who fulfill the ethical mandate of education.