If there’s one thing to be said about AI, it’s that it is highly agreeable — pleasant algorithms willing to validate all sorts of stupidity in the name of engagement optimisation. So, it’s no wonder that it will make up a thing or two to vindicate your arguments. And when you’re using AI as a therapy chatbot then, sure, these sycophantic arguments are no problem. But increasingly, AI hallucinations are showing up in serious places, from doctor’s scripts to case law.
The law is meant to be a realm of truth, but it is steadily being peppered with incoherent arguments and hallucinated precedents. In South Africa, acting high court judge Themba Khaba was recently taken to task after 11 “discrepancies” were found in a ruling he wrote that was potentially AI generated.
Judge Ingrid Opperman flagged these issues, noting that Khaba had in some instances used the correct name of a case but the wrong citation. In other instances, the correct case number and reference were given, but the passage quoted was incorrect. Most notable was his quotation of a nonexistent case.
It’s far from the first instance of garbled AI legal papers in South Africa, notes Dario Milo, a media law specialist at Webber Wentzel. He cites a case where “counsel’s heads of argument relied on an AI research tool that produced fabricated case citations. The court identified these as AI hallucinations and referred the legal team’s conduct to the Legal Practice Council [LPC] for investigation, despite there being no intention to mislead the court.”
Then there’s the case of the Molawa judgment, in which Labour Court judge Kelsey Allen-Yaman had to stay disciplinary proceedings after finding that a ruling relied entirely on AI-generated, fictitious case law. Of the four sources cited by the chair of the inquiry, two were entirely made up, and a third was materially misrepresented.
For judges, the matter is particularly serious. As law firm Cliffe Dekker Hofmeyr notes: “When judges pronounce on matters, their words have binding effect beyond the immediate parties.”
It goes to the integrity of the entire judicial process. “Judges sit at the pinnacle of the legal hierarchy, and mistakes made at this level can actually alter the course of our jurisprudence … If rigorous verification of AI-generated citations (or other AI content for that matter) is important for other legal role-players, it stands to reason that for the judiciary it is an indispensable safeguard for preserving the integrity of the judicial process.”
Proceed with caution
So how does one even begin to regulate or police AI use in the law?
It is impossible to outright forbid its use, experts agree, and this would actually do more harm than good. “Aside from being extremely difficult to enforce in practice, an outright prohibition would not make sense,” says Milo. “The legal profession is evolving, and it is important to adapt and keep pace with technological developments, provided this occurs within appropriate legal and ethical boundaries.”
Anthony Norton, MD of law firm Nortons Inc, agrees. “It’s difficult to be too prescriptive about it,” he says. “These AI tools are evolving so quickly that what is considered appropriate today might be very different in a year’s time.”
On the whole, AI is seen as mostly beneficial to legal practice. Norton views it as “brilliant” for searching extensive documents, picking out extracts, and as a screening tool to see what is potentially useful. Even the office of the chief justice “welcomes” the technological advancements in the legal sphere and notes it provides a host of opportunities to “improve efficiencies in the administration of justice”.
Says Milo: “I have no doubt that AI is a positive advancement for the legal profession — provided it is used responsibly.”
Indeed, the catch is that AI needs to be used correctly. It may be helpful in research, but things quickly fall apart when lawyers use it to draft heads of argument or write up pleadings and affidavits. AI “has a creative licence of its own”, says Norton, and its output cannot be blindly trusted. “There is no substitute for actually going into the case law itself and getting the actual judgments, doing the proper research and reading the academic articles.”
As Milo puts it: “AI systems can produce responses that are coherent, confident — and entirely incorrect.”
The risk, says Pearl Arnold Mfusi, acting executive officer of the LPC, is the “uncritical reliance on AI-generated outputs without proper verification and oversight by a legal practitioner”.
This is the crux of the problem: AI itself is not a danger to the integrity of the legal profession; it’s the practitioners who put too much trust in it.
Sometimes, this is less a result of incapability and more of pressure to deliver. “I think if you are resource-constrained and cost-constrained, there’s a huge temptation to use it to generate full sets of pleadings and documents,” says Norton. “It’s a shortcut way of doing a lot of this stuff.”
Ultimately, though, it is a lawyer’s responsibility to ensure that their shortcut contains valid information.
Arnold Mfusi gives a non-AI-related example: “Practitioners may use textbooks to identify authorities [the sources of law used to support arguments], but it has always been considered poor practice to rely on a textbook’s references without consulting the underlying judgment. Similarly, AI-generated references should be treated as a starting point for further research rather than as authoritative sources in their own right.”
Rules of engagement
The quagmire is that AI use is almost impossible to directly regulate, and yet some sort of regulation is demanded if the integrity of legal practice is to be safeguarded.
On a smaller level, Norton muses that regulation could come from within firms or from entities like the bar; his own firm has “very strict internal guidelines” around the use of AI. On a macro level, he argues that “it would be helpful if there were more extensive guidelines generated at a professional level about this, whether by the LPC or by the judiciary”.
Arnold Mfusi notes that while the LPC “does not currently have a specific code of conduct dealing exclusively with AI”, she believes its existing ethical and professional obligations are “sufficient to regulate the use of AI in legal practice”.
“If a practitioner were to rely on fictitious authorities, fail to verify legal sources, or mislead a court through negligence or recklessness, such conduct could potentially constitute professional misconduct,” she explains. “Existing disciplinary processes would then apply, just as they would where inaccurate information was sourced from any other research method.”
It’s a fair point, though the issue here is that AI will still undoubtedly be used — legal practitioners will just suffer for it after the fact. The LPC’s way around this is by hosting “regular awareness sessions” on the proper use of AI.
The judiciary, on the other hand, is in the process of developing an AI policy. “This policy will not only provide guidance, it will also outline the processes to be enacted when the prescripts that have outlined safeguards are not followed,” says the office of the chief justice.
Both Milo and Norton welcome the idea of a broad-spectrum AI policy from an entity such as the judiciary but agree that some sort of more formal, mandatory training needs to take place for lawyers to learn about the limitations of AI, the risks of hallucinations, confidentiality and privilege concerns, bias and discrimination, as well as cybersecurity considerations.
In fact, Norton thinks AI is not being used to its full potential in the legal sphere, and training could help lawyers make the best use of it. But it remains to be seen whether this will make a real difference or if AI hallucinations will remain endemic for as long as lawyers lean on it as a crutch.
