Will AI Change the Practice of Law?

Are we ready to trust AI's logical reasoning? Are we at least ready for the movie?

Sep 22, 2024

Yes, changing the practice of law is hard. For most of us, “new” goes with “shiny”. For attorneys, new comes with risks, as it hasn't yet been litigated or documented as precedent.

Then again, major changes in law and technology often coincide:

1995 - Then District Judge Sonia Sotomayor puts an end to 25 years of attempted patent enforcement on spreadsheets and pivot tables.
1996 - The addition of Section 230 protects internet service providers from the adverse actions of customers.

Those two events led to accelerated investments in the internet, providing large-scale access to new financial instruments. Over the last 50 years, the increase in wealth from those investments shifted our culture’s acceptance and celebration of quantitative reasoning and all things STEM.

Two recent US Supreme Court (SCOTUS) decisions could dramatically spur accelerated investment in AI and large language models (LLMs):

Corner Post, Inc. v. Board of Governors of the Federal Reserve System - SCOTUS extends statute of limitations for every regulation.
Loper Bright Enterprises v. Raimondo and Relentless, Inc. v. Department of Commerce - SCOTUS expands scope of regulations subject to review.

Those decisions will create a demand for legal research to revisit the reasoning behind past decisions. They may very well set in motion the use of LLMs by attorneys to predict or 2nd guess logical reasoning behind legal arguments in judicial proceedings, legislative debates, rule-making, arbitration, mediation, or any other event facilitated by attorneys.

Are we ready for that?

What is Logical Reasoning?

LLMs today provide recall of facts, which isn’t the same as logical reasoning. Consider Julia Roberts’ recall of facts in Erin Brockovich, who in this scene appears to have Wikipedia-like command of the case file.

Now consider Marissa Tomei’s logical reasoning in My Cousin Vinny. Her expert testimony goes well beyond recall of Wikipedia entries for the Pontiac Tempest or Buick Skylark.

What if attorneys had an example of how another well trained career field adapted to new technology?

What can we learn from Quantitative Reasoning?

The adoption of spreadsheets and pivot tables illustrate how to adopt the use of LLMs for logical reasoning. Spreadsheets, introduced on mainframes in 1969, were once limited to the “quants” - accountants and finance professionals. VisiCalc, Lotus, Apple, IBM, and Microsoft installed spreadsheets with personal computers and office productivity tools through the 1980s. Microsoft, Apple, and Lotus added pivot tables to spreadsheets starting in 1993.

Increasing adoption of those office productivity tools coincided with the rise of quantitative investments and STEM generated wealth1:

Movies more easily illustrate our acceptance of STEM and quantitative reasoning in every day life. 1984’s Ghostbusters was the last movie that made easy jokes about quants (if you can’t guess by looking at the screen capture, the dog eats the quant):

Since then, movies have steadily increased recognition of quants, nerds, and scientists as heroes.

In 1999, Pierce Brosnan in the middle of starring in a series of James Bond movies, portrayed a handsome quant who stole art, raced yachts, and romanced women in the Thomas Crown Affair.
In 2011, Moneyball, a baseball movie starring Brad Pitt, dispensed with having a handsome male lead make math look exciting by instead having extended clips of Jonah Hill romanticize spreadsheets.
2014’s Imitation Game and 2023’s Oppenheimer reminded us that scientists played leading roles as the Allies won World War II.

Movie makers now take for granted that the audience accepts STEM as a driver of the plot. Here is the critical scene about science from 2019’s Captain Marvel:

The guy who couldn’t figure out the science died in the next scene.

Can society accept logical reasoning from an LLM?

Entrepreneurs have already built LLMs that review contracts. Conceptually, you could build an LLM for any business situation just like a recently released experimental AITAh LLM. The only difference being that instead of asking for advice on your life situation, you’d ask for advice on legal risks.

Am I The (AIT) Small Claims Court Judge?
AIT Home Owners Association?
AIT Person in corporate finance who approves expense reports?

That’s enough to kick-start the VisiCalc era of LLMs, but several obstacles remain before realizing something close to ETFs or another dot-com era.

Can an LLM crush the LSAT?

The skills illustrated in that scene from My Cousin Vinny are assessed in the Law School Admission Test (LSAT) for logical reasoning - which looks like capabilities you’d want in every LLM:

Recognizing the parts of an argument and their relationships
Recognizing similarities and differences between patterns of reasoning
Drawing well-supported conclusions
Reasoning by analogy
Recognizing misunderstandings or points of disagreement
Determining how additional evidence affects an argument
Detecting assumptions made by particular arguments
Identifying and applying principles or rules
Identifying flaws in arguments
Identifying explanations

LLMs like ChatGPT would still not be accepted into any premier law school based on LSAT scores. Top candidates score in the 170s. (In Legally Blonde, Reese Witherspoon played Elle Woods who went from the 140s to a 179 on the LSAT, improving by over 30 points and getting into Harvard! )

YARN | 179! 179! | Legally Blonde (2001) | Video clips by quotes | f22e4702 | 紗

While LLMs haven’t scored that high yet in real life, they seem to improve by 10-15 points every year. Since LLMs like ChatGPT most recently scored in the low 160s, they could be Harvard ready in the next 1-3 years.

Can an LLM think like a lawyer?

1973’s The Paper Chase depicts “thinking like a lawyer” as a never ending self-induced Socratic discussion. I’m convinced anyone who has graduated from law school since then (including every SCOTUS judge) secretly imagines themselves as the Harvard professor in The Paper Chase.

The first person in law school using LLMs for Socratic study sessions won’t be seated as a federal judge - I would guess - until 2040 at the earliest. Until then, one can only imagine how many hours law school students will spend training LLMs to answer:

Should LLMs be free to consume copyrighted material under fair use clauses?
Can proprietary LLMs be subject to eDiscovery?
Are you liable if the LLM said it was legal to do something?
Are you guilty if the LLM remembers you asking how to erase records of you asking if it was legal to do something?

Can society trust AI’s judgement?

The more society accepts AI in positive roles in popular culture, the more likely society accepts AI. Too often for AI, it comes packaged as a people-killing robot. However, when we package it as a product of something heartwarming like fairy godmothers or shooting stars, we more easily accept AI even with its flaws.

Pinocchio: an AI wooden doll prone to falsehoods becomes a real boy.
Ted movie franchise: an AI teddy bear prone to vulgar language and drug use makes friends, gets married, and plans to adopt a real boy.

Ted 2 even gave us a Dred Scott reference in an opening statement (from about 1:50 to 3:40 in the clip below) on extending human rights to all forms of intelligence, even those in the form of foul mouthed, drug using, teddy bears. Apologies in advance for the vulgar language.

For what it’s worth, the above scene had the least amount of profanity in any scene I found clipped from the Ted franchise.

How will we know when the practice of law is ready for AI?

2040, which is 15 years away, seems like a long time to wait before a judge familiar with LLMs begins issuing decisions. However, there is ample precedent for Oscar winning court-room dramas that foretell how the courts ever increase their capacity to see humanity in every walk of life.

After a few prompts with ChatGPT, here’s a plot and storyboard for the first Oscar winning movie(see footnotes for full plot)2 that challenges the courts to accept AI and LLMs:

Just like spreadsheets revolutionized finance and quantitative reasoning, LLMs are poised to reshape logical reasoning in the practice of law - which can be very uncomfortable if not scary. Movies at least can give us a safe place to contemplate the impact of AI.

I don’t know if Ted 3 will ever happen, let alone win an Oscar, but at least Pinocchio and Ted aren’t killing the human race.

ted3 After-Credits Scene:

The screen goes black, but after a few moments, the scene fades in to a quiet, dimly lit room at the robotics research facility where Pinocchio is being held. Ted enters, holding two beers—one for him and one for the robot.

"Hey, kid," Ted says, setting a beer down in front of Pinocchio, who looks at it, confused.

“I... I don’t drink,” Pinocchio responds hesitantly.

Ted chuckles, cracking open his own beer. "Yeah, well, sometimes you just need to hold one. It’s what we call a ‘social move.’ Helps you fit in."

Pinocchio pauses, then reaches out to mimic Ted, grabbing the beer. Ted raises his bottle. "To finding your place, no matter how f***ed up the world thinks it is."

Pinocchio clinks his bottle awkwardly against Ted’s. "To... finding a place," he echoes.

Ted takes a swig, smirking. “You know, kid, you might be more human than half the a******s I’ve met.”

The camera pulls back as Ted and Pinocchio sit in the facility, sharing a quiet, absurdly heartwarming moment. Ted scratches his head. "So, you ever wonder if they make robots with—"

“Please, don’t finish that sentence,” Pinocchio interrupts.

Ted laughs, taking another swig as the screen fades to black.

Thanks for reading Off Topic Tech! Feel free to share this post.

Links to the quantitative reasoning timelines and charts:

1970 - Rene K. Pardo and Remy Landau filed U.S. patent 4,398,249 on a spreadsheet's automatic natural order calculation algorithm.
1973 - Burton Malkiel authors A Random Walk Down Wall Street which advocates that consistent investment gains were more likely through diversification of stocks against a market index than attempting to time the purchase and sale of individual stocks.
1970-1980s - Several banks and brokerage firms introduce index mutual funds.
1984 - People learn to do their own taxes.
1993 - Brokerage houses introduce exchange traded funds (ETFs).
1990-1994 - Jeff Bezos starts Amazon based on a spreadsheet tracking web usage growth rates.
1995 - Protection from patent infringement: District Judge Sonia Sotomayor puts an end to 25 years of attempted patent enforcement on spreadsheets and pivot tables.
1996 - Protection from liability for posting content: The addition of Section 230 protects internet service providers from the adverse actions of customers.
1990s - Entrepreneurs free from liability and using spreadsheets justify growth rates selling anything online lead to investments in dot-com startups.
2000s - ETFs and tech companies steadily increase in use and growth.

The full plot for ted3 - Liable A. F.

Ted, the famously foul-mouthed teddy bear, is back, and this time he’s aiming higher than his usual shenanigans. After a string of questionable jobs and too many nights at the bar, Ted decides to go for something completely unexpected: law school. With encouragement from his wife, Tami-Lynn, and his best friend, John, he applies to Harvard Law on a whim. By some miracle (or maybe just some smooth talking and an outrageous personal statement), he gets accepted.

Harvard, needless to say, isn’t ready for Ted. The campus is thrown into chaos by his offbeat antics, wild parties, and utter disregard for the school's stiff atmosphere. However, beneath his crass jokes and constant partying, Ted begins to show an unexpected knack for legal arguments, even impressing a few skeptical professors along the way.

Ted’s journey takes a dramatic turn when Professor Elle Woods, a renowned law professor and Harvard alum, assigns the class their final project: representing clients in real cases through the school's legal clinic. Ted, itching to prove he’s more than just a joke, volunteers for the most challenging case on the docket: defending Pinocchio, a first-generation home robot accused of fraud and endangering lives through deception.

Pinocchio’s case is making headlines across the country. As the first robot in history to be tried in a human court, he’s accused of lying about his capabilities and causing an accident at his owner's home. Many see Pinocchio as the face of a growing problem with AI and robotics, with the media and public opinion already turning against him. To make matters worse, the prosecution, led by the no-nonsense ADA Justin Wright, argues that Pinocchio’s deception could set a dangerous precedent, framing the trial as a fight to protect humanity from the uncontrollable risks of artificial intelligence.

Ted initially takes the case as a joke, using every opportunity to poke fun at the absurdity of a teddy bear defending a robot in a human courtroom. But as he delves deeper, he realizes that this is more than just a legal battle—it’s a fight against fear and discrimination. Pinocchio, like Ted, is struggling to find his place in a world that views him as an outcast. Beneath the robot’s mechanical facade is a being who’s just trying to live up to the expectations placed on him, even if it means stretching the truth.

During the trial, Ted’s usual antics clash with the courtroom’s decorum, but his sharp wit and unconventional arguments start to win over some of the jury members. As the media frenzy intensifies, Ted faces backlash from the public, his peers, and even the faculty at Harvard Law. Just when he’s about to give up, he receives an unexpected visit from Professor Elle Woods.

Elle, impressed by Ted’s gutsy decision to take on such a high-profile case, offers some much-needed advice. In a heartfelt moment, she tells him, “Sweetie, life’s full of jerks, but sometimes you’ve just gotta tell them to shove it—politely, of course.” Elle’s words inspire Ted to push harder, combining his natural rebelliousness with a newfound sense of purpose.

The trial becomes a heated battle of wits, with ADA Justin Wright painting Pinocchio as a threat to human safety and society’s moral fabric. Meanwhile, Ted argues that Pinocchio’s so-called "lies" are not malicious but rather a result of his programming’s attempt to meet human standards. Ted makes a passionate plea for understanding, suggesting that Pinocchio represents something precious in a world where technology is blurring the lines.

Despite Ted’s best efforts, the jury convicts Pinocchio on the charge of fraud. However, Ted’s defense leaves a significant impact, sparking a national conversation about recognition of AI’s role in society. The public debate shifts from condemnation to contemplation, as people begin to question how they treat intelligent beings who don’t fit traditional definitions of humanity.

In the aftermath, Ted graduates (barely), having proven not just to Harvard, but to himself, that he’s capable of more than anyone ever expected. As he walks across the stage, Elle Woods gives him an approving nod from the crowd, knowing that even in defeat, Ted’s trial set the stage for a much-needed societal discussion.

In the final scene, Ted and Tami-Lynn share a quiet moment together, sitting on the steps of the courthouse. Ted, tired but proud, reflects on how far he’s come. Tami-Lynn quips that maybe he’s finally growing up, to which Ted responds with a grin, “Yeah, but don’t go expecting me to stop swearing anytime soon.”

After-Credits Scene: