While
law
firms
struggle
to
keep
a
lid
on
AI
security
threats
and
otherwise
control
lawyers
using
AI
out
of
turn,
Linklaters
has
a
forward-thinking
solution
that
more
firms
should
embrace.
As
RollOnFriday
notes,
Linklaters
puts
its
prospective
robot
lawyers
through
a
UK
law
exam
of
their
own
design,
crafting
50
questions
from
10
different
practice
areas
geared
toward
a
“competent
mid-level
lawyer
specialised
in
that
practice
area”
and
asked
the
AI
to
hash
it
out.
It’s
the
Baby
Bar
for
robots.
So,
like,
a
Nano
Bar?
Whatever
it
is,
it’s
working
better
than
the
California
Bar.
It’s
not
like
Linklaters
is
alone
on
this.
Tech
professionals
across
Biglaw
perform
tests
like
these
all
the
time.
The
distinction
is
that
Linklaters
is
doing
this
out
in
the
open.
And
senior
lawyers
graded
the
responses
for
substance,
introducing
important
stakeholders
to
the
AI
evaluation
process
that
can
get
overlooked
when
tools
are
considered
exclusively
by
IT
teams
and
firm
tech
committees.
Which
isn’t
a
knock
on
the
conscientious
approach
firm
staff
and
tech-savvy
attorneys
bring
to
these
decisions
but…
sometimes
you
need
to
take
these
issues
outside
the
nerd
circle.
The
good
news
for
firms
is
that
AI
is
getting
better:
Linklaters
noted
there
was
a
“significant
improvement”
in
the
results
when
compared
with
tests
it
ran
in
October
2023.
The
firm
said
that
the
AI
models
are
starting
to
perform
at
a
level
where
they
should
be
able
to
assist
in
legal
research,
such
as
providing
a
first
draft
or
“as
a
cross-check
on
an
existing
answer,”
and
could
also
be
useful
“for
tasks
that
involve
summarising
relatively
well-known
areas
of
law”.
OpenAI
delivered
the
best-performing
AI
model
on
the
Linklaters
test.
That
said,
the
tool
only
scored
6.4
out
of
10
and
bet-the-company
lawyering
doesn’t
grade
on
a
curve.
The
report
doesn’t
provide
Grok’s
score,
but
based
on
its
stated
approach
to
legal
reasoning,
we’re
guessing
it
made
OpenAI
look
like
William
Blackstone
crawled
out
of
the
grave
for
a
spot
of
tea
and
a
run
at
some
contracts.
But
putting
AI
through
the
wringer
in
such
a
public
way
signals
to
lawyers
that
the
firm
both
sees
potential
in
AI
while
still
making
the
clear
case
to
everyone
that
AI
isn’t
ready
for
prime
time.
This
approach
should
push
back
against
the
“shadow
AI
economy”
problem
Hill
Dickinson
recently
addressed.
When
firms
downplay
or
outright
shun
AI,
unruly
lawyers
are
going
to
start
experimenting
with
AI
tools
on
their
own.
And
that’s
how
you
end
up
with
confidential
client
data
being
uploaded
to
random
servers
in
hostile
countries.
Transparency
keeps
everyone
on
the
same
page
when
it
comes
to
when
and
where
AI
fits
into
a
modern
legal
practice.
Linklaters
makes
AI
sit
law
exams
[RollOnFriday]
Joe
Patrice is
a
senior
editor
at
Above
the
Law
and
co-host
of
Thinking
Like
A
Lawyer.
Feel
free
to email
any
tips,
questions,
or
comments.
Follow
him
on Twitter or
Bluesky
if
you’re
interested
in
law,
politics,
and
a
healthy
dose
of
college
sports
news.
Joe
also
serves
as
a
Managing
Director
at
RPN
Executive
Search.