The Three Principles of Responsible AI Development, and Other Takeaways from the Everlaw Summit

At
the

Everlaw
Summit
in
San
Francisco
last
week,
the
annual
customer
conference
of
the
e-discovery
company

Everlaw,
founder
and
CEO

AJ
Shankar
delivered
a
keynote
address
in
which
he
announced
the
general
availability
of
three
generative
AI
features
the
company
first
introduced
last
year
and
had
been
developing
in
beta
ever
since.

In
the
course
of
delivering
that
address
(see
featured
image
above),
Shankar,
a
computer
scientist
by
training,
detailed
the
core
principles
that
guide
the
company’s
AI
development
–
principles
that
he
said
are
“table
stakes”
to
ensuring
responsible
AI
development
and
the
best
long-term
outcomes
for
customers.

The
three
features
announced,
all
under
the
umbrella
name
Everlaw
AI
Assistant,
are
now
live
on
the
Everlaw
platform,
although
customers
must
purchase
credits
beyond
their
standard
subscriptions
to
use
them.
They
are:

• Review
  Assistant,
  for
  reviewing,
  summarizing
  and
  prioritizing
  documents.
• Coding
  Suggestions,
  for
  coding
  and
  categorizing
  documents
  based
  on
  criteria
  provided
  by
  the
  user.
• Writing
  Assistant,
  for
  analyzing
  and
  brainstorming
  against
  documents,
  evidence
  and
  depositions.

Three
Core
Principles

At
a
time
when
many
legal
professionals
still
question
the
safety
and
accuracy
of
generative
AI,
it
was
notable
that
Shankar
devoted
a
substantial
portion
of
his
keynote
to
talking
not
about
the
products,
per
se,
but
about
the
three
core
principles
that
guided
their
development
and
Everlaw’s
development
of
other
AI
products
still
to
come.
Those
principles
are:

• Privacy
  and
  security.
• Control.
• Confidence.

With
regard
to
privacy
and
security,
Shankar
said
that
Everlaw
ensures
that
providers
of
the
large
language
models
it
uses
adhere
to
strict
data
retention
policies.
Everlaw
prevents
LLM
providers
from
storing
any
user
data
beyond
the
immediate
query
and
from
using
that
data
for
model
training.

“We
ensure
that
they
apply
zero
data
retention
to
your
data,
which
means
that
when
you
send
data
to
them,
they’re
not
allowed
to
store
it
for
any
reason
past
when
they’ve
answered
your
query,
as
well
as
no
training,
so
they
can’t
use
the
data
to
train
their
models
in
any
way.”

With
regard
to
control,
Shankar
said
Everlaw
is
committed
to
enabling
users
to
maintain
control
over
their
data
and
tool
usage
through
features
that
allow
them
to
manage
visibility,
access,
and
project-specific
settings.
Everlaw’s
approach
to
transparency
includes
notifying
users
when
they
are
using
AI-powered
features
and
making
it
clear
which
models
are
in
use.

Administrative-level
control
allows
admins
to
control
access
to
AI
features
as
well
as
consumption
of
AI
credits
at
various
organizational
and
project
levels.

“Your
users
should
always
know
when
they’re
using
gen
AI,”
Shankar
said.
“We’ll
tell
you
what
models
we
use.
We
want
you
to
have
that
kind
of
transparency
and
control
in
your
interactions
here,
so
you
can
best
devise
how
to
use
a
tool.”

The
third
principle
–
that
of
enabling
customers
to
have
confidence
in
using
these
tools
–
is
the
hardest,
Shankar
said.
“We
know
gen
AI
can
provide
immense
value,
but
it
can
also
make
mistakes,
right.
We
all
know
about
the
potential
for
so
called
hallucinations.”

Shankar
outlined
two
ways
Everlaw’s
development
of
AI
seeks
to
establish
confidence
in
the
AI’s
results.

• 
  Play
  to
  AI’s
  strengths.
  “The
  first
  thing
  we
  do
  is
  that
  we
  design
  experiences
  that
  play
  to
  the
  strengths
  of
  large
  language
  models
  and,
  to
  the
  extent
  possible,
  avoid
  their
  weaknesses.”
  That
  means
  focusing
  on
  use
  cases
  where
  LLMs
  have
  reliable
  innate
  capabilities,
  such
  as
  natural
  language
  fluency,
  creativity,
  and
  even
  some
  reasoning.
  Even
  then,
  he
  said,
  “we’re
  really
  wary.”
  For
  that
  reason,
  Everlaw
  avoids
  uses
  that
  require
  embedded
  knowledge
  of
  the
  law
  and
  instead
  delivers
  results
  that
  rely
  on
  the
  four
  corners
  of
  the
  document
  set
  on
  which
  the
  customer
  is
  working
  –
  documents
  provided
  to
  the
  model
  when
  it
  is
  queried,
  not
  when
  it
  is
  being
  trained.
  “That
  makes
  a
  far
  more
  reliable
  experience.”
• 
  Embed
  into
  existing
  workflows.
  By
  embedding
  the
  AI
  into
  customers’
  existing
  workflows,
  rather
  than
  in
  a
  conversational
  chat
  interface
  that
  gives
  open-ended
  answers,
  the
  AI
  is
  able
  to
  deliver
  answers
  with
  greater
  precision.
  “We
  don’t
  want
  users
  having
  to
  learn
  how
  to
  prompt
  engineer
  to
  get
  what
  they
  want.
  They
  basically
  will,
  in
  many
  cases,
  just
  click
  a
  button
  and
  we’ve
  done
  the
  work
  for
  that
  precise
  use
  case
  to
  ensure
  it’s
  going
  to
  be
  reliable.”
  This
  embedding
  into
  workflows
  also
  means
  that
  the
  necessary
  context
  is
  provided
  to
  more
  precisely
  answer
  the
  question.
  “So,
  together,
  being
  able
  to
  have
  precise
  use
  cases
  and
  having
  all
  the
  context
  you
  need
  allows
  for
  protective
  guardrails
  and
  higher
  quality
  outputs.”

But
he
said
there
is
a
third
aspect
of
building
confidence
in
the
AI,
and
it
is
something
customers
have
to
do
for
themselves,
which
is
to
change
their
mental
model.

“What
you
basically
have
to
do
is
think
about
using
a
computer
a
little
bit
differently
from
how
we’ve
all
been
trained
to
do
for
many
years.
You
have
to
move
from
an
interaction
model
where
you
have
very
repeatable
interactions
that
are
also
largely
inflexible,
like
a
calculator,
to
a
variable-interactions
model,
where
things
might
be
a
little
different,
but
it’s
highly
flexible.
It’s
much
more
like
a
human.”

‘A
Smart
Intern’

In
fact,
he
urged
the
audience
to
think
of
gen
AI
as
a
“smart
intern”
–
very
capable
and
very
hard
working,
but
still
able
to
make
mistakes.
Over
time,
you
need
to
learn
what
the
intern
is
capable
of
and
determine
your
personal
comfort
level
with
its
capabilities,
but
in
the
meanwhile,
you
need
to
continue
to
check
its
work.

“In
this
new
world,
it’s
neither
good
to
just
blindly
trust
the
output
of
a
gen
AI
tool,
nor
is
it
good
to
just
say,
hey,
one
mistake
and
it’s
out.
It’s
like
a
person,
and
that’s
a
fundamental
shift
in
how
we
want
you
to
think
about
these
tools.”

Just
as
you
would
with
an
intern,
in
order
to
build
confidence
in
the
AI,
you
need
to
check
its
work,
to
learn
what
it
is
good
at
and
what
it
is
not.
For
that
reason,
he
said,
Everlaw
builds
its
AI
products
with
features
that
make
it
easy
for
users
to
check
the
outputs.

“Our
answers
will
cite
specific
passages
in
a
document
or
specific
documents
when
you’re
looking
at
many
documents
at
once,
and
so
you
can
check
that
work.”

A
specific
example
of
this
ability
to
check
the
AI’s
work
can
be
found
in
the
new
Coding
Suggestions
feature,
which
will
evaluate
and
code
each
document
in
a
set
based
on
instructions
you
provide,
much
like
human
reviewers
would
do.

Unlike
predictive
coding,
it
will
actually
provide
an
explanation
for
why
it
coded
a
document
a
certain
way,
and
cite
back
to
specific
snippets
of
text
within
the
source
document
that
support
its
coding
decisions.
This
allows
the
user
to
quickly
verify
the
results
and
understand
why
the
document
was
coded
as
it
was.

“It
has
a
richer
semantic
understanding
of
the
context
of
each
document,
which
allows
for
a
unique
insight
like
a
human,
potentially
beyond
what
predictive
coding
could
provide
by
itself,”
Shankar
said.

A
Skeptic
Converted

During
his
keynote,
Shankar
invited
onto
the
stage
two
customers
who
had
participated
in
the
beta
testing
of
these
AI
products.

Of
particular
interest
was
customer

Cal
Yeaman,
project
attorney
at
Orrick,
Herrington
&
Sutcliffe,
who
admitted
he
had
been
highly
skeptical
of
using
gen
AI
for
review
before
testing
the
Review
Assistant
and
the
related
Coding
Suggestions
features
for
himself.

In
his
testing,
he
compared
the
results
of
the
gen
AI
review
tool
against
the
results
of
both
human
review
and
predictive
coding
for
finding
responsive
and
privileged
documents.

“I
was
surprised
to
find
that
the
generative
AI
coding
suggestions
were
more
accurate
than
human
review
by
a
statistically
significant
margin,”
he
reported.

He
speculated
that
others
might
get
different
results
when
using
the
gen
AI
review
tool,
depending
on
their
criteria
for
the
case,
the
nature
of
the
case,
and
the
underlying
subject
matter.

“But
the
more
subject
matter
expertise
is
required,
the
more
it’s
going
to
favor
something
like
the
generative
AI
model,”
he
said.

Another
way
in
which
the
gen
AI
review
impressed
him
was
its
consistency
in
coding
documents.
“If
it
was
right,
it
was
consistently
right
the
whole
way
through.
If
it
was
wrong,
it
was
consistently
wrong
the
whole
way
through.”
That
consistency
meant
less
QC
on
the
back
end,
he
said.

He
also
commented
on
the
speed
of
the
gen
AI
tool
compared
to
other
review
options.
In
just
a
few
hours,
he
was
able
to
complete
two
tranches
of
review
of
some
4,000-5,000
documents,
including
privilege
review.

Even
for
someone
who
is
inefficient
in
their
use
of
gen
AI,
the
review
would
have
cost
less
than
half
that
of
a
managed
review,
and
for
someone
who
is
proficient
in
these
tools,
the
cost
would
be
only
5-20%
of
the
cost
of
managed
review.
“So
it
was
a
massive
savings
to
the
client,”
he
said.

Of
course,
cost
doesn’t
matter
if
the
product
can’t
do
the
job,
he
said.
On
this,
he
said,
of
all
the
documents
that
the
model
suggested
were
not
relevant,
the
partner
who
reviewed
the
results
as
the
subject
matter
expert
found
only
one
that
he
considered
was
relevant,
and
that
was
a
lesser-inclusive
email
that
was
already
represented
in
the
production
population.

He
said
it
was
also
highly
impressive
in
its
identification
of
privileged
documents,
catching
several
communications
among
lawyers
who
the
review
team
had
not
been
aware
of
or
who
had
moved
on
to
other
positions.
In
one
instance,
it
flagged
an
email
based
only
on
a
snippet
of
text
that
a
client
had
copied
from
one
email
chain
and
pasted
into
another
email
with
only
the
lawyer’s
first
name
to
identify
him
and
no
reference
to
him
as
an
attorney.

“There’s
no
indication
that
it
was
an
email
to
an
attorney.
There’s
no
indication
that
it’s
necessarily
privileged.
Nothing
in
the
metadata.
No
nothing.”

Overall,
he
said,
there
was
close
alignment
between
the
gen
AI
coding
suggestions
and
the
predictive
coding,
with
their
suggestions
generally
varying
by
no
more
than
5-10%.

However,
in
those
cases
where
there
was
sharp
contrast
between
the
generative
AI
suggestions
and
the
machine
learning
models,
he
said,
then
in
every
instance
the
subject
matter
expert
found
that
the
gen
AI
had
gotten
it
right.

“Those
documents
tended
to
be
something
that
needed
some
sort
of
heuristic
reasoning,
where
you
need
some
sort
of
nuance
to
the
reasoning,”
he
said.

Other
New
Products

For
all
the
focus
on
generative
AI
at
the
Everlaw
Summit,
Shankar
noted
that
only
20%
of
the
company’s
development
budget
is
devoted
to
gen
AI,
with
the
rest
going
to
enhancing
and
developing
other
features
and
products.

In
a
separate
presentation,
two
of
the
company’s
product
leads
gave
an
overview
of
some
of
the
other
top
features
rolled
out
this
year.
They
included:

• Multi-matter
  models
  for
  predictive
  coding.
  This
  provides
  the
  ability
  to
  leverage
  predictive
  coding
  models
  created
  in
  one
  matter
  to
  be
  reused
  in
  subsequent
  similar
  matters,
  making
  it
  possible
  to
  generate
  prediction
  scores
  on
  new
  matters
  almost
  immediately.
  Over
  time,
  customers
  will
  be
  able
  to
  create
  libraries
  of
  predictive
  coding
  models.
• Microsoft
  Directory
  Integration
  for
  Legal
  holds.
  This
  feature
  allows
  users
  to
  create
  dynamic
  legal
  hold
  directories
  by
  connecting
  a
  Microsoft
  Active
  Directory
  to
  their
  legal
  holds
  on
  Everlaw.
  That
  can
  streamline
  the
  process
  of
  creating
  a
  legal
  hold
  and
  keep
  custodian
  information
  in
  existing
  legal
  holds
  up
  to
  date.
• Enhancements
  to
  Everlaw’s
  clustering
  and
  data
  visualization
  tools.

A
Note
on
the
Conference

This
was
my
first
time
attending
the
Everlaw
Summit.
As
it
generally
the
case
with
customer
conferences,
there
would
be
little
reason
to
attend
for
those
who
are
not
either
customers
or
considering
becoming
customers.

That
said,
the
more
than
350
attendees
(plus
Everlaw
staff
and
others)
got
their
money’s
worth.
The
programs
that
I
attended
were
substantive
and
interesting,
and
many
covered
issues
that
were
not
product
focused,
but
of
broad
interest
to
legal
professionals.
(I
moderated
one
such
panel,
looking
at
the
discovery
issues
and
strategies
in
two
high-profile
litigations
that
have
been
in
the
news.)

The
conference
also
featured
two
fascinating
“big
name”
speakers
–

Shankar
Vedantam,
creator
and
host
of
the

Hidden
Brain
podcast,
and

Kevin
Roose,
technology
columnist
for

The
New
York
Times.

An
unfortunate
sidebar
to
the
conference
was
the
strike
by
workers
at
The
Palace
Hotel,
the
Marriott-owned
hotel
where
the
conference
was
held.
Just
a
couple
days
before
the
conference
started,
they
started
picketing
outside
the
hotel,
joining
a
strike
and
picket
lines
that
are
ongoing
at
Marriott
hotels
throughout
the
United
States.

Workers
are
seeking
new
collective
bargaining
agreements
providing
higher
wages
and
fair
staffing
levels
and
workloads.

You
can
read
more
about
the
hotel
workers’
campaign
at

UnitedHere!
and
find
hotels
endorsed
by
UniteHere
at

FairHotel.org.