The next generation of the bar exam is coming. Well, hopefully.
The National Conference of Bar Examiners (NCBE) released its Testing Task Force’s Phase 1 report this week. The report summarizes what people throughout the legal profession think about the current licensing process and exam. While readers will not be surprised by the general sentiments — people are generally dissatisfied with the current bar exam — they may be surprised that the NCBE published such forceful critiques.
But first some background.
The NCBE is the nonprofit organization that state regulators in all U.S. jurisdictions outsource between some and a lot of the work that goes into determining whether someone should receive a law license. The NCBE is the group behind the Multistate Professional Responsibility Exam (MPRE), Multistate Bar Exam (MBE), Multistate Essay Exam (MEE), and the Multistate Peformance Test (MPT). The NCBE also coordinates the Uniform Bar Exam (UBE), a compact among 36 jurisdictions to uniformly administer, grade, and score the MBE, MEE, and MPT. A test-taker’s results can then be transferred to any of the other UBE jurisdictions. Each state still sets its own passing score, so you can pass the UBE in one state but not another.
The NCBE created its Task Force as part of its efforts “to sustain confidence by all stakeholders in the testing process.” While some consternation stems from falling bar pass rates, legitimate criticism of the exam’s validity from some of legal education’s best thinkers has plagued the bar exam for several decades. That is, the exam may not measure what it says it does: whether an individual is minimally competent to practice law.
From my conversations with NCBE’s new leadership over the past few years, they are deadly serious about the exam’s validity. The Testing Task Force’s transparent process is a tacit acknowledgment that the NCBE has not listened very well in the past, and that there are meaningful questions about how the exam measures minimum competence.
Phase I involved listening sessions with lawyers, judges, examiners, and educators. Phase II involves a practice analysis to “gather current, empirical data on the knowledge, skills, abilities, other characteristics, and technologies that newly licensed lawyers use to accomplish the job tasks they perform.” Phase III will involve turning the results of Phase I and II into exam design recommendations by the end of 2020.
(If you practice law, go take the 20 minute practice analysis survey.)
That brings us to the Phase I report and some common threads throughout all the listening sessions.
- The exam tests both too much and too little. That is, it should emphasize more lawyering skills and less subject matter knowledge.
- The exam should utilize more writing, less multiple-choice, and additional methods like simulation.
- Jurisdictions should consider breaking the exam into different parts over more than just a few days.
The report also includes notes for each session. The following bullets are directly from the report but paraphrased from attendees.
- “The MEE is the least valuable component of the bar exam because it is not realistic: it requires answering short essays based upon memorization of the law, which is not consistent with how lawyers practice (e.g., with access to electronic databases like Westlaw or LexisNexis)”
- “The inconsistent passing scores implemented across states raise questions about the legitimacy of exam results; states need to agree on a definition of minimum competence”
- “The MBE tests arcane, obscure, or trivial aspects of the law that new practitioners should not be expected to know and are not reflective of minimum competence; the MBE tests too deeply on subjects; memorizing black-letter law for the MBE to answer multiple-choice questions (MCQs) does not mimic real practice because lawyers would look up the law and not rely only on memory in representing clients; too much focus on memorization; the MBE tests only memorization and no skills; the MBE questions are full of red herrings and intentionally tricky”
- “MCQs are not realistic or an effective way to test what lawyers do; if retaining MCQs for the MBE, reduce the number of questions or increase the amount of time allowed”
I’ve focused on these negative sentiments because they tell us something important about NCBE’s approach. Because NCBE paraphrased what session attendees said, it could have softened these remarks without anyone being the wiser. Session attendees also provided enough material that NCBE could have told a completely different story in its executive summary and declined to publish such extensive session notes.
Instead, the report showcases an array of dissatisfied stakeholders who directly question whether the exam actually measures minimum competence at all. Each of these critiques cut right to the heart of the bar exam itself. If the test methods and content are not effective or realistic or relevant to law practice, it’s difficult to imagine how the exam could be valid.
We operate in a world of standardized tests for some good reasons, and the bar exam is no exception. Not only do state regulators expect the exam to be valid, but also to be reliable and thus able to yield stable scores over time. Often, however, test-makers sacrifice validity for reliability because the principal means for achieving reliability is through multiple-choice questions that don’t quite reflect what people in the real world do. I’m hopeful that the bar exam of the future will focus significantly more on validity.
NCBE is not the only organization concerned about validity. I am working with a team of researchers on the Building a Better Bar project at the Institute for the Advancement of the American Legal System (IAALS) to explore the work that new lawyers do during their first year after licensing. We’re holding 60 focus groups in 12 states to shed light on the “minimum competence” that lawyers need to serve clients in a wide variety of practice settings. Our focus-group approach, we believe, will complement NCBE’s job analysis surveys.
The only fair reason to limit licensure to those who pass the bar is to protect the public. If the bar exam does not measure minimum competence, it needs to change or be eliminated. Fortunately the NCBE seems poised to oversee that change in the near future.
Kyle McEntee is the executive director of Law School Transparency, a 501(c)(3) nonprofit with a mission to make entry to the legal profession more transparent, affordable, and fair. You can follow him on Twitter @kpmcentee and @LSTupdates.