Predictive Coding—A Dispatch From the Front Lines of E-Discovery
"Predictive coding" is an increasingly popular tool in productions of electronically stored information. Predictive coding holds the promise of dramatically reducing the costs of discovery in cases involving large volumes of data. Moreover, there is evidence that predictive coding—when used properly and in the right circumstances—may be more reliable than the more expensive methods it replaces.
To date, there have been only two judicial decisions discussing the use of predictive coding: Da Silva Moore and Global Aerospace.[1] Jones Day argued in opposition to the predictive coding motion in Global Aerospace. That motion will hardly be the last discovery dispute involving this new technology, and the lessons learned in Global Aerospace can be instructive.
Predictive Coding
In recent years, discovery costs have ballooned as people increasingly write, transmit, and store documents electronically. As a result of this "electronically stored information," or "ESI," clients store vastly more documents than ever before. Predictive coding, which is also known as "computer-assisted review," is a means of fighting back against this expense by enlisting computer technology to help identify responsive or privileged documents. Predictive coding can also generate work product such as lists of significant documents.
Definitions of "predictive coding" vary, but a common form of predictive coding includes the following steps. First, the data is uploaded onto a vendor's servers. Next, representative samples of the electronic documents are identified. These "seed sets" can be created by counsel familiar with the issues, by the predictive coding software, or both. Counsel then review the seed sets and code each document for responsiveness or other attributes, such as privilege or confidentiality. The predictive coding system analyzes this input and creates a new "training set" reflecting the system's determinations of responsiveness. Counsel then "train" the computer by evaluating where their decisions differ from the computer's and then making appropriate adjustments regarding how the computer will analyze future documents.
This process is repeated until the system's output is deemed reliable. Reliability is determined by statistical methods that measure recall—the percentage of responsive documents in the entire data set that the computer has located—and precision—the percentage of documents within the computer's output set that are actually responsive. (That is, "recall" tests the extent to which the predictive coding system misses responsive documents, while "precision" tests the extent to which the system is mixing irrelevant documents in with the production set.) The resulting output can be either produced as is or further refined by subsequent human review.
In large productions, predictive coding can provide real cost savings. If humans need not look at a significant percentage of the collected documents, the savings over millions of documents is tremendous. Proponents of predictive coding, citing published studies, also assert that it is more accurate than having humans review every document.
Of course, like any tool, predictive coding has its disadvantages, and it is not the right tool for every case. Setting up a predictive coding system is expensive. While there can be benefits from predictive coding even in ordinary-sized cases, the cost savings are amplified as volume rises. Furthermore, the amount of "training" necessarily increases when the predictive coding system is asked to find documents responsive to multiple concepts. Thus, the system works best where the ratio of documents to document requests is high.
Where the savings are marginal—where the number of electronic documents is substantial but not overwhelming—counsel should evaluate other means of being efficient. Where electronic documents are well organized, through folders or otherwise, it may be relatively easy to determine the irrelevancy of entire folders. Thus, if a client has distinct lines of business or distinct projects that are irrelevant to the litigation, the corresponding folders might not warrant review. Of course, concepts important to litigation often cut across the organizational structures used for business purposes, and the strength of predictive coding lies in dealing with those situations.
Even where predictive coding is used, counsel must still evaluate, for each document request, the best approach to locating responsive documents. A resolution of the board of directors might be best found by looking in the minute book, a request for an accounting report might be best fulfilled by asking the appropriate employee to generate a report, etc.
Keyword (Boolean) searches yield cruder results than predictive coding systems but may nevertheless be helpful, either in place of or in addition to other techniques. For example, a witness may be of such importance that it is worthwhile to look at all documents bearing his or her name or email address. A keyword search might also be warranted for a distinctly named project that lies at the heart of the litigation.
The Global Aerospace Litigation
On February 6, 2008, an airplane hangar known as the Dulles Jet Center collapsed, damaging and destroying many airplanes. The airplane owners (or their insurers) filed numerous lawsuits in the nearby state court, the Circuit Court for Loudoun County, Virginia. These cases were consolidated under the name Global Aerospace. The principal defendants are Landow & Company Builders, Inc. and Landow Aviation Limited Partnership, affiliated entities that built and owned the hangar. Dozens of subcontractors and suppliers were also named as defendants or third-party defendants.
At first blush, Global Aerospace was an unlikely vehicle for a groundbreaking decision on electronic discovery techniques. The decision emerged from a courthouse that does not permit electronic filing. The circuit court's civil docket overwhelmingly reflects the troubles of ordinary citizens rather than complex business litigation. Global Aerospace was also an unlikely source of a predictive coding dispute, given the size of the parties that sought to employ the tool.
The Landow defendants were, and remain, fairly small businesses. During the construction of the Dulles Jet Center, they operated out of the second floor of a suburban apartment building, a space that Landow & Company currently shares with three other businesses. Only seven Landow employees (including a secretary and a receptionist) ever worked on the construction project. The Landows' production of hard-copy documents was only about 55,000 pages, a significant portion of which consisted of photographs and technical drawings.
Early this year, counsel for the Landows asked Jones Day and counsel for other plaintiffs whether we would agree to the use of predictive coding. The Landows had accumulated between 250 and 300 GB of data, and they asserted that the only reasonable method of reviewing the data was through predictive coding.
To be sure, there was no rule requiring the Landows to ask for our consent, nor were they obligated to provide any information to substantiate the need for predictive coding. But there was equally no rule requiring the plaintiffs to provide an advance waiver of any objections to the use of predictive coding. Jones Day and counsel for the other plaintiffs were reluctant to provide any such waiver without first obtaining information showing that the proposed use of predictive coding was the best method for locating responsive documents.
We never received this comfort; many of our questions about the data went unanswered. The Landows' counsel refused to state how many emails they had accumulated, how many emails dated from the key time period, how many emails they had collected from the principal officer of the Landow entities, or how any of the documents were foldered or organized. The plaintiffs therefore did not consent to the use of predictive coding. The Landows then filed a motion for a protective order.
The Predictive Coding Ruling in Global Aerospace
In both Da Silva Moore and Global Aerospace, the courts entered orders permitting the use of predictive coding. Both courts did so in advance; in neither case had anyone produced any documents that had been located through a predictive coding tool. The similarities between the two rulings ends there. In Da Silva Moore, Magistrate Judge Peck held three conferences, resolved numerous technical disputes, and wrote a 15-page decision approving and mandating a detailed, 11-page protocol. In Global Aerospace, Judge Chamblin ruled from the bench after declining to hear expert testimony, and he then signed a two-sentence order.
The Landows' motion sought "an Order approving the use of predictive coding technology." However, in court they clarified that the relief sought was far more limited. The Landows' counsel told the court that "you don't have to approve the protocol" and that "you don't even have to approve of any parameters."
You simply have to let me do predictive coding because I will have this done in about two months' time, and I will be able to come back and show that I've coded the documents, trained the machine, separated the relevant from irrelevant documents and, frankly, validated the extent to which I've been able to do that and confirm the effectiveness of the tool….
All I'm asking you is to give me that two-month period to do this and to demonstrate that it works.
Judge Chamblin granted this request. His ruling was not an endorsement of a proposed protocol, nor did he rule that predictive coding was the tool best suited for the task at hand. Rather, he indicated that a party has the right to choose the method by which it would satisfy its discovery obligations and that there is no intrinsic bar to choosing a new technology.
When these issues that involve modern technology come up, I always like to compare them to what is the modern technology taking the place of? What is it doing that was done the old fashioned way?
And really … isn't what predictive coding [does] the same thing, as … a bunch of lawyers sitting in a room looking at a bunch of documents?
So why isn't this just the method that the Landow defendants want to use…?
The court concluded that it would give the Landows "two months to be able to do this." But "if you [plaintiffs] don't think it works right, then bring it back to the Court."
The parties then conferred to determine the wording of the order. At that point, the Landows' counsel stated that they could not commit to producing documents before 120 days. The parties agreed on 60 days for processing and up to 60 additional days for production. To document this agreement, the Landows' counsel marked up a form of order that they had previously prepared, and the court then endorsed it.
On the 60th day after entry of this order, Jones Day received an email from the Landows' counsel stating that processing was not complete and explaining that "we were seriously delayed in commencing predictive coding due to difficulties in processing such a large amount of data." The email then went on to indicate what progress had been made and stated that production would be "completed in accordance with the Court's schedule." Our client settled its claims shortly thereafter, before the 120-day deadline for production. After settling, Jones Day no longer had a clear window on discovery activity.
Lessons for the Future
Negotiating with Opposing Counsel. Virtually every judge encourages the parties to cooperate on discovery matters. Further, almost every court mandates a meet-and-confer in advance of discovery motions. But in implementing these mandates, the litigators must make judgment calls regarding when and how they will seek agreement.
Obtaining early agreement on discovery matters has real advantages. If the receiving party can become comfortable that the producing party is searching for the right documents in the right way, the producing party can undertake an efficient and targeted search, with the assurance that the production will not need to be redone later. But there are, equally, pitfalls to opening discussions too early.
Determining the best way of searching for documents cannot be done in the abstract. Especially in a large case, designing an efficient and thorough document search requires first learning the details about what documents the client keeps, how they are kept, and how many there are. Moreover, whether or not one is using predictive coding, even the best-designed search protocol will need to be tweaked as more is learned about the documents. Thus, one needs significant information even before initiating discussions, and starting negotiations too early could lead to negotiating against a moving target.
From the perspective of the receiving party, early negotiations of a predictive coding protocol can be even more difficult. The receiving party will generally know virtually nothing about the other side's documents and will not have direct access to the files, the employees who keep them, or the experts running the predictive coding tool. As the failed negotiations in Global Aerospace demonstrate, it is difficult to gain the comfort necessary to bless a particular search protocol without receiving extensive information about the documents.
Strategy will depend greatly on the nature of the documents, the document requests, and the producing party's ability to supplement the main production. But—at least regarding the technical details of predictive coding—all parties may be better off waiting until production is complete, and then negotiating whether and how the production should be supplemented.
Seeking Judicial Rulings. In both Da Silva Moore and Global Aerospace, the producing parties sought, and obtained, judicial approval of predictive coding in advance of performing the work. Will seeking advance approval be advisable in the bulk of the disputes that will follow? As with most discovery issues, the answer is a fact-dependent judgment call. But this judgment can be informed by the experience in the Global Aerospace litigation.
Keep in mind, first of all, the exact nature of the relief that the Landows obtained. They did gain something important—they learned that the court has no bias against parties making use of the latest technology. Beyond that, what Judge Chamblin actually ordered was not particularly helpful to the Landows. The written order states that the permission to use predictive coding "is without prejudice to a receiving party raising with the Court an issue as to completeness or the contents of the production or the ongoing use of predictive coding." That was the situation before the Landows filed their motion. Absent an order to the contrary, a producing party can always use any methodology that it believes to be a good-faith response to the pending requests, subject to the receiving party's right to object.
And at what cost did the Landows obtain their ruling? There is, of course, the money spent: The Landows filed a 19-page brief and flew in three experts and a senior partner for the hearing. Beyond that, the Landows committed to a timetable that they did not meet. They further promised a very high level of accuracy ("better than" "the 75 or 80 percent range"), verification of the results, and full transparency.
In deciding whether to seek an advance ruling on a predictive coding technique, keep in mind three questions. First, what is at stake? If, for example, in a big-document case there is a challenge to the right to use predictive coding at all, a producing party may want an advance ruling, rather than risking the possibility that the production would later need to be redone from scratch. However, as predictive coding techniques mature and improve, and as courts gain more experience with them, the viability of the technology will no longer be subject to serious question. With time, there will therefore be diminishing benefits from merely checking to see whether the presiding judge is allergic to the notion of computer-assisted document review.
Second, is the issue sufficiently concrete? Or in other words, why wouldn't the judge, as Judge Chamblin did, allow the producing party to attempt any method it wanted, with the caveat that "if [the receiving parties] don't think it works right, then bring it back to the Court." After all, such a ruling avoids questions that may be mooted by later developments, and to issue such a ruling, the court need not examine evidence or hear witnesses. Even Judge Peck, who spent considerable effort resolving technical questions, held that significant "types of questions are better decided 'down the road,' when real information is available to the parties and the Court."
Perhaps time is of the essence. Or perhaps it makes sense to resolve a dispute regarding which categories of documents are relevant, or which documents are sufficiently important to justify the cost of looking for them. But when asking for a judge's attention before the document production, one needs to justify why an advance ruling would be helpful.
Third, what are the costs of action or inaction? The costs of a motion include the out-of-pocket expenses and any resulting difficulties if the ruling is wholly or partially adverse. Bringing a motion might tax the judge's patience or might lead to making promises that are difficult to keep. On the other hand, waiting until after the production holds other dangers, in addition to potentially lengthening the discovery process. The producing party risks having to entirely redo the production. Even if this possibility is remote, there are many scenarios that could result in court-ordered supplementation of the production. The receiving party, in some circumstances, might face arguments that a delay in bringing objections would result in a waiver or estoppel.
Conclusion
Predictive coding, when properly used, can be a valuable tool in effecting Rule 1's goal of "the just, speedy, and inexpensive determination of every action and proceeding." But no computer system can replace the litigator's judgment: in deciding how to gather and produce documents, in evaluating the quality of the production one receives, and in navigating conflicts with opposing counsel.
Lawyer Contact
For further information, please contact your principal Firm representative or the lawyer listed below. General email messages may be sent using our "Contact Us" form, which can be found at www.jonesday.com.
Jonathan Berman
Washington
+1.202.879.3669
jberman@jonesday.com
Jonathan Berman presented the argument against the predictive coding motion in Global Aerospace.
Jones Day publications should not be construed as legal advice on any specific facts or circumstances. The contents are intended for general information purposes only and may not be quoted or referred to in any other publication or proceeding without the prior written consent of the Firm, to be given or withheld at our discretion. To request reprint permission for any of our publications, please use our "Contact Us" form, which can be found on our web site at www.jonesday.com. The mailing of this publication is not intended to create, and receipt of it does not constitute, an attorney-client relationship. The views set forth herein are the personal views of the author and do not necessarily reflect those of the Firm.
[1] There has also been briefing and argument regarding predictive coding in the Kleen Products litigation. However, there do not appear to have been any resulting rulings in Kleen Products, at least not in written form.
