Recently, CRC Press published Data-Driven Law: Data Analytics and the New Legal Services by Edward J. Walters. The volume’s contributed chapters cover a wide range of topics at various levels of mathematical rigor, but they all underscore the importance of how data science can greatly improve the quality and efficiency of the legal process from document discovery to predicting potential judgments. While Data-Driven Law does not specifically discuss arbitration or other alternative dispute resolution mechanisms in which Dispute Resolution Data (DRD) specializes, we nonetheless believe that the book’s publication is a strong leading indicator of the growing role that data analytics will play throughout the legal profession. Certainly, we recognize that many strategic decisions in this profession are informed by the invaluable wisdom that practitioners have accumulated over a career’s worth of experience. What we aim to provide is a means to help augment that wisdom with knowledge derived from the analysis of data that have been collected thoughtfully, thoroughly, and systematically. In this way, practitioners need not rely exclusively on hunches or educated guesses to guide their decision-making processes. With the power of data, they can now provide their clients with more definitive and fact-based assessments of how likely a case will result in a particular outcome, given a set of defined input parameters.
To that end, we presented the first in a series of blog posts discussing the results of data-driven analyses which support the notion that the most frequently observed outcome of international arbitration cases is settlement or withdrawal. In most cases, settlement/withdrawal is reached relatively quickly, often less than a year following the claim date, and also prior to any counter-claim, preliminary hearing, or hearing on the case’s merits. Our results were derived from an aggregate view of all available international commercial arbitration case data in the DRD repository, which presently consists of nearly 4,000 cases dating back to 2005.
In our September 2018 follow-up blog post, we demonstrated that because this data sample was relatively large, we can be quite confident that our statistical estimate of cases ending in settlement/withdrawal (currently 54%) has a low margin of error (hereafter abbreviated “MOE”). That is, our estimate (specifically, 54% ± 2%, with 95% confidence) is likely to be representative of the entire “population” of all international arbitration cases, when viewed as an aggregate across all case types and case regions. We invite the interested reader to review our September 2018 blog post, which includes a primer on how to compute the MOE while explaining, in accessible language, what the MOE actually means from a statistical standpoint.
However, just as having a large sample size yields a narrow MOE, a smaller sample size typically results in a wider MOE, given the same 95% level of confidence. And while the aggregate estimate of 54% (±2%) of cases ending in settlement/withdrawal speaks to the potential power of arbitration as a settlement tool in a broad sense, parties are likely to be more interested in estimates that pertain to a specific case type, or to a specific geographic region in which the arbitration took place, or perhaps to other, even narrower data segments extracted from the spectrum of international commercial arbitration. More specific criteria will, necessarily, reduce the size of the data sample to be analyzed. Here, we use selected results from analyzing the DRD database to demonstrate the effect of sample size on MOE, and we discuss how stakeholders may interpret such results with varying degrees of confidence.
For example, let us consider how the proportion of cases resulting in settlement/withdrawal varies by case type. In the DRD repository, the four largest sectors (by number of cases entered in the database) presently include commercial contracts (817 cases), hospitality & travel (456), construction (369), and wholesale & retail trade (314). From Figure 1, we can see that in all but one of these case types, settlement/withdrawal is the most frequent outcome, but in addition, the MOE varies inversely with the number of cases.
Figure 1: Percentages of international commercial arbitration cases (since 2005) resulting in settlement/withdrawal for each of four major case types (with a similar result for all case types shown for comparison). Margin of error (indicated by error bars) is computed at the 95% confidence level. To simplify the MOE computation, other case outcomes are grouped into a single “not settled/withdrawn” category.
For the four major case types shown in Figure 1, the MOE is about ±5%, which is arguably an acceptable degree of uncertainty for parties exploring arbitration as a dispute resolution mechanism. Such a relatively narrow MOE mainly results from each of these case types having a relatively large sample size (about 300 or more). In contrast, Figure 2 shows the arbitration case outcomes (and associated margins of error) for four selected case types that, at least presently, have smaller sample sizes. For example, there are currently 39 cases in the agriculture sector in the DRD database, approximately half of which resulted in settlement/withdrawal. The associated MOE of ±16% means that the actual percentage of all international commercial arbitration cases in the agriculture sector reaching settlement/withdrawal ranges between 33% and 65%, based on a 95% level of confidence.
Such a wide range may still guide the decision-making processes of the stakeholders exploring arbitration as a settlement tool. For example, in the mining & raw materials sector, the MOE is relatively high at ±11%, but the likely maximum proportion of cases reaching settlement/withdrawal is no higher than 50%. This suggests that, at least for this case type, one can be reasonably confident that the chances of reaching settlement prior to a hearing (or, for that matter, reaching settlement at any point in the arbitration process) is not insignificant, but it is still relatively low.
Figure 2: Percentages of international commercial arbitration cases (since 2005) resulting in settlement/withdrawal for four selected case types currently represented by fewer than 300 cases in the DRD database.
The fact that the degree of uncertainty of the case outcome is larger for reduced sample sizes does not imply the data are of poor quality; rather, the uncertainty is a function of the quantity of data. Consequently, in an effort to provide DRD’s customers with helpful guidance as to whether the sample size for a given scenario is large enough so that uncertainty is relatively minimal, DRD is introducing a “signal strength” metric intended to provide its users with a simple visual indicator of the confidence that a given DRD data sample is likely to be representative of the “population” of all cases associated with the category of data to which the sample corresponds, whether or not those cases are recorded in the DRD database.
Aesthetically, the signal strength indicator resembles the familiar “signal bars” that are a ubiquitous feature of the modern mobile phone’s user interface. Functionally speaking, the number of bars is based on a proprietary computation involving such inputs as sample size, the margin of error associated with the reported statistic, and the level of confidence on which the MOE is based. In short: the greater the number of bars, the greater the degree of confidence that the reported statistic lies within an acceptably low MOE (at most ±5%). Figure 3 illustrates several examples of the DRD “signal strength” as determined for statistical results selected from those presented in Figures 1 and 2.
Figure 3: Selected results annotated with DRD’s new “Signal Strength” metric: an “at-a-glance” indicator of the level of confidence that the associated statistic represents all cases meeting the same criteria.
We at DRD acknowledge the value of practitioner experience. We do not claim that our results should replace the wisdom of lawyers and other legal professionals who are keenly aware of the myriad possibilities of how cases play out in practice. Rather, we believe that our data-driven offerings can enhance “the human factor” with additional, largely quantitative knowledge that can inform the decision-making process, particularly within the alternative dispute resolution spectrum. In our next blog post, we will share further examples of results derived from a continually finer-grained analysis of DRD’s dataset. For example, we will present results demonstrating how the likely outcome of an international commercial arbitration case changes when settlement is not reached prior to a hearing on the merits. We will also report on how the magnitude of the claim amount can affect the case’s chances of yielding an award that meets or exceeds some threshold percentage of the original claim amount. These examples, combined with those presented in our prior blog posts, merely scratch the surface of what a comprehensive dataset like ours can offer, and we look forward to enabling the alternative dispute resolution community to learn the analytical tools needed to extract valuable insights that can potentially guide future strategic decisions.
________________________
To make sure you do not miss out on regular updates from the Kluwer Arbitration Blog, please subscribe here. To submit a proposal for a blog post, please consult our Editorial Guidelines.
