Kluwer Arbitration Blog

In virtually every sector of modern business, data is enhancing if not replacing intuition as the basis for making decisions. This trend holds even for assessments as seemingly subjective and rarified as predicting the quality—and hence price—of an exquisite French Bordeaux.

In selecting international arbitrators, however, intuition still predominates. For example, a recent industry survey by Bryan Cave Leighton Paisner found that the most important qualities in selecting an arbitrator are identified as “expertise” (according to 93% of respondents) and “efficiency” (according to 91%). Expertise and efficiency, however, are not easy to measure or quantify.

These qualities are not credentials that are listed on arbitrators’ CVs. Instead, expertise and efficiency are cumulative, largely intuitive assessments that are drawn from a number of sources and metrics. Moreover, what constitutes the best expertise or means for achieving efficiency may vary from case to case depending on a client’s needs.

Given the confidential nature of arbitration, gathering the relevant information means personal phone calls with individuals who have appeared before a potential arbitrator or, better yet, sat as a co-arbitrator with that person. This kind of ad hoc individual research largely confines assessment of potential arbitrators to feedback from a limited number of individuals. Despite this limited scope, ad hoc research can be time-consuming (and therefore costly), but not always reliable. Without broad data against which to evaluate these inputs, however, it is impossible to determine whether the feedback is broadly representative, readily transferrable to the case at hand, or just an outlier.

Another problem with ad hoc information gathering is that it creates an information bottleneck. Newer and more diverse arbitrators cannot readily develop international reputations as long as personal references are the primary means for determining expertise and efficiency. This informational bottleneck is increasingly intolerable in light of concerns about the lack of diversity among international arbitrators and in-house counsel with corporate benchmarks and applying greater pressure to find newer arbitrators about whom there is a scarcity of information.

Arbitrator Intelligence (AI) seeks to solve these problems by bringing data-driven analysis to arbitrator appointments. The means to these ends is the recently launched Arbitrator Intelligence Questionnaire, or AIQ.

The AIQ

The idea behind the AIQ is simple. The AIQ seeks to replicate, through systematically collected feedback, the same kinds of information currently sought through personal-to-person inquiries. Data from the AIQ will not eliminate altogether the value of individualized ad hoc inquiries, but it will allow parties and counsel to tap into the collective intelligence of the global international arbitration community.

The AIQ is designed for parties, in-house counsel, external law firms and even third-party funders to complete at the end of each arbitration. The web-based questionnaire asks a number of background questions about the case, and then inquires about a number of features that are relevant for future arbitrator selection. For example (to paraphrase a few questions from the AIQ): Did the arbitrators grant document production? If so, what standard did they use? Did the arbitrators ask questions that demonstrated familiarity with the record? Did contract interpretation in the award reflect a plain meaning analysis of the words in the contract? Or did it consider the drafting history? Or did it seek to adopt a more flexible interpretation to achieve fairness and equity in the outcome of the dispute?

As a practical matter, the AIQ is divided into two phases, and each phase can be completed in 10 minutes or less. Phase I concentrates on objective background information about the case, and can be completed by anyone who has access to the award or case file. Phase II contains questions that relate to the conduct of the arbitration and, in some instances, seek professional assessments. As a consequence, Phase II should be completed by an attorney or party who actively participated in the proceedings. Certain background information from Phase I questions automatically prefills the relevant questions in Phase II to make it even faster to complete.

In developing the questions for the AIQ, AI employed state-of-the-art survey design (in coordination with the Penn State Survey Research Center), as well as extensive public and expert input. The ultimate goals were multiple and ambitious: to ensure quality feedback, to avoid questions that even implicitly preferenced certain cultures or legal traditions, to ensure fairness to arbitrators, and to promote systematic responses.

Achieving systematic completion of AIQs is Arbitrator Intelligence’s biggest challenge. To that end, AI is entering into collaboration agreements with various arbitral institutions around the world. Under these agreements, institutions agree to forward the AIQ to parties and lawyers at the end of each arbitration, and in exchange AI will give collaborating institutions free access to AI Reports (see below).

To date, AI has formally entered into such agreements with a few institutions (such as Singapore International Arbitration Centre and AMCHAM Quito), and is in discussions with more than a dozen other institutions. So watch for emails coming to you from arbitral institutions at the end of your arbitration!

AI is also inviting parties and law firms to support it by signing The Arbitrator Intelligence Pact. By signing the AI Pact, parties, law firms, individual counsel, arbitrators, arbitral institutions, and arbitration organizations commit to supporting AI’s goals of transparency, accountability, and diversity by helping to promote completion of AIQs regularly at the conclusion of arbitrations.

Notably, one of the world’s leading law firms has not only signed The Pact, but also agreed to provide retrospective AIQs on cases completed in the last few years. AI is currently in discussions with several other firms that are also considering providing retrospective AIQs. AIQ data is essential for AI to develop AI Reports, so consider joining these industry leaders by completing AIQs on recently completed arbitrations.

Once sufficient information has been collected through the AIQ, Arbitrator Intelligence will begin publishing AI Reports, through its partner WoltersKluwer.

Arbitrator Intelligence Reports

AI Reports are still in the development phase, and the nature and scope of AI Reports will inevitably evolve over time, particularly as AI’s base of data expands. Nevertheless, it is already easy to see from some preliminary mock-ups how AI Reports will help promote more data-driven decisions about arbitrator appointments.

By way of preview, consider the following chart regarding a (hypothetical) arbitrator’s approach to document production:

Figure A
(based on hypothetical data—for illustrative purposes only)

This basic chart provides a systematic comparison of the arbitrator’s historical practice in granting document production (the light blue bars to the left), as compared with the document production practices of all arbitrators in the sample oil and gas cases (the dark blue bars to the right).

There are several advantages to this approach over ad hoc individual inquiries, or self-reporting by arbitrators. First, when asked to comment on their own practices, most arbitrators explain that their approach will vary depending on the type of case. This chart examines disputes within a particular industry (oil and gas), but it could alternatively evaluate the data based on case size, applicable law, or some combination of these or other variables.

Even more importantly, Figure A above and Figure B below demonstrate the benefits of assessing individual cases in comparison to a baseline of data in similar cases.

Figure B
(based on hypothetical data—for illustrative purposes only)

In Figure B, the y-axis indicates how many days an award is rendered after close of proceedings (defined in the AIQ as the last day of hearings or the day of the last post-hearing submissions). The x-axis indicates the size of the case as a proxy for complexity (on the assumption that more time is needed to draft awards in more complex cases). The blue line shows the relationship between amount at stake and length of hearing for all arbitrators presiding in oil and gas cases in the sample. Each red x is a case decided by the arbitrator of interest.

Like Figure A above on document production, the independent baseline in Figure B (the blue line) provides a valuable check against mistaken assumptions about the representativeness of performance in a particular case. For example, by luck of the draw, ad hoc research may reveal two examples of cases in which an arbitrator rendered awards more than 200 days after the close of proceedings. Based on this feedback, a client may conclude that this arbitrator is simply too slow and thus disqualified from consideration. But that assessment may be different if broader data reveals that only a few of the arbitrator’s awards took longer than 200 days or that, depending on the size of the case, a 200-day timeframe is well within the norm for all similar cases.

These charts and graphs are prototypes for off-the-shelf AI Reports and, again, are based on hypothetical data. AI Reports will provide numerous forms of data analysis on various topics, and the range will inevitably grow and develop over time as more data is generated.

In the future, AI also anticipates being able to produce customized reports as more data is available. For example, in some cases, the ability to obtain (or avoid) document production may be the lynchpin of a party’s strategy. In that case, a party may want a bar chart similar to the Figure A above, but instead each of the three arbitrators on its shortlist.

Of course, AI Reports will identify the limitations of the data, particularly in production of early AI Reports. More generally, there are a number of challenges in analyzing data from phenomena as complex as arbitral disputes. Such challenges include accounting for different institutional rules, differences in appointment of the arbitrator (was the arbitrator party-appointed, or sitting as a chair or sole arbitrator?), and changes in data and to arbitration practice over time.

As an academically affiliated entity, however, Arbitrator Intelligence is uniquely positioned to meet these complex challenges. AI’s Board of Directors will oversee development of the AI Reports and the software needed to generate them. The Board is composed primarily of university professors who collectively possess the essential range of expertise in relevant fields, including empirical research in international arbitration, data analytics in the legal profession, mass data collection and strategic decision-making, econometrics, artificial intelligence, and information systems.

In addition to its Board of Directors, AI also has a Board of Advisors that brings to the project diverse perspectives from among in-house and external counsel, leading arbitrators, institutional representatives, and academics specializing in international arbitration.

Conclusion

When Arbitrator Intelligence was first conceived, major law firms stated (unabashedly!) that they hoped this project would fail. AI would be seeking to gather and make widely available information that they sold to their clients, information that signaled their value-added expertise, information that distinguished them from lesser competitors. And they did not want the competition.

Today, given the size and complexity of the market, the reaction is quite different. Even the leading law firms with the largest networks for collecting information recognize that there is no such thing as “enough information” about arbitrators. In-house counsel are increasingly demanding more than mere intuition to justify arbitrator appointments. They want concrete data and analysis that their colleagues use in making other business decisions and that they will especially need if they have to explain an unexpected result to management. Even arbitral institutions, which also appoint arbitrators, increasingly need more information to optimize their appointments and remain competitive.

For those of us who enjoy drinking good wine, but not necessarily investing in wine futures, we may still prefer the tasting notes of well-known aficionados and recommendations from a sommelier’s tastevin. But for parties selecting the individuals who will pass judgment on their most important disputes, precision is critical and should not be left to intuition alone. Arbitrator Intelligence will liberate arbitrator selection from the 19th Century’s telephone and introduce it to the 21st Century’s data-driven analytic solutions.

To make sure you do not miss out on regular updates on the Kluwer Arbitration Blog, please subscribe here.

________________________

To make sure you do not miss out on regular updates from the Kluwer Arbitration Blog, please subscribe here. To submit a proposal for a blog post, please consult our Editorial Guidelines.