Kluwer Arbitration Blog

You might be forgiven if you thought “moneyball” was the name of a new lottery game. It’s an easy mistake if you have not read Moneyball, Michael Lewis’ critically acclaimed book or seen the 6-time Academy-Award-nominated film starring Brad Pitt.¹⁾The analogy for this post is adapted from Professor Chris Zorn, my Penn State colleague, Member of Arbitrator Intelligence’s Board of Directors, and co-founder of Lawyer Metrics, a company he describes as “Moneyball for Lawyers.” The content of this essay was developed for presentations on July 2-3 in Costa Rica at the Centro Internacional de Conciliación y Arbitraje (CICA), and a conference on Innovation, Technology and Law, co-sponsored by CICA, AmCham San Jose, and arbitration specialist Herman Duarte.

If you have read Lewis’ book or seen the movie, however, you would know that the term “moneyball” is slang for sabermetrics, or the practice of crunching data to pick baseball players who are dismissed by conventional wisdom and hence undervalued by the market. As told in the book, sabermetrics enabled the Oakland Athletics baseball team to put together an exceptionally talented team with a pathetically meager budget.²⁾Moneyball is not without its skeptics and detractors. Instead of spending millions to buy baseball’s flashiest superstars, the Athletics concentrated on players who consistently got on base and brought in understated runs that added up to victories.

“Wait,” you say, “I know nothing about baseball!” Don’t worry. Neither do I.³⁾An apology is perhaps also in order. Most of the world is more focused on G-O-O-O-O-A-Ls in the World Cup, not on America’s favorite pastime. Unfortunately, I know even less about football than I do about baseball and Michael Lewis has yet to write a book about selecting football players. Fortunately, Lewis’ book is about much more than just baseball.

The book is also about how information can put underfunded outsiders on a level playing field with rich insiders, about how a new, verifiably superior approach can be irrationally rejected by traditionalists, and about the “ruthless drive for efficiency that capitalism demands.” These themes, as it turns out, apply as much to the process of selecting arbitrators as they do to the process of selecting baseball players.

To give some background, future professional baseball players are traditionally recruited after they have been identified by insider “scouts.” In the course of their scouting, these (most often) retired baseball players reject out of hand, and fail to see the value of, players they consider to be misfits. Conventional wisdom counts among these so-called misfits “short right-handed pitchers,” “skinny little guys who get on base,” or “fat catchers.”

For example, in Moneyball we are told that scouts laughed at a catcher who “wears big underwear” (i.e., is overweight), so they overlooked his uncanny ability to “control the strike zone” and hence earn “walks” to first base that accumulate into scored runs.⁴⁾In baseball, a batter earns a “walk” if he receives four pitches that the umpire determines were “balls,” meaning that they were outside of the “strike zone” (the space between the batter’s shoulders and knees). When a batter earns a “walk,” he can go directly to first base and cannot be called out, as can only occur if the batter hits the ball and runs to first base. The scouts likewise dismissed a rather puny center fielder who nevertheless had a “gift for getting on base.” And they also considered a “bizarre sight” the double-jointed pitcher (cruelly nicknamed “The Creature”) who had an “84-mph fastball” and ended up being named “the closer on the rookie league All-Star team.”

In addition to failing to see hidden talent, traditional scouts are also reluctant to admit any misjudgment on their own part. They are apparently so confident in their ability to see talent in young ball players, even when their favored player flames out, they talk “as if he’d become exactly what they all said he would be and it was only by some piece of sorcery that he didn’t have the numbers to prove it.”

While obviously not quite the same thing, the process of “scouting” for international arbitrators has something in common with scouting for baseball players. In international arbitration, parties and lawyers often report that they have picked a “big name” arbitrator for the reassurance that comes with a prominent reputation and, presumably, the relative ease with which they can be identified. These big name arbitrators certainly look the part, and have all the conventional credentials that are hard to argue about.

But when, for whatever reason, the outcome of an arbitration is unexpectedly disappointing,⁵⁾Losing may not be the only disappointing outcome. Increasingly, in-house counsel are complaining about long waiting times to schedule hearings and inexplicable delays in the time for rendering the award. the big name arbitrator is still regarded as having been a proper, “safe choice.” It must have been some piece of sorcery that made the arbitration come out wrong.

Going forward, the expanding and diversifying the pool of arbitrators will require us to reconsider what makes for a good arbitrator and how they should be identified. Parties often say they want the “the best person for the job,”⁶⁾Anonymous posting to OGEMID@mailtalk.ac.uk (9 February 2012, 03.27 CST), cited in Lucy Greenwood & Mark Backer, Getting a Better Balance on International Arbitration Tribunals, 28 Arbitration INTERNATIONAL 653, 661 at n. 42 (2012). but then they fail to inquire further what exactly that means. Let’s consider a few possible examples of features that might make an “unconventional” arbitrator the best choice.

Younger arbitrators who are eager to establish a reputation may redouble their preparation for hearings—a particularly valuable trait in a case with complex facts. An African arbitrator may have unique insights about trade usages in the region that would otherwise require expert testimony,⁷⁾This is a point made in Won Kidane’s book, The Culture of International Arbitration (Oxford 2017). and an Asian arbitrator may be more willing to take on allegations of corruption by a multinational. Overall, newer arbitrators may have fewer cases, which will allow them to transform arbitral decisions quickly into written awards.

In Moneyball, Lewis quips that “baseball scouting was at roughly the same stage of development in the twenty-first century as professional medicine had been in the eighteenth.” The same could be said of modern international arbitrator selection. Much like the “fraternity of old scouts” who use their gut instincts to identify future baseball players, parties and attorneys select international arbitrators based on intuition, and supplement that intuition with ad hoc, person-to-person research, usually over the telephone.

One primary obstacle to parties and attorneys making less obvious, but potentially more strategic, choices about arbitrators is that the critical data about newer arbitrators is simply not available in the same way it is about baseball players (or more senior arbitrators). But statistics about baseball players were not always available either. As the forefather of sabermetrics, Bill James, “never turns loose of a statistic unless they get a dollar for it.” This approach was self-defeating, according to James, because:

The entire basis of professional sports is the public’s interest in what is going on. To deny the public access to information that it cares about is the logical equivalent of locking the stadiums and playing the games in private so that no one will find out what is happening.

A similar impatience now exists regarding the unavailability of arbitrator data. Even the most sophisticated lawyers acknowledge that you can never have “enough” information about arbitrators. Meanwhile, a whopping 92% of respondents in a recent survey by Berwin Leighton Paisner (now Bryan Cave Berwin Leighton Paisner, or BCLP), said they want more information about arbitrators.

Some arbitral institutions and outside groups (like ArbitralWomen) are working to expand the pool of arbitrators by making names of arbitrators more readily available for consideration. The problem is that without information to go with these names, even if an arbitrator makes a short list, he or she is unlikely to be chosen, particularly if up against a more known and established arbitrator.

What kind of information is needed? In the same BCLP survey, responders identified the most important qualities in an arbitrator as “expertise” (according to 93% of respondents) and “efficiency” (according to 91%). Expertise and efficiency, however, are not easy to measure or quantify. These qualities are not quantifiable data points listed on arbitrators’ CVs. Instead, expertise and efficiency are cumulative, largely intuitive assessments that are drawn from a number of sources and metrics. Moreover, what constitutes the best expertise or means for achieving efficiency may vary from case to case depending on a client’s needs.

Arbitrator Intelligence (AI), through the AI Questionnaire or AIQ seeks to disaggregate these qualities into data that parties and attorneys can more readily use. For example, in one case, a party might regard “efficiency” as the ability to effectively limit document production, while in another case, that same party may regard “expertise” as the ability to discern that the party can only effectively prove its case with documents held by the opposing party.

To hone in on the critical information, the AIQ asks whether document production was requested (by whom) and, if so, whether it was granted. It then asks what standard was used:

Which of the following describe(s) the document production ordered by the tribunal (please select all that apply)?*
*Descriptions of document categories are based on art. 3(3) of the IBA Rules on the Taking of Evidence in International Arbitration (2010)

• Production was ordered in accordance with the agreement of the parties
• Production was ordered of a limited number of individually identified documents
• Production was ordered of “narrow and specific requested category[ies] of Documents that are reasonably believed to exist”
• Production was ordered of documents maintained in electronic form based on identification of “specific files, search terms, individuals or other means of searching for such Documents in an efficient and economical manner”
• Production was ordered of broad categories of documents based on general statements of materiality and relevance

 

When enough data is collected, responses to this question can be triangulated against data from other questions in the AIQ, such as the size of the case, the industry in which the dispute arose, the legal seat, etc. Responses can also be compared with cumulative average among arbitrators in similar cases, or with the specific track records of other arbitrators.

In another example, some cases hinge on interpretation of contract, statutory, or treaty provisions, or on trade usages. The AIQ collects information on arbitrators’ treatment of those issues in the arbitral award. For example, with respect to contract interpretation, the AIQ asks responders:

 

In your professional judgment, which of the following describe(s) the tribunal’s contract interpretation (please select all that apply)?

• The award reflects a plain meaning analysis of the specific words of the contract
• The award considers the negotiation and drafting history of the contract
• The award relies primarily on precedents in relevant cases
• The award reflects a flexible interpretation of the specific words of the contract in order to give the contract its common sense or commercial sense meaning
• The award reflects a flexible interpretation of the specific words of the contract in order to achieve fairness and equity in the outcome of the dispute
• Other (please specify):

Again, responses to this question can be triangulated against various other data points collected through the AIQ, and used to compare particular arbitrators’ past rulings to overall averages or other arbitrators under consideration for appointment.

This data will soon be available in “Arbitrator Intelligence Reports” (AI Reports) that will be available for a fee through Kluwer.

AI Reports will ultimately be based on thousands of data points, not the millions of data points sabermetrics relies on. For this reason, AI Reports will always be a far cry from moneyball. And that is both a necessary and a good thing. Arbitration is not a game, and arbitrators are not players following a set of prescribed rules on a clearly defined field. The complexity of international arbitration, and the task of arbitrating, will defies an easy algorithmic process for selecting arbitrators. But AI Reports will provide a more meaningful starting point for both insiders and outsiders.

In addition to making arbitrator selection more precise and predictable, AI Reports will allow parties and attorneys to consider a broader range of criteria, and open up information about a broader pool of candidates. AI Reports will also force parties and attorneys to at least begin their selection process with a more objective assessment of arbitrators and their professional qualities.

So, at the end of your next international arbitration, take a few minutes to fill out an AIQ.⁸⁾The AIQ can be previewed and accessed on the AI website. You may not end up being mentioned in a book by Michael Lewis or be played by Brad Pitt (or Jennifer Lawrence) in the movie version. But you can help contribute the much-needed data to upgrade international arbitrator selection.

________________________

To make sure you do not miss out on regular updates from the Kluwer Arbitration Blog, please subscribe here. To submit a proposal for a blog post, please consult our Editorial Guidelines.