The Arbitrator Intelligence Questionnaire (AIQ) is a feedback questionnaire that will be used by Arbitrator Intelligence (AI) to collect information on case management and decision making practices of international arbitrators. After pilot testing and extensively vetting the AIQ, we are now asking for public comments to help us further refine it.
A demonstration version of the draft AIQ is now available for review on our website. There, you will also find an interactive annotation tool that will allow you to register your comments directly onto the text of the AIQ. Come give us your input!
Why Is the AIQ Needed?
The AIQ is a potential solution to some of international arbitration’s most salient challenges. For example, a recent survey by Berwin Leighton Paisner investigating the lack of diversity in international arbitration found that a staggering 92% of respondents wanted more information about new and less well-known arbitrators, and a whopping 81% of respondents wanted to give feedback about arbitrators at the end of cases.
Respondents to the 2015 Queen Mary Survey on Improvements and Innovations in International Arbitration identified the third worst characteristic of international arbitration as the “lack of insight into arbitrators’ efficiency.” Meanwhile, most responses to the Survey about how institutions could improve international arbitration involved providing more information about arbitrators, how they are appointed, and their decision making.
Finally, and most recently, three out of the 10 Hot Topics in International Arbitration for 2017 identified in a Kluwer blog post – transparency, the arbitrator selection process, and diversity – go to the heart of AI’s mission.
The AIQ is our means to accomplish these aspects of our mission. The idea behind the AIQ is relatively simple: to replicate through systematically gathered feedback the kinds of information that are currently obtainable only through ad hoc, person-to-person phone calls during the arbitrator selection process. More, and more accurate, information about how arbitrators decide cases will empower parties, counsel, and institutions to make better informed choices in selecting arbitrators. It will also reduce information asymmetries that undermine the fairness of arbitrator appointments and facilitate greater diversity by allowing newer arbitrators meaningful opportunities to establish reputations based on their actual performance.
Challenges in Creating the AIQ
To effectively replicate the essential characteristics of telephone research, we faced tremendous challenges in developing both the content of the AIQ and a strategy for implementing it. For starters, when you conduct telephone research about arbitrators, you can ask case-specific questions and follow up questions to fill in details. By contrast, the AIQ must cover all the topics that may be relevant in any particular case and anticipate potentially relevant follow up questions.
Meanwhile, when you make telephone inquiries, you know the identity of the person on the other side of the line and can assess the quality of responses in light of your confidence in that person’s experience and judgment. For the AIQ, we have to develop alternative means for determining relevance and ensuring confidence in the quality of responses, while still maintaining the confidentiality of the responder and of identifying details of the case.
Another challenge in developing the AIQ is preventing what we might call “The Disgruntled Losing Party Problem.” The concern is that, instead of providing fair and objective feedback, disappointed parties might misuse a survey to exact revenge against arbitrators who rendered an unfavorable award. To avoid this problem, our AIQ has to include control questions and other mechanisms to reduce the potential for unfair or inaccurate responses.
Ultimately, all these challenges must be met, but the questionnaire can’t be too long! Meeting these various challenges forced us to think and rethink details of the AIQ, and find innovative solutions to protect confidentiality while assuring quality.
To meet these challenges, we worked for months refining and redrafting questions in consultation with the Penn State Survey Research Center and other experts who are trained in empirical research methodologies. We then extensively pilot tested the AIQ with advisors and friends of AI, and we previewed the AIQ to select groups of in-house counsel, arbitrators, law firms, and representatives from arbitral institutions. The result is a draft version of the AIQ, which is now posted for public comment. Through this process, we hope to get additional feedback from around the globe so we can further refine and improve the AIQ.
Overview of the AIQ
The AIQ is divided into two phases. The first Phase focuses on general background about the case, and can be completed by anyone who has access to the arbitral award or case file. Key data from Phase I will then be prefilled into in the questions in Phase II. For example, the arbitrators’ names will be input in Phase I, along with key dates (like the date the request for arbitration was filed, the date the proceedings were closed, and the date the award was rendered). These data then will be incorporated into the questions in Phase II, which seek more evaluative and analytical feedback and are designed to be completed by an attorney or party involved with the case.
There are several reasons we divided the AIQ into 2 phases. First, it soon became clear that we could not ask all the essential questions we needed to ask and still have a questionnaire that would not be unduly burdensome. Now, each phase can be completed in 15 minutes or less.
Second, because Phase I seeks only objective information derived from awards, it can be used not only to gather information an ongoing basis, but also to extract data from past awards collected by AI. This data from past awards will provide a valuable starting point for analytics about arbitrator decision making.
Give Your Feedback on the AIQ
We need you! Come visit the AI website, where a static presentation version of the AIQ is posted along with annotation software that will enable you and others in the international arbitration community to provide us with interactive comments on the AIQ. Is there any question or response that is unclear? Are we missing a question you think should be included? Or are there questions you think are superfluous, unnecessary, or simply unhelpful?
Our website features Phase II for comment because it asks the most nuanced and complex questions—the questions on which your input will be most valuable. If you only have time to review one of the AIQ’s phases, Phase II is it! But we also welcome your feedback on Phase I, which is available here.
You can also find additional information about the AIQ in our answers to Frequently Asked Questions, which are available here.
Your feedback and input will help us finalize our AIQ before we formally launch it later this summer. In addition to annotating the AIQ on our website, you can also submit comments or questions directly to catherine.rogers@arbitratorintelligence.org.
Conclusion
International arbitration is no longer a cozy little practice among an elite group of insiders who can exchange information by telephone. We now see over 10,000 international arbitration cases annually (just among the major institutions), and thousands of arbitrators who regularly sit in those cases. The telephone is simply not a good means for keeping up with the scope of information implicated by these numbers. The result, unsurprisingly, is that even the largest firms and parties report finding themselves forced to consider or appear before arbitrators with whom they are unfamiliar and about whom they have trouble finding information.
The time has come for a technological and informational upgrade to the entire international arbitration regime. Parties, counsel, arbitral institutions, and even arbitrators need a more reliable, neutral, data-driven resource for sharing information about arbitrators and their decisional history. As an academic-based enterprise, Arbitrator Intelligence is uniquely positioned to meet these challenges on behalf of all stakeholders in the international arbitration community.
________________________
To make sure you do not miss out on regular updates from the Kluwer Arbitration Blog, please subscribe here. To submit a proposal for a blog post, please consult our Editorial Guidelines.
