The CAT’s new Practice Direction on Expert Evidence provides very helpful guidance on what the CAT expects of experts. It covers a wide range of topics including how experts should work with each other and the Tribunal.
However, two aspects of the new guidance are likely to have a particularly significant practical impact. In short, it seems to me that (1) the Practice Direction will require economists to be much more transparent in documenting how they have built their models, and (2) it will also tend to mean that experts are kept more at arm’s length from the legal team.
Talking of experts, before unpacking those points can I record my thanks to Joseph Bell of Oxera for his invaluable comments on an earlier draft. All errors and opinions of course remain my own.
Background
I argued on this blog a couple of years ago that the CAT should give more guidance to experts. Up until now, the main way that the Tribunal has clarified its expectations has been by criticising experts in judgments when they veer too far off-piste. That is fine as far as it goes, but public criticism is a blunt tool.
In the absence of guidance, a lot has been left to the judgment of individual experts and legal teams, with the result that a range of approaches are taken to some quite basic issues (e.g. how much input should the legal team have into an expert report). It is reasonable to expect that different judges would also have different views on such things.
The new guidance clarifies a lot. But along the way, it makes two big calls and leaves some questions unanswered.
How the sausage is made
Many of us will have had the experience of reading an expert report, with its nice step-by-step descriptions of how the data and specifications were selected, culminating in a model which produces robust results, and wondering: but how many previous data and specification choices were tried and discarded before arriving at this neat final outcome?
That question is now out in the open. Paragraphs 17 and 18 of the Practice Direction say this:
“17. Where an expert carries out econometric analysis or other quantitative analysis involving mathematical manipulation of data, the Tribunal will expect the expert to justify their choice of model as well as the specifications of that model, to be prepared to answer questions on those points, and to be prepared to consider how alternative models and/or specifications might affect the results. The expert should also disclose any initial analysis that materially informed their subsequent choice of methodology, particularly if the earlier analysis produced a result contrary to the expert’s client’s interests.
18. Econometric or other analytical techniques, particularly tests and criteria used to decide between alternative models, that are identified in advance of any data being analysed are likely to be given more weight by the Tribunal than techniques selected only once their effect on the outcome can be observed.”
Those paragraphs are going to need very careful consideration, and the challenge will be in the details.
The starting point is clear enough: the Tribunal would like to know that experts have followed a clear and logical process to choose their methodology, and if that methodology is identified and explained in advance of being applied then all the better.
One does however have to acknowledge that judgments between different models or specifications can be highly complex. Economists often do not know the best way to analyse data until they actually have the data, and exploratory data analysis to inform methodological choices can be good practice.
Similarly, if a model produces irrational outcomes then working out why, and what needs to be modified, is not always straightforward. The kinds of judgments that need to be applied are not always capable of prior explanation (or at least, not within what the CAT would consider acceptable page limits).
Furthermore, the process may involve trial and error not because of any attempt to bias the outcome, but just because that can be part of the process to build a more accurate model. Indeed, it is normal for a model to be cleaned at the outset, looking for coding errors or problems caused by incomplete data. Even after those first stages there are many reasons why running different variations can be useful and entirely proper. Iteration does not always imply manipulation.
One point which is clear is that the more an expert can describe their methodology in advance, the better. To benefit from what is said in paragraph 18 of the PD, it may even be useful for the expert to produce a written description of their intended approach before applying it. However, in light of the points made above, that is going to require pragmatism as well as ambition. It is right to expect an expert to be able to say more, than “I intend to carry out a during and after regression”. On the other hand, it is probably not realistic to expect them to be able to identify in advance a sort of “full pre-specification” of their proposed model.
Experts are therefore going to need to reach for something between those extremes. That is likely to encompass a description of the principles and the process to be used in choosing their method. So for example:
- Not, ‘I will use these control variables,’ but, ‘this is how I will choose my control variables’
- Not, ‘this is going to be my functional form,’ but, ‘I will choose a functional form based on a series of sensitivities and seeing which performs best to post estimation tests’.
Experts are also going to need to think hard about the new requirement to “disclose any initial analysis that materially informed their subsequent choice of methodology, particularly if the earlier analysis produced a result contrary to the expert’s client’s interests.”
I do not think that can be read as applying only to the “choice of methodology” at a high level, such as the choice to use a regression analysis rather than some other methodology to test for overcharge. Rather, it could also cover more micro methodological choices, for example if data sets are dropped, or if model specifications are changed because a sensitivity analysis is failed. On the other hand, a sensible line will need to be drawn by the expert to distinguish between those sorts of amendments, as distinct from more minor refinements.
It is therefore going to be necessary for the expert to keep a record of the initial analyses, and to explain what they showed and how they fed into the eventual methodological choices. It is also important to balance that against the risk of the process being overwhelmed by too much disclosure of different model iterations. What is needed is a clear description of the significant developments.
The role of lawyers
The other big point is the following:
“25. Unless the Tribunal directs otherwise, the involvement of the parties’ legal teams in any expert report or joint expert statement should be limited to providing guidance as to the format of the report or joint statement and the issues to be addressed, and having sight of the draft report or statement to comment on the accuracy of any factual and legal points, as well as any typographical corrections or stylistic amendments to ensure comprehension of the document. The parties’ lawyers must not, however, seek to influence the substantive views expressed by the experts.”
As I have mentioned above, the lack of guidance up until now has meant that a variety of approaches have been taken to questions such as, how closely involved should lawyers be in the expert process? However, my own experience is that it is standard for lawyers and expert economists to work closely together. There are commonly meetings to discuss the case, including for the lawyers to understand the evolving expert analysis and the sort of evidence that the expert may need. I do not think the new PD prevents any of that, although it may well be that the process becomes more formalised and documented.
What about if, during such a discussion, a lawyer were to say: “Have you thought about this other factor which may be relevant to your economic analysis?” Or, “What if you used a different data series?” Or, “Have you tried using a different methodology?” Or, “Your market definition seems flawed to me because you’re not taking account of XYZ”. Or even, to take an example which would be hard to describe as a ‘legal’ or ‘factual’ point, “Those regression results do not seem reliable because XYZ”.
Whether any of those comments are permissible may depend on the context. Clearly if the lawyer’s intention is to influence the substantive views expressed in the expert report then they are impermissible. If the intention is to ensure that all relevant issues have been considered and investigated then in principle that the questions may be permissible, but there can be a fine line between those two things.
It also strikes me that comments of the kind described above may be more likely to be appropriate during the early phases of litigation, when issues are being identified, rather than following the provision of a draft report which is the stage of the process which paragraph 25 of the PD is more directly aimed at. It may be that more “front-loading” is appropriate, so that issues can be identified at an earlier stage in the process rather than after a draft report has been produced. It may also be that some of the dangers can be navigated by drafting even more comprehensive letters of instruction and/or by providing formal instructions earlier in the process, rather than (as sometimes now happens) them being provided in an informal way in the early stages and then only formalised just at the point when the expert report is being finalised.
Consequences
The new Practice Direction is going to need careful thought, including in cases currently before the Tribunal where teams may need to review what has happened to date. At an event I attended yesterday someone suggested that parties should agree to wipe the slate clean as regards what has happened to date, to avoid mutually assured destruction. They were joking (I think), but it is certainly true that a period of adjustment should be expected.
Going forward, the main consequence of the PD will be to bring greater transparency in the expert process, as well as consistency. However, for reasons which I have touched on above, it is also likely to result in significant changes to working practices.
I expect that legal teams will need to be much more cautious about their interactions with experts, keeping them more at arm’s length. Experts are likely to have less input into the case, and lawyers less input into the expert work. And experts are going to need to find a way to satisfy the need for a detailed description of their process, whilst also keeping in mind the Tribunal’s frequent requests for more concise reports.
