Predicting Appellate Outcomes in the 5th Circuit

Litigation forecasting is big business. Law firms, corporations, and litigation funders aim to harness any information and data that can inform their litigation strategy and contribute to a thorough assessment of risk.

But which of the many variables in all of that data is useful in predicting litigation outcomes? Bloomberg Law’s Litigation Analytics includes biographical and historical data for federal judges. An examination of the data for judges on the U.S. Court of Appeals for the Fifth Circuit can offer attorneys and litigation finance professionals some interesting insights into appellate outcomes and the likelihood a judge’s decision will be wholly affirmed by the court of appeals (the judge’s “affirmance rate”). Although this modeling can help narrow the odds, the analysis was unable to explain the vast majority (80%) of the variance in affirmance rates or the reasons underlying any individual outcome. Applying regression modeling to thousands of appellate outcomes demonstrates that, in the aggregate, three factors together can help explain roughly 20% of the variance in affirmance rates between federal court judges in the Fifth Circuit:

•  the district judge’s age,
•  the job held immediately before ascending to the federal bench, and
•  the political party of the president who appointed the judge.

Put another way, the Fifth Circuit—which has a relatively high case load and a reputation as a politically conservative court—is slightly more likely to reverse decisions issued by older district court judges appointed by Democrats and who previously presided in state trial or appellate courts.

Methodology

Bloomberg Law has a collection of over 13.5 million court opinions in its database, and Appeal Outcome analytics are based on opinions authored by each federal district court judge and an analysis of their subsequent appellate history. A review of 6,615 appellate outcomes from 104 active Fifth Circuit district court judges captured by Bloomberg Law’s Judicial Analytics, indicates that, on average, district judges in the Fifth Circuit have their rulings affirmed 75.7% of the time. The modeling sought to identify variables that help predict whether judges would have affirmance rates higher or lower than average. To explain any differences, or “variance,” among judges’ affirmance rates, the following data was analyzed:

•  Background information obtained from the Federal Judicial Center, including:
•  The judge’s year of birth;
•  The law school where the judge obtained a JD;
•  The President who appointed the judge and that President’s political party;
•  The year the judge was appointed to the federal bench; and
•  The position the judge held prior to his or her appointment.
•  The ABA ranking each judge obtained from the American Bar Association’s Standing Committee on the Federal Judiciary.
•  2018 law school rankings from U.S. News and World Report.

This data generated a number of categorical (e.g. appointing President, prior position) and numeric (e.g. age, law school ranking) variables associated with each judge. Analyses of variance and numerous linear regression models were applied to determine whether there were any statistically significant relationships between these variables (using a 0.05 significance level) and what combination of these variables can best help explain the difference in affirmance rates among district court judges.

Age is More Than Just a Number

The most prevalent relationship uncovered by the analysis is a negative relationship between age and affirmance rate: Older judges are reversed more often.

The relationship is statistically significant: There is only a one-tenth of one percent (0.1%) chance that age and affirmance rate are unrelated. This relationship alone can help explain 10% of the total variance in affirmance rates between judges.

The relationship between age and affirmance rate appears in other ways, too. For example, there is a statistically significant difference in affirmance rates between judges who have assumed senior status, who have an average affirmance rate of 72.8%, and judges who have not, who have an average affirmance rate roughly five percentage points higher. Similarly, a statistically significant, negative relationship exists when it comes to years of federal judicial service: The longer a judge has served on the bench, the higher the likelihood of reversal. However when the impact of age is factored out through regression modeling, senior status and years of judicial service are no longer relevant given their close ties with age.

The negative relationship between age and affirmance rate could be due to a number of factors. Although the analysis can confirm that the relationship doesexist, it cannot explain why it exists. Due to the statistical strength of the relationship between age and affirmance rate and its predictive power, however, age served as the backbone of the regression analysis. Other variables were then added to increase the model’s accuracy.

The Impact of a Prior Job

A statistically significant relationship also exists between affirmance rate and the job the judge held prior to being appointed to the federal bench. Using data provided by the Federal Judicial Center, the 104 district judges were categorized based on the job the judge held before being appointed to the federal bench. The different job categories included:

•  private practitioner (53);
•  state court judge (30);
•  U.S. Magistrate Judge (10);
•  former prosecutor (5);
•  “other” (5); and
•  bankruptcy judge (1).

Of these, former state court judges had an average affirmance rate of 71.1%—4.6 percentage points lower than the group mean and the lowest amongst all judges measured on this variable. Former prosecutors, both state and federal, had a mean affirmance rate of 73.4%, slightly below the group mean of 75.7%. The average affirmance rates for private practitioners and former U.S. Magistrates, 77.4% and 78.4% respectively, tended to exceed the overall group mean, and lawyers in the “other” category, which includes former public interest lawyers and municipal attorneys, among others, had an average affirmance rate significantly higher, at 87.1%. There were no appellate outcomes reported by Bloomberg Law’s Judicial Analytics for the bankruptcy judge.

Former state trial or appellate judges are less likely to be affirmed. As can be seen by reference to the line in the scatterplot of age below, the majority of purple dots—former state judges—fall short of the expected affirmance rate based on their age.

Importantly, the judge’s prior job remained relevant even after regression modeling controlled for the effect of other variables. When prior job is added to a regression model that accounts for age, both variables retain their statistical significance and the proportion of the variance in affirmance rates among Fifth Circuit district judges that can be explained jumps from 10% to 17%.

The Power of Politics

Politics inform the final piece of the model. The Fifth Circuit is generally perceived as a politically conservative court, and the numbers help justify this perception. Republican presidents appointed 18 out of 26 of the Circuit’s appellate judges (69%) and 66 out of 104 (63%) of its district judges. The average affirmance rate of judges appointed by Republican presidents is slightly (a little over 1 percentage point) higher than the average affirmance rate for judges appointed by Democrats, but this variable on its own is not significant.

Although these differences may not seem substantial, when combined with age and prior job the political party of the appointing president becomes statistically significant and improves the ability to explain the variance in affirmance rates. The fact that judges who are appointed by Republicans are slightly more likely to be affirmed may seem unsurprising given the Fifth Circuit’s conservative reputation. Adding this variable to a model that accounts for age and prior job adds another 3% of explanatory power. So it has a significantly smaller impact on the ability to explain differences in affirmance rates than either age or prior job.

Nevertheless, a linear regression that accounts for these three variables can explain 20% of the variance between affirmance rates and is the most predictive model generated.

Trivial Pursuits

Statistical analysis also demonstrates that a number of seemingly-relevant inputs have no relationship with Fifth Circuit affirmance rates.

The rating each judge received from the majority of the American Bar Association’s Standing Committee on the Federal Judiciary is one such variable. Of the 104 judges surveyed, the 44 jurists who earned a majority “Well Qualified” rating had an average affirmance rate of 76.6%, the 41 who received a majority “Qualified” rating had an average rate of 76.2%, and the 19 who were appointed before the 101st Congress (the earliest online records), and therefore received “no rating” for the purposes of our analysis, had an average rate of 73.6% percent. Although there is some variability between these groups, regression modeling indicates ABA approval ratings only complicate the ability to predict appellate outcomes.

So too with law schools. Despite some theories that the best law schools make the best lawyers, the 2018 U.S. News & World Report ranking of a judge’s law school cannot help predict his or her affirmance rate in any meaningful manner. Although there are slight differences between the affirmance rates of judges when the U.S. News ranking of their law schools is taken into account, these differences are not statistically significant. Moreover, the U.S. News ranking of law schools does not materially improve the predictive abilities of linear regression models when combined with other variables.

Conclusion

Eighty percent of the variance in affirmance rates in the Fifth Circuit cannot be explained by a regression model of measurable data and presumably is related to the merits of each case. Even though the model can only explain 20% of the variance in district court affirmance rates, lawyers and litigation finance professionals interested in predicting appellate outcomes now have some additional data to help assess risk. For litigation funders in particular, that 20% could have an impact when assessing a risk profile for multiple matters. Future analyses will explore whether these data points, or others, indicate statistically significant predictive factors in each of the other federal appellate circuits.

— With assistance from Senior Data Analytics Manager Tom Shen.

To read more about AI-powered litigation, click here.