Predicting Appellate Outcomes in the 2nd Circuit

Litigation forecasting is big business. Law firms, corporations, and litigation funders aim to harness any information and data that can inform their litigation strategy and contribute to a thorough assessment of risk. Bloomberg Law’s Judicial Analytics can help. An analysis of appellate outcomes for district judges in the U.S. Court of Appeals for the Second Circuit shows that—more than any other factor—age impacts the likelihood that a trial court judge’s ruling will be reversed. Also, the job a judge held before accepting his or her commission can help predict the likelihood of reversal on appeal by the Second Circuit, with prior prosecutors tending to get better outcomes than others.

An analysis of biographical and historical data for each district court judge in the Second Circuit may help predict the likelihood that a judge’s decision will be wholly affirmed by the court of appeals (the judge’s “affirmance rate”). Although it cannot account for any individual case, applying weighted regression modeling to thousands of appellate outcomes demonstrates that, in the aggregate, the district judge’s age and the job held immediately before ascending to the bench can help explain roughly 20% of the variance in affirmance rates between Second Circuit district judges. Put another way, the Second Circuit is slightly more likely to affirm decisions issued by younger district court judges who are former prosecutors. Although modeling can help narrow the odds, even the best model cannot explain the vast majority (80%) of the variance in affirmance rates or the reasons underlying any individual outcome (this is not uncommon in the social sciences).

Methodology

A review of 7,970 appellate outcomes from 97 Second Circuit district court judges as reported by Bloomberg Law’s Judicial Analytics indicates that, on average, district judges in the Second Circuit have their rulings affirmed 75.6% of the time. The modeling sought to identify variables that help predict whether judges would have affirmance rates higher or lower than average. Simply put, the difference among judges’ affirmance rates is called “variance” and the goal was to explain as much of it as possible. To do so, the following data was analyzed:

Background information obtained from the Federal Judicial Center, including:

• The judge’s year of birth;
• The law school where the judge obtained a JD;
• The President who appointed the judge and that President’s political party;
• The year the judge was appointed to the federal bench; and
• The position the judge held prior to appointment. The ABA ranking each judge obtained from the American Bar Association’s Standing Committee on the Federal Judiciary. 2018 law school rankings from U.S. News and World Report.

This data generated a number of categorical (e.g. appointing President, prior position) and ordinal (e.g. age, law school ranking) variables associated with each judge. Numerous linear regression models were applied to determine whether there were any statistically significant relationships between these variables (using a 0.05 significance level) and what combination of these variables can best help explain the difference in affirmance rates among district court judges. The model was also weighted to take into account the number of appellate outcomes available for each district judge.

Age Strikes Again

In the Second Circuit, the most prevalent relationship uncovered is a negative relationship between age and affirmance rate. In other words, older judges are reversed more often.

This relationship was statistically significant and—standing alone—can help explain more than 10% of the total variance in affirmance rates between judges.

The relationship between age and affirmance rate also appears in other ways. For example, there is a statistically significant difference in affirmance rates between judges who have assumed senior status, who have an average affirmance rate of 73.8%, and judges who have not, who have an average affirmance rate almost four percentage points higher. A statistically significant, negative relationship exists when it comes to years of federal judicial service: the longer a judge has served on the bench, the higher the likelihood of reversal. However, when the impact of age is factored out of these relationships through regression modeling, its relevance falls away.

It bears repeating that the negative relationship between age and affirmance rate could be due to a number of factors. Although the analysis can confirm that the relationship does exist, it cannot explain why it exists. Because of the statistical strength of the relationship between age and affirmance rate and its predictive power, age served as the backbone of the regression analysis. Other variables were then combined with it in an effort to increase the model’s accuracy.

Judges Who Were Prosecutors Fare Better

A statistically significant relationship also exists between affirmance rate and the job the judge held prior to accepting an appointment to the federal bench. Using data provided by the Federal Judicial Center, the 97 district judges were categorized based on the job the judge held before being appointed to the federal bench.

The different job categories included:

• private practitioners (41);
• state court judges (21);
• U.S. Magistrate Judges (7);
• former prosecutors (16);
• “others” (11); and
• a sole bankruptcy judge.

Although most of these categories had no relationship with the likelihood a judge would be affirmed on appeal, one category had an appreciable effect in improving the predictive power of the model. Judges who had been prosecutors immediately before ascending to the federal bench had a mean affirmance rate of 80%, more than5 percentage points higher than the average for everyone else (74.71%).

The relationship was statistically significant and the judge’s prior job remained relevant even after regression modeling controlled for the effect of other variables. When prior job is added to a regression model that accounts for age, both variables retain their statistical significance and the proportion of the variance in
affirmance rates among Second Circuit district judges that can be explained increases from 11% to 13%.

Weighting Results

Weighting regression modeling based on the number of appellate outcomes available in Bloomberg Law’s Judicial Analytics also helped to improve the model’s accuracy. Results for judges with more than 100 reported appellate outcomes were given the highest weight. Results for judges with less than 100 reported appellate outcomes were weighted on a sliding scale, with the weight increasing in direct proportion to the number of reported appellate outcomes. This increased the predictive power of the model significantly, and the amount of variance it was able to explain rose from 13% to 20%.

In sum, a weighted linear regression that takes into account a judge’s age and prior job can explain 20% of the variance between affirmance rates among Second Circuit judges and is the most predictive model generated.

Trivial Pursuits

Statistical analysis also demonstrates that a number of seemingly-relevant inputs have no relationship with Second Circuit affirmance rates. The rating each judge received from the American Bar Association’s Standing Committee on the Federal Judiciary has no appreciable impact on the model’s ability to predict Second Circuit affirmance rates.

Law school ranking is similarly overrated. The 2019 U.S. News & World Report ranking of a judge’s law school cannot help predict his or her affirmance rate in any meaningful manner.

Although the data demonstrates a statistically significant relationship between attending a “top 20” law school and affirmance rate, this relationship disappears when the effects of other variables are removed. Simply put, the U.S. News ranking of law schools does not materially improve the predictive abilities of linear regression
models.

The Circuits Compared

A prior analysis of the appellate outcomes for federal district judges in the Fifth Circuit indicated that older judges who were appointed by Democratic presidents and previously presided in state trial and appellate courts were more likely to be reversed by the Fifth Circuit Court of Appeals.

There are notable differences between the Second and Fifth Circuit district court judges. To start, compared to the Fifth Circuit, the cadre of Second Circuit district judges includes three times more judges who held prosecutorial positions before their ascension to the district court level. The Fifth Circuit, on the other hand,
includes a higher number of former private practitioners and state court judges.

Another key difference between these circuits is their political makeup, as determined by the political party of the appointing president. More judges in the Second Circuit were appointed under presidents Clinton and Obama, whereas more judges in the Fifth Circuit were appointed by presidents George W. Bush and Trump.

Finally, there is a difference in the age distribution between Second and Fifth Circuit district judges. District judges in the Second Circuit are older, with an average age that’s 3.78 years older than the age of the average district judge in the Fifth Circuit. Moreover, in the Second Circuit, slightly more than half of the judges have
assumed senior status, compared to roughly one third in the Fifth.

Despite these differences, the two circuits share one important similarity: their average affirmance rates. Taking into account 14,585 appellate outcomes, there is only a .1% difference between the average affirmance rate of Second Circuit district judges (75.6%) and those who sit in the Fifth Circuit (75.7%). The overall likelihood of reversal in these two circuits is remarkably consistent.

Conclusion

Once again, regression modeling using background variables to predict affirmance rates can only explain 20% of the variance between judges on this variable. Presumably, the remaining 80% is related to the merits of each case. Even though the model can only explain 20% of the variance in district court affirmance rates, lawyers and litigation finance professionals interested in predicting appellate outcomes now have some additional data to help assess risk. For litigation funders in particular, that 20% could have an impact when assessing a risk profile for multiple matters. Future analyses will explore whether these data points, or others, indicate statistically significant predictive factors in each of the other federal appellate circuits.

— With assistance from Senior Data Analytics Manager Tom Shen.

To read more about AI-powered litigation, click here.