Case study on evaluating statistical tools that predict recidivism.

View the Project on GitHub AllenDowney/RecidivismCaseStudy

Recidivism Case Study

This case study is based on two articles that were published in 2016:

Both articles are about COMPAS, a statistical tool used in the justice system to assign defendants a “risk score” that is intended to reflect the risk that they will commit another crime if released.

The ProPublica article evaluates COMPAS as a binary classifier and compares its error rates for black and white defendants. It concludes that COMPAS is unfair to black defendants because they are more likely to be misclassified as high risk.

In response, the Washington Post article shows that COMPAS has the same predictive value for black and white defendants. And they explain that the test cannot have the same predictive value and the same error rates at the same time.

The purpose of this case study is to understand these conflicting claims, to learn about classification algorithms and the metrics we use to evaluate them, and to think about fairness and the ethics of data science.

The notebooks

These three notebooks are intended to support a module in a data science class that engages students in the context and ethical challenges of machine learning.


I used these notebooks for a module of my Data Science class at Olin College.

Over the course of three class sessions, I presented these slides and led a discussion with students. This happened in Spring 2020 when classes were run remotely, so the discussions were not as effective as they could have been. For next time I hope to develop a richer set of discussion questions.

Additional notebooks

This repository contains three additional notebooks with additional explorations that you might be interested in. They are not essential to understand the issues, and they are less complete than the first three notebook.

I include these notebook in part to resist the temptation to hide my development process. I worked on this case study on and off over several years. I explored a lot of things and took a lot of wrong turns. It took me a long time to find the story, get it organized, and strike a balance between two conflicting goals: maintaining the scientific detachment that lets us tackle difficult topics while keeping sight of the context, the people, and the human consequences.

I hope these materials will be engaging and informative for readers, and useful for teaching and learning the ethical practice of data science.