openpolitics.com

2020 update: The specific flaws of Roland Fryer’s paper have now been characterized in two studies (by other scholars, not myself). Knox, Lowe, and Mummolo (2019) reanalyze Fryer’s data to find it understates racial biases. Ross, Winterhalder, and McElreath (2018) do something similar through a statistical simulation.

 

Roland Fryer, an economics professor at Harvard University, recently published a working paper at NBER on the topic of racial bias in police use of force and police shootings. The paper gained substantial media attention – a write-up of it became the top viewed article on the New York Times website. The most notable part of the study was its finding that there was no evidence of racial bias in police shootings, which Fryer called “the most surprising result of [his] career”. In his analysis of shootings in Houston, Texas, black and Hispanic people were no more likely (and perhaps even less likely) to be shot relative to whites.

Fryer’s analysis is highly flawed, however. It suffers from major theoretical and methodological errors, and he has communicated the results to news media in a way that is misleading. While there have long been problems with the quality of police shootings data, there is still plenty of evidence to support a pattern of systematic, racially discriminatory use of force against black people in the United States.

Breaking down the analysis of police shootings in Houston

There should be no argument that black and Latino people in Houston are much more likely to be shot by police compared to whites. I looked at the same Houston police shooting dataset as Fryer for the years 2005-2015, which I supplemented with census data, and found that black people were over 5 times as likely to be shot relative to whites. Latinos were roughly twice as likely to be shot versus whites.

Fryer was not comparing rates of police shootings by race, however. Instead, his research asked whether these racial differences were the result of “racial bias” rather than merely “statistical discrimination”. Both terms have specific meanings in economics. Statistical discrimination occurs when an individual or institution treats people differently based on racial stereotypes that ‘truly’ reflect the average behavior of a racial group. For instance, if a city’s black drivers are 50% more likely to possess drugs than white drivers, and police officers are 50% more likely to pull over black drivers, economic theory would hold that this discriminatory policing is rational. If, however, police were to pull over black drivers at a rate that disproportionately exceeded their likelihood of drug possession, that would be an irrational behavior representing individual or institutional bias.

Once explained, it is possible to find the idea of “statistical discrimination” just as abhorrent as “racial bias”. One could point out that the drug laws police enforce were passed with racially discriminatory intent, that collectively punishing black people based on “average behavior” is wrong, or that – as a self-fulfilling prophecy – bias can turn into statistical discrimination (if black people’s cars are searched more thoroughly, for instance, it will appear that their rates of drug possession are higher). At the same time, studies assessing the extent of racial bias above and beyond statistical discrimination have been able to secure legal victories for civil rights. An analysis of stop-and-frisk data by Jeffrey Fagan, which found evidence racial bias, was an important part of the court case against the NYPD, and helped secure an injunction against the policy.

Even if one accepts the logic of statistical discrimination versus racial bias, it is an inappropriate choice for a study of police shootings. The method that Fryer employs has, for the most part, been used to study traffic stops and stop-and-frisk practices. In those cases, economic theory holds that police want to maximize the number of arrests for the possession of contraband (such as drugs or weapons) while expending the fewest resources. If they are acting in the most cost-efficient, rational manner, the officers may use racial stereotypes to increase the arrest rate per stop. This theory completely falls apart for police shootings, however, because officers are not trying to rationally maximize the number of shootings. The theory that is supposed to be informing Fryer’s choice of methods is therefore not applicable to this case. He seems somewhat aware of this issue. In his interview with the New York Times, he attributes his ‘surprising’ finding to an issue of “costs, legal and psychological” that happen following a shooting. In what is perhaps a case of cognitive dissonance, he seems to not have reflected on whether the question of cost renders his choice of methods invalid.

Economic theory aside, there is an even more fundamental problem with the Houston police shooting analysis. In a typical study, a researcher will start with a previously defined population where each individual is at risk of a particular outcome. For instance, a population of drivers stopped by police can have one of two outcomes: they can be arrested, or they can be sent on their way. Instead of following this standard approach, Fryer constructs a fictitious population of people who are shot by police and people who are arrested. The problem here is that these two groups (those shot and those arrested) are, in all likelihood, systematically different from one another in ways that cannot be controlled for statistically (UPenn Professor Uri Simonsohn expands on this point here). Fryer acknowledges this limitation in a brief footnote, but understates just how problematic it is. Properly interpreted, the actual result from Fryer’s analysis is that the racial disparity in arrest rates is larger than the racial disparity in police shootings. This is an unsurprising finding, and proves neither a lack of bias nor a lack of systematic discrimination.

Even if the difference in the arrest vs. shooting groups could be accounted for, Fryer tries to control for these differences using variables in police reports, such as if the suspect was described as ‘violently resisting arrest’. There is reason to believe that these police reports themselves are racially biased. An investigation of people charged with assaulting a police officer in Washington, DC found that this charge was applied disproportionately towards black residents even for situations in which no assault actually occurred. This was partly due to an overly broad definition of assault against police in DC law, but the principle – that police are likely to describe black civilians as more threatening – is applicable to other jurisdictions.

I’ll also briefly note that there was another analysis, using data from multiple cities, that looked at racial differences in whether or not civilians attacked officers before they were shot. Fryer himself downplays the credibility of this analysis, because it relied on reports from police who had every incentive to misrepresent the order of events.

Racial inequality in police shootings

Fryer’s study is far from the first to investigate racial bias or discrimination in police shootings. A number of studies have placed officers in shooting simulators, and most have shown a greater propensity for shooting black civilians relative to whites. Other research has found that cities with black mayors and city councilors have lower rates of police shootings than would otherwise be expected. A recent analysis of national data showed wide variation in racial disparities for police shooting rates between counties, and these differences were not associated with racial differences in crime rates. This is just a small sample of the dozens of studies on police killings published since the 1950s, most of which suggests that racial bias is indeed a problem.

It is a failure of journalism that the New York Times heavily promoted this study without seeking critical perspectives from experts in the field. Fryer makes basic methodological errors, overstates the quality of his results, and casually uses the term “racial bias” in a way that is nearly guaranteed to be misinterpreted by anyone who isn’t an economist.