Integrity of ‘independent’ state traffic stop analysis questioned by researcher’s opposite conclusions

A Northeastern University professor who found a persistent and widespread discrepancy in how often Massachusetts police ticket white drivers and drivers of color is questioning the analysis of a report released by the state Executive Office of Public Safety and Security.

That report, made public Tuesday (Feb. 27), included an analysis of the same data and found “no support … for a pattern of racial disparity in total Massachusetts stops.”

“I think, clearly, there are some pretty large differences in terms of mainline findings from my report and the EOPSS report,” says Matthew B. Ross, an associate professor of public policy and economics at Northeastern.

“At best, one could imagine that those differences are driven by the fact that there were some relatively non-standard decisions made in the analysis process,” he says.

Ross questions if those decisions were made to shape the overarching findings in the report.

“At worst, given that EOPSS is really working in partnership with the research team, I think it raises questions about the integrity of the entire process,” he says.

An EOPSS spokesperson did not directly answer questions about the discrepancy and methodology, but referred to the organization’s report and its executive summary, “which outlines in robust detail the methodology and limitations of the two-year analysis.”

Tuesday’s report analyzes statewide traffic stop data collected by the Registry of Motor Vehicles from 2021 and 2022: examining demographics of stopped motorists including age, gender and race; reviewing the date and time and location of the offense and whether the vehicle was searched; and classifying whether the stop resulted in a warning, citation or arrest. The analysis involved a total of 1,270,129 traffic stops, which resulted in 1,358,720 citations.

The analysis was done by “independent researchers” at Salem and Worcester State universities whom EOPSS contracted, the report says.

The research applies much of the “Connecticut Model,” which Ross and colleagues developed in the early 2010s, and which has been used widely since across the country.

This model prominently uses a Veil of Darkness test—an approach that compares traffic stops made in daylight to stops made when it’s dark outside. The test is based on the assumption that traffic stops after dark should show no discrepancy between minority and white drivers because police cannot racially identify drivers when there is less daylight.

“The results of this test for all stops indicates that white motorists are more often stopped in daylight than in darkness whereas non-white motorists are less often stopped in daylight than in darkness,” the executive summary of Tuesday’s report reads. “For this analysis, no support was shown for a pattern of racial disparity in total Massachusetts stops.”

Ross, however, analyzed the traffic stop data from 2014 to 2022 and applied the Veil of Darkness test as part of an investigation by the U.S. Today Network. In November, Ross came to the opposite conclusion as Tuesday’s state report (as well as to a 2022 state report conducted using data from 2020 traffic stops)—finding “fairly large and persistent disparities” between how often Massachusetts police stop white drivers and, specifically, Black and Hispanic drivers.

So why the discrepancies?

Ross says that researchers contracted by the state did not uniformly follow best practices methodology established by the Connecticut Model.

“It does look like they’re trying to follow the best practice methodology, but if you actually get under the hood and look at some of the decisions that they made methodologically, they’re not,” Ross says. “I think the reason that they found the complete opposite finding is simply that their methodologically didn’t apply the test in the standard way that everybody else does it, and that there’s documented scholarly literature that says that you need to do it this way.”

Ross cited traffic stops involving light violations as an example.

Typically, these are excluded from the analysis because they are correlated with darkness (a broken tail light is more noticeable when it’s dark out) and socioeconomic status, Ross explained.

But Ross says the appendix of the state’s recent report, however, suggests that these stops are potentially included, potentially biasing the result.

The researchers also added a control for season of the year, Ross says, something that is “just not correct” and “nonsensical.”

“That is particularly problematic because in the Veil of Darkness, all of the variation that you’re using to identify potential discrimination is coming from seasonal changes in the timing of the sunset,” Ross says. “If you include controls for season of the year, you’re effectively nuking all of the variation in the variable that you’re using to identify discrimination.”

Ross again questioned the independence of the contracted researchers.

“It is presented as an independent analysis, but if you read the acknowledgments in both of the reports, it basically acknowledges that it was really done in partnership with EOPSS,” Ross says, adding that other states typically involve an advisory board including diverse stakeholders. “There’s an input process and the whole thing is transparent versus a policing agency like EOPSS handling the entire thing behind closed doors.”

Ross says he plans to present his concerns through written testimony at an upcoming public hearing on the report.

That being said, Ross did praise the recommendations section of the report as possibly reactive to some of the criticism that he and reporters at U.S. Today Network raised previously.

“They look like they were shaped by some of the things that we were saying critically,” Ross says. “I do think that those recommendations are really a step in the right direction.”

This story is republished courtesy of Northeastern Global News news.northeastern.edu.