Panel review finds FMCSA Safety Measurement System ‘defendable’ but could be improved

The Federal Motor Carrier Safety Administration’s Safety Measurement System is “sound,” but the agency should consider a “more statistically principled approach” for evaluating the safety of commercial motor vehicle carriers, the National Academies of Sciences concluded after an 18-month review of the Compliance, Safety, Accountability program.

The review was mandated by the Fixing America’s Surface Transportation (FAST) Act of 2015.

FMCSA uses the CSA program to rank CMV carriers on how safely they operate, mainly measured by the amount of violations found during roadside inspections and the number of crashes a carrier experiences. Based on that, FMCSA then uses its Safety Measurement System to target carriers that may be at risk for future violations or crashes.

The trucking industry has been highly critical of the CSA program, claiming it uses “faulty methodology” – including not accounting for crashes that are not the fault of the CMV driver or using measures that don’t reflect a carrier’s efforts to improve its safety performance – to assess risk potential.

As a result, an NAS panel was tasked with “analyzing the ability of SMS measures to discriminate between low- and high-risk carriers, assess the public usage of SMS, review the data and methodology used to calculate the measures, and provide advice on additional data collection and safety assessment methodologies.”

The panel determined that “FMCSA has instead adopted a sensible, related approach based on prevention rather than prediction. … The current SMS implementation is defendable as being fair and not overtly biased against various types of carriers.”

However, the panel went on to say that “some features of SMS implementation can be improved upon, and some of the details of the implementation are ad hoc and not fully supported by empirical studies. Many of these details of implementation would be easily addressed if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem.”

Over the next two years, the panel states, FMCSA should develop an item response theory model, and, if it works well, replace its current program with the IRT model. According to the panel, the IRT model offers many advantages over SMS:

• The item discrimination parameters are estimated based on a combination of current observed data and expert opinion, and ultimately on data alone.
• It supports the direct estimation of variability of scores and ranks.
• It can enhance the transparency of the evaluation system.
• It can provide a basis with which to evaluate how data insufficiency could affect safety ratings of carriers.

The American Trucking Associations applauded the findings of the NAS review.

“This report has confirmed much of what we have said about the program for some time. The program, while a valuable enforcement tool, has significant shortcomings that must be addressed and we look forward to working with FMCSA to strengthen the program,” ATA President and CEO Chris Spear said in a press release.

Further, “We strongly agree with the National Academies’ conclusion that [the Compliance, Safety, Accountability program] should be based less on the ‘subject-matter expertise’ of enforcement and more on the empirically validated data,” Sean Garney, ATA’s director of safety policy, said in the release.

Rep. Eleanor Holmes Norton (D-DC), ranking member of the House Highways and Transit Subcommittee, has requested a hearing on the NAS review.