Evaluating Countermeasures: Before–After and Cross-Sectional Studies

Evaluating Countermeasures: Before–After and Cross-Sectional Studies

Estimate the effect against a counterfactual

A treatment evaluation asks what would have happened at the treated sites without the treatment. That unobserved outcome is the counterfactual. Before–after and cross-sectional methods estimate it in different ways. Neither a favorable after-period count nor a plausible mechanism alone establishes a causal treatment effect. The evaluation plan must address exposure, time trends, site selection, reporting, and other changes that could explain the result.

The FHWA Guide to Developing Quality Crash Modification Factors discusses study methods and quality. The CMF Clearinghouse records methods, applicability, and uncertainty for published effects. Use those details when interpreting a claimed percentage reduction. The most credible method depends on the available data and how sites were selected, not on the method's name alone.

Simple before–after: useful description, weak counterfactual

A simple comparison divides observed after crashes by observed before crashes, using comparable periods. If a site has 20 before crashes and 12 after crashes, the raw ratio is 0.60, describing a 40% reduction in observed counts. It does not account for a traffic increase, regional trend, altered reporting threshold, concurrent project, or regression to the mean.

Regression to the mean is especially important when sites were chosen because of unusually high recent counts. Part of the later decrease may have occurred without intervention. Equal before and after durations address one source of imbalance, but do not remove this selection effect. Similarly, dividing both counts by traffic volume does not necessarily solve the problem: safety may have a nonlinear relationship with exposure, and other conditions still change.

Use a simple comparison to report what was observed and identify questions. Avoid calling its ratio an unbiased CMF solely because the computation is straightforward. Short periods and very small counts can also generate unstable percentages; a reduction from two crashes to one is a 50% observed change with substantial uncertainty.

Comparison-group before–after studies

A comparison group estimates background changes affecting untreated sites. Suppose crashes at treated sites decrease while similar untreated sites also decrease. The treatment's net effect is smaller than the treated sites' raw decline if the comparison trend is applicable. Choose comparison sites with relevant similarities in facility type, exposure, environment, and time trends, and ensure they did not receive the treatment or experience spillover effects.

A poorly chosen group can introduce bias. Selecting treated sites for high recent crashes but comparison sites without comparable selection can leave regression-to-the-mean problems. Different land-use changes or reporting systems can invalidate a common-trend assumption. Explain the selection method and inspect trends rather than treating “comparison group included” as proof of study quality.

Empirical Bayes before–after studies

An empirical Bayes (EB) approach combines observed before crashes with an SPF prediction to estimate the expected before-period frequency. It then projects that expectation to the after period under without-treatment conditions, considering the adopted method's exposure and time adjustments. Compare observed after crashes with this projected counterfactual. EB is a strong method for addressing regression to the mean when suitable models and data are available, but it does not automatically eliminate every bias.

Consider an illustrative site with 20 observed before crashes, an SPF prediction of 10, and a supplied overdispersion parameter $k=0.1$. Using the simplified consistent-period convention $w=1/(1+kP)$ gives $w=0.5$. The EB expected before count is $wP+(1-w)O=15$. Assume the valid without-treatment projection multiplier is 1.2, giving 18 expected after crashes. With 12 observed after crashes, the approximate effect ratio is $12/18=0.667$, or a 33.3% reduction. The naive observed-count comparison suggested 40%.

This is a teaching calculation, not a complete publication-quality EB estimator. Formal procedures include variance and bias adjustments, pooled sites where appropriate, and uncertainty calculations. The multiplier is supplied here; it is not automatically equal to an AADT growth ratio for every SPF. A poor model, incorrect crash-category match, or unmeasured concurrent treatment can compromise the result despite using EB.

Cross-sectional studies

Cross-sectional studies compare different sites with and without a feature during a study period. A regression model can adjust for measured differences in exposure, geometry, and other covariates. For example, analysts might compare intersections with different control types while adjusting for traffic volumes and configuration.

The challenge is confounding: the feature may have been installed precisely because a site had unusual risks. A coefficient can reflect those selection differences rather than a pure treatment effect. Unmeasured pedestrian activity, land use, or enforcement can also matter. Examine functional form, covariate quality, matching or other adjustment methods, and the range of conditions represented. A cross-sectional association is not automatically transferable to a before–after project decision.

Study designs estimate different comparisons

• Simple before–after: compare observed counts across periods.
• Comparison-group: use untreated trends to estimate background changes.
• EB before–after: combine history and models, then project the counterfactual.
• Cross-sectional: compare different sites while addressing confounding.

Interpret uncertainty and report limits

Report the outcome category, sites, periods, treatment implementation, model, effect estimate, and uncertainty. A point estimate below 1.0 can be compatible with no reduction if its interval includes 1.0. An imprecise result does not prove the treatment is ineffective; it may indicate insufficient information. Conversely, statistical significance does not make a small benefit practically important or erase study bias.

Randomization, when feasible and appropriate, can strengthen causal inference; ethical, operational, or funding constraints may limit its use in roadway projects. No design guarantees perfect inference. The sound conclusion explains what the evidence supports, which assumptions are needed, and whether the result applies to the proposed site and crash types.