This post is in dialogue with the recent papers “Syringe exchange programs and harm reduction: New evidence in the wake of the opioid epidemic” published in the Journal of Public Economics and “The Effects of Naloxone Access Laws on Opioid Abuse, Mortality, and Crime” published in the Journal of Law and Economics.
I’ve been meaning to write something about two papers by economists that undermine harm reduction narratives. Both papers are policy relevant, timely (in the sense that they relate to ongoing drug crises – the analysis period in both ends in 2016 and 2015), and contradict public health orthodoxy (and their scholarly literature, which is often empirically less sophisticated than economics). Both articles were controversial and attracted significant media coverage. Like the public health folks, I’m not inclined to update my beliefs about the effect of harm reduction services based on these papers, but I think my reasoning is only partially similar to theirs.
I hope this post differentiates itself from prior discourse by making some novel methodological points. I describe “feedback” between past outcomes and current treatment and unmodeled time-varying confounding in drug supply as general problems for studying the effect of harm reduction services, though addressing confounding is somewhat more tractable. I also describe measurement error and implausible theory that are more specific to the papers.
Feedback Effects
The methodological issue that has been most neglected is “feedback” between past outcome and treatment assignment. The strict exogeneity assumption in unit fixed effect designs “strictly forbids a ‘feedback’ effect from past outcomes to treatment assignment” (Chiu et al 2023). This observation has been noted previously such as in Kim & Imai 2019 (pg 472, “lack of feedback effects over time represents another key causal assumption required for the unit fixed effects models”) and Liu, Wang, and Xu 2021:“the strict exogeneity assumption… rules out the possibility that past outcomes directly affect current treatment assignment (no feedback)” but feedback remains rarely if ever discussed in empirical studies.
In the SEP paper, treatment is defined as a county opening a new syringe exchange for intravenous drug users. New services for drug users open in places with drug problems when these problems are increasing. And overdose deaths from intravenous drug use (most fentanyl/heroin use is injection drug use) is the outcome of interest here. It strikes me as incontrovertible that past levels (and trends) of drug overdose deaths are related to opening harm reduction services for injection drug users. There are significant political barriers to opening these services (locally unwanted land uses, NIMBYism, etc), and it’s inefficient and sort of nonsensical to open them in places and times that are not experiencing drug problems. This strikes me as a general problem for studying harm reduction services with studies of this kind (county panel data) – the services open where and when problems are increasing because problems are increasing.
Time Varying COnfounders
A second endogeneity issue that might be even more important, though is a little more specific to these papers, is time-varying confounders – namely that SEP openings are spatially correlated in the study period and occur around the time that fentanyl saturates the drug market in the Midwest and eastern US (~2014) – see my Fentanyl Shock paper. The abstract of the SEP paper notes “Effects are largest in rural counties and in counties that adopted SEPs after the influx of fentanyl to the US.” But the paper doesn’t model fentanyl exposure directly. Omitting this explanatory factor correlated with exposure and the outcome will inflate the estimate of SEP openings on opioid mortality.
The second paper in a similar vein – economists examining naloxone distribution and overdose deaths, reporting “moral hazard” effects – makes the same observation that results are mediated by fentanyl – including a stunning claim that naloxone access laws caused an 84% increase in fentanyl deaths in the Midwest. However, like the SEP paper, this paper does not model fentanyl exposure directly, an incredible omission of a time-varying confounder in a model of changes in fentanyl mortality during a period when fentanyl rapidly saturated Midwest drug supply. Ohio, Pennsylvania, Michigan, Wisconsin, and Minnesota adopted naloxone laws in 2014, so there’s a massive time varying confound not included in the model. You don’t see this in their analysis of the Northeast because many though not all naloxone laws passed before fentanyl penetrated, e.g. MA in 2012 and DC/NJ/VT in 2013.
Theory/Mechanism
My other issue with the naloxone distribution paper is that its model of moral hazard applied to injection drug use of street heroin – that people will start injecting street heroin or inject more heroin at once when naloxone is abundant because they might get rescued if they overdose – is not plausible for several reasons. A lot has been said about this already. And the most obvious problem is that there is no evidence people start using opioids because a naloxone law passed. One additional issue that the model neglects is that overdose reversal by naloxone, an opioid antagonist, is painful; it immediately precipitates opioid withdrawal symptoms and can lead to agitation, nausea, and vomiting. SAMSHA’s guidance to first-responders cooly notes “withdrawal triggered by naloxone can feel unpleasant. Some people may become agitated or confused.” When people with chronic pain are revived from opioid overdose with naloxone, naloxone administration can lead to “abrupt onset of significant physical pain.” So even with assurance that a friend could rescue them from an overdose, the rationalist pleasure-maximizing injection drug user they have in mind would still have reason to avoid overdosing because it’s more pleasant to get high than to get too high, overdose, and suffer from opioid withdrawal induced by naloxone administration.
The SEP model of drug use is odd, too. The textbook SEP is an exchange program – used needles are exchanged 1 to 1 for clean needles. Why would a person who does not inject drugs start doing so when this program becomes available? Where are they getting used needles? (Of course, SEPs vary, and some will dispense clean needles without requiring dirty needles. But some SEPs also dispense MOUDs, which we have strong evidence reduce overdose mortality, or provide other health care services such as pregnancy screenings. This heterogeneity is not modeled in the paper.)
Measurement Error
Measurement error, in many senses, seems prevalent. Many treated counties are labeled as never-treated because the source data is not a comprehensive database of SEPs. In Massachusetts, based on the map in Figure A1, 6 counties (Suffolk, Middlesex, Essex, Barnstable, Bristol, Hampden) are labeled as ever treated and the remaining 8 are labeled never treated as of 2016, the last year of the study. I found that three of the 8 labeled never treated as of 2016 had needle exchanges during the study period. Hampshire County Massachusetts is labeled as never having an SEP when a needle exchange has operated in Northampton, the county seat, since 1995. In 2016, AIDS Project Worcester began operating a city-funded SEP; Worcester County is incorrectly labeled “never treated” in the paper, likely because it hadn’t yet appeared in NASENs directory. Also in 2016, BAMI COPE Center began a syringe exchange program in Brockton; its county, Plymouth County, is also incorrectly labeled “never treated” in the paper, again likely because it hadn’t yet appeared in NASENs directory.
SEP opening is not a direct measurement of “needles exchanged” or “clients who used SEP services.” For example, an existing SEP might increase capacity.
People travel between counties for services (unmodeled spillovers). Based partly on survey data from clients of 1 SEP that, on average, clients traveled 14 miles for services at that SEP, that inter-county migration is not a big concern. But the standard deviation on this number is 40 miles. These numbers are consistent with crossing county lines.
Facial Validity of Parallel Trends
Treatment status of units is changing rapidly during and just after the study period. In 2017, the year after the analysis period, SEPs opened in several of the remaining untreated counties in Massachusetts: Both North Adams and Pittsfield in Berkshire County and Greenfield in Franklin County opened SEPs in 2017 (that were approved in 2016). An SEP has also opened on Martha’s Vineyard (Dukes County). If you were writing this paper today, the only source of “never treated” within-state county-level variation from Massachusetts would come from comparing Nantucket to the rest of the state. The analytic point here is that whatever pre-trend plot you show (and these are apparently often underpowered), the plausibility of parallel trends absent a facility opening is undermined by the places without facilities being quite different.
Replication Materials
Finally, both papers don’t share replication datasets or analysis code – they use CDC restricted mortality data, so neither one can publish the full replcation dataset. But I wish they had considered ways to share at least some replication materials (The author of the SEP paper did share the list of facilities!), such as sharing the final, aggregated data file without the restricted mortality files fields. They might have appended to this publicly available CDC mortality data from CDC Wonder, or publicly available “modeled” county overdose mortality data, or the aggregated restricted data from the CDC with random noise added. None of these are without their issues (public mortality data omits small cells, which in context would be counties with small populations and/or fewer drug deaths; modeled deaths obviously have error, though so do death certificates) but it would make the process more transparent and make it easier to check things like pre-trends and sensitivity to particular states.
Michael Zoorob 