There’s been much discussion and debate about the cuts to Medicaid eligibility that Congress just passed and, in particular, what they may mean for current Medicaid recipients. A key piece of evidence in this debate has been results from the Oregon Health Insurance Experiment (OHIE), a randomized trial, which I helped lead, examining the impact of covering low-income uninsured adults with Medicaid for one to two years. While it’s always gratifying to see one’s work used in policy deliberations, it’s frustrating when the results are misinterpreted.
An important sticking point is the interpretation of so-called “null results” — estimates of Medicaid’s impact that we cannot statistically distinguish from no effect. In the case of the OHIE, we found no evidence of statistically significant impacts of Medicaid coverage on mortality, or on several measures of physical health, such as hypertension, high cholesterol, or diabetes.
Unfortunately, people are making a common mistake: They are misinterpreting the lack of evidence of impacts as evidence of no impact.
For example, two economists recently wrote a letter to the Wall Street Journal noting that:
“The best evidence on the health effects of the Medicaid expansion comes from the Oregon Health Insurance Experiment. The OHIE is a randomized controlled trial, or RCT — the gold standard for such research. … The OHIE found no improvements in mortality or any other physical health outcome from expanding Medicaid.”
How the tax-cut bill that’s headed to Trump’s desk would upend health care
Null results can be extremely valuable. They can make us question what we think we know and spur innovation. When making evidence-informed decisions, understanding what doesn’t work is just as critical as understanding what works.
However, interpreting null results correctly is essential. The results from the OHIE indicated no statistically significant impact of Medicaid on several physical health measures or on mortality.
But we cannot say we found evidence that Medicaid has no effect on these outcomes. The difference between no evidence of impact and evidence of no impact may seem like wordplay, but when almost 12 million people are at risk of losing health insurance, understanding this difference is crucial.
We need to look beyond a simplistic summary of whether or not there is evidence of a statistically significant effect of Medicaid to consider the magnitude of the estimated effect and the amount of uncertainty around it. Every research result comes with a range of plausible values around it (a confidence interval) that represents statistical uncertainty about the true effect. If this range includes zero, we can’t rule out no effect. But we also can’t rule out any of the other values within the plausible range.
Consider some of the health outcomes in the OHIE for which there was no evidence of a statistically significant impact of Medicaid. Some of these “null results” were sufficiently statistically precise to be informative.
One example of an informative null result was the study’s findings for hypertension. My co-authors and I found no impact of Medicaid coverage in reducing hypertension, and the results were sufficiently precise to rule out much larger estimates of Medicaid’s ability to reduce hypertension that had been found in previous, quasi-experimental studies.
In other words, even the maximum possible benefit in our range of plausible values was smaller than what previous research had found. So, from this “null result” on hypertension, we learned that the effect of Medicaid on reducing high blood pressure may be smaller than what was previously thought. (Again, though, it is not evidence that Medicaid has no effect on hypertension.) That’s a useful addition to the discussion.
However, the null results of the impact of Medicaid on rates of uncontrolled diabetes (i.e., high rates of glycated hemoglobin) and on mortality were not informative. This stems from a combination of the relatively small sample size of the Oregon experiment (only about 10,000 individuals gained Medicaid coverage) and the (fortunately) low rates of diabetes (about 5%) and mortality (less than 1%) in the study population. The result was a high degree of uncertainty.
For diabetes, the range of plausible impacts for Medicaid included zero, but also included the improvements one might expect given the estimates from the amount Medicaid increased use of diabetes medication and the estimates from the clinical literature on what such an increase in medication would predict for improvements in glycated hemoglobin levels.
GOP works to square making America healthy with taking away health insurance
So we couldn’t rule out either no effect on diabetes or the possibility that Medicaid had the very effect we would have expected based on its impact on diabetes medication. I’d call this type of null result uninformative.
The study’s mortality results were likewise uninformative. They were unable to rule out the possibility that Medicaid reduced or increased mortality by a substantial amount. A subsequent, much larger, randomized controlled trial in which almost 4 million people were encouraged to enroll in health insurance found that health insurance has a statistically significant impact on reducing mortality among 45- to 64-year-olds. The authors of that study explicitly noted the results were perfectly consistent with the findings from the OHIE, due to the wide range of plausible mortality effects we had estimated.
Randomized evaluations can provide some of the most compelling evidence on program impacts, as the authors of the Wall Street Journal letter pointed out. But the appropriate use of that evidence to inform policy debates requires understanding that having a null result does not necessarily mean a program has no impact. Researchers and policymakers alike have a duty to represent and use evidence, including null results, responsibly.
Amy Finkelstein is a professor of economics at MIT and the co-Scientific Director of J-PAL North America.
