All About That Basic: Why Adherence to Reporting Standards Matters More Than Financial Conflict of Interest in Quality-of-Evidence Assessment

Scholars familiar with reporting standards of the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network are aware of their importance: compliance with these guidelines facilitates transparency and accuracy in published work. What may be less obvious is that when these guidelines are followed, financial conflicts of interest in research are, arguably, rendered mostly or entirely harmless. Why? Because when authors report the critical information they should, and describe findings and limitations transparently as they should, readers are provided with accurate information, regardless of study sponsorship. Conflicts of interest should be disclosed, of course, but ultimately may have little effect on the credibility and usability of results published in compliance with transparency standards.

A recent case in point provides, unfortunately, a great example of the opposite situation, in which the authors of a systematic review article claimed to follow Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) standards but did not actually do so [1,2]. The article in question examined the effects of out-of-pocket cost sharing on  prescription-filling behaviors[1]. Its authors concluded that the “published literature shows consistent [negative] impacts of higher cost sharing on initiation and continuation of medications” in a dose-response fashion. Regrettably, these conclusions were grounded in a combination of failure to conform to PRISMA standards and selective reporting of the available information, resulting in the provision of inaccurate information to the journal’s readers. 

Nonadherence to PRISMA Standards 

No Publication Bias Assessment. Foremost, the systematic review included no assessment of publication bias, deemed a PRISMA-essential standard because systematic exclusion of studies from available evidence threatens the core validity of any attempt to summarize the literature[2,3]. This omission is particularly concerning because publication bias is a known problem in research on cost-sharing policy[4]. 

One example is the MHealthy copayment-reduction cohort study, completed in 2009, whose a priori outcomes—medication adherence, health care costs, outpatient visits, and emergency/inpatient utilization—were never published[5]. Also never published were several important a priori outcomes in a pivotal randomized controlled trial (RCT) included in the review, the Post-Myocardial Infarction Free Rx and Economic Evaluation (Post MI-FREEE) study, including utilization rates for emergency departments, hospitals, and physicians[6-8]. Reporting of standard publication-bias metrics (e.g., funnel plot, Egger test) would have provided readers with critical information about the degree to which available evidence represents actual cost-sharing effects versus reporting proclivities and interests of study investigators[3,9]. 

No Risk-of-Bias Assessment. Another important but missing PRISMA element was quality (risk-of-bias) assessment[2,3,9]. The authors indicated they performed risk-of-bias assessments but omitted essential information: rating method, number of raters, resolution of inter-rater disagreement, and—most remarkable—results of the assessments, although these are critically important indicators of the validity of any systematic review[1,3,9]. 

No Certainty-of-Evidence Assessment Despite Published Cochrane Report. Finally, contradicting the authors’ assertion of consistency in results reported in the literature, they neither conducted an assessment of study heterogeneity nor reported the specific quantitative results of individual studies, with measures of precision around the estimates (e.g., confidence intervals), as recommended in PRISMA guidelines[2,3]. Notably, a 2015 Cochrane review on this topic found so much heterogeneity in available evidence that its authors reported both difficulty in summarizing results and low to very low evidentiary certainty[10]. 

Selectively Reported Results 

Primary Outcome of Pivotal Trial Not Mentioned. Contrary to PRISMA guidelines for reporting outcomes of individual studies[2], the review authors omitted mention of the Post-MI FREEE RCT’s primary between-group result: a nonsignificant difference, comparing patients randomized to free medication versus usual coverage[7]. The authors chose instead to describe only a secondary outcome between-group result, characterized by the authors as “statistically significantly fewer major vascular events, revascularizations, or strokes” despite an extremely small effect size: rates of 21.5 versus 23.3 per 100 person-years (HR=0.89, 95% CI=0.80-0.99)[1,7]. Only 12.1% of patients randomized to free medication, versus 8.9% in usual coverage, were fully adherent (medication possession ratio >80%) to all 3 recommended medication classes[7]. 

These notably unimpressive results of providing free medication in Post-MI FREEE, although not mentioned by the authors of the systematic review, were highlighted in an editorial accompanying the Post-MI FREEE report, which referred to copayment-elimination effects as “distressingly modest”[11] and by one Post-MI FREEE principal investigator, who commented: “we gave these people the medicines for free and only half took it.”[12] Although seemingly counterintuitive, this relative lack of medication price sensitivity is expected because cost is not commonly endorsed by patients as a key cause of medication nonadherence. More commonly mentioned causes, described in one systematic review of patients after myocardial infarction, are beliefs about illness and medication, side effects, forgetfulness, and need for provider support and communication[13]. 

Inaccurate Description of the RAND Health Insurance Experiment (HIE). The review authors’ claim that their conclusions were “consistent with [those] found 50 years ago in the HIE” is directly contraindicated by the HIE findings, as described by HIE investigators[14]. Price elasticity (responsiveness) in the HIE, calculated from a comparison of 95% coinsurance with free care, was 0.2[15], similar to previously reported elasticities in health care and far less than the standard of 1.0 considered by economists to represent price sensitivity[8]. Despite evidence that both high- and low-value services were equally reduced by coinsurance in the HIE, the coinsurance group had fewer activity-restricted days, including time spent seeking health care, and less anxiety about health[14]. The only negative health effects of coinsurance, observed only in lower-income persons, were clinically minor and—importantly—due to higher case-finding rates, not to better treatment adherence[4,14,15]. The HIE investigators concluded that increased cost sharing for “the kind of people who typically are covered under employer health insurance” would likely result in “enormous potential savings“ with “little apparent health impact”[14]. 

What Went Wrong in This Systematic Review? 

The problems in this review can be summarized in two phrases: poor-quality source literature and lack of transparency in reporting. Most research on cost-sharing policy has been plagued by serious methodological flaws including channeling bias, publication bias, and selective reporting[4,8,10]. PRISMA standards exist to inform readers of such limitations, facilitating accurate synthesis of research quality and outcomes[9]. That these standards were not followed in this systematic review is unfortunate and make its results uninterpretable.

Perhaps more important, though, is the process question of how errors so substantial were able to be reported and go uncorrected at all. I suggest that no one was served well by this process--not the journal's readers, who were provided with inaccurate information; not the authors, whose work could have benefited from the corrections that peer review should provide but did not; and not the study sponsor, the National Pharmaceutical Council, because the publication of information that lacks credibility does not benefit those who paid for it. 

Only the journal’s editorial team can assess the root cause of this incident, and it is to be hoped they will choose to do so. Regardless, we should view the incident as a lesson learned about the importance of accurate and transparent research reporting: when it comes to quality-of-evidence assessment, it really is all about that basic. 

References  

[1] Fusco N, Sils B, Graff JS, Kistler K, Ruiz K. Cost-sharing and adherence, clinical outcomes, health care utilization, and costs: a systematic literature review. J Manag Care Pharm. 2022 April 7. doi: 10.18553/jmcp.2022.21270 

[2] Enhancing the QUAlity and Transparency of health Research (EQUATOR) Network. PRISMA 2020 checklist. https://www.equator-network.org/reporting-guidelines/prisma/.

See also the expanded checklist at https://prisma-statement.org/documents/PRISMA_2020_expanded_checklist.pdf. 

[3] Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. 

[4] Fairman KA, Curtiss FR. What do we really know about VBID? Quality of the evidence and ethical considerations for health plan sponsors. J Manag Care Pharm. 2011;17(2):156-174. 

[5] Spaulding A, Fendrick AM, Herman WH, et al. A controlled trial of value-based insurance design – The MHealthy Focus on Diabetes (FOD) trial. Implement Sci. 2009;4:19. 

[6] Choudhry NK, Brennan T, Toscano M, et al. Rationale and design of the Post-MI FREEE trial: a randomized evaluation of first-dollar drug coverage for post-myocardial infarction secondary preventive therapies. Am Heart J. 2008;156(1):31-36. 

[7] Choudhry NK, Avorn J, Glynn RJ, et al. Full coverage for preventive medications after myocardial infarction. N Engl J Med. 2011p;365(22):2088-2097. 

[8] Fairman KA, Curtiss FR. VBID, the PPACA, and FREEE medications: Did politics trump the evidence about cost sharing? J Manag Care Pharm. 2012;18(2):146-155.

[9] Haber SL, Fairman KA, Sclar DA. Principles in the evaluation of systematic reviews. Pharmacotherapy. 2015;35(11):1077-1087. 

[10] Luiza VL, Chaves LA, Silva RM, et al. Pharmaceutical policies: effects of cap and co-payment on rational use of medicines. Cochrane Database Syst Rev. 2015;(5):CD007017. 

[11] Goldman L, Epstein AM. Improving adherence—money isn’t the only thing. N Engl J Med. 2011;365(22):2131-33. 

[12] Marchione M. Study finds many patients shun free heart drugs. AP. November 15, 2011. https://www.bostonglobe.com/news/nation/2011/11/15/study-finds-many-patients-shun-free-heart-drugs/vBUPZ18azF0zqICmnRlJOL/story.html 

[13] Piekarz H, Langran C, Raza A, Donyai P. Medication-taking for secondary prevention of acute myocardial infarction: a thematic meta-synthesis of patient experiences. Open Heart. 2022;9(1):e001939. 

[14] Keeler EB. Effects of cost sharing on use of medical services and health. Rand Corporation. 1992. https://www.rand.org/pubs/reprints/RP1114.html#:~:text=It%20concludes%20that%20cost%20sharing,restrain%20spending%20on%20health%20care. 

[15] Manning WG, Newhouse JP, Duan N, Keeler EB, Liebowitz A, Marquis MS. Health  insurance and the demand for medical care: evidence from a randomized experiment. Am Econ Rev. 1987;77(3):251-77.

  

Were COVID-19 “Lockdowns” Helpful or Harmful? The Answer Shouldn’t Depend on Whom You Ask

 Last month, the U.S. Centers for Disease Control and Prevention (CDC) announced troubling findings of the Adolescent Behaviors and Experiences Survey (ABES) administered during the first half of 2021: during the 12 months prior to the survey, 37.1% of teens reported poor mental health, 44.2% had “experienced persistent feelings of sadness or hopelessness,” and 19.9% had “seriously considered” suicide[1]. This was not the first bad behavioral health news to come out of the pandemic. Multiple children’s hospitals and pediatric health organizations declared a national state of emergency in children’s mental health in October 2021, citing rising rates of emergency and hospital use for mental health emergencies, including suicide attempts, and implicating “the stress brought on by COVID-19 and the ongoing struggle for racial justice” as causal factors, particularly for children of color[2]. Also disproportionately affecting people of color, especially youth, were skyrocketing rates of psychoactive substance use, misuse, and overdose deaths[3,4].

Social isolation was an important risk factor for these problems in the ABES, as in previous work[4-8]. In the ABES, feelings of being close to people at school were associated with lower rates of all the mental health problems measured, including serious consideration of suicide (14.0% for those connected versus 25.6% for those not connected). Virtual connection was also associated with somewhat reduced rates (18.4% vs. 24.9%). A growing body of evidence has linked these trends, which were rising prior to the pandemic, with public safety measures for COVID-19 mitigation, such as lockdowns and school closures[4-6]. Similarly, loneliness and lack of access to treatment have long been recognized risk factors for substance misuse and its consequences[7,8].

Were these pandemic-period increases in mental health and substance use disorders unavoidable, the combination of trends in place prior to the pandemic and completely necessary, life-saving governmental “lockdown” actions? Or were they avoidable, the results of governmental decisions unsupported by scientific evidence? Likely, you have formed an opinion on these issues. Many of us have.

The truth is, though, that from an evidentiary perspective, we don’t know the answer to these questions. To answer them adequately would require calculating the net benefits versus harms of “lockdowns.” We need a comprehensive assessment of death and illness from multiple causes—substance misuse rates, overdoses, mental health problems, deaths from infectious disease, and deaths from other causes—comparing regions with varying pandemic policies. This evidence does not exist[9,10]. More importantly from an evidentiary perspective, though, is what may be an underlying process failure in research practice: scientists from different disciplines generally do not work together but work in their own professional silos.

For example, a highly publicized study finding that pandemic lockdowns “had little to no effect on COVID-19 mortality” was conducted entirely by a team of economists[11]. Other studies of COVID-19 policies and effects were conducted by teams of behavioral scientists, with no biomedical scientists, or vice versa, even when the work is labeled “multidisciplinary” by its authors[4,12-14]. And, as I point out in a historical review and commentary published in the Annals of Epidemiology, federal public health leadership, both on the White House COVID-19 Task Force and at the CDC, consisted—and still consists—almost entirely of infectious disease specialists, with no specialists in behavioral health or education[15].

What if we approached public health science differently? What if we recognized that, just as health itself is holistic, public health science should also be holistic? What if we applied to public health science the same standards of multidisciplinary engagement already in use in medical care for chronic diseases[16,17]? What if we put into practice a recognition that specialists in behavioral health, education, and biomedicine each have unique areas of expertise and efficiencies in identifying, measuring, and interpreting health outcomes?

The result, I believe, would be use of multidisciplinary teams to formulate and study proposed public health policies, producing better quality evidence—and better policy. In my experience working as a behavioral health researcher in conjunction with an exceptional clinical team at the Midwestern University College of Pharmacy, the research product always benefits from different professional points of view. If you get an opportunity to work with a person in a discipline whose vocabulary and methods are very different from your own—please consider it. I promise you and your work will both benefit.

Read my article suggesting a need for multidisciplinary approaches to public health science: Pandemics, policy, and the power of paradigm: will COVID-19 lead to a new scientific revolution? at this link.

 

1. Jones SE, Ethier KA, Hertz M, et al. Mental health, suicidality, and connectedness among high school students during the COVID-19 pandemic—adolescent behaviors and experiences survey, United States, January-June 2021. MMWR Suppl. 2022;71(3):16-21. Available at: https://www.cdc.gov/mmwr/volumes/71/su/su7103a3.htm.

[2] American Academy of Pediatrics, American Academy of Child and Adolescent Psychiatry and Children’s Hospital Association. AAP-AACAP-CHA declaration of a national emergency in child and adolescent mental health. October 19, 2021. https://www.aap.org/en/advocacy/child-and-adolescent-healthy-mental-development/aap-aacap-cha-declaration-of-a-national-emergency-in-child-and-adolescent-mental-health/?_ga=2.249206535.1219731009.1653672604-2027483972.1649716825

[3] American Medical Association. Issue brief: nation’s drug-related overdose and death epidemic continues to worsen. May 12, 2022. https://www.ama-assn.org/system/files/issue-brief-increases-in-opioid-related-overdose.pdf

[4] Ross JA, Malone PK, Levy S. The impact of the SARS-CoV-2 pandemic on substance use in the US. Clin Infect Dis. 2022;ciac311. Online ahead of print. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9129126/pdf/ciac311.pdf

[5] Viner R, Russell S, Saulle R, et al. School closures during social lockdown and mental health, health behaviors, and well-being among children and adolescents during the first COVID-19 wave. JAMA Pediatr. 2022;176(4):400-409. Available at: https://jamanetwork.com/journals/jamapediatrics/fullarticle/2788069

[6] Benton T, Njoroge WFM, Ng WYK. Sounding the alarm for children’s mental health during the COVID-19 pandemic. JAMA Pediatr. 2022;176(4):e216295. Available at: https://jamanetwork.com/journals/jamapediatrics/fullarticle/2788911?resultClick=1

[7] Ingram I, Kelly PJ, Deane FP, et al. Loneliness among people with substance use problems: a narrative systematic review. Drug Alcohol Rev. 2020;39(5):447-483. Available at: https://onlinelibrary.wiley.com/doi/10.1111/dar.13064

[8] Bolinski R, Ellis K, Zahnd WE, et al. Social norms associated with nonmedical opioid use in rural communities: a systematic review. Transl Behav Med. 2019;9(6):1224-1232. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6875642/

[9] Haber NA, Clarke-Deelder E, Feller A, et al. Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. BMJ Open. 2022;12(1):e053820. Available at: https://bmjopen.bmj.com/content/12/1/e053820.long

[10] Cristea IA, Naudet F, Ioannidis JPA. Preserving equipoise and performing randomized trials for COVID-19 social distancing interventions. Epidemiol Psychiatr Sci. 2020;29:e184. https://doi.org/10.1017/S2045796020000992

[11] Herby J, Jonung L, Hanke SH. A literature review and meta-analysis of the effects of lockdowns on COVID-19 mortality. Studies in Applied Economics. January 2022. https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf

[12] Chams N, Chams S, Badran R, et al. COVID-19: a multidisciplinary review. Front Public Health. 2020;8:383. Available at: https://www.frontiersin.org/articles/10.3389/fpubh.2020.00383/full 

[13] Holmes EA, O’Conhor RC, Perry VH, et al. Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry. 2020;7(6):547-560. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7159850/

[14] Woolf SH, Chapman DA, Sabo RT. Excess deaths from COVID-19 and other causes in the US, March 1, 2020, to January 2, 2021. JAMA. 2021;325(17):1786-1789. Available at: https://jamanetwork.com/journals/jama/fullarticle/2778361

[15] Fairman KA. Pandemics, policy, and the power of paradigm: will COVID-19 lead to a new scientific revolution? Ann Epidemiol. 2022;69:17-23. Available at: https://reader.elsevier.com/reader/sd/pii/S1047279722000230?token=937D66A3B192E124236DFC7133F0DCEC0BF147BDCA80BF302A17A7B686856E2F60CBC363052290C8D6D0135656EE5953&originRegion=us-east-1&originCreation=20220530173333

[16] American Diabetes Association. Standards of medical care in diabetes—2021. Available at: https://care.diabetesjournals.org/content/diacare/suppl/2020/12/09/44.Supplement_1.DC1/DC_44_S1_final_copyright_stamped.pdf.

 

[17] National Kidney Foundation. Clinical practice guidelines for chronic kidney disease. evaluation, classification, and stratification. 2002. Available at: https://www.kidney.org/sites/default/files/docs/ckd_evaluation_classification_stratification.pdf. 

On Being a "Numbers Person" in the Covid-19 Era

On September 8, 1994, my husband and I traveled with our 3-year-old son and his 9-month-old twin brothers to my parents’ home near Rhode Island. Waiting at the airport for our red-eye flight to the airline’s hub in Pittsburgh, we learned that a plane bound there, operated by the same airline, had crashed in the woods a few hours earlier, killing all 132 people on board. When we arrived at Green Airport the next morning, I was surprised to see a television crew waiting for us. “Were you frightened?” a reporter asked me. I had been up all night with three small children and said the first thing that popped into my head. “No, I wasn’t frightened,” I replied. “It’s a low-probability event.”

My calm response aired that evening, alongside that of a woman who was sobbing. “Oh, those poor people!” she cried. Of course, I had not been asked whether I was sorry for the loss of life; I had been asked if I was frightened by the crash, and statistically, there was no reason to be. Yet, with some amusement, I realized that sometimes, being a “numbers person” can make one appear to be … well … a little uncaring.

I have sensed this perception in responses to biostatisticians who have questioned the evidentiary basis behind governmental response to the Covid-19. Some appear to suggest that these biostatisticians are unaware of either the numbers of “coffins of Covid-19 victims” or the exponential nature of infectious disease transmission that puts an increasing number of people at risk of death. I would suggest the opposite is true. The issue is, it seems to me, not lack of awareness; it is a heightened awareness that defies comfort—an all-too-complete picture of the consequences, not just of the disease, but of the lockdowns intended to combat it.

When I hear calls for protection of the vulnerable, I echo them, of course. But which vulnerable people? We are all familiar with the demographic and clinical conditions that put one at higher risk for severe Covid-19 illness. Yet, as I write this, an estimated 22 million U.S. residents have been laid off because of business and school shutdowns, and more layoffs are coming. If prolonged, the resulting economic deprivation will dramatically increase their risk of numerous negative health outcomes, including a doubling of their odds of cardiovascular disease, our nation’s top cause of death. The immunosuppressive effects of chronic stress are well-documented, ironically meaning that some paycheck-to-paycheck earners may be put at a heightened risk of infection from the very policies intended to help them avoid it. Suicide hotlines are already seeing spikes in calls from persons who are anxious, isolated, or worried about their economic futures, and persons of low income and educational levels are reporting more financial stress than are their higher-income/education counterparts.

Covid-19 is an important public health issue. So, too, are unemployment, cardiovascular risk, chronic stress, loneliness, and despair.

What is the actual risk associated with Covid-19 infection? Legitimate questions on this point merit more than the usual vague bromides about listening to “the experts.” One might first ask, which experts? From the beginning of the Covid-19 outbreak, most top biostatisticians have been making essentially the same observations: (1) without an accurate denominator of the total number of infections, both symptomatic and asymptomatic, it is impossible to estimate disease severity; (2) it is important to account for sources of regional variation (e.g., population age, hospital bed occupancy) and competing causes of death (i.e., not every death with Covid-19 infection is a death from Covid-19 infection); (3) despite media coverage suggesting otherwise, the risk of Covid-19 death is remarkably low for those not in the known vulnerable groups; and (4) exaggerated estimates of harms and some extreme measures themselves can be harmful to public health (e.g., public panic, hoarding of essential supplies). Elsewhere, a research intern and I observed that the Imperial College model that was the basis for the shutdowns did not meet fundamental professional standards and had likely systematically overestimated projected ICU stays and deaths. A much higher-quality model from Oxford reached very different conclusions and, in a sign of a solid evidence-based approach, called for population serological surveys to validate or refute the modelers’ results. Other experienced modelers are examining disease trajectories by country, finding few differences between countries with versus without lockdown policies.

All this evidence is preliminary, and there is certainly no shortage of disagreement about data interpretation. We do not yet have the most important information needed, estimates of disease severity with accurate denominators, although one preliminary study in the Gangelt region of Germany based on antibody testing concluded that the infection fatality rate was 0.37%. While we wait for more antibody test results, what is a “numbers person” to do?

For a “numbers person” who has the misfortune of having a tender heart, as I do, watching the unfolding of a process not based on reliable evidence has been difficult. The policy disagreement has never been about whether we "social distance." It has always been about how we do so, and whether reliance on better evidence could have produced more targeted, smarter strategies with less collateral damage to our nation's economically vulnerable.

In that context, it seems to me that the best response of a “numbers person” is to rely on the methodologically higher-quality work in reaching conclusions about Covid-19, update those conclusions as new evidence comes in, and advocate for highest-quality data to support decision making. Lest that sound too much like my measured response to the reporter 25 years ago, I would suggest that when one considers the harms as well as the benefits of Covid-19 lockdowns, calling for an evidentiary basis for decision making is the very opposite of uncaring. It is, in fact, the only caring thing for a "numbers person" to do.

Numerators, Denominators, and COVID-19

Numerators, Denominators, and COVID-19

After a seven-year absence, this blog is returning, just in time to tackle what will surely become one of the most discussed debates in public health for years to come—how best to manage the COVID-19 infection pandemic currently gripping the world’s attention. A recent provocative article in STAT by widely respected expert in evidence-based medicine John Ioannidis suggests that sweeping and consequential public policy decisions are being made “without reliable data.” The result, he suggests, may be “a fiasco in the making.” The article is well worth reading for an evidence-based perspective on the data we have—and the critically important data that we do not have—on  COVID-19. Surprisingly, the question Ioannidis is raising may be described as one of numerators and denominators.

At JMCP, the most common mathematical errors we identified in submitted work were not made in sophisticated models or multivariate analyses; they were made in calculating and describing simple percentages. The main problem was—and seems to be today—confusion over numerators and denominators used to estimate risks. The fundamental question raised by Ioannidis isn’t whether COVID-19 is a pandemic, as it has been named as such by the World Health Organization. The question is how medically serious the pandemic is, and to know that, one requires the data needed for an empirically accurate calculation of infection-associated risk. Ioannidis argues convincingly that we do not have the data to make such a calculation.

When I teach analytics, I make three key points about percentages:

(1)    The most important word in describing a percentage is “of,” which identifies the group you are describing—who is at risk? This is the denominator. Ioannidis observes that the only known denominator today is the count of people who are both symptomatic and test positive. As Ioannidis points out, this is the well-known methodological problem of selection bias. The people selected for testing are those with symptoms; they do not represent the total infected population. So, the correct statement to describe most of the percentages cited for COVID-19 today should be: “Of those who are so sick that they seek health care and test positive for COVID-19, what percentage experience an adverse outcome?”

(2)    For a percentage to be accurate, the numerator must be an accurately described subset of the denominator. Estimates of the effects of the COVID-19 pandemic may be flawed by numerators mismatched to the denominators they are intended to represent. The 16% rate of serious illness cited by the CDC means that 16% of those who became so sick that they were tested in a hospital experienced serious illness (in China). Unfortunately, the State of Ohio, among the first to impose widespread closures, increased that number to 20% and applied it to all infected people, not just those who get sick. The result of this mathematical error is likely a substantial overestimate of the total number of persons who will require hospitalization.  

(3)    For a percentage to be accurate, both the numerator and denominator must be known and measured accurately. Neither is true of the numbers used to represent the effects of COVID-19 in the United States. Numbers reported to date reflect health systems and environments that differ greatly from those of the United States:

-The rate of health care-associated infection in Italy is nearly double that of the United States.
-The rate of daily smoking is 25% in China, 20% in Italy, and 11% in the United States.
-The death rate from air pollution (per 100,000 population) is 140 in China, 49 in Italy, and 24 in the United States.

I will leave it to others to draw your own opinions about the effects and merits of the policies in place now. While we await data, the members of my family and I are adhering to the CDC guidelines. Time—and accurate information, I hope—will provide us all with the empirical evidence we need.

BD, ICD, and GIGO:

Why “Big Data” May Be Less than Meets the Eye

With both funding and pressure piling onto the business of comparative effectiveness research, the old mantra that you cannot improve what you do not measure has seemingly never been more critically important in health care than it is today. Many would argue that measuring health care outcomes has also never been easier.

And, in a sense, they’d be right. Health care researchers today have sophisticated equipment with supercomputing capabilities that many of us couldn’t even dream of ten or fifteen years ago. We also have access to tens of millions of data points—medical and pharmacy claims, mortality records, survey data, and even detailed genetic data, depending on the organization in which we are working. So much information, available so conveniently, at such a low cost, and analyzable at such speed. It’s enough to make a policy analyst positively giddy. Perhaps it is not surprising then, that one commentator, a Health Policy and Life Sciences Group manager at Intel, assessed Big Data’s potential to “revolutionize health care” in this way:

Big Data provides us an opportunity to transition to a personal care system. Rather than making assumptions based on what has worked for other people, this personal view would allow us to take data about a patient’s genomes, medical history and behaviors to construct a virtual model that would help predict which treatments will be most effective and customize them to an individual — improving quality of life for the patient and saving the delivery system money.

With all this excitement about the potential of large datasets to unlock the secrets of greater longevity at lower cost, it’s easy to forget a crucial pitfall encountered by researchers who use them unawares: those beautifully packaged data were collected by human beings. And, because many of those human beings did not have research on their mind when they collected and recorded the data, they may have been motivated to treat our valuable information in ways that we did not expect or want. One example, which I discussed in Chapter 7 of Health Care Research Done Right, is billing (claims) data. Because the main purpose of billing data is to generate payment for health care services, claims coding is subject to “upcoding” and deliberate miscoding to enhance reimbursement.

This example is known to many claims database researchers. However, not all similar threats to study validity are as widely recognized or publicized, despite their potential effect on the well-being of patients whose plans or health care providers forget that real-world data collection may affect ideal-world research in unexpected ways. Several examples have been highlighted in recent press articles. I’ll talk about two in this post.

First is updated information about the long-awaited transition to ICD-10, which will increase the number of available diagnosis codes from about 14,000 to more than 68,000, the number of procedure codes from about 4,000 to more than 72,000, and the number of pages in the American Family Practice Association “superbill” (a standard form intended to list most of the diagnosis codes encountered in a typical practice, used for the physician’s convenience) from 2 to a whopping 9 pages. If used properly, the coding system has the potential to improve the accuracy of electronic diagnostic record-keeping, reimbursement, and, ultimately, quality of care, because—in theory—physicians who are given more accurate feedback will have greater incentive to use evidence-based procedures and treatment protocols.

But I said if used properly, and that big “if” is looking iffier all the time. The deadline for compliance with ICD-10 coding, originally scheduled by the Centers for Medicare & Medicaid Services (CMS) for October 2011, has been pushed back several times and is now slated for October 2014. Except that a survey of “providers, payers and health information technology vendors” conducted in February 2013 found that about one-half of participating vendors reported less than 50% progress toward ICD-10 readiness, and more than 40% did not know when they would begin detailed steps toward final implementation, with about one-quarter reporting being nearly finished.

And those are just the data processing problems. Industry insiders are reporting that experienced diagnosis coders are choosing to retire rather than learn the new system, leaving an unknown proportion of the implementation of ICD-10 in the hands of newbies. How long it will take for coders to become familiar with the new system is unknown. Which will bring researchers to a critically important question: when analyzing claims data coded with ICD-10, how accurate are the diagnoses?

A second example involves a completely different data source but a similar problem. An anonymous, Internet-based survey of New York City hospital residents found that 49% had knowingly reported cause of death inaccurately when completing a death certificate.[1] Of residents who had completed at least eleven death certificates in the previous three years, nearly six in ten reported deliberate inaccuracy. About three-quarters said that the computer system “would not accept the correct cause” of death, 41% said that they were told to “put something else” by the hospital admitting office, and 31% said that the medical examiner told them to report the diagnosis incorrectly. Among the more common actual causes of death associated with deliberately inaccurate reporting was septic shock, management of which is a quality-of-care indicator.[2] Noting that death certificates “contain critical information for epidemiology, public health research, disease surveillance, and community health programs,” the researchers noted that the routine reporting of inaccurate causes of death “may have lasting effects on the public health priorities of the community.”

All of which should give us pause as we consider the current level of enthusiasm for “big data.” Are automated data a silver bullet for all that ails the American health care system, or are they GIGO (garbage in, garbage out)? Savvy researchers should recognize that either possibility exists and should know how to investigate the data prior to using them.

[1] Wexelman BA, Eden E, Rose KM. Survey of New York City resident physicians on cause-of-death reporting, 2010. Prev Chronic Dis. 2013;10:E76.

[2] NQF #0500 Severe Sepsis and Septic Shock: Management Bundle, Last Updated Date: Oct 05, 2012.

Value-Based Insurance Design Defined as Copayment Reduction:

Where Have All the Studies Gone? Part 2

In Part 1 of this posting, I explained the recent shift in the meaning of the term “value-based,” hinted at missing research evidence, and made a passing reference to an award-winning song included on one of the few folk albums in U.S. history to attain #1 status for a full month—Peter, Paul, and Mary, released in 1962 by the artists of the same name. Today, on the fifty-first anniversary of John F. Kennedy’s commencement address at Yale, also in 1962, I draw on his key theme and ask: when it comes to VBID, have we “enjoyed the comfort of opinion without the discomfort of thought”?

A characterization that no “discomfort” associated with hard work has been experienced in VBID research would be unfair because extensive analysis has been done—much (but not the majority) of high quality.[1] However, some key information has gone unreported, as shown in the table below:

Table 1. Copayment Reduction Studies

First Author (Publication Year)—Description	Current Status
Chernew (2008). Quasi-experimental (difference-in-difference times series analysis with nonequivalent comparison group)[2]   Copayment of $0 for generic, 50% reduction for brand   Outcome measure was MPR change.	Reported. Copayment reductions were associated with a 2.6 to 4.0 MPR point difference (i.e., an added 7 to 14 days of therapy per year).   Key information missing from publication included: *Pre-intervention cost-sharing amounts in the comparison plan *Comparability of relevant benefit features in the study plans *Industry sectors of employees in study plans
Original analytic design of planned follow-up to Chernew 2008 (study described in previous table row). Outcomes included hospitalizations and ER visits, which had been predicted by the investigators to decline as a result of the medication adherence improvements observed.[2]	Results not reported. Instead, after initial analysis found “considerable uncertainty surrounding estimates of the impact of the [VBID] intervention on aggregate spending,” the authors developed a pharmacoeconomic model to estimate outcomes, describing it as “evidence that value-based insurance can be effective.”[3]
Spaulding (2009). MHealthy intervention for University of Michigan employees and dependents with diabetes; prospective trial with nonrandomized comparison group. Primary outcomes were medication utilization and adherence. Secondary outcomes were health care costs and utilization rates for hospitalizations, ER visits, and outpatient visits.[4]	Results not reported, although the study was initiated in 2006 and follow-up was scheduled to end in January 2009. Only exception was a poster described in a review article that reported “a 7 percent increase in adherence to blood pressure lowering medication and a nonsignificant 4 percent increase in adherence to statins.”[5]
Observational studies of copayment reductions conducted between 2010 and 2012[1]	Reported. Methodologically strongest of these studies indicated average MPR effect sizes of 1.4 to 4.0 percentage points, or about 5 to 15 added days of therapy per year.   Common problems identified in an editorial by Fairman and Curtiss included:   *No information about generic drug utilization *No information about net payer cost *Incomplete reporting of key study details (e.g., sample composition method) *Failure to control for or report mail order utilization rates or 90-day fill policies in all but one study
Choudhry (2011). Post-MI FREEE study of providing evidence-based medications to heart attack survivors free of charge after hospital discharge; block-randomized controlled trial (health plans, rather than patients, were randomized).[6] Nonequivalent communications to study groups (see below).   Primary outcome was occurrence of revascularization procedure or major vascular event (heart attack, stroke, unstable angina, CHF, or in-hospital death from CVD)	Primary outcome reported. *No significant difference between the study groups. *No pre-intervention values reported for either group.
Secondary outcome: Rate of major events as defined above	One secondary outcome reported. *Slightly and significantly lower for free medication group – 21.5 versus 23.3 events per 100 person-years   *No pre-intervention values reported for either group
Secondary outcome: Published analysis plan indicated that utilization outcomes would include annual rates of doctor visits, ER visits, and hospital admissions.	*Secondary outcome results not reported, although follow-up ended in November 2010
Secondary outcome: Published analysis plan indicated that investigators would report the outcome including out-of-hospital death data from the CDC.	*Secondary outcome results not yet reported; out-of-hospital death data are subject to a 1.5 to 2-year time lag and therefore should have been available for all patients by November 2012
Results for originally planned minimum follow-up time of one year	*Results not reported; results reflect minimum three-month follow-up.

CHF=congestive heart failure; CVD=cardiovascular disease; MPR=medication possession ratio, a commonly used measure of medication adherence, defined as days supply of dispensed medication divided by calendar time in days.

As Fred Curtiss and I pointed out in an editorial in April 2012, the most important of these studies is Post-MI FREEE, which, as its name implies, was an assessment of the effect of providing free medications to patients after hospital discharge for a heart attack. Despite Post-MI FREEE’s unique status as the first randomized study of copayment reductions, its value was greatly diminished by three critically important limitations.[7]

First was that the utilization outcomes pre-specified in the analysis plan have never been reported, although patient follow-up ended about two and one-half years ago. Nonreporting of pre-specified outcomes is always a cause for concern but was especially problematic in Post-MI FREEE because of the block-randomized design that assigned entire plans (rather than individual members) to free medication or usual coverage (copayments).  Specifically, because there was no control on medical reimbursement levels in the block randomization process, it is possible that the study groups did not have equivalent payment rates for the same medical services at baseline. Thus, it is possible that the post-intervention cost outcomes give an incomplete or even misleading picture of the effect of the intervention on service utilization. Compounding the problem, no baseline (pre-intervention) costs were reported for either study group.

Second patients in only one group—the group that did not receive free medications—were told that they would be on medications for a long time and would have to give up a lot of fun stuff. For both groups, letters advising patients about the study contained a list of medications that are recommended “to keep your heart strong,” but the usual coverage (control group) letters advised patients that “You may be on some medications for many years …” The usual coverage group letters also, unlike the free coverage letter, advised patients of American Heart Association recommendations for diet, exercise, and limitations on alcohol intake.

Third was a change in the study analysis plan to reduce the minimum length of follow-up from twelve months to three months because of lower-than-expected participation. A typical and sensible approach to handling this unavoidable development would have been a sensitivity analysis limited to study patients who had the originally intended twelve month minimum follow-up. However, although 65% of the study sample met that criterion, meaning that plenty of cases were available to perform a sensitivity analysis, none was performed.[7] As a result, we don’t know if the intervention’s effects would have persisted beyond the short follow-up period. Previous research suggests cause for concern on this point because of the large amount of noncompliance that takes place after the first three months of therapy but before the end of the first year. This decision was especially unfortunate because previous research in a similar patient population suggests that the survival benefits of statins are not observed until about 24 months of therapy.[8]

And—even with all of those potential biases toward finding optimal results for the free-medication group—the reported findings showed that only 12% of patients given free drugs, compared with 9% of those who paid copayments, were fully adherent to their drugs.[6]

Key Takeaway Points for Health Care Payers:

What To Do When the Studies Have “Gone to Flowers, Every One …” 

 

The history of VBID (copayment reduction) promotion is a notable but certainly not isolated example of noise outstripping scientific evidence in modern culture.  Much of the promised data have never been reported, and those that have been published suggest that the effects of copayment reductions are modest and may not be worth the added cost. It’s been said before, but it bears repeating: the strongest voices do not necessarily have the most important or valuable message.

So, when you read or hear about a proposed policy change in health care, whether it is VBID or a different proposal, consider the following questions:

• What was the follow-up time for the study being cited as evidence for the policy change? Does it represent a time frame during which one can reasonably expect meaningful benefits to occur?  If not, the results may represent confounding or the effects of another factor not measured in the study and/or presented in the report.
      
• Does the study follow-up period represent the time frame during which you will have to pay for the recommended intervention? (The investigators may have “lived with” this intervention for just a few short months; if you implement it, you may have its costs with you for a long time—perhaps long after intervention benefits stop accruing).
      
• To what degree do the characteristics reported by the authors match to those of the group to which you might apply the proposed intervention? For example, an intervention tested in a group of university professors may not have the same effect if implemented in an auto-workers’ union.
      
• Do the outcome measures represent your cost? Some copayment reduction studies report only total cost, which does not reflect the actual cost of the program to the payer. It is essential to report the net cost after taking loss of copayment revenues (offsets) from patients into account.
      
• If a pre-intervention versus post-intervention study was performed, did the investigators report the pre-intervention values for the outcome measures? Without this information, it’s hard for a decision maker to understand the actual effects of an intervention relative to the baseline status of the study sample.
      
• How was the intervention—such as “value-based”—defined? For example, the results of a “value-based” intervention that promoted a high-efficiency provider network provide little or no information about a “value-based” program that uses copayment reduction to incentivize medication use.
      
• Does the program described in the report include multiple components that you might or might not want to implement simultaneously? For example, if the program combined copayment reductions with other strategies, such as gym club memberships or educational interventions, it is hard to tell which program components were responsible for the outcomes. (And, if you cannot tell from reading the reporting what features were included in the program, it is reasonable to be skeptical about its findings.)
      
• Have relevant details for both study groups been presented in the study report? For example, for an intervention targeted to improve drug adherence, have the authors reported whether formularies, copayment levels, and utilization management policies (e.g., step therapy and prior authorization) were reasonably equivalent at baseline or otherwise explained the comparability of the groups (e.g., both groups were drawn from the same health plan or employer group with a uniform benefit design)?
      
• Have all the pre-specified outcomes been reported? (Note: This one takes a little detective work but is extremely helpful.) Use PubMed or, for randomized trials, www.clinicaltrials.gov to find the investigators’ pre-specified analysis plan.[9] If an analysis plan is available, compare it to the study report. If outcomes are missing from the report, query the investigators. If the outcomes are still not reported a considerable length of time after the end of follow-up, it is reasonable to consider the possibility of publication bias—that is, the outcomes were not reported because they did not fit the predispositions of the investigators, study sponsors, journal peer reviewers, or editors.
      
• Have previous studies of the same topic gone unreported, as has been the case in VBID? If so, it is reasonable to be skeptical about even the most promising research findings. For those of you familiar with probability theory—the underlying logic of statistical significance testing is that a certain number of results within a sampling distribution (a hypothetical sequence of repeated samples) will be statistically significant based on chance (sampling error) alone. If the results for actual samples have gone unreported, information necessary to interpret the results of statistical tests for any one sample is missing.

For health care payers, who are routinely bombarded today with assertions about "evidence-based" policies or services, these considerations should be viewed as critically important factors in decision making about value-based designs—or any proposed intervention, for that matter. When making decisions that affect not only pocketbooks, but also patient well-being, the best policy is circumspection: review the evidence that is reported, take into account the absence of evidence that has gone unreported, carefully examine the applicability of the available information for your population and setting, and don’t be afraid to ask questions.

 

[1] Fairman KA, Curtiss FR. What do we really know about VBID? Quality of the evidence and ethical considerations for plan sponsors. J Manag Care Pharm. 2011;17(2):156-174.

[2] Chernew ME, Shah MR, Wegh A, et al. Impact of decreasing copayments

on medication adherence within a disease management environment. Health

Aff (Millwood). 2008;27(1):103-12; Fairman KA, Curtiss FR. Making the world safe for evidence-based  policy: let’s slay the biases in research on value-based insurance design. J Manag Care Pharm. 2008;14(2):198-204.

[3] Chernew ME, Juster IA, Shah M, et al. Evidence that value-based insurance can be effective. Health Aff (Millwood). 2010;29(3):530-36.

[4] Spaulding A, Fendrick AM, Herman WH, et al. A controlled trial of value-based insurance design - the MHealthy: Focus on Diabetes (FOD) trial. Implement Sci. April 2009.

[5] Choudhry NK, Rosenthal MB, Milstein A. Assessing the evidence for

value-based insurance design. Health Aff (Millwood). 2010;29(11):1988-94.

[6] Choudhry NK, Avorn J, Glynn RJ, et al.; Post-Myocardial Infarction Free Rx Event and Economic Evaluation (MI FREEE) trial. Full coverage for preventive medications after myocardial infarction. N Engl J Med. 2011;365(22):2088-2097; Choudhry NK, Brennan T, Toscano M, et al. Rationale and design of the Post-MI FREEE trial: a randomized evaluation of first-dollar drug coverage for post-myocardial infarction secondary preventive therapies. Am Heart J. 2008;156(1):31-37.

[7] Fairman KA, Curtiss FR. VBID, the PPACA, and FREEE medications: Did politics trump the evidence about cost sharing? J Manag Care Pharm. 2012;18(2):146-156.

[8] Bavry AA, Mood GR, Kumbhani DJ, Borek PP, Askari AT, Bhatt DL.

Long-term benefit of statin therapy initiated during hospitalization for an

acute coronary syndrome: a systematic review of randomized trials. Am J

Cardiovasc Drugs. 2007;7(2):135-41.

[9] At www.clinicaltrials.gov, you can search by name of a medical condition, drug, or investigator.