Scholars familiar with reporting standards of the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network are aware of their importance: compliance with these guidelines facilitates transparency and accuracy in published work. What may be less obvious is that when these guidelines are followed, financial conflicts of interest in research are, arguably, rendered mostly or entirely harmless. Why? Because when authors report the critical information they should, and describe findings and limitations transparently as they should, readers are provided with accurate information, regardless of study sponsorship. Conflicts of interest should be disclosed, of course, but ultimately may have little effect on the credibility and usability of results published in compliance with transparency standards.
A recent case in point provides, unfortunately, a great example of the opposite situation, in which the authors of a systematic review article claimed to follow Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) standards but did not actually do so [1,2]. The article in question examined the effects of out-of-pocket cost sharing on prescription-filling behaviors[1]. Its authors concluded that the “published literature shows consistent [negative] impacts of higher cost sharing on initiation and continuation of medications” in a dose-response fashion. Regrettably, these conclusions were grounded in a combination of failure to conform to PRISMA standards and selective reporting of the available information, resulting in the provision of inaccurate information to the journal’s readers.
Nonadherence to PRISMA Standards
No Publication Bias Assessment. Foremost, the systematic review included no assessment of publication bias, deemed a PRISMA-essential standard because systematic exclusion of studies from available evidence threatens the core validity of any attempt to summarize the literature[2,3]. This omission is particularly concerning because publication bias is a known problem in research on cost-sharing policy[4].
One example is the MHealthy copayment-reduction cohort study, completed in 2009, whose a priori outcomes—medication adherence, health care costs, outpatient visits, and emergency/inpatient utilization—were never published[5]. Also never published were several important a priori outcomes in a pivotal randomized controlled trial (RCT) included in the review, the Post-Myocardial Infarction Free Rx and Economic Evaluation (Post MI-FREEE) study, including utilization rates for emergency departments, hospitals, and physicians[6-8]. Reporting of standard publication-bias metrics (e.g., funnel plot, Egger test) would have provided readers with critical information about the degree to which available evidence represents actual cost-sharing effects versus reporting proclivities and interests of study investigators[3,9].
No Risk-of-Bias Assessment. Another important but missing PRISMA element was quality (risk-of-bias) assessment[2,3,9]. The authors indicated they performed risk-of-bias assessments but omitted essential information: rating method, number of raters, resolution of inter-rater disagreement, and—most remarkable—results of the assessments, although these are critically important indicators of the validity of any systematic review[1,3,9].
No Certainty-of-Evidence Assessment Despite Published Cochrane Report. Finally, contradicting the authors’ assertion of consistency in results reported in the literature, they neither conducted an assessment of study heterogeneity nor reported the specific quantitative results of individual studies, with measures of precision around the estimates (e.g., confidence intervals), as recommended in PRISMA guidelines[2,3]. Notably, a 2015 Cochrane review on this topic found so much heterogeneity in available evidence that its authors reported both difficulty in summarizing results and low to very low evidentiary certainty[10].
Selectively Reported Results
Primary Outcome of Pivotal Trial Not Mentioned. Contrary to PRISMA guidelines for reporting outcomes of individual studies[2], the review authors omitted mention of the Post-MI FREEE RCT’s primary between-group result: a nonsignificant difference, comparing patients randomized to free medication versus usual coverage[7]. The authors chose instead to describe only a secondary outcome between-group result, characterized by the authors as “statistically significantly fewer major vascular events, revascularizations, or strokes” despite an extremely small effect size: rates of 21.5 versus 23.3 per 100 person-years (HR=0.89, 95% CI=0.80-0.99)[1,7]. Only 12.1% of patients randomized to free medication, versus 8.9% in usual coverage, were fully adherent (medication possession ratio >80%) to all 3 recommended medication classes[7].
These notably unimpressive results of providing free medication in Post-MI FREEE, although not mentioned by the authors of the systematic review, were highlighted in an editorial accompanying the Post-MI FREEE report, which referred to copayment-elimination effects as “distressingly modest”[11] and by one Post-MI FREEE principal investigator, who commented: “we gave these people the medicines for free and only half took it.”[12] Although seemingly counterintuitive, this relative lack of medication price sensitivity is expected because cost is not commonly endorsed by patients as a key cause of medication nonadherence. More commonly mentioned causes, described in one systematic review of patients after myocardial infarction, are beliefs about illness and medication, side effects, forgetfulness, and need for provider support and communication[13].
Inaccurate Description of the RAND Health Insurance Experiment (HIE). The review authors’ claim that their conclusions were “consistent with [those] found 50 years ago in the HIE” is directly contraindicated by the HIE findings, as described by HIE investigators[14]. Price elasticity (responsiveness) in the HIE, calculated from a comparison of 95% coinsurance with free care, was 0.2[15], similar to previously reported elasticities in health care and far less than the standard of 1.0 considered by economists to represent price sensitivity[8]. Despite evidence that both high- and low-value services were equally reduced by coinsurance in the HIE, the coinsurance group had fewer activity-restricted days, including time spent seeking health care, and less anxiety about health[14]. The only negative health effects of coinsurance, observed only in lower-income persons, were clinically minor and—importantly—due to higher case-finding rates, not to better treatment adherence[4,14,15]. The HIE investigators concluded that increased cost sharing for “the kind of people who typically are covered under employer health insurance” would likely result in “enormous potential savings“ with “little apparent health impact”[14].
What Went Wrong in This Systematic Review?
The problems in this review can be summarized in two phrases: poor-quality source literature and lack of transparency in reporting. Most research on cost-sharing policy has been plagued by serious methodological flaws including channeling bias, publication bias, and selective reporting[4,8,10]. PRISMA standards exist to inform readers of such limitations, facilitating accurate synthesis of research quality and outcomes[9]. That these standards were not followed in this systematic review is unfortunate and make its results uninterpretable.
Perhaps more important, though, is the process question of how errors so substantial were able to be reported and go uncorrected at all. I suggest that no one was served well by this process--not the journal's readers, who were provided with inaccurate information; not the authors, whose work could have benefited from the corrections that peer review should provide but did not; and not the study sponsor, the National Pharmaceutical Council, because the publication of information that lacks credibility does not benefit those who paid for it.
Only the journal’s editorial team can assess the root cause of this incident, and it is to be hoped they will choose to do so. Regardless, we should view the incident as a lesson learned about the importance of accurate and transparent research reporting: when it comes to quality-of-evidence assessment, it really is all about that basic.
References
[1] Fusco N, Sils B, Graff JS, Kistler K, Ruiz K. Cost-sharing and adherence, clinical outcomes, health care utilization, and costs: a systematic literature review. J Manag Care Pharm. 2022 April 7. doi: 10.18553/jmcp.2022.21270
[2]
Enhancing the QUAlity and Transparency of health Research (EQUATOR) Network.
PRISMA 2020 checklist. https://www.equator-network.org/reporting-guidelines/prisma/.
See also the expanded checklist at https://prisma-statement.org/documents/PRISMA_2020_expanded_checklist.pdf.
[3] Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160.
[4] Fairman KA, Curtiss FR. What do we really know about VBID? Quality of the evidence and ethical considerations for health plan sponsors. J Manag Care Pharm. 2011;17(2):156-174.
[5] Spaulding A, Fendrick AM, Herman WH, et al. A controlled trial of value-based insurance design – The MHealthy Focus on Diabetes (FOD) trial. Implement Sci. 2009;4:19.
[6] Choudhry NK, Brennan T, Toscano M, et al. Rationale and design of the Post-MI FREEE trial: a randomized evaluation of first-dollar drug coverage for post-myocardial infarction secondary preventive therapies. Am Heart J. 2008;156(1):31-36.
[7] Choudhry NK, Avorn J, Glynn RJ, et al. Full coverage for preventive medications after myocardial infarction. N Engl J Med. 2011p;365(22):2088-2097.
[8] Fairman KA, Curtiss FR. VBID, the PPACA, and FREEE medications: Did politics trump the evidence about cost sharing? J Manag Care Pharm. 2012;18(2):146-155.
[9] Haber SL, Fairman KA, Sclar DA. Principles in the evaluation of systematic reviews. Pharmacotherapy. 2015;35(11):1077-1087.
[10] Luiza VL, Chaves LA, Silva RM, et al. Pharmaceutical policies: effects of cap and co-payment on rational use of medicines. Cochrane Database Syst Rev. 2015;(5):CD007017.
[11] Goldman L, Epstein AM. Improving adherence—money isn’t the only thing. N Engl J Med. 2011;365(22):2131-33.
[12] Marchione M. Study finds many patients shun free heart drugs. AP. November 15, 2011. https://www.bostonglobe.com/news/nation/2011/11/15/study-finds-many-patients-shun-free-heart-drugs/vBUPZ18azF0zqICmnRlJOL/story.html
[13] Piekarz H, Langran C, Raza A, Donyai P. Medication-taking for secondary prevention of acute myocardial infarction: a thematic meta-synthesis of patient experiences. Open Heart. 2022;9(1):e001939.
[14] Keeler EB. Effects of cost sharing on use of medical services and health. Rand Corporation. 1992. https://www.rand.org/pubs/reprints/RP1114.html#:~:text=It%20concludes%20that%20cost%20sharing,restrain%20spending%20on%20health%20care.
[15] Manning WG, Newhouse JP, Duan N, Keeler EB, Liebowitz A, Marquis MS. Health insurance and the demand for medical care: evidence from a randomized experiment. Am Econ Rev. 1987;77(3):251-77.
