On Being a "Numbers Person" in the Covid-19 Era

On September 8, 1994, my husband and I traveled with our 3-year-old son and his 9-month-old twin brothers to my parents’ home near Rhode Island. Waiting at the airport for our red-eye flight to the airline’s hub in Pittsburgh, we learned that a plane bound there, operated by the same airline, had crashed in the woods a few hours earlier, killing all 132 people on board. When we arrived at Green Airport the next morning, I was surprised to see a television crew waiting for us. “Were you frightened?” a reporter asked me. I had been up all night with three small children and said the first thing that popped into my head. “No, I wasn’t frightened,” I replied. “It’s a low-probability event.”

My calm response aired that evening, alongside that of a woman who was sobbing. “Oh, those poor people!” she cried. Of course, I had not been asked whether I was sorry for the loss of life; I had been asked if I was frightened by the crash, and statistically, there was no reason to be. Yet, with some amusement, I realized that sometimes, being a “numbers person” can make one appear to be … well … a little uncaring.

I have sensed this perception in responses to biostatisticians who have questioned the evidentiary basis behind governmental response to the Covid-19. Some appear to suggest that these biostatisticians are unaware of either the numbers of “coffins of Covid-19 victims” or the exponential nature of infectious disease transmission that puts an increasing number of people at risk of death. I would suggest the opposite is true. The issue is, it seems to me, not lack of awareness; it is a heightened awareness that defies comfort—an all-too-complete picture of the consequences, not just of the disease, but of the lockdowns intended to combat it.

When I hear calls for protection of the vulnerable, I echo them, of course. But which vulnerable people? We are all familiar with the demographic and clinical conditions that put one at higher risk for severe Covid-19 illness. Yet, as I write this, an estimated 22 million U.S. residents have been laid off because of business and school shutdowns, and more layoffs are coming. If prolonged, the resulting economic deprivation will dramatically increase their risk of numerous negative health outcomes, including a doubling of their odds of cardiovascular disease, our nation’s top cause of death. The immunosuppressive effects of chronic stress are well-documented, ironically meaning that some paycheck-to-paycheck earners may be put at a heightened risk of infection from the very policies intended to help them avoid it. Suicide hotlines are already seeing spikes in calls from persons who are anxious, isolated, or worried about their economic futures, and persons of low income and educational levels are reporting more financial stress than are their higher-income/education counterparts.

Covid-19 is an important public health issue. So, too, are unemployment, cardiovascular risk, chronic stress, loneliness, and despair.

What is the actual risk associated with Covid-19 infection? Legitimate questions on this point merit more than the usual vague bromides about listening to “the experts.” One might first ask, which experts? From the beginning of the Covid-19 outbreak, most top biostatisticians have been making essentially the same observations: (1) without an accurate denominator of the total number of infections, both symptomatic and asymptomatic, it is impossible to estimate disease severity; (2) it is important to account for sources of regional variation (e.g., population age, hospital bed occupancy) and competing causes of death (i.e., not every death with Covid-19 infection is a death from Covid-19 infection); (3) despite media coverage suggesting otherwise, the risk of Covid-19 death is remarkably low for those not in the known vulnerable groups; and (4) exaggerated estimates of harms and some extreme measures themselves can be harmful to public health (e.g., public panic, hoarding of essential supplies). Elsewhere, a research intern and I observed that the Imperial College model that was the basis for the shutdowns did not meet fundamental professional standards and had likely systematically overestimated projected ICU stays and deaths. A much higher-quality model from Oxford reached very different conclusions and, in a sign of a solid evidence-based approach, called for population serological surveys to validate or refute the modelers’ results. Other experienced modelers are examining disease trajectories by country, finding few differences between countries with versus without lockdown policies.

All this evidence is preliminary, and there is certainly no shortage of disagreement about data interpretation. We do not yet have the most important information needed, estimates of disease severity with accurate denominators, although one preliminary study in the Gangelt region of Germany based on antibody testing concluded that the infection fatality rate was 0.37%. While we wait for more antibody test results, what is a “numbers person” to do?

For a “numbers person” who has the misfortune of having a tender heart, as I do, watching the unfolding of a process not based on reliable evidence has been difficult. The policy disagreement has never been about whether we "social distance." It has always been about how we do so, and whether reliance on better evidence could have produced more targeted, smarter strategies with less collateral damage to our nation's economically vulnerable.

In that context, it seems to me that the best response of a “numbers person” is to rely on the methodologically higher-quality work in reaching conclusions about Covid-19, update those conclusions as new evidence comes in, and advocate for highest-quality data to support decision making. Lest that sound too much like my measured response to the reporter 25 years ago, I would suggest that when one considers the harms as well as the benefits of Covid-19 lockdowns, calling for an evidentiary basis for decision making is the very opposite of uncaring. It is, in fact, the only caring thing for a "numbers person" to do.

Numerators, Denominators, and COVID-19

Numerators, Denominators, and COVID-19

After a seven-year absence, this blog is returning, just in time to tackle what will surely become one of the most discussed debates in public health for years to come—how best to manage the COVID-19 infection pandemic currently gripping the world’s attention. A recent provocative article in STAT by widely respected expert in evidence-based medicine John Ioannidis suggests that sweeping and consequential public policy decisions are being made “without reliable data.” The result, he suggests, may be “a fiasco in the making.” The article is well worth reading for an evidence-based perspective on the data we have—and the critically important data that we do not have—on  COVID-19. Surprisingly, the question Ioannidis is raising may be described as one of numerators and denominators.

At JMCP, the most common mathematical errors we identified in submitted work were not made in sophisticated models or multivariate analyses; they were made in calculating and describing simple percentages. The main problem was—and seems to be today—confusion over numerators and denominators used to estimate risks. The fundamental question raised by Ioannidis isn’t whether COVID-19 is a pandemic, as it has been named as such by the World Health Organization. The question is how medically serious the pandemic is, and to know that, one requires the data needed for an empirically accurate calculation of infection-associated risk. Ioannidis argues convincingly that we do not have the data to make such a calculation.

When I teach analytics, I make three key points about percentages:

(1)    The most important word in describing a percentage is “of,” which identifies the group you are describing—who is at risk? This is the denominator. Ioannidis observes that the only known denominator today is the count of people who are both symptomatic and test positive. As Ioannidis points out, this is the well-known methodological problem of selection bias. The people selected for testing are those with symptoms; they do not represent the total infected population. So, the correct statement to describe most of the percentages cited for COVID-19 today should be: “Of those who are so sick that they seek health care and test positive for COVID-19, what percentage experience an adverse outcome?”

(2)    For a percentage to be accurate, the numerator must be an accurately described subset of the denominator. Estimates of the effects of the COVID-19 pandemic may be flawed by numerators mismatched to the denominators they are intended to represent. The 16% rate of serious illness cited by the CDC means that 16% of those who became so sick that they were tested in a hospital experienced serious illness (in China). Unfortunately, the State of Ohio, among the first to impose widespread closures, increased that number to 20% and applied it to all infected people, not just those who get sick. The result of this mathematical error is likely a substantial overestimate of the total number of persons who will require hospitalization.  

(3)    For a percentage to be accurate, both the numerator and denominator must be known and measured accurately. Neither is true of the numbers used to represent the effects of COVID-19 in the United States. Numbers reported to date reflect health systems and environments that differ greatly from those of the United States:

-The rate of health care-associated infection in Italy is nearly double that of the United States.
-The rate of daily smoking is 25% in China, 20% in Italy, and 11% in the United States.
-The death rate from air pollution (per 100,000 population) is 140 in China, 49 in Italy, and 24 in the United States.

I will leave it to others to draw your own opinions about the effects and merits of the policies in place now. While we await data, the members of my family and I are adhering to the CDC guidelines. Time—and accurate information, I hope—will provide us all with the empirical evidence we need.