Why COVID-19 case counts are so unreliable

Early in this pandemic, a relative of mine developed a fever, difficulty breathing, and a persistent cough. We are here in the original epicenter of the disease in the United States — King County, Washington — and so there was heightened fear among health systems about being overwhelmed by patients coming in with the disease and a lack of tests to accurately diagnose the disease. Being a journalist by training, I called everyone I could to try to get my relative tested. Finally, we were able to get my relative seen by a health care provider. But the doctors refused to test them for COVID-19, saying that the strict criteria helping health care professionals ration tests had not been met.

So, we will never know. And, more importantly, that is an example why case counts are so unreliable with COVID-19. Even people who are sick and think they have COVID-19 have not been able to get tested for the disease.

As headlines proclaim that the number of cases has risen above 1 million in the United States, reporters should use the opportunity to explain to their audiences how misleading case counts are.

We are getting a real time lesson in all aspects of epidemiology, an exciting time for public health professionals but a potentially confusing time for everyone else. Many people may assume that there is some sort massive system for accurately counting what is making people sick at any given time. Most reporting reinforces this idea when we see stories proclaiming very specific numbers of cases for different diseases.

There are four important things to remember and try to convey to your audiences as you continue to report on the COVID-19 pandemic and on health conditions more broadly.

1. Everything is an estimate. There is no central repository of disease counts. Everything you are seeing in the scientific literature, in reports from advocacy groups, in press releases from government agencies, is an estimate. The only way to truly count each illness would be for every person to be tested and accurately diagnosed whenever they felt symptoms of an illness and then for that information to be tracked in a centralized system. Setting aside whether this type of system would be desirable, it’s certainly not practical. The next best thing is a system that tracks people who seek health care. Again, accurate testing and diagnoses are critical. 

Even in countries that have comprehensive health care systems that cover care for everyone, there are people who are never diagnosed with a disease — even when they have the disease. And even in those systems, there are people who are misdiagnosed with a disease. (More on that below.) But in the United States, disease counting is especially difficult because of the patchwork health care system we have. You can base some estimates on Medicare numbers, but that skews toward older Americans. A combination of Medicare and Medicaid will result in numbers that disproportionately represent people who are older, lower income, or have disabilities. Every individual private health care system will have its own patient profiles, too. So, at the heart of every estimate you see out there is an incomplete set of source data.

2. Counts rely on reports. Not only are very few people being tested for this disease, but the reports on those who are being tested or diagnosed (an important distinction) are not being seamlessly delivered to the public. One thing that has surprised me, despite so much sophisticated reporting on this pandemic, is the inability of journalists and pundits to understand that numbers will spike simply because of a backlog of cases being reported all at once. Think about a hospital setting. If patients meet testing criteria, it may take days for them – or the hospital – to receive results. The hospitals themselves are often overwhelmed with patients. They don’t necessarily report the test results they do receive immediately to some government agency that may be compiling counts. So those tests and diagnoses pile up. We have seen the effect of case counts dropping over the weekend as hospital staffs take a break and then surge early in the week as the backlog of cases end up being reported. As a result, the case curves can look oddly erratic, with big spikes that are actually artifacts of how we’re testing and reporting those results.

3. Cases are often misdiagnosed. This isn’t just true with COVID-19, it’s true with diseases that have been around for decades. A recent study in BMJ showed that Parkinson’s disease might be wrongly diagnosed a third of the time. The researchers wrote, “Pathological examination of the brains of patients with a clinical diagnosis of Parkinson's disease shows a different diagnosis in up to 35% of cases.” False negatives and false positives have been a persistent problem with COVID-19 tests.

4. We are building the plane while we are flying. As you can see from everything above, it’s hard enough to nail down an accurate count of diseases that are well known and understood. Magnify those problems by 100 when you have an emerging disease. We have seen this with every disease that has emerged in modern history. If you look back at the coverage of HIV/AIDS, for example, you will see case counts that were wildly fluctuating. And the case count problems with HIV/AIDS persist to this day because not everyone is accurately diagnosed. With COVID-19 we don’t fully understand how the novel coronavirus works. We don’t fully understand when people are most likely to transmit the disease. We don’t fully understand why some people seem to be more susceptible to becoming ill than others. And we don’t fully understand whether developing antibodies to the disease actually makes you immune. Those kinds of unknowns — and there are many more — have an impact on estimates of the number of cases.

So don’t be surprised when you see the number of COVID-19 cases surge or drop or stay flat and then go up or down. Use this time to explain to your audience why case counting is harder than one might think. Those become essential caveats when states are considering relaxing social distancing orders based on such counts.