One of the national computer models being used to help Memphis-area hospitals predict and prepare for the surge in COVID-19 cases comes with a warning label:

“There is a high degree of uncertainty about the details of COVID-19 infection, transmission, and the effectiveness of social distancing measures,” states the University of Pennsylvania’s disclaimer for its CHIME computer model. “Long-term projections made using this simplified model of outbreak progression should be treated with extreme caution.”

They are.

Local COVID-19 effort takes critical look at optimistic virus modeling

That’s why local epidemiologists are spending sometimes several hours a day discussing and debating the merits of that and other surge prediction models.

That’s why local and state government and health officials have disagreed in recent weeks about which models to use and whether they can and should create a model just for Shelby County or just for Tennessee.

That’s why scientists at Vanderbilt University have been working for weeks to develop a complex predictive model of the spread of COVID-19 within Tennessee, with region-specific projections. The results are expected this week.

And that’s why local officials were concerned when the national model state and federal officials have been relying on — the IHME model from the University of Washington —released a dramatically more optimistic COVID-19 forecast on Monday.

”The model that was presented (Monday) assumed 100 percent compliance on social distancing through the end of May,” Doug McGowen, the city’s chief operating officer and task force leader, said Tuesday. “We all know that that is not occurring today. That is the basis for us to re-examine that.”

Local officials are hoping the more optimistic numbers reported Monday show that strict social distancing policies are working.

They also fear that the less-dire numbers — which predicted a peak for the virus locally April 15, sooner than expected, with a lower number of cases and deaths that will not exceed the capacity of local hospitals — will give the public a false sense of security.

“We know there is a surge coming, that we have to prepare for that,” McGowen said. “Irrespective of the height of the peak, irrespective of the duration of the peak, we are preparing our health care system to be responsive to the needs of our community to the best of our ability.”

Local health and government leaders have been working to develop their own — and they believe more accurate —predictive model.

But those efforts have been stymied by Gov. Bill Lee and state officials, who began pushing Vanderbilt nearly three weeks ago to develop a state-specific model.

“I have asked for a modeling that is unique to our state,” Lee said on March 26, “and we think we’ll have that model within a day or so, having informed decisions around next steps.”

A day turned into a week as city and county officials waited for the new model to help them predict what the Memphis-area health care system would be facing in the weeks ahead.

The task turned out to be a lot more complicated than anyone imagined.

“Doing this so fast, and with such a novel and fast-moving disease, has been a new challenge for us,” Dr. John Graves, Vanderbilt’s associate professor of health policy, said Monday.

“Our goal, now that we have a model up and running, is to, in short order, provide policymakers across the state regular modeling estimates they can use to make decisions. This model can predict the trajectory of infections, and we can use that information, along with data on the effects of policies, to estimate what we need to do to keep the spread and consequences of COVID-19 in check. The ultimate goal is to know when it is possible to start to return to something more like a normal life.

******

While state officials have been waiting for the Vanderbilt model, local officials in Memphis and across the state have been pushing the state to help them prepare for the COVID-19 surge.

On April 1, in lieu of having its own state model, the state released surge predictions from the model developed by the University of Washington’s Institute for Health Metrics and Evaluation, or IHME.

It’s the nation’s most influential and controversial COVID-19 model.

It’s the same model the White House relied on to predict a national surge by April 15.

At a March 30 press briefing, Dr. Deborah Birx, the White House coordinator for its coronavirus task force, said government scientists had reviewed a dozen predictive models.

But she singled out the IHME model, which she said was rich in real time data about the effects of efforts to “flatten the curve” of demand on medical resources. It “assumes full mitigation”, is adjusted every morning with new data, and depends on the behavior of the population.

Strickland: New COVID optimism based on social distancing

Birx said the IHME model, in particular, was being used to guide President Trump’s decision to extend stay-at-home guidelines through April 30.

That’s the same model Gov. Lee used last week when he ordered all Tennesseans to stay home at least through April 14.

“I want to speak directly to every Tennessean,” Lee said. “The month of April stands to be a very tough time for our state. COVID-19 is an imminent threat and we need you to understand that staying home isn’t an option. It is a requirement for the swift defeat of COVID-19 in Tennessee.”

Early Monday morning, the University of Washington released updated projections that were dramatically less dire than those that were used by Trump, Lee and others.

The original model suggested that Tennessee would exceed its hospital capacity by Thursday, April 9.

The updated model said it wouldn’t even come close to doing that at any point.

The original model predicted that Tennessee would suffer more than 3,000 COVID-19-related deaths.

The updated model said the number of deaths caused by the novel coronavirus will not exceed 600 statewide.

“The model is encouraging,” Lee said Monday.

Health Commissioner Lisa Piercey agreed. “Our new cases each day appears to be slowing down,” she said. “That means what you’re doing is working. This is not the time to let up on the effort.”

Memphis Mayor Jim Strickland said Monday that the numbers were encouraging, but he cautioned that “models change every day” and that the public should continue to limit social contacts.

To emphasize the point, he extended the city’s “Safer at Home” order for two weeks until April 21.

”We can’t let up,” Strickland said. “We certainly hope for the best but we must continue to plan for the worst. ... The virus feeds on social interaction. We need to starve the virus.”

******

Federal, state and local officials across the country are using various computer models to help them predict and prepare for COVID-19 surges.

None are perfect and all have flaws, said Dr. Xinhua Yu, an epidemiologist and biostatistician at the University of Memphis. He has been working with local officials to assess various models.

”Currently, almost all models are based on aggregated data and target population dynamics, and did not consider race, gender, health status, income, etc.,” Yu said.

”Some may consider age and population density in the model, but the most popular models do not incorporate into these factors. Ideally, these are important factors that should be included in the models.”

Demographic variables such as race, income and underlying health status can severely impact the spread and impact of the disease.

That matters, especially in a predominantly minority community like Memphis.

Local social distancing policies seem to be working; new surge estimates show fewer deaths

County health officials announced Wednesday that COVID-19 is having a disproportionate impact on African Americans in Shelby County.

An analysis of 230 of the confirmed 897 COVID-19 cases in the county showed that African Americans account for 68 percent of those who have tested positive for the disease, and 71 percent of virus-related fatalities.

African Americans make up about 54 percent of the county’s population.

Those numbers reflect similar racial disparities being found among COVID-19 victims in Illinois, Michigan, North Carolina, South Carolina and other states, according to an analysis by The New York Times.

In Shelby County, the analysis also showed the people with underlying heart conditions made up 83 percent of the cases, those with diabetes 39 percent, and those with asthma and other lung issues 22 percent.

Locally and nationally, African Americans are more likely to be uninsured and have existing health conditions.

They’re also more likely to be living in poverty, and to have jobs that don’t allow them to work from home, the Times reported.

”This pandemic is highlighting the health inequities that have existed forever,” said Jenny Bartlett-Prescott, chief operating officer of Church Health who is leading the local task force’s testing subgroup.

”The poor and under-served are more likely to be suffering the economic, social and health issues that make them more vulnerable to the coronavirus and less able to fight it.”

******

Those design gaps will make it even more difficult for any computer model to predict the COVID-19 surge in Memphis.

The goal is to help government officials decide how many more beds, equipment, supplies and personnel are needed to prevent hospitals from being overwhelmed by COVID-19 cases.

Some models predict the surge will hit local and state hospitals early next week. Others predict it will hit later in the month or early next month.

The models also differ on estimates of how many people with COVID-19 will need to be hospitalized, and how many might die.

The IHME model predicts the surge will hit Tennessee much sooner (April 15), and be much steeper, but be over in about a month.

Another widely used model, the CHIME model predicts the surge will hit much later (mid-May to early August), and last longer (until November), but won’t be nearly as steep or lethal.

Predictive models are important and helpful, but can’t be taken as gospel, said Dr. Manoj Jain, the infectious disease expert and epidemiologist who is advising the local COVID-19 Task Force.

”You can make health care improvements if you don’t look at the data,” he said. “It’s important to analyze all of the numbers, but to keep in mind that models vary and numbers change, especially during an epidemic. We need all of the information we can get, but no one model will not have it all.”

That’s the warning label local health experts and government officials want to apply to any and all models being used to predict the surge.

”Any prediction relies on the available data, and at the time of emerging epidemic, things evolves very fast,” Yu said.

”Over the past few weeks, we have seen many models, some are rigorously done by academics and published in peer-reviewed journals, some are produced by amateur modelers. None of them is perfect, as statistician George Box once said: ‘All models are wrong, but some are useful.’”

Jain, Yu and other local epidemiologists have been analyzing several models, but are particularly attuned to one created last month by the University of Pennsylvania Health System.

It’s called the COVID-19 Hospital Impact Model for Epidemics, or CHIME model.

CHIME is based on something called SIR modeling. It predicts the number of people in a particular location who are Susceptible, likely to become Infected, and likely to be Removed — to recover or die.

Those calculations are based on real and changing numbers and ratios of people in a particular location who actually have been infected, hospitalized, and recovered or died.

Those numbers change daily by hospital, city, county, state and country.

The reporting of those numbers also varies from place to place, and can lead the model to underestimate or overestimate.

Variables include the number of people who are being tested, when test results are received (one day after testing, seven days after, and so on), how many who test positive never go to the hospital, and more.

In Shelby County, some test results are received within hours; others have taken more than two weeks.

That can dramatically effect a key variable known as the “doubling rate” — the rate at which the reported number of COVID-19 cases are doubling.

Currently, Shelby County’s doubling time is 8 days, Alisa Haushalter, executive director of the county health department, said Tuesday, April 7.

The number of new cases is reported daily, but those numbers vary based on when test results are received. Some test results are received within a day, others have taken up to two weeks or longer.

The daily case numbers also vary depending on the availability of testing, which has increased in fits and starts.

It’s also unknown how many people actually have COVID-19, because some cases are very mild and aren’t tested.

”In this COVID-19 epidemic, we have some unique challenges,” said Yu. “For example, the number of cases are heavily affected by the availability of testing kits and detection ability. At the early stage, there is a significant under-report and delay of reporting in COVID-19 cases due to detection issues.”

******

The more controversial IHME model, used by the U.S. and Tennessee, differs from the CHIME model in several significant ways.

The IHME model “assumes 100 percent social distancing through May 20.

”That is unrealistic and that is not happening today,” McGowen said April 7.

The CHIME model does not attempt to estimate the impact of social distancing and other mitigation policies.

”The SIR model itself is a simplified epidemic model that does not account for the structure of contact networks and resultant transmission,” UPenn acknowledged.

”This may be particularly important in the case of COVID-19, because of rapidly shifting regional policies on physical distancing, school and university closures, cancellation of public gatherings, and shelter-in-place advisories.”

The CHIME model does allow hospitals to provide their own estimates of “the percent reduction in social contact going forward.”

It also allows each hospital to factor in its community’s own particular “doubling rate” — the rate at which the reported number of COVID-19 cases are doubling.

The range is from two days (as reported in Italy) to 10 days (as reported in South Korea).

By providing a range of inputs, the CHIME model produces a range of outcomes for each hospital and location — from “best-case” to “worst-case” scenario.

As a result, the CHIME model provides a range of surge dates, surge cases and hospital needs, and a range of potential deaths.

The IHME model also produces a range of dates and estimated numbers. But its overall projections tend to be more optimistic — because it assumes 100 percent compliance with social distancing.

Another key difference: The IHME model is the only model that offers state-by-state estimates. But each state’s numbers are deeply impacted by national and international numbers.

The original IHME model, funded by the Bill & Melinda Gates Foundation, was based on the trending curve of deaths from China, where the novel coronavirus was first detected.

Those numbers from China were sort of superimposed over emerging data from U.S. cities and counties to predict what might happen here.

The dramatic changes in Monday’s projections were attributed to the fact that the model was updated.

It now includes surge curves from Italy and Spain where epidemics have peaked, as well as states such as Florida, Virginia and Louisiana where cases are peaking sooner than expected.

The revised forecasts reflect “a massive infusion of new data,” Dr. Christopher Murray, IHME director at the University of Washington’s School of Medicine, said in a statement that accompanied the updated figures.

“As we obtain more data and more precise data, the forecasts we at IHME created have become more accurate,” Murray said. “And these projections are vital to health planners, policymakers, and anyone else associated with caring for those affected by and infected with the coronavirus.”

The CHIME model is more localized and can be used by any individual hospital or group of hospitals to make predictions.

For example, the CHIME model can be used to account for the likelihood that Shelby County’s health care system also will be taking care of COVID-19 patients from North Mississippi and East Arkansas.

Epidemiologists on the local task force have been analyzing both the IHME and CHIME models. They’ve also been looking at a third, much more pessimistic model called Covid Act Now, or CAN.

Some state officials are concerned about how the federal government is using IHME’s lower estimates to deny states’ increasingly desperate requests for equipment and help in preparations, according to The Washington Post.

“It’s unclear exactly what the White House is doing on this front,” Dylan George, who helped the Obama White House develop models to guide its Ebola response in 2014, told the Post.

“As a result, you have every state trying to create their own models to anticipate their needs. And you have one model like IHME being adopted as the national guide.”

Some states are using multiple models to predict and prepare for the surge.

Illinois is using a version of the CHIME model as well as models from three universities. New York is using at least four different models, including IHME.

Vanderbilt’s model will incorporate aspects of several other models, including IHME. But it will use data specific to Tennessee and its counties.

“Our team has access to Medicare data related to the specific geographic locations of health care providers and patient flows from ZIP codes to hospitals,” Graves said.

“One of the unique things about our model is that it is tailored to Tennessee, and each sub-state region even has its own model we can use to project out where things are heading. The fact that it is a Tennessee-specific model allows us to do more refinement based on our own local experience, and we don’t have to rely on a national model.”

UPDATE: The local task force is looking at a third, even more pessimistic model that hasn’t received as much attention as the first two.

It’s called Covid Act Now, or the CAN model. It was developed by a large team of data scientists from several universities and such companies as Google, Yelp and The New York Times.

The model “was developed to provide decision-makers at the state, local, and federal levels the information they need to advance effective policy responses to COVID-19,” the team explains on their website. “The model and its predictions are not intended to predict the future and should not be used to assume specific numbers of cases, hospitalizations, or deaths.”

Like IHME, the CAN model provides county-level forecasts. It also factors in varying levels of social distancing compliance. But unlike the others, CAN tries to factor in longer-range estimates, including probable additional waves of infection.

The IHME model predicts that Shelby County won’t need any additional hospital beds to handle the surge that will hit in about 10 days.

The CHIME model predicts that the county might need up to 1,000 additional beds by the time the surge hits next month.

The CAN model suggests that Memphis-area hospitals will be overloaded in four to eight weeks, and will need up to 2,000 additional hospital beds, depending on when and how much stay-at-home orders are relaxed.

Local task force members are debating which models to use and whether any of them can be trusted.