Problems with the Santa Clara Study
One of the first studies of its kind, researchers from Stanford recruited 3,330 people via online advertising (Facebook and Reddit) for coronavirus antibody testing. They concluded that the actual number of cases may be 50 times the number of confirmed cases in the County, and the mortality rate may be as low as the flu (0.12%). There were many problems with this study.
- The advertisements read like this:
Why get tested? (1) Knowledge — Peace of mind. You will know if you are immune. If you have antibodies against the virus, you are FREE from the danger of a) getting sick or b) spreading the virus. In China and U.K. they are asking for proof of immunity before returning to work. If you know any small business owners or employees that have been laid off, let them know — they no longer need to quarantine and can return to work without fear. If you don’t want to know the results, we don’t need to send you the results. (2) Research — Contribute to knowledge of the prevalence of virus spread in Santa Clara County. This allows researchers to plan hospital bed needs and forecasting where to allocate public health resources. This will help your neighbors and family members.”
This ad encourages people who think that they have been exposed or were sick to get the testing. This selects for people who are worried about having it or having had it. While the study asked the test subjects if they had symptoms, the researchers didn’t actually use that information in a meaningful way. Basically, this type of ad selects for a higher proportion of COVID-19 positive people.
2. The test has a false positive rate of 0.5%. That means when you test 3330 people, 17 people will have a false positive. (3330 * 0.005 = 17) The total number of people who tested positive in this study was 50. Of the 50 positives, a very significant portion (17) may have been false positives. This was not discounted in the results and analysis. The researcher then extrapolated the 50 positives to say that there are thousands of positives in the County.
3. To calculate mortality rate, they used a reference study from Seattle of 24 ICU patients (reference 22). It was a case series to look at symptoms and clinical course for ICU patients, not a study dedicated to determine length of time to death. The patient selection criteria was simply an ICU stay at one of three hospitals. 12 of the patients died. The study did not report the length of time from infection to death. The only data they had was when the patient reached the hospital. They estimated that the average time from symptoms to hospitalization was 7 days. The Stanford authors must have taken the data from the 12 who died to extrapolate that average time to death was 21 days. But there was no way to know that for sure based on what was reported. Since this disease transmits exponentially, being a few days off for time to death will drastically change the projected mortality rate.
4. The researchers assume that Santa Clara County has accounted for all COVID-19 deaths. They report 50 deaths as of April 10, 2020. We all know testing has been inadequate. There were likely people who died of COVID-19 prior to us starting to test for community spread. Santa Clara County was the site of one of the first confirmed cases of community spread in the country. So the virus was there, and it was killing before we knew to test for it. There are likely elderly people who died at home, never presenting to the hospital for testing. The virus causes end-points other than fever and coughing, such as heart infections, strokes, falls, that confuse the situation and may not have resulted in testing in February-March. (By April, there was enough awareness of these confounding factors that most hospitalizations are likely being tested now.) There have been far more than just 50 deaths caused by the novel coronavirus in the County.
Summary:
- This study overestimates the number of cases in Santa Clara County.
- This study underestimates the mortality rate.
The paper was published pre-print, prior to peer review, which means that other scientists had not had a chance to help the study authors more critically look at their data. Meanwhile, the researchers have embarked on a media campaign, rather than addressing the numerous short-comings that peers are now pointing out. It suggests that they are serving other interests — perhaps those of the study’s donors. Of note, there is a hedge-fund manager (Andrew Bogan) listed as an author.
This study is being used by some people to say that the mortality rate isn’t very high, and that we should loosen the social distancing measures. The problem all along hasn’t been the actual mortality rate. The problem has been that 10–20% of adults (depending on age) who catch the virus need to be hospitalized. If our hospitals are overwhelmed, as they were in certain cities, the mortality rate will be over 1%. If our hospitals can accommodate the ill, have the oxygen, the ventilators, the medications, and the staff, we can keep the mortality rate to under 1%, perhaps even closer to 0.5%.
This article calculates the fatality rate to be 0.12–0.2%. The flu is about 0.1%. These calculated rates do not jive with this chart based on national mortality data. While the research is greatly appreciated, the statistical analysis is deeply flawed.
The countries that have been able to keep the virus suppressed through testing and quarantine have been able to keep their mortality rates down. And they’ve been able to keep businesses running. The key is to suppress transmission through increased testing, quarantine, and tracing contacts. Get those right, and we can and should be able to re-open ASAP.
Wei-Shin Lai M.D. is the co-founder and CEO of AcousticSheep LLC, makers of SleepPhones® headphones, the most comfortable headphones you can wear in bed. She was once offered a job with the CDC as an epidemic intelligence officer.
