A mere two weeks ago, I noted how evidence from randomized controlled clinical trials (RCTs) can’t seem to kill the claims of advocates that the antimalarial drug hydroxychloroquine is a highly effective treatment for COVID-19. There were two bits of disinformation that I discussed in that post. First was the Newsweek article by a Yale epidemiologist named Harvey Risch that was rife with bad arguments, cherry-picked studies, and a reliance on less rigorous retrospective observational studies over the RCTs (linked to in my post) that have been published over the last couple of months. The RCTs have consistently failed to find a benefit to using the drug to treat COVID-19, be it in hospitalized patients, in patients with disease mild enough to be treated as outpatients (with or without the magic drug azithromycin, which Didier Raoult insists to be an essential component of treatment), or as post-exposure prophylaxis to prevent the development of symptomatic COVID-19 in patients who had had close contact with patients with known COVID-19. Second was a truly ridiculously incompetent “study” touted by the John Birch Society-like group masquerading as a medical professional society, the Association of American Physicians and Surgeons (AAPS), that claimed to find that countries that adopted the use of hydroxychloroquine to treat COVID-19 had fewer fatalities than those that did not use the drug.
Given the drip-drip-drip of negative studies of hydroxychloroquine to treat COVID-19 and my having written about the issue so recently, I didn’t think I’d have to revisit it again so soon. Boy, was I mistaken! Never did I suspect that somewhere, somehow, someone would double down on the disinformation first spread by AAPS, in order to continue to tout hydroxychloroquine to treat COVID-19. This comparison of various countries, as you might recall from my recent post, first made an appearance in a court filing by the AAPS in its lawsuit against the FDA to get it to reinstate its ill-advised emergency use authorization (EUA) for hydroxychloroquine to treat COVID-19, which was revoked in June as evidence accumulated that it did not work and might actually be dangerous, due to the cardiac arrhythmias that the drug can cause. You might also recall that Harvey Risch himself alluded to a claim that countries that used hydroxychloroquine had fewer fatalities from coronavirus than those that had banned its use to treat COVID-19, a claim so ludicrous that Risch really deserves to have his title of epidemiologist stripped from him unceremoniously. Certainly, many faculty at Yale are embarrassed by his poorly constructed arguments.
But let’s get to the “study,” shall we?
A shadowy cabal of “scientists” @CovidAnalysis try to do science on a website
If there’s one thing that I’ve learned, it’s that denialists never admit when their argument is refuted or so ridiculous as to deserve mockery. Instead, they double down, as a shadowy cabal claiming to represent legitimate scientists did late last week:
Here we a professional-looking website claiming to present results from a randomized controlled trial of hydrochloroquine.
No authors or affiliations are listed, just a twitter account. t.co/dYGzqr6mLz
— Carl T. Bergstrom (@CT_Bergstrom) August 7, 2020
The “study” on the website spread far and wide among the usual sources, such as FOX News:
Notice what the “study” is being called: “Early treatment with hydroxychloroquine: A country-randomized trial”. Here’s a hint: There’s no such thing as a “country-randomized trial”. It’s a meaningless term. Subjects were not “randomized by country.” They couldn’t have been. This is a retrospective study. (More on that later.) As biology professor Carl Bergstrom notes:
So what is a country-randomized controlled trial?
It seems to be something *made up* entirely for this article.
The term “country-randomized controlled trial” has NEVER been used prior to this article.
— Carl T. Bergstrom (@CT_Bergstrom) August 8, 2020
I agree. I’d never heard the term “country-randomized trial” before this website. None of this stops the website authors from publishing this “abstract“:
Many countries either adopted or declined early treatment with HCQ, forming a large country-randomized controlled trial with 2.0 billion people in the treatment group and 663 million in the control group. As of August 8, 2020, an average of 39.6/million in the treatment group have died, and 443.7/million in the control group, relative risk 0.089. After adjustments, treatment and control deaths become 82.0/million and 637.0/million, relative risk 0.13. Confounding factors affect this estimate, including varying degrees of spread between countries. Accounting for predicted changes in spread, we estimate a relative risk of 0.21. The treatment group has a 79.1% lower death rate. We examined diabetes, obesity, hypertension, life expectancy, population density, urbanization, testing level, and intervention level, which do not account for the effect observed.
This is all utter rubbish, methods, conclusion, and all, as you will see. It’s so bad that it reminds me of a study by two antivaxxers without any qualifications in epidemiology, Neil Z. Miller and Gary S. Goldman, that tried to correlate the number of vaccines in the recommended vaccine schedules of various countries with those countries’ infant mortality rates. So bad was the study that Miller and Goldman actually tried to use linear regression to correlate the number of vaccines with infant mortality rates, while cherry picking the countries chosen to be included in the analysis. @CovidAnalysis at least spared us the shenanigans misusing linear regression analysis, but he/she/it/they sure did do the same sort of cherry picking, as you will see.
Before coming back to this part, let’s go to the HCQ Trial website to look at a bit more, namely the claimed “methods”. There’s so much wrong in this website that it’s hard to know where to start, but I guess the best place is in the “conclusions” made by this “study”. No, on second thought, the best place to start is to mention that this is not a peer-reviewed study. It’s a website, nothing more. (It’s also a Twitter account, @CovidAnalysis.) It’s also a website whose creators are not just unknown but who went to a fair amount of trouble to hide who they are. The domain’s WHOIS entry is masked by Contact Privacy Inc. No one (that I know of) has yet been able to figure out who’s behind this website, but the FAQ claims:
Who is @CovidAnalysis? We are PhD researchers, scientists, people who hope to make a contribution, even if it is only very minor. You can find our research in journals like Science and Nature. For examples of why we can’t be more specific search for “raoult death threats” or “simone gold fired”. We have little interest in adding to our publication lists, being in the news, or being on TV (we have done all of these things before but feel there are more important things in life now).
They’re published in Science and Nature? Sure they are. Sure they are. Alternatively, maybe one of them has gotten published in one of those journals, just not on a topic having anything to do with COVID-19, coronaviruses, SARS-CoV-2, epidemiology, clinical trials, or any other scientific discipline relevant to determining if hydroxychloroquine is efficacious against COVID-19. My guess is that there are probably some scientists in this group who, like the scientists who attack climate science or evolution, are not actually climate scientists or evolutionary biologists and demonstrate it with their obviously shallow understanding of the relevant disciplines. I will also say, right here, right now, quite clearly that the proper way to do this sort of thing, if you’re a real scientist, is to publish it in the peer-reviewed biomedical scientific literature, not to publish an anonymous website, with no names and no affiliations. That reeks of astroturfing. In fact, this whole website reeks of astroturfing. (Astroturfing, for those not familiar with the term, means the promotion of a message as though it’s coming from the grassroots when in reality it’s coming from a company, political party, political group, or ideological group.)
Let’s dive in, though. It’s important to call out astroturfing, and it’s also important to post, for the record, an explanation of why this astroturf “study” is such bad science and does not show what its creators (whoever they are) want you to think it shows.
The “HCQ Trial”
The first thing that I noticed about this “study” is that it very aggressively co-opts the use of the language of RCTs. This is clearly to give the impression that what this group, whoever is in it, is doing is akin to an actual RCT. It’s not. Here is the current language:
Treatment. We investigate early or prophylactic treatment for COVID-19 with hydroxychloroquine (HCQ), which has been adopted or declined in different countries. Since the severity of COVID-19 varies widely based on age and comorbidities, treatment was generally only initiated in higher risk individuals. The primary endpoint was death.
Treatment groups. Entire countries made different decisions regarding treatment with HCQ, essentially assigning themself to the treatment or control group. For the purposes of this study, selection into the treatment or control group was based on the same information and is essentially random.
However, biology professor Carl Bergstrom caught @CovidAnalysis doing this:
Given the aggressive misuse of randomized controlled trial terminology—a mistake no public health professional would make—and the subsequent half-retreat without notification, I find it extremely hard to see how this could be anything other than a deliberate attempt to confuse.
— Carl T. Bergstrom (@CT_Bergstrom) August 8, 2020
I’m with Prof. Bergstrom. This backtracking implies that the authors of the website realized after seeing all the criticism online and on Twitter that they had gone a bit too far co-opting the language of RCTs to their deceptive purpose. Again, this reeks of astroturfing.
Let’s go through the “trial” itself, though. Certainly, whoever wrote this website is good at sounding “science-y” and as though they know what they’re talking about with respect to clinical trials. They did, for instance, cite a primary endpoint, death, and all good clinical trials have to have a primary endpoint that is measured.
Let’s consider, though, what constitutes a “randomized controlled clinical trial”. First off, RCTs are interventional studies, meaning that they study an intervention, be it a drug, a surgical procedure, implantation of a device, or whatever treatment. Most importantly, RCTs study interventions whose use can be controlled by the people doing the study. Say, for instance, that I want to determine if Drug X works to treat Condition Z. The classical RCT design would be to randomly assign subjects with Condition Z to receive either Drug X or a placebo, and ideally the study would be double blind, meaning that neither the subjects nor the investigators would know which subjects were getting Drug X or placebo. (The reason for this is because, for subjective symptoms, there are “placebo effects”, in which some subjects receiving placebo will report improvement even though they are receiving an inactive compound.) Even better would be “triple blind,” in which the statisticians analyzing the data from the trial don’t know which group received the experimental drug and which received the placebo until after they had done the analysis, after which everyone is “unblinded.”
Another important point here is that the randomization occurs at the level of the individual, and it also tries to balance out the control group and the group receiving Drug X in such a way that they are comparable in age, sex distribution, disease severity, and comorbidities, so that there isn’t a result that’s not due to the use of the drug; e.g., if one group is significantly older than the other or has significantly more serious disease than the other. Indeed, much effort goes into making sure that treatment assignment results in groups that are well-balanced in age, disease severity, and other potential confounding factors that might affect the primary outcome being studied. Finally, RCTs are prospective. That means that the subjects are assigned to treatment group before treatments are administered, not after, and then outcome observations are made. Moreover, the variable (treatment) being studied is independent, and the outcomes are the dependent variable.
None of these characteristics apply to the HCQ Trial. Countries were not “randomized” to receive hydroxychloroquine or not. Countries made decisions about whether to use hydroxychloroquine based on their leadership, their situation, belief of the medical profession in each country regarding whether hydroxychloroquine might work, and many other factors, known and unknown. This doesn’t even account for differences in the use of hydroxychloroquine in different areas of the same country. I’ll refer to Harvey Risch’s claim that I mentioned in my recent post, where Risch referred to a “natural experiment” in Brazil in which the northern Brazil state of Pará purchased 75,000 doses of azithromycin and 90,000 doses of hydroxychloroquine in April, supposedly resulting in a 7/8 decline in mortality within a month. The bottom line is that there is no way one can “randomize” countries to use a drug or not. It’s gibberish. It’s nonsense. The claim is an insult to the intelligence of epidemiologists and clinical trialists, who will immediately recognize it for the BS that it is. Unfortunately, if you don’t know anything about epidemiology or clinical trials, you might think this characterization sounds reasonable, leading to the conclusion that the use of hydroxychloroquine resulted in 79% fewer deaths per capita than in nonuse of hydroxychloroquine did, on a country-by-country basis.
If we were to give @CovidAnalysis the benefit of the doubt (which I’m not inclined to do but will do for a moment for the sake of discussion), taking this “study” at face value, I would characterize it as a retrospective ecological study, as was pointed out on Twitter:
You can also tell this, in part, by this passage:
We focus here on countries that chose and maintained a clear assignment to one of the groups for a majority of the duration of their outbreak, either adopting widespread use, or highly limiting use. Some countries have very mixed usage, and some countries have joined or left the treatment group during their outbreak. We searched government web sites, Twitter, and Google, with the assistance of several experts in HCQ usage, to confirm assignment to the treatment or control group, locating a total of 193 relevant references, shown in Appendix 12. We excluded countries with <1M population, and countries with <0.5% of people over the age of 80. COVID-19 disproportionately affects older people and the age based adjustments are less reliable when there are very few people in the high-risk age groups. We also excluded countries that adopted early widespread use of masks because these countries tend to have significantly lower spread, which we discuss in detail below.
Notice how the authors are doing all sorts of adjustments for confounders after assignment? That’s something that’s necessary in retrospective studies. I will admit to being somewhat amused by the mention of excluding countries that adopted early widespread use of masks. In general hydroxychloroquine believers tend also to be antimaskers (i.e., people who falsely claim that the widespread use of masks doesn’t slow the spread of COVID-19 and might even be harmful.) It’s funny how, when the rubber hits the road and hydroxychloroquine believers want to try to sound convincing, they actually concede that the widespread use of masks can dramatically slow the rate of COVID-19 spread. I also can’t help but cite Edward Nirenberg here:
If it isn’t, then I demand to see proof of informed consent for study participation for the 2.7 billion people in the clinical trial.
Anyway, from this first sentence alone, you can be sure that everything that follows from this is at best prevarication, but let’s continue.
— Edward Nirenberg (@ENirenberg) August 8, 2020
Yes, it’s utterly ridiculous to claim 2.7 billion subjects as the sample size for this study. Even if such a study could be done, it would be utterly unethical, absent informed consent, and, like Ed, I’d love to see those 2.7 billion signed informed consent pages.
I also note, as Ed does too, that per capita mortality is not the appropriate primary endpoint, given that per capita mortality will be highly dependent on the prevalence of COVID-19:
this metric may quite literally just indicate the difference in the prevalence of COVID-19.
There is nothing redeeming about this piece.
— Edward Nirenberg (@ENirenberg) August 8, 2020
But let’s get back to what this trial really is, a retrospective observational ecological trial, and that’s even assuming that the authors did things correctly, something I do not assume. What is an ecological trial? Basically, it’s an epidemiological study in which the unit of analysis is not the individual person, but rather the group. I’ve discussed this issue before (here and here, among other times), in particular the “ecological fallacy,” which states that ecological studies are particularly prone to false positives. One of the best explanations of the ecological fallacies I’ve seen is from an epidemiologist by the ‘nym of EpiWonk. Unfortunately, the article is no longer there. Fortunately, there is the almighty Wayback Machine at Archive.org, where EpiWonk defines the ecological fallacy as “thinking that relationships observed for groups necessarily hold for individuals””:
The ecological fallacy was first described by the psychologist Edward Thorndike in 1938 in a paper entitled, “On the fallacy of imputing the correlations found for groups to the individuals or smaller groups composing them.” (Kind of says it all, doesn’t it.) The concept was introduced into sociology in 1950 by W.S. Robinson in 1950 in a paper entitled, “Ecological correlations and the behavior of individuals,” and the term Ecological Fallacy was coined by the sociologist H.C. Selvin in 1958. The concept of the ecological fallacy was formally introduced into epidemiology by Mervyn Susser in his 1973 text, Causal Thinking in the Health Sciences, although group-level analyses had been published in public health and epidemiology for decades.
To show you one example of the ecological fallacy, let’s take a brief look at H.C. Selvin’s 1958 paper. Selvin re-analyzed the 1897 study of Emile Durkheim (the “father of sociology”), Suicide, which investigated the association between religion and suicide. Although it’s difficult to find Selvin’s 1958 paper, the analyses are duplicated in a review by Professor Hal Morgenstern of the University of Michigan. Durkheim had data on four groups of Prussian provinces between 1883 and 1890. When the suicide rate is regressed on the percent of each group that was Protestant, an ecologic regression reveals a relative risk of 7.57, “i.e. it appears that Protestants were 7½ times as likely to commit suicide as were other residents (most of whom were Catholic)….ln fact, Durkheim actually compared suicide rates for Protestants and Catholics living in Prussia. From his data, we find that the rate was about twice as great among Protestants as among other religious groups, suggesting a substantial difference between the results obtained at the ecologic level (RR = 7.57) and those obtained at the individual level (RR = 2).” Thus, in Durkheim’s data, the effect estimate (the relative risk) is magnified by 4 by ecologic bias. In a recent methodological investigation of bias magnification in ecologic studies, Dr. Tom Webster of Boston University shows that effect measures can be biased upwards by as much as 25 times or more in ecologic analyses in which confounding is not controlled.
Another epidemiologist, Gideon M-K (a.k.a. the Health Nerd) explains this fallacy thusly:
The basic idea of the fallacy is this: you cannot directly infer the properties of individuals from the average of a group. Sounds complicated, but what that means is that if you measure something about lots of people — say, height — you can’t take the average measurement as an indication of any particular person’s status.
There’s a really simple example of this to do with means, or averages. Imagine you’ve got two groups of ten people, A and B. Group A has an average height of 170cm, and group B has an average height of 168cm. If you randomly select one person from each group, who is more likely to be taller, someone from group A or B?
The intuitive reaction is to say that someone from A is going to be taller than B, because the mean height is higher. However, this is not necessarily true. You can have a mean height of 170cm caused by two 200cm giants and eight 162.5cm people, and a mean of 168cm with six 170cm people and four 165cm people. In this case, 80% of group A is shorter than everyone in group B, which means that you’ll almost always get a taller person in group B if you pick randomly.
In other words, the average of a group isn’t always representative of the individuals.
Despite claiming this is a billion-subject sample (it is not, it’s an association of country policy w/ death rate), using averaged death rates for 36 countries gives you 36 data points. This is your true sample size: 36. You now have a small to low-medium sized sample.
— Arrivederci (@PapaGlider) August 8, 2020
That’s right. The sample size is actually rather small. It’s not billions. It’s 36.
@CovidAnalysis does try to control for some confounders, but even then they do it badly. For instance, in trying to control for obesity, a known risk factor for death from COVID-19, the authors cite the CIA World Factbook:
16/n The obesity rates are referenced to the CIA world factbook
Problem is, this data was last entered in 2016, and often is cited to sources up to a decade old
So it’s really very out of date!
— Health Nerd (@GidMK) August 9, 2020
Let’s just put it this way. You can’t use out-of-date information to apply to a population and control for a potential confounder.
@CovidAnalysis also notably leaves out several countries (such as France, where hydroxychloroquine use was quite high, spurred on by Didier Raoult, whose atrociously awful science about hydroxychloroquine and azithromycin as treatments for COVID-19 have been a frequent topic of this blog and my not-so-super-secret-other blog) and doesn’t explain why, as @BadCOVID19Takes points out:
In addition, as several critics have noted, using the number of deaths normalized to the entire population (deaths per capita), rather than the case fatality rate (deaths per cases of symptomatic COVID-19 infection) or infection fatality rate (deaths per cases of all COVID-19 infections, symptomatic or asymptomatic), is meaningless because the number of deaths per capita will depend a lot on how many infections there are in the population. Worse, the authors didn’t control for other measures undertaken by the various governments to control the spread of COVID-19 other than mask wearing, and—guess what!—they didn’t even control for that very well:
And they didn’t control for lockdowns at all:
Said they removed mask use as a confound (they didn’t, see above). But not quarantine as a confound? Lockdown orders associated with 40-60% reduction in COVID spread/mortality.
Le et al. t.co/dQxoc3kk41
— Arrivederci (@PapaGlider) August 8, 2020
Oops! Also: Incompetence or dishonesty? You be the judge!
Before you answer, look at Appendix 12. Look at the sources. They include primarily Tweets from anonymous pro-hydroxychloroquine accounts like Covid19Crusher and ChloroquineGuerilla, and news stories.
There are so many things wrong with this “study” that it’s hard to keep track of them all:
- It’s not a randomized controlled trial by any stretch of the imagination
- It’s an observational ecological study (sort of) prone to the ecological fallacy
- Confounders are not properly controlled for, particularly mask wearing and lockdowns
- It’s not peer-reviewed
- Its sample size is 36, not 2.7 billion
- It is not known why only 36 countries were selected, and why these 36, which reeks of cherry picking
- It misreports the results of at least one study, if not several
- It doesn’t actually determine how much hydroxychloroquine was actually used in the “HCQ countries” or whether and how much it was used in some of the “non-HCQ countries”
- It does not define what it means by “early” versus “late” treatment
I could go on and on and on, but I’m already tired. Suffice to say that this is an utterly worthless “study” whose “results” are utterly meaningless.
Who is @CovidAnalysis
As I’ve said before, the HCQTrial.com website reeks of astroturfing. It’s amusing to look at the Acknowledgments section, though:
It doesn’t, however, get us closer to who this is. Also, as noted here, this is just one website that appears to be part of a shady web of websites by one person or entity:
I do have my suspicions, though. Remember how two weeks ago (and earlier in this post) I pointed out that AAPS was the first to use this sort of analysis in its court filings for its lawsuit against the FDA trying to get the FDA to reinstate its EUA for hydroxychloroquine for COVID-19? Yes, I’m very suspicious that AAPS is behind @CovidAnalysis and the HCQTrial.com website and its incredibly deceptive “analysis”. On the AAPS website, there is an article from August 6 promoting hydroxychloroquine for COVID-19 that concludes:
For information on the “natural experiment” of early use vs. nonuse of HCQ—a 79% difference in mortality—see hcqtrial.com.
That’s admittedly weak evidence, although suggestive. Also, in one of the AAPS filings (dated July 20, 2020), it states:
None of Defendants’ arguments justifies their senseless interference with access by the public to hydroxychloroquine, a medication having a 65-year track of safety with numerous studies demonstrating its effectiveness as an early treatment against COVID-19 as compiled independently on the c19study.com website.
Playing with the CSS, this once had a label on it as if the website was being reviewed by one or more others. The blue graphic was also under the “DRAFT 4” label before being hidden pic.twitter.com/WKVTAi5Zmi
— 🧬 Adam Grant (@NiceGoingAdam) August 8, 2020
None of this, of course, is any sort of slam-dunk evidence that AAPS is behind this. It’s highly circumstantial. It’s also possible that whoever is behind this web of pro-hydroxychloroquine sites is has nothing to do with AAPS, but that the c19study.com and HCQTrial.com websites are simply useful repositories of disinformation for AAPS to use. Either way, it’s clear that these sites are all related and that they are all spreading the same sort of disinformation.
Unfortunately, there is a network of websites and news media willing to amplify this disinformation, and social media further facilitates its spread. The HCQTrial.com website is obvious pseudoscience to anyone who has any expertise in epidemiology and/or clinical trials, but unfortunately it has spread far and wide faster than experts could debunk its disinformation. Such is the age we are living in.