Let's get rid of this argument

I checked out some of the other early reactions to the 60 Minutes report looked like, and immediately came upon a standard criticism of Kirsch. It’s one that I find particularly annoying, because people who make it often don’t understand the situation and it takes a while to explain why this is wrong, and most people would understand if they read the book. In this case, though, the author of the post, John Grohol, links to Kirsch’s book, so his raising this objection is either disingenuous, and therefore offensive, or sloppy.

Here it is:

What wasn’t mentioned in the 60 Minutes piece, because it was opinion journalism forwarding a specific viewpoint, is that Kirsch’s research is selective. He hasn’t looked at every antidepressant study ever done (now numbering in the thousands). He only looked at the clinical trials required to gain U.S. Food and Drug Administration approval for 6 antidepressant drugs (there are over a dozen on the market).

I’ve seen this argument on many occasions, sometimes – as in this case – hinting that Kirsch was trying to pass off a biased sample and sometimes claiming it outright. There are of course legitimate criticisms that can be made of Kirsch’s work, as of anyone’s, but this particular criticism is ridiculous.

That the published literature on antidepressants, were that to include 10 studies or hundreds, is completely biased in favor of the pharmaceutical makers is not in dispute by anyone even slightly knowledgeable about this. The techniques that the drugmakers use not only to slant research outcomes but to exaggerate effectiveness and hide side effects in the published literature are extremely well documented, and described in great detail by Kirsch, Ben Goldacre, Marcia Angell, and many others (as I’ve discussed numerous times here). The overwhelming publication bias in favor of the drugs’ effectiveness is also well established, and even stated by the British commissioner in the report. The data Kirsch obtained demonstrated it, but his is far from the only research to do so.

We should pause here for a moment, because people often skip past or brush off this point. It’s well demonstrated that these companies manipulate research, its presentation, and its publication in favor of showing effectiveness and minimizing side effects of the products they sell. These products are drugs, and the effectiveness and side effects have serious, in some cases life or death, consequences for people. They've paid researchers large amounts of money. They’ve been caught lying about what the research shows, including in cases that involve drugs that are going to be given to children. And they are allowed to retain control over the data.

I’m amazed at the lack of outrage, or even concern, about this situation in discussions of psychiatric drugs. Sometimes people go so far as to excuse these practices, which can ruin lives, with a shrugging “corporations will be corporations.” Reasonable people who care about their own and others’ health, or even just about the integrity of science, should be up in arms about this. They should join Kirsch, Goldacre, Angell and others who’ve long been calling for a real clinical trials registry. (There is one in the US, but from what I’ve seen it’s a joke. Try searching for, for example, SSRIs and see what you find. If it were an effective tool for keeping track of and providing public access to the data – and you can forget about the raw data – no one would be talking about the available published literature, because we’d have the data.) They would advocate for the denial of approval of a drug or its continuation if all of the data are not made public.

In any case, this is the reality Kirsch faced, and an insuperable problem faced by anyone who wants to perform a meta-analysis of the research on these drugs. But the one situation in which the companies can’t suppress negative findings (although of course they can and do still manipulate the research designs themselves) is in submitting clinical trials to the FDA for approval. That’s why Kirsch used FOIA – and that he had to do this should itself make people angry – to gain access to the entire set of pre-approval trials. He wasn’t selecting studies that would suit a pre-formed argument. He was trying to get the most unbiased set of studies possible, while continuing to argue for public access to all of the pre- and post-approval data. I’d like people suggesting that this is “biased” to explain how a less “biased” set of studies could be obtained. Moreover, as he says and as is worthwhile to point out,

these were the data upon which the antidepressants that are on the market today were approved for doctors to prescribe. If there was anything wrong with those data, then arguably the drugs should not have been approved in the first place. (p. 25)

So any claims – which, incidentally, Kirsch addresses in the book – about these studies not being long enough or short enough or whatever such that Kirsch’s meta-analysis was biased don’t hold up. We know that the published literature is biased, and we know the reason for and direction of the bias. An analysis of all or a representative sample of the published literature would be biased. Don’t criticize Kirsch and other researchers for being denied access to data we should all have access to, especially when the complete data can't reasonably be expected to contradict his findings. The suggestion that his work is biased because it doesn’t include, or that it should include, all of the published literature, either generally or for any specific sort of study, is simply unreasonable.

There are other problems with the post I linked to, errors of both omission and commission (the author seems confused about the difference between “conclusions based on one's reading of the evidence” and “bias”), and I expect to see them repeated many times. The 60 Minutes report – which, having to fit the story into a 20-minute frame, was necessarily limited – will likely contribute to some,* but my hope is that more people will at least read the book and criticize it based on a fair analysis of the substantive arguments.