Nonpublication of Neurology Trials for Stalled Drugs & the Ironic Nonpublication of Data on those Stalled Drugs

In my experience, peer review greatly improves a manuscript in the vast majority of cases. There are times, however, when peer review improves a manuscript on one less important axis, while impoverishing it in another more important one. This is the case with our recent article in Annals of Neurology.

Briefly, our manuscript created a sample of FDA-approved neurological drugs, as well as a matched sample of neurological drugs that did not receive FDA approval- but instead stalled in development (i.e. a 3 year pause in testing). We then used clinicaltrials.gov to identify trials of drugs in both groups, and determined the proportion of trials that were published for all approved drugs, as well as FDA non-approved drugs. We found- not surprisingly- that trials involving stalled neurological drugs were significantly less likely to publish. What- for us- was the bigger surprise was that the proportion of trials published at 5 years or more after closure was a mere 32% for stalled neurological drugs (56% for licensed). Think about what that means in terms of the volume of information we lose, and the disrespect we show to neurological patients who volunteer their bodies to test drugs that show themselves to be ineffective and/or unsafe.

We shopped the manuscript around – eventually landing at Annals of Neurology. The paper received glowing reviews. Referee 1: “The research is careful and unbiased and the conclusions sound and impactful.” Referee 2: “This is an excellent and very important paper. It rigorously documents a very important issue in clinical trial conduct and reporting. The authors have done a superb job of identifying a crucial question, studying it carefully and fairly with first-rate quantification, and presenting the results in a clear, well-written, and illuminating manner… I have no major concerns, but some small points may be helpful…” Ka-ching!

However, after submitting small revisions, the manuscript was sent to a statistical referee who was highly critical of elements that seemed minor, given the thrust of the manuscript. [Disclosure: from here forward – this blog reflects my opinion but not necessarily the opinion of my two co-authors]. We were told to expunge the word “cohort” from the manuscript (since there was variable follow-up time). Odd but not worth disputing. We were urged “to fit a Cox model from time of completion of the trial to publication, with a time-varying covariate that is set to 0 until the time of FDA approval, at which time it is changed to 1. The associated coefficient of this covariate is the hazard ratio for publication comparing approved drugs to unapproved drugs.” That seemed fastidious – we’re not estimating survival of a drug to make policy here – but not unreasonable. We were told we must remove our Kaplan-Meier curves of time to publication. I guess. So we did it and resubmitted- keeping some of our unadjusted analyses in (of course labeling them as unadjusted).

The reviewer pressed further. He/she wanted all presentation of proportions and aggregate data removed (here I will acknowledge a generous aspect of the referee and editors- he/she agreed to use track changes to cut content from the ms [I am not being snarky here – this went beyond normal protocol at major journals]). We executed a “search and destroy” mission for just about all percentages in the manuscript: in this case we cut two tables’ worth of data describing the particular drugs, characteristics of trials in our sample, and proportions of trials for which data were obtainable in abstract form, or on company websites. Although one referee had signed off (“My high regard for this paper persists. Differences in views concerning the statistical approach are understandable. I see the paper as providing very important data about the trajectory of publication or non-publication of data depending on the licensing fate of the drug being studied, and see the survival analysis as bolstering that approach”) the editors insisted on our making revisions requested by the reviewer.

So in the end- we had to present what we believe to be an impoverished, data-starved, and somewhat less accessible version in Anals of Neurology. And not surprisingly, upon publication, we were (fairly) faulted online for not providing enough information about our sample. To our mind, the **real** version- and the one we think incorporates the referee’s productive suggestions while respecting our discretion as authors can be accessed here. And we are making our complete dataset available here.

2017 June 5 | Leave a comment

@Manual{stream2017-1325,
    title = {Nonpublication of Neurology Trials for Stalled Drugs & the Ironic Nonpublication of Data on those Stalled Drugs},
    journal = {STREAM research},
    author = {Jonathan Kimmelman},
    address = {Montreal, Canada},
    date = 2017,
    month = jun,
    day = 5,
    url = {http://www.translationalethics.com/2017/06/05/nonpublication-of-neurology-trials-for-stalled-drugs-the-ironic-nonpublication-of-data-on-those-stalled-drugs/}
}