Concordance between preclinical and clinical outcomes and laboratory practices

by

Published on: Oct 9, 2015 @ 12:21

Preamble

Encouraging drug efficacy claims in preclinical cancer research often produce only modest outcomes in rigorous clinical trials. This study will investigate whether xenograft tumor response experiments correlate with response rates in clinical trials of matched indications for two targeted therapies—sorafenib and sunitinib. As a secondary analysis, we are assessing the effect of preclinical design characteristics on concordance between preclinical and clinical outcomes. Our objective is to provide insight to its clinical predictive value and factors that influence the relationship between preclinical and clinical treatment outcomes.

Primary Analyses

Comparing pooled effect sizes between clinical and preclinical trials within and across malignancy types

1A. Within malignancy types:

For each malignancy type, we will calculate the mean difference and 95% CIs between the pooled SMDs of preclinical experiments and clinical trials for monotherapy studies involving the two anti-cancer drugs sunitinib and sorafenib. For each preclinical experiment, we will calculate a SMD and 95% CI for tumor volume. Preclinical trials within a malignancy type will then be meta-analyzed to produce a pooled SMD and 95%CI for tumour volume.

Likewise, for each clinical trial, we will calculate a Peto odds ratio and 95% CI for objective response rate.1 Calculation of odds ratios for controlled trials are straightforward. For single arm trials, we will impute a 0% objective response rate for a control arm and match the sample size of the treatment arm. To facilitate comparisons, odds ratios of objective response rates in clinical trials will be converted to SMDs. Clinical trials within a malignancy type- which will be pooled prior to the launch of any analysis- will then be meta-analyzed to produce a pooled SMD and 95% CI for objective response rate.

For each malignancy type, we will evaluate differences between preclinical and clinical pooled SMDs by calculating a mean difference in the pooled SMDs and their respective 95% CIs. We will use a 5% alpha level.
For each malignancy type, we will plot individual preclinical and clinical SMDs on x and y axes respectively.

1B. Across malignancy types:

Meta-analyze pooled differences in SMDs between preclinical (tumor volume) and clinical (objective response) adjusting for malignancy type to test for differences.
We will plot pooled clinical effect size (objective response) on y-axis and preclinical pooled effect size (tumour volume) on the x-axis. Inferential tests will ste from the pooled differences in the SMDs and respective 95% confidence intervals, as described previously.

 

Secondary Analyses

Assessing impact of measures of validity within and across malignancy types

2A. Internal validity:

To assess the impact of major items of a preclinical trial’s internal validity, we will create a six-point scale based on whether studies implement each of the 6 IV items in our dataset (i.e. presence or absence of: 1) exact sample size for all groups, 2) performance of a sample size calculation; 3) random allocation, 4) concealed allocation, 5) blinded outcome assessment, 6) addressing animal flow). We will score every preclinical experiment based on that scale. We will perform meta-regression assessing the effect of score on the pooled pre-clinical measure of effect within a malignancy, within a drug, as well as for all experiments, regardless of malignancy and drug, and assess its impact on the mean difference between pre-clinical and clinical effect.
Analyze and plot as per 1B.
Repeat 1B for each of the 6 internal validity items via subgroup analyses.

2B. Construct validity:

As above, except we will use the following variables, as derived from Henderson et al 20152 and Henderson et al 2013.2
– Mouse vs. Rat
– Juvenile vs. Adult or Aged animal
– Immuno-compromised vs. Immunocompetent
– Xenograft vs. non-xenograft
– Early stage vs. late stage
– Not Molecular and Physiological vs. Molecular and Physiological
Repeat 1B for each of the 6 internal validity items individually via subgroup analyses.

2C. External validity:

To assess the relationship between external validity and concordance of outcomes, we will restrict our analysis to xenograft studies only, examining whether malignancies testing higher than the median number of xenografts have greater concordance than those testing less than the median.
Repeat 1B for each of the 6 internal validity items via subgroup analyses.

2D. Effect of Publication bias.

To assess whether correction for publication bias improves concordance of outcomes, we will create trim and fill analyses/plots to produce corrected effect sizes for every malignancy in the sunitinib and sorafenib datasets. We will perform steps 1A and 1B (across malignancies) and plot corrected and uncorrected estimates. Calculate average % of over or underestimation across all malignancies.  Determine whether- if we use effect sizes that have been corrected using trim and fill, the correlation of effect is improved vs. when we use the uncorrected.

2E. Further Exploratory Analyses

All further analysis will be conducted in an exploratory manner. We intend to perform the following analyses:

i. Role of funding:

Question: Are preclinical studies supported by the makers of sunitinib and sorafenib more likely to be concordant with trial outcomes than those for which makers have not provided support?
Hypothesis: We expect that studies supported by the private sector will be more concordant with clinical outcomes than those not supported by the private sector.
Approach: Code all preclinical studies based on whether they received support from makers or not (yes/no). Then, test whether level of concordance for one is distinguishable from the other.

ii. Role of drug:

Question: If we repeat primary analyses above, does one drug show greater concordance than the other?
Hypothesis: Because sorafenib is cytostatic, we expect sunitinib studies will show greater concordance with clinical outcomes (if any) than sorafenib.
Approach: Perform analyses as described above, separating sorafenib and sunitinib studies.

iii. Role of dose-response curves:

Question: Are malignancies for which a dose-response has been demonstrated more likely to show concordance with clinical outcomes than malignancies for which there is no dose-response information?
Hypothesis: We expect studies supported by dose response curves will be more concordant with clinical outcomes.
Approach: Divide malignancies into those for which dose-response curves have been performed and shown to have a positive relation, and those for which there is no dose-response curves. Determine whether the former shows greater concordance with clinical outcomes (if any) than the latter.

iv. Recommendation for clinical testing:

Relationship between studies that recommend clinical testing and ones that do not
Hypothesis: We expect studies that conclude with a recommendation for continued pursuit of the drug in clinical testing will be more concordant with clinical outcomes.
Approach: Divide studies into those that explicitly recommend further testing in a clinical population, and those for which there is no explicit recommendation or a recommendation against clinical testing. Determine whether the former shows greater concordance with clinical outcomes (if any) than the latter.

v. Challenger drug:

Relationship between studies where drug is challenger (no other arms) vs. incumbent (monotherapy compared with another drug, OR combination therapy).
Hypothesis: We expect studies where the drug is the incumbent will be more concordant with clinical outcomes.
Approach: Divide malignancies into those for which drug is a challenger (no other arms) vs. incumbent (monotherapy compared with another drug, OR combination therapy). Determine whether the latter shows greater concordance with clinical outcomes (if any) than the former.

Questions

Should you have questions or concerns about our study, please contact Jonathan Kimmelman, head of the STREAM research group and associate professor in the Biomedical Ethics Unit / Social Studies of Medicine department at McGill University. Email: jonathan.kimmelman@mcgill.ca

PROJECT PROGRESS AND UPDATES

Oct 9, 2015 @ 13:19

We have collected and individually meta-analyzed data from sorafenib (unpublished data) and sunitinib2 preclinical experiments. We have extracted and meta-analyzed data from sunitinib clinical trials4 and are in the process of meta-analysis of sorafenib clinical trial data (unpublished data). To date, we have not performed analyses combining preclinical and clinical data, nor have we combined data between drugs. We have not yet investigated the correspondence between preclinical design, reporting practices, and tumour response and clinical treatment outcomes for either sorafenib or sunitinib.

Oct 16, 2015 @ 15:09

Sunitinib preclinical data were published Oct. 13 in Elife.2 See References section for hyperlink to online article.

Dec 9, 2015 @ 15:57

Sunitinib clinical data were published online Nov. 7, 2015 in JNCI (print Jan. 2016).4 See References section for hyperlink to online article.

 

Jun 20, 2016 @ 15:32

Sorafenib preclinical data were published online June 3, 2016 in Cancer Research.5  See References section for hyperlink to online article.

After consulting with a statistician, we are conducting our analyses using relative differences (ratio of means) presenting the overestimation of preclinical SMD as compared to clinical SMD, plotted on a log scale. The original analysis plotting individual preclinical and clinical SMDs on x and y axes, respectively, is not a coherent illustration especially when incorporating 95% CIs. However, these analyses can be included as supplemental data in our final report.

Finally, we are adding an exploratory analysis investigating whether available survival data in our preclinical dataset shows correlation to clinical results in matched indications [methods to be added shortly].

 

REFERENCES

1          Yusuf, S, Peto, R, Lewis, J, Collins, R, & Sleight, P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Progress in cardiovascular diseases 1985; 27: 335-371.

2          Henderson VC, Demko N, Hakala A, et al. A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib. Elife 2015; 4: e08351.

3          Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med 2013; 10: e1001489.

4          Carlisle B, Demko N, Freeman G, et al. Benefit, Risk and Outcomes in Drug Development: A Systematic Review of Sunitinib. Journal of the National Cancer Institute 2015; 108: djv292.

5          Mattina J, MacKinnon N, Henderson VC, Fergusson D, Kimmelman J. Design and Reporting of Targeted Anti-Cancer Preclinical Studies: A Meta-Analysis of Animal Studies Investigating Sorafenib Antitumor Efficacy. Cancer research. 2016 Jun 3:canres-3455.

BibTeX

@Manual{stream2015-856,
    title = {Concordance between preclinical and clinical outcomes and laboratory practices},
    journal = {STREAM research},
    author = {James Mattina},
    address = {Montreal, Canada},
    date = 2015,
    month = oct,
    day = 9,
    url = {http://www.translationalethics.com/concordance-between-preclinical-and-clinical-outcomes-and-laboratory-practices/}
}

MLA

James Mattina. "Concordance between preclinical and clinical outcomes and laboratory practices" Web blog post. STREAM research. 09 Oct 2015. Web. 11 Dec 2018. <http://www.translationalethics.com/concordance-between-preclinical-and-clinical-outcomes-and-laboratory-practices/>

APA

James Mattina. (2015, Oct 09). Concordance between preclinical and clinical outcomes and laboratory practices [Web log post]. Retrieved from http://www.translationalethics.com/concordance-between-preclinical-and-clinical-outcomes-and-laboratory-practices/


Comments are closed.

Search STREAM

Old blog posts


All content © STREAM research

admin@translationalethics.com
Twitter: @stream_research
3647 rue Peel
Montreal QC H3A 1X1