Non-Scientific Factors on
Research and Education: Publication Bias
|
|
|
Seth S. Leopold, MD |
|
|
|
Associate Professor |
|
Department of Orthopaedics and Sports
Medicine |
|
University of Washington |
Industry and Orthopaedics
|
|
|
|
Physician — Public |
|
Collaborative outreach / public
education |
|
Physician — Patient |
|
Research to determine best practice |
|
Physician — Physician |
|
CME |
Headlines: Public
Concerns
|
|
|
|
“Company tried to bar report that HIV
vaccine failed” |
|
“MDs and the pharmaceutical industry: A
growing embarrassment and liability” |
|
“Premature discontinuation of clinical
trials for reasons not related to efficacy, safety, or feasibility” |
|
“MDs and the pharmaceutical industry:
Is a gift ever just a gift?” |
|
|
|
|
|
|
Data: Funding and
Research
|
|
|
|
Authors of practice guidelines |
|
87% funded through industry |
|
59% rec’d products from supporting
companies |
|
Denied relationship affected choice |
|
Meta-Analysis: 1140 studies |
|
Industry sponsorship related to
positive outcome |
|
Also related to limits on publication
and data sharing |
|
|
|
Orthopaedic Results |
|
Industry funding related to positive
outcome |
|
Most severe in arthroplasty (83% vs
45%, p<0.004) |
Data: Funding and CME
|
|
|
|
>40% CME funding is from industry |
|
Up from 17% in 1994 |
|
$569 million in 2001; 22% increase
since 2000 |
|
Evidence that industry uses CME as a
“marketing tool” |
|
Physicians attending company’s courses
disproportionately prescribe their products |
Orthopaedic Relationships
|
|
|
|
|
Public: Confidence? |
|
Bias, or mere appearance of bias? |
|
In this context, does that matter? |
|
Headlines: MD’s “on the take” from
industry |
|
|
|
Patients: Best Practice? |
|
Quality research or publication bias? |
|
Physicians: CME? |
|
Education or marketing? |
Issue: The Relationship
|
|
|
“A good relationship is based on trust… |
Issue: The Relationship
|
|
|
“Good name in man and woman, dear my
lord, is the immediate jewel of their souls…” |
Academic Orthopaedic
Goals
|
|
|
|
How do we |
|
Earn / keep public’s confidence? |
|
Discover best practice? |
|
Disseminate results of our research? |
|
What effect do non-scientific factors
have on these goals? |
|
|
“CAGE” for Industry
Relations
Example: MIS THA
|
|
|
How many of you do hip replacements? |
|
Of those, how many have had a patient
ask you about “Minimally Invasive”? |
Example: Less-Invasive
THA
|
|
|
|
Peer-reviewed papers on Mini THA: 0 |
|
Peer-reviewed papers on MIS THA: 0 |
|
MD’s marketing MIS to public: Countless |
|
Courses teaching MIS to surgeons: Countless |
|
|
|
|
Patients in the Middle
|
|
|
|
Public: Confidence? |
|
Patients: Best Practice? |
|
Publication Bias? |
|
Our Research…Past and Future |
|
Physicians: CME? |
External Factors and
Outcome
|
|
|
|
Evidence from non-surgical specialties |
|
Non-scientific factors associated with
positive outcomes |
|
Pharmaceutical industry funding |
|
Country of origin |
|
Relationship with tobacco/EtOH industry |
External Factors and
Outcome
|
|
|
|
Orthopaedics: No data, despite |
|
Strong industry presence |
|
Apparent positive-outcome bias (AAOS
2000) |
Positive Outcome Bias
|
|
|
Increased conditional probability that
a research will be
•Funded
•Accepted for presentation
•Published
If conclusions are positive |
|
|
Positive Outcome Bias
|
|
|
|
Insidious: Hard to detect |
|
Need to look at large number of studies |
|
Harmful: Overestimation of treatment
effects |
|
Review articles |
|
Meta-analysis |
|
Economic analysis |
|
Evidence-based medicine |
|
|
|
|
Hypotheses
|
|
|
|
Receipt of commercial funding |
|
Country of origin |
|
Statistician as co-investigator
Are associated with positive outcome |
|
|
|
|
|
|
|
|
|
|
Methods
|
|
|
|
All papers over a one-year period |
|
3 Journals |
|
JBJS-A |
|
J Arthroplasty |
|
Am J Sports Med |
Methods
|
|
|
|
Two experienced reviewers |
|
“Rotating” review process |
|
|
|
3rd reviewer adjudicated prn |
Methods
|
|
|
|
Blinded review |
|
Authors’ names |
|
Departmental affiliations |
|
Countries of origin |
|
Sources of funding |
|
Presence of a statistician |
Data Collected
|
|
|
|
Classified by study design |
|
Outcome (positive or negative) |
|
315 articles reviewed |
|
95.6% were classifiable |
Definition of “Positive”
|
|
|
Emphasizes safety or efficacy |
|
Emphasizes cost-effectiveness |
|
Statistically significant difference
found |
|
|
|
If specific criteria not met, study not
considered “positive” |
|
|
|
|
Description of Literature
|
|
|
11 of 315 (3.5%) randomized |
|
|
|
32 of 315 (10.2%) stated hypothesis |
|
66 of 315 (21%) prospective |
|
|
|
159 of 315 (50.5%) controlled |
Associated w/Positive
Outcome
|
|
|
|
Receipt of commercial funding |
|
78.9% of industry-funded studies were
positive |
|
63.3% of non-industry-funded studies
were positive |
|
p=0.025 |
|
|
|
|
Industry Funding and
Outcome
|
|
|
|
Largest contribution: |
|
J Arthroplasty |
|
83% vs. 45% positive if funded in JA |
Not Associated with
Outcome
|
|
|
|
|
|
Presence of statistician (p=0.935) |
|
|
|
Country of origin (p=0.248) |
Reasons for Industry
Effect
|
|
|
|
Benign |
|
In-house research before collaboration |
|
More $ ® More specimens ® Less b-error |
Reasons for Industry
Effect
|
|
|
|
Troubling |
|
Restrictive covenants |
|
Early termination of studies by funding
source |
|
Psychological or economic influences on
investigator |
Conclusions
|
|
|
Industry funding associated with
positive outcome |
|
Potential source of positive-outcome
bias |
What Did We Really Show?
|
|
|
Industry-funded papers more likely to
conclude positively |
|
Positive papers more likely to have
received industry funding |
Smoke = Fire?
|
|
|
|
|
Possibly |
|
F/U work in other specialties: Bias,
not just appearance of bias, was confirmed |
|
ML Callaham (JAMA v280), JM Stern (BMJ
v315) |
|
Documented misbehaviors already
discussed |
|
But not proven |
|
Consider a different denominator |
|
Published papers (Our study): Appearance
of bias |
|
Abstracts submitted for presentation |
|
Manuscripts submitted for publication |
|
|
Bias vs. Appearance of
Bias
|
|
|
|
Consider universe of all experiments
submitted |
|
Address alternative explanations for
“apparent bias” |
|
Statistical power |
|
Study quality |
|
Actual publication bias |
Power
|
|
|
Is there a difference between positive
and negative trials in terms of power? |
|
Control for any systematic differences
in power between positive and negative studies |
Study Quality
|
|
|
|
Are positive studies “better” studies
as a group? |
|
Industry says “yes,” cites different |
|
Designs |
|
Funding levels |
|
Support networks |
|
Need to control for potential
systematic differences in quality |
|
|
Publication Bias: If So,
Where?
|
|
|
|
Investigator |
|
Avoid studies to show equivalence ($,
time) |
|
Unexpectedly negative result?
Self-censor |
|
Reviewer |
|
Framing effects |
|
Journal |
|
Funding sources? |
Follow-Up Study
|
|
|
|
|
H1: Non-scientific factors do not
affect likelihood that submitted studies will conclude positively |
|
Control for: |
|
Sample size |
|
Study quality |
|
H2: Positive outcome does not affect
the likelihood that study will be published |
|
Denominator: All submissions to a
journal |
|
Comment on more than just “apparent
bias” |
Conclusions
|
|
|
|
Industry collaboration is here to stay |
|
Study to determine its impact and
effects |
|
Realize unique partnership
opportunities |
|
Research on publication bias critical |
|
If non-scientific factors impact
outcomes, these MUST be identified / controlled as any other confounding
variable |
|
|
|
Peer-review process may need change |
|
Disclose funding sources earlier in
review |
THANK YOU
|
|
|
ACKNOWLEDGEMENTS: |
|
Winston J. Warme, MD |
|
E. Fritz Braunlich, MD |
|
Susan Shott, Ph D |
|
Fredric M. Wolf, Ph D |
|
UW Friends of Orthopaedic Research and
Education |
|
|