1	Non-Scientific Factors on Research and Education: Publication Bias Seth S. Leopold, MD Associate Professor Department of Orthopaedics and Sports Medicine University of Washington
2	Industry and Orthopaedics Physician — Public Collaborative outreach / public education Physician — Patient Research to determine best practice Physician — Physician CME
3	Headlines: Public Concerns “Company tried to bar report that HIV vaccine failed” “MDs and the pharmaceutical industry: A growing embarrassment and liability” “Premature discontinuation of clinical trials for reasons not related to efficacy, safety, or feasibility” “MDs and the pharmaceutical industry: Is a gift ever just a gift?”
4	Data: Funding and Research Authors of practice guidelines 87% funded through industry 59% rec’d products from supporting companies Denied relationship affected choice Meta-Analysis: 1140 studies Industry sponsorship related to positive outcome Also related to limits on publication and data sharing Orthopaedic Results Industry funding related to positive outcome Most severe in arthroplasty (83% vs 45%, p<0.004)
5	Data: Funding and CME >40% CME funding is from industry Up from 17% in 1994 $569 million in 2001; 22% increase since 2000 Evidence that industry uses CME as a “marketing tool” Physicians attending company’s courses disproportionately prescribe their products
6	Orthopaedic Relationships Public: Confidence? Bias, or mere appearance of bias? In this context, does that matter? Headlines: MD’s “on the take” from industry Patients: Best Practice? Quality research or publication bias? Physicians: CME? Education or marketing?
7	Issue: The Relationship “A good relationship is based on trust…
8	Issue: The Relationship “Good name in man and woman, dear my lord, is the immediate jewel of their souls…”
9	Academic Orthopaedic Goals How do we Earn / keep public’s confidence? Discover best practice? Disseminate results of our research? What effect do non-scientific factors have on these goals?
10	“CAGE” for Industry Relations
11	Example: MIS THA How many of you do hip replacements? Of those, how many have had a patient ask you about “Minimally Invasive”?
12	Example: Less-Invasive THA Peer-reviewed papers on Mini THA: 0 Peer-reviewed papers on MIS THA: 0 MD’s marketing MIS to public: Countless Courses teaching MIS to surgeons: Countless
13	Patients in the Middle Public: Confidence? Patients: Best Practice? Publication Bias? Our Research…Past and Future Physicians: CME?
14	External Factors and Outcome Evidence from non-surgical specialties Non-scientific factors associated with positive outcomes Pharmaceutical industry funding Country of origin Relationship with tobacco/EtOH industry
15	External Factors and Outcome Orthopaedics: No data, despite Strong industry presence Apparent positive-outcome bias (AAOS 2000)
16	Positive Outcome Bias Increased conditional probability that a research will be •Funded •Accepted for presentation •Published If conclusions are positive
17	Positive Outcome Bias Insidious: Hard to detect Need to look at large number of studies Harmful: Overestimation of treatment effects Review articles Meta-analysis Economic analysis Evidence-based medicine
18	Hypotheses Receipt of commercial funding Country of origin Statistician as co-investigator Are associated with positive outcome
19	Methods All papers over a one-year period 3 Journals JBJS-A J Arthroplasty Am J Sports Med
20	Methods Two experienced reviewers “Rotating” review process 3rd reviewer adjudicated prn
21	Methods Blinded review Authors’ names Departmental affiliations Countries of origin Sources of funding Presence of a statistician
22	Data Collected Classified by study design Outcome (positive or negative) 315 articles reviewed 95.6% were classifiable
23	Definition of “Positive” Emphasizes safety or efficacy Emphasizes cost-effectiveness Statistically significant difference found If specific criteria not met, study not considered “positive”
24	Description of Literature 11 of 315 (3.5%) randomized 32 of 315 (10.2%) stated hypothesis 66 of 315 (21%) prospective 159 of 315 (50.5%) controlled
25	Associated w/Positive Outcome Receipt of commercial funding 78.9% of industry-funded studies were positive 63.3% of non-industry-funded studies were positive p=0.025
26	Industry Funding and Outcome Largest contribution: J Arthroplasty 83% vs. 45% positive if funded in JA
27	Not Associated with Outcome Presence of statistician (p=0.935) Country of origin (p=0.248)
28	Reasons for Industry Effect Benign In-house research before collaboration More $ ® More specimens ® Less b-error
29	Reasons for Industry Effect Troubling Restrictive covenants Early termination of studies by funding source Psychological or economic influences on investigator
30	Conclusions Industry funding associated with positive outcome Potential source of positive-outcome bias
31	What Did We Really Show? Industry-funded papers more likely to conclude positively Positive papers more likely to have received industry funding
32	Smoke = Fire? Possibly F/U work in other specialties: Bias, not just appearance of bias, was confirmed ML Callaham (JAMA v280), JM Stern (BMJ v315) Documented misbehaviors already discussed But not proven Consider a different denominator Published papers (Our study): Appearance of bias Abstracts submitted for presentation Manuscripts submitted for publication
33	Bias vs. Appearance of Bias Consider universe of all experiments submitted Address alternative explanations for “apparent bias” Statistical power Study quality Actual publication bias
34	Power Is there a difference between positive and negative trials in terms of power? Control for any systematic differences in power between positive and negative studies
35	Study Quality Are positive studies “better” studies as a group? Industry says “yes,” cites different Designs Funding levels Support networks Need to control for potential systematic differences in quality
36	Publication Bias: If So, Where? Investigator Avoid studies to show equivalence ($, time) Unexpectedly negative result? Self-censor Reviewer Framing effects Journal Funding sources?
37	Follow-Up Study H1: Non-scientific factors do not affect likelihood that submitted studies will conclude positively Control for: Sample size Study quality H2: Positive outcome does not affect the likelihood that study will be published Denominator: All submissions to a journal Comment on more than just “apparent bias”
38	Conclusions Industry collaboration is here to stay Study to determine its impact and effects Realize unique partnership opportunities Research on publication bias critical If non-scientific factors impact outcomes, these MUST be identified / controlled as any other confounding variable Peer-review process may need change Disclose funding sources earlier in review
39	THANK YOU ACKNOWLEDGEMENTS: Winston J. Warme, MD E. Fritz Braunlich, MD Susan Shott, Ph D Fredric M. Wolf, Ph D UW Friends of Orthopaedic Research and Education