Evidence Based Practice - How good is the evidence?

The Trauma Register Experience - A Quest for Better Quality Data through Direct Physician and Patient Structured Data (i.e. point-and-click, menu-driven) Entry.

AAOS OKO Evidence Based Practice Portal

Bill Burman, MD, HWB Foundation

Background:

Gillott et al.¹ have stated: "Maintenance of an active registry must be viewed as important as the medical care rendered, if the right person is going to receive the right treatment in a timely fashion without undue cost to society."

Despite a growing demand for such surveillance, "best practices" for the routine capture of data adequate for this purpose have yet to be determined. Champion and Cushing ² have noted that despite the expenditure of tens of millions of dollars on the development and maintenance of trauma databases, the poor quality and lack of clinical detail of the data within these registries imposed great limitation of their use for sophisticated research and outcome analysis.

Furthermore, Keller³ has surmised "It has become increasingly clear that much of the clinical research that has long been published and on which we base much of our education and practice activity is, in fact, severely flawed."

After 21 years of development spanning 7 versions of the Anderson Orthopaedic Clinic Total Joint Registry, Engh⁴ concluded that data quality was not so much a function of hardware or software but rather the diligence of those who input the system.

Direct Structured Data Entry:

The best evidence should be obtainable from those who can give the most expert testimony and in whom optimal data entry diligence can be induced.

After a review of the Yale-New Haven SNOWMED orthopaedic departmental database, Brand et al.⁵ concluded that: "The most sophisticated system is useless if physicians refuse to register their patients." Anyone other than the physician involved in the direct care of the patient may have "difficulty in identifying the salient features of a patient's diagnosis and procedures and describing them ...". This assertion has been borne out by studies which have shown significant discrepancies in relatively simple ICD-9 coding of diagnoses between direct care physicians and coding technicians^{6, 7}.

In an era of practice profiling and pay-for-performance quality metrics, the physician has become a major stakeholder in the specificity and quality of the documentation of care. Just as most physicians would be reluctant to let someone else "do the talking for them" in the medical record (because of possible medical-legal consequences), there should be a similar impetus for direct entry into databases which may have an impact on the bottom line.

Ellwood⁸ has cautioned if outcomes are to be fairly compared and interpreted, there must first be an adjustment for:

1) risk factors - e.g. allergies, diabetes, hypertension, infection, genetic predisposition, age, sex, socioeconomic status, natural and occupational exposures

2) severity of illness

3) complexity of care

Ellwood concludes "This adjustment is far from a trivial undertaking". Such a determination requires considerable clinical judgement.

Another important reason for physicians to be directly involved with data entry is to provide guidance with the design and utility of the input process and interface. Just as the questioning of a resident by an attending can help to emphasize certain key clinical parameters, the line of inquiry of an information system can affect clinical awareness - for better or for worse. Patel et al.⁹ found that exposure to the computer-based patient record was associated with changes in physicians' information gathering and reasoning strategies and concluded that the process of data entry can have a profound influence in shaping cognitive behavior.

Finally, direct physician structured data entry is the missing link to the full implementation of the Electronic Medical Record (EMR). According to medical informatics pioneer Clement McDonald¹⁰:

"If 'everyone' wants EMRs, and the sources of electronic patient data are so abundant, why are they so scarce? The answer is twofold. First, the sources of electronic patient information that do exist (e.g., laboratory data, pharmacy data, and physician dictation) reside on many isolated islands that have been very difficult to bridge; and second, we have not quite figured out how to capture the data from the physician in a structured and computer understandable form... Research is still needed to solve this last problem."

According to Bleeker et al,¹¹ "the potential benefits of structured data entry (SDE), i.e. data entry based on selection of predefined medical concepts, in particular are uniformity of data, easier reporting of data (e.g. by a standard letter to the general practitioner), and more advanced decision support (e.g. by embedding clinical guidelines in the EMR), quality assessment, and patient-oriented clinical research."

Methods:

The Trauma Register is an orthopaedic trauma registry instrument designed to obtain direct, senior-level, physician-specialist input of detailed, structured data entry to multiply classify and describe diagnosis, treatment, complication and outcome of orthopaedic injury. It also records all visceral injuries, co-morbidities and non-traumatic orthopaedic diagnoses.

Recognizing the high cost and fatigue-failure rate of duplicated input required by stand-alone data systems, the Trauma Register promotes the integration of the registry input with practice management functions through input load-sharing data exchange with other information resources on the hospital network. Such a process is necessary to the enrichment and validation of data entry.

Since structured data entry is more demanding than often inconsistent and incomplete free text entry, it is necessary that the data collection system is programmed to provide a substantial return on investment of the extra input effort. There can be reformatting of the input to a narrative format for export to the EMR with a resulting reduction of data entry duplication and transcription costs. Error and medical-legal alerts can work to reduce malpractice premiums. Links to "just-in-time" educational resources at the "teachable moment" can provide more effective and less costly CME which potentially improves the quality of care at the point of care and the quality of observations entered into the database. Practice management functions related to scheduling, procedure and pre-certification coding may be performed. Certain requirements of RRC, ACGME, ABOS and JCAHO certification and logging may be addressed. Input load-sharing, data exchanges which produce rounding lists and other useful reports, can be made available.

In order to deliver the incentives for physician structured data entry, the data collection system should be connected to the hospital network. This may require considerable negotiation but is of critical importance if the data collection system is to become a readily available, indispensable, information and patient care asset.

In general, the key to success in the gathering of evidence lies more within the realm of institutional politics (i.e. network access, data exchange, HIPAA, the IRB, etc) than information technology. Reed Gardner, a former president of American Medical Informatics Association, acknowledged this fact in the 1998 Davies Lecture¹²:

"In my opinion, the success of a project is 80% dependent on the social and political interaction skills of the developer and 20% or less on the implementation of the hardware and software technology."¹³

Court-Brown and McQueen¹⁴ reported that only 2% of orthopaedic surgeons over the age of 55 to have a major interest in trauma. "As it is these surgeons who tend to be politically influential, it is clear that orthopaedic trauma is considerably disadvantaged." This lack of political clout compounds the difficulty of establishing and sustaining integrated, orthopaedic trauma registries.

Results:

Over a period of 16 years, the Trauma Register (TR) system has been deployed in 16 trauma centers. A total of 47,688 records of variable incompleteness have been created. Hospital network connection was achieved in 5 trauma centers. In 3/5 of these hospitals, direct physician structured data entry has been sustained for as long as 16 years. In 2/5 of these hospitals, direct physician structured data entry stopped shortly after major departmental leadership and staffing changes - but not before a total of 20,000 records had been created.

Direct physician structured data entry has not been sustained in 10/11 hospitals where hospital network integration was not achieved.

Despite a potential for outcome determination, in settings where the TR has been sustained, it has thus far been used mainly as a practice and patient management adjunct.

A few studies have made partial use of the TR direct structured data but to date there have been no studies which were entirely based on the TR structured data entry.

Discussion:

In examining the failure to obtain complete input of direct physician and patient structured data entry, three major components of outcome determination are taken into account. Click the links for methods and samples of data collection.

1) Diagnosis and Treatment

Direct physician structured data entry of diagnosis and treatment has been most successfully obtained and maintained. Besides specific orthopaedic trauma classification, the generation of ICD-9, CPT and RVU values linked to reimbursement practice management functions drives the data entry. In centers, such as Bellevue Hospital, where the program has been networked and integrated accordingly, approximately 1000 patient records and 2000 procedure records containing structured data entry are created each year.

Diagnostic and treatment data entry can be augmented with data exchanges with other information resources on the hospital network (e.g. Operating Room, ACS Trauma and PACS Radiology database inputs were achieved at Vanderbilt).

2) Complications

The acquisition of complications data (structured or otherwise) has been much more challenging. Typically less than 10 complication records per hospital per year have been created. There seem to be only disincentives for the reporting of mishaps.

However, JCAHO and RRC accreditation requirements can be substantial drivers of complications data entry. Both the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and Residency Review Committee (RRC) have morbidity and mortality (M&M) reporting requirements which can be merged with complications database entry. For a number of years this was implemented and sustained at Vanderbilt and Elmhurst Hospitals.

Specific structured data entry of complications can provide classification and linkage to on-line references which can enhance the educational impact and reporting of the M&M conference. This was actually a component of EA Codman's "End-Result System".¹⁵

"He detailed the clinical history and outcomes of each of his patients on a set of cards and used this information to review adverse events systematically and categorize their precedent errors."

3) Cinical Follow-up - Patient & Physician

In the follow-up clinic, both direct patient and physician structured outcome data entry have been frustrated by a number of logistical problems. The constraints of time, space and lack of portable, wireless input devices make the clinic a particularly challenging setting within which to acquire the data. With enlistment of administrative support, it is possible to establish many of the aforementioned incentives (non-duplicative EMR integration, lower transcription costs, alerts, just-in-time education, reimbursement and scheduling practice management functions) plus immediate outcome scoring in support of pay-for-performance quality metrics, compensated disability determinations and the potential for clinical research and academic promotion.

Recently at UC Davis, the Trauma Register program has become available on all hospital fixed and wireless workstations. Direct physician and patient structured outcome data entry is now possible and has been accomplished in 232/1586 patient records created there thus far.

Summary:

Middleton et al¹⁶ have identified a number of stumbling blocks in sustaining clinical information systems. They include concerns about data quality, malaligned or missing incentives, uncertain product viability (vaporware) and the lack of standards and interoperability (integration). All clinical information system development must continually address these concerns.

Without direct structured data entry, complete and consistent evaluation cannot be assured. In order to sustain the delivery of substantial input incentives and input load sharing data exchange via the hospital network, high levels of administrative support are usually required.

Although the Trauma Register program has thus far been unable to combine direct physician and patient structured data entry of the basic outcome analysis components at a single institution, the potential for doing so has been demonstrated by the accomplishment of the separate collection of these same components at different institutions. Eventually, a confluence of favorable political and technical factors should permit the occurence of routine, complete, outcome determinations based on direct structured data entry in the same clinical setting.

Conclusion:

The patient and the physician must be provided the means to fully express themselves with well specified, uniform, direct structured data entry into information systems - particularly those from which clinical guidelines will be derived. In doing so, the guidance of Hill¹⁷ from 1953 still applies - "One must go seek more facts, paying less attention to technique of handling the data and far more to the development and perfection of the method for obtaining them."

Links:

• AAOS OKO Evidence Based Practice Portal
• Data Collection Analysis, HWB Annual Meeting 2005

Other Orthopaedic Trauma Registries:

• MGH OrthoDude
• Edinburgh Orthopaedic Trauma Database
• OTA Web Database
• SIGN Web Database

References:

1. Gillott AR, Thomas JM, Forrester C. Development of a statewide trauma registry. J Trauma. 1989 Dec;29(12):1667-72. PMID: 2593198 [MEDLINE]

2. Champion HR, Cushing B., Trauma data bases. J Trauma. 1995 Nov;39(5):813-4. No abstract available. PMID: 7473994 [MEDLINE]

3. Bourne RB, Keller RB., Controversy - Outcomes Research. Spine. 1995 Feb 1;20(3):384-387. No abstract available.

4. Engh CA, Engh CA Jr, Nagowski JP, Hopper RH Jr., Database production and maintenance. Clin Orthop Relat Res. 2004 Apr;(421):35-42. PMID: 15123923 [MEDLINE]

5. Brand DA, Krag MH, Hausman MR, Trainor KF, Akelman E, Rudicel SA, Southwick WO., A patient registry for orthopedic surgery. Clin Orthop Relat Res. 1990 Mar;(252):262-9. PMID: 2302892 [MEDLINE]

6. Pace et al,. Validity of ICD-9-CM database for THRA, AAOS Atlanta Paper 398 2/25/96

7. Burns CM,. The 1990 Fraser Gurd Lecture: a Canadian trauma registry system--nine years experience. PMID: 2056552 [MEDLINE]

8. Ellwood PM,. Shattuck lecture--outcomes management. A technology of patient experience. N Engl J Med. 1988 Jun 9;318(23):1549-56. No abstract available. PMID: 3367968 [MEDLINE]

9. Patel VL, Kushniruk AW, Yang S, Yale JF., Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. J Am Med Inform Assoc. 2000 Nov-Dec;7(6):569-85. PMID: 11062231 [MEDLINE]

10. McDonald CJ., The Barriers to Electronic Medical Record Systems and How to Overcome Them, J Am Med Inform Assoc. 1997 May­Jun; 4(3): 213­221. [FULL TEXT]

11. Sacha E Bleeker, Gerarda Derksen-Lubsen, Astrid M van Ginneken, Johan van der Lei and Henriëtte A Moll, BMC Medical Informatics and Decision Making 2006, 6:29 [FULL TEXT]

12. Gardner R. Davies keynote lecture. Proceedings of the Computer-based Patient Record Institute Conference. Washington, DC: CPRI, 1998.

13. Lorenzi NM, Riley RT, Managing Change - An Overview, J Am Med Inform Assoc. 2000 Mar­Apr; 7(2): 116­124. [FULL TEXT]

14. C. Court-Brown and M. M. McQueen, Trauma management in the United Kingdom, J Bone Joint Surg Br 1997 79-B: 1-3. [PDF]

15. Liu, AMA Professional Resources - Medical Ethics - April 2005. - with reference to Mallon WJ. Ernest Amory Codman: The End Result of a Life in Medicine. Philadelphia, Pa: WB Saunders; 2000.

16. Middleton B, Hammond WE, Brennan PF, Cooper GF, Accelerating U.S. EHR adoption: how to get there from here. recommendations based on the 2004 ACMI retreat., J Am Med Inform Assoc. 2005 Jan-Feb;12(1):13-9. Epub 2004 Oct 18. [FULL TEXT]

17. Hill AB, Observation and Experiment, N Engl J Med. 1953:248:995-1001 No abstract available.

Commentary: