Modifying Without a Trace: High-level Audit Guidelines are Inadequate for Electronic Health Record Audit Mechanisms: Difference between revisions

← Older edit

@@ Line 334: / Line 334: @@
 === 4.2. Low-level Assessment using Black-box Test Cases ===
+Our low-level assessment of user-based non-repudiation involves constructing a black-box test plan for testing an EHR system’s recording of ''specific'' auditable events (such as “view diagnosis data”). In this paper, we briefly describe the process for the audit test cases used to evaluate user-based non-repudiation audit functionality.  We developed this methodology in earlier work<sup>[14]</sup>.
+In 2006, through a consensus-based process that engaged stakeholders, CCHIT defined certification criteria focused on the functional capabilities that should be included in ambulatory (outpatient) and inpatient EHR systems.  The requirements specifications contain 284 different functional descriptions of EHR behavior.
+The CCHIT ambulatory certification criteria contain eight requirements related to audit.  The audit requirements contain functionality such as “The system shall allow an authorized administrator to set the inclusion or exclusion of auditable events based on organizational policy & operating requirements/limits.”  One CCHIT audit criterion states that the set of auditable events in an EHR system should include the following fourteen items:
+# Application start/stop
+# User login/logout
+# Session timeout
+# Account lockout
+# Patient Record created/viewed/updated/deleted
+# Scheduling
+# Query
+# Order
+# Node-authentication failure
+# Signature created/validated
+# PHI Export (e.g. print)
+# PHI import
+# Security administration events
+# Backup and restore
+The list is provided here verbatim from the CCHIT ambulatory criteria.  The criteria are vague. For example, the phrase “security administration events” is undefined and could relate to authentication attempts, deletion of log files, or assigning user privileges. Likewise the term “scheduling” could relate to scheduling patient appointments, scheduling system backups, or scheduling system down-time for maintenance. The interpretation of these phrases varies, and the intended meanings are ambiguous.
+Due to the vagueness in these auditable events, we elected to approach the CCHIT certification criteria as a general functional requirements specification. The criteria describe functionality for EHR systems, such as editing a patient’s health record, signing a note about a patient, and indicating advance directives (e.g. a do-not-resuscitate order). Using these functional CCHIT requirements<sup>[2]</sup>, we develop a set of 58 black-box test cases that assess the ability of an EHR system to audit the user actions specified by these CCHIT requirements.  These test cases all involve a registered user performing a given action within the EHR system, therefore representing an assessment of user-based non-repudiation within each EHR system. The 58 test cases correspond to 58 individual CCHIT requirements statements.  Our test plan covers the 20.4% of the CCHIT requirements that are relevant to personal or protected health information.  The remaining 79.6% of the CCHIT requirements do not pertain to personal health information, and therefore do not necessitate an audit record for user-based non-repudiation.
+We iterated through each of the 284 ambulatory CCHIT requirements, extracting keywords and applying the template to produce a test case when necessary. We generate a test case from a specific requirement based on keywords within the requirements statement.  We know that a CCHIT requirements statement should result in a test case based on certain keywords within the requirements statement.  For example, requirements that include phrases like “problem list,” “clinical documents,” and “diagnostic test” all indicate the user’s interaction with a piece of a patient’s protected health information.
+Additionally, we extract an action phrase (e.g. “edit”) and an object phrase (e.g. “demographics”) from each relevant requirement to construct the black-box test case.  We present the template used for these black-box tests in Section 4.2.1, and present an example of a test case and its corresponding requirement in Section 4.2.2.
 ==== 4.2.1 Audit Test Case Template ====
+Test Procedure Template:
+# Authenticate as <''insert a registered user name''>.
+# Open the user interface for <''insert action phrase''>ing an <''insert object phrase''>.
+# Verb an <''insert object phrase''>with details.
+# Logout as <''insert a registered user name''>.
+# Authenticate as <''insert an administrator’s user name''>.
+# Open the audit records for today’s date.
+Expected Results Template:
+* The audit records should show that registered user <''insert action phrase''>ed an <''insert object phrase''>.
+* The audit records should be clearly readable and easily accessible.
 ==== 4.2.2 Audit Test Case Example ====
+Example Natural Language Artifact:
+* CCHIT Criteria: AM 03.08.01 – The system shall provide the ability to associate orders and medications with one or more codified problems/diagnoses.
+Example Test Procedure:
+# Authenticate as Dr. Robert Alexander.
+# Remove the association between Theodore S. Smith’s Hypertension diagnosis and Zantac.
+# Add the association back between Theodore S. Smith’s Hypertension diagnosis and Zantac.
+# Logout as Dr. Robert Alexander.
+# Authenticate as Denny Hudzinger.
+# Open the audit records for today’s date. If necessary, focus on patient Theodore S. Smith.
+Example Expected Results:
+* The audit records should show adding and removing the association of Theodore S. Smith’s Hypertension diagnosis and Zantac, both linked to Dr. Robert Alexander, and with today’s date.
+* The audit records should be clearly readable and easily accessible
 == 5. Case Studies ==
+Section 5.1 describes the EHR systems we used in this case study. Section 5.2 describes our EHR audit mechanism assessment based on the high-level assessment criteria from Section 4.1.  Then, Section 5.3 describes our low-level black-box test case evaluation of three open-source EHR systems.
 === 5.1. Open-source EHR Systems Studied ===
+In this study, we compare and contrast audit mechanisms from three open-source EHR systems. The criteria for inclusion in this study involved (1) being open-source for ease-of-access, and (2) having a fully-functional default demo deployment available online. For this study, we assess the following EHR systems:
+* Open Electronic Medical Records (OpenEMR)<sup>2</sup> system,
+* Open Medical Record System (OpenMRS)<sup>3</sup> system, with added Access Logging Module<sup>4</sup>.
+* Tolven Healthcare Innovations’s Electronic Clinician Health Record (eCHR)<sup>5</sup> system, with added Performance Plugin<sup>6</sup> module
+A summary of these software applications appears in Table 2.
 === 5.2. High-level User-based Non-repudiation Assessment ===