Modifying Without a Trace: High-level Audit Guidelines are Inadequate for Electronic Health Record Audit Mechanisms: Difference between revisions

← Older edit Newer edit →

@@ Line 334: / Line 334: @@
 === 4.2. Low-level Assessment using Black-box Test Cases ===
+Our low-level assessment of user-based non-repudiation involves constructing a black-box test plan for testing an EHR system’s recording of ''specific'' auditable events (such as “view diagnosis data”). In this paper, we briefly describe the process for the audit test cases used to evaluate user-based non-repudiation audit functionality.  We developed this methodology in earlier work<sup>[14]</sup>.
+In 2006, through a consensus-based process that engaged stakeholders, CCHIT defined certification criteria focused on the functional capabilities that should be included in ambulatory (outpatient) and inpatient EHR systems.  The requirements specifications contain 284 different functional descriptions of EHR behavior.
+The CCHIT ambulatory certification criteria contain eight requirements related to audit.  The audit requirements contain functionality such as “The system shall allow an authorized administrator to set the inclusion or exclusion of auditable events based on organizational policy & operating requirements/limits.”  One CCHIT audit criterion states that the set of auditable events in an EHR system should include the following fourteen items:
+# Application start/stop
+# User login/logout
+# Session timeout
+# Account lockout
+# Patient Record created/viewed/updated/deleted
+# Scheduling
+# Query
+# Order
+# Node-authentication failure
+# Signature created/validated
+# PHI Export (e.g. print)
+# PHI import
+# Security administration events
+# Backup and restore
+The list is provided here verbatim from the CCHIT ambulatory criteria.  The criteria are vague. For example, the phrase “security administration events” is undefined and could relate to authentication attempts, deletion of log files, or assigning user privileges. Likewise the term “scheduling” could relate to scheduling patient appointments, scheduling system backups, or scheduling system down-time for maintenance. The interpretation of these phrases varies, and the intended meanings are ambiguous.
+Due to the vagueness in these auditable events, we elected to approach the CCHIT certification criteria as a general functional requirements specification. The criteria describe functionality for EHR systems, such as editing a patient’s health record, signing a note about a patient, and indicating advance directives (e.g. a do-not-resuscitate order). Using these functional CCHIT requirements<sup>[2]</sup>, we develop a set of 58 black-box test cases that assess the ability of an EHR system to audit the user actions specified by these CCHIT requirements.  These test cases all involve a registered user performing a given action within the EHR system, therefore representing an assessment of user-based non-repudiation within each EHR system. The 58 test cases correspond to 58 individual CCHIT requirements statements.  Our test plan covers the 20.4% of the CCHIT requirements that are relevant to personal or protected health information.  The remaining 79.6% of the CCHIT requirements do not pertain to personal health information, and therefore do not necessitate an audit record for user-based non-repudiation.
+We iterated through each of the 284 ambulatory CCHIT requirements, extracting keywords and applying the template to produce a test case when necessary. We generate a test case from a specific requirement based on keywords within the requirements statement.  We know that a CCHIT requirements statement should result in a test case based on certain keywords within the requirements statement.  For example, requirements that include phrases like “problem list,” “clinical documents,” and “diagnostic test” all indicate the user’s interaction with a piece of a patient’s protected health information.
+Additionally, we extract an action phrase (e.g. “edit”) and an object phrase (e.g. “demographics”) from each relevant requirement to construct the black-box test case.  We present the template used for these black-box tests in Section 4.2.1, and present an example of a test case and its corresponding requirement in Section 4.2.2.
 ==== 4.2.1 Audit Test Case Template ====