An Empirical Evaluation of Assessing Software Requirements for Legal Compliance

Abstract—Software engineers regularly build systems that are required to comply with laws and regulations. To this end, software engineers must determine which requirements have met or exceeded their legal obligations and which requirements have not. Research is needed to better understand how to support software engineers in making these determinations. In this paper, we describe an experiment in which we asked graduate-level software engineering students to determine whether a set of software requirements for an electronic health record system met or exceeded their corresponding legal obligations as expressed in regulations created pursuant to the U.S. Health Insurance Portability and Accountability Act (HIPAA). We compare the determinations made by graduate students with the determinations made by HIPAA compliance subject matter experts. Additionally, we contrast these results with those generated by a legal requirements triage algorithm. Our findings suggest that the average graduate-level software engineer is ill-prepared to write legally compliant software with any confidence and that domain experts are an absolute necessity. Our findings also indicate the potential utility of legal requirements metrics in aiding software engineers as they make legal compliance decisions. Keywords-legal requirements, requirements metrics I. INTRODUCTION How do software engineers make legal compliance decisions? The common approach to domain-specific decisions in requirements engineering is to hire or consult with domain experts [25]. Lawyers serve as extremely capable domain experts for legal compliance concerns, but their services are expensive. As a result, lawyers need to be consulted only in situations where software engineers cannot provide accurate or reliable answers. To our knowledge, no researchers have empirically examined if software engineers are able to accurately identify whether software requirements meet or exceed their legal obligations. The increasingly regulated environments in which software systems must operate makes legal compliance a critical concern for software engineers. In the United States, for example, there are extensive laws and regulations governing software used in areas such as healthcare, corporate accounting, and financial systems. The Health Insurance Portability and Accountability Act of 1996 (HIPAA) prompted the adoption of regulations governing electronic health records (EHR) systems. Three years later, passage of the Graham-Leach-Bliley Act (GLBA) spurred regulations governing the financial services industry. The Sarbanes-Oxley Act of 2002 (SOX) regulates corporate accounting practices. Each of these laws, along with their accompanying regulations, governs a myriad of software systems that must be updated and maintained by software engineers. When new laws and regulations take effect, or existing laws and regulations are amended, software engineers need to quickly assess the ways in which those legal changes will impact software system requirements. Otto and Antón highlight the need to manage the evolution of laws and regulations over time in order to facilitate legal compliance efforts [29]. For example, in 2009 Congress amended HIPAA with the HITECH Act. Among the HITECH Act’s provisions were new data breach notification requirements. In addition, administrative agencies update regulations regularly in response to new laws and changing conditions. Again using the HITECH Act as an example, both the Department of Health & Human Services and the Federal Trade Commission promulgated new regulations pursuant to the HITECH Act within six months of the act becoming law. In summary, requirements for a software system must comply with governing laws and regulations, which also requires that software engineers stay current with changes in those laws and regulations. As a result, software engineers must be able to discern those requirements that are implementation-ready from those that require further disambiguation and/or elaboration. A requirement is considered to be Legally Implementation Ready (LIR) if it meets or exceeds its legal obligations as expressed in relevant regulations. In this paper, we discuss an experiment in which we employ a set of EHR system requirements, coupled with a particular HIPAA regulatory section with which the system must comply. In the experiment, thirty-two graduate-level computer science students evaluated the EHR requirements for legal implementation readiness. The results of this study are compared with the findings of three subject matter experts—individuals with both software engineering and HIPAA compliance expertise—who evaluated the same set of requirements for legal implementation readiness. In addition, we employed our legal requirements triage algorithm [21, 24], which uses legal requirements metrics to assess the same set of requirements for legal implementation readiness and compare the algorithm’s results to both the student and expert results. Our legal requirements metrics comprise simple, consistent attributes of a legal text that measure the dependency, complexity, and maturity of each attribute as it relates to the requirements for the software system. We also created a statistical model using our legal requirements metrics to determine whether or not they aid legal compliance decisions. Our study reveals that graduate-level computer science students exhibit little consensus about LIR requirements, and they poorly identified which requirements were LIR. Our study validates the need for subject matter experts or additional guidance for individuals making legal compliance decisions. Finally, our study reveals that the metrics used in our legal requirements triage algorithm are useful in identifying LIR requirements; the strengths and limitations of the metrics and algorithm are presented herein. The remainder of this paper is organized as follows. In Section II, we discuss relevant related work. In Section III, we describe our experimental design. In Section IV, we present the results of our experiment. In Section V, we describe potential threats to the validity of our experiment. Finally, in Section VI, we discuss the meaning of our results and potential future work.