Proposing SQL Statement Coverage Metrics

Ben Smith, Younghee Shin, and Laurie Williams

Abstract

An increasing number of cyber attacks are occurring at the application layer when attackers use malicious input. These input validation vulnerabilities can be exploited by (among others) SQL injection, cross site scripting, and buffer overflow attacks. Statement coverage and similar test adequacy metrics have historically been used to assess the level of functional and unit testing which has been performed on an application. However, these currently-available metrics do not highlight how well the system protects itself through validation. In this paper, we propose two SQL injection input validation testing adequacy metrics: target statement coverage and input variable coverage. A test suite which satisfies both adequacy criteria can be leveraged as a solid foundation for input validation scanning with a blacklist. To determine whether it is feasible to calculate values for our two metrics, we perform a case study on a web healthcare application and discuss some issues in implementation we have encountered. We find that the web healthcare application scored 96.7% target statement coverage and 98.5% input variable coverage

1. Introduction

According to the National Vulnerability Database (NVD)¹, more than half of all of the ever-increasing number of cyber vulnerabilities reported in 2002-2006 were input validation vulnerabilities. As Figure 1 shows, the number of input validation vulnerabilities is still increasing.

PLACEHOLDER FOR FIGURE 1²

Figure 1 illustrates the number of reported instances of each type of cyber vulnerability listed in the series legend for each year displayed in the x-axis. The curve with the square shaped points is the sum of all reported vulnerabilities that fall into the categories “SQL injection”, “XSS”, or “buffer overflow” when querying the National Vulnerability Database. The curve with diamond shaped points represents all cyber vulnerabilities reported for the year in the x-axis. For several years now, the number of reported input validation vulnerabilities has been half the total number of reported vulnerabilities. Additionally, the graph demonstrates that these curves are monotonically increasing; indicating that we are unlikely to see a drop in the future in ratio of reported input validation vulnerabilities.

Input validation testing is the process of writing and running test cases to investigate how a system responds to malicious input with the intention of using tests to mitigate the risk of a security threat. Input validation testing can increase confidence that input validation has been properly implemented. The goal of input validation testing is to check whether input is validated against constraints given for the input. Input validation testing should test both whether legal input is accepted, and whether illegal input is rejected. A coverage metric can quantify the extent to which this goal has been met. Various coverage criteria have been defined based on the target of testing (specification or program as a target) and underlying testing methods (structural, fault-based and error-based)^[19]. Statement coverage and branch coverage are well-known program-based structural coverage criteria^[19].

However, current structural coverage metrics and the tools which implement them do not provide specific information about insufficient or missing input validation. New coverage criteria to measure the adequacy of input validation testing can be used to highlight a level of security testing. Our research objective is to propose and to validate two input validation testing adequacy metrics related to SQL injection vulnerabilities. Our current input validation coverage criteria consist of two experimental metrics: input variable coverage, which measures the percentage of input variables used in at least one test; and target statement coverage, which measures the percentage of SQL statements executed in at least one test.

An input variable is any dynamic, user-assigned variable which an attacker could manipulate to send malicious input to the system. In the context of the Web, any field on a web form is an input variable as well as any number of other client-side input spaces. Within the context of SQL injection attacks, input variables are any variable which is sent to the database management system, as will be illustrated in further detail in Section 2. A target statement is any statement in an application which is subject to attack via malicious input; for this paper, our target statements will be all SQL statements found in production code. Other input sources can be leveraged to form an attack, but we have chosen not to focus on them for this study because they comprise less than half of recently reported cyber vulnerabilities (see Figure 1 and explanation).

In practice, even software development teams who use metrics such as traditional statement coverage often do not achieve 100% values in these metrics before production^[1]. If the lines left uncovered contain target statements, traditional statement coverage could be very high while little to no input validation testing is performed on the system. A target statement or input variable which is involved in at least one test might achieve high input validation coverage metrics yet still remain insecure if the test case(s) did not utilize a malicious form of input. However, a system with a high score in the metrics we define has a foundation for thorough input validation testing. Testers can relatively easily reuse existing test cases with multiple forms of good and malicious input. Our vision is to automate such reuse.

We evaluated our metrics on the server-side code of a Java Server Pages web healthcare application that had an extensive set of JUnit³ test cases. We manually counted the number of input variables and SQL statements found in this system and dynamically recorded how many of these statements and variables are used in executing a given test set. The rest of this paper is organized as follows: First, Section 2 defines SQL injection attacks. Then, Section 3 introduces our experimental metrics. Section 4 provides a brief summary of related work. Next, Section 5 describes our case study and application of our technique. Section 6 reports the results of our study and discusses their implications. Then, Section 7 illustrates some limitations on our technique and our metrics. Finally, Section 8 concludes and discusses the future use and development of our metrics.

2. Background

Section 2.1 explains the fundamental difference between traditional testing and security testing. Then, Section 2.2 describes SQL injection.

2.1 Testing for Security

Web applications are inherently insecure^[15] and web applications’ attackers look the same as any other customer to the server^[12]. Developers should, but typically do not, focus on building security into web applications ^[6]. Security has been added to the list of web application quality criteria^[11] and the result is that companies have begun to incorporate security testing (including input validation testing) into their development methodologies^[3]. Security testing is contrasted from traditional testing, as illustrated by Figure 2: Functional vs. Security Testing, adapted from^[17].

Figure 2. Intended vs. Actual Behavior, (adapted from ^[17])

Represented by the left-hand circle in Figure 2, the current software development paradigm includes a list of testing strategies to ensure the correctness of an application in functionality and usability as indicated by a requirements specification. With respect to intended correctness, verification typically entails creating test cases designed to discover faults by causing failures. Oracles tell us what the system should do and failures tell us that the system does not do what it is supposed to do. The right-hand circle in Figure 2 indicates that we validate not only that the system does what it should, but also that the system does not do what it should not: the right-hand circle represents a failure occurring in the system which causes a security problem. The circles intersect because some intended functionality can cause indirect vulnerabilities because privacy and security were not considered in designing the required functionality^[17]. Testing for functionality only validates that the application achieves what was written in the requirements specification. Testing for security validates that the application prevents undesirable security risks from occurring, even when the nature of this functionality is spread across several modules and might be due to an oversight in the application’s design. To adapt to the new paradigm, companies have started to incorporate new techniques. Some companies use vulnerability scanners, which behave like a hacker to make automated attempts at gaining access or misusing the system to discover its flaws^[4]. A blacklist is a representative or comprehensive set of all input validation attacks of a given type (such as SQL injection, see Section 2.2). These vulnerability scanners typically use a blacklist to test potential vulnerabilities against all attacks (or a set of representative attacks). Coverage criteria for target statements can help companies assess how much of their system has the framework for a range of input validation testing. A vulnerability scanner is ineffective if its blacklist is not tested against every target statement in the system.

2.2 SQL Injection Attacks

A SQL injection attack is performed when a user exploits a lack of input validation to force unintended system behavior by altering the logical structure of a SQL statement with special characters. The lack of input validation to prevent SQL injection attacks is known as a SQL injection vulnerability^{[2, 5, 6, 8, 9, 13-16]}. Our example of this type of input validation vulnerability begins with the login form presented in Figure 3.

Figure 3. Example login form

Usernames typically consist of alphanumeric characters, underscores, periods and dashes. Passwords also typically consist of these character ranges and additionally allow for some other non-alphanumeric characters such as $, ^ or #. The authentication mechanism functions by a code segment resembling the one in Figure 4. Assume there exists some table maintaining a list of all usernames, passwords, and most likely some indication of the role of each unique username.

 //for simplicity, this example is given in PHP. 
 //first, extract the input values from the form 
 $username = $_POST[‘username’]; 
 $password = $_POST[‘password’]; 
 //query the database for a user with username/pw 
 $result = mysql_query( 
“select * from users where username = 
‘$username’ AND password = ‘$password’”); 
 //extract the first row of the resultset 
 $firstresult = mysql_fetch_array($result); 
 //extract the “role” column from the result 
 $role = $firstresult[‘role’]; 
 //set a cookie for the user with their role 
 setcookie(“userrole”, $role); 

Figure 4. Example authentication code

The code in Figure 4 performs the following. First, query the database for every entry with the entered username and password. Typically, we use the first row of returned SQL results (which is retrieved by mysql_fetch_array and stored in $firstresult) because the web application (or the database management system) will ensure that there are no duplicate usernames and will ensure that every user name is given the appropriate role. Finally, we extract the role field from the first result and give the user a cookie⁴, which allows the login to be persistent (i.e., the user does not have to login to view every protected page).

The example we have presented in Figure 4 performs no input validation, and as a result the example contains at least three input validation vulnerability locations. The first two are the username and password fields as given in the web form in Figure 3. An attacker could cause the code fragment change shown in Figure 5 simply by entering the SQL command fragment “‘ OR 1=1 -- AND" in the input field instead of any valid user name in Figure 3.

 //from Figure 4; original code 
 $result = mysql_query( 
  “select * from users where username = 
  ‘$username’ AND password = ‘$password’”);

 //code with inserted attack parameters 
 $result = mysql_query( 
  “select * from users where username = 
  ‘’ OR 1=1 -- AND password = ‘PASSWORD’”); 

Figure 5. Example SQL statement, before and after

The single quotation mark (') indicates to the SQL parser that the character sequence for the username column is closed, the fragment OR 1=1 is interpreted as always true, and the hyphens (--) tells the parser that the SQL command is over and the fragment of the query after the hyphens is a comment. With these values, the $result variable contains a list of every user in the table (and their associated role) because the where clause is always true. The first listing returned from the database is unknown and will vary based on the database configuration. Regardless, the role of the user in the first returned row will be extracted and assigned to a cookie on the attacker’s machine. The consequence is as follows: Assuming the attacker is not a registered user of the system, he or she has just been granted unauthorized access to the system with the role (and identity) associated with the first username in the table. The password field shown in Figure 3 is also vulnerable, but we do not demonstrate this attack for space reasons. Because no input validation was performed, the system can be exploited for a use that was unintended by its developers.

The exploitation of the third vulnerability requires slightly more work than the first two, but is more threatening. Presumably, the developer of this example web application provides different content to a given web user (or provides no content at all) depending on the role parameter, which is stored in a cookie. An example code for the design decision of using a cookie is Figure 6.

The $_COOKIE[‘role’] macro extracts the value stored on the user’s machine for the parameter passed (in this case “role”). The web application provides one set of content for users with the administrator role and another set of content for those with the employee role. If the role parameter is anything else, the user is redirected to authrequired.html, which presumably contains some type of message to the user that authentication is required to access the requested page. The vulnerability stems from the relatively well-known fact that HTTP cookies are usually stored in a text file on the user’s machine. In this case, the attacker need only to edit this file and see that there is a parameter named “role” and a reasonable guess for the authentication value would be “admin”. The consequence is as follows: If the attacker succeeds in guessing the correct value, the system provides content to a user who was unauthorized to view it and the system has been exploited.

 if ($_COOKIE[‘role’] == ‘admin’) 
 { 
  //give admin access 
 } 
 else if ($_COOKIE[‘role’] == ‘employee’) 
 { 
  //give employee access 
 } 
 else 
 { 
  //no role or unrecognizable role, 
  //redirect to an error page. 
  header(“Location: authrequired.html”); 
 } 

Figure 6. Example authentication persistence

A countermeasure for the form input field vulnerability is simply to escape all control characters (such as ‘ or #) from the input variables. For the cookie vulnerability, a countermeasure would be to dynamically generate a unique identifier for the current session and store that in the cookie as well as the associated user role. Because these vulnerabilities can be prevented with input validation, they are known as input validation vulnerabilities. Figure 6 is not a SQL injection attack; however it still represents an input validation vulnerability. We have included it here in the interest of completeness, but we will not focus on this type of vulnerability in the rest of this paper.

Although a number of techniques exist to mitigate the risks posed by SQL injection vulnerabilities^{[2, 6, 8, 9, 13, 14]}, none of these techniques propose a methodology of adequacy as ensured by measuring how many commands issued to a database management system are tested by the test suite.

3. Coverage Criteria

We define two criteria for input validation testing coverage. Client-side input validation can be bypassed by attackers ^[7]. Therefore, we only measure the coverage of server-side code. The followings are basic terms to be used to define input validation coverage criteria.

• Target statement: A target statement (within our context) is a SQL statement which could cause a security problem when malicious input is used. For example, consider the statement

 java.sql.Statement.executeQuery(String sql) 

A SQL injection attack can happen when an attacker uses maliciously-devised input as explained in Section 2. Let T be the set of all the SQL statements in an application.

• Input variable: An input variable is any variable in the serverside production code which is dynamically user-assigned and sent to the database management system. Let F represent the set of all input variables in all SQL statements occurring in the production code.

3.1 Target Statement Coverage

Target statement coverage measures the percentage of SQL statements executed at least once during execution of the test suite.

Definition: A set of input validation tests satisfies target statement coverage if and only if for every SQL statement t ∈ T, there exists at least one test in the input validation test cases which executes t.

Metric: The target statement coverage criterion can be measured by the percentage of SQL statements tested at least once by the test set out of total SQL statements.

Server-side target statement coverage = PLACEHOLDER

where Test(t) is a SQL statement tested at least once.

Coverage interpretation: A low value for target statement coverage indicates that testing was insufficient. Programmers need to add more test cases to the input validation set for untested SQL statements to improve target statement coverage.

3.2 Input Variable Coverage

Input variable coverage measures the percentage of input variables used in at least one test at the server-side. Input variable coverage does not consider all the constraints for the input variable.

Definition: A set of tests satisfies input variable coverage criterion if and only if for every input variable f ∈ F, there exists at least one test that uses that input variable at least once.

Metric: The input variable coverage criterion can be measured by the percentage of input variables tested at least once by the test set out of total number of input variables found in any target statement in the production code of the system.

Input variable coverage = PLACEHOLDER

where Test(f) is an input variable used in at least one test.

Coverage interpretation: A low value for input variable coverage indicates that input validation testing is insufficient. Programmers need to add more test cases for untested input variables to improve input variable coverage.

We note here that a test set which achieves 100% input variable coverage and 100% target statement coverage may not contain any tests with malicious input. Consider a test set which satisfies both coverage criteria and leverages a blacklist to test for input validation attacks. This test set ensures that every input variable in every target statement is tested with every attack in the blacklist.

The relationship between target statement coverage and input variable coverage is not yet known; however, we contend that input variable coverage is a useful, finer-grained measurement.

Input variable coverage has the effect of weighting a target statement which has more input variables more heavily. Since most input variables are each a separate potential vulnerability if not adequately validated, a target statement which contains more input variables is of a higher threat level.

9. References

^[1] B. Beizer, Software testing techniques: Van Nostrand Reinhold Co. New York, NY, USA, 1990.

^[2] S. W. Boyd and A. D. Keromytis, "SQLrand: Preventing SQL injection attacks," in Proceedings of the 2nd Applied Cryptography and Network Security (ACNS) Conference, Yellow Mountain, China, pp. 292-304, 2004.

^[3] B. Brenner, "CSI 2007: Developers need Web application security assistance," in SearchSecurity.com, 2007.

^[4] M. Cobb, "Making the case for Web application vulnerability scanners," in SearchSecurity.com, 2007.

^[5] W. G. Halfond, J. Viegas, and A. Orso, "A Classification of SQL-Injection Attacks and Countermeasures," in Proceedings of the International Symposium on Secure Software Engineering, March, Arlington, VA, 2006.

^[6] W. G. J. Halfond and A. Orso, "AMNESIA: analysis and monitoring for NEutralizing SQL-injection attacks," in Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, Long Beach, CA, USA, pp. 174-183, 2005.

^[7] W. G. J. Halfond and A. Orso, "Command-Form Coverage for Testing Database Applications," Proceedings of the IEEE and ACM International Conference on Automated Software Engineering, pp. 69–78, 2006.

^[8] Y. W. Huang, S. K. Huang, T. P. Lin, and C. H. Tsai, "Web application security assessment by fault injection and behavior monitoring," in Proceedings of the 12th International Conference on World Wide Web, Budapest, Hungary, pp. 148-159, 2003.

^[9] S. Kals, E. Kirda, C. Kruegel, and N. Jovanovic, "SecuBat: a web vulnerability scanner," in Proceedings of the 15th international conference on World Wide Web, Edinburgh, Scotland pp. 247-256, 2006.

^[10] G. McGraw, Software Security: Building Security in. Upper Saddle River, NJ: Addison-Wesley Professional, 2006.

^[11] J. Offutt, "Quality attributes of Web software applications," IEEE Software, vol. 19, no. 2, pp. 25-32, 2002.

^[12] E. Ogren, "App Security's Evolution," in DarkReading.com, 2007.

^[13] T. Pietraszek and C. V. Berghe, "Defending against injection attacks through context-sensitive string evaluation," in Recent Advances in Intrusion Detection (RAID). Seattle, WA, 2005.

^[14] F. S. Rietta, "Application layer intrusion detection for SQL injection," in Proceedings of the 44th annual southeast regional conference, New York, NY, pp. 531-536, 2006.

^[15] D. Scott and R. Sharp, "Developing secure Web applications," Internet Computing, IEEE, vol. 6, no. 6, pp. 38-45, 2002.

^[16] Z. Su and G. Wassermann, "The essence of command injection attacks in web applications," in Proceedings of the Annual Symposium on Principles of Programming Languages, Charleston, SC, pp. 372-382, 2006.

^[17] H. H. Thompson and J. A. Whittaker, "Testing for software security," Dr. Dobb's Journal, vol. 27, no. 11, pp. 24-34, 2002.

^[18] D. Willmor and S. M. Embury, "Exploring test adequacy for database systems," in Proceedings of the 3rd UK Software Testing Research Workshop, Sheffield, UK, pp. p123-133, 2005.

^[19] H. Zhu, P. A. V. Hall, and J. H. R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, 1997.

10. Footnotes

1. http://nvd.nist.gov/
2. In Figure 1, we counted the reported instances of vulnerabilities by using the keywords "SQL injection", "cross-site scripting", "XSS", and "buffer overflow" within the input validation error category from NVD.
3. http://www.junit.org
4. A cookie is a piece of information that is sent by a web server when a user first accesses the website and saved to a local file. The cookie is then used in consecutive requests to identify the user to the server. See http://www.ietf.org/rfc/rfc2109.txt.