Proposing SQL Statement Coverage Metrics: Difference between revisions

Line 4:

== 1. Introduction ==

According to the ~~[http://nvd.nist.gov/~~ National Vulnerability Database] (NVD), more than half of all of the ever-increasing number of cyber vulnerabilities reported in 2002-2006 were input validation vulnerabilities. As Figure 1 shows, the number of input validation vulnerabilities is still increasing.

According to the National Vulnerability Database '''PUT REFERENCE HERE SOMEHOW''' (NVD), more than half of all of the ever-increasing number of cyber vulnerabilities reported in 2002-2006 were input validation vulnerabilities. As Figure 1 shows, the number of input validation vulnerabilities is still increasing.

'''PLACEHOLDER FOR FIGURE 1'''

Line 17:

An ''input variable'' is any dynamic, user-assigned variable which an attacker could manipulate to send malicious input to the system. In the context of the Web, any field on a web form is an input variable as well as any number of other client-side input spaces. Within the context of SQL injection attacks, input variables are any variable which is sent to the database management system, as will be illustrated in further detail in Section 2. A target statement is any statement in an application which is subject to attack via malicious input; for this paper, our target statements will be all SQL statements found in production code. Other input sources can be leveraged to form an attack, but we have chosen not to focus on them for this study because they comprise less than half of recently reported cyber vulnerabilities (see Figure 1 and explanation).

In practice, even software development teams who use metrics such as traditional statement coverage often do not achieve 100% values in these metrics before production [1]. If the lines left uncovered contain target statements, traditional statement coverage could be very high while little to no input validation testing is performed on the system. A target statement or input variable which is involved in at least one test might achieve high input validation coverage metrics yet still remain insecure if the test case(s) did not utilize a malicious form of input. However, a system with a high score in the metrics we define has a foundation for thorough input validation testing. Testers can relatively easily reuse existing test cases with multiple forms of good and malicious input. Our vision is to automate such reuse.

In practice, even software development teams who use metrics such as traditional statement coverage often do not achieve 100% values in these metrics before production[1]. If the lines left uncovered contain target statements, traditional statement coverage could be very high while little to no input validation testing is performed on the system. A target statement or input variable which is involved in at least one test might achieve high input validation coverage metrics yet still remain insecure if the test case(s) did not utilize a malicious form of input. However, a system with a high score in the metrics we define has a foundation for thorough input validation testing. Testers can relatively easily reuse existing test cases with multiple forms of good and malicious input. Our vision is to automate such reuse.

We evaluated our metrics on the server-side code of a Java Server Pages web healthcare application that had an extensive set of [http://www.junit.org/ JUnit] test cases. We manually counted the number of input variables and SQL statements found in this system and dynamically recorded how many of these statements and variables are used in executing a given test set. The rest of this paper is organized as follows: First, Section 2 defines SQL injection attacks. Then, Section 3 introduces our experimental metrics. Section 4 provides a brief summary of related work. Next, Section 5 describes our case study and application of our technique. Section 6 reports the results of our study and discusses their implications. Then, Section 7 illustrates some limitations on our technique and our metrics. Finally, Section 8 concludes and discusses the future use and development of our metrics.

Line 26:

=== 2.1 Testing for Security ===

Web applications are inherently insecure ~~~~[~~[#scottSharp|[4]]~~]~~~~ and web applications’ attackers look the same as any other customer to the server [[#Ogren|[5]]]. Developers should, but typically do not, focus on building security into web applications [[#mcgraw|[6]]]. Security has been added to the list of web application quality criteria [[#offutt|[7]]] and the result is that companies have begun to incorporate security testing (including input validation testing) into their development methodologies [[#brenner|[8]]]. Security testing is contrasted from traditional testing, as illustrated by Figure 2: Functional vs. Security Testing, adapted from [[#thompson|[9]]].

Web applications are inherently insecure [15] and web applications’ attackers look the same as any other customer to the server [[#Ogren|[5]]]. Developers should, but typically do not, focus on building security into web applications [[#mcgraw|[6]]]. Security has been added to the list of web application quality criteria [[#offutt|[7]]] and the result is that companies have begun to incorporate security testing (including input validation testing) into their development methodologies [[#brenner|[8]]]. Security testing is contrasted from traditional testing, as illustrated by Figure 2: Functional vs. Security Testing, adapted from [[#thompson|[9]]].

[[File:Sess-figure-2.png|thumb|'''Figure 2. Intended vs. Actual Behavior, (adapted from [[#thompson|[9]]])''']]

Line 53:

: [18] D. Willmor and S. M. Embury, "Exploring test adequacy for database systems," in Proceedings of the 3rd UK Software Testing Research Workshop, Sheffield, UK, pp. p123-133, 2005.

: [19] H. Zhu, P. A. V. Hall, and J. H. R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, 1997.

: [20] http://nvd.nist.gov/

: [21] http://www.junit.org

@@ Line 4: / Line 4: @@
 == 1. Introduction ==
-According to the [http://nvd.nist.gov/ National Vulnerability Database] (NVD), more than half of all of the ever-increasing number of cyber vulnerabilities reported in 2002-2006 were input validation vulnerabilities. As Figure 1 shows, the number of input validation vulnerabilities is still increasing.
+According to the National Vulnerability Database '''PUT REFERENCE HERE SOMEHOW''' (NVD), more than half of all of the ever-increasing number of cyber vulnerabilities reported in 2002-2006 were input validation vulnerabilities. As Figure 1 shows, the number of input validation vulnerabilities is still increasing.
 '''PLACEHOLDER FOR FIGURE 1'''
@@ Line 17: / Line 17: @@
 An ''input variable'' is any dynamic, user-assigned variable which an attacker could manipulate to send malicious input to the system. In the context of the Web, any field on a web form is an input variable as well as any number of other client-side input spaces. Within the context of SQL injection attacks, input variables are any variable which is sent to the database management system, as will be illustrated in further detail in Section 2. A target statement is any statement in an application which is subject to attack via malicious input; for this paper, our target statements will be all SQL statements found in production code. Other input sources can be leveraged to form an attack, but we have chosen not to focus on them for this study because they comprise less than half of recently reported cyber vulnerabilities (see Figure 1 and explanation).
-In practice, even software development teams who use metrics such as traditional statement coverage often do not achieve 100% values in these metrics before production [1]. If the lines left uncovered contain target statements, traditional statement coverage could be very high while little to no input validation testing is performed on the system. A target statement or input variable which is involved in at least one test might achieve high input validation coverage metrics yet still remain insecure if the test case(s) did not utilize a malicious form of input. However, a system with a high score in the metrics we define has a foundation for thorough input validation testing. Testers can relatively easily reuse existing test cases with multiple forms of good and malicious input. Our vision is to automate such reuse.
+In practice, even software development teams who use metrics such as traditional statement coverage often do not achieve 100% values in these metrics before production<sup>[1]</sup>. If the lines left uncovered contain target statements, traditional statement coverage could be very high while little to no input validation testing is performed on the system. A target statement or input variable which is involved in at least one test might achieve high input validation coverage metrics yet still remain insecure if the test case(s) did not utilize a malicious form of input. However, a system with a high score in the metrics we define has a foundation for thorough input validation testing. Testers can relatively easily reuse existing test cases with multiple forms of good and malicious input. Our vision is to automate such reuse.
 We evaluated our metrics on the server-side code of a Java Server Pages web healthcare application that had an extensive set of [http://www.junit.org/ JUnit] test cases. We manually counted the number of input variables and SQL statements found in this system and dynamically recorded how many of these statements and variables are used in executing a given test set. The rest of this paper is organized as follows: First, Section 2 defines SQL injection attacks. Then, Section 3 introduces our experimental metrics. Section 4 provides a brief summary of related work. Next, Section 5 describes our case study and application of our technique. Section 6 reports the results of our study and discusses their implications. Then, Section 7 illustrates some limitations on our technique and our metrics. Finally, Section 8 concludes and discusses the future use and development of our metrics.
@@ Line 26: / Line 26: @@
 === 2.1 Testing for Security ===
-Web applications are inherently insecure <sup>[[#scottSharp|[4]]]</sup> and web applications’ attackers look the same as any other customer to the server <sup>[[#Ogren|[5]]]</sup>. Developers should, but typically do not, focus on building security into web applications <sup>[[#mcgraw|[6]]]</sup>. Security has been added to the list of web application quality criteria <sup>[[#offutt|[7]]]</sup> and the result is that companies have begun to incorporate security testing (including input validation testing) into their development methodologies <sup>[[#brenner|[8]]]</sup>. Security testing is contrasted from traditional testing, as illustrated by Figure 2: Functional vs. Security Testing, adapted from <sup>[[#thompson|[9]]]</sup>.
+Web applications are inherently insecure [15] and web applications’ attackers look the same as any other customer to the server <sup>[[#Ogren|[5]]]</sup>. Developers should, but typically do not, focus on building security into web applications <sup>[[#mcgraw|[6]]]</sup>. Security has been added to the list of web application quality criteria <sup>[[#offutt|[7]]]</sup> and the result is that companies have begun to incorporate security testing (including input validation testing) into their development methodologies <sup>[[#brenner|[8]]]</sup>. Security testing is contrasted from traditional testing, as illustrated by Figure 2: Functional vs. Security Testing, adapted from <sup>[[#thompson|[9]]]</sup>.
 [[File:Sess-figure-2.png|thumb|'''Figure 2. Intended vs. Actual Behavior, (adapted from <sup>[[#thompson|[9]]]</sup>)''']]
@@ Line 53: / Line 53: @@
 : [18] D. Willmor and S. M. Embury, "Exploring test adequacy for database systems," in Proceedings of the 3rd UK Software Testing Research Workshop, Sheffield, UK, pp. p123-133, 2005.
 : [19] H. Zhu, P. A. V. Hall, and J. H. R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, 1997.
+: [20] http://nvd.nist.gov/
+: [21] http://www.junit.org

Proposing SQL Statement Coverage Metrics: Difference between revisions

Programsam (talk | contribs)