Learning Better Inspection Optimization Policies

Markus Lumpe, Rajesh Vasa Faculty of ICT, 
Swinburne University of Technology
Hawthorn, Australia
{mlumpe,rvasa}@swin.edu.au 
Tim Menzies 
CSEE, 
West Virginia University
Morgantown, West Virginia
tim@menzies.us 
Burak Turhan 
Info. Processing Science, 
University of Oulu
Oulu, Finland
turhanb@computer.org 

Abstract- Recent research has shown the value of social metrics for defect prediction. Yet many repositories lack the information required for a social analysis. So, what other means exist to infer how developers interact around their code?  One option is static code metrics that have already demonstrated their usefulness in analyzing change in evolving software systems. But do they also help in defect prediction?  To answer this question we selected a small set of static code metrics to determine what classes are most “active” (i.e., the classes where the developers spend much time interacting with each other’s design and implementation decisions) in open-source Java systems that lack details about individual developers. In particular, we assessed the merit of these activity-centric measures in the context of “inspection optimization” – a technique that allows for reading the fewest lines of code in order to find the most defects. For the task of inspection optimization these activity measures perform as well as (usually, within 4%) a theoretical upper bound on the performance of any set of measures. As a result, we argue that activity-centric static code metrics are an excellent predictor for defects.
Keywords – Data Mining, defect predition, static measures

I.  Introduction
A remarkable recent discovery is that social metrics (that model the sociology of the programmers working on the code) can be an effective predictor for defect injection and removal [1]-[4]. For example, Guo et al. [4] demonstrate that the reputation of the developer who reported a bug relates to the odds that that bug will get fixed. 
A premise of the social metrics research is that code repositories contain more than just static code measures. But, not all code repositories contain detailed knowledge about how developers interact around a code base. Consider, for example, the Helix project [5], which has studied 40+ multi-year large open-source systems under active development. Many developers contributed to those systems but their code repositories are very weak sources for information regarding a developer’s social context.
Nevertheless, we show here that it is possible use static code measures to capture how programmers interact with their code. Specifically, it is feasible to find what parts of the code are most “active,” that is, are the focus of much of the shared attention of all developers working to organize behavior and functionality at suitable system-specific levels [6]-[8]. This opens intriguing options for guiding quality assurance processes. In particular, we demonstrate that a small set of activity-based static code metrics can serve as a good predictor for defects in object-oriented software.
The rest of this paper is structured as follows. The next section expands on this notion of “activity.” After that, the general field of defect prediction and inspection optimization is discussed. We then turn to the experiments showing the value of activity measures. We demonstrate that in open-source Java projects activity-based defect predictors work within 4% of a theoretical upper bound on predictor performance (this is the basis for our claim that a small set of static metrics can generate an excellent performance within the context of inspection optimization). The validity of our conclusions is then discussed, which will lead into a review of possible future directions for this work.  
II.  Activity
When do we call a software artifact, say a class, active?  We contend that activity arises when code is being modified, typically via enhancement or correction. This is change and we can detect and measure it through the evolution of the associated volumetric and structural properties of a class [6]. 
However, one surprising observation from the Helix studies [6]-[8] has been that (a) only a small set of highly active classes undergoes change frequently and (b) predictable patterns of modification emerge very early in the lifetime of a software system. Therefore, we ask whether the metrics used in Helix can also guide defect discovery, since change and defects are closely related concepts. In particular, we argue that change can lead to defects via:

•	Defect discovery: Since active classes are used more frequently by developers, then developers are most likely to discover their defects earlier. 
•	Defect injection: When developers work with active classes, they make occasional mistakes, some of which lead to defects. Since developers work on active classes more than other classes, then most developer defects accumulate in the active classes. 

(The second point was first proposed by Nagappan & Ball who say “code that changes many times prerelease will likely have more post-release defects than code that changes less over the same period of time” [9].)

Figure 1 shows our choice of measures of activity, each tagged with a rationale motivating its selection. These measures capture volumetric and the structural properties of a class and provide us with an empirical component for detecting and measuring change. Furthermore, these measures are sufficiently broad to encompass, from a design perspective, the amount of functionality as well as how the developers have structurally organized the solution, and how they chose to decompose the functionality.

Measure	Description	Rationale for selection	
Bugs	Number of bugs as annotated in the source code control logs	Used to check our predictions	
LOC	Lines of code in the class	Used to estimate inspection effort	
Getters	Number of get methods	Read responsibility allocation	
Setters	Number of set methods	Write responsibility allocation	
NoM	Number of methods	Breadth of functional decomposition	
InDegree	Number of other classes depending on this class	Coupling within design	
OutDegree	Number of other classes this class depends upon	Breadth of delegation	
Clustering Coefficient	Degree to which classes tend to cluster together	Density of design	

Figure 1: Measures used in this study (collected separately for each class).

The measures NoM, Getters, and Setters define simple class-based counts (see figure 2). For the complexity measures InDegree, OutDegree, and Clustering Coefficient, however, we need to construct a complete class dependency graph first. The class dependency graph captures the dependencies between these classes. That is, when a class uses either data or functionality from another class, there is a dependency between these classes. In the context of Java software, a dependency is created if a class inherits from a class, implements an interface, invokes a method on another class (including constructors), declares a field or local variable, uses an exception, or uses class types within a method declaration. Thus, a class dependency graph is an ordered pair , where N is a finite, nonempty set of types (i.e., classes and interfaces) and L is a finite, possibly empty, set of directed links between types (i.e.,) expressing the dependencies between classes. For the purpose of the metrics extraction, we analyze each node  in the graph to compute the structural complexity metrics of class C type node n represents as shown in Figure 2.
An important feature of these measures is that they are relatively easy to collect. For example, one measure we rely on for defect prediction is the Number of Getter Methods that developers have added to a class. Parsers for such simple measures are easy to obtain from early design representation (e.g., UML models) and can, with little effort, be adapted to new languages. Moreover, all measures are pairwise independent [7]-[8] (measured using Spearman’s rank correlation). In particular, and do not occur in pairs and are not being used as a means to expose simply the private fields of a class [8] In general, the odds are only 1:3 that if a class defines a getter, then this class will also provide a matching setter method.
III.  Defect Prediction
In this section, we step back from the specific issue of activity to discuss the more general field of defect prediction.
A typical, object-oriented, software project can contain hundreds to thousands of classes. In order to guarantee general and project-related fitness attributes for those classes, it is commonplace to apply some quality assurance (QA) techniques to assess the classes’s inherent quality. These techniques include inspections, unit tests, static source code analyzers, etc. A record of the results of this QA is a defect log. We can use these logs to learn defect predictors if the information contained in the data provides not only a precise account of the encountered faults (i.e., the “bugs"), but also a thorough description of static code features such as Lines of Code (LOC), complexity measures (e.g., McCabe’s cyclomatic complexity), and suitable object-oriented design metrics [6]-[8], [10].
For this, data miners can learn a predictor for the number of defective classes from past projects so that it can be applied for QA assessment in future projects. Such a predictor allows focusing the QA budgets. This is an important task as, during development, developers have to skew their quality assurance activities towards artifacts they believe require most effort due to limited project resources. For example, it is a common QA practice at NASA to focus on on-board guidance systems rather than ground-based databases that store data collected from the satellite [11].

Metrics	Definition	
NoM	Counts all member functions defined by class C.	
Getters	Counts all non-overloaded member functions in class 
C with arity zero, whose name starts with “get."	
Setters	Counts all non-overloaded member functions in class 
C with arity one, whose name starts with “set."	
InDegree	Let n be the type node for class C. Then 
is the in-degree of class C.	
OutDegree	Let n be the type node for class C. Then 
is the out-degree of class C.	
Clustering Coefficient	Let n be the type node for class C. Then 

is the clustering coefficient of class C, where  is the 
neighborhood of n with 
. 
	
Figure 2: Activity metrics definitions.

Skewing can introduce an inappropriate QA bias. Consider n project classes . If the QA team only verifies  with , using some ordering and inspection threshold of the analyzed classes, then constitute a blind spot that can compromise the system’s required quality attributes. Leveson remarks that, in modern complex systems, unsafe operations often result from an unstudied interaction between components [12]. For example, Lutz and Mikulski [13] found a blind spot in NASA deep-space missions, where most of the mission critical in-flight anomalies resulted from errors in the ground software that failed to correctly collect in-flight data. To avoid blind spots, one option is to rigorously assess all aspects of all software modules. But this is impractical. Exponential costs quickly exhausts finite QA resources. We must, therefore, devise a technique to manage them. 
Static code defect predictors are a lightweight sampling policy that guides the exploration of a system and raises an alert on sections that appear problematic. This approach is incomplete by definition. But, it is the only option when resource limits prevent us from performing a complete assessment of the code base.
One reason to favor static code measures is that they can be automatically extracted from the code base, with very little effort even for very large systems [14]. The industrial experience is that defect prediction scales well to a commercial context. Defect predicting technology has been commercialized in Predictive [15] a scalable product suite to analyze and predict defects in software projects. One company used it to manage the safety critical software for a fighter aircraft (the software controlled a lithium ion battery, which can over-charge and possibly explode). After applying a more expensive tool for structural code coverage, the company ran Predictive on the same code base. Predictive produced results consistent with the more expensive tool. But, Predictive was able to faster process a larger code base than the more expensive tool [15].
In addition, defect predictors developed at NASA [11] have also been used in software development companies outside the US (in Turkey). When the inspection teams focused on the modules that trigger the defect predictors, they found up to 70% of the defects using just 40% of their QA effort (measured in staff hours) [16].
Finally, a subsequent study on the Turkish software compared how much code needs to be inspected using random selection vs. selection via defect predictors. Using random testing, 87% of the files would have to be inspected in order to detect 87% of the defects. However, if the inspection process was restricted to the 25% of the files that trigger the defect predictors, then 88% of the defects could be found. That is, the same level of defect detection (after inspection) can be achieved using  less effort [17].

Figure 3: Percentage of defects found after sorting the code using different inspection ordering policies. Note that, in this case, developers were continually modifying a small number of very active classes handling complex interfacing tasks. Hence, for the blue curve, reading just this 1% of the code found nearly a quarter of the defects.
IV.  Inspection Optimization
How can our selected metrics help us in constructing an effective approach for fault localization in object-oriented software systems?  To answer this question, we first need to develop a suitable success criterion for defect prediction. For this paper, we adopt a proposal by Arisholm & Briand [18] as follows: 
 If X% of the classes are predicted to be defective, then the actual faults identified in those classes must account for more than X% of all defects in the system being analyzed. Otherwise, the costs of generating the defect predictor is not worth the effort. 
In essence, this is inspection optimization – find some ordering to project artifacts such that humans have to read the least code in order to discover the most faults, which we model as follows: 
•	After a data miner predicts a class is defective, then a secondary human team examines the code. 
•	This team correctly recognizes of the truly defective classes (and means that the inspection team is perfect at its task and finds every defect present). 
•	A good learner is one that finds the most defective classes (measured in terms of probability of detection, pd) in the smallest classes (measured in terms of lines of code, LOC).  
Inspection optimization can be visualized using Figure 3 that illustrates three plausible inspection ordering policies: 
•	The blue optimal policy combines knowledge of class size and the location of the actual defects. 
•	The green activity policy guesses defect locations using a defect predictor learned from the activity measures. 
•	The red baseline policy ignores defect counts and just sorts the classes in ascending order of size.  
Each of these ordering policies sorts the code base along the x-axis. The code is then inspected, left to right, across that order, so that, by the end of the x-axis, we have read 100% of the code. Along the way, we encounter classes containing y% of defects (a.k.a. recall). A better policy finds more defects sooner, that is, it yields a larger area under the curve of %LOC-vs-recall. In Figure 3, we note that that the green activity policy does better than the red baseline (and comes close, within 95%, of the blue optimal).
These three policies are defined by one equation modeling distance to some utopia point of most defects and smallest LOC: 
  (1)		(1)
Here,  and  are the number of defects and lines of code in class C (normalized zero to one). Also, α is a constant controlling the sorting. At α=0, we ignore defects and sort only on LOC. This implements the baseline policy. This baseline policy is the Koru ordering advocated by researchers who argue that smaller classes have a relatively higher density of errors [19]-[21]. Note that if the activity policy cannot out-perform baseline, then our notion of activity is superfluous.
The other policies use α=1. For the activity policy, we: 
•	Train a learner using the measures of metrics without LOC, 
•	Set  via the learned model, 
•	Sort using score, , LOC, and α=1, 
•	Calculate auc and determine the area under the %LOC-vs-recall curve.  

The optimal policy does the same, but sets  using the historical defect logs. Note that optimal is different to activity since the former knows exactly where the defects are, whereas the latter must guess the defect locations using the learned model.
In practice, the optimal policy is impossible to apply since it implies that we were to know the number of defects before the classes would be inspected. However, it is the theoretical upper-bound on performance for inspection optimization. Hence, we report activity and baseline performances as a ratio of the area under the curve of optimal.
This ratio calculation has another advantage. Note that the Δ effectiveness of the secondary human inspection team is same, regardless of the oracle that sorts the code. Hence, in the ratio calculation, Δ cancels out and we can ignore it from our analysis.
V.  Activity and Inspection Optimization
To assess the value of the selected metrics in Figure 1, we distilled them for 33 open-source Java projects from the Helix project and applied them to build defect predictors. As shown below, the median value for the  ratio will be 96%, that is, very close to the theoretical upper bound possible for any defect predictor for the the task of inspection optimization.

System	  Description	
ant	  Build management system	
ivy	  Dependency manager	
jedit	  Text editor	
lucene	  Text search engine	
poi	  API for Office Open XML standards	
synapse	  Enterprise service bus	
velocity	  Template language engine	
xalan	  XSLT processor	
xerces	  XML processor	

 

Figure 4: Java systems used in this study.
A.  Data
The data used in this studies was built as a join between two complementary data sets:
•	The PROMISE repository [22] contains defect information for various open-source object-oriented systems. The defect data for this study was collected by Jureczko [23].
•	The Helix repository [5], [6] provides static source code metrics for a compilation of release histories of non-trivial Java open-source software systems.  
The joined data sets represent releases of the projects listed in Figure 4. All projects are “long term" (at least 15 releases span over a development period of 36 months or more) and comprise more than 100 classes each. In addition, every project can be characterized as either application, framework, or library, a broad “binning" strategy that reflects the inherent, yet recurring, differences in software design and composition. For a detailed description of these data sets, see [6].
For LOC (i.e., the Lines of Codes) we use an estimator based on the size of the compiled byte code rather than the actual source code. The byte code provides us with a noise-free image of the class’s defined functionality. LOC of a class C is given as the sum of the following components extracted from the binaries:
•	out-degree of C line(s) for import statements 
•	1 line for the class declaration 
•	1 line for super class declaration if not java.lang.Object 
•	1 line for each interface implemented by C 
•	1 line for each field defined in class C 
•	1 line for each method m defined in class C, plus
•	# parameters of m 
•	# throws defined by m 
•	MaxLocals attribute (i.e., local variables) of m 
•	# byte code instructions in m

We selected these components as they provide a very consistent approximation of the size of source code independent of the actual coding style used. The LOC estimator correlates very well with the lines of source code (cf. Figure 5). Furthermore, for the purpose of inspection optimization, an added benefit of processing byte code rather than source is that the data miner will only report those classes that actually appear in the released version. That is, the secondary human inspection team is given further guidance to focus its QA effort. Previous research [24]-[27] found that, in general, not all parts of the code base are included in the final release build. This is due to the release build configuration settings used. Hence, processing 10% of the classes as per byte code, is equivalent to analyzing 10% of the active source code classes (i.e., the classes that must be inspected in the QA process)

Figure 5: Lines of code () extracted from byte code is a very strong approximation of the extracted directly from source code
The joined, activity-based, data sets are constructed as follows:
1)	From the PROMISE repository we fetch the bug information for release N per class.
2)	We extract from the Helix repository the static code metrics, including LOC, for release N per class.
3)	Using the fully-qualified class name as key, both information is merged into the activity data set for release N per class.
Figure 6 shows the distribution of defects seen in our classes. Usually, most classes have no defects, but in 10% of cases, each class has more than 1 to 5 recorded defects. 
B.  Experimental Setup
This study used a N-way cross-evaluation study [28]. The data set is divided into N=10 buckets. For each bucket in the N-way, a predictor is learned on the nine of the buckets, then tested on the remaining bucket. These N studies implement N hold out studies where a model is tested on data not used in training.

		percentiles	
data	# classes	10%	30%	50%	70%	90%	
ant-1.3	125	0	0	0	0	1	
ant-1.4	178	0	0	0	0	1	
ant-1.5	293	0	0	0	0	1	
ant-1.6	351	0	0	0	0	2	
ant-1.7	493	0	0	0	0	2	
ivy-1.4	241	0	0	0	0	0	
ivy-2	352	0	0	0	0	1	
jedit-3.2	272	0	0	0	1	4	
jedit-4	306	0	0	0	0	2	
jedit-4.1	312	0	0	0	0	2	
jedit-4.2	367	0	0	0	0	1	
jedit-4.3	492	0	0	0	0	0	
lucene-2	195	0	0	0	1	4	
poi-2	314	0	0	0	0	1	
synapse-1	157	0	0	0	0	0	
synapse-1.1	222	0	0	0	0	1	
synapse-1.2	256	0	0	0	1	2	
velocity-1.6	229	0	0	0	1	2	
xalan-2.4	428	0	0	0	0	1	
xalan-2.5	763	0	0	0	1	2	
xalan-2.6	875	0	0	0	1	2	
xerces-1	162	0	0	0	2	2	
xerces-1.2	438	0	0	0	0	1	
xerces-1.3	452	0	0	0	0	1	
lucene-2.2	247	0	0	1	2	4	
lucene-2.4	428	0	0	1	2	5	
poi-1.5	237	0	0	1	1	4	
poi-2.5	384	0	0	1	2	2	
poi-3	442	0	0	1	1	2	
velocity-1.4	196	0	1	1	1	2	
velocity-1.5	214	0	0	1	2	4	
xalan-2.7	908	1	1	1	1	2	
xerces-1.4	329	0	0	1	2	4	

Figure 6: Percentile distributions, defects per class. For example, in xalan-2.7 the median (50th percentile) defects per class is 1. In lucene-2.4, 10% of classes have 5 defects or more. In xalan-2.7, most classes have at least 1 defect. The table is sorted by the median defects (see the 50% percentile column).

To appreciate cross-validation, consider another approach called self-test where the learned model is assessed on the same data that was used to create it. Self-tests are deprecated by the research community [28]. If the goal is to understand how well a defect predictor will work on future projects, it is best to assess the predictor via hold-out modules not used in the generation of that predictor. 
In the WEKA 3.7.3 implementation of the cross-val procedure used in this study, results are reported once for each test-instance as that instance appears in one of the N hold-outs. So a data set containing C examples will generated C predictions, regardless of the value of N used for the number of hold-outs.

C.  Selection of Learners
As mentioned above, there are very many methods for converting static code measures into defect predictors [11], [29]-[39] We adopted Holte’s simplicity-first heuristic [40] and applied a simple linear regression (LSR) algorithm available in WEKA [28], with no pre-processing.

Note that WEKA’s LSR tool uses a simple greedy back-select, which is applied after the linear model has been generated. In that back-select, WEKA steps through all the attributes removing the one with the smallest standardized coefficient until no improvement is observed in the estimate of the model error given by the Akaike information criterion. As a consequence, some attributes may be absent from the final learned model. 
Initially, we planned to test various learners, feature extractors, instance selectors, and discretization methods (as we have done in the past [11], [38], [39]). But our results were so encouraging that there was little room for further improvement over simple LSR.

	percentiles	
data	10%	30%	50%	70%	90%	
lucene-2.4	-2.0	-0.9	-0.3	0.4	2.6	
velocity-1.6	-1.4	-0.6	-0.3	0.0	1.4	
xerces-1.0	-1.1	-0.6	-0.3	0.7	1.0	
ant-1.4	-0.4	-0.3	-0.2	-0.1	0.8	
lucene-2.0	-2.0	-0.8	-0.2	0.1	1.7	
poi-1.5	-1.6	-0.9	-0.2	0.0	2.3	
synapse-1.2	-0.8	-0.4	-0.2	0.0	1.0	
xalan-2.5	-0.9	-0.5	-0.2	0.5	0.8	
xalan-2.6	-0.9	-0.5	-0.2	0.4	1.2	
xalan-2.7	-0.5	-0.3	-0.2	0.1	0.7	
xerces-1.2	-0.5	-0.2	-0.2	0.0	0.9	
xerces-1.4	-2.2	-0.4	-0.2	0.4	0.8	
ant-1.3	-0.5	-0.3	-0.1	0.0	0.6	
ant-1.7	-0.8	-0.3	-0.1	0.0	0.8	
ivy-2.0	-0.3	-0.1	-0.1	0.0	0.3	
lucene-2.2	-2.5	-0.8	-0.1	0.4	2.0	
poi-2.0	-0.2	-0.1	-0.1	-0.1	0.7	
poi-3.0	-1.1	-0.4	-0.1	0.0	1.0	
synapse-1.0	-0.3	-0.2	-0.1	0.0	0.0	
synapse-1.1	-0.7	-0.4	-0.1	0.0	0.9	
velocity-1.5	-1.5	-0.7	-0.1	0.3	1.5	
xalan-2.4	-0.5	-0.2	-0.1	0.0	0.7	
ant-1.5	-0.3	-0.1	0.0	0.0	0.3	
ant-1.6	-0.8	-0.3	0.0	0.0	0.9	
ivy-1.4	-0.2	-0.1	0.0	0.0	0.0	
jedit-3.2	-2.3	-0.8	0.0	0.0	1.8	
jedit-4.0	-1.3	-0.5	0.0	0.0	1.0	
jedit-4.1	-1.1	-0.4	0.0	0.0	1.0	
jedit-4.2	-0.6	-0.2	0.0	0.0	0.3	
jedit-4.3	-0.1	0.0	0.0	0.0	0.0	
poi-2.5	-1.3	-0.6	0.0	0.7	0.9	
velocity-1.4	-1.2	-0.2	0.0	0.1	1.0	
xerces-1.3	-1.4	-0.2	0.0	0.0	0.7	

Figure 7: Percentile distributions of actual−predicted number of defects per class. For example, the median (50th percentile) value of actual−predicted is -0.3 to 0.

Figure 8: Performance results expressed as a ratio of the optimal policy. Data sets are sorted according the the activity results. Median values for baseline, activity are 91%, 96% of optimal, respectively. 
VI.  Results
A.  Sanity Check
Figure 7 shows the distribution of actual−predicted defects for our classes where actual comes from historical logs and predicted comes from the C predictions seen in our 10-way. This result is our sanity check: if the actual−predicted values were large, then we would doubt the value of activity-based defect prediction. Note that, in the median case (shown in the middle 50% column), the predictions are very close to actuals (-0 to -0.3). Since our estimates are close to actuals, we may continue.
B.  Baseline and Activity vs. Optimal
Figure 8 shows the ratio of the optimal policy achieved with the activity policy (the green curve) and the baseline policy (the red curve). These curves are statistically significantly different (Wilcoxon, 95% confidence). For both curves, the result are expressed in as a ratio of the optimal policy that uses historical knowledge to determine the number of defects in each class.
We observe that the results of the baseline policy are far more erratic than for the activity policy. The spread of a distribution is the difference between the 75% and 25%th percentile range. The spread of the values in pm are: 
•	Activity: 98 - 91 = 7, 
•	Baseline: 95 - 82 = 13. 
That is, the results of the activity policy are more predictable (fall into a narrower range). On the other hand, the results from baseline can spread nearly twice as far.
Not only are the activity results more predictable, they out-perform the baseline policy. The median value of the red baseline policy results (i.e., inspecting the code based on increasing class size) is 91% of optimal. Note that baseline is rarely any little better than activity, and often it is much worse: 

•	When baseline out-performs activity (in only  of our comparisons) it does so only by a small margin. 
•	In the  data sets where baseline does worse than activity, sometimes it does much worse (see the velocity-1.5 and velocity-1.6 results which fall to 70% of optimal). 

 On the other hand, the median value of the activity policies results are 96%, which is within 4% of optimal. Further, the top ten results of activity all score 100% of optimal (see the right-hand side of the green curve in Figure 8). That is, for the purposes of optimizing inspection, there is little to no room for improvement on top of the activity-based measures.
Hence, we strongly recommend the activity policy.
C.  Summary
Our key observations in this study are as follows:
•	According to Figure 7, activity-based measures combined with linear regression lead to defect predictors with low error rates in open-source object-oriented systems. 
•	According to Figure 8, for the task of inspection optimization, activity-based defect prediction works significantly better than the baseline and very close to the optimum. 
VII.  Discussion
The results here are quite unequivocal – activity is a strong predictor for software defects, and this effect can be detected with a simple model such as linear regression. Hence, we need to explain why this effect has not been reported before. We conjecture that the use of a small set of activity-centric static metrics is too simple and too novel a concept to be reported previously. 
A.  Too Simple? 
We can broadly classify object-oriented software quality research as (a) studies with more focus on prediction models than the metrics, and (b) studies with more focus on metrics validation than the models (as in this study). It is no surprise that the former kind of studies did not explicitly investigate the concept of activity, as they usually operate within existing sets of common metrics in order to choose the best model among many. The literature offers many complex methods for data mining such as support-vector machines, random forests, and tabu search to tune the parameters of a genetic algorithm (i.e., [30]-[33]). In this era of increasing learner complexity, something as easy as linear regression on a small set of static code measures aiming especially on activity may have been discounted before being explored rigorously. Therefore, our first explanation is that the use of activity as a concept is so simple that it escaped the attention of this type of research. 
Nevertheless, we cannot ignore the latter type of studies, in which the focus has usually been validating object-oriented metrics as predictors of defects through correlational methods (i.e., [41]-[45]). Briand & Wüst provide an extensive survey of empirical studies of quality in object-oriented systems, and observe that the majority of studies falls in this category [46]. However, they also state that only half of the studies employ multivariate prediction models, and the other half just reports univariate relations between object-oriented metrics and defects. Further, only half of the studies with a prediction model, conduct a proper performance analysis through cross-validation. After this filtering, remaining work contains hard-to-compare empirical studies, where the size and the number of data sets are so small that the combined results are conflicting and do not reveal a common trend possibly due to varying contexts of the studies. 
Another aspect of related studies is that they consider certain subgroups of object-oriented metrics relating to concepts such as coupling, cohesion, inheritance and polymorphism, and size [46]. Briand & Wüst report that the significance of the relation between different subgroups of metrics and defects are mostly inconclusive, and only a number of size and coupling measures are consistent. We have further run a smaller-scale review of major studies conducted with the guidelines of the original survey [18], [36], [41]-[43], [45], [47]-[50]. Similar to Briand & Wüst, we observed that the table of metrics vs. different systems used to assess those metrics were sparsely populated. 
B.  Too Novel? 
The starting point for this research was the observation in the Helix data sets that most classes stabilize very early in their life cycle while a very small number of active classes garner the most attention by developers [6], [7]. As discussed above in Section II, this is not the standard picture of the life cycle of a class. To us, this observation was so unique that it prompted the question “does the amount a class is used by developers predict for system defects” (i.e., this study). However, without that initial surprising observation, we would not have conducted the study reported in this paper. 
Compared to other studies (e.g., studies surveyed in [46]), the size and number of data sets used in our study is extensive and reveals a clear benefit of using activity-centric metrics in the context of open-source object-oriented systems. In contrast to our concept of activity, Turhan et al. [51] investigate popularity. Their approach is to augment standard static code metrics within a call graph-based ranking framework, which is inspired by the PageRank algorithm [52]. Rather than constructing learners with a standard set of metrics that value each module equally, Turhan et al. first rank the modules using the dependency graph information and weigh the information learned from “popular” modules more. Their approach reduced the false alarm rates significantly. However, this technique is an indirect way of utilizing activity, and does not include explicit activity metrics that are used in this study. Similarly, Zimmermann  et al. include eigenvector centrality, a measure of closeness centrality of network nodes similar to PageRank, in their analysis of complexity vs. network metrics for predicting defects from software dependencies [53]. Though, eigenvector centrality is found to be correlated with defects for the Windows system they have evaluated, this metric did not stand out among other network or complexity based metrics to allow a discussion on “activity” (see the next section below for a possible cause). Finally, Kpodjodo et al. monitored their proposed, again PageRank inspired, Class Rank metric among several versions of a single system and found moderate evidence in favor [54]. In this paper, we handle activity as a concept rather than rely on a single measure, and we achieve near optimal results compared to moderate improvements of similar work. 
C.  Hidden? 
Thirdly, it is possible that activity was buried under other effects. When we look at the attributes we have used in previous studies (e.g., [11]), we can see some overlap between those attributes and those of metrics. Miller [55], Witten & Frank [28], and Wagner [56] offer a theoretical analysis discussing how an excess of attributes containing multiple strong predictors for the target class can confuse learning. For example, both Wagner and Miller note that in a model comprising N variables, any noise in variable  adds to the noise in the output variables. 
We have also observed supporting evidence for this explanation in our small scale quality-in-object-oriented-systems review. In all cases, where both a univariate and a multivariate analysis is utilized, it is common for metrics verified by the univariate model not to be included in the multivariate model for the same data [41], [42], [45], [47], [49], [50]. El Emam et al. use this phenomenon to control for the confounding effects of size on metrics believed to serve as suitable predictors for defects [44]. Similarly, the multivariate model metrics may include those that are not verified by the univariate model [18], [36], [47], for which Guyon provide simple examples showing that the prediction power can be significantly increased when features are used together rather than individually [57]. Hence, even though some measures exist in a data set, noise from the other variables may have drowned out their effect. 
VIII.  Validity & Future Work
    Internal Validity: Apart from joining the PROMISE data sets (for defect counts) with the Helix data sets (for the activity measures), we did not pre-process the datasets in any way. This was done to enable replication of our results. 
Construct Validity: We have made the case above that the measures of metrics reflect the “activity” of different classes, that is, how often a developer will modify or extend the services of a class as an expression of the attractiveness of this class for the developer’s design choices. This case has not been tested here. Hence:
Future work 1: Analyze participant observation of developers to determine what classes they inspect as part of their workflow. 
External Validity: Our use of cross-validation means that all the results reported above come from the application of our models to data not seen during training. This gives us some confidence that these results will hold for future data sets.
As to our selection of data sets, the material used in this study represents real-world use, collected from real-world projects. Measured in terms of number of data sets, this paper is one of the largest defect prediction studies that we are aware of. Nevertheless, there is a clear bias in our sample: open-source Java systems. Hence:

Future work 2: Test the validity of our conclusions to close-sourced, non-object-oriented, and non-Java projects. 
Conclusion Validity: We take great care to only state our conclusions in terms of areas under a %LOC-vs-recall curve. For the purpose of finding the most defects after inspecting the fewest lines of code (i.e., the inspection optimization criterion proposed by Arisholm & Briand [18]), the “activity” measures of metrics exhibit an excellent performance (median results, within 96% of the optimum).
While area under a %LOC-vs-recall is an interesting measure, it is not the only seen in the literature. Hence:
Future work 3: Explore the value of activity for other evaluation criteria. 
Those other criteria include:
•	Counting the number of files inspected, rather than the total LOC, as done, for example, by Weyuker and Ostrand [58], [59],
•	 Precision, as advocated, for example, by Zhang & Zhang [60] (but depreciated by Menzies et al. [61]), 
•	Area under the curve of the pd-vs-pf curves, as used, for example, by Lessmann  et al. [30]. 
IX.  Conclusion
We have shown above that a repository containing just static code measures can be still be used to infer interaction patterns amongst developers. Specifically, we studied the “active” classes, that is, the classes where the developers spend much time interacting with each other’s design and implementation decisions. In 33 open-source Java systems, we found that defect predictors based on static code measures that model “activity” perform within 96% of a theoretical upper bound. This upper bound was derived assuming that the goal of the detectors was “inspection optimization,” that is, read the fewest lines of code to find the most defects.
There is another aspect of activity measures that recommends their use. In this paper, we show that simple linear regression over these measures works very well indeed. That is, the machinery required to convert these measures into defect predictors is far less complex than alternative approaches, such as:
•	Lessmann’s random forests and support-vector machines [30],
•	 The many methods explored by Khoshgoftaar [31]-[33],
•	 Defect prediction via multiple explanatory variables [36], [37],
•	 Our own defect predictors via feature selection [11], instance selection [38], or novel learners built for particular tasks [39].  

The comparative simplicity of activity-based prediction, suggests that previous work [29]-[37], including our own research [11], [38], [39] may have needlessly complicated a very simple concept, that is, defects are introduced and discovered due to all the activity around a small number of most active classes.
Acknowledgment
   This work was conducted at Swinburne University of Technology, West Virginia University, and University of Oulu with partial funding from (1) the New Zealand Foundation for Research, Science and Technology, (2) the United States National Science Foundation, CISE grant 71608561, (3) a research subcontract with the Qatar University NPRP 09-1205-2-470, and (4) TEKES under the Cloud-SW project in Finland.

References
[1]	C. Bird, N. Nagappan, H. Gall, B. Murphy, and P. Devanbu, “Putting It All Together: Using Socio-technical Networks to Predict Failures,” in Proceedings of 20th International Symposium on Software Reliability Engineering, Nov. 2009,pp. 109–119.
[2]	C. Bird, N. Nagappan, P. Devanbu, H. Gall, and B. Murphy, “Does distributed development affect software quality? An empirical case study of Windows Vista,” in Proceedings of 31st International Conference on Software Engineering, May2009, pp. 518 –528.
[3]	N. Nagappan, B. Murphy, and V. Basili, “The Influence of Organizational Structure on Software Quality: An Empirical Case Study,” in Proceedings of the 30th International Conference on Software Engineering, May 2008, pp. 521–530.
[4]	P. J. Guo, T. Zimmermann, N. Nagappan, and M. Brendan, “Characterizing and predicting which bugs get fixed: An empirical study of Microsoft Windows,” in Proceedings of the 32nd International Conference on Software Engineering, May 2010, pp. 495–504.
[5]	R. Vasa, M. Lumpe, and A. Jones, “Helix - Software Evolution Data Set,” http://www.ict.swin.edu.au/research/projects/helix, Dec. 2010.
[6]	R. Vasa, “Growth and Change Dynamics in Open Source Software Systems,” Ph.D. dissertation, Swinburne University of Technology, Faculty of Information and Communication Technologies, Oct. 2010.
[7]	R. Vasa, M. Lumpe, P. Branch, and O. Nierstrasz, “Comparative Analysis of Evolving Software Systems Using the Gini Coefficient,” in Proceedings of 25th IEEE International Conference on Software Maintenance. Edmonton, Alberta: IEEE Computer Society, Sep. 2009, pp. 179–188.
[8]	M. Lumpe, S. Mahmud, and R. Vasa, “On the Use of Properties in Java Applications,” in Proceedings of the 21stAustralian Software Engineering Conference, Auckland, New Zealand, Apr. 2010, pp. 235–244.
[9]	N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in ICSE’05, 2005,pp. 284–292.
[10]	S. R. Chidamber and C. F. Kemerer, “A Metrics Suite for Object Oriented Design,” IEEE Transactions on Software Engineering, vol. 20, no. 6, pp. 476–493, Jun. 1994.
[11]	 T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” IEEE Transactions on Software Engineering, vol. 33, no. 1, pp. 2–13, Jan. 2007.
[12]	N. Leveson, Safeware System Safety And Computers. Addison-Wesley, 1995.
[13]	R. Lutz and C. Mikulski, “Operational anomalies as a cause of safety-critical requirements evolution,” Journal of Systems and Software, vol. 65, no. 2, pp. 155–161, Feb. 2003.
[14]	N. Nagappan and T. Ball, “Static analysis tools as early indicators of pre-release defect density,” in Proceedings of the 27th International Conference on Software Engineering, May 2005, pp. 580–586.
[15]	J. Turner, “A predictive approach to eliminating errors in software code,” 2006, available from http://www.sti.nasa.gov/ tto/Spinoff2006/ct 1.html.
[16]	A. Tosun, B. Turhan, and A. Bener, “Practical considerations of deploying ai in defect prediction: A case study within the turkish telecommunication industry,” in Proceedings of the 5th International Conference on Predictor Models in Software Engineering.	ACM, May 2009, pp. 1–9.
[17]	A. Tosun, A. Bener, and R. Kale, “Ai-based software defect predictors: Applications and benefits in a case study,” in Twenty-Second IAAI Conference on Artificial Intelligence, Jul. 2010, pp. 1748–1755.
[18]	E. Arisholm and L. Briand, “Predicting fault-prone components in a java legacy system,” in Proceedings of the 5th ACM/IEEE International Symposium on Empirical Software Engineering, Sep. 2006, pp. 8–17.
[19]	A. Koru, D. Zhang, K. El Emam, and H. Liu, “An investigation into the functional form of the size-defect relationship for software modules,” Software Engineering, IEEE Transactions on, vol. 35, no. 2, pp. 293–304, Mar. 2009.
[20]	A. Koru, K. E. Emam, D. Zhang, H. Liu, and D. Mathew, “Theory of relative defect proneness: Replicated studies on the functional form of the size-defect relationship,” Empirical Software Engineering, pp. 473–498, Oct. 2008.
[21]	A. Koru, D. Zhang, and H. Liu, “Modeling the effect of size on defect proneness for open-source software,” in International Workshop on Predictor Models in Software Engineering (PROMISE’07), May 2007, article No. 10.
[22]	G. Boetticher, T. Menzies, and T. Ostrand, “The PROMISE Repository of Empirical Software Engineering Data,” 2007, http://promisedata.org/.
[23]	M. Jureczko and D. Spinellis, “Using Object-Oriented Design Metrics to Predict Software Defects,” Models and Methods of System Dependability. Oficyna Wydawnicza Politechniki Wrocławskiej, pp. 69–81, 2010.
[24]	J. Krinke, “Identifying Similar Code with Program Dependence Graphs,” in Proceedings of the Eighth Working Conference on Reverse Engineering (WCRE’01).	IEEE Computer Society, Oct. 2001, pp. 301–309.
[25]	G. Antoniol, G. Canfora, G. Casazza, and A. De Lucia, “Information Retrieval Models for Recovering Traceability Links Between Code and Documentation,” in Proceedings of the International Conference on Software Maintenance (ICSM’00), 2000, pp. 40–49.
[26]	K. Kontogiannis, R. DeMori, E. Merlo, M. Galler, and M. Bernstein, “Pattern Matching for Clone and Concept Detection,” Journal of Automated Software Engineering, vol. 3, pp. 77–108, 1996.
[27]	J. H. Johnson, “Substring Matching for Clone Detection and Change Tracking,” in Proceedings of the International Conference on Software Maintenance (ICSM 94), 1994, pp. 120–126.
[28]	I. H. Witten and E. Frank, Data mining. 2nd edition.	Los Altos, US: Morgan Kaufmann, 2005.
[29]	T. McCabe, “A Complexity Measure,” IEEE Transactions on Software Engineering, vol. 2, no. 4, pp. 308–320, Dec. 1976.
[30]	S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Bench-marking classification models for software defect prediction: A proposed framework and novel findings,” IEEE Transactions on Software Engineering, vol. 34, no. 4, pp. 485–496, Aug. 2008.
[31]	T. M. Khoshgoftaar, N. Seliya, and K. Gao, “Assessment of a
New Three-Group Software Quality Classification Technique: An Empirical Case Study,” Empirical Software Engineering, vol. 10, pp. 183–218, Apr. 2005.
[32]	T. M. Khoshgoftaar, S. Zhong, and V. Joshi, “Enhancing software quality estimation using ensemble-classifier based noise filtering,” Intell. Data Anal., vol. 9, pp. 3–27, Jan. 2005.
[33]	T. M. Khoshgoftaar, X. Yuan, and E. B. Allen, “Balancing Misclassification Rates in Classification-Tree Models of Soft-ware Quality,” Empirical Software Engineering, vol. 5, pp.313–330, Dec. 2000.
[34]	R. Harrison, S. Counsell, and R. Nithi, “An investigation into the applicability and validity of object-oriented design metrics,” Empirical Software Engineering, vol. 3, no. 3, pp.255–273, Sep. 1998.
[35]	L. C. Briand, S. Morasca, and V. R. Basili, “Defining and Validating Measures for Object-Based High-Level Design, ”IEEE Transactions on Software Engineering, vol. 25, no. 5,pp. 722–743, Sep. 1999.
[36]	L. Briand, J. Wu ̈st, J. Daly, and D. Victor Porter, “Exploring the relationships between design measures and software quality in object-oriented systems,” Journal of Systems and Software, vol. 51, no. 3, pp. 245–273, May 2000.
[37]	N. Nagappan, T. Ball, and A. Zeller, “Mining metrics to predict component failures,” in Proceedings of the 28th Inter-national Conference on Software Engineering.	ACM, May2006, pp. 452–461.
[38]	B. Turhan, T. Menzies, A. Bener, and J. Di Stefano, “On the relative value of cross-company and within-company data for defect prediction,” Empirical Software Engineering, vol. 14,no. 5, pp. 540–578, Oct. 2009.
[39]	T. Menzies, O. Jalali, J. Hihn, D. Baker, and K. Lum, “Stable rankings for different effort models,” Automated Software Engineering, vol. 17, no. 4, pp. 409–437, Dec. 2010.
[40]	R. Holte, “Very simple classification rules perform well on most commonly used datasets,” Machine Learning, vol. 11,pp. 63–90, Apr. 1993.
[41]	V. R. Basili, L. C. Briand, and W. L. Melo, “A validation of object-oriented design metrics as quality indicators,” IEEE Transactions on Software Engineering, vol. 22, no. 10, pp.751–761, Oct. 1996.
[42]	L. C. Briand, J. Wu ̈st, S. V. Ikonomovski, and H. Lounis, “Investigating quality factors in object-oriented designs: an industrial case study,” in Proceedings of 21st International Conference of Software Engineering.	Los Alamitos, CA, USA: IEEE Computer Society Press, May 1999, pp. 345–354.
[43]	M. Cartwright and M. Shepperd, “An empirical investigation fan object-oriented software system,” IEEE Transactions on Software Engineering, vol. 26, pp. 786–796, Aug. 2000.
[44]	K. El Emam, S. Benlarbi, N. Goel, and S. N. Rai, “The confounding effect of class size on the validity of object-oriented metrics,” IEEE Transactions on Software Engineering, vol. 27, no. 7, pp. 630–650, Jul. 2001.
[45]	 K. K. Aggarwal, Y. Singh, A. Kaur, and R. Malhotra, “Empirical analysis for investigating the effect of object-oriented metrics on fault proneness: a replicated case study,” Software Process: Improvement and Practice, vol. 14, no. 1, pp. 39–62, Jan. 2009.
[46]	L. C. Briand and J. Wu ̈st, “Empirical studies of quality models in object-oriented systems,” Advances in Computers, vol. 56, pp. 98–167, 2002.
[47]	K. E. Emam, W. Melo, and J. C. Machado, “The prediction of faulty classes using object-oriented design metrics,” System and Software, vol. 56, pp. 63–75, Feb. 2001.
[48]	L. C. Briand, W. L. Melo, and J. Wu ̈st, “Assessing the applicability of fault-proneness models across object-oriented software projects,” IEEE Transactions on Software Engineering, vol. 28, no. 7, pp. 706–720, Jul. 2002.
[49]	 K. K. Aggarwal, Y. Singh, A. Kaur, and R. Malhotra, “Investigating the effect of coupling metrics on fault proneness in object-oriented systems,” Software Quality Professional, vol. 8, no. 4, pp. 4–16, Sep. 2006.
[50]	——, “Investigating effect of Design Metrics on Fault Proneness in Object-Oriented Systems,” Journal of Object Technology, vol. 6, no. 10, pp. 127–141, Nov. 2007.
[51]	B. Turhan, G. Kocak, and A. Bener, “Software Defect Prediction Using Call Graph Based Ranking (CGBR) Framework, ”in Proceedings of 34th Euromicro Conference on Software Engineering and Advanced Applications, Sep. 2008, pp. 191–198.
[52]	S. Brin and L. Page, “The anatomy of a large-scale hypertextual Web search engine,” in Proceedings of the 7thInternational Conference on World Wide Web, Apr. 1998, pp.107–117.
[53]	T. Zimmermann and N. Nagappan, “Predicting defects using network analysis on dependency graphs,” in Proceedings of the 30th international conference on Software engineering, ser. ICSE ’08.	New York, NY, USA: ACM, 2008, pp. 531–540.
[54]	S. Kpodjedo, F. Ricca, G. Antoniol, and P. Galinier, “Evolution and search based metrics to improve defects prediction, ”Search Based Software Engineering, International Symposium on, vol. 0, pp. 23–32, 2009.
[55]	A. Miller, Subset Selection in Regression (second edition). Chapman & Hall, 2002.
[56]	S. Wagner, “Global Sensitivity Analysis of Predictor Models in Software Engineering,” in International Workshop on Predictor Models in Software Engineering (PROMISE’07), May2007, article No. 3.
[57]	I. Guyon, A. Elisseefi, and L. Kaelbling, “An introduction to variable and feature selection,” Journal of Machine Learning Research, vol. 3, no. 7-8, pp. 1157–1182, Mar. 2003.
[58]	T. J. Ostrand, E. J. Weyuker, and R. M. Bell, “Where the bugs are,” in Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis.	New York, NY, USA: ACM, Jul. 2004, pp. 86–96.
[59]	E. Weyuker, T. Ostrand, and R. Bell, “Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models,” Empirical Software Engineering, vol. 13, no. 5, pp. 539–559, Oct. 2008.
[60]	H. Zhang and X. Zhang, “Comments on ’Data Mining Static Code Attributes to Learn Defect Predictors’,” IEEE Transactions on Software Engineering, vol. 33, no. 9, pp. 635–637,Sep. 2007.
[61]	 T. Menzies, A. Dekhtyar, J. Distefano, and J. Greenwald, “Problems with Precision: A Response to ”Comments on ’Data Mining Static Code Attributes to Learn Defect Predictors’”,” IEEE Transactions on Software Engineering, vol. 33,no. 9, pp. 637–640, Sep. 2007.