\section{Related Work}
\subsection{Neighborhood Reasoning}
Our use of nearest neighbor is somewhat similar to {\em k-NN classifier} or
{\em reasoning by analogy}.
A {\em k-NN classifier} returns the majority class in the ``k'' nearest neighbors to the 
test instance. Lessman et.al. benchmark k-NN against 19 other learners and found
that it was one of the four worst methods for defect prediction~\cite{lessmann09}.

In {\em reasoning by analogy}, inference about some new
case is made by partial match to prior cases, then applying some repair or generalization
procedure to the matched cases.
Like k-NN, reasoning by analogy makes use of some distance metric that finds old
cases similiar (i.e. nearer) to the the new case. While reasoning by analogy is a common
effort estimation method (e.g.~\cite{shepperd97}), it is far less common
to see it applied to defect prediction.
One reason for this is the typical size of the training set: effort estimation
typically deals with dozens of prior examples while defect training sets can have
thousands of examples. We conjecture that as the data set grows in size,
simple induction over many instances may be more powerful than complex analogical
reasoning over a handful of examples.

Our work is most similar to that of 
Khoshgoftaar and Seliya~\cite{khoshgoftaar03} who augmented k-NN with a counting
procedure that was a slighly more
sophisticated decision procedure than ``return the majority class''. Our approach differs
to theirs in that we 
bundle together the k-nearest neighbors, then pass that bundle to
a full-fledged classifier (Naive Bayes).

\subsection{WC vs CC in Effort Estimation}
The value of WV vs CC data has been 
investigated in the effort estimation community, with inconclusive results:
\bi
\item
Mendes et al.~\cite{mendes07} found within-company data
performed much better than cross-company data for predicting
estimation effort of web-based projects.  They only recommend using
cross-company data in the special case when that ``data are obtained
using rigorous quality control procedures''.  
\item
A similar conclusion was reached by Abrahamsson et al.
who  discussed learning effort predictors in the context
of an agile development process~\cite{abrahamsson07}. They strongly advocate
the use of WC-data.
\ei
However,
other studies are not so clear in their conclusions:
\bi
\item
MacDonnel \& Shepperd tried
to find trends in a set of papers relating to project 
management and effort estimation~\cite{macdonell07}. However, the
 papers studied by MacDonnel \& Shepperd used a wide range of data
sets so these authors found it hard to offer  a definitive combined
conclusion.  
\item
In other work, after a review of numerous case studies, 
Kitchenham
et al.~\cite{kitch07} concluded that the value of CC vs WC data for effort estimation
is
unclear: \begin{quote} {\em 
{\ldots}some organizations would benefit from using models derived from
cross-company benchmarking databases but others would not}~\cite{kitch07}.
\end{quote}
\ei
How can these contradictory results be explained?
One possibility is that 
effort estimation requires the collection of project data, some of which have
ambiguous definitions. For example, one of the features of the COCOMO-family~\cite{boehm00a}
of effort predictors is ``applications experience'' (aexp).
According to one on-line source\footnote{\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/drivers.html}}, this feature is defined as follows: 
``the project team's equivalent level of experience with 
{\em this type} of application''. 
No guidance is offered regarding how to characterize ``this type of application''.
Hence, there is some degree of ambiguity in this definition.
We conjecture that the ambiguity of the effort estimation features is one reason
for the variance in the results reported by MacDonnel \& Shepperd and Kitchenham et al.~\cite{macdonell07, kitch07}.


%To test this hypothesis, the rest of this paper assesses
%%the relative merits of WC-vs-CC data for defect prediction.
%
\subsection{On Defect Prediction Using Static Code Features}\label{sec:dp}
%
%This section updates a literature review
%from a prior publication on defect prediction~\cite{me07b}.




%We take care to distance our work on defect prediction   from
%Musa-style software reliability engineering (SRE)~\cite{musa87}. In standard
%SRE, there is a track
%record of post-release {\em failures} and the goal is to predict
%the probability of failure on demand in the next call on the software.
%One of the major results of the SRE literature is that as the software
%is developed and bugs are removed, this probability forms a reliability
%growth curve with predictable properties~\cite{musa87}.

%The standard defect prediction task is  different from standard SRE in that we 
%learn defect predictors from
%logs of pre-release {\em defects} found by, say, a V\&V team. Such
%defects may be seen in code from systems that is not yet complete.
%Hence, many of the features used in this kind of analysis are static
%code features that can be extracted from as-yet-un-executing code.

%Note that the patterns  found in pre-release {\em defects} or post-release
%{\em failures} can be used in different ways.
%Reliability growth
%curves can be used to predict how long before the running
%software will reach some required reliability of, say, one failure
%in 10,000 calls.  
%Defect predictors
%can be used to bias the ordering
%of modules to be inspected by   verification and validation teams:
%\bi
%\item
%In the case where {\em there are insufficient resources to inspect all
%code} (which is a very common situation in industrial developments),
%defect predictors can be used to increase the odds that the inspected
%code will have more defects. 
%\item
%In the case where {\em all the code is to
%be inspected}, but that inspection process will take weeks to months
%to complete, defect predictors can be used to increase the odds
%that defective modules will be inspected earlier. This is useful
%since it gives the development team earlier notification of what
%modules require rework, hence giving them more time to complete
%that rework prior to delivery.
%\ei

We study defect predictors learned from
static code attributes since they are {\em useful}, {\em easy to
collect}, and {\em widely-used}.

%{\em Useful}: 
%This paper  finds defect predictors
%with a probability of detection of 80\%, or higher.
%This is 
%higher than currently-used
%industrial
%methods such as manual code reviews:
%\bi
%\item
%A {\em IEEE Metrics
%2002}~\cite{shu02} panel
%concluded that manual software  reviews can find ${\approx}60\%$ 
%of defects\footnote{That panel supported neither Fagan's
%claim~\cite{fagan86} that inspections can find 95\% of defects before
%testing or Shull's claim that specialized directed inspection methods
%can catch 35\% more defects that other
%methods~\cite{shull00a}.}  
%\item
%In 2004, Raffo (personnel communication) reports
%that the defect detection capability of
%industrial review methods can vary from \mbox{probability of detection: $pd=TR(35, 50,
%65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
%min/mode/max of $a,b,c$.}.
% for full Fagan inspections~\cite{fagan76} to
%\mbox{ $pd=TR(13, 21, 30)\%$} for less-structured inspections.
%\ei

%{\em Easy to collect}: 
%static code attributes like lines of code
%and  the McCabe/Halstead
%attributes can be automatically and cheaply collected, even for very large systems~\cite{conf/icse/NagappanB05a}.
%By contrast:
%\bi
%\item 
%Other methods such as manual code reviews are labor-intensive;
%e.g.
%8 to 20 LOC/minute can be
%inspected and this effort repeats for all members of the review team,
%which can be as large as four or six~\cite{me02f}. 
%\item
%Other features (e.g. number of developers,
%the software development practices used to develop the code) may be unavailable
%or hard to characterize.
%\ei

%{\em Widely used}:
%Many researchers use static attributes to guide software 
%quality predictions  
%(see \cite{halstead77,mccabe76,chapman02,me04g,polyspace,hall00,nikora03,conf/icse/NagappanB05a,khoshgoftaar01, Khoshgoftaar:2004p1877,conf/ictai/TangK04,journals/ese/KhoshgoftaarS03,me03a,me02e,me03k,me03q,porter90,tiang95,khoshgoftaar99,srinivasan95}).
%Verification and validation (V\&V) textbooks
%(\cite{rakitin01}) advise using static code complexity attributes
%to select modules worthy of manual inspections.  

{\em Useful}: 
Defect predictors are considered useful, if they provide a prediction performance that is comparable to or better than manual reviews \cite{moser08}. 
This paper  finds defect predictors
with a probability of detection of 80\%, or higher.
This is 
higher than currently-used
industrial
methods such as manual code reviews:
\bi
\item
A {\em IEEE Metrics
2002}~\cite{shu02} panel
concluded that manual software  reviews can find ${\approx}60\%$ 
of defects\footnote{That panel supported neither Fagan's
claim~\cite{fagan86} that inspections can find 95\% of defects before
testing or Shull's claim that specialized directed inspection methods
can catch 35\% more defects that other
methods~\cite{shull00a}.}  
\item
In 2004, Raffo (personnel communication) reports
that the defect detection capability of
industrial review methods can vary from \mbox{probability of detection: $pd=TR(35, 50,
65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
min/mode/max of $a,b,c$.}.
 for full Fagan inspections~\cite{fagan76} to
\mbox{ $pd=TR(13, 21, 30)\%$} for less-structured inspections.
\ei
Further, defect predictors based on static code features are considered as static analysis, since they do not require the execution of code. Zheng et.al. compares automated static analysis (ASA) with manual inspections on Nortel software and conclude that {\em
...our results indicate that ASA is an economical complement to other verification and validation techniques}

%\bsi
%\item
%A {\em IEEE Metrics
%2002}~\cite{shu02} panel
%concluded that manual software  reviews can find ${\approx}60\%$ 
%of defects.  
%\item
%In 2004, Raffo (personnel communication) reports
%that the defect detection capability of
%industrial review methods can vary from \mbox{probability of detection: $pd=TR(35, 50,
%65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
%min/mode/max of $a,b,c$.}.
% for full Fagan inspections~\cite{fagan76} to
%\mbox{ $pd=TR(13, 21, 30)\%$} for less-structured inspections.
%\esi

{\em Easy to collect}: 
Employing defect predictors in practice should not take too much time \cite{moser08} for data collection and constructing the models themselves. 
An advantage of static code features is that they can be quickly and automatically collected
from the source code, even if no other information is available.
Static code attributes like lines of code
and  the McCabe/Halstead
attributes can be automatically and cheaply collected, even for very large systems~\cite{conf/icse/NagappanB05a}.
By contrast, other methods such as manual code reviews are labor-intensive;
e.g.
8 to 20 LOC/minute can be
inspected and this effort repeats for all members of the review team,
which can be as large as four or six~\cite{me02f}. 
Furthermore,
other features (e.g. number of developers,
the software development practices used to develop the code) may be unavailable
or hard to characterize.

{\em Widely used}:
Many researchers use static features to guide software 
quality predictions  
(see \cite{halstead77,mccabe76,chapman02,me04g,polyspace,hall00,nikora03,conf/icse/NagappanB05a, Khoshgoftaar:2004p1877,conf/ictai/TangK04,porter90,tiang95,srinivasan95}).
Verification and validation (V\&V) textbooks
advise using static code features
to select modules worthy of manual inspections \cite{rakitin01}.  

%For several  years, the first author worked on-site at the NASA software Independent Verification
%and Validation facility
%and he
%knows of several large government software contractors that won't
%review software modules {\em unless} tools like McCabe predict that
%they are fault prone.  
%For example, 
%in Feburary 2008 the first author attended two briefings by NASA and European Space Agency test engineers
%who both cited McCabe's $v(g)\ge 10$ as one of their triggers for modules requiring closer inspections.





\begin{table}[!t]
\caption{Some representative $pd$s and $pf$s for prediction problems
 from the UC Irvine machine learning
database~\protect\cite{Blake+Merz:1998}. These values were generated
using the
standard
settings of a state-of-art decision tree learner (J48). For each data set, ten
experiments were conducted, where a decision tree was learned on 90\% of
the data, then tests are done of the remaining 10\%. 
The numbers shown here are the average results across ten such experiments.}\label{tbl:uci}
\scriptsize
\begin{center}
\begin{tabular}{r|rr|}\cline{2-3}
\multicolumn{1}{c|}{}&\multicolumn{2}{c|}{probability of}\\\hline
data&	detection&	false alarm\\\hline
pima diabetes&	60&	19\\
sonar	&	71	&29\\
horse-colic	&71	&7\\
heart-statlog &	73	&21\\
rangeseg	&76	&30\\
credit rating	&88&	16\\
sick	&88&	1\\
hepatitis	&94	&56\\
vote	&95	&3\\
ionosphere	&96	&18\\\hline
mean &81	& 20\\\cline{2-3}
\end{tabular}
\end{center}
\end{table}



Nevertheless, there are many reasons to doubt the value
of static code attributes for defect prediction. Static code attributes 
are hardly a complete characterization of
the internals of a function. Fenton offers an insightful example where
{\em the same} functionality is achieved using {\em different}
programming language constructs resulting in {\em different} static
measurements for that module~\cite{fenton96}.  
Fenton uses this example to argue the uselessness
of static code attributes. 
Further,
Fenton~\&~Pfleeger  
note that the main McCabe's attribute
(cyclomatic complexity, or $v(g)$) 
is highly correlated with lines of code~\cite{fenton96}.
Shepperd~\&~Ince repeated that result, commenting that ``for a large class of software it
(cyclomatic complexity) is no more than a proxy for, and in many cases
outperformed by, lines of code''~\cite{shepperd94}.

However, if 
the above criticisms are correct then we would predict
that,
in general, the performance of a defect predictor learned by a data miner should be very poor.
More specifically,
the supposedly better static code attributes such as Halstead and Mccabe should
perform no better than just simple thresholds on lines of code. 

Neither of these predictions are true, at least for the data sets used in this study.
The defect predictors learned from static code attributes
perform surprisingly well.
Formally, learning a defect predictor is a {\em binary prediction problem} where each
module in a project has been labeled as "defect-free" or "defective".
The learning problem is to build some predictor which guesses the labels
for as-yet-unseen modules. In this paper we find predictors (i.e. for SOFTLAB projects) with
a probability of detection (pd) and probability of false alarm (pf) of
\[median(pd) = 88; median(pf) = 29\]

\tbl{uci} lets us compare our new results against
standard binary prediction results from the UC Irvine machine learning
repository of standard test sets for data miners~\protect\cite{Blake+Merz:1998}.
Our $median(pd,pf)$
are very close to
the standard results of 
 $mean(pd,pf)$=(81\%,20\%) 
which is noteworthy in two ways:
\be
\item It is unexpected. If static code attributes capture so little
about source code (as argued by 
Shepherd, Ince, Fenton and Pfleeger), then we would expect lower probabilities
of detection and much higher false alarm rates. 
\item
These $(pd,pf)$ results are 
better than currently
used industrial methods such as the $pd{\approx}60\%$ reported
at the 2002 IEEE Metrics panel or the $median(pd)=21..50$ reported
by Raffo\footnote{
Please note that we can only relatively compare the defect {\em detection}
properties of automatic vs manual methods.
Unlike automatic defect
prediction via data mining, the above manual inspection methods
don't just report ``true,false'' on a module, Rather, the manual
methods also provide specific debugging information.  Hence, a
complete comparison of automatic vs manual defect prediction would
have to include both an analysis of the time to {\em detect} potential
defects {\em and} the time required to {\em fix} them.
Manual methods might score higher
to automatic methods
since they can offer more clues back to the developer about
what is wrong with the method.  However, such an analysis is beyond
the scope of this paper.
Here, we focus only on the relative merits of different methods for 
{\em predicting} error prone modules.
}.
\ee

\section{Practical Implications}	
While
\tbl{uci} shows that our defect detectors work nearly as well as standard data mining methods,
it does not necessarily demonstrate that 
false alarm rates of around $29\%$ 
are useful in an industrial context. 
For example, Arisholm and Briand have certain concerns on the practical usage of defect predictors~\cite{arisholm06}. They argue that
if X\% of the modules are predicted to be faulty and if those modules contain less than X\% of the defects,
then the costs of generating the defect predictor is not worth the effort.

Let us analyze the testing efforts on MW1 project from Arisholm and Briand's point of view. For MW1, there are a total of 403 modules with 31 defective and 372 defect-free ones. CC model yields 90\% pd and 68\% pf, and one should examine 280 modules, which is around a 31\% reduction in inspection efforts compared to examining all modules. Yet, we argue that 68\% pf rate is quite high and using NN we are able to reduce it to 33\% along with 68\% pd. This corresponds to examining 144 modules, a reduction of 47\% compared to exhaustive testing (and we assume an exhaustive test should examine 274 modules for detecting 68\% defects, as Arisholm and Briand suggests). 

This analysis can be extended for all projects used in this paper.
For instance, the company from which SOFTLAB data in \tbl{data1} are collected, is keen to use our detectors, arguing that they
operate
	in a highly competitive market segment where profit margins are
	very tight. Therefore reducing the cost of the product even by 1\%
	can make a major difference both in market share and profits. 
Their
	applications are embedded systems where, over the last decade, the software components
	have taken precedence over the hardware. Hence
	their problem is a software engineering problem. 
According to Brooks~\cite{brooks95}, 
half the cost of software development is in unit and systems testing. 
The company also believes that 
	their main challange is
	the testing phase and they seek predictors that indicate
	where the defects might exist {\em before} they start testing. Their expectation from the predictor is not to detect all defects, but to guide them to the problematic modules so that they can detect more defects in shorter times. Hence, any reduction in their testing efforts allows
them to efficiently use their scarce resources. 

Considering  the results of our experiments, if a company lacks local data,
we would suggest a two-phase approach.
In phase one, that organization uses imported CC data, filtered via NN.
Also, during phase one, the organization should start a data collection program to collect
static code attributes. Phase two commences when there is {\em enough} local WC data to learn
defect predictors. During phase two, the organization would switch to new defect predictors
learned from the WC data. 

An important issue worth more mentioning is the concern about the time required for setting up a metric program (i.e. in order to collect data for building
actual defect predictors).
Our incremental WC results suggest that, in the case of defect prediction,
this concern may be less than previously believed. 
Kitchenham {\em et al.}~\cite{kitch07} argue that
organizations use
cross-company data since within-company data
can be so hard to collect:
\bi
\item
The time required to collect enough data on past projects from within a  company
may be prohibitive.
\item
Collecting within-company data may take so long
that 
technologies change and older
projects do not represent current practice.
\ei
In most of our experiments, as few as 100 modules may be enough to learn
adequate defect predictors. 
When so few examples are enough,
it is possible that
projects can learn local defect predictors that are relevant to their
current technology in just a few months. 

Further, our experiences with our industry partners show that data collection is not necessarily a major concern. Static code attributes can be automatically and quickly collected with relatively little effort. We have found that when there is high level management commitment, it becomes a relatively simple process. For the three projects of SOFTLAB data, neither the static code attributes, nor the mapping of defects to software modules were available when the authors attempted to collect these data. Since these were smaller scale projects, it was sufficient to spend some time with the developers and going through defect reports. 
Although not all projects have 100 modules individually, the company has a growing repository from several projects and enough data to perform defect prediction.

We also have experience with a large scale telecommunication company, where a long-term metric program for monitoring complex projects (around 750.000 lines of code) requires introducing automated processes. Again with high level management support, it was possible to employ appropriate tool support and these new processes were introduced easily and invisible to the staff. For that project, we have now a growing repository of defects mapped with source code (around 25 defects per month). Software in that project are being developed for more than 10 years and have very low defect rates. We have obtained the first results in the $8^{th}$ month of the project and it is planned to be completed in $12$ months. In summary setting up a metric program for defect prediction can be done more quickly than it is perceived.
