\section{Motivation}
\subsection{Why Study WC vs CC?}

The genesis of this work was several frustrating years 
working at NASA trying to collect data on new projects. This proved
difficult to do, for a variety of reasons (including feuding NASA centers and
contractors trying to maintain some control over the information flow
to the client). Several times in that process, we asked ``Is it necessary
to collect new data? Can't we just apply old data?''.

In the middle of 2007, we had an opportunity to experiment with using CC
data on a Turkish software development company. That company worked
in a highly competitive market and a decrease in their bug rate of even 1\%
had an economic benefit to them.
However, that company had no historical data with which to train its defect detectors.
We therefore trained a defect detector on NASA data and applied it to the Turkish
software.
Much to our surprise,
that defector could ``adequately'' predict errors within Turkish software
(``adequate'' in the sense that the Turkish client looked at the code
that triggered our detectors, found bugs, and was subsequently
 interested enough to request
more studies of this nature).  

Based on that small informal experiment, 
 we saw:
\bi
\item The business benefits in using CC
data: no need to wait for local data collection;
\item  The adequacy of the learned detectors; 
\ei
A single case study is not enough to justify a general principle. Accordingly, we conducted the experiments
described in this paper.

\subsection{Why Study Defect Predictors?}
Defect predictors are subtley 
different to other quality assurance practices.
This
section highlights those differences and explains why
we elect to study them.

It is insightul to contrast defect
prediction 
from Musa-style software reliability engineering (SRE)~\cite{musa87}.
In standard SRE, there is a track record of post-release failures 
and the goal is to predict the probability of failure on demand 
in the next call on the software. One of the major results of the 
SRE literature is that as the software is developed and bugs 
are removed, this probability forms a reliability growth curve 
with predictable properties.

The standard defect prediction task is different to standard 
SRE.
Our task is to predict defective modules during development and before delivery.
Such early lifecycle detection is important since, as Boehm reports, repair
cost increases exponentially, the longer a bug remains in the system~\cite{boehm88}.
Hence,
we learn defect predictors from logs of pre-release 
defects found by a verification and validation (V\&V) team. Such defects may be seen 
in code from systems that is not yet complete. Hence, many 
of the features used in this kind of analysis are static code 
features that can be extracted from as-yet-un-executing code. 

The patterns found in pre-release defects or post-release failures can be used in different ways. Reliability 
growth curves can be used to predict how long before 
the running software will reach some required reliability of, 
i.e. one failure in 10,000 calls. Defect predictors, on the other 
hand, can be used to bias the ordering of modules to be 
inspected by V\&V teams: 

\bi
\item
In the case where there is insufficient resources to inspect 
all code (which is a very common situation in industrial 
developments), defect predictors can be used to increase 
the odds that the inspected code will have more defects. 
\item
In the case where all the code is to be inspected, but that inspection process will
take weeks to months to complete, defect predictors can be used to increase the 
chances that defective modules will be inspected earlier. This is useful since it 
gives the development team earlier notification of what modules require rework, 
hence giving them more time to complete that rework prior to delivery.
\ei
The above two cases are quite common.
This research team comes from the United
States and Turkey and we only began exploring defect detectors when
we realized that our user community was making extensive use of these
detectors.

For example, for several  years, one of the authors worked on-site
at the NASA software Independent Verification and Validation facility
and he knows of several large government software contractors that
will not review software modules {\em unless} tools like McCabe predict
that they are fault prone.  In Feburary 2008, that
author attended two briefings by NASA and European Space Agency
test engineers who both cited McCabe's $v(g)\ge 10$ as one of their
triggers for modules requiring closer inspections.

What is significant about the NASA experience is that NASA is not ``one'' organization,
in the conventional sense. Rather, NASA is an umbrella organization that loosely
monitors and guides large group of sub-contractors. 
That is, data collected from ``NASA'' is really data collected from numerous large
software engineering companies like Lockheed-Martin, Boeing, Rockwell-Collins, etc.

As per our Turkish experience, the software development industry in Turkey has been growing very rapidly in the last couple of years as the domestic economy stabilizes. We observe more and more software development houses that are delivering projects to European and U.S markets. As the companies mature and they compete in the global market place they need to improve their processes, take measures to lower their costs and increase the quality of their products. Therefore they realize that there needs to be a consistent methodology for software development and they need to collect product and project related metrics data for measurement and evaluation. In this context, most of our Turkish industrial partners name their major challenges as effectively allocating their verification resources for locating defects before product delivery. Specifically, our Turkish industrial partner uses the outcomes of this research in order to guide their decision making processes during testing.

What may surprise the reader is that our detectors are widely used,
despite being quite ``coarse-grained''.
In the following study, our only knowledge of modules will be that they are defective
or defect-free. This work makes no use of other knowledge about a module such
as its criticality or the dates on which it failed.
We look forward to the day when we can apply
our data miners to access comprehensive failure logs as well as  
accurate criticality predictions.
However, that day is not yet here:
\bi
\item
{\em Regarding failure occurence logs:} 
When working with large teams of contractors, it is often difficult to access
precise logs on when failures occured
since such information is often corporate critical. The NASA
coarse-grained
information used
in this study, for example, 
took years of careful negotiation to acquire. All the data passed
to us was highly sanitized; e.g. this research team could not
even access project or module names.
\item
{\em Regarding criticality knowledge:}  
Our goal is to predict defective modules during development and before delivery.
Goseva and Hamill~\cite{goseva07} report that pre-delivery
perceptions
of module criticality can be widely misleading. In their work, they trace 
errors back to the modules that caused runtime faults. They find that
a remarkably small percent of modules are  implicated in all faults and that
set can change, radically, if the input space to the program alters.
\ei
For further notes on related work on defect predictors from static code attributes,
see Section 8.3.
