%\documentclass[twocolumn,journal,draftcls,compsoc]{IEEEtran}
\documentclass[two,journal,compsoc]{IEEEtran}

\usepackage{pifont}
\usepackage{times}
\usepackage{alltt}
\usepackage{graphicx}
\usepackage{url}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\makeatletter
\makeatother


\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}

\usepackage[nocompress]{cite}


% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
\begin{document}

\title{On the Relative Value of Cross-company and Within-Company Data
for Defect Prediction}


\author{Burak~Turhan,~\IEEEmembership{Student~Member,~IEEE,}
        Tim~Menzies,~\IEEEmembership{Member,~IEEE,}
        Ayse~Bener,~\IEEEmembership{Member,~IEEE,}\\ 
		and Justin~Distefano
\thanks{Mr. Turhan and Dr. Bener are with the 
Department of Computer Engineering, Bogazici University, Turkey.
Emails:
\protect\url{turhanb@boun.edu.tr},
\protect\url{bener@boun.edu.tr}.
}
\thanks{Dr. Menzies and  Mr. Distefano
are with the Lane Department of Computer Science and Electrical Engineering, 
West Virginia
University.
Mr. Distefano is also Chief Programmer at Integrated Software Metrics.
Emails: \protect\url{tim@menzies.us}
and \protect\url{jdistefano@ismwv.com}
}
\thanks{The research described in this paper
was supported by
Bogazici University research fund under grant number BAP-06HA104, by TUBITAK under grant number EEEAG 108E014
and at
West Virginia  University
under grants with NASA's Software Assurance Research Program.
Reference herein to any specific commercial product, process, or service
by trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States Government.
}\thanks{See 
See \protect\url{http://menzies.us/pdf/07interra.pdf} for an earlier draft of this paper.
Manuscript received June, 2008; revised XXX,}} 

\markboth{Journal of ???,~Vol.~6, No.~1, January~2008}%
{Turhan \MakeLowercase{\textit{et al.}}: On the Comparative Value of Local Data for Defect Prediction}

\maketitle

\begin{abstract}
We report experiments where defect predictors from cross-company
(CC) data result in  dramatically high probabilities of detecting faulty
modules  (median value of 7 data sets: 97\%).  Sadly,
this CC data also yield very large false alarm rates  (median
value: 64\%).

We explain this high false alarm rate by hypothesizing that CC data
mixes useful data with an excess of
{\em extraneous information}.  In support of this
hypothesis, we show that simple   nearest neighbor (NN)
sampling of CC data can filter the extraneous information to yield detectors with performance close
to, but not better than, within-company (WC) data.  That is, NN-filtered CC data
can be a viable stop-gap technique
while local data is being collected.  

That stop-gap can be quite
brief: we demonstrate in this paper that the minimum number of data
samples required to build effective defect predictors can be quite
small (often, just 100 modules).

Hence, for defect prediction, we recommend a two-phase approach. In phase one,
companies should use NN-filtered CC data to initiate defect
prediction process and simultaneously start collecting WC (local)
data. Once enough WC data is collected, organizations
should switch to phase 2 and use predictors learned from WC data.~\\ \\
KEYWORDS: defect prediction;learning; metrics (product metrics); cross-company; within-company
\end{abstract}

\section{Introduction}
What is the generality of our theories for common problems in empirical software engineering? Are the lessons learned
from {\em Project-X} in {\em Company-A} relevant to {\em Project-Y} in 
{\em Company-B}? Theoretically, 
the answer is ``no''.
Organizations can work in different domains, have
different processes, and define/measure defects and other aspects of
their products and processes in different ways. 
Worse, all too often, organizations do not precisely define their
processes, products, measurements, etc. 
Hence, given data from {\em Project-X} from {\em Company-A}
it is not clear to what extent the domain, process, and measurement 
varies from that project to other projects.


Our view is that cross-company data must be used with 
great care in any software engineering problem.
For example, previously~\cite{me06d}, we have built elaborate tool sets for converting supposedly 
general {\em effort estimation}
models to local conditions. 
That work concluded that data imported from other organizations must be extensively pruned (removal
of extraneous features and projects) before it can be tuned to local conditions.

In this paper, we turn the attention to {\em defect prediction}. 
Specifically,
this paper 
assesses the relative merits
of cross-company (CC) vs within-company (WC) data for  defect prediction.
To the best of our knowledge, this kind of analysis is a novel one in defect prediction literature.
We perform several experiments to answer the following questions:

\subsection{Are the defect predictors learned from CC data beneficial for organizations?}

Our first experiment
identifies the conditions under which
cross-company  data are preferred
to within-company  for the purposes of learning
defect predictors.
Those conditions will turn out to be quite extreme; so much
that they 
hold in only a small number of organizations
 (e.g.
organizations would have to tolerate extremely high false alarm rates).
Hence,
except in very rare cases,
this first experiment deprecates the use of unfiltered cross-company data for defect prediction.

\subsection{Can we still make use of CC data for defect prediction?}

Our explanation for the poor performance of CC data is that it mixes useful information with an excess
of extraneous information.
To explore, we applied
nearest neighbor (NN) filtering to the CC data.
This second experiment shows that some of the drawbacks of CC data can be removed. While,
WC data still performs best, NN-filtered CC data can be viable alternative, while an organization
starts its own local WC collection program.

\subsection{How much data do organizations need for constructing a local model for defect prediction?}

Kitchenham {\em et al.}~\cite{kitch07} argue that
organizations use
cross-company data since within-company data
can be so hard to collect:
\bi
\item
The time required to collect enough data on past projects from within a  company
may be prohibitive.
\item
Collecting within-company data may take so long
that 
technologies change and older
projects do not represent current practice.
\ei
We show below that
predictors learned from a 
mere one hundred examples  perform as well as predictors learned from many more examples.
That is:
\bi
\item 
Predictors tuned to the particulars of one company can
be learned using very little data, collected in a very small amount of time;
\item
The above two reasons~\cite{kitch07}  for avoiding WC-data
in {\em effort estimation}
may not neccessarily hold for
{\em defect prediction}.
\ei

\subsection{Can our theories and results be generalized?}

The above experiments use data collected from NASA projects. In order to check the  external validity of our
conclusions, we replicated all the above three experiments on
data from a company that has no ties with NASA: specifically, a Turkish company writing software controllers
for Turkish whitegoods. All the results described above also hold for the Turkish data. While this does not
conclusively prove the external validty of our conclusions, it does suggest that these results are not
just some fluke of one domain.

\section{Related Work}
\subsection{Effort Estimation}
Before beginning, it is important to question the value of WC vs CC studies. Intuitivaly,
it could be argued that
there is no issue here since {\em of course} local data is always better than imported data.

Prior to this article, there was no test of this intuition in the domain of {\em defect prediction}.
However, in the domain of {\em effort estimation}, it turns out that this intuition has only mixed support:
\bi
\item
Mendes et al.~\cite{mendes07} found within-company data
performed much better than cross-company data for predicting
estimation effort of web-based projects.  They only recommend using
cross-company data in the special case when that ``data are obtained
using rigorous quality control procedures''.  
\item
A similar conclusion was reached by Abrahansson et al.
who  discussed learning effort predictors in the context
of an agile development process~\cite{abrahamsson07}. They strongly advocate
the use of WC-data.
\ei
However,
other studies are not so clear in their conclusions:
\bi
\item
MacDonnel \& Shepperd tried
to find trends in a set of papers relating to project 
management and effort estimation~\cite{macdonell07}. However, the
 papers studied by MacDonnel \& Shepperd used a wide range of data
sets so these authors found it hard to offer  a definitive combined
conclusion.  
\item
In other work, after a review of numerous case studies, 
Kitchenham
et al.~\cite{kitch07} concluded that the value of CC vs WC data for effort estimation
is
unclear: \begin{quote} {\em 
{\ldots}some organizations would benefit from using models derived from
cross-company benchmarking databases but others would not}~\cite{kitch07}.
\end{quote}
\item
Premraj and Zimmermann suspects that the reason for the contradictory results are due to heterogeneity in data. Therefore they build business specific cost models to have homogeneity in data. They compare within company, cross company and business specific cost models and report that although cross company models perform slightly worse, neither model is significantly better than others~\cite{Premraj:2007p1626}.

\ei
How can these contradictory results be explained?
One possibility is that 
effort estimation requires the collection of project data, some of which has
ambiguous definitions. For example, one of the features of the COCOMO-family~\cite{boehm00a}
of effort predictors is "applications experience'' (aexp).
According to one on-line source\footnote{\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/drivers.html}}, this feature is defined as follows: 
``the project team's equivalent level of experience with 
{\em this type} of application''. 
No guidance is offered regarding how to characterize ``this type of application''.
Hence, there is some degree of ambiguity in this definition.
We conjecture that the ambiguity of the effort estimation features is one reason
for the variance in the results reported by MacDonnel \& Shepperd and Kitchenham et al.~\cite{macdonell07, kitch07}.

Static code features, on the other hand, are not so ambiguous. Simple toolkits
can be used to collect these features in a rapid, automatic, 
and uniform manner across multiple projects.
Therefore, in theory, conclusions reached from these features should be less
ambiguous than those reached from effort estimation features.

To test this hypothesis, the rest of this paper assesses
the relative merits of WC-vs-CC data for defect prediction.

\subsection{Defect Prediction}\label{sec:dp}

This section updates a literature review
from a prior publication on defect prediction~\cite{me07b}.

For the two experiments in this paper, 
we learn defect predictors from  tables of static code
features defined by McCabe~\cite{mccabe76} and
Halstead~\cite{halstead77}. McCabe (and Halstead) are ``module''-based
metrics. Each row in the table stores information from one
module;  the smallest unit of functionality\footnote{
In other languages, modules may be called ``function'' or
``method''.}.  
To learn such predictors,  the data tables are
augmented with 
one column holding  boolean
values for
``defects detected''.
The data mining task is to find combinations of code features that predict
for the value in the defects column.



Defect predictors
can be used to bias the ordering
of modules to be inspected by   verification and validation teams:
\bi
\item
In the case where {\em there are insufficient resources to inspect all
code} (which is a very common situation in industrial developments),
defect predictors can be used to increase the odds that the inspected
code will have more defects. 
\item
In the case where {\em all the code is to
be inspected}, but that inspection process will take weeks to months
to complete, defect predictors can be used to increase the odds
that defective modules will be inspected earlier. This is useful
since it gives the development team earlier notification of what
modules require rework, hence giving them more time to complete
that rework prior to delivery.
\ei

We study defect predictors learned from
static code attributes since they are {\em useful}, {\em easy to
use}, and {\em widely-used}.

{\em Useful}: 
This paper  finds defect predictors
with a probability of detection of 80\%, or higher.
This is 
higher than currently-used
industrial
methods such as manual code reviews:
\bi
\item
A {\em IEEE Metrics
2002}~\cite{shu02} panel
concluded that manual software  reviews can find ${\approx}60\%$ 
of defects\footnote{That panel supported neither Fagan's
claim~\cite{fagan86} that inspections can find 95\% of defects before
testing or Shull's claim that specialized directed inspection methods
can catch 35\% more defects that other
methods~\cite{shull00a}.}  
\item
In 2004, Raffo (personnel communication) reports
that the defect detection capability of
industrial review methods can vary from \mbox{probability of detection: $pd=TR(35, 50,
65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
min/mode/max of $a,b,c$.}.
 for full Fagan inspections~\cite{fagan76} to
\mbox{ $pd=TR(13, 21, 30)\%$} for less-structured inspections.
\ei

{\em Easy to use}: 
static code attributes like lines of code
and  the McCabe/Halstead
attributes can be automatically and cheaply collected, even for very large systems~\cite{conf/icse/NagappanB05a}.
By contrast:
\bi
\item 
Other methods such as manual code reviews are labor-intensive;
e.g.
8 to 20 LOC/minute can be
inspected and this effort repeats for all members of the review team,
which can be as large as four or six~\cite{me02f}. 
\item
Other features (e.g. number of developers,
the software development practices used to develop the code) may be unavailable
or hard to characterize.
\ei

{\em Widely used}:
Many researchers use static attributes to guide software 
quality predictions  
(see \cite{halstead77,mccabe76,chapman02,me04g,polyspace,hall00,nikora03,conf/icse/NagappanB05a,khoshgoftaar01, Khoshgoftaar:2004p1877,conf/ictai/TangK04,journals/ese/KhoshgoftaarS03,me03a,me02e,me03k,me03q,porter90,tiang95,khoshgoftaar99,srinivasan95}).
Verification and validation (V\&V) textbooks
(\cite{rakitin01}) advise using static code complexity attributes
to select modules worthy of manual inspections.  
For several  years, the first author worked on-site at the NASA software Independent Verification
and Validation facility
and he
knows of several large government software contractors that won't
review software modules {\em unless} tools like McCabe predict that
they are fault prone.  
For example, 
in Feburary 2008 the first author attended two briefings by NASA and European Space Agency test engineers
who both cited McCabe's $v(g)\ge 10$ as one of their triggers for modules requiring closer inspections.

Nevertheless, there are many reasons to doubt the value
of static code attributes for defect prediction.
Descriptions of software modules {\em only} in terms of static
code attributes can overlook some important aspects of software
including
the type of application
domain; the skill level of the individual
programmers involved in system development;
 contractor development practices;
the variation in measurement practices;
and the validation of the measurements and instruments used
to collect the data.
For this reason some researchers augment, or replace
static code measures with other information such as the history
of past faults or changes to code or number of
developers who have worked on the code~\cite{graves00}.

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r|rr|}\cline{2-3}
\multicolumn{1}{c|}{}&\multicolumn{2}{c|}{probability of}\\\hline
data&	detection&	false alarm\\\hline
pima diabetes&	60&	19\\
sonar	&	71	&29\\
horse-colic	&71	&7\\
heart-statlog &	73	&21\\
rangeseg	&76	&30\\
credit rating	&88&	16\\
sick	&88&	1\\
hepatitis	&94	&56\\
vote	&95	&3\\
ionosphere	&96	&18\\\hline
mean &81	& 20\\\cline{2-3}
\end{tabular}
}
\end{center}
\caption{Some representative $pd$s and $pf$s for prediction problems
 from the UC Irvine machine learning
database~\protect\cite{Blake+Merz:1998}. These values were generated
using the
standard
settings of a state-of-art decision tree learner (J48). For each data set, ten
experiments were conducted, where a decision tree was learned on 90\% of
the data, then tests are done of the remaining 10\%. 
The numbers shown here are the average results across ten such experiments.}\label{fig:uci}
\end{figure}

\begin{figure*}[!b]
\scriptsize
\begin{center}
\begin{tabular}{l|l|llrrrrr@{.}r}
&     &       &   & (\# modules)         &          &  & & &\\
source
&project & language & description & examples & features  & loc &   \multicolumn{2}{c}{\%defective}\\\hline
NASA&pc1  & C++ &Flight software for earth orbiting satellite& 1,109    & 21       &25,924&&    6&94\\
NASA&kc1  & C++ &Storage management for ground data& 845      & 21       &42,965&&    15&45\\
NASA&kc2  & C++ &Storage management for ground data& 522      & 21        &19,259&&   20&49\\
NASA&cm1  & C++ &Spacecraft instrument& 498      & 21        &14,763&&   9&83\\
NASA&kc3  & JAVA &Storage management for ground data& 458      & 39      &7749&&     9&38\\
NASA&mw1  & C++ &A zero gravity experiment related to combustion& 403      & 37      &8341&&     7&69\\
SOFTLAB& ar4 & C &Embedded controller for white-goods&107 &  30       &9196&     & 18 & 69\\
SOFTLAB& ar3& C &Embedded controller for white-goods& 63 &    30       &5624&   & 12 & 70\\
NASA&mc2  & C++ &Video guidance system & 61      & 39         &6134& & 32&29\\
SOFTLAB & ar5 & C &Embedded controller for white-goods& 36 &  30         &2732&  & 22 & 22\\
\cline{5-4}
     \multicolumn{4}{c}{~}     &  4,102 

\end{tabular}
\end{center}
\caption{Ten tables of data, sorted in order of number of examples.
The rows labeled ``NASA'' come from NASA aerospace projects while the rows
labeled ``SOFTLAB'' come  from a Turkish software company writing
applications for domestic appliances.
For details on the features used in each data set, see  \fig{attr}.
}\label{fig:data}
\end{figure*} 


Also, static code attributes 
are hardly a complete characterization of
the internals of a function. Fenton offers an insightful example where
{\em the same} functionality is achieved using {\em different}
programming language constructs resulting in {\em different} static
measurements for that module~\cite{fenton96}.  
Fenton uses this example to argue the uselessness
of static code attributes. 
Further,
Fenton~\&~Pfleeger  
note that the main McCabe's attribute
(cyclomatic complexity, or $v(g)$) 
is highly correlated with lines of code~\cite{fenton96}.
Shepperd~\&~Ince repeated that result, commenting that ``for a large class of software it
(cyclomatic complexity) is no more than a proxy for, and in many cases
outperformed by, lines of code''~\cite{shepperd94}.

If 
the above criticisms are correct then we would predict
that,
in general, the performance of a predictor learned by a data miner should be very poor.
More specifically,
the supposedly better static code attributes such as Halstead and Mccabe should
perform no better than just simple thresholds on lines of code. 

Neither of these predictions are true, at least for the data sets used in this study.
The defect predictors learned from static code attributes
perform surprisingly well.
Formally, learning a defect predictor is a {\em binary prediction problem} where each
modules in a database has been labeled "defect-free" or "defective".
The learning problem is to build some predictor which guesses the labels
for as-yet-unseen modules. In this paper we find predictors with
a probability of detection (pd) and probability of false alarm (pf) of
\[median(pd) \ge 80; median(pf)\le 26\]

\fig{uci} lets us compare our new results against
standard binary prediction results from the UC Irvine machine learning
repository of standard test sets for data miners~\protect\cite{Blake+Merz:1998}.
Our $(pd,pf)$
are very close to
the standard results of 
 ($pd$,$pf$)=(81\%,20\%) 
which is noteworthy in two ways:
\be
\item It is unexpected. If static code attributes capture so little
about source code (as argued by 
Shepherd, Ince, Fenton and Pfleeger), then we would expect lower probabilities
of detection and much higher false alarm rates. 
\item
These $(pd,pf)$ results are 
better than currently
used industrial methods such as the $pd{\approx}60\%$ reported
at the 2002 IEEE Metrics panel or the $median(pd)=21..50$ reported
by Raffo\footnote{
But note that we can only relatively compare the defect {\em detection}
properties of automatic vs manual methods.
Unlike automatic defect
prediction via data mining, the above manual inspection methods
don't just report ``true,false'' on a module, Rather, the manual
methods also provide specific debugging information.  Hence, a
complete comparison of automatic vs manual defect prediction would
have to include both an analysis of the time to {\em detect} potential
defects {\em and} the time required to {\em fix} them.
Manual methods might score higher
to automatic methods
since they can offer more clues back to the developer about
what is wrong with the method.  However, such an analysis is beyond
the scope of this paper.
Here, we focus only on the relative merits of different methods for 
{\em detecting} errors.
}.
\ee


While
\fig{uci} shows that our defect detectors work nearly as well as standard data mining methods,
it does not demonstrate that 
false alarm rates of around $\frac{1}{4}$ 
are useful in an industrial context. 
Assessing such detectors in an industrial context depends on the industry. But
one of our clients
(the company from which SOFTLAB data in \fig{data} are collected) is keen to use our detectors, arguing that they
operate
	in a highly competitive market segment where profit margins are
	very tight. Therefore reducing the cost of the product even by 1\%
	can make a major difference both in market share and profits. 
Their
	applications are embedded systems where, over the last decade, the software components
	have taken precedence over the hardware. Hence
	their problem is a software engineering problem. 
According to Brooks~\cite{brooks95}, 
half the cost of software development is in unit and systems testing. 
The company also believes that 
	their main challange is
	the testing phase and they seek predictors that indicate
	where the defects might exist {\em before} they start testing. This allows
them to efficiently use their scarce resources. 

\section{Data}


This paper
focuses solely on predictors learned from static code features (These features are shown in \fig{attr} and explained in  \fig{halstead} and \fig{mccabe}). It is therefore
prudent to digress and discuss the value of other types of features.

Defect predictors have been successfully learned from wide variety of measures such as:
\bi
\item static code features~\cite{me07b};
\item churn metrics (rates of code change)~\cite{nikora03};
\item personnel details about the development team~\cite{nach08};
\item features extracted from requirements documents~\cite{jiang07}.
\ei
This list is hardly exhaustive and can be extended be either adding
new feature types
of combining existing ones.
For example,  Jiang, Cukic \& Menzies~\cite{jiang07} found that
defect predictors learned from
requirements {\em and} code measures can create predictors that out-perform predictors learned
from just code {\em or} just requirements metrics. 

Nevertheless, we hesitate to demand defect prediction should {\em always} be based on 
churn, requirements, code, or personnel features.
We have some theoretical evidence suggesting that the ``best'' set of features
may change from project to project (see the  feature subset selection experiments of
\cite{me07c}).
More pragmatically, we note that
different projects use different tools and build software
using different processes. Hence, different projects have access to
different sets of features which may be useful for predicting where
faults hide; for example:
\bi
\item An open source software (OSS) or agile process may have no access to the
detailed requirement documents used to build model by Jian, Cukic, \& Menzies;
\item The data used in this study (see \fig{data}) comes 
from NASA sub-contractors who do not report
the personnel information used by Nagappan et.al. in their defect predictions~\cite{nach08}.
\ei
Debates over the value of one kind of feature over another is 
orthogonal to the main point of this paper.
In this work, we identify a particular type of feature (i.e. static code features) 
which we can collect from multiple projects. With that data in hand, we now explore
the value of CC vs WC learning. For future work, we would repeat this analysis
with those other kinds of features, if and when they become available from multiple projects.

An advantage of static code features is that they can be quickly and automatically collected
from the source code, even if no other information is available.
The experiments of this paper use the static code data of \fig{data}, which are
downloaded from the PROMISE repository\footnote{\url{http://promisedata.org/repository}}.
We considered using static code features collected from OSS and examined
the defect logs of the Mozilla project. 
This experiment was abandoned when we realized that, at least for the Mozilla project,
that those issue reports did not accurately map between defect reports and modules;  e.g. 
a developer may checking many modules as part of a bug fix, some of which may actually
be enhancements generated as a side-effect of the bug fix.
Other researchers have had more success with OSS defect prediction~\cite{kim08}
and in future work we will try again to apply
our methods to OSS.




\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{l@{~}|l@{~}|l@{~}|c@{~}p{3.9cm}|}\cline{2-5}
		&\multicolumn{2}{l}{m = Mccabe }       &$v(g)$  &cyclomatic\_complexity  \\
        &        \multicolumn{2}{l}{~}                                 & $iv(G)$&design\_complexity\\
        &         \multicolumn{2}{l}{~}                                & $ev(G)$&essential\_complexity\\\cline{2-5}
        &locs&{loc}                         &   &loc\_total (one line = one count\\\cline{3-5}
	&	&loc(other)                      &   &loc\_blank           \\
        &       &                                &   &loc\_code\_and\_comment\\
        &       &                                &   &loc\_comments         \\
        &        &                                &   &loc\_executable\\
        &       &                                &    &number\_of\_lines (opening to closing brackets)\\\cline{2-5}
	&{Halstead} & h& $N_1$ &num\_operators\\
        &       &                                 &$N_2$ &num\_operands \\
        &       &                                 &$\mu_1$ &num\_unique\_operators\\
        &       &                                 &$\mu_2$ &num\_unique\_operands\\\cline{3-5}
        &       &  { H}&$N$  &length: $N= N_1 + N_2$\\
        &       &                                 & $V$  &volume: $V=N * log_{2}\mu$\\
        &       &                                 & $L$  &level: $L=V^*/V$ where\newline $V^*=(2+\mu{_2}^{*})log_{2}(2+\mu{_2}^{*})$\\
        &       &                                 & $D$  &difficulty: $D=1/L$\\
        &       &                                 &$I$   &content: $I=\hat{L}*V$ where\newline $\hat{L}=\frac{2}{\mu_{1}} * \frac{\mu_{2}}{N_{2}}$ \\
        &       &                                 & $E$  &effort: $E=V/\hat{L}$\\
        &       &                                 & $B$  &error\_est\\
        &       &                                 & $T$  &prog\_time: $T=E/18$ seconds\\\cline{2-5}
\end{tabular}}
\end{center}
\caption{Features used in this study. The Halstead features are explained in \fig{halstead}
and the Mccabe features are explained in \fig{mccabe}.}\label{fig:attr}
\end{figure}


\begin{figure}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
The Halstead features were proposed by Maurice Halstead in 1977. He
argued that modules
that are hard to read are more likely to be fault
prone~\cite{halstead77}. Halstead estimates reading complexity by
counting the number of operators and operands in a module: see the
$h$ features of \fig{attr}.  These three raw $h$  Halstead
features were 
then used to compute the $H$: the eight derived Halstead features
using the equations shown in
\fig{attr}.  In between the raw and derived Halstead features are
certain intermediaries:
 \bi
\item
$\mu= \mu_1 + \mu_2$;
\item
minimum operator count: $\mu_1^*=2$;
\item
$\mu_2^*$  is the minimum operand count and equals the number of module parameters.
\ei\\\hline
\end{tabular}
\caption{Notes on the Halstead features}\label{fig:halstead}
\end{figure}
 

\begin{figure}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
An alternative to the Halstead features of \fig{halstead}
are the complexity features proposed by Thomas McCabe in 1976.
Unlike Halstead,
McCabe
 argued that the complexity of
pathways {\em between} module symbols are more insightful than just a count of the symbols~\cite{mccabe76}. 

The first three lines of \fig{attr}
shows McCabe three
main features for this pathway complexity. These are defined as follows.
\bi
\item
A
module is said to have a {\em flow graph}; i.e. a directed graph where each node corresponds to a
program statement, and each arc indicates the flow of control from
one statement to another.
\item
The {\em cyclomatic complexity} of a module is   $v(G) = e - n +
2$ where $G$ is a program's flow graph, $e$ is the number of arcs in
the flow graph, and $n$ is the number of nodes in the
flow graph~\cite{fenton96}.
\item
The {\em essential complexity}, ($ev(G)$) of a module is  the extent to which a
flow graph can be ``reduced'' by decomposing all the subflowgraphs
of $G$ that are {\em D-structured primes}
(also sometimes referred to as ``proper one-entry
one-exit subflowgraphs''~\cite{fenton96}). $ev(G) = v(G) - m$ where $m$ is the number of subflowgraphs of
$G$ that are D-structured primes~\cite{fenton96}.
\item
Finally, the
{\em design complexity}   ($iv(G)$) of a module is the cyclomatic complexity of a
module's reduced flow graph.
\ei\\\hline
\end{tabular}
\caption{Notes on the McCabe features}\label{fig:mccabe}
\end{figure}
 
Our data are taken from software developed
in
different geographical locations across North America (NASA) and Turkey (SOFTLAB).
Within a system, the sub-systems
shared some common code base but did not pass personnel or code
between sub-systems. 
While NASA and SOFTLAB are one single {\em source} of data, there are several projects within each source.
For example, NASA 
is really an umbrella organization  used to co-ordinate and fund
a large and diverse set of projects:
\bi
\item
The NASA data was
collected from across the United States over a 
a period of five years from numerous
NASA contractors working at different geographical centers.  
\item
These projects
represent a wide
array of projects, including satellite instrumentation, ground control
systems and partial flight control modules (i.e. Attitude Control).  
\item
The
data sets also represent a wide range of code reuse: some of the projects
are 100\% new, and some are modifications to previously deployed code. 
\ei
That is, even if we explore just (say) the NASA data sets, we can still
examine issues of cross- vs within- company data use.

The external validity of generalizing from NASA examples
has been discussed elsewhere~\cite{me07b}. In summary, 
NASA uses 
contractors who are contractually obliged (ISO-9O01) to demonstrate
their understanding and usage of current industrial best
practices. These contractors service many other industries;
for example, Rockwell-Collins builds systems for many government and
commercial organizations.  For these reasons, other noted researchers
such as Basili, Zelkowitz, et al.~\cite{basili02} have argued that
conclusions from NASA data are relevant to the general software
engineering industry.

Nevertheless, it is always wise to test claims of external validity. Hence:
\bi
\item
The SOFTLAB data are, initially,  in reserve.
 Our first three experiments
will be based solely on the aerospace applications found in the NASA data.
\item
Our last experiment will check if the  SOFTLAB data exhibit the same pattern as the 
NASA data.
\ei
The results are shown below.	

\section{Experiment \#1: WC-vs-CC}
\subsection{Design}
Our first WC-vs-CC experiments repeated the following procedure
for all 7 NASA tables of \fig{data}.
For each table,
test sets were built from
10\% of the rows, selected at random.  Defect
predictors were then learned from:
\bi
\item Treatment 1 (CC): all rows from the other 11 tables.
\item Treatment 2 (WC):  just the other 90\% rows of this table;
\ei
Most of the \fig{data} tables come from systems written in ``C/C++''
but at
least one of the systems was written in JAVA.
For cross-company data, an industrial practitioner may not have access to 
detailed meta-knowledge (e.g. whether it was developed in ``C'' or JAVA).
They may only be aware
that data, from an unknown
source, are available for download from a certain url.
To replicate that scenario,  we will make no
use of our meta-knowledge about \fig{data}. As we shall see, 
a clear stable effect will occur
across all the tables, regardless of (say) the implementation language.

In order to control for   {\em order effects} (where the 
learned theory is unduly affected by the order of the examples)
our procedure was repeated 20 times,
randomizing the order of the rows in the table each time. Therefore, randomly selected training and test sets did not stay constant throughout the repetitions,  
In all, we ran 280 experiments to compare WC-vs-CC:
\[\left(\mbox{2 treatments}\right) * \left(\mbox{20 randomized orderings}\right) * \left(\mbox{7 tables}\right)\]

All the numeric distributions in the \fig{data}
data
are exponential. A ``log-filter'' replaces all numerics $N$ with $log(N)$.
This 
spreads out exponential
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
This ``spreading'' can significantly
improve the effectiveness of data mining~\cite{me07b}.

In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem work best for the 
\fig{data} data~\cite{me07b}
(for notes on Bayes classifiers, see \fig{bayes}).
Since that study, we have tried to find  better data mining
algorithms for defect prediction. To date, we have failed. Our
recent (as yet, unpublished) experiments have found no additional
statistically significant improvement from the application of the
following data mining methods: logistic regression; average
one-dependence estimators; under- or
over-sampling~\cite{drummond03}, random forests,
RIPPER~\cite{cohen95r}, decision tree learning with 
J48~\cite{quinlan92}, rule learning with OneR~\cite{holte93}
and bagging~\cite{brieman96}. Only boosting~\cite{FreSch97} on
discretized data offers a statistically better result than a Bayes
classifier. However, we cannot recommend boosting: the median
improvement is quite negligible and  boosting is orders of
magnitude slower than our simple Bayes classifier.

Other researchers have also failed to improve our results. For example, Lessmann et al. 
investigate the statistical difference of the results between 19 learners, including naive Bayes, on the same datasets~\cite{lessmann09}. \fig{lessmann} shows that the simple Bayesian method discussed above ties in
first place along with 15 other methods.


\begin{figure}
\begin{center}
\includegraphics[width=3in]{lessmann.pdf}
\end{center}
\caption{Range of ranks seen in 19 learners building defects predictors when, 
10 times, a random 66\% selection of the data are used for training and
the remaining data are used for testing. In ranked data, values from one
method are replaced by their rank in space of all sorted values (so smaller ranks
mean better performance). In this case, the performance value
was area under the false positive vs true positive curve (and larger
values are better). Vertical lines divide the results into regions where
the results are statistically similar. For example, all the methods
whose top ranks are 4 to 12 are statistically insignificantly different.
From~\cite{lessmann09}.
}\label{fig:lessmann}
\end{figure}



\begin{figure}[!t]
\footnotesize
\begin{tabular}{|p{.95\linewidth}|}\hline
Bayesian classifiers
offers a relationship between  fragments of evidence $E_i$,
a prior probability for
a class $P(H)$, and a posteriori  probability
$P(H|E)$. Note that the likelihood $P(H|E)$ is approximated by the product term due to the i.i.d. (independent and identically distributed) assumption: 
\begin{equation}\label{eq:bayes}
P(H|E)= \left(\prod_iP(E_i|H)\right) \frac{P(H)}{P(E)}
\end{equation}
For example, in our data sets, there are two hypotheses: modules are either
defective or not; i.e. \mbox{$H\in \left\{ defective,nonDefective \right\}$}.
Also, if a particular module has $numberofSymbols=27$ and
$LOC=40$ and was previously classified
as ``defective'' then
\[
\begin{array}{lcl}
E_1&:& numberOfSymbols=27\\
E_2& :& LOC=40\\
H&:&  defective
\end{array}
\]
When building defect predictors,
the
posterior probability of each class (``defective'' or ``defect-free'')
is calculated, given the features
extracted from a 
module.
So, if a data set has 100 modules and 25 of them are faulty, then:
\[P(defective)=0.25\]
When testing new data, a module is assigned to the class
with the higher probability, calculated from \eq{bayes}.

For
numeric features, 
a feature's mean $\mu$ and standard deviation $\sigma$ is used in
a
Gaussian  probability
function~\cite{witten05}:
\[f(x) =
1/(\sqrt{2\pi}\sigma)e^{-\frac{(x-\mu)^2}{2\sigma^2}}\] 

Simple  Bayes classifiers are often called ``naive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(frequency
counts are required only for each feature),
 it is possible
that
correlated events are missed by this ``naive'' approach.
Potentially, this is a significant problem for our kinds of data sets where
the static code measures are highly correlated (e.g. the number of
symbols in a module increases linearly with module lines of code). 

In a previous work, we have investigated variants of Bayesian models considering this issue and observed that the independence assumption of naive Bayes is safe for the Nasa datasets~\cite{turhan07qsic}. In another work, we have also performed extensive feature subset selection (FSS) experiments, where similar prediction performances were obtained using small subsets of features selected by Information Gain~\cite{me07b}. However, we did not observe an increase in the prediction performances when we used whole available data. 
Furthermore, 
Domingos and Pazzani show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, seemingly naive Bayes classifiers perform as well
as other seemingly more sophisticated schemes (e.g. see Table~1 in~\cite{domingos97optimality}).

The Domingos and Pazzani result
also explains our prior experiments where naive Bayes did not perform
worse than other learners that continually re-sample the data for dependent
instances (e.g. decision-tree learners that recurse
on each ``split'' of the data~\cite{quinlan92}).\\\hline
\end{tabular}
\caption{About Bayes classifiers.}\label{fig:bayes}
\end{figure}


Data mining effectiveness was measured using $pd$, $pf$ and $balance$~\cite{me07b,me07e}.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% detection ($pd=1$).

Other measures such as $accuracy$  and $precision$
were not used since, as shown in \fig{data}, the percent of defective examples
in our tables was usually very small (median value around 8\%).
Accuracy and precision
are poor indicators
of performance for data where the target class is so rare
(for more on this issue, see~\cite{me07b,me07e}). 

The WC and CC results were visualized using
{\em
quartile charts}.  To generate these charts, the performance
deltas for some treatment
are sorted to isolate the median and the lower and upper quartile of numbers.
For example:

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In our quartile charts,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the 50\% percentile value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
The Mann-Whitney U test~\cite{mann47} was used to test
for statistical difference between treatments.
This non-parametric test replaces (e.g.) $pd$ values with their
rank inside the population of all sorted $pd$ values. 
Such non-parametric tests are recommended in data mining since many of the performance
distributions are non-Gaussian~\cite{demsar06}. 

We report one deviation from our prior procedure. The tables of data 
come from different sources and, hence,
have different features. For this study, all the tables were pruned such
that they only contained features that appear in all the seven tables.


\subsection{Results from Experiment \#1}

\fig{ccwsnum} shows the $\{pd,pf\}$ quartile charts for CC vs WC data averaged over seven NASA datasets.
The trend is very clear: CC data dramatically increases {\em both}
the probability of detection and the probability of false alarms. The $pd$
results are particularly striking. 

For cross-company data: 
\bi
\item 50\% of the pd values are at or above 97\% 
\item 75\% of the pd values are at or above 83\%; 
\item And all the pd values are at or over 50\%.
\ei

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&      50&  83&  97&  100& 100& \boxplot{50}{33}{97}{100}{ 0}\\
    & WC&      17&  63&  75&   82& 100& \boxplot{17}{46}{75}{82}{18}\\\hline
 pf & CC&      14&  53&  64&   91& 100& \boxplot{14}{39}{64}{91}{ 9}\\
    & WC&       0&  24&  29&   36&  73& \boxplot{ 0}{24}{29}{36}{36}\\
\hline\end{tabular}

}
\end{center}
\caption{Experiment \#1 results averaged over seven NASA tables.
Numeric results on left; quartile charts on right. ``Q1'' and ``Q3'' denote
the 25\% and 75\% percentile points (respectively). 
The upper quartile of the first row is  not visible  since
it runs from 100\% to 100\%; i.e. it has zero length.}\label{fig:ccwsnum}
\end{figure}


\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{|r|c|c|p{2cm}|l|}\hline
&$pd$    & $pf$ & &  \\
group & $WC \rightarrow CC$    &  $WC \rightarrow CC$& tables & $|tables|$ \\\hline\hline
a& increased               &        increased & CM1 KC1 KC2 MC2 MW1 PC1  & 6\\\hline
b& same                    & same             & KC3& 1\\\hline
\end{tabular}}
\end{center}
\caption{Summary of
U-test results (95\% confidence): moving from WC to CC.
For full results, see~\fig{ccwsprojs}. }\label{fig:ccwc}
\end{figure}



By way of comparison, recall from the above that
our previous result had an  average $pd$ of $71\%$~\cite{me07b}.

To the best of our knowledge, \fig{ccwsnum} are the largest $pd$ values ever reported
from these data.
However, these very high $pd$ values come at some considerable cost. Note in \fig{ccwsnum}
that the median false alarm
rate has changed from 29\% (with WC) to 64\% (with CC) and the maximum $pf$ rate
now reaches 100\%.

We explain these increases in $pd, pf$ with the following 
{\em extraneous hypothesis}:
Using a large training set (e.g. seven of the tables in 
Figure 2) informs not only all the causes of errors, but also of 
numerous irrelevancies (e.g. applying statistics gathered from 
JAVA programs to ``C'' programs). Hence, 
large training sets 
increase the probability of detection (since there are more 
known sources of errors) as well as the probability of false 
alarms (since there are more {\em  extraneous factors} introduced to 
the analysis). We test the extraneous hypothesis in the next experiment.

\subsection{Sanity Checks on Experiment \#1}
This section explores 
threats to the external validity
of the Experiment\#1 conclusions. It can be skipped at first reading
of this paper.

Once a {\em general result} is defined (e.g. CC dramatically
increases both $pf$ and $pd$), it is good practice to check for 
specific exceptions
to that pattern. 
\fig{ccwc} shows a summary of results when U tests
were applied to
test results from each of the 7 tables,  {\em
in isolation} and \fig{ccwsprojs} shows the $\{pd,pf\}$ quartile charts for 
Experiment \#1 for each NASA table:
\bi
\item 
Usually  ($\frac{6}{7}$), the general pattern still holds
(see group $a$). 
\item
In one case (see group $b$), there was no difference in the results of the different
treatments. 
\ei
Overall, the {\em general result} 
holds in the majority of cases (i.e. $\frac{6}{7}$ ), which is not a 100\% internally consistent, however 
it is still a very clear effect.

\begin{figure*}[!t]

\begin{center}
\begin{tabular}{cc}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($cm1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       80& 100&  100&  100& 100& \boxplot{80}{20}{100}{100}{0}\\
    & WC&       40&  60&  80&   100& 100& \boxplot{40}{20}{80}{100}{0}\\\hline
 pf & CC&        87&  91&  96&   96& 98& \boxplot{87}{4}{96}{96}{2}\\
   & WC&        24&  27&  33&   38&  47& \boxplot{24}{3}{33}{38}{9}\\
\hline\end{tabular}


}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 88&  94&  94& 100& \boxplot{82}{6}{94}{94}{6}\\
    & WC&       64&  73&  82&   85& 97& \boxplot{64}{9}{82}{85}{12}\\\hline
 pf & CC&        47&  49&  51&   53& 57& \boxplot{47}{2}{51}{53}{4}\\
   & WC&        27&  34&  36&   38&  40& \boxplot{27}{7}{36}{38}{2}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 91&  91&  100& 100& \boxplot{82}{9}{91}{100}{0}\\
    & WC&       55&  73&  82&   91& 100& \boxplot{55}{18}{82}{91}{9}\\\hline
 pf & CC&        57&  62&  64&   74& 81& \boxplot{57}{5}{64}{74}{7}\\
   & WC&        14&  24&  31&   33&  45& \boxplot{14}{10}{31}{33}{12}\\
\hline\end{tabular}


}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       60& 80&  80&  80& 100& \boxplot{60}{20}{80}{80}{20}\\
    & WC&       40&  60&  80&   80& 100& \boxplot{40}{20}{80}{80}{20}\\\hline
 pf & CC&        14&  19&  24&   31& 38& \boxplot{14}{5}{24}{31}{7}\\
   & WC&        10&  17&  21&   26&  36& \boxplot{10}{7}{21}{26}{10}\\
\hline\end{tabular}


}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       50& 67&  83&  100& 100& \boxplot{50}{17}{83}{100}{0}\\
    & WC&       17&  33&  67&   67& 83& \boxplot{17}{10}{67}{67}{18}\\\hline
 pf & CC&        55&  64&  73&   73& 100& \boxplot{55}{9}{73}{73}{27}\\
   & WC&        0&  27&  36&   45&  73& \boxplot{0}{27}{36}{45}{28}\\
\hline\end{tabular}

}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mw1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       75& 75&  100&  100& 100& \boxplot{75}{0}{100}{100}{0}\\
    & WC&       25&  50&  75&   75& 100& \boxplot{25}{25}{75}{75}{25}\\\hline
 pf & CC&        50&  55&  63&   66& 82& \boxplot{50}{5}{63}{66}{16}\\
   & WC&        13&  18&  21&   29&  37& \boxplot{13}{5}{21}{29}{8}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       88& 100&  100&  100& 100& \boxplot{88}{12}{100}{100}{0}\\
    & WC&       38&  63&  63&   75& 88& \boxplot{38}{25}{63}{75}{13}\\\hline
 pf & CC&        89&  92&  93&   95& 99& \boxplot{89}{3}{93}{95}{4}\\
   & WC&        17&  25&  27&   30&  34& \boxplot{17}{8}{27}{30}{4}\\
\hline\end{tabular}

}&
\end{tabular}
\end{center}
\caption{Project-wise Experiment \#1 results for NASA tables.}\label{fig:ccwsprojs}
\end{figure*}




\subsection{Discussion of Experiment \#1}
When practitioners use defect predictors with high false alarm rates (e.g.the 64\% reported above),
they must allocate a large portion of their debugging budget to 
the unfruitful exploration of erroneous alarms.

In our industrial work, we have sometimes seen 
several situations
where detectors with high false alarms  are useful:\begin{itemize}\item{\em
When the cost of missing the target is prohibitively expensive.} In
mission critical or security  applications, the goal of
100\% detection
may be demanded in all situations, regardless of the  cost of chasing false alarms.\item
{\em When only a small  fraction
the data is returned.}
Hayes, Dekhtyar,
\& Sundaram call this fraction $selectivity$ and offer an extensive
discussion of the merits of this measure~\cite{hayes06}.  \item {\em
When there is little or no cost in checking false alarms.}  For
example,  a detector we have found useful in industrial settings
is to check modules  where \[\frac{lines\; of\; comments}{lines\;
of\; code} > 0.25\]This detector triggers on complex functions that
programmers comment extensively, instead of splitting up into
smaller, more maintainable, functions.
This detector suffers from high false alarms-
it often triggers on well-written functions with detailed comments.
On the other hand,
it may be fast
and simple for a human agent to inspect  the identified modules and
discern which
ones were well-written and which were over-commented to compensate for
being
badly coded. \end{itemize}

Having said that, we observe that most sites do {\em not} accept
false alarm rates as high as 64\%. Therefore, the conditions under which the
benefits of CC data (high probabilities of detection) outweigh their high
costs (high false alarm rates) are quite rare.


In summary,
 for most software applications, very high  $pf$ rates like the CC results
of \fig{ccwsnum} make the predictors impractical to use.

\section{Experiment \#2: NN-Filtered CC }
The results of the first experiment limits the use of CC data to a limited domain (i.e. mission critical) and may look discouraging at first glance. In the first experiment we explained our observations with the {\em extraneous hypothesis}, that the results are affected by the irrelevancies in CC data. In effort estimation literature, Kitchenham {\em et al.} argues the same concept as the heterogeneity of CC data ~\cite{kitch07}. Further, Premraj and Zimmermann builds business specific cost models to avoid such heterogeneity~\cite{Premraj:2007p1626}. However, the effects of more homogeneous data for cost estimation are observed in a range of patterns that vary in different companies. (Recall that we explain this with the ambiguity in cost estimation features). Here, we will perform similar analysis for defect prediction to test the {\em extraneous hypothesis}.

\subsection{Design}
In this experiment, we try to construct more homogeneous {\em defect} datasets from CC data. For this purpose we use a simple sampling method (i.e. NN).
Our idea behind sampling is to collect similar instances together in order to construct a learning set that is homogeneous with the validation set. We simply use the k-Nearest Neighbor (k-NN) method to measure the similarity between the validation set and the training candidates. The expected outcome of the sampling part is to obtain a subset of available CC data that shows similar characteristics to the local code culture.
 

We first calculate the pairwise distances between the validation set and the {\em candidate} training set samples (i.e. all CC data). Let $N$ be the number of validation set size. For each validation sample, we pick its $k =10$ nearest neighbors from candidate training set. Then we come up with a total of  $10\times N$ similar samples. Note that these $10\times N$ samples may not be unique (i.e. a single data sample can be a nearest neighbor of many data samples in the validation set). Using only unique ones, we form the training set and use a random $90\%$ of it for training a predictor. We repeat the last step $100$ times.    
\subsection{Results}


\begin{figure}
\renewcommand{\baselinestretch}{0.5}
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

CM1 &1&   CC&98&98&98&98&98&\boxplot{  98.0}{0}{  98.0}{  98.0}{0}\\
&2& NN&76&82&82&84&84&\boxplot{  76.0}{6}{  82.0}{  84.0}{0}\\
&3&   WC&20&60&80&80&100&\boxplot{  20.0}{40}{  80.0}{  80.0}{20}\\\hline


MW1&1&   CC&90&90&90&90&90&\boxplot{  90.0}{0}{  90.0}{  90.0}{0}\\
&2& NN&68&68&68&68&71&\boxplot{  68.0}{0}{  68.0}{  68.0}{3}\\
&2&   WC&0&50&50&75&100&\boxplot{   0.0}{50}{  50.0}{  75.0}{25}\\\hline

PC1&1&   CC&99&99&99&99&99&\boxplot{  99.0}{0}{  99.0}{  99.0}{0}\\
&2& NN&74&77&77&77&78&\boxplot{  74.0}{3}{  77.0}{  77.0}{1}\\
&3&   WC&38&62&62&75&100&\boxplot{  38.0}{24}{  62.0}{  75.0}{25}\\\hline


\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
 Experiment \#2 PD results where
$NN_{pd} \ge WC_{pd}$. 
Rankings computed via Mann-Whitney (95\% 
confidence) comparing each row to all other rows.}\label{fig:pd1}
\end{figure}
\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

KC1 &1&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\
&2&   WC&64&76&82&85&94&\boxplot{  64.0}{12}{  82.0}{  85.0}{9}\\
&3& NN&60&64&65&66&69&\boxplot{  60.0}{4}{  65.0}{  66.0}{3}\\\hline

KC2&1&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\
&2&   WC&45&73&82&91&100&\boxplot{  45.0}{28}{  82.0}{  91.0}{9}\\
&2& NN&77&78&79&79&80&\boxplot{  77.0}{1}{  79.0}{  79.0}{1}\\\hline

KC3&1&   CC&81&81&81&84&84&\boxplot{  81.0}{0}{  81.0}{  84.0}{0}\\
&2&   WC&20&60&80&100&100&\boxplot{  20.0}{40}{  80.0}{ 100.0}{0}\\
&3& NN&60&63&65&67&70&\boxplot{  60.0}{3}{  65.0}{  67.0}{3}\\\hline

MC2&1&   CC&83&83&83&83&85&\boxplot{  83.0}{0}{  83.0}{  83.0}{2}\\
&2&   WC&17&50&67&83&100&\boxplot{  17.0}{33}{  67.0}{  83.0}{17}\\
&3& NN&56&56&56&56&58&\boxplot{  56.0}{0}{  56.0}{  56.0}{2}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
Experiment \#2 PD results where
$NN_{pd} < WC_{pd}$. 
}\label{fig:pd2}
\end{figure}
\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

KC1&1& NN&22&23&24&25&27&\boxplot{  22.0}{1}{  24.0}{  25.0}{2}\\
&2&   WC&26&32&35&37&43&\boxplot{  26.0}{6}{  35.0}{  37.0}{6}\\
&3&   CC&59&60&60&60&60&\boxplot{  59.0}{1}{  60.0}{  60.0}{0}\\\hline

KC2&1& NN&24&25&25&25&27&\boxplot{  24.0}{1}{  25.0}{  25.0}{2}\\
&1&   WC&10&21&26&31&40&\boxplot{  10.0}{11}{  26.0}{  31.0}{9}\\
&2&   CC&67&67&67&67&67&\boxplot{  67.0}{0}{  67.0}{  67.0}{0}\\\hline

KC3&1& NN&17&18&18&19&20&\boxplot{  17.0}{1}{  18.0}{  19.0}{1}\\
&2&   WC&7&17&21&26&31&\boxplot{   7.0}{10}{  21.0}{  26.0}{5}\\
&3&   CC&26&27&27&27&27&\boxplot{  26.0}{1}{  27.0}{  27.0}{0}\\\hline

MC2&1& NN&29&30&31&32&35&\boxplot{  29.0}{1}{  31.0}{  32.0}{3}\\
&2&   WC&0&27&36&45&73&\boxplot{   0.0}{27}{  36.0}{  45.0}{28}\\
&3&   CC&71&71&71&71&71&\boxplot{  71.0}{0}{  71.0}{  71.0}{0}\\\hline

\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{Experiments \#2 PF results 
where $NN_{pf} \le WC_{pf}$.}\label{fig:pf1}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline
CM1&1&   WC&16&29&33&38&49&\boxplot{  16.0}{13}{  33.0}{  38.0}{11}\\
&2& NN&40&43&44&45&46&\boxplot{  40.0}{3}{  44.0}{  45.0}{1}\\
&3&   CC&90&91&91&91&93&\boxplot{  90.0}{1}{  91.0}{  91.0}{2}\\\hline

MW1&1&   WC&8&21&26&29&47&\boxplot{   8.0}{13}{  26.0}{  29.0}{18}\\
&2& NN&30&32&33&33&36&\boxplot{  30.0}{2}{  33.0}{  33.0}{3}\\
&3&   CC&67&68&68&69&70&\boxplot{  67.0}{1}{  68.0}{  69.0}{1}\\\hline

PC1&1&   WC&16&24&28&31&40&\boxplot{  16.0}{8}{  28.0}{  31.0}{9}\\
&2& NN&45&48&48&49&53&\boxplot{  45.0}{3}{  48.0}{  49.0}{4}\\
&3&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\\hline

\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{  Experiment \#2 PF results where $NN_{pf} > WC_{pf}$.  }\label{fig:pf2}
\end{figure}

The ideal result would be that NN can be used as a surrogate for local WC data.
This, in turn, would mean that developers
could avoid the tedium and expense of local data collection.
If NN out-performed WC then two observations would appear:
\begin{itemize}
\item {\em Observation1:} NN would have $pd$ values above or equal to WC's $pd$.
The examples displaying $Observation1$ are shown in \fig{pd1}.
\item {\em Observation2:} NN would have $pf$ values below or equal to WC's $pf$.
The examples displaying $Observation2$ are shown in \fig{pf1}.
\end{itemize}
Note that the conjunction of
$Observation1$ and $Observation2$ is uncommon.
In fact, our results
suggest that $Observation1$ and $Observation2$ are somewhat mutually exclusive:
\begin{itemize}
\item
As shown in Figures \ref{fig:pd1} and \ref{fig:pf2}, the examples where $NN$ increases
the probability of detection are also those where it increases the probability of false alarms.
\end{itemize}
Hence, we cannot recommend NN as a replacement for WC.

Nevertheless, if local WC data is unavailable, 
then we would recommend processing foreign CC data with NN.

In all cases, NN dramatically reduces the high false alarm rates associated with
the use of cross-company data.
Often that reduction  halves
the false alarm range. For example, in MW1, the false alarm rate drops
between CC to NN from 68\% to 33\%.




\subsection{Discussions}
In raw cross company models we see that the false alarm rates substantially increase compared to the within company models. Our new experiment shows that NN sampling removes the increased false alarm rates. Now we argue that, using NN sampling instead of random sampling, helps choosing training examples that are similar to problem at hand. Thus, the irrelevant information in non-similar examples are avoided. However, this also removes the rich sample base and yields a slight decrease in detection rates. Mann-Whitney tests reveal that NN sampling is 
\bi
\item far better than random sampling cross company data, 
\item and still worse than using within company data.
\ei

NN Sampling picks training examples that are similar to the examples about which we want to make predictions. Without introducing irrelevant information, NN sampling populates a set that has the same characteristics with the problem. That means, the likelihood of implementing similar software modules and having similar defects in these modules is high. Thus, NN sampling simulates that the selected cross company data are coming from within the company. Considering the {\em extraneous hypothesis}, NN sampling introduces homogeneity whereas random sampling introduces heterogeneity to cross company data.

Combining these results, if a company lacks local data,
we would suggest a two-phase approach.
In phase one, that organization uses imported CC data, filtered via NN.
Also, during phase one, the organization should start a data collection program to collect
static code attributes. Phase two commences when there is {\em enough} local WC data to learn
defect predictors. During phase two, the organization would switch to new defect predictors
learned from the WC data. 

Our next experiment is, therefore, designed to determine the number of examples required to build defect predictors from WC data.   

\section{Experiment \#3: Incremental WC }
\begin{figure*}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{incrementalcm1.pdf}&
\includegraphics[angle=90,width=2in]{incrementalkc1.pdf}&
\includegraphics[angle=90,width=2in]{incrementalkc2.pdf}\\
\includegraphics[angle=90,width=2in]{incrementalkc3.pdf}&
\includegraphics[angle=90,width=2in]{incrementalmc2.pdf}&
\includegraphics[angle=90,width=2in]{incrementalmw1.pdf}\\
\includegraphics[angle=90,width=2in]{incrementalpc1.pdf}&

\end{tabular}
\end{center}
\caption{Results from experiment \#3. Training set size grows in units of 100 examples,
 moving left to right
over the x-axis. The MC2 results only appear at the maximum x-value since MC2 
has less than 200 examples.
}\label{fig:inc}

\end{figure*}

In the introduction, the case was made that CC data are attractive to
organizations if it avoids a time-consuming or expensive local
data collection program.  Also, our results of Experiment \#1 and \#2 reveals the necessity for WC data.  In this section,  we will show that adequate defect predictors can be learned from very
small samples of WC data.

\subsection{Design}
A curious aspect of the above results is that defect predictors
were learned using only a handful of defective modules.
For example, consider a 90\%/10\% train/test split on
$pc1$ with 1,109 modules, only 6.94\% of which are defective.
On average, the training set will only contain \mbox{$1109 * 0.9 * 6.94/100=69$}
defective modules. Despite this, $pc1$ yields an adequate median $\{pd,pf\}$ results
of $\{63,27\}$\%. 

Experiment \#3 was therefore designed to check how {\em little} data are required
to learn defect predictors. Experiment \#3 was essentially the same
as the first experiment, but
without treatment \#1 (the
cross-company study). Instead,
experiment \#3
took the 7 example tables of \fig{data}
and 
learned predictors using:
\bi
\item Treatment 3 (reduced WC):  a randomly selected
subset of up to 90\% of the rows.
\ei
After randomizing the order of the rows, training sets
were built using just the first 100,200,300,\ldots rows in the tables.
After training, the learned theory was applied to the remaining rows
not used in training.

Experiment \#1 only used the features found in all tables of data.
For this experiment, we imposed no such restrictions and used whatever features
were available in each data set.
\subsection{Results from Experiment \#3}
Recall that
\eq{bal} defined ``balance'' to be a combination of $\{pd,pf\}$
that decreases if $pd$ decreases or $pf$ increases.
As shown in \fig{inc}, there was very little change in balanced 
performance  after learning from 100,200,300,... examples.
Indeed, there is some evidence that
learning from larger training sets had detrimental effects: the more
training data, the larger the variance in the performance of the learned predictor.
Observe how, in $kc1$ and $pc1$, as the training set size increases (moving right along the x-axis)
the dots showing the balance performance start spreading out.

The Mann-Whitney U test was applied to check the visual trends seen in \fig{inc}.
For each table, all results from training sets of size 100,200,300\ldots were
compared to all other results from the same table. The issue was ``how much data are enough?''
 i.e.
what is the $min$imum
training set size that never lost to other training set of a larger size. Usually, that 
$min$ value 
was quite small:

\bi
\item In five tables $\{cm1,kc2,kc3,mc2,mw1\}$, $min=100$;
\item In  \{$kc1,pc1$\}, $min=\{200,300\}$ instances, respectively.
\ei


We explain the experiment \#3 results as follows.
These experiments used simplistic static code features such as lines of code, number of
unique symbols in the module, etc.  
Such simplistic static code features are hardly a complete characterization of
the internals of a function. Recall Fenton's example where
{\em the same} functionality is achieved using {\em different}
programming language constructs resulting in {\em different} static
measurements for that module~\cite{fenton97}.  
We would characterize such static code features as having 
{\em limited information content}~\cite{me08d}. 
Limited content
is soon exhausted by repeated sampling.
Hence, such simple features reveal all they can reveal after
a small sample.

\begin{figure}
\begin{center} 
\includegraphics[width=2.4in]{plateaudots.pdf}
\includegraphics[width=2.4in]{plateaudotsC.pdf} 
\end{center}
\caption{Y-axis shows plateau point after learning from data sets
that have up to $X$ examples (from~\cite{orrego04}).
The left plot shows results from using
Naive Bayes (nbk) or a decision tree learner (j48)~\cite{quinlan92}
to predict for discrete classes. Right plot shows results from
using linear regression (lsr) or model trees (m5)~\cite{quinlan92b}
to learn predictors for continuous classes. In this study, data
sets were drawn from the UC Irvine data repository~\cite{Blake+Merz:1998}.}\label{fig:plateau} 
\end{figure}

\subsection{Sanity Checks on Experiment \#3}
This section checks for precedents on the Experiment \#3 results
and can be skipped at first reading of this paper.

There is also some evidence that the  results of Experiment 3 (that
performance improvements stop after a few hundred examples) have
been seen previously in the data mining literature (caveat: to the
best of our knowledge, this is the first report of this effect in the
defect prediction literature): \bi \item In their discussion on how
to best handle numeric features, Langley and John offers plots of
the accuracy of Naive Bayes classifiers after learning on 10,20,40,..200
examples. In those plots, there is little change in performance
after 100 instances~\cite{john95}.  \item Orrego~\cite{orrego04}
applied four data miners (including Naive Bayes) to  20 data sets
to find the {\em plateau point}: i.e. the point after which there
was little net change in the performance of the data miner.  To
find the plateau point, Oreggo used t-tests to compare the results
of learning from $Y$ or $Y+\Delta$ examples. If, in  a 10-way
cross-validation, there was no statistical difference between $Y$
and $Y+\Delta$, the plateau point was set to $Y$.  As shown in
\fig{plateau}, many of those plateaus were found at $Y\le100$ and
most were found at $Y\le200$. Note that these plateau sizes are
consistent with the results of Experiment 3.  \ei 

\subsection{Discussion of Experiment \#3}
In the majority case, predictors learned from as little as one hundred
examples perform as well as predictors learned from many more examples.
This suggests that the effort associated with learning defect predictors
from within-company data may not be overly large.
For example, \fig{estimate} estimates that the effort required to build and test
100 modules may be as little as 2.4 to 3.7 months.  Considering the proposed two-phase approach, these are the optimistic waiting times before switching to phase two (i.e. local predictors from WC data).

\begin{figure}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
100  modules may take as little as two to four months to construct. This estimate
was generated as follows:
\bi
\item
In the $cm1$ data base, the median module size is 17 lines. 100 randomly
selected modules would 
therefore use 1700 LOC.
\item
To generate an effort estimate for these modules,
we used the on-line COCOMO~\cite{boehm00a} effort estimator (\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/expert_cocomo2000.html}). Estimates were generated
assuming 1700 LOC and
the required reliability varying from very low to very high.
\item
The resulting estimates ranged from
between 2.4 and 3.7 person months to build and test those modules.
\ei\\\hline
\end{tabular}
\caption{An estimate of the effort required to build and test 100 modules.}\label{fig:estimate}
\end{figure}

\section{Experiment Replication}
\label{sec:exp4}
Experiments \# 1 to \# 3 were based on NASA data.
In order to search evidence for the external validity of the conclusions of those experiments,
we obtained new data sets. Those data sets were deliberately chosen to be as
far removed as possible from American aerospace software applications.
Using our connections with the Turkish software industry, we collected
three data sets in the format of \fig{attr} from a Turkish white- goods manufacturer. The new datasets (\{ar3, ar4, ar5\}), marked
as ``SOFTLAB'' in \fig{data},
are the 
controller software for:
\bi
\item
A washing machine;
\item
 A dishwasher;
\item And a
refrigerator. 
\ei
Note that this software was developed via methods that are both
culturally and organizationally different to NASA aerospace software. Turkish domestic appliances company software are developed by a small team of 2-3 people. The development is carried out in an ad-hoc, informal way rather than formal, process oriented approach in NASA. Furthermore, the company is a profit and revenue driven commercial organization, whereas NASA is a cost driven government entity. 

For each SOFTLAB data set, we follow the same procedure as in Experiments \# 1 and \#2; i.e.
10\% of the rows of each data set are selected at random for constructing test sets and then defect
predictors are learned from:
\bi
\item Treatment 1 (CC): all rows from 7 NASA tables.
\item Treatment 2 (WC):  random 90\% rows of remaining SOFTLAB tables\footnote{In order to reflect the use in practice, we do not use the remaining 90\% of the same project for training, we rather use a random 90\% of data from other projects. Note that all WC experiments in this paper reflects within-company, not within project simulations. Since SOFTLAB data are collected from a single company, learning a predictor on some projects and to test it on a different one does not violate within company simulation.};
\item Treatment 3 (NN): {\em similar} rows from 7 NASA tables.
\ei 

The SOFTLAB tables include 29 static code features, 19 of which
 common with the NASA tables. In order to simplify the comparison between these new
data and Experiment \#1 and \#2, 
 we used only these shared attributes in our CC experiments. 
On the other hand we use all available attributes in WC experiments for SOFTLAB tables. 
In the following external validity experiment, we treated each NASA table as 
cross- company data for SOFTLAB tables. 

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Average results on SOFTLAB data}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  95&  100& 100& \boxplot{88}{0}{95}{100}{0}\\
    & WC&       35&  40&  88&   100& 100& \boxplot{35}{5}{88}{100}{0}\\\hline
 pf & CC&       52&  59&  65&   68& 68& \boxplot{52}{7}{65}{68}{0}\\
    & WC&        3&  5&  29&   40&  42& \boxplot{3}{2}{29}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  88&  88& 88& \boxplot{88}{0}{88}{88}{0}\\
    & WC&       88&  88&  88&   88& 88& \boxplot{88}{0}{88}{88}{0}\\\hline
 pf & CC&       62&  65&  65&   65& 65& \boxplot{62}{3}{65}{65}{0}\\
    & WC&        40&  40&  40&   40&  42& \boxplot{40}{0}{40}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar4$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       95&  95&  95&  95& 95& \boxplot{95}{0}{95}{95}{0}\\
    & WC&       35&  40&  40&   40& 40& \boxplot{35}{5}{40}{40}{0}\\\hline
 pf & CC&       52&  55&  56&   59& 60& \boxplot{52}{3}{56}{59}{1}\\
    & WC&        3&  3&  3&   5&  5& \boxplot{3}{0}{3}{5}{0}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar5$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       100&  100&  100&  100& 100& \boxplot{100}{0}{100}{100}{0}\\
    & WC&       88&  100&  100&   100& 100& \boxplot{88}{12}{100}{100}{0}\\\hline
 pf & CC&       57&  68&  68&   68& 68& \boxplot{57}{11}{68}{68}{0}\\
    & WC&        29&  29&  29&   29&  29& \boxplot{29}{0}{29}{29}{0}\\
\hline\end{tabular}
}\\
\end{tabular}
\end{center}
\caption{Experiment \#1 results for the SOFTLAB tables. Averaged and individual results are shown respectively.}\label{fig:ccwcturkey}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   CC&35&90&100&100&100&\boxplot{  35.0}{55}{ 100.0}{ 100.0}{0}\\
&2& NN&65&65&70&70&70&\boxplot{  65.0}{0}{  70.0}{  70.0}{0}\\
&3&   WC&35&40&40&40&45&\boxplot{  35.0}{5}{  40.0}{  40.0}{5}\\\hline

AR3&1&   CC&75&88&88&88&88&\boxplot{  75.0}{13}{  88.0}{  88.0}{0}\\
&1& NN&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\
&1&   WC&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\\hline

AR5&1&   CC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\
&1& NN&100&100&100&100&100&\boxplot{ 100.0}{0}{ 100.0}{ 100.0}{0}\\
&1&   WC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
 Experiment \#2 PD results for the SOFTLAB tables where
$NN_{pd} \ge WC_{pd}$. 
Rankings computed via Mann-Whitney (95\% 
confidence) comparing each row to all other rows.}\label{fig:arpd1}
\end{figure}



\begin{figure}[!t]
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR3&1& NN&38&38&38&38&40&\boxplot{  38.0}{0}{  38.0}{  38.0}{2}\\
&2&   WC&40&40&40&40&42&\boxplot{  40.0}{0}{  40.0}{  40.0}{2}\\
&3&   CC&38&55&56&60&80&\boxplot{  38.0}{17}{  56.0}{  60.0}{20}\\\hline

AR5&1& NN&21&25&25&25&32&\boxplot{  21.0}{4}{  25.0}{  25.0}{7}\\
&2&   WC&29&29&29&29&29&\boxplot{  29.0}{0}{  29.0}{  29.0}{0}\\
&3&   CC&29&46&50&50&79&\boxplot{  29.0}{17}{  50.0}{  50.0}{29}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{Experiments \#2 PF results for the SOFTLAB tables
where $NN_{pf} \le WC_{pf}$.}\label{fig:arpf1}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   WC&3&3&3&5&7&\boxplot{   3.0}{0}{   3.0}{   5.0}{2}\\
&2& NN&24&25&25&25&28&\boxplot{  24.0}{1}{  25.0}{  25.0}{3}\\
&3&   CC&6&47&62&67&78&\boxplot{   6.0}{41}{  62.0}{  67.0}{11}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{  Experiment \#2 PF results for the SOFTLAB tables where $NN_{pf} > WC_{pf}$.  }\label{fig:arpf2}
\end{figure}


Figure \ref{fig:ccwcturkey} shows the results:
\bi
\item
The {\em pd} values for CC treatment increases compared to WC treatment
with the cost of increased {\em pf}. 
\item
The CC treatment shifts \{Q1,
median\} of {\em pf} from \{5, 29\} to \{59, 65\}.  
\item
For the CC treatment:
\bi
\item 25\% of the $pd$ values are at 100\%.
\item 50\% of the $pd$ values are above 95\%
\item And all the $pd$ values are at or over 88\%.
\ei
\ei

These results also provide suggestive evidence for the generality of our conclusions for Experiment \# 3
(where we discuss that the minimum number of instances for training
a predictor is merely $100-200$ examples). Note that SOFTLAB tables
{\em ar3, ar4} and {\em ar5} have \{63,107, 36\} modules respectively,
with a total of 206 modules. Thus, WC results in Figure
\ref{fig:ccwcturkey} are achieved using a minimum of $(63+36) *
0.90 = 90$ examples (i.e. learn a predictor on {\em (ar3 $+$ ar5)} and test it on {\em ar4}) and a maximum of $(63+107) *
0.90 = 153$ examples (i.e. learn a predictor on {\em (ar3 $+$ ar4)} and test it on {\em ar5}).

\fig{arpd1} to \fig{arpf2} shows {\em Observation1} (i.e. $NN_{pd} \ge WC_{pd} $) and {\em Observation2} (i.e. $NN_{pf} \le WC_{pf} $) for SOFTLAB data. Recall that these observations were mutually exclusive for NASA data. The pattern is similar in SOFTLAB data:
\bi
\item for {\em ar4} mutual exclusiveness hold: $NN_{pd} \ge WC_{pd}$ and $NN_{pf} > WC_{pf}$
\item  for {\em ar3} and {\em ar5}: $NN_{pf} \le WC_{pf}$. If the observations were mutually exclusive, we would expect  $NN_{pd} < WC_{pd}$. Yet, for {\em pd} $NN_{pd} \ge WC_{pd}$, however this inequality holds with the equity (see \fig{arpd1}) and $NN_{pd} \not > WC_{pd}$
\ei

In summary, 
   the WC, CC and NN patterns found in American NASA rocket software
   are also observed in software controllers of Turkish domestic appliances. While this is not 
   the definitive proof of the external validity of our
   results, we find it a very compelling result that provides suggestive evidence for the generality for our observations.


\section{Conclusion}
In this study, we have found a clear and unambiguous conclusion for defect prediction amongst
our static code features:
\bi
\item CC-data dramatically increase the probability of detecting defective modules;
\item But CC-data also dramatically increase the false alarm rate. 
\item This can be explained by the {\em extraneous hypothesis};
\item NN-filtering CC data avoids the high false alarm rates by removing irrelevancies in CC data;
\item Yet WC-data models are still the best and they can be constructed with small amounts of data (i.e. 100 examples).
\ei
Interpreting these in terms of the posed questions in the introduction, we conclude that:
\bi
\item Companies can benefit from raw CC data in extreme cases such as mission critical projects, where the cost of false alarms can be afforded.
\item Pruning CC data with NN-filter allows the use of CC data for constructing practical defect predictors in other domains. NN-filtered CC data yields much better results than raw CC data, yet closer but worse results than WC data.
\item The best option of using WC data requires the collection of a mere hundred examples from within a company and can be done in a short time (i.e. a few months).
\item We observe the same patterns not only in aerospace software from NASA, but also in software from a completely different company round the globe. While this is a strong evidence of generality, we take great care {\em not to} interpret it as a proof of external validity. 
\ei

An important issue worth more mentioning is the concern about the time required for setting up a metric program (i.e. in order to collect data for building
actual defect predictors).
Our incremental WC results suggest that, in the case of defect prediction,
this concern may be less than previously believed. 
In most of our experiments, as few as 100 modules may be enough to learn
adequate defect predictors. 
When so few examples are enough,
it is possible that
projects can learn local defect predictors that are relevant to their
current technology in just a few months. 

Further, our experiences with our industry partners show that data collection is not necessarily a major concern. Static code attributes can be automatically and quickly collected with relatively little effort. We have found that when there is high level management commitment, it becomes a relatively simple process. For example, in an extreme case, the three projects of SOFTLAB data were collected in less than a week's time. Neither the static code attributes, nor the mapping of defects to software modules were available when the authors attempted to collect these data. Since these were smaller scale projects, it was sufficient to spend some time with the developers and going through defect reports. 
Although not all projects have 100 modules individually, the company has a growing repository from several projects and enough data to perform defect prediction.

We also have experience with a large scale telecommunication company, where a long-term metric program for monitoring complex projects (around 750.000 lines of code) requires introducing automated processes. Again with high level management support, it was possible to employ appropriate tool support and these new processes were introduced easily and invisible to the staff. For that project, we have now a growing repository of defects mapped with source code (around 25 defects per month). Note that the software in that project are being developed for more than 10 years and have very low defect rates. We have obtained the first results in the $8^{th}$ month of the project and it is planned to be completed in $12$ months. In summary setting up a metric program for defect prediction can be done more quickly than it is perceived.

We conclude our findings by proposing a two-phase approach for initiating defect prediction process:
\bi
\item {\em Phase1:} Use NN filtered CC data to make local predictions and start to collect WC data.
\item {\em Phase2:} After a few hundred examples are available in the local repository (usually a few months), discard the predictor learned on CC data and switch to those learned from WC data.
\ei 

\bibliography{refs}
\bibliographystyle{IEEEtran}

\end{document}
