%\documentclass[twocolumn,journal,draftcls,compsoc]{IEEEtran}
\documentclass[final]{svjour3}

\usepackage{rotating}
\usepackage{pifont}
\usepackage{times}
\usepackage{alltt}
\usepackage{graphicx}
\usepackage{url}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\usepackage{moreverb}
\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
\makeatother

%\documentclass[technote]{IEEEtran}
%\documentclass[twocolumn,technote,9pt,compsoc]{IEEEtran}
\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		\put(0,0){\line(0,1){8}}
		\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tbl}[1]{Table~\ref{tbl:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage[nocompress]{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
%\usepackage[nomarkers]{endfloat}
%\let\MYoriglatexcaption\caption
%\renewcommand{\caption}[2][\relax]{\MYoriglatexcaption[#2]{#2}}
\journalname{Empirical Software Engineering}
\begin{document}

\title{On the Relative Value of Cross-Company and Within-Company Data
for Defect Prediction}
\author{Burak Turhan         \and
        Tim Menzies \and
        Ayse Bener \and
        Justin Distefano%etc.
}

%\authorrunning{Short form of author list} % if too long for running head

\institute{Burak Turhan \at
              Department of Computer Engineering, Bogazici University, Istanbul, Turkey \\
%              Tel.: +90-212-3597227\\
  %            Fax: +90-212-287-2461\\
              \email{turhanb@boun.edu.tr}           %  \\
%             \emph{Present address:} of F. Author  %  if needed
           \and
           Tim Menzies \at
              Lane Department of Computer Science and Electrical Engineering, West Virginia\\
              \email{tim@menzies.us} 
           \and
              Ayse Bener \at
              Department of Computer Engineering, Bogazici University, Istanbul, Turkey \\
%              Tel.: +90-212-3597226\\
%              Fax: +90-212-287-2461\\
              \email{bener@boun.edu.tr} 
            \and
           Justin Distefano \at
              Lane Department of Computer Science and Electrical Engineering, West Virginia\\
              \email{jdistefano@ismwv.com} }



\date{Received: Sept'08 / Accepted: Nov'08 }
% The correct dates will be entered by the editor




\maketitle

\begin{abstract}

We propose a practical defect prediction approach for companies that do not track 
defect related data. Specifically, we investigate the applicability of cross-company (CC) data for building localized defect
predictors using static code features. 

Firstly, we analyze the conditions, where
CC data can be used as is. These conditions turn out to be quite few. Then we apply principles of analogy-based learning (i.e. nearest neighbor (NN) filtering) to CC data, in order to fine tune these models for localization. We compare the performance of these models with that of defect predictors learned from within-company (WC) data. As expected, we observe that defect predictors learned from WC data outperform the ones learned from CC data. However, our analyses also yield defect predictors learned from NN-filtered CC data, with performance close
to, but still not better than, WC data.  
Therefore, we perform a final analysis for determining the minimum number of local defect reports in order to learn WC defect predictors. We demonstrate in this paper that the minimum number of data
samples required to build effective defect predictors can be quite
small and can be collected quickly within a few months.


Hence, for companies with no local defect data, we recommend a two-phase approach that allows them to employ the defect prediction process instantaneously. 
In phase one,
companies should use NN-filtered CC data to initiate the defect
prediction process and simultaneously start collecting WC (local)
data. Once enough WC data is collected (i.e. after a few months), organizations
should switch to phase two and use predictors learned from WC data.

KEYWORDS: defect prediction; learning; metrics (product metrics); cross-company; within-company; nearest-neighbor filtering
\end{abstract}

%------------------------------------------------------------------------- 


\section{Introduction}

Defect prediction studies usually focus on building models with available local data (i.e. within company predictors). To employ these models, a company should have a data repository, where project metrics and defect information from past projects are stored. However, few companies apply this practice. We suspect that a common reason for not using defect predictors in practice, is the lack of local data repositories. Constructing such a repository requires keeping track of project metrics together with related defect information. When automated tools are not used, manual effort is inevitable to maintain these repositories. 
%Our view is that managers do not take the effort to collect metrics for process improvement due to tight schedule and budget constraints.

On the other hand, there are public data repositories including projects from companies such as NASA \cite{me07c}. In this context, we investigate whether these public project data can be helpful for other companies for building localized defect predictors\footnote{Throughout the paper, the following notation is used: a defect predictor (or predictor) means a binary classification method that categorizes software modules as either defective or defect-free; data refers to MxN matrices of raw measurements of N metrics from M software modules; these N metrics are referred to as features.}, especially for those with limited or no defect data repository. 

Defect prediction literature contains many examples where predictors are learned from within company data~\cite{Arisholm:2006p516, Bell:2006p517, Ostrand:2007p519, me07e,  chen05, dekhtyar04, jiang07, nagappan05}.  However, there exists no case that attempts to harness cross company data. 
In this paper, we focus on {\em binary defect prediction} and perform analyses to check if we can reach a conclusion in favor of either CC or WC data. 
Specifically,
this paper 
assesses the relative merits
of cross-company (CC) vs within-company (WC) data for  defect prediction.


We perform a series of analyses on 10 publicly available project data from different companies.
Our analyses use the Naive Bayes data miner and are based on static code features\footnote{ Therefore, throughout the paper, the term "data" refers to static code features.}.

 Our analyses aim at answering the following research questions: 



\bi
\item {\em Are CC data ever useful for organizations?}:

Our goal is to identify the conditions under which
cross-company  data should be preferred
to within-company 
for the purposes of structuring software inspections using outcomes of defect predictors.
In the first analysis, we compare the performance of defect predictors learned from WC data to those learned from CC data.


\item{\em How can companies filter CC data for local tuning?}:


CC data includes information from many diverse projects and are heterogeneous compared to WC data.  The goal of the second analysis is to select a subset of the available CC data that is similar to WC data and to investigate the effect of data homogeneity on the defect prediction performance. 
We apply  a simple nearest neighbor (NN) filtering to CC data for automatically 
constructing a locally tuned repository.
We use the Euclidean distance between static code features of WC and CC data for measuring similarity and automated selection of neighbors.\footnote{We should carefully note that we do not make use of any conceptual similarities, since our analysis is based on static code features. As to the issue of conceptual connections within the code, we refer the reader to the concept location and cohesion work of Marcus et al.~\cite{marcus08}.}

\item{\em What is the smallest amount of local data needed for constructing a 
model?}:

In the third analysis, we focus on WC models and determine the smallest amount of local data needed for constructing a model. We employ an incremental learning approach to  WC data in order determine the number of samples in local repositories for building defect prediction models.


\item{\em Can our results be generalized?}:

We initially use only NASA projects in our three analyses to answer the above questions.
In order to check the  external validity of our
results, our last analysis replicates all three analyses on
projects from a company that has no ties with NASA: specifically, a Turkish company writing software controllers
for home appliances. 

\ei

The contributions of this paper are relevant for both academia and practice. Prior to this paper, no study was performed to investigate the relative merits of using cross company or within company data for constructing defect predictors. In our analysis, in favor of the common belief, we empirically show that within company data is better for defect prediction problem. Further, on the contrary to the common belief, we empirically show that, the required local data for constructing defect predictors can be easily and quickly collected. For our analysis, we have collected data from industry and made it publicly available for the use of other researchers and practitioners. As a practical contribution, our analysis of cross company data and the proposed methodology allow the construction of defect predictors even for companies with no local defect data. In summary, we observe that the defect detectors learned from site A are found to be useful at site B, at least for the datasets we analyzed. Furthermore, it is a result of major significance that other project's data can be used locally (after applying a very simple filter).
Nevertheless, there may be some
reluctance on the part of programmers and managers to use data
collected at other sites. Such caution is to be expected amongst professionals striving to deliver quality products
to their clients. In this paper, we suggest that it can be quite beneficial to use
data from other sites, particularly when no local data is available. Professional developers should apply
our advise cautiously- perhaps via an intial case study on a small part of their system.

This paper is organized as follows: The next section motivates
our work with examples of industrial groups that use these detectors.
Then, in Section 3, we explain our methodology by describing the data sources, data mining algorithm and the performance evaluation criteria.  In Sections 4, 5 and 6, we describe our analyses, results and discussions for the first three research questions stated above. In Section 7, we replicate all three analyses in order to check the external validity of our results. Sections 8, 9 and 10, discusses the related work, practical implications and threats to validity  respectively. We conclude our work in Section 11.

\section{Motivation}
\subsection{Why Study WC vs CC?}

The genesis of this work was several frustrating years 
working at NASA trying to collect data on new projects. This proved
difficult to do, for a variety of reasons (including feuding NASA centers and
contractors trying to maintain some control over the information flow
to the client). Several times in that process, we asked ``Is it necessary
to collect new data? Can't we just apply old data?''.

In the middle of 2007, we had an opportunity to evaluate CC
data usage on a Turkish software development company. That company worked
in a highly competitive market and a decrease in their bug rate of even 1\%
had an economic benefit to them.
However, that company had no historical data with which to train its defect detectors.
We therefore trained a defect detector on NASA data and applied it to the Turkish
software.
Much to our surprise,
that detector could ``adequately'' predict errors within Turkish software
(``adequate'' in the sense that the Turkish client looked at the code
that triggered our detectors, found bugs, and was subsequently
 interested enough to request
more studies of this nature).  

Based on that small informal experience, 
 we saw:
\bi
\item The business benefits in using CC
data: no need to wait for local data collection;
\item  The adequacy of the learned detectors; 
\ei
A single case study is not enough to justify a general principle. Accordingly, we conducted the analyses
described in this paper.

\subsection{Why Study Defect Predictors?}
Defect predictors are subtley 
different to other quality assurance practices.
This
section highlights those differences and explains why
we elect to study them.

It is insightul to contrast defect
prediction 
from Musa-style software reliability engineering (SRE)~\cite{musa87}.
In standard SRE, there is a track record of post-release failures 
and the goal is to predict the probability of failure on demand 
in the next call on the software. One of the major results of the 
SRE literature is that as the software is developed and bugs 
are removed, this probability forms a reliability growth curve 
with predictable properties.

The standard defect prediction task is different to standard 
SRE.
Our task is to predict defective modules during development and before delivery.
Such early lifecycle detection is important since, as Boehm reports, repair
cost increases exponentially, the longer a bug remains in the system~\cite{boehm88}.
Hence,
we learn defect predictors from logs of pre-release 
defects found by a verification and validation (V\&V) team. Such defects may be seen 
in code from systems that is not yet complete. Hence, many 
of the features used in this kind of analysis are static code 
features that can be extracted from as-yet-un-executing code. 

The patterns found in pre-release defects or post-release failures can be used in different ways. Reliability 
growth curves can be used to predict how long before 
the running software will reach some required reliability of, 
i.e. one failure in 10,000 calls. Defect predictors, on the other 
hand, can be used to bias the ordering of modules to be 
inspected by V\&V teams: 

\bi
\item
In the case where there is insufficient resources to inspect 
all code (which is a very common situation in industrial 
developments), defect predictors can be used to increase 
the odds that the inspected code will have more defects. 
\item
In the case where all the code is to be inspected, but that inspection process will
take weeks to months to complete, defect predictors can be used to increase the 
chances that defective modules will be inspected earlier. This is useful since it 
gives the development team earlier notification of what modules require rework, 
hence giving them more time to complete that rework prior to delivery.
\ei
The above two cases are quite common.
The authors of this paper are from the United
States and Turkey and only began exploring defect detectors when
they realized that their user community was making extensive use of these
detectors.
%For example, for several  years, one of the authors worked on-site
%at the NASA software Independent Verification and Validation facility
%and he knows of several large government software contractors that
%will not review software modules {\em unless} tools like McCabe predict
%that they are fault prone.  In Feburary 2008, that
%author attended two briefings by NASA and European Space Agency
%test engineers who both cited McCabe's $v(g)\ge 10$ as one of their
%triggers for modules requiring closer inspections.

%What is significant about the NASA experience is that NASA is not ``one'' organization,
%in the conventional sense. Rather, NASA is an umbrella organization that loosely
%monitors and guides large group of sub-contractors. 
%That is, data collected from ``NASA'' is really data collected from numerous large
%software engineering companies like Lockheed-Martin, Boeing, Rockwell-Collins, etc.

%As per our Turkish experience, the software development industry in Turkey has been growing very rapidly in the last couple of years as the domestic economy stabilizes. We observe more and more software development houses that are delivering projects to European and U.S markets. As the companies mature and they compete in the global market place they need to improve their processes, take measures to lower their costs and increase the quality of their products. Therefore they realize that there needs to be a consistent methodology for software development and they need to collect product and project related metrics data for measurement and evaluation. In this context, most of our Turkish industrial partners name their major challenges as effectively allocating their verification resources for locating defects before product delivery. Specifically, our Turkish industrial partner uses the outcomes of this research in order to guide their decision making processes during testing.
Furthermore, these detectors are widely used despite the fact that they are quite ``coarse-grained''. 
In the following study, our only knowledge of modules will be that they are defective
or defect-free. This work makes no use of other knowledge about a module such
as its criticality or the dates on which it failed.
\bi
\item
{\em Regarding failure occurence logs:} 
When working with large teams of contractors, it is often difficult to access
precise logs on when failures occured
since such information is often corporate critical. The NASA
coarse-grained
information used
in this study, for example, 
took years of careful negotiation to acquire. All the data passed
to us was highly sanitized; e.g. this research team could not
even access project or module names.
\item
{\em Regarding criticality knowledge:}  
Our goal is to predict defective modules during development and before delivery.
Goseva and Hamill~\cite{goseva07} report that pre-delivery
perceptions
of module criticality can be widely misleading. In their work, they trace 
errors back to the modules that caused runtime faults. They find that
a remarkably small set of modules are implicated in all faults and that
set can change, radically, if the input space to the program alters.
\ei
For further notes on related work on defect predictors from static code attributes,
see Section 8.3.

\section{Methodology}
\subsection{Data}
The analyses of this paper use the static code features of 10 projects tabulated in \tbl{data1} and \tbl{data2}, which are
downloaded from the PROMISE repository\footnote{\url{http://promisedata.org/repository}}.
Available static code features are shown in \tbl{attr1}. 
An advantage of static code features is that they can be quickly and automatically collected
from the source code, even if no other information is available.

\begin{table}[ht]
\caption{Descriptions of ten software projects used in the paper.
The rows labeled ``NASA'' come from NASA aerospace projects while the rows
labeled ``SOFTLAB'' come  from a Turkish software company writing
applications for domestic appliances.}
\begin{tabular}{l|lll}
\scriptsize
&     &       &   \\
source
&project & language & description \\ \hline
NASA&pc1  & C++ &Flight software for earth orbiting satellite\\
NASA&kc1  & C++ &Storage management for ground data\\
NASA&kc2  & C++ &Storage management for ground data\\
NASA&cm1  & C++ &Spacecraft instrument\\
NASA&kc3  & JAVA &Storage management for ground data\\
NASA&mw1  & C++ &A zero gravity experiment related to combustion\\
SOFTLAB& ar4 & C &Embedded controller for white-goods\\
SOFTLAB& ar3& C &Embedded controller for white-goods\\
NASA&mc2  & C++ &Video guidance system \\
SOFTLAB & ar5 & C &Embedded controller for white-goods\\
\end{tabular}
\label{tbl:data1}
\end{table}

\begin{table}[ht]
\caption{Summary of data from ten software projects of \tbl{data1}, sorted in order of number of functional units.}
\begin{tabular}{l|lrrrrr@{.}l}
\scriptsize
&     &     (\# modules)         &          &  & & &\\
source &project  & examples & features  & loc &   \multicolumn{3}{r}{\%defective}\\\hline
NASA&pc1  & 1,109    & 21       &25,924&&    6&94\\
NASA&kc1  & 845      & 21       &42,965&&    15&45\\
NASA&kc2  & 522      & 21        &19,259&&   20&49\\
NASA&cm1  & 498      & 21        &14,763&&   9&83\\
NASA&kc3  & 458      & 39      &7749&&     9&38\\
NASA&mw1  & 403      & 37      &8341&&     7&69\\
SOFTLAB& ar4 & 107 &  29       &9196&     & 18 & 69\\
SOFTLAB& ar3& 63 &    29       &5624&   & 12 & 70\\
NASA&mc2  & 61      & 39         &6134& & 32&29\\
SOFTLAB & ar5 & 36 &  29       &2732&  & 22 & 22\\
\cline{5-4}
     \multicolumn{4}{c}{~}     &  4,102 
\end{tabular}
\label{tbl:data2}
\end{table}


\begin{table}
\begin{center}
\caption{Static code features available in \tbl{data2} projects.}\label{tbl:attr1}
\scriptsize
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|}\hline
\#&Feature &pc1 &kc1 &kc2 &cm1 &kc3 &mw1 & ar4& ar3&mc2&ar5\\\hline \hline
1&$branch count $ &X &X &X &X &X &X &X & X&X & X \\\hline 
2&$code and comment loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
3&$comment loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
4&$cyclomatic complexity $&X &X &X &X &X &X &X & X&X & X \\\hline 
5&$design complexity $ &X &X &X &X &X &X &X & X&X & X \\\hline 
6&$halstead difficulty $ &X &X &X &X &X &X &X & X&X & X \\\hline 
7&$halstead effort $ &X &X &X &X &X &X &X & X&X & X \\\hline 
8&$halstead error $ &X &X &X &X &X &X &X & X&X & X \\\hline 
9&$halstead length $ &X &X &X &X &X &X &X & X&X & X \\\hline 
10&$halstead time $ &X &X &X &X &X &X &X & X&X & X \\\hline 
11&$halstead volume $ &X &X &X &X &X &X &X & X&X & X \\\hline 
12&$total operands $ &X &X &X &X &X &X &X & X&X & X \\\hline 
13&$total operators $ &X &X &X &X &X &X &X & X&X & X \\\hline 
14&$unique operands $ &X &X &X &X &X &X &X & X&X & X \\\hline 
15&$unique operators $ &X &X &X &X &X &X &X & X&X & X \\\hline 
16&$executable loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
17&$total loc$  &X &X &X &X &X &X &X & X&X & X \\\hline 
18&$halstead content $ &X &X &X &X & X &X & & &X& \\\hline
19&$essential complexity$ &X &X &X &X & X &X & & &X&  \\\hline 
20&$halstead vocabulary $ & X& X& X&X  &  & &X & X&  & X \\\hline
21&$blank loc$  &X &X &X &X &X &X &X & X&X & X  \\\hline
22&$call pairs $ & & & & &X &X &X & X& X& X  \\\hline
 23&$condition count $ & & & & &X &X &X & X& X& X  \\\hline
 24&$cyclomatic density $ & & & & &X &X &X & X& X& X  \\\hline
 25&$decision count $ & & & & &X &X &X & X& X& X  \\\hline
 26&$decision density $ & & & & &X &X &X & X& X& X  \\\hline
27&$halstead level $ & & & & &X &X &X & X& X& X  \\\hline
28&$multiple condition count $ & & & & &X &X &X & X& X& X  \\\hline
29&$design density $ & & & & &X &X &X & X& X& X \\\hline
30&$norm cyclomatic comp. $ & & & & &X &X &X & X& X& X \\\hline 
31&$formal parameters $ & & & & &X &X &X & X& X& X  \\\hline
32&$modified condition count $ & & & & & X &X & & &X&   \\\hline 
33&$maintenance severity$ & & & & & X &X & & &X&   \\\hline
 34&$edge count $ & & & & & X &X & & &X&   \\\hline
 35&$node count $ & & & & & X &X & & &X&   \\\hline
 36&$essential density $ & & & & & X &X & & &X&   \\\hline
 37&$global data complexity$ & & & & & X & & & &X&   \\\hline
  38&$global data density $ & & & & & X & & & &X&   \\\hline 
 39&$percent comment $ & & & & & X &X & & &X&  \\\hline 
  40&$number of lines $ & & & & & X &X & & &X&   \\\hline \hline
&Total &21 &21 &21 &21 &39 &37 &29 & 29&39 & 29  \\\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}
\begin{center}
\caption{Static code features shared by NASA and SOFTLAB projects.}\label{tbl:attr2}
\scriptsize
\begin{tabular}{|c|c|c|c|}\hline
\#&Feature &NASA shared&All Shared \\\hline \hline
1&$branch count $ &X&X\\\hline 
2&$code and comment loc$ &X&X \\\hline 
3&$comment loc$ &X&X\\\hline 
4&$cyclomatic complexity $&X&X\\\hline 
5&$design complexity $ &X&X\\\hline 
6&$halstead difficulty $ &X&X\\\hline 
7&$halstead effort $ &X&X\\\hline 
8&$halstead error $ &X&X\\\hline 
9&$halstead length $ &X&X\\\hline 
10&$halstead time $ &X&X\\\hline 
11&$halstead volume $ &X&X\\\hline 
12&$total operands $ &X&X\\\hline 
13&$total operators $ &X&X\\\hline 
14&$unique operands $ &X&X\\\hline 
15&$unique operators $ &X&X\\\hline 
16&$executable loc$ &X&X\\\hline 
17&$total loc$  &X&X\\\hline 
18&$halstead content $ &X&\\\hline
19&$essential complexity$ &X&  \\\hline 
\hline
&Total &19&17 \\\hline
\end{tabular}
\end{center}
\end{table}

Our project data are taken from software developed
in
different geographical locations across North America (NASA) and Turkey (SOFTLAB). Therefore, the static code features that are available for each project vary. While \tbl{attr1} shows the features available for each project, \tbl{attr2} shows the common features for both sources.
While NASA and SOFTLAB are one single {\em source} of data, there are several projects within each source.
For example, NASA 
is really an umbrella organization  used to co-ordinate and fund
a large and diverse set of projects:
\bi
\item
The NASA data was
collected from across the United States over a 
a period of five years from numerous
NASA contractors working at different geographical centers.  
\item
These projects
represent a wide
array of projects, including satellite instrumentation, ground control
systems and partial flight control modules (i.e. Altitude Control).  
\item
The
data sets also represent a wide range of code reuse: some of the projects
are 100\% new, and some are modifications to previously deployed code. 
\ei

That is, even if we explore just the NASA data sets, we can still
examine issues of cross- vs within- company data use.
Nevertheless, using our connections with the Turkish software industry, we collected
new data sets in the format of \tbl{attr1} from a Turkish white- goods manufacturer. 
The SOFTLAB datasets (\{ar3, ar4, ar5\}) in \tbl{data1},
are the 
controller software for:
\bi
\item
A washing machine;
\item
 A dishwasher;
\item And a
refrigerator. 
\ei

In summary, 7 datasets are from NASA projects developed at different sites by different teams, hence we treat each of them as if they were from 7 different companies. Remaining 3 datasets are from a Turkish company collected from software for domestic home appliances. Therefore we use 10 projects from 8 different companies. 

In all analyses, we used all available features to learn defect predictors. One question is whether it might be better to focus on just
some special subset of those features.
In previous work~\cite{me07b}, we have found that there is no best single set of features. In fact, this
{\em best} set is highly unstable and changes dramatically from data set to data set.
That study is summarized, below. The main message of that study is that it is ``best''
to never assume a {\em best} set of features. Rather, give them all to a learner and let
it work out which ones are best to use or ignore for a particular application.

Feature subset selection is a technique that explores subsets of the available features.
The simplest and fastest
subsetting
method is to rank attributes from the most
informative to least informative. After discretizing numeric 
data\footnote{E.g. given an attribute's
minimum and maximum values, replace a particular value $n$ with $(n - min)/((max - min)/10)$.
For more on discretization, see~\cite{dou95}.}
then if $F$ is a set of features, the number of bits required to encode
a class after observing a feature is:
\begin{equation}\label{eq:ent}
H(C|F) = -{\sum}_{f{\in}F} p(f) {\sum}_{c{\in}C} p(c|f)log_2(p(c|f)
\end{equation}
The highest ranked
feature $F_i$ is the one with the largest {\em information gain};
i.e the one that most
reduces the encoding required for the data {\em after}
using that feature; i.e.
\begin{equation}\label{eq:infogain}
InfoGain(F_i) = H(C) - H(C|F_i)
\end{equation}
where $H(C)$ comes from \eq{ent}.
In {\em iterative InfoGain subsetting}, predictors are  learned
using the  $i=1,2...,N$-th top-ranked attributes.
Subsetting  terminates when $i+1$ attributes perform no better than
$i$.
In {\em exhaustive InfoGain subsetting},
the attributes
are first ranked using iterative subsetting.
Next, predictors are built using
all subsets of the  top $j$ ranked
attributes.
For both iterative and exhaustive subsetting, the process is repeated 10 times
using 90\% of the  data (randomly selected).
Iterative subsetting takes time 
linear on the number of attributes $N$ while exhaustive subsetting takes time
$2^j$ (so it is only practical for small $j\le N$).

\begin{figure}
{\footnotesize
\begin{center}
%XX remove accuracy
\begin{tabular}{rr|rr|c|c}
\multicolumn{2}{c}{~}& \multicolumn{2}{c}{\%}&selected 
attributes&selection\\\cline{3-4}
data  &  N  & pd      & pf  &   (see\fig{attrs})& method  \\\hline
pc1   & 100&48 & 17 &  3, 35, 37&exhaustive  subsetting\\
mw1   & 100&52 & 15 &  23, 31, 35&iterative subsetting\\
kc3   &  100&69 & 28 & 16, 24, 26&iterative subsetting\\
cm1   & 100&71 & 27  & 5, 35, 36&iterative subsetting\\
pc2   & 100&72 & 14 & 5, 39&iterative subsetting\\
kc4   & 100&79 & 32 &  3, 13, 31&iterative subsetting\\
pc3   & 100&80 & 35 & 1, 20, 37&iterative subsetting\\
pc4   & 100&98  & 29 & 1, 4, 39&iterative subsetting\\\hline
all   & 800& 71& 25 &\multicolumn{2}{c}{}
\end{tabular}
\end{center}
\caption{Best defect predictors learned in ~\cite{me07b}. Mean
results from 
Naive Bayes after a 10 repeats of (i)~randomize the order of the data;
(ii)~divide that data into ten 90\%:10\% splits for
training:test. Prior to learning,
all numerics where replaced with logarithms.  InfoGain was then
used to select the best two or three attributes shown in the right-hand
column (and if ``three'' performed
as well as ``two'', then this table shows the results using ``two'').}
\label{fig:best}}
\end{figure}

\begin{figure} {\footnotesize
\begin{center}
\begin{tabular}{llll}
  &frequency&\\
ID&in \fig{best}&what& type\\\hline
1&2&  {loc\_blanks}  & locs\\
3&2 &  {call\_pairs} & misc\\
4&1 &  {loc\_code\_and\_command} & locs\\
5&2 &  {loc\_comments} & locs\\
13&1 &  {edge\_count}   & misc\\
16&1 &  {loc\_executable} & locs\\
20&1 &  {I} & H (derived Halstead)\\
23&1 &   {B}& H (derived Halstead)\\
24&1 &   {L}& H (derived Halstead)\\
26&1 &   {T}& H (derived Halstead)\\
31&2 & {node\_count} & misc\\
35&3 & {$\mu_2$}& h (raw Halstead)\\
36&1 & {$\mu_1$}& h (raw Halstead)\\
37&2 &  {number\_of\_lines} & locs\\
39&2 & {percent\_comments}& misc\\
\end{tabular}
\end{center}
\caption{Attributes used in \fig{best}. }\label{fig:attrs}}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=3.5in]{infogain.pdf}
%Burak: see http://now.unbox.org/all/trunk/doc/05/gg/infogain.pdf
\end{center}
\caption{InfoGain for KC3 attributes. Calculated from \eq{infogain}.
Lines show means and t-bars show standard deviations after 10 trials
on 90\% of the training data (randomly selected).}\label{fig:sorted}
\end{figure}

\fig{best} shows the feature subset selection results reported previously~\cite{me07b}.
Note that this study was run on many of the same NASA data sets used in this study.
The right-hand-side column of that figure reports which attributes used
in that study were selected by feature subset selection. In a result consistent
with many of the static code measures being highly correlated, note that most
of the features were removed and using 3 features was often as good as using many more.

In terms of ``which features are best to use?'', the 
key feature of 
\fig{best} is that different data sets selected very different
``best'' features. \fig{attrs} tallies the number of times each
feature was selected: no feature was selected all the time and many
of the features were never selected at all. This
aspect can be explained by \fig{sorted} which shows the InfoGain of
all the featues in one of our  data sets (KC3).  Note how the highest ranked
attributes (those on the left-hand-side) offer very similar
information.  That is, there are no clear winners so minor changes in
the training sample (the 90\% sub-sampling used in subsetting or a
cross-validation study) can result in the selection of very different
``best'' attributes.

The pattern of InfoGain values of \fig{sorted} (where there are many
alternative ``best'' features) repeats in all our data sets.
This pattern explains a prior observation of Shepperd \& Ince who
found 18 publications where an equal number of studies reporting that the
McCabe cyclomatic complexity is the same; is better; or is worse than
lines of code in predicting defects~\cite{shepperd94}.  
\fig{sorted} motivates the following 
principles: 
\bi
\item
Do not seek ``best'' subsets of static code attributes.
\item
Rather, seek instead for learning methods that can combine multiple
partial defect indicators like  the statistical methods of
Naive Bayes. 
\ei

Therefore, we do not perform feature subset selection, rather we use all available features in our analyses.

\subsection{Naive Bayes Classifier}
Each project in \tbl{data2} contains information from many
modules;  the smallest unit of functionality\footnote{
In other languages, modules may be called ``function'' or
``method''.}.  
To learn defect predictors,  the project data are
augmented with 
one dependent variable holding  boolean
values for
``defects detected''.
The data mining task is to find combinations of static code features that predict the boolean value of the dependent variable.

In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem yields better performance than rule based methods (i.e. decision trees, oneR), for the 
\tbl{data2} NASA data\footnote{SOFTLAB data were not available at that time.}~\cite{me07b}.
An extensive study by Lessmann et.al also shows that Naive Bayes performs equivalently well with 15 other methods~\cite{lessmann09}. Therefore, we find Naive Bayes as a viable choice as a classifier to use in our analysis. 

A N{aive} Bayes classifier~\cite{duda76}
is
based on Bayes' Theorem. 
Informally, the
theorem says $next=old*new$ i.e. what we'll believe {\em next} comes
from how {\em new} evidence effects {\em old} beliefs.  More formally:
\begin{equation}\label{eq:nb}
P(H|E)=\frac{P(H)}{P(E)}\prod_iP(E_i|H)
\end{equation}
i.e. given fragments of evidence $E_i$ and a prior probability for
a class $P(H)$, the theorem lets us calculate a  posterior  probability
$P(H|E)$. 
For example, in our data sets, there are two hypotheses: modules are either
defective or not:
\mbox{$H\in \left\{ defective,nonDefective \right\}$}.
Also, if a particular module has $numberofSymbols=27$ and
$LOC=40$ and was previously classified
as ``defective'' then
\[
\begin{array}{lcl}
E_1&:& numberOfSymbols=27\\
E_2& :& LOC=40\\
H&:&  defective
\end{array}
\]

When building defect predictors,
the
prior probability of each class (``defective'' or ``defect-free'')
is calculated, given the defect content of each 
module.
So, if a data set has 100 modules and 25 of them are faulty, then:
\[P(defective)=0.25\]
When testing new data, a module is assigned to the class
with the higher posterior probability, calculated from \eq{nb}.


N{aive} Bayes classifiers are called ``naive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(only the frequency
counts are required for each feature),
 it is possible
that
correlated events are missed by this ``naive'' approach.
Domingos and Pazzani show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, N{aive} Bayes classifiers perform as well
as other seemingly more sophisticated schemes\footnote{ Details of this issue are out of the scope of this paper. For more, please see Table~1 in~\cite{domingos97optimality}}.


\eq{nb} offers a simple method for handling missing values. Generating a
posterior probability means of tuning a prior probability
to new evidence. If that evidence is missing,
then no tuning is needed. In this case \eq{nb} sets $P(E_i|H)=1$ which, in effect,
makes no change to $P(H)$.

When estimating the prior probability of hypothesis $H$, 
it is common practice~\cite{yang02} to  use 
 an {\em M-estimate} as follows.
Given 
that the total number of hypothesis is $C$,
the total number of training instances is $I$, and  $N(H)$ is the frequency the hypothesis $H$ within
$I$, then
\begin{equation}\label{eq:b}
P(H) = \frac{N(H) + m}{I + m\cdot C}
\end{equation}
Here $m$ is a small non-zero constant (often, $m=1$). Three special cases of \eq{b} are:
\bi
\item
For high frequency hypothesis 
in
large training sets,
$N(H)$ and $I$ 
 are much larger than 
$m$ and $m\cdot C$, so  \eq{b} simplifies
to 
$P(H)= \frac{N(H)}{I}$, as one might expect. 
\item
For low frequency classes
in large training sets,
$N(H)$ is small, $I$ is large,
and 
the prior probability for a rare class is never less than $\frac{1}{I}$;
i.e. the inverse of the number
of instances. If this were not true, rare classes would never appear in predictions.  
\item
For very small data sets, $I$ is small and $N(H)$ is even smaller.
In this case,
\eq{b} approaches the inverse of the number of classes; i.e. $\frac{1}{C}$.
This is a useful approximation when learning from very small data sets 
when all the data relating to a certain class has not yet been seen.
\ei
The prior probability calculated in \eq{b} is a useful lower bound for $P(E_i|H)$.
If some value $v$ is seen $N(f=v|H)$ times in feature $f$'s observations for 
hypothesis $H$, then\begin{equation}\label{eq:laplace}
P(E_i|H) = \frac{N(f=v|H) +l\cdot P(H)}{N(H) + l} 
\end{equation} 
Here, $l$ is the {\em L-estimate} and is set to a 
small constant (Yang \&Webb~\cite{yang02} recommend $l=2$).
Two special cases of are:
\bi
\item
A common situation is when there are many examples of 
an  hypothesis 
and numerous observations have been made
for a particular value. In that situation, 
$N(H)$ and $N(f=v|H)$ are large and
\eq{laplace}
approaches \mbox{$\frac{N(f=v|H)}{N(H)}$}, as one might expect. 
\item
In the case of very
little evidence 
for a rare hypothesis,
\mbox{$N(f=v|H)$} and $N(H)$ are small and \eq{laplace}
approaches $\frac{l\cdot P(H)}{l}$; i.e. the default frequency of an observation in  a hypothesis
is a fraction of
the probability of that hypothesis. This is a useful approximation when very
little data is available. 
\ei


For
numeric features it is common practice for N{aive} Bayes classifiers
to use the Gaussian probability
density function~\cite{witten05}:
\begin{equation}\label{eq:normal}
g(x) =
\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\end{equation} where
$\{\mu,\sigma\}$ are the feature's \{mean,standard
deviation\}, respectively.
To be precise, the probability of a
continuous feature having exactly the  value $x$ is zero,
but the probability that it lies within a small region, say $x \pm
\epsilon/2$, is $\epsilon \times g(x)$.  Since $\epsilon$ is a
constant that weighs across all possibilities, it cancels out and
needs not be computed. 

All the static code features of  \tbl{attr1} are numeric and 
are highly skewed. Therefore, we replace all numeric values with a "log-filter", i.e. $N$ with $ln(N)$.
This 
spreads out skewed
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
This "spreading'' can significantly
improve the effectiveness of data mining, since the distribution of log-filtered feature values fits better
to the normal distribution assumption ~\cite{me07b}.


\subsection{Performance Evaluation}
Data mining effectiveness was measured using {\em probability of detection} ($pd$), {\em probability of false alarm} ($pf$) and {\em balance} ($bal$)~\cite{me07b,me07e}.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% detection ($pd=1$).

Other measures such as $accuracy$  and $precision$
were not used since, as shown in \tbl{data2}, the percent of defective examples
in our projects was usually very small (median value around 13\%).
Accuracy and precision
are poor indicators
of performance for data where the target class (i.e. defective) is so rare
(for more on this issue, see~\cite{me07b,me07e}). 

The results were visualized using
{\em
quartile charts}.  To generate these charts, the performance
measures for an analysis
are sorted to isolate the median and the lower and upper quartile of numbers.
For example:

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In our quartile charts,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the 50\% percentile value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
The Mann-Whitney U test~\cite{mann47} was used to test
for statistical difference between results.
This non-parametric test replaces performance measure values (i.e. pd, pf, bal) with their
rank inside the population of all sorted values. 
Such non-parametric tests are recommended in data mining since many of the performance
distributions are non-Gaussian~\cite{demsar06}. 




\section{Analysis \#1: Are CC data ever useful for organizations?}
\subsection{Design}
Our goal is to determine whether using cross company data is beneficial for constructing defect predictors and to identify the conditions under which
cross-company data should be preferred
to within-company data.
Our first WC-vs-CC analysis follows the pseudo code given in \tbl{exp1},
for all 7 NASA projects of \tbl{data1}.
For each project,
test sets were built from
10\% of the data, selected at random.  Defect
predictors were then learned from:
\bi
\item CC data: all data from the other 6 projects.
\item WC data:  remaining 90\% data of that project;
\ei
Most of the \tbl{data1} data come from systems written in ``C/C++''
but at
least one of the systems was written in JAVA.
For cross-company data, an industrial practitioner may not have access to 
detailed meta-knowledge (e.g. whether it was developed in ``C'' or JAVA).
They may only be aware
that data, from an unknown
source, are available for download from a certain url.
To replicate that scenario,  we will make no
use of our meta-knowledge about \tbl{data1}. 


In order to control for   {\em order effects} (where the 
learned theory is unduly affected by the order of the examples)
our procedure was repeated 20 times,
randomizing the order of data in each project each time. 
In all, we ran 280 tests to compare WC-vs-CC:
\[\left(\mbox{2 data sources}\right) * \left(\mbox{20 randomized orderings}\right) * \left(\mbox{7 projects}\right)\]

\begin{center}
\begin{table}[ht]
\caption{Pseudocode for Analysis 1}
\scriptsize
\begin{tabular}{|p{.95\linewidth}|}\hline
\begin{verbatimtab}
DATA = [PC1, KC1, KC2, CM1, KC3, MW1, MC2]	// all available data
LEARNER = [Naive Bayes]		 		//  defect predictor 

C_FEATURES <- Find common features IN DATA   
FOR EACH data IN DATA
	data = Select C_FEATURES in data	// use common features
END
REPEAT 20 TIMES
	FOR EACH data in DATA
		CC_TRAIN = DATA - data		// cross company training data
		WC_TRAIN = random 90% of data   // within company training data
		TEST = data - WC_TRAIN		// shared test data
	
		//construct predictor from CC data
		CC_PREDICTOR = Train LEARNER with CC_TRAIN    
		// construct predictor from WC data
		WC_PREDICTOR = Train LEARNER with WC_TRAIN   
		//Evaluate both predictors on the same test data
		[cc_pd, cc_pf, cc_bal] = CC_PREDICTOR on TEST         
		[wc_pd, wc_pf, wc_bal] = WC_PREDICTOR on TEST   
	END
END
\end{verbatimtab}
\\\hline
\end{tabular}
\label{tbl:exp1}
\end{table}
\end{center}

The projects' data 
come from different sources and, hence,
have different features. For this analysis, only the features that are common in all NASA projects are used, a total of 19 features. These features are marked in "NASA Shared" column of~\tbl{attr2}. 

\subsection{Results from Analysis \#1}

\fig{ccwsnum} shows the $\{pd,pf\}$ quartile charts for CC vs WC data averaged over seven NASA projects.
The pattern is very clear: CC data dramatically increases {\em both}
the probability of detection and the probability of false alarms. The $pd$
results are particularly striking. 


For cross-company data: 
\bi
\item 50\% of the pd values are at or above 97\% 
\item 75\% of the pd values are at or above 83\%; 
\item And all the pd values are at or over 50\%.
\ei

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&      50&  83&  97&  100& 100& \boxplot{50}{33}{97}{100}{ 0}\\
    & WC&      17&  63&  75&   82& 100& \boxplot{17}{46}{75}{82}{18}\\\hline
 pf & CC&      14&  53&  64&   91& 100& \boxplot{14}{39}{64}{91}{ 9}\\
    & WC&       0&  24&  29&   36&  73& \boxplot{ 0}{24}{29}{36}{36}\\
\hline\end{tabular}

}
\end{center}
\caption{Analysis \#1 results averaged over seven NASA tables.
Numeric results on left; quartile charts on right. ``Q1'' and ``Q3'' denote
the 25\% and 75\% percentile points (respectively). 
The upper quartile of the first row is  not visible  since
it runs from 100\% to 100\%; i.e. it has zero length.}\label{fig:ccwsnum}
\end{figure}




To the best of our knowledge, \fig{ccwsnum} are the largest $pd$ values ever reported
from these data.
However, these very high $pd$ values come at some considerable cost. We observe in \fig{ccwsnum}
that the median false alarm
rate has changed from 29\% (with WC) to 64\% (with CC) and the maximum $pf$ rate
now reaches 100\%. Note that a 100\% $pf$ rate means that all defect-free modules are classified as defective, which yields inspection of all these modules unnecessarily and contradicts with the purpose of defect prediction. However, it is not right to assess the general behavior of the CC defect predictors with such an extreme case. We mention this issue in order to clarify that high false alarm rates may be prohibitive in the practical application of defect predictors.   

We explain these increases in $pd, pf$ with the  
{\em extraneous factors} in CC data. More specifically,
using a large training set (e.g. seven projects in 
\tbl{data2}) informs of not only the sources of errors, but also 
numerous irrelevancies. For example, the defect characteristics of software modules with different complexity levels or sizes may differ~\cite{koru08}. In this context, a complicated search algorithm's metrics are irrelevant to the defective behavior of a simple $sum$ function. Since there are few modules with extreme characteristics (i.e. complexity, size) in a single project, their effect on the overall model are limited. However, when data from multiple companies are combined, the number of these extreme cases, and hence their cumulative effect on the overall model increase significantly. Therefore, factors such as programming language constructs (i.e. object oriented vs. procedural) and project specific requirements (i.e. availability, speed) that have impacts on the module characteristics can be considered as extraneous factors.

Hence, 
large training sets 
increase the error detection rates (i.e. $pd$) since there are more 
known sources of errors. However, they also increase the probability of false 
alarms ($pf$) since there are more {\em  extraneous factors} introduced to 
the analysis. We will test the validity of this claim in the next analysis.

\subsection{ Checking the  Analysis \#1 Results}


Once a {\em general result} is defined (e.g. defect predictors learned from CC projects dramatically
increase both $pf$ and $pd$), it is good practice to check for 
specific exceptions
to that pattern. 
\tbl{ccwc} shows a summary of results when U tests with $\alpha=0.05$ 
were applied to
test results from each of the 7 projects,  {\em
in isolation} and \fig{ccwsprojs} shows the $\{pd,pf\}$ quartile charts for 
Analysis \#1 for each NASA project:
\bi
\item 
For six projects, the general result holds (i.e. both $(pd,pf)$ increases if defect predictors learned from CC projects are used rather than defect predictors learned from WC projects. See group $a$ in \tbl{ccwc}). 
\item
For one project, there is no difference in the performances of the defect predictors learned from CC and WC projects (see group $b$ in \tbl{ccwc}). 
\ei

%Overall, the {\em general result} 
%holds in the majority of cases (i.e. $6$ projects out of $7$ ). This is not a 100\% internally consistent effect, however 
%it very clear and encouraging.

\begin{table}
\caption{Summary of
U-test results (95\% confidence): moving from WC to CC.
For all projects' results, see~\fig{ccwsprojs}. }\label{tbl:ccwc}
\begin{center}
{\footnotesize
\begin{tabular}{|r|c|c|p{2cm}|l|}\hline
&$pd$    & $pf$ & &  \\
group & $WC \rightarrow CC$    &  $WC \rightarrow CC$& tables & $|tables|$ \\\hline\hline
a& increased               &        increased & CM1 KC1 KC2 MC2 MW1 PC1  & 6\\\hline
b& same                    & same             & KC3& 1\\\hline
\end{tabular}}
\end{center}
\end{table}



\begin{figure}[!t]

\begin{center}
\begin{tabular}{c}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($cm1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       80& 100&  100&  100& 100& \boxplot{80}{20}{100}{100}{0}\\
    & WC&       40&  60&  80&   100& 100& \boxplot{40}{20}{80}{100}{0}\\\hline
 pf & CC&        87&  91&  96&   96& 98& \boxplot{87}{4}{96}{96}{2}\\
   & WC&        24&  27&  33&   38&  47& \boxplot{24}{3}{33}{38}{9}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 88&  94&  94& 100& \boxplot{82}{6}{94}{94}{6}\\
    & WC&       64&  73&  82&   85& 97& \boxplot{64}{9}{82}{85}{12}\\\hline
 pf & CC&        47&  49&  51&   53& 57& \boxplot{47}{2}{51}{53}{4}\\
   & WC&        27&  34&  36&   38&  40& \boxplot{27}{7}{36}{38}{2}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 91&  91&  100& 100& \boxplot{82}{9}{91}{100}{0}\\
    & WC&       55&  73&  82&   91& 100& \boxplot{55}{18}{82}{91}{9}\\\hline
 pf & CC&        57&  62&  64&   74& 81& \boxplot{57}{5}{64}{74}{7}\\
   & WC&        14&  24&  31&   33&  45& \boxplot{14}{10}{31}{33}{12}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       60& 80&  80&  80& 100& \boxplot{60}{20}{80}{80}{20}\\
    & WC&       40&  60&  80&   80& 100& \boxplot{40}{20}{80}{80}{20}\\\hline
 pf & CC&        14&  19&  24&   31& 38& \boxplot{14}{5}{24}{31}{7}\\
   & WC&        10&  17&  21&   26&  36& \boxplot{10}{7}{21}{26}{10}\\
\hline\end{tabular}


}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       50& 67&  83&  100& 100& \boxplot{50}{17}{83}{100}{0}\\
    & WC&       17&  33&  67&   67& 83& \boxplot{17}{10}{67}{67}{18}\\\hline
 pf & CC&        55&  64&  73&   73& 100& \boxplot{55}{9}{73}{73}{27}\\
   & WC&        0&  27&  36&   45&  73& \boxplot{0}{27}{36}{45}{28}\\
\hline\end{tabular}

}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mw1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       75& 75&  100&  100& 100& \boxplot{75}{0}{100}{100}{0}\\
    & WC&       25&  50&  75&   75& 100& \boxplot{25}{25}{75}{75}{25}\\\hline
 pf & CC&        50&  55&  63&   66& 82& \boxplot{50}{5}{63}{66}{16}\\
   & WC&        13&  18&  21&   29&  37& \boxplot{13}{5}{21}{29}{8}\\
\hline\end{tabular}


}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       88& 100&  100&  100& 100& \boxplot{88}{12}{100}{100}{0}\\
    & WC&       38&  63&  63&   75& 88& \boxplot{38}{25}{63}{75}{13}\\\hline
 pf & CC&        89&  92&  93&   95& 99& \boxplot{89}{3}{93}{95}{4}\\
   & WC&        17&  25&  27&   30&  34& \boxplot{17}{8}{27}{30}{4}\\
\hline\end{tabular}


}\\

\end{tabular}
\end{center}
\caption{Project-wise Analysis \#1 results for NASA projects.}\label{fig:ccwsprojs}
\end{figure}




\subsection{Discussion of Analysis \#1}
When practitioners use defect predictors with high false alarm rates (e.g.the 64\% reported above),
they must allocate a large portion of their debugging budget to 
the unfruitful exploration of erroneous alarms.

In our industrial work, we have sometimes seen 
several situations
where detectors with high false alarms  are useful:
\bi
\item{\em
When the cost of missing the target is prohibitively expensive.} In
mission critical or security  applications, the goal of
100\% detection
may be demanded in all situations, regardless of the  cost of chasing false alarms.
\item{\em When only a small  fraction of
the data is returned.}
Hayes, Dekhtyar,
\& Sundaram call this fraction $selectivity$ and offer an extensive
discussion of the merits of this measure~\cite{hayes06}.  
\item {\em
When there is little or no cost in checking false alarms.}  

\ei

Having said that, we have shown these results
to five of our user groups in the United States and Turkey
and none of them could accept
false alarm rates as high as 64\%. Therefore, the conditions under which the
benefits of CC data (high probabilities of detection) outweigh their high
costs (high false alarm rates) are not common.


In summary,
 for most software applications, very high  $pf$ rates like the CC results
of \fig{ccwsnum} make the predictors impractical to use.



\section{Analysis \#2:  How can companies filter CC data for local tuning? }
The results of the first analysis restricts the use of CC data to a limited domain (i.e. mission critical) and may look discouraging at first glance. In the first analysis we explained our observations with the {\em extraneous factors}, that the results are affected by the irrelevant factors in CC data. In this section we hypothesize this claim and test for its validity. 


\subsection{Design}
In this analysis, we try to construct more homogeneous {\em defect} datasets from CC data in an automated manner. For this purpose we use a simple filtering method (i.e. NN) which is to be described next. The analysis design is given in~\tbl{exp2}. For this analysis, we use all common features available in NASA projects which is a total of
19 features. These features are marked in �NASA Shared� column of Table 4.

\begin{center}
\begin{table}
\caption{Pseudocode for Analysis 2}
\scriptsize
\begin{tabular}{|p{.95\linewidth}|}\hline
\begin{verbatimtab}
DATA = [PC1, KC1, KC2, CM1, KC3, MW1, MC2]
LEARNER = [Naive Bayes]

C_FEATURES <- Find common features IN DATA
FOR EACH data IN DATA
	data = Select C_FEATURES in data
END
REPEAT 100 TIMES
	FOR EACH data in DATA
		WC_TRAIN = random 90% of data
		TEST = data - WC_TRAIN
		CC_TRAIN = DATA - data
		
	//NN-Filtering: Select k=10 neighbors for each validation data
	
		FOR EACH test IN TEST
			dist = L2_DISTANCE{test, CC_TRAIN}
			NNCC_TRAIN <- 10 Samples in CC_TRAIN with min{dist)
		END
		
	//Use only unique samples from the union of all selected neighbors
	
		NNCC_TRAIN = UNIQUE(NNCC_TRAIN)
		
		NNCC_PREDICTOR = Train LEARNER with NNCC_TRAIN
		CC_PREDICTOR = Train LEARNER with CC_TRAIN
		WC_PREDICTOR = Train LEARNER with WC_TRAIN
		[nncc_pd, nncc_pf, nncc_bal] = NNCC_PREDICTOR on TEST 
		[cc_pd, cc_pf, cc_bal] = CC_PREDICTOR on TEST 		
		[wc_pd, wc_pf, wc_bal] = WC_PREDICTOR on TEST 
	END
END
\end{verbatimtab}
\\\hline
\end{tabular}
\label{tbl:exp2}
\end{table}
\end{center}

\subsubsection{Nearest Neighbor (NN) Filtering}
Our idea behind filtering is to collect similar instances together in order to construct a learning set that is homogeneous with the validation set. More formally, we try to introduce a bias in the model by using training data that are similar to the validation data characteristics. While a bias is not desired in general, it is what we seek on purpose, since in our case we can control the bias against removing the noise in CC data (i.e. extraneous factors). We simply use the k-Nearest Neighbor (k-NN) method to measure the similarity between the validation set and the training {\em candidates}. The similarity measure is the Euclidean distance between the static code features of validation and training candidate sets. As mentioned before, we do not perform any feature selection and use all available common features for filtering. The expected outcome of the filtering part is to obtain a subset of available CC data that shows similar characteristics to the local code culture.

The details of NN filter are as follows: We calculate the pairwise Euclidean distances between the validation set and the {\em candidate} training set (i.e. all CC data). Let $N$ be the number of validation set size. For each validation data, we pick its $k =10$ nearest neighbors from candidate training set\footnote{Caveat: We did not optimize the value of $k$ for each project. We simply used a constant $k=10$. We consider it as a future work to dynamically set the value of $k$ for a given project~\cite{baker07}. }. Then we come up with a total of  $10\times N$ similar data points (i.e. module feature vectors). These $10\times N$ samples may not be unique (i.e. a single data sample can be a nearest neighbor of many data samples in the validation set). Using only unique ones, we form the training set and use a random $90\%$ of it for training a predictor. 

Please note that we do not use the class information (i.e. a module is defective or defect-free) while measuring similarity and selecting neighbors. This corresponds to a real life case, where the development of some modules are completed and they are ready for testing: there is no defect related information available, however static code features are collected with automated tools. 

\subsection{Results}
If the results of Analysis \#1 are due to the extraneous factors in CC data (as we suspect), then we would expect lower pf's in NN results than CC results. Therefore, for pd, we define the null hypothesis as:

{ $H_{0}: NN_{pf} \ge CC_{pf} $ 

$H_{0}$ is rejected by the U test with $\alpha=0.05$. Therefore, using NN filtered CC data significantly decreases the false alarms compared to CC data. Yet, we observe that pd's have also decreased. However, false alarm rates are more dramatically decreased than detection rates as seen in \fig{pd1} to \fig{pf2}. For example, in $CM1$ project, the median false alarm rate decreases nearly one third, from 91\% to 33\%, whereas the median detection rate slightly decreases from 94\% to 82\%. In all cases, predictors learned from NN data dramatically reduce the high false alarm rates associated with
the use of cross-company data.
Often that reduction  halves
the false alarm rate. For example, in MW1, the median false alarm rate drops
from 68\% (CC) to 33\% (NN).


\begin{figure}
\renewcommand{\baselinestretch}{0.5}
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

CM1 &1&   CC&98&98&98&98&98&\boxplot{  98.0}{0}{  98.0}{  98.0}{0}\\
&2& NN&76&82&82&84&84&\boxplot{  76.0}{6}{  82.0}{  84.0}{0}\\
&3&   WC&20&60&80&80&100&\boxplot{  20.0}{40}{  80.0}{  80.0}{20}\\\hline


MW1&1&   CC&90&90&90&90&90&\boxplot{  90.0}{0}{  90.0}{  90.0}{0}\\
&2& NN&68&68&68&68&71&\boxplot{  68.0}{0}{  68.0}{  68.0}{3}\\
&2&   WC&0&50&50&75&100&\boxplot{   0.0}{50}{  50.0}{  75.0}{25}\\\hline

PC1&1&   CC&99&99&99&99&99&\boxplot{  99.0}{0}{  99.0}{  99.0}{0}\\
&2& NN&74&77&77&77&78&\boxplot{  74.0}{3}{  77.0}{  77.0}{1}\\
&3&   WC&38&62&62&75&100&\boxplot{  38.0}{24}{  62.0}{  75.0}{25}\\\hline

\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
 Analysis \#2 PD results where
$NN_{pd} \ge WC_{pd}$. 
Rankings computed via Mann-Whitney (95\% 
confidence) comparing each row to all other rows.}\label{fig:pd1}
\end{figure}
\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

KC1 &1&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\
&2&   WC&64&76&82&85&94&\boxplot{  64.0}{12}{  82.0}{  85.0}{9}\\
&3& NN&60&64&65&66&69&\boxplot{  60.0}{4}{  65.0}{  66.0}{3}\\\hline

KC2&1&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\
&2&   WC&45&73&82&91&100&\boxplot{  45.0}{28}{  82.0}{  91.0}{9}\\
&2& NN&77&78&79&79&80&\boxplot{  77.0}{1}{  79.0}{  79.0}{1}\\\hline

KC3&1&   CC&81&81&81&84&84&\boxplot{  81.0}{0}{  81.0}{  84.0}{0}\\
&2&   WC&20&60&80&100&100&\boxplot{  20.0}{40}{  80.0}{ 100.0}{0}\\
&3& NN&60&63&65&67&70&\boxplot{  60.0}{3}{  65.0}{  67.0}{3}\\\hline

MC2&1&   CC&83&83&83&83&85&\boxplot{  83.0}{0}{  83.0}{  83.0}{2}\\
&2&   WC&17&50&67&83&100&\boxplot{  17.0}{33}{  67.0}{  83.0}{17}\\
&3& NN&56&56&56&56&58&\boxplot{  56.0}{0}{  56.0}{  56.0}{2}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
Analysis \#2 PD results where
$NN_{pd} < WC_{pd}$. 
}\label{fig:pd2}
\end{figure}
\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

KC1&1& NN&22&23&24&25&27&\boxplot{  22.0}{1}{  24.0}{  25.0}{2}\\
&2&   WC&26&32&35&37&43&\boxplot{  26.0}{6}{  35.0}{  37.0}{6}\\
&3&   CC&59&60&60&60&60&\boxplot{  59.0}{1}{  60.0}{  60.0}{0}\\\hline

KC2&1& NN&24&25&25&25&27&\boxplot{  24.0}{1}{  25.0}{  25.0}{2}\\
&1&   WC&10&21&26&31&40&\boxplot{  10.0}{11}{  26.0}{  31.0}{9}\\
&2&   CC&67&67&67&67&67&\boxplot{  67.0}{0}{  67.0}{  67.0}{0}\\\hline

KC3&1& NN&17&18&18&19&20&\boxplot{  17.0}{1}{  18.0}{  19.0}{1}\\
&2&   WC&7&17&21&26&31&\boxplot{   7.0}{10}{  21.0}{  26.0}{5}\\
&3&   CC&26&27&27&27&27&\boxplot{  26.0}{1}{  27.0}{  27.0}{0}\\\hline

MC2&1& NN&29&30&31&32&35&\boxplot{  29.0}{1}{  31.0}{  32.0}{3}\\
&2&   WC&0&27&36&45&73&\boxplot{   0.0}{27}{  36.0}{  45.0}{28}\\
&3&   CC&71&71&71&71&71&\boxplot{  71.0}{0}{  71.0}{  71.0}{0}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{Analysis s \#2 PF results 
where $NN_{pf} \le WC_{pf}$.}\label{fig:pf1}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline
CM1&1&   WC&16&29&33&38&49&\boxplot{  16.0}{13}{  33.0}{  38.0}{11}\\
&2& NN&40&43&44&45&46&\boxplot{  40.0}{3}{  44.0}{  45.0}{1}\\
&3&   CC&90&91&91&91&93&\boxplot{  90.0}{1}{  91.0}{  91.0}{2}\\\hline

MW1&1&   WC&8&21&26&29&47&\boxplot{   8.0}{13}{  26.0}{  29.0}{18}\\
&2& NN&30&32&33&33&36&\boxplot{  30.0}{2}{  33.0}{  33.0}{3}\\
&3&   CC&67&68&68&69&70&\boxplot{  67.0}{1}{  68.0}{  69.0}{1}\\\hline

PC1&1&   WC&16&24&28&31&40&\boxplot{  16.0}{8}{  28.0}{  31.0}{9}\\
&2& NN&45&48&48&49&53&\boxplot{  45.0}{3}{  48.0}{  49.0}{4}\\
&3&   CC&94&94&94&94&94&\boxplot{  94.0}{0}{  94.0}{  94.0}{0}\\\hline


\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{  Analysis \#2 PF results where $NN_{pf} > WC_{pf}$.  }\label{fig:pf2}
\end{figure}
Showing that defect predictors learned from NN-filtered CC data are significantly better than the ones learned from CC data, 
the ideal result would be that models based on NN-filtered can be used as a surrogate for WC data based models.
This, in turn, would mean that 
developers could avoid the tedious and expensive work of local data collection. We now investigate the
relation between the NN-filtered CC data and the WC data based defect predictors. 


If defect predictors learned from NN-filtered CC data out-performed the ones learned from WC data then we would expect to notice two observations:
\begin{itemize}
\item {\em Observation1:} NN would have $pd$ values above or equal to WC's $pd$.
The examples displaying $Observation1$ are shown in \fig{pd1}.
\item {\em Observation2:} NN would have $pf$ values below or equal to WC's $pf$.
The examples displaying $Observation2$ are shown in \fig{pf1}.
\end{itemize}
Please note that the conjunction of
$Observation1$ and $Observation2$ is uncommon.
In fact, our results
suggest that $Observation1$ and $Observation2$ are somewhat mutually exclusive:
\begin{itemize}
\item
As shown in Figures \ref{fig:pd1} and \ref{fig:pf2}, the examples where $NN$ increases
the probability of detection are also those where it increases the probability of false alarms.
\end{itemize}
Hence, we cannot recommend NN as a replacement for WC.
Nevertheless, if local WC data are not available, 
then we would recommend processing foreign CC data with NN.


\subsection{Discussions}
In Analysis \#1 we have used random samples of CC data and observed that the false alarm rates substantially increased compared to the WC models. Our new analysis shows that NN filtering CC data removes the increased false alarm rates. Now we argue that, using NN filtering instead of using all available CC data, helps choosing training examples that are similar to problem at hand. Thus, the irrelevant information in non-similar examples is avoided. However, this also removes the rich sample base and yields a slight decrease in detection rates. Mann-Whitney tests reveal that NN filtering is 
\bi
\item far better than random sampling cross company data, 
\item and still worse than using within company data.
\ei

The performances of defect predictors based on the NN-filtered CC vs. WC data do not give necessary empirical evidence to make a strong conclusion. Sometimes NN data based models may perform better than WC data based models. A possible reason may be hidden in the processes that are implemented those projects. May be, a group of new developers were working together for the first time and corresponding WC data included more heterogeneity, which is reduced by NN. May be the development methodology changed during the project, producing a different code culture. However, we do not have access to the internals of these projects that allows a discussion of these observations.


\section{Analysis \#3: What is the smallest amount of local data needed for constructing a model? }
\begin{figure*}
\begin{center}
\begin{tabular}{cc}
\includegraphics[angle=90,width=2in]{incrementalcm1.pdf}&
\includegraphics[angle=90,width=2in]{incrementalkc1.pdf}\\
\includegraphics[angle=90,width=2in]{incrementalkc2.pdf}&
\includegraphics[angle=90,width=2in]{incrementalkc3.pdf}\\
\includegraphics[angle=90,width=2in]{incrementalmc2.pdf}&
\includegraphics[angle=90,width=2in]{incrementalmw1.pdf}\\
\includegraphics[angle=90,width=2in]{incrementalpc1.pdf}&
\end{tabular}
\end{center}
\caption{Results from analysis \#3. Training set size grows in units of 100 examples,
 moving left to right
over the x-axis. The MC2 results only appear at the maximum x-value since MC2 
has less than 200 examples.
}\label{fig:inc}
\end{figure*}

Our results of Analyses \#1 and \#2 reveal that WC data models are better if data are available. In this section,  we will show that defect predictors can be learned from very
small samples of WC data.

\subsection{Design}
An important aspect of the Analyses \#1 and \#2 results is that defect predictors
were learned using only a handful of defective modules.
For example, consider a 90\%/10\% train/test split on
$pc1$ with 1,109 modules, only 6.94\% of which are defective.
On average, the training set will only contain \mbox{$1109 * 0.9 * 6.94/100=69$}
defective modules. Despite this, $pc1$ yields an adequate median $\{pd,pf\}$ results
of $\{63,27\}$\%. 

Analysis \#3 was therefore designed to check the smallest amount of data needed
to learn defect predictors. The design is given in~\tbl{exp3}. Analysis \#3 is essentially the same
as the first analysis, but
without the
cross-company study. Instead,
analysis \#3
took the 7 NASA projects of \tbl{data1}
and 
learned predictors using:
\bi
\item reduced WC data:  a randomly selected
subset of up to 90\% of each project data.
\ei
After randomizing the order of the data, training sets
were built using just the first 100, 200, 300,
\ldots data samples in the project.
After training the defect predictor, its performance is tested on the remaining data samples
not used in training.

\begin{center}
\begin{table}
\caption{Pseudocode for Analysis 3}
\scriptsize
\begin{tabular}{|p{.95\linewidth}|}\hline
\begin{verbatimtab}
DATA = [PC1, KC1, KC2, CM1, KC3, MW1, MC2]
LEARNER = [Naive Bayes]

REPEAT 100 TIMES
 FOR EACH data IN DATA
    WC_TRAIN = random 90% of data
   	TEST = data - WC_TRAIN		
   	FOR i IN {100, 200, 300, ...}
   	    WC_TRAIN_INCREMENTAL <- Random i Examples from WC_TRAIN
   	    WC_INC_PREDICTOR = Train LEARNER with WC_TRAIN_INCREMENTAL
   		[iwc_pd, iwc_pf, iwc_bal] = WC_INC_PREDICTOR on TEST 
   	END
  END
END
\end{verbatimtab}
\\\hline
\end{tabular}
\label{tbl:exp3}
\end{table}
\end{center}


Analysis \#1 only used the features found in all NASA projects.
For this analysis, we imposed no such restrictions and used whatever features
were available in each data set.
\subsection{Results from Analysis \#3}
\eq{bal} defined ``balance'' to be a combination of $\{pd,pf\}$
that decreases if $pd$ decreases or $pf$ increases.
As shown in \fig{inc}, there was very little change in balanced 
performance  after learning from 100,200,300,... examples.
Indeed, there is some evidence that
learning from larger training sets had detrimental effects: the more
training data, the larger the variance in the performance of the learned predictor.
Observe how, in $kc1$ and $pc1$, as the training set size increases (moving right along the x-axis)
the dots showing the balance performance start spreading out.
We conjecture that this effect is due to the presence of occasional larger outliers in the training
data (the probability of discovering these increases as the training set grows inside and, if the learner
trains on them, then the resulting theory has more variance.)

The Mann-Whitney U test was applied to check the visual trends seen in \fig{inc}.
For each project, all results from training sets of size 100,200,300\ldots were
compared to all other results from the same project. The issue was ``how much data are enough?''
 i.e.
what is the $min$imum
training set size that never lost to other training set of a larger size. Usually, that 
$min$ value 
was quite small:

\bi
\item In five tables $\{cm1,kc2,kc3,mc2,mw1\}$, $min=100$;
\item In  \{$kc1,pc1$\}, $min=\{200,300\}$ instances, respectively.
\ei


We explain the analysis \#3 results as follows.
This analysis used simplistic static code features such as lines of code, number of
unique symbols in the module, etc.  
Such simplistic static code features are hardly a complete characterization of
the internals of a function.
We would characterize such static code features as having 
{\em limited information content}~\cite{me08d}. 
Limited content
is soon exhausted by repeated sampling.
Hence, such simple features reveal all they can reveal after
a small sample.

\begin{figure}
\begin{center} 
\includegraphics[width=2.4in]{plateaudots.pdf}
\includegraphics[width=2.4in]{plateaudotsC.pdf} 
\end{center}
\caption{Y-axis shows plateau point after learning from data sets
that have up to $X$ examples (from~\cite{orrego04}).
The left plot shows results from using
Naive Bayes (nbk) or a decision tree learner (j48)~\cite{quinlan92}
to predict for discrete classes. Right plot shows results from
using linear regression (lsr) or model trees (m5)~\cite{quinlan92b}
to learn predictors for continuous classes. In this study, data
sets were drawn from the UC Irvine data repository~\cite{Blake+Merz:1998}.}\label{fig:plateau} 
\end{figure}

\subsection{Checking the  Analysis \#3 Results}


There is also some evidence that the  results of Analysis \#3 (that
performance improvements stop after a few hundred examples) have
been seen previously in the data mining literature. To the
best of our knowledge, this is the first report of this effect in the
defect prediction literature. \bi \item In their discussion on how
to best handle numeric features, Langley and John offer plots of
the accuracy of Naive Bayes classifiers after learning on 10,20,40,..200
examples. In those plots, there is little change in performance
after 100 instances~\cite{john95}.  \item Orrego~\cite{orrego04}
applied four data miners (including Naive Bayes) to  20 data sets
to find the {\em plateau point}: i.e. the point after which there
was little net change in the performance of the data miner.  To
find the plateau point, Oreggo used t-tests to compare the results
of learning from $Y$ or $Y+\Delta$ examples. If, in  a 10-way
cross-validation, there was no statistical difference between $Y$
and $Y+\Delta$, the plateau point was set to $Y$.  As shown in
\fig{plateau}, many of those plateaus were found at $Y\le100$ and
most were found at $Y\le200$. Please note that these plateau sizes are
consistent with the results of Analysis \#3.  \ei 

\subsection{Discussion of Analysis \#3}
In the majority of cases, predictors learned from as little as one hundred
examples perform as well as predictors learned from many more examples.
This suggests that the effort associated with learning defect predictors
from within-company data may not be overly large.
For example, \tbl{estimate} estimates that the effort required to build and test
100 modules may be as little as 2.4 to 3.7 person months. However, practitioners should use this approach cautiously. The populations of one hundred examples in our experiments are randomly selected from completed projects {\em with} stratification. Therefore, in practice, any one hundred sample may not necessarily reflect the company characteristics and constructing this initial set may take longer than expected.

\begin{table}
\caption{An estimate of the effort required to build and test 100 modules.}\label{tbl:estimate}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
100  modules may take as little as two to four person months to construct. This estimate
was generated as follows:
\bi
\item
In the $cm1$ data base, the median module size is 17 lines. 100 randomly
selected modules would 
therefore use 1700 LOC.
\item
To generate an effort estimate for these modules,
we used the on-line COCOMO~\cite{boehm00a} effort estimator (\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/expert_cocomo2000.html}). Estimates were generated
assuming 1700 LOC and
the required reliability varying from very low to very high.
\item
The resulting estimates ranged from
between 2.4 and 3.7 person months to build and test those modules.
\ei\\\hline
\end{tabular}
\end{table}

\section{Replication: Can our results be generalized?}
\label{sec:exp4}
Analyses  \# 1 to \# 3 were based on NASA projects.
For the external validity of the conclusions of those Analyses,
we replicate the same analyses on SOFTLAB projects of \tbl{data1}. 

For each SOFTLAB project, we follow the same procedure as in Analyses \# 1 and \#2; i.e.
10\% of the rows of each data set are selected at random for constructing test sets. Then three different types of defect predictors are constructed. First type are defect predictors trained with cross company data (i.e. all 7 NASA projects). Second type of defect predictors are trained with within company data (i.e. random 90\% rows of remaining SOFTLAB projects\footnote{In order to reflect the use in practice, we do not use the remaining 90\% of the same project for training, we rather use a random 90\% of data from other projects. Please note that all WC analysis in this paper reflects within-company, not within project simulations. Since SOFTLAB data are collected from a single company, learning a predictor on some projects and to test it on a different one does not violate within company simulation.}). Finally, the third type are defect predictors trained with nearest neighbor filtered cross company data (i.e.  {\em similar} rows from 7 NASA tables).

The SOFTLAB projects include 29 static code features, 17 of which are
 common with the NASA projects. In order to simplify the comparison between these new
projects and Analyses \#1 and \#2, 
 we used only these shared attributes in our CC analyses. 
On the other hand we use all available features in WC analyses for SOFTLAB projects. 
In the following external validity analysis, we treated each NASA project as 
cross- company data for SOFTLAB projects. 

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Overall results on SOFTLAB data}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  95&  100& 100& \boxplot{88}{0}{95}{100}{0}\\
    & WC&       35&  40&  88&   100& 100& \boxplot{35}{5}{88}{100}{0}\\\hline
 pf & CC&       52&  59&  65&   68& 68& \boxplot{52}{7}{65}{68}{0}\\
    & WC&        3&  5&  29&   40&  42& \boxplot{3}{2}{29}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  88&  88& 88& \boxplot{88}{0}{88}{88}{0}\\
    & WC&       88&  88&  88&   88& 88& \boxplot{88}{0}{88}{88}{0}\\\hline
 pf & CC&       62&  65&  65&   65& 65& \boxplot{62}{3}{65}{65}{0}\\
    & WC&        40&  40&  40&   40&  42& \boxplot{40}{0}{40}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar4$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       95&  95&  95&  95& 95& \boxplot{95}{0}{95}{95}{0}\\
    & WC&       35&  40&  40&   40& 40& \boxplot{35}{5}{40}{40}{0}\\\hline
 pf & CC&       52&  55&  56&   59& 60& \boxplot{52}{3}{56}{59}{1}\\
    & WC&        3&  3&  3&   5&  5& \boxplot{3}{0}{3}{5}{0}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar5$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       100&  100&  100&  100& 100& \boxplot{100}{0}{100}{100}{0}\\
    & WC&       88&  100&  100&   100& 100& \boxplot{88}{12}{100}{100}{0}\\\hline
 pf & CC&       57&  68&  68&   68& 68& \boxplot{57}{11}{68}{68}{0}\\
    & WC&        29&  29&  29&   29&  29& \boxplot{29}{0}{29}{29}{0}\\
\hline\end{tabular}
}\\
\end{tabular}
\end{center}
\caption{Analysis \#1 results for the SOFTLAB projects. Overall and individual results are shown respectively.}\label{fig:ccwcturkey}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   CC&35&90&100&100&100&\boxplot{  35.0}{55}{ 100.0}{ 100.0}{0}\\
&2& NN&65&65&70&70&70&\boxplot{  65.0}{0}{  70.0}{  70.0}{0}\\
&3&   WC&35&40&40&40&45&\boxplot{  35.0}{5}{  40.0}{  40.0}{5}\\\hline

AR3&1&   CC&75&88&88&88&88&\boxplot{  75.0}{13}{  88.0}{  88.0}{0}\\
&1& NN&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\
&1&   WC&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\\hline

AR5&1&   CC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\
&1& NN&100&100&100&100&100&\boxplot{ 100.0}{0}{ 100.0}{ 100.0}{0}\\
&1&   WC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
 Analysis \#2 PD results for the SOFTLAB projects where
$NN_{pd} \ge WC_{pd}$. 
Rankings computed via Mann-Whitney (95\% 
confidence) comparing each row to all other rows.}\label{fig:arpd1}
\end{figure}



\begin{figure}[!t]
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR3&1& NN&38&38&38&38&40&\boxplot{  38.0}{0}{  38.0}{  38.0}{2}\\
&2&   WC&40&40&40&40&42&\boxplot{  40.0}{0}{  40.0}{  40.0}{2}\\
&3&   CC&38&55&56&60&80&\boxplot{  38.0}{17}{  56.0}{  60.0}{20}\\\hline

AR5&1& NN&21&25&25&25&32&\boxplot{  21.0}{4}{  25.0}{  25.0}{7}\\
&2&   WC&29&29&29&29&29&\boxplot{  29.0}{0}{  29.0}{  29.0}{0}\\
&3&   CC&29&46&50&50&79&\boxplot{  29.0}{17}{  50.0}{  50.0}{29}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{Analysis \#2 PF results for the SOFTLAB projects
where $NN_{pf} \le WC_{pf}$.}\label{fig:arpf1}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   WC&3&3&3&5&7&\boxplot{   3.0}{0}{   3.0}{   5.0}{2}\\
&2& NN&24&25&25&25&28&\boxplot{  24.0}{1}{  25.0}{  25.0}{3}\\
&3&   CC&6&47&62&67&78&\boxplot{   6.0}{41}{  62.0}{  67.0}{11}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{  Analysis \#2 PF results for the SOFTLAB projects where $NN_{pf} > WC_{pf}$.  }\label{fig:arpf2}
\end{figure}


Figure \ref{fig:ccwcturkey} shows the results:
\bi
\item
The {\em pd} values for CC data increase compared to WC data
with the cost of increased {\em pf}. 
\item
CC data shifts \{Q1,
median\} of {\em pf} from \{5, 29\} to \{59, 65\}.  
\item
For CC data:
\bi
\item 25\% of the $pd$ values are at 100\%.
\item 50\% of the $pd$ values are above 95\%
\item And all the $pd$ values are at or over 88\%.
\ei
\ei

These results also provide evidence for the validity of our conclusions for Analysis \# 3.
In Analysis \# 3, we conclude that the minimum number of instances for training
a defect predictor is around $100-200$ data samples. Please note that three SOFTLAB projects
{\em ar3, ar4} and {\em ar5} have \{63,107, 36\} modules respectively. Thus, the minimum number of training samples occurs when a predictor is trained from {\em (ar3 $+$ ar5)} projects. In this case, only $(63+36) * 0.90 = 90$ training samples are used to construct a defect predictor for the {\em ar4} project. Similarly, the maximum number of training samples occurs when a predictor is trained from {\em (ar3 $+$ ar4)} projects. Then $(63+107) * 0.90 = 153$ training samples are used to construct a defect predictor for the {\em ar5} project. Therefore, the WC results in Figure
\ref{fig:ccwcturkey} are achieved using a minimum of $90$ and a maximum of $153$ training samples.

\fig{arpd1} to \fig{arpf2} shows {\em Observation1} (i.e. $NN_{pd} \ge WC_{pd} $) and {\em Observation2} (i.e. $NN_{pf} \le WC_{pf} $) for SOFTLAB projects. Please recall that these observations were mutually exclusive for NASA data. The pattern is similar in SOFTLAB projects:
\bi
\item for {\em ar4} mutual exclusiveness hold: $NN_{pd} \ge WC_{pd}$ and $NN_{pf} > WC_{pf}$
\item  for {\em ar3} and {\em ar5}: $NN_{pf} \le WC_{pf}$. If the observations were mutually exclusive, we would expect  $NN_{pd} < WC_{pd}$. While this is not the case for {\em pd}, we observe that $NN_{pd} = WC_{pd}$  and $NN_{pd} \not > WC_{pd}$  (see \fig{arpd1}).
\ei

In summary, 
   the WC, CC and NN patterns found in American NASA rocket software
   are also observed in software controllers of Turkish domestic appliances. While this is not 
   the definitive proof of the external validity of our
   results, we find it a very compelling result that is reproducable in different companies.

\section{Related Work}
\subsection{Neighborhood Reasoning}
Our use of nearest neighbor is somewhat similar to {\em k-NN classifier} or
{\em reasoning by analogy}.
A {\em k-NN classifier} returns the majority class in the ``k'' nearest neighbors to the 
test instance. Lessman et.al. benchmark k-NN against 19 other learners and found
that it was one of the four worst methods for defect prediction~\cite{lessmann09}.

In {\em reasoning by analogy}, inference about some new
case is made by partial match to prior cases, then applying some repair or generalization
procedure to the matched cases.
Like k-NN, reasoning by analogy makes use of some distance metric that finds old
cases similiar (i.e. nearer) to the the new case. While reasoning by analogy is a common
effort estimation method (e.g.~\cite{shepperd97}), it is far less common
to see it applied to defect prediction.
One reason for this is the typical size of the training set: effort estimation
typically deals with dozens of prior examples while defect training sets can have
thousands of examples. We conjecture that as the data set grows in size,
simple induction over many instances may be more powerful than complex analogical
reasoning over a handful of examples.

Our work is most similar to that of 
Khoshgoftaar and Seliya~\cite{khoshgoftaar03} who augmented k-NN with a counting
procedure that was a slighly more
sophisticated decision procedure than ``return the majority class''. Our approach differs
to theirs in that we 
bundle together the k-nearest neighbors, then pass that bundle to
a full-fledged classifier (Naive Bayes).

%\subsection{WC vs CC in Effort Estimation}
%The value of WC vs CC data has been 
%investigated in the effort estimation community, with inconclusive results:
%\bi
%\item
%Mendes et al.~\cite{mendes07} found within-company data
%performed much better than cross-company data for predicting
%estimation effort of web-based projects.  They only recommend using
%cross-company data in the special case when that ``data are obtained
%using rigorous quality control procedures''.  
%\item
%A similar conclusion was reached by Abrahamsson et al.
%who  discussed learning effort predictors in the context
%of an agile development process~\cite{abrahamsson07}. They strongly advocate
%the use of WC-data.
%\ei
%However,
%other studies are not so clear in their conclusions:
%\bi
%\item
%MacDonell \& Shepperd tried
%to find trends in a set of papers relating to project 
%management and effort estimation~\cite{macdonell07}. However, the
% papers studied by MacDonell \& Shepperd used a wide range of data
%sets so these authors found it hard to offer  a definitive combined
%conclusion.  
%\item
%In other work, after a review of numerous case studies, 
%Kitchenham
%et al.~\cite{kitch07} concluded that the value of CC vs WC data for effort estimation
%is
%unclear: \begin{quote} {\em 
%{\ldots}some organizations would benefit from using models derived from
%cross-company benchmarking databases but others would not}~\cite{kitch07}.
%\end{quote}
%\ei
%How can these contradictory results be explained?
%One possibility is that 
%effort estimation requires the collection of project data, some of which have
%ambiguous definitions. For example, one of the features of the COCOMO-family~\cite{boehm00a}
%of effort predictors is ``applications experience'' (aexp).
%According to one on-line source\footnote{\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/drivers.html}}, this feature is defined as follows: 
%``the project team's equivalent level of experience with 
%{\em this type} of application''. 
%No guidance is offered regarding how to characterize ``this type of application''.
%Hence, there is some degree of ambiguity in this definition.
%We conjecture that the ambiguity of the effort estimation features is one reason
%for the variance in the results reported by MacDonell \& Shepperd and Kitchenham et al.~\cite{macdonell07, kitch07}.


\subsection{On Defect Prediction Using Static Code Features}\label{sec:dp}


We study defect predictors learned from
static code attributes since they are {\em useful}, {\em easy to
collect}, and {\em widely-used}.


{\em Useful}: 
Defect predictors are considered useful, if they provide a prediction performance that is comparable to or better than manual reviews \cite{moser08}. 
This paper  finds defect predictors
with a probability of detection of 80\%, or higher.
This is 
higher than currently-used
industrial
methods such as manual code reviews:
\bi
\item
A {\em IEEE Metrics
2002}~\cite{shu02} panel
concluded that manual software  reviews can find ${\approx}60\%$ 
of defects\footnote{That panel supported neither Fagan's
claim~\cite{fagan86} that inspections can find 95\% of defects before
testing or Shull's claim that specialized directed inspection methods
can catch 35\% more defects that other
methods~\cite{shull00a}.}  
\item
In 2004, Raffo (personnel communication) reports
that the defect detection capability of
industrial review methods can vary from \mbox{probability of detection: $pd=TR(35, 50,
65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
min/mode/max of $a,b,c$.}.
 for full Fagan inspections~\cite{fagan76} to
\mbox{ $pd=TR(13, 21, 30)\%$} for less-structured inspections.
\ei
Further, defect predictors based on static code features are considered as static analysis, since they do not require the execution of code. Zheng et.al. compares automated static analysis (ASA) with manual inspections on Nortel software and conclude that {\em
...our results indicate that ASA is an economical complement to other verification and validation techniques}


{\em Easy to collect}: 
Employing defect predictors in practice should not take too much time \cite{moser08} for data collection and constructing the models themselves. 
An advantage of static code features is that they can be quickly and automatically collected
from the source code, even if no other information is available.
Static code attributes like lines of code
and  the McCabe/Halstead
attributes can be automatically and cheaply collected, even for very large systems~\cite{conf/icse/NagappanB05a}.
By contrast, other methods such as manual code reviews are labor-intensive;
e.g.
Each member of a review team can inspect 8 to 20 LOC/minute~\cite{me02f}. 
Furthermore,
other features (e.g. number of developers,
the software development practices used to develop the code) may be unavailable
or hard to characterize.

{\em Widely used}:
Many researchers use static features to guide software 
quality predictions  
(see \cite{halstead77,mccabe76,chapman02,me04g,polyspace,hall00,nikora03,conf/icse/NagappanB05a, Khoshgoftaar:2004p1877,conf/ictai/TangK04,porter90,tiang95,srinivasan95}).
Verification and validation (V\&V) textbooks
advise using static code features
to select modules worthy of manual inspections \cite{rakitin01}.  


\begin{table}[!t]
\caption{Some representative $pd$s and $pf$s for prediction problems
 from the UC Irvine machine learning
database~\protect\cite{Blake+Merz:1998}. These values were generated
using the
standard
settings of a state-of-art decision tree learner (J48). For each data set, ten
experiments were conducted, where a decision tree was learned on 90\% of
the data, then tests are done of the remaining 10\%. 
The numbers shown here are the average results across ten such experiments.}\label{tbl:uci}
\scriptsize
\begin{center}
\begin{tabular}{r|rr|}\cline{2-3}
\multicolumn{1}{c|}{}&\multicolumn{2}{c|}{probability of}\\\hline
data&	detection&	false alarm\\\hline
pima diabetes&	60&	19\\
sonar	&	71	&29\\
horse-colic	&71	&7\\
heart-statlog &	73	&21\\
rangeseg	&76	&30\\
credit rating	&88&	16\\
sick	&88&	1\\
hepatitis	&94	&56\\
vote	&95	&3\\
ionosphere	&96	&18\\\hline
mean &81	& 20\\\cline{2-3}
\end{tabular}
\end{center}
\end{table}



Nevertheless, there are many reasons to doubt the value
of static code attributes for defect prediction. Static code attributes 
are hardly a complete characterization of
the internals of a function. Fenton offers an insightful example where
{\em the same} functionality is achieved using {\em different}
programming language constructs resulting in {\em different} static
measurements for that module~\cite{fenton96}.  
Fenton uses this example to argue the uselessness
of static code attributes. 
Further,
Fenton~\&~Pfleeger  
note that the main McCabe's attribute
(cyclomatic complexity, or $v(g)$) 
is highly correlated with lines of code~\cite{fenton96}.
Shepperd~\&~Ince repeated that result, commenting that ``for a large class of software it
(cyclomatic complexity) is no more than a proxy for, and in many cases
outperformed by, lines of code''~\cite{shepperd94}.

However, if 
the above criticisms are correct then we would predict
that,
in general, the performance of a defect predictor learned by a data miner should be very poor.
More specifically,
the supposedly better static code attributes such as Halstead and Mccabe should
perform no better than just simple thresholds on lines of code. 

Neither of these predictions are true, at least for the data sets used in this study.
The defect predictors learned from static code attributes
perform surprisingly well.
Formally, learning a defect predictor is a {\em binary prediction problem} where each
module in a project has been labeled as "defect-free" or "defective".
The learning problem is to build some predictor which guesses the labels
for as-yet-unseen modules. In this paper we find predictors (i.e. for SOFTLAB projects) with
a probability of detection (pd) and probability of false alarm (pf) of
\[median(pd) = 88; median(pf) = 29\]

\tbl{uci} lets us compare our new results against
standard binary prediction results from the UC Irvine machine learning
repository of standard test sets for data miners~\protect\cite{Blake+Merz:1998}.
Our $median(pd,pf)$
are very close to
the standard results of 
 $mean(pd,pf)$=(81\%,20\%) 
which is noteworthy in two ways:
\be
\item It is unexpected. If static code attributes capture so little
about source code (as argued by 
Shepherd, Ince, Fenton and Pfleeger), then we would expect lower probabilities
of detection and much higher false alarm rates. 
\item
These $(pd,pf)$ results are 
better than currently
used industrial methods such as the $pd{\approx}60\%$ reported
at the 2002 IEEE Metrics panel or the $median(pd)=[21..50]$ reported
by Raffo\footnote{
Please note that we can only compare the defect {\em detection}
properties of automatic vs manual methods.
Unlike automatic defect
prediction via data mining, the above manual inspection methods
don't just report ``true,false'' on a module, Rather, the manual
methods also provide specific debugging information.  Hence, a
complete comparison of automatic vs manual defect prediction would
have to include both an analysis of the time to {\em detect} potential
defects {\em and} the time required to {\em fix} them.
Manual methods might score higher
to automatic methods
since they can offer more clues back to the developer about
what is wrong with the method.  However, such an analysis is beyond
the scope of this paper.
Here, we focus only on the relative merits of different methods for 
{\em predicting} error prone modules.
}.
\ee

\section{Practical Implications}	
While
\tbl{uci} shows that our defect detectors work nearly as well as standard data mining methods,
it does not necessarily demonstrate that 
false alarm rates of around $29\%$ 
are useful in an industrial context. 
For example, Arisholm and Briand have certain concerns on the practical usage of defect predictors~\cite{arisholm06}. They argue that
if X\% of the modules are predicted to be faulty and if those modules contain less than X\% of the defects,
then the costs of generating the defect predictor is not worth the effort.

Let us analyze the testing efforts on MW1 project from Arisholm and Briand's point of view. For MW1, there are a total of 403 modules with 31 defective and 372 defect-free ones. CC model yields 90\% pd and 68\% pf, and one should examine 280 modules, which is around a 31\% reduction in inspection efforts compared to examining all modules. Yet, we argue that 68\% pf rate is quite high and using NN we are able to reduce it to 33\% along with 68\% pd. This corresponds to examining 144 modules, a reduction of 47\% compared to exhaustive testing (and we assume an exhaustive test should examine 274 modules for detecting 68\% defects, as Arisholm and Briand suggests). 

This analysis can be extended for all projects used in this paper.
For instance, the company from which SOFTLAB data in \tbl{data1} are collected is keen to use our detectors, arguing that they
operate
	in a highly competitive market segment where profit margins are
	very tight. Therefore reducing the cost of the product even by 1\%
	can make a major difference both in market share and profits. 
Their
	applications are embedded systems where, over the last decade, the software components
	have taken precedence over the hardware. Hence
	their problem is a software engineering problem. 
According to Brooks~\cite{brooks95}, 
half the cost of software development is in unit and systems testing. 
The company also believes that 
	their main challange is
	the testing phase and they seek predictors that indicate
	where the defects might exist {\em before} they start testing. Their expectation from the predictor is not to detect all defects, but to guide them to the problematic modules so that they can detect more defects in shorter times. Hence, any reduction in their testing efforts allows
them to efficiently use their scarce resources. 

Considering the results of our analyses, if a company lacks local data,
we would suggest a two-phase approach.
In phase one, that organization uses imported CC data, filtered via NN.
Also, during phase one, the organization should start a data collection program to collect
static code attributes. Phase two commences when there is {\em enough} local WC data to learn
defect predictors. During phase two, the organization would switch to new defect predictors
learned from the WC data. 

An important issue worth more mentioning is the concern about the time required for setting up a metric program (i.e. in order to collect data for building
actual defect predictors).
Our incremental WC results suggest that, in the case of defect prediction,
this concern may be less than previously believed. 
Kitchenham {\em et al.}~\cite{kitch07} argue that
organizations use
cross-company data since within-company data
can be so hard to collect:
\bi
\item
The time required to collect enough data on past projects from within a  company
may be prohibitive.
\item
Collecting within-company data may take so long
that 
technologies change and older
projects do not represent current practice.
\ei
In our analysis we observe that as few as 100 modules are enough to learn
adequate defect predictors. 
When so few examples are enough,
it is possible that
projects can learn local defect predictors that are relevant to their
current technology in just a few months. 

Further, our experiences with our industry partners show that data collection is not necessarily a major concern. Static code attributes can be automatically and quickly collected with relatively little effort. We have found that when there is high level management commitment, it becomes a relatively simple process. For the three projects of SOFTLAB data, neither the static code attributes, nor the mapping of defects to software modules were available when the authors attempted to collect these data. Since these were smaller scale projects, it was sufficient to spend some time with the developers and going through defect reports. 
Although not all projects have 100 modules individually, the company has a growing repository from several projects and enough data to perform defect prediction.

We also have experience with a large scale telecommunication company, where a long-term metric program for monitoring complex projects (around 750.000 lines of code) requires introducing automated processes. Again with high level management support, it was possible to employ appropriate tool support and these new processes were introduced easily and invisibly to the staff. For that project, we have now a growing repository of defects mapped with source code (around 25 defects per month). Software in that project are being developed for more than 10 years and have very low defect rates. We have obtained the first results in the $8^{th}$ month of the project and it is planned to be completed in $12$ months. In summary setting up a metric program for defect prediction can be done more quickly than it is perceived.

\section{Threats to validity}
In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem work best for the 
\tbl{data1} data~\cite{me07b}.
Since that study, we have tried to find  better data mining
algorithms for defect prediction. To date, we have failed. 
Other researchers have also failed to improve our results. For example, Lessmann et al. 
investigate the statistical difference of the results between 19 learners, including naive Bayes, on the NASA projects~\cite{lessmann09}. \fig{lessmann} shows that the simple Bayesian method discussed above ties in
first place along with 15 other methods.


\begin{figure}
\begin{center}
\includegraphics[width=3in]{lessmann.pdf}
\end{center}
\caption{x-axis shows the rank of the data miners given in the y-axis. All the methods
whose top ranks are 4 to 12 are statistically insignificantly different.
From~\cite{lessmann09}.
}\label{fig:lessmann}
\end{figure}

Our analyses do not involve a value-based approach. We do not consider that fixing defects in a module may be more critical than fixing defects in another one. A reason is that our data sources do not consistently include 
necessary data (i.e. the severity level of defects). If and when data are available, same kind of analysis should be repeated considering this issue. 

Another point is that descriptions of software modules {\em only} in terms of static
code attributes can overlook some important aspects of software
including:
the type of application
domain; the skill level of the individual
programmers involved in system development;
 contractor development practices;
the variation in measurement practices;
and the validation of the measurements and instruments used
to collect the data.
For this reason some researchers augment, or replace
static code measures with other information such as the history
of past faults or changes to code or number of
developers who have worked on the code~\cite{graves00}. Yet again, we have used all available data in our data sources, which are the static code features of the projects.
 
The external validity of generalizing from NASA examples
has been discussed elsewhere~\cite{me07b}. In summary, 
NASA uses 
contractors who are contractually obliged to demonstrate
their understanding and usage of current industrial best
practices. These contractors service many other industries;
for example, Rockwell-Collins builds systems for many government and
commercial organizations.  For these reasons, other noted researchers
such as Basili, Zelkowitz, et al.~\cite{basili02} have argued that
conclusions from NASA data are relevant to the general software
engineering industry.

Nevertheless, it is always wise to test claims of external validity. Hence:
\bi
\item
The SOFTLAB data were, initially,  in reserve.
 Our first three analyses
are based solely on the aerospace applications found in the NASA data.
\item
Our last analysis checked if the  SOFTLAB data exhibit the same pattern as the 
NASA data.
\ei

SOFTLAB data sets were deliberately chosen to be as
far removed as possible from American aerospace software applications
(i.e. Turkish home appliances controller software).
Please note that this software was developed via methods that are both
culturally and organizationally different to NASA aerospace software. Turkish domestic appliances company software are developed by a small team of 2-3 people. The development is carried out in an ad-hoc, informal way rather than formal, process oriented approach in NASA. Furthermore, the company is a profit and revenue driven commercial organization, whereas NASA is a cost driven government entity. This implies that our approach is widely applicable among different development practices. More precisely, our approach is independent of the processes that yield the final product, at least for the wide range of projects that we have analyzed.  



\section{Conclusion}
In this study, we have analyzed defect predictors of static code features, constructed by cross company vs. within company data and found clear, unambiguous conclusions:
\bi
\item CC-data dramatically increase the probability of detecting defective modules (i.e. from median value 75 to 92);
\item But CC-data also dramatically increase the false alarm rate (i.e. from median value 29 to 64). 
\item NN-filtering CC data avoids the high false alarm rates by removing irrelevancies in CC data (i.e. from median value 64 to 32). This removal takes place by automatically selecting similar project data in terms of available static code features and discarding non-similar ones.  
\item Yet WC-data models are still the best and they can be constructed with small amounts of data (i.e. 100 examples).
\ei
Interpreting these in terms of the posed questions in the introduction, we conclude that:
\bi
\item CC data are useful in extreme cases such as mission critical projects, where the cost of false alarms can be afforded. Therefore, CC data should be used only when WC data are not available.
\item Pruning CC data with NN-filter allows the use of CC data for constructing practical defect predictors in other domains. NN-filtered CC data yields much better results than raw CC data, yet closer but worse results than WC data.
\item The best option of using WC data requires the collection of a mere hundred examples from within a company and can be done in a short time (i.e. a few months).
\item We observe the same patterns not only in aerospace software from NASA, but also in software from a completely different company located in another country. While this is a strong evidence of generality, we take great care {\em not to} interpret it as a proof of external validity. 
\ei


We conclude our findings by proposing a two-phase approach for initiating defect prediction process:
\bi
\item {\em Phase1:} Use NN filtered CC data to make local predictions and start to collect WC data.
\item {\em Phase2:} After a few hundred examples are available in the local repository (usually a few months), discard the predictor learned on CC data and switch to those learned from WC data.
\ei 
\begin{acknowledgements}
This research was supported in part by Turkish Scientific Research Council (TUBITAK) under grant \#EEEAG 108E014 and was partially funded by a National Science Foundation grant \#CCF-0810879.
\end{acknowledgements}
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@rmstyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}

\bibitem{me07c}
G.~Boetticher, T.~Menzies, and T.~Ostrand, ``The {PROMISE} {R}epository of
  {E}mpirical {S}oftware {E}ngineering {D}ata,'' 2007,
  \url{http://promisedata.org/repository}.

\bibitem{Arisholm:2006p516}
\BIBentryALTinterwordspacing
E.~Arisholm and L.~Briand, ``Predicting fault-prone components in a java legacy
  system,'' \emph{ISESE '06: Proceedings of the 2006 ACM/IEEE international
  symposium on International symposium on empirical software engineering}, Sep
  2006. [Online]. Available:
  \url{http://portal.acm.org/citation.cfm?id=1159733.1159738}
\BIBentrySTDinterwordspacing

\bibitem{Bell:2006p517}
\BIBentryALTinterwordspacing
R.~Bell, T.~Ostrand, and E.~Weyuker, ``Looking for bugs in all the right
  places,'' \emph{ISSTA '06: Proceedings of the 2006 international symposium on
  Software testing and analysis}, Jul 2006. [Online]. Available:
  \url{http://portal.acm.org/citation.cfm?id=1146238.1146246}
\BIBentrySTDinterwordspacing

\bibitem{Ostrand:2007p519}
\BIBentryALTinterwordspacing
T.~Ostrand, E.~Weyuker, and R.~Bell, ``Automating algorithms for the
  identification of fault-prone files,'' \emph{ISSTA '07: Proceedings of the
  2007 international symposium on Software testing and analysis}, Jul 2007.
  [Online]. Available:
  \url{http://portal.acm.org/citation.cfm?id=1273463.1273493}
\BIBentrySTDinterwordspacing

\bibitem{me07e}
T.~Menzies, A.~Dekhtyar, J.~Distefano, and J.~Greenwald, ``Problems with
  precision,'' \emph{IEEE Transactions on Software Engineering}, September
  2007, \url{http://menzies.us/pdf/07precision.pdf}.

\bibitem{chen05}
Z.~Chen, T.~Menzies, and D.~Port, ``Feature subset selection can improve
  software cost estimation,'' in \emph{PROMISE'05}, 2005, available from
  \url{http://menzies.us/pdf/05/fsscocomo.pdf}.

\bibitem{dekhtyar04}
A.~Dekhtyar, J.~H. Hayes, and T.~Menzies, ``Text is software too,'' in
  \emph{International Workshop on Mining Software Repositories (submitted)},
  2004, available from \url{http://menzies.us/pdf/04msrtext.pdf}.

\bibitem{jiang07}
Y.~Jiang, B.~Cukic, and T.~Menzies, ``Fault prediction using early lifecycle
  data,'' in \emph{ISSRE'07}, 2007, available from
  \url{http://menzies.us/pdf/07issre.pdf}.

\bibitem{nagappan05}
N.~Nagappan and T.~Ball, ``Static analysis tools as early indicators of
  pre-release defect density,'' in \emph{ICSE 2005, St. Louis}, 2005.

\bibitem{marcus08}
A.~Marcus, D.~Poshyvanyk, and R.~Ferenc, ``Using the conceptual cohesion of
  classes for fault prediction in object-oriented systems,'' \emph{Software
  Engineering, IEEE Transactions on}, vol.~34, no.~2, pp. 287--300, March-April
  2008.

\bibitem{musa87}
J.~Musa, A.~Iannino, and K.~Okumoto, \emph{Software Reliability: Measurement,
  Prediction, Application}.\hskip 1em plus 0.5em minus 0.4em\relax McGraw Hill,
  1987.

\bibitem{boehm88}
B.~Boehm and P.~Papaccio, ``Understanding and controlling software costs,''
  \emph{IEEE Trans. on Software Engineering}, vol.~14, no.~10, pp. 1462--1477,
  October 1988.

\bibitem{goseva07}
``Architecture-based software reliability: Why only a few parameters matter?''

\bibitem{me07b}
T.~Menzies, J.~Greenwald, and A.~Frank, ``Data mining static code attributes to
  learn defect predictors,'' \emph{IEEE Transactions on Software Engineering},
  January 2007, available from \url{http://menzies.us/pdf/06learnPredict.pdf}.

\bibitem{dou95}
J.~Dougherty, R.~Kohavi, and M.~Sahami, ``Supervised and unsupervised
  discretization of continuous features,'' in \emph{International Conference on
  Machine Learning}, 1995, pp. 194--202, available from
  \url{http://www.cs.pdx.edu/~timm/dm/dougherty95supervised.pdf}.

\bibitem{shepperd94}
M.~Shepperd and D.~Ince, ``A critique of three metrics,'' \emph{The Journal of
  Systems and Software}, vol.~26, no.~3, pp. 197--210, September 1994.

\bibitem{lessmann09}
S.~Lessmann, B.~Baesens, C.~Mues, and S.~Pietsch, ``Benchmarking classification
  models for software defect prediction: A proposed framework and novel
  findings,'' \emph{accepted for publication IEEE Transactions on Software
  Engineering}, 2009.

\bibitem{duda76}
R.~Duda, P.~Hart, and N.~Nilsson, ``Subjective bayesian methods for rule-based
  inference systems,'' in \emph{Technical Report 124, Artificial Intelligence
  Center, SRI International}, 1976.

\bibitem{domingos97optimality}
\BIBentryALTinterwordspacing
P.~Domingos and M.~J. Pazzani, ``On the optimality of the simple bayesian
  classifier under zero-one loss,'' \emph{Machine Learning}, vol.~29, no. 2-3,
  pp. 103--130, 1997. [Online]. Available:
  \url{citeseer.ist.psu.edu/domingos97optimality.html}
\BIBentrySTDinterwordspacing

\bibitem{yang02}
Y.~Yang and G.~Webb, ``Weighted proportional k-interval discretization for
  naive-bayes classifiers,'' in \emph{Proceedings of the 7th Pacific-Asia
  Conference on Knowledge Discovery and Data Mining (PAKDD 2003)}, 2003,
  available from \url{http://www.csse.monash.edu/~webb/Files/YangWebb03.pdf}.

\bibitem{witten05}
I.~H. Witten and E.~Frank, \emph{Data mining. 2nd edition}.\hskip 1em plus
  0.5em minus 0.4em\relax Los Altos, US: Morgan Kaufmann, 2005.

\bibitem{mann47}
H.~B. Mann and D.~R. Whitney, ``On a test of whether one of two random
  variables is stochastically larger than the other,'' \emph{Ann. Math.
  Statist.}, vol.~18, no.~1, pp. 50--60, 1947, available on-line at
  \url{http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&hand%
le=euclid.aoms/1177730491}.

\bibitem{demsar06}
J.~Demsar, ``Statistical comparisons of clasifiers over multiple data sets,''
  \emph{Journal of Machine Learning Research}, vol.~7, pp. 1--30, 2006,
  avaliable from \url{http://jmlr.csail.mit.edu/papers/v7/demsar06a.html}.

\bibitem{koru08}
A.~G. Koru, K.~E. Emam, D.~Zhang, H.~Liu, and D.~Mathew, ``Theory of relative
  defect proneness,'' \emph{Empirical Softw. Engg.}, vol.~13, no.~5, pp.
  473--498, 2008.

\bibitem{hayes06}
\BIBentryALTinterwordspacing
J.~H. Hayes, A.~Dekhtyar, and S.~K. Sundaram, ``Advancing candidate link
  generation for requirements tracing: The study of methods,'' \emph{IEEE
  Trans. Software Eng}, vol.~32, no.~1, pp. 4--19, 2006. [Online]. Available:
  \url{http://doi.ieeecomputersociety.org/10.1109/TSE.2006.3}
\BIBentrySTDinterwordspacing

\bibitem{baker07}
D.~Baker, ``A hybrid approach to expert and model-based effort estimation,''
  Master's thesis, Lane Department of Computer Science and Electrical
  Engineering, West Virginia University, 2007, available from
  \url{https://eidr.wvu.edu/etd/documentdata.eTD?documentid=5443}.

\bibitem{me08d}
T.~Menzies, B.~Turhan, A.~Bener, G.~Gay, B.~Cukic, and Y.~Jiang, ``Implications
  of ceiling effects in defect predictors,'' in \emph{Proceedings of PROMISE
  2008 Workshop (ICSE)}, 2008, available from
  \url{http://menzies.us/pdf/08ceiling.pdf}.

\bibitem{orrego04}
A.~Orrego, ``Sawtooth: Learning from huge amounts of data,'' Master's thesis,
  Computer Science, West Virginia University, 2004.

\bibitem{quinlan92}
R.~Quinlan, \emph{C4.5: Programs for Machine Learning}.\hskip 1em plus 0.5em
  minus 0.4em\relax Morgan Kaufman, 1992, iSBN: 1558602380.

\bibitem{quinlan92b}
J.~R. Quinlan, ``Learning with {C}ontinuous {C}lasses,'' in \emph{5th
  Australian Joint Conference on Artificial Intelligence}, 1992, pp. 343--348,
  available from \url{http://citeseer.nj.nec.com/quinlan92learning.html}.

\bibitem{Blake+Merz:1998}
C.~Blake and C.~Merz, ``{UCI} repository of machine learning databases,'' 1998,
  uRL: \url{http://www.ics.uci.edu/~mlearn/MLRepository.html}.

\bibitem{john95}
G.~John and P.~Langley, ``Estimating continuous distributions in bayesian
  classifiers,'' in \emph{Proceedings of the Eleventh Conference on Uncertainty
  in Artificial Intelligence Montreal, Quebec: Morgan Kaufmann}, 1995, pp.
  338--345, available from
  \url{http://citeseer.ist.psu.edu/john95estimating.html}.

\bibitem{boehm00a}
B.~Boehm, ``Safe and simple software cost analysis,'' \emph{IEEE Software}, pp.
  14--17, September/October 2000, available from
  \url{http://www.computer.org/certification/beta/Boehm_Safe.pdf}.

\bibitem{shepperd97}
M.~Shepperd and C.~Schofield, ``Estimating software project effort using
  analogies,'' \emph{IEEE Transactions on Software Engineering}, vol.~23,
  no.~12, November 1997, available from
  \url{http://www.utdallas.edu/~rbanker/SE_XII.pdf}.

\bibitem{khoshgoftaar03}
T.~M. Khoshgoftaar and N.~Seliya, ``Analogy-based practical classification
  rules for software quality estimation,'' \emph{Empirical Softw. Engg.},
  vol.~8, no.~4, pp. 325--350, 2003.

\bibitem{moser08}
\BIBentryALTinterwordspacing
R.~Moser, W.~Pedrycz, and G.~Succi, ``A comparative analysis of the efficiency
  of change metrics and static code attributes for defect prediction,''
  \emph{ICSE '08: Proceedings of the 30th international conference on Software
  engineering}, May 2008. [Online]. Available:
  \url{http://portal.acm.org/citation.cfm?id=1368088.1368114}
\BIBentrySTDinterwordspacing

\bibitem{shu02}
F.~Shull, V.~Basili, B.~Boehm, A.~Brown, P.~Costa, M.~Lindvall, D.~Port,
  I.~Rus, R.~Tesoriero, and M.~Zelkowitz, ``What we have learned about fighting
  defects,'' in \emph{Proceedings of 8th International Software Metrics
  Symposium, Ottawa, Canada}, 2002, pp. 249--258, available from
  \url{http://fc-md.umd.edu/fcmd/Papers/shull_defects.ps}.

\bibitem{fagan86}
M.~Fagan, ``Advances in software inspections,'' \emph{IEEE Trans. on Software
  Engineering}, pp. 744--751, July 1986.

\bibitem{shull00a}
F.~Shull, I.~Rus, and V.~Basili, ``How perspective-based reading can improve
  requirements inspections,'' \emph{IEEE Computer}, vol.~33, no.~7, pp. 73--79,
  2000, available from
  \url{http://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.77.pdf}.

\bibitem{fagan76}
M.~Fagan, ``Design and code inspections to reduce errors in program
  development,'' \emph{IBM Systems Journal}, vol.~15, no.~3, 1976.

\bibitem{conf/icse/NagappanB05a}
\BIBentryALTinterwordspacing
N.~Nagappan and T.~Ball, ``Static analysis tools as early indicators of
  pre-release defect density,'' in \emph{ICSE}, 2005, pp. 580--586. [Online].
  Available: \url{http://doi.acm.org/10.1145/1062558}
\BIBentrySTDinterwordspacing

\bibitem{me02f}
T.~Menzies, D.~Raffo, S.~on~Setamanit, Y.~Hu, and S.~Tootoonian, ``Model-based
  tests of truisms,'' in \emph{Proceedings of IEEE ASE 2002}, 2002, available
  from \url{http://menzies.us/pdf/02truisms.pdf}.

\bibitem{halstead77}
M.~Halstead, \emph{Elements of Software Science}.\hskip 1em plus 0.5em minus
  0.4em\relax Elsevier, 1977.

\bibitem{mccabe76}
T.~McCabe, ``A complexity measure,'' \emph{IEEE Transactions on Software
  Engineering}, vol.~2, no.~4, pp. 308--320, Dec. 1976.

\bibitem{chapman02}
M.~Chapman and D.~Solomon, ``The relationship of cyclomatic complexity,
  essential complexity and error rates,'' 2002, proceedings of the NASA
  Software Assurance Symposium, Coolfont Resort and Conference Center in
  Berkley Springs, West Virginia. Available from
  \url{http://www.ivv.nasa.gov/business/research/osmasas/conclusion2002/Mike_C%
hapman_The_Relationship_of_Cyclomatic_Complexity_Essential_Complexity_and_Erro%
r_Rates.ppt}.

\bibitem{me04g}
T.~Menzies, J.~DiStefano, A.~Orrego, and R.~Chapman, ``Assessing predictors of
  software defects,'' in \emph{Proceedings, workshop on Predictive Software
  Models, Chicago}, 2004, available from \url{http://menzies.us/pdf/04psm.pdf}.

\bibitem{polyspace}
``Polyspace verifier$^{\mbox \textregistered}$,'' 2005, available from
  \url{http://www.di.ens.fr/~cousot/projects/DAEDALUS/synthetic_summary/POLYSP%
ACE/polyspace-daedalus.htm}.

\bibitem{hall00}
G.~Hall and J.~Munson, ``Software evolution: code delta and code churn,''
  \emph{Journal of Systems and Software}, pp. 111 -- 118, 2000.

\bibitem{nikora03}
A.~Nikora and J.~Munson, ``Developing fault predictors for evolving software
  systems,'' in \emph{Ninth International Software Metrics Symposium
  (METRICS'03)}, 2003.

\bibitem{Khoshgoftaar:2004p1877}
T.~Khoshgoftaar and N.~Seliya, ``Comparative assessment of software quality
  classification techniques: An empirical case study,'' \emph{Empirical
  Software Engineering}, vol.~9, no.~3, pp. 229--257, 2004.

\bibitem{conf/ictai/TangK04}
\BIBentryALTinterwordspacing
W.~Tang and T.~M. Khoshgoftaar, ``Noise identification with the k-means
  algorithm,'' in \emph{ICTAI}, 2004, pp. 373--378. [Online]. Available:
  \url{http://doi.ieeecomputersociety.org/10.1109/ICTAI.2004.93}
\BIBentrySTDinterwordspacing

\bibitem{porter90}
A.~Porter and R.~Selby, ``Empirically guided software development using
  metric-based classification trees,'' \emph{IEEE Software}, pp. 46--54, March
  1990.

\bibitem{tiang95}
J.~Tian and M.~Zelkowitz, ``Complexity measure evaluation and selection,''
  \emph{IEEE Transaction on Software Engineering}, vol.~21, no.~8, pp.
  641--649, Aug. 1995.

\bibitem{srinivasan95}
K.~Srinivasan and D.~Fisher, ``Machine learning approaches to estimating
  software development effort,'' \emph{IEEE Trans. Soft. Eng.}, pp. 126--137,
  February 1995.

\bibitem{rakitin01}
S.~Rakitin, \emph{Software Verification and Validation for Practitioners and
  Managers, Second Edition}.\hskip 1em plus 0.5em minus 0.4em\relax Artech
  House, 2001.

\bibitem{fenton96}
N.~E. Fenton and S.~Pfleeger, \emph{Software Metrics: A Rigorous \& Practical
  Approach (second edition)}.\hskip 1em plus 0.5em minus 0.4em\relax
  International Thompson Press, 1995.

\bibitem{arisholm06}
E.~Arisholm and L.~Briand, ``Predicting fault-prone components in a java legacy
  system,'' in \emph{5th ACM-IEEE International Symposium on Empirical Software
  Engineering (ISESE), Rio de Janeiro, Brazil, September 21-22}, 2006,
  available from
  \url{http://simula.no/research/engineering/publications/Arisholm.2006.4}.

\bibitem{brooks95}
F.~P. Brooks, \emph{The Mythical Man-Month, Anniversary edition}.\hskip 1em
  plus 0.5em minus 0.4em\relax Addison-Wesley, 1995.

\bibitem{kitch07}
B.~A. Kitchenham, E.~Mendes, and G.~H. Travassos, ``Cross- vs. within-company
  cost estimation studies: A systematic review,'' \emph{IEEE Transactions on
  Software Engineering}, pp. 316--329, May 2007.

\bibitem{graves00}
T.~L. Graves, A.~F. Karr, J.~S. Marron, and H.~P. Siy, ``Predicting fault
  incidence using software change history,'' \emph{IEEE Trans. Software Eng.},
  vol.~26, no.~7, pp. 653--661, 2000, available on-line at
  \url{www.niss.org/technicalreports/tr80.pdf}.

\bibitem{basili02}
V.~Basili, F.~McGarry, R.~Pajerski, and M.~Zelkowitz, ``Lessons learned from 25
  years of process improvement: The rise and fall of the {NASA} software
  engineering laboratory,'' in \emph{Proceedings of the 24th International
  Conference on Software Engineering (ICSE) 2002, Orlando, Florida}, 2002,
  available from
  \url{http://www.cs.umd.edu/projects/SoftEng/ESEG/papers/83.88.pdf}.

\end{thebibliography}


\end{document}
