\section{Introduction}

Several research in defect prediction focus on building models with available local data (i.e. within company predictors). To employ these models, a company should have a data repository, where project metrics and defect information from past projects are stored. However, few companies apply this practice. We suspect that a common reason for not using defect predictors in practice, is the lack of local data repositories. Constructing such a repository requires keeping track of project metrics together with related defect information. In the absence of certain infrastructure like automated tools to collect these metrics, manual effort is unavoidable to accomplish this process. Our view is that managers do not take the effort to collect metrics for process improvement due to tight schedule and budget constrains.

On the other hand, there are public data repositories including projects from companies such as NASA \cite{me07c}. In this context, we investigate whether these public project data can be helpful for other companies for building localized defect predictors, especially for those with limited or no defect data repository. 

While there exists numerous research in learning predictors using within company data \cite{Arisholm:2006p516, Bell:2006p517, Ostrand:2007p519, me07e,  chen05, dekhtyar04, jiang07, nagappan05}, utilizing cross-company data has not been investigated in defect prediction research. 
%However, this issue has been throughly analyzed in a closely related area, that is cost estimation. In their systematic review, Kitchenham et.al. argue that it is not possible to reach a conclusion whether to use within or cross company data for cost estimation purposes \cite{kitchenham07}.
%In this context, we recently investigated this issue and on the contrary to the cost estimation literature, we reached the conclusion that use of third party data is helpful if the cost of investigating false alarms can be managed \cite{menzies07tr}. This result is applicable to domains where mission/ safety critical projects are carried out. It is also applicable to embedded software domain where the cost of patching defective software in millions of released products is not quite affordable. 
%What is the generality of our theories for common problems in empirical software engineering? Are the lessons learned
%from {\em Project-X} in {\em Company-A} relevant to {\em Project-Y} in 
%{\em Company-B}? Theoretically, 
%the answer is ``no''.
%Organizations can work in different domains, have
%different processes, and define/measure defects and other aspects of
%their products and processes in different ways. 
%Worse, all too often, organizations do not precisely define their
%processes, products, measurements, etc. 
%Hence, given data from {\em Project-X} from {\em Company-A}
%it is not clear to what extent the domain, process, and measurement 
%varies from that project to other projects.
%Despite this,
% much work assumes cross-company generality:
%\bi
%\item In 1976, McCabe defined a measure of internal module complexity called {\em v(g)} (the ``cyclomatic
%complexity) and claimed that modules with $v(g)\ge 10$ have a far greater likelihood of containing defects~\cite{mccabe76}.
%\item  
% In 1981, Boehm's COCOMO effort estimation research produced a table
%of {\em effort multipliers} and {\em scale factors} modeling how decisions about a software
%project
%affect, say, the development time.
%In his preferred {\em local calibration}
%method, this table is fixed while a pair of ``tuning'' factors can fine tune the estimates generated
%by this model~\cite{boehm81}. Subsequent work updated those tables~\cite{boehm00} but the essential
%assumption remained; i.e. that these tables were a useful and general place to start tuning effort estimates
%across multiple companies.
%\ei
%Despite their associated theoretical problems,
%these two approaches are widely applied across different companies.
%The present authors have extensive contacts
%with large software organizations 
%working on government projects.
%In that community we have observed:
%\bi
%\item
%When that group of contractors is required to produce audit-able estimates for completion of software projects, they often use
%off-the-shelf COCOMO (with little or no local tuning) to justify those estimates\footnote{
%For example, Dabney et al.'s research into the cost-effectiveness of independent verification and validation used 
%figures from untuned COCOMO~\cite{dabney07}.}.
%\item
%Also, as described in \tion{dp},
%McCabe's $v(g)$ rule
%is frequently cited as how test engineers select which modules should be selected for closer inspection.
%\ei
In this paper, we focus on {\em binary defect prediction} and perform experiments to check if we can reach a conclusion in favor of either CC or WC data. 
Specifically,
this paper 
assesses the relative merits
of cross-company (CC) vs within-company (WC) data for  defect prediction.
To the best of our knowledge, this kind of analysis is a novel one in defect prediction literature.

We perform three experiments with 10 publicly available project data from different companies.
Our analysis use the Naive Bayes data miner and are based on static code features\footnote{ Therefore, throughout the paper, the term "data" refers to static code features.}.

 Our experiments are designed  to answer the following research questions: 



%Our view is that cross-company data must be used with 
%great care in any software engineering problem.
%For example, previously~\cite{me06d}, we have built elaborate tool sets for converting supposedly 
%general {\em effort estimation}
%models to local conditions. 
%That work concluded that data imported from other organizations must be extensively pruned (removal
%of extraneous features and projects) before it can be tuned to local conditions.


%We perform several experiments to answer the following questions:

\bi
\item {\em When are defect predictors learned from CC data beneficial for organizations?}:

Our goal is to identify the conditions under which
cross-company  data should be preferred
to within-company 
for the purposes of structuring software inspections using outcomes of defect predictors.
In the first experiment, we compare the performance of defect predictors learned from WC data to those learned from CC data.
%Those conditions will turn out to be quite extreme; so much
%that they 
%hold in only a small number of organizations
% (e.g.
%organizations would have to tolerate extremely high false alarm rates).
%Hence,
%except in very rare cases,
%this first experiment deprecates the use of unfiltered cross-company data for defect prediction.

\item{\em How can companies filter CC data for local tuning?}:

%We anticipate that results from this first experiment will not be enough to stop the use of 
%cross-company data. With the {\em extraneous hypothesis}, 
%Our explanation for the poor performance of CC data is that it mixes useful infomration wiah an excess
%of extraneous information.
%To explore, 
CC data includes information from many diverse projects and are heterogeneous compared to WC data.  The goal of the second experiment is to select a subset of the available CC data that is similar to WC data and to investigate the effect of data homogeneity on the defect prediction performance. 
We apply  a simple nearest neighbor (NN) filtering to CC data for constructing a locally tuned repository.
We use the Euclidean distance between static code features of WC and CC data for measuring similarity and determining neighbors.  

\item{\em How much local data do organizations need for constructing a model for defect prediction?}:

A common belief is that collecting local data takes too much time and effort~\cite{kitch07}. In the third experiment, we check whether this belief is valid or not for defect data. We employ an incremental learning approach to  WC data in order determine the number of samples in local repositories for building defect prediction models.
%Kitchenham {\em et al.}~\cite{kitch07} argue that
%organizations use
%cross-company data since within-company data
%can be so hard to collect:
%\bi
%\item
%The time required to collect enough data on past projects from within a  company
%may be prohibitive.
%\item
%Collecting within-company data may take so long
%that 
%technologies change and older
%projects do not represent current practice.
%\ei
%We show below that
%predictors learned from a 
%mere one hundred examples  perform as well as predictors learned from many more examples.
%That is:
%\bi
%\item 
%Predictors tuned to the particulars of one company can
%be learned using very little data, collected in a very small amount of time;
%\item
%The above two reasons~\cite{kitch07}  for avoiding WC-data
%in {\em effort estimation}
%may not neccessarily hold for
%{\em defect prediction}.
%\ei

\item{\em Can our results be generalized?}:

We initially use only NASA projects in our three experiments to answer the above questions.
In order to check the  external validity of our
results, our last experiment replicates all three experiments on
projects from a company that has no ties with NASA: specifically, a Turkish company writing software controllers
for home appliances. 
%All the results described above also hold for the Turkish data. While this does not
%conclusively prove the external validty of our conclusions, it does suggest that these results are not
%just some fluke of one domain.
\ei

The contributions of this paper are relevant for both academia and practice. Prior to this paper, no study was performed to investigate the relative merits of using cross company or within company data for constructing defect predictors. In our analysis, in favor of the common belief, we empirically show that within company data is better for defect prediction problem. Further, on the contrary to the common belief, we empirically show that, the required local data for constructing defect predictors can be easily and quickly collected. For our analysis, we have collected data from industry and made it publicly available for the use of other researchers and practitioners. As a practical contribution, our analysis of cross company data and the proposed methodology allow the construction of defect predictors even for companies with no local defect data. In summary, we observe that the defect detectors learned from site A are found to be useful at site B. Furthermore, it is a result of major significance that other project's data can be used locally (after applying a very simple filter).

%Our results have important implications for software practitioners. Our results suggests that in the absence of local data, companies can {\em instantaneously} benefit from cross company data with 
%\bi
%\item high probability of detecting defective methods, 
%\item affordable false alarm rates,
%\item a minor investment, which is to use an automated tool to collect local static code measurements,
%\ei

%and they can also collect and start using local data in a few months.

This paper is organized as follows: The next section motivates
our work with examples of industrial groups that use these detectors.
Then, in Section 3, we explain our methodology by describing the data sources, data mining algorithm and the performance evaluation criteria.  In Sections 4, 5 and 6, we describe our experiment designs, results and discussions for the first three research questions stated above. In Section 7, we replicate all three experiments in order to check the external validity of our results. Sections 8, 9 and 10, discusses the related work, practical implications and threats to validity  respectively. We conclude our work in Section 11.
