%\documentclass[onecolumn,draftcls,journal,compsoc]{IEEEtran}
%\documentclass[technote]{IEEEtran}
\documentclass[twocolumn,technote,9pt,compsoc]{IEEEtran}
\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		%\put(0,0){\line(0,1){8}}
		%\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{5cm}{5cm}\selectfont{Draft v5}}}}}
%\makeatother
%
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage[nocompress]{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}

\begin{document}

\title{Data Sampling for Cross Company Defect Predictors}


\author{ Burak~Turhan,
       Ayse~Bener,\\ Tim~Menzies%, Turgay~Aytac
\thanks{Mr. Turhan and Dr. Bener are with the 
Department of Computer Engineering, Bogazici University, Turkey.
Emails:
\protect\url{turhanb@boun.edu.tr},
\protect\url{bener@boun.edu.tr}.
}\thanks{The research described in this paper
is supported in part by Arcelik Corp. and
Bogazici University research fund under grant number BAP-06HA104
}}


\maketitle

\begin{abstract}
Defect prediction models are helpful for QA activities in terms of allocating limited resources for testing efforts. 
Several research in this area focus on building models with available local data (i.e. within company predictors). To employ these models, a company should have a data repository, where measurements and defect information from past projects are stored. In the industry, few companies apply this practice. In a recent work, we have shown that cross company data can be used for building a predictor with the cost of increased false alarms. Thus, the practical application of that work is limited with certain domains such as mission critical and embedded projects.

In this paper, we provide empirical evidence from an industrial case study showing the applicability of our previous work. Furthermore, we extend the advantages of using cross company data. We show that using nearest neighbor (NN) sampling techniques, cross company defect predictors can perform as good as within company predictors. Our results show that NN sampling removes the increased false alarm rate introduced by cross company data. We conclude that all companies can benefit from defect prediction only by investing on an automated metric collection tool.
\end{abstract}

%------------------------------------------------------------------------- 

\section{Introduction}
Numerous research in defect prediction focus on building effective models on within company data \cite{menzies07, turhanicsoft, turhanqsic, kutlubay07}. Making use of third party, i.e.  cross company, data has been throughly analyzed in cost estimation literature.  In their systematic review, Kitchenham et.al. argues that it is not possible to reach a conclusion whether to use within or cross company data for cost estimation purposes \cite{kitchenham07}. In defect prediction context, we recently address the same problem and on the contrary to the cost estimation literature, we reach the conclusion that use of third party data is helpful if the cost of investigating false alarms can be managed \cite{menzies07tr}. This result is applicable to domains where mission/ safety critical projects are carried out. It is also applicable to embedded software domain where the cost of patching defective software in millions of released products is not quite affordable. Furthermore, the reputation of such a company may be significantly affected in these cases. 

It is tempting to benefit from third party data, but the following question rises immediately: Do we need the whole available data? In our previous research, we show that similar performance can be achieved by using only a small portion of the available data in within company studies \cite{menzies07tr}. We also note that as the number of training data increases, variance of the learned theories also tend to increase. Boetticher and Pelayo\&Dick reports that sampling strategies, i.e. nearest neighbor/ over/ under sampling, can improve the performance of defect predictors for within company studies \cite{boetticher05, pelayo07}. 

In this paper we look for a solution to wisely select a small subset of available third party data in order to construct a defect prediction model to be used in practice. While we benefit from the outcomes of the above-mentioned studies, our research are different from these in many aspects.
Our research focus on data selection from third party repositories, while these research are within company studies. Our previous cross company research is about the usability of third party data and does not investigate data sampling.
Use of third party data comes with the cost of increased false alarms. While the application domain allows for false alarms, we aim at removing this cost by determining third party data similar to company characteristics.
We propose to use a nearest neighbor based sampling method, rather than using randomized under/over sampling. Pelayo\&Dick use only class labels for sampling, whereas we employ non-class attributes.
Boetticher makes use of class and non-class attributes together, however his work is a within company study and focuses on determining the experimental difficulty by data characteristics. 

The problem addressed in this paper has emerged from a collaborative research with a white-goods manufacturer (Arcelik Corp.). Arcelik develops embedded software for its white-good products. The nature of the collaborative research is to determine defects in the software without too much manual labor. Thus, there is a need for initiating a metric data collection and defect prediction process. However, it turned out that data collection part progresses rather slowly, since Arcelik's software team can not spend the required time. Although static code measurements can be achieved with automatic tools, someone has to manually label these measurements as defective or non-defective instances. Currently, Arcelik has only managed to construct a relatively small data repository. So, we make use of public Nasa MDP projects to construct a defect predictor for Arcelik \cite{nasa}. 

Our contribution in this paper is two folds. First, we present the application of our previous results in a case study. In this sense, we build a cross company model for Arcelik and achieve $\{pd, pf\}=\{95, 55\}$. Compared to the within company model, where we achieve $\{pd, pf\}=\{88, 29\}$, our results are consistent with our previous conclusions, which is cross company data increases the rate of false alarms. Second, we show that using NN methods to construct a training set from cross company data not only decreases the false alarm rate, but also achieves equivalent performance with the within company model. We achieve  $\{pd, pf\}=\{88, 25\}$ with NN sampling cross company model for Arcelik. We also verify this result using Nasa datasets, where we achieve $\{pd, pf\}=\{78, 28\}$, $\{pd, pf\}=\{92, 63\}$ and $\{pd, pf\}=\{77, 32\}$ for within company, cross company and NN sampling cross company models respectively. Our results have an important practical implication: All companies can employ defect prediction process only by investing in an automated metric collection tool.  

This paper is organized as folows: We introduce the datasets and performance measures used in this study in Section 2 and Section 3 respectively. In Section 4 we explain the NN sampling strategy. Rest of the paper (i.e. Section 5, 6 and 7) presents the experimental setup and discusses the results. Finally, we conclude in Section 8.


\section{Data}
We have evaluated 13 datasets from different project domains. Among these, 10 public datasets are obtained from Nasa MDP and Promise Repositories \cite{nasa, promise}. These datasets are accepted to reflect the common industrial software engineering practice. In our previous work, we have argued that Nasa is an umbrella organization and the projects used in that study (also used in this work) have been developed at different sites, so each project can be assumed to be developed by different companies \cite{menzies07tr}. Additionally,  we have also formed 3 more datasets (i.e. Arcelik datasets) that are collected from Arcelik. In the experiments, we treat each Nasa dataset as third party data (i.e. cross- company) for every other dataset. On the other hand, Arcelik datasets are collected from a single company and are treated as within- company data. 

A list of datasets used in this research is given in Figure \ref{fig:data}. Sample sizes of the projects vary from 36 to 5,589 modules, which enables experiments in a broad spectrum of project sizes. Each Nasa dataset has 22 attributes static code attributes (See \cite{nasa, menzies07}). Arcelik datasets include 29 static code attributes. 19 of these attributes are common in all datasets, so we use only these shared attributes in our experiments. Finally, In both datasets modules with error counts greater than zero are assumed to be defective. 

\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{l|lrrr@{.}l}
     &          & (\# modules)         &          &  &\\
data & language & examples & features  & \multicolumn{2}{c}{\%defective}\\\hline
%pc5  & C++ & 17,186   & 22          & 3&0\\
%mc1  & C++ & 9,466    & 22          & 0&71\\
pc2  & C++ & 5,589    & 22         & 0&41 \\
pc3  & C++ & 1,563    & 22          & 10&23\\
pc4  & C & 1,458    & 22          & 12&2\\
pc1  & C++ & 1,109    & 22          & 6&94\\
kc1  & C++ & 845      & 22          & 15&45\\
kc2  & C++ & 522      & 22          & 20&49\\
cm1  & C++ & 498      & 22          & 9&83\\
kc3  & JAVA & 458      & 22          & 9&38\\
mw1  & C++ & 403      & 22          & 7&69\\
mc2  & C++ & 61      & 22          & 32&29\\
ar3 & C & 63 & 29 & 12&70\\
ar4 & C & 107 & 29 & 18&69\\
ar5 & C & 36 & 29 & 22&22\\\cline{3-3}
     \multicolumn{2}{c}{~}     &  12,712 

\end{tabular}
\end{center}
\caption{Datasets used in this study.}\label{fig:data}
\end{figure} 

\section{Performance Measures}
\begin{table}

\renewcommand{\arraystretch}{1.3}
\caption{Confusion Matrix}
\label{perfmes}
\centering

\begin{tabular}{|c|c|c|}
\hline
 & \multicolumn{2}{c|}{Estimated} \\ \hline
Real & Defective & Nondefective \\ \hline
Defective & A & C \\ \hline
NonDefective & B & D \\ \hline
\end{tabular}
\end{table}
\begin{eqnarray}
\label{pd}
pd = (A)/(A+C) \\
pf = (B)/(B+D) \nonumber
\end{eqnarray}
We have used probability of detection (pd) and probability of false alarm (pf) as the performance measures \cite{menzies07}. Formal definitions for these performance criteria are given in Equation \ref{pd} respectively and they are derived from the confusion matrix given in Table \ref{perfmes}. pd is a measure of accuracy for correctly detecting the defective modules. Therefore, higher pd's are desired. pf is a measure for false alarms and it is an error measure for incorrectly detecting the nondefective modules. pf is desired to have low values. Since we need to optimize two parameters, pd and pf, a third performance measure called balance is used to choose the optimal (pd, pf) pairs. balance is defined as the normalized Euclidean distance from the desired point (0,1) to (pd, pf) in a ROC curve \cite{menzies07}. 

Zhang and Zhang argue that using (pd,pf) performance mesures in imbalanced classification problems is not practical due to low precisions \cite{zhang07}. On the contrary, Menzies et.al argues that precision has an unstable nature and it can be misleading to determine the better predictor \cite{menzies07a}. They also give examples of low precision predictors that are in use in SE industry. We also think that predictors with high pd rate can be practical even when they have high pf rate. Especially in mission critical and safety critical systems, detecting defects accurately with the cost of many false alarms is affordable \cite{menzies07a,turhanicsoft,oral07}. Furthermore, we believe that this strategy is still more efficient than using exhaustive testing. Thus, we argue that it would not defeat the purpose of defect prediction as Zhang and Zhang claims \cite{zhang07}.


\section{Sampling Strategy}

Our idea behind sampling is to collect similar instances together in order to construct a training set that is homogeneous with the validation set. We use the k-Nearest Nieghbor (k-NN) method to measure the similarity between the validation set and the training candidates. Standard k-NN uses Euclidean distance to measure similarity. Besides k-NN, we use a rather conceptual sampling strategy, where we select the training set from third party datasets that have similar application domain with the validation set. Arcelik's projects are from embedded systems domain. Using the descriptions of Nasa datasets, we have come up with 4 datasets (i.e. cm1, pc1, pc3, pc4) that shows embedded systems characteristics \cite{oral07}.  

The expected outcome of the sampling part is to obtain a subset of available third party data that shows similar characteristics to Arcelik's coding culture. If the data have class labels then we can make use of this information to build a model. However, in practice labeled data is limited (or does not exist) and most of the time, it requires manual labeling. On the other hand automated tools can process all past projects and produce lots of unlabeled measurements with no manual effort. As explained in the previous section, Arcelik has limited labeled data. In our experiments we assume that they have no class labels. 

\section{Experiment I: Within Company Model}
Experiment I is designed as a baseline to compare the performance of other experiments. With no usage of third party data, this experiment presents the baseline performance of Arcelik's defect prediction model. The limited labeled data repository is used to train a model and to test its performance.
Each Arcelikx dataset is tested with predictors that are trained on the combined version of the remaining Arcelikx' datasets. For each validation set, the training candidates are randomized $100$ times and 90\% of them are used for training. 

Figure \ref{wcar} shows the boxplots of {\em pd, pf, bal} for $3\times 100$ experiments for Arcelik data. With the within company model, we achieve $\{pd, pf\}=\{88, 29\}$ considering the three projects available in Arcelik's repository. These values are adequate for practical use compared to manual code reviews \cite{menzies07}. However, constructing such a model requires the availability of labeled data, where source code measurements are matched with defects. When automated tools are not used, labeling should be carried out manually, which undermines the ease of automation in defect prediction. We will use these results as a baseline to compare with the following experiments.

For the sake of completeness, we repeat the same experiment on Nasa datasets. We treat each Nasa project individually assuming each project is developed by different companies. However, we report the performance measures of the combined results (Recall that we treat Nasa as an umbrella organization). Figure \ref{wcnasa} shows that a median of $\{pd, pf\}=\{78, 28\}$ rate is achieved with a within company model for Nasa datasets.

\begin{figure}
\centering
\includegraphics[width=3in]{wc-ar.png}
\caption{Within company performance measures (pd, pf, bal) for Arcelik datasets}
\label{wcar}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=3in]{wc-nasa.png}
\caption{Within company performance measures (pd, pf, bal) for NASA datasets}
\label{wcnasa}
\end{figure}

\section{Experiment II: Cross Company Models}
In this experiment, we discard the class information of Arcelik's data and use a random subset of third party data to train a model. In practice, this approach corresponds to measuring source code with automated tools, yet there are no labels, assuming there are not enough resources to spare time on labeling these measurements.  Thus, this random subset can be used to train a predictor and can be tested on Arcelik's data. This experiment is a replication of our previous cross company study, where we used Nasa datasets. In this experiment, we similarly use Nasa datsets for training, yet the validation set comes from Arcelik's projects. To the best of our knowledge, this experiment is the first real application of a cross company defect prediction. We also repeat this experiment with Nasa datasets in order to compare with the sampling results.

There are three variations of this experiment. 
\begin{enumerate}
\item{Use a random portion of third party data: This is the replicated cross company study.}
\item{Use projects of third party data with similar application domain (Conceptual Sampling): The selected subset of third party data shows similar application domain characteristics with Arcelik's data.}
\item{Sample from similar instances of available third party data: The training subset of third party data is selected with k-NN}
\end{enumerate}

\subsection{Experiment II.a: Random Sampling}
Figure \ref{ccar} and Figure \ref{ccnasa} shows boxplots for {\em random sampling cross company} results for Arcelik and Nasa datasets respectively. The results for Nasa datasets are consistent with our previous work.  Furthermore, Arcelik results support those findings. Comparing with Figure \ref{wcar}, we see a pattern of increased probability of detection at the cost of  increased probability of false alarms. $pd$ for Arcelik has increased by 7\%, however, $pf$ has nearly doubled to 55\%. This increase in $pf$ can be afforded in the case of Arcelik. Because missing a defect costs much more than redundantly  testing a non-defective module.

With this experiment, we show that cross company models are applicable in practice. We make use of Nasa datasets in order to train a predictor and use that predictor to determine the defects in Arcelik's software. However, this comes with the cost of increased false alarms, which is conveniently consistent with the domain of Arcelik. On the other hand, not all companies may afford this increased testing cost. Therefore, remaining experiments deal with this issue. We will show that, by using wiser sampling strategies, this overhead can be avoided. In the following experiments, we investigate 2 sampling strategies. 


\begin{figure}
\centering
\includegraphics[width=3in]{cc-ar.png}
\caption{Cross company performance measures (pd, pf, bal) for Arcelik datasets}
\label{ccar}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=3in]{cc-nasa.png}
\caption{Cross company performance measures (pd, pf, bal) for Nasa datasets}
\label{ccnasa}
\end{figure}

\subsection{Experiment II.b: Conceptual Sampling}
In order to choose similar projects from Nasa, we have derived the characteristics of the Nasa projects from available requirements and application areas. After examining the domain requirements, we have included CM1, PC1, PC3 and PC4 in our training candidates \cite{oral07}. CM1 is a spacecraft instrument and owns the characteristics of embedded systems. Other projects are from a flight software system, which is one area embedded software is necessarily implemented. These projects are used to training a predictor to be tested on Arcelik projects.

Results for conceptual sampling are quite interesting and given in Figure \ref{dsccar}. With conceptual sampling, median $pd$ is around $88\%$. Further, median $pf$ has dropped to $39\%$.  We explain this phenomenon as follows: In our cross company study, we associate the increased false alarm rate with the irrelevant information introduced by the cross company data \cite{menzies07tr}. Kitchenham et al. explains similar behaviour in cost estimation by arguing that cross company data are heterogeneous, whereas within company data are homogeneous \cite{kitchenham07}. Our results show that, even by using a conceptual sampling among projects according to their application domain, the heterogeneity of cross company data is reduced. Thus, the amount of irrelevant information is also reduced together with the probability of false alarms. In the next experiment, we look for further evidence of this explanation with k-NN sampling method.  

\begin{figure}
\centering
\includegraphics[width=3in]{domaincc-ar.png}
\caption{Conseptual sampling cross company performance measures (pd, pf, bal) for Arcelik datasets}
\label{dsccar}
\end{figure}

\subsection{Experiment II.c: NN Sampling}
Previous experiment defines similarity of data samples at the level of related projects' application domain. Although it is a conceptual, even subjective, definition of similarity,  this approach yields improved results. In this experiment we use formal definitions of similarity at the data sample level. We employ k-NN in order to determine similar data samples. We first calculate the pairwise distances between the validation set and the candidate training set samples. Let $N$ be the number of validation set size. For each validation sample, we pick its $k =10$ nearest neighbors from candidate training set. Then we come up with a total of  $10\times N$ similar samples. Note that these $10\times N$ samples may not be unique (i.e. a single data sample can be a nearest neighbor of many data samples in the validation set). Using only unique ones, we form the training set and use a random $90\%$ of it for training a predictor. We repeat the last step $100$ times.    

Figure \ref{nnccar} and Figure \ref{nnccnasa} shows the performance measures for Arcelik and Nasa datasets respectively. In Arcelik, we achieve median $\{pd, pf\}=\{88, 25\}$ and in Nasa we have  $\{pd, pf\}=\{77, 32\}$. These results are most interesting. Recall the within company performances from Figure \ref{wcar} and Figure \ref{wcnasa}. For both companies (Arcelik and Nasa), the results for NN sampling cross company predictors are equivalent with within company models. This observation has very important practical implications. Building a within company model requires a local repository where source code measurements and defect information should be matched. We have discussed that without using automated tools, this matching process is a labor intensive one. On the other hand, only gathering static code attributes via tool support and sampling third party repositories for building a model perform as well as building a within company model. Yet, it is much cheaper and it does not require any investment other than metric extractor tools. 


\begin{figure}
\centering
\includegraphics[width=3in]{nncc-ar.png}
\caption{NN Sampling cross company performance measures (pd, pf, bal) for Arcelik datasets}
\label{nnccar}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=3in]{nncc-nasa.png}
\caption{NN Sampling cross company performance measures (pd, pf, bal) for Nasa datasets}
\label{nnccnasa}
\end{figure}

\section{Discussion}

In random sampling cross company models we see that the false alarm rates also substantially increase compared to the within company models. In our previous work, we have argued that this is due to the irrelevancies introduced by cross company data \cite{menzies07tr}. Experiment IIb and Experiment IIc shows that using conceptual or NN sampling removes the increased false alarm rates. Now we argue that, using these sampling methods instead of random sampling, helps choosing training examples that are similar to problem at hand. Thus, the irrelevant information in non-similar examples are avoided. 

A surprising outcome of our results is that cross company models with NN sampling performs as good as within company models. On a second thought, this should not be surprising. NN Sampling picks training examples that are similar to the unlabeled examples about which we want to make predictions. Without introducing irrelevant information, NN sampling populates a set that has the same characteristics with the problem. That means, the likelihood of both implementing similar software modules and having similar defects in these modules is high. Thus, NN sampling simulates that the selected cross company data are coming from within the company. In other words, NN sampling preserves the homogeneity of within company data whereas random sampling introduces heterogeneity.    

Another point worth mentioning is about the programming languages of the datasets. Arcelik projects are implemented in C language and most Nasa projects are implemented in C++. Although Arcelik uses a procedural language, it can make use of Nasa projects that are implemented in an object-oriented language. We explain this as follows: The source code attributes extracted from these projects are categorized in three classes: LOC, Halstead and McCabe metrics \cite{mccabe76, halstead77}. These are {\em base} metrics that reflect the characteristics of a code piece. They contain no language dependent measures except LOC. However in our case, it seems that LOC metrics does not have significant differences between C and C++ languages. 


\section{Conclusion}

In our previous  research , we have compared cross vs. within company defect predictors and concluded that cross company predictors should be preferred in certain project domains such as mission critical and embedded systems. That conclusion was based on the fact that cross company predictors increase the probability of detecting defects at the cost of increasing the rate of false alarms. Thus, in other domains, where the cost of handling more false alarms may not be afforded, we advised using within company models. 

Our results in this research shows that sampling cross company repositories with simple NN methods removes the increased false alarm rate introduced by the cross company data. Furthermore, we show that using cross company predictors perform as good as within company models. Thus, we extend our previous conclusion and state that cross company models can be safely used in all project domains. 

A common reason for not using defect predictors in practice is the lack of local data repositories. Constructing such a repository requires keeping track of project metrics together with related defect information. In the absence of certain infrastructure like automated tools, manual effort is unavoidable to accomplish this process. In practice, managers do not take the effort to collect metrics for process improvement due to tight schedule and budget constrains. However, based on our results, investing on an automated tool that extracts source code metrics is sufficient for defect prediction purposes. 


% use section* for acknowledgement
%\section*{Acknowledgment}
%This research is supported in part by Bogazici University research fund under grant number BAP-06HA104.

\bibliographystyle{IEEEtran}
\bibliography{ccnn}
\end{document}
