\section{Component vs. Whole-based Defect Prediction}

\subsection{The Experiment}

In order to test the implications of learning using components dense with software defects, an experiment was constructed using five NASA defect data sets (CM1, KC1, MC1, PC1, PC3). These data sets were chosen because they have been studied in the field extensively, and also that they are widely available to the PROMISE community. Five were chosen due to the limited number of data sets containing noteworthy numbers of components.

For each data set, components are extracted by first determining both defective and non-defective modules contained in that data set. Once the modules are obtained, those components (named after a unique identifier) containing these modules are selected for further analyses. 

After extracting the components and corresponding number of defects, components were retrieved for further analysis from each data set if the number of defective modules per component exceeded the median number of defects across $all$ components in that data set. For example, in \fig{kc1-defects} the bottom horizontal line represents the median number of defects in the KC1 data set. Thus, those components lying under this line are not used in further stages of the experiment. The components selected (at or above the median number of defects) are considered $dense$ components. The pseudocode in \fig{dense-comp-experiment} illustrates the remaining setup of the experiment:

\begin{figure}
\begin{verbatim}

1 For run = 1 to 10
2   For each dense component C in data set D
  
3    Let Train = C
4    Let Test = All components in D except for C
   
5     For bin = 1 to 10
6      Train' = Randomly select 90% modules from Train
7      Test' = Randomly select 10% modules from Test
     
8      Learn using the Naive Bayes classifier using Train' and Test'
9     end bin
10   end component
11 end run
\end{verbatim}
\caption[Training on Dense Components Vs. All Components]{Training on dense components versus all components. The experiment performs training on modules residing in dense components, and testing on modules contained in all other components in the data set.}\label{fig:dense-comp-experiment}
\end{figure}

Lines 1 and 5 of \fig{dense-comp-experiment} illustrate the use of the 10X10-way cross validation used in the experimental process. The standard 10X10-way cross validation operates by selecting 90\% of the data randomly for training, and the remaining 10\% for testing. This process is then repeated 10 times for consistency. The experiment shown in \fig{dense-comp-experiment}, however, handles this operation in a slightly different manner. Since the objective is to analyze the performance of training on modules in components containing a high number of defective modules while testing on all other components' modules, a minute alteration was made to the cross-validation of the experiment. A ``pool'' of training data was constructed by focusing on only those instances $within$ a dense component, as in line 3 of the psuedocode. The available pool of testing instances, thus, are gathered from the $remaining$ components in the data set. This is employed to prevent training, and then testing on modules within the same component. Lines 6 and 7 illustrate collecting 90\% of the current dense component's instances as the final training set $Train'$, and 10\% of the modules from the available instances in components not labeled $dense$ as $Test'$.

While this represents a slight modification to the standard pratice of performing a cross-validation, it is within our engineering judgement to apply techniques that best mimick current methods in an area of experimentation still in its infancy. Thus, the recentness of this specific area of research invites further techniques to be discovered and implemented.

Line 8 of \fig{dense-comp-experiment} executes the classifier (in this case, \nbc) on the previously created training and testing sets $Train'$ and $Test'$. \nbc was used because of its speed, and also because it has been shown to perform well on PROMISE defect data against other learners ~\cite{lessmann08}.

As the overall goal is to determine if training our classifiers using fewer, but more densely-packed components is advantageous to the usual practice of learning on a pool of all components (and thus all modules), comparisons are made between the two approaches, as well as  standard over and under sampling of the data. The results are shown in the following section.

\begin{figure}
\begin{center}
\includegraphics[width=6in]{figures/cs-kc1-dist.pdf}
\end{center}
\caption[Component defect distribution example]{Defect distributions of components found in the KC1 data set. Note that only a small number of components contain a relatively high number of defects.}\label{fig:kc1-defects}
\end{figure}

\subsection{Results}

The metrics used in the analysis of comparing results from training on dense components over the traditional method of using all components in the data set are $pd$ (Probability of Detection), $pf$ (Probability of False Alarm) and $precision$. If $A$,$B$,$C$, and $D$ denote the true negatives, false negatives, false positives and true positives (respectively) found by a classifier, then:

\begin{equation}
pd = Recall = \frac{D}{(B+D)}
\end {equation}

and

\begin{equation}
pf = \frac{C}{(A+C)}
\end {equation}

and

\begin{equation}
precision = \frac{D}{(D+C)}
\end{equation}

Therefore, $pd$ and $precision$ values are best if maximized, while $pf$ results should be minimized.

\fig{pds-all}, \fig{pfs-all} and \fig{prec-all} show statistical rankings of each treatment, as well as quartile charts displaying the median and variance of each metric for the $combined$ data sets, as a whole, used in the experiment. Note that training on components containing a higher number of defective modules maintains higher or tied ranks with the traditional method, and yields similar medians; while $precision$ and $pd$ medians lose 3\% and 2\% respectively, learning on dense areas provides much better $pf$ medians -- almost half. 

Perhaps more interestingly are the analyses of data sets separately. Table \ref{plus-minus} demonstrates the outcome of each treatment for each data set independently. A summary table of these results is shown in \fig{summz}, where``+'' denotes a $win$ for a particular treatment against the other, per data set. Conversely, a ``-'' indicates a $loss$, and ``0'' represents a $tie$. For example, the fifth row in Table \ref{plus-minus} (data set PC3), shows that learning on dense components wins over all other methods for $pf$, but loses in recall and ties in precision. A win, loss or tie is assigned to a treatment by examining its statistical rank value (according to the Mann-Whitney test) in comparison to the opposing treatments. 

The results from this table demonstrate that learning on only components containing higher numbers of defective modules is beneficial because

\bi
\item{defect prediction performance is improved significantly}
\item{less data is required during the training phase, meaning faster runtimes and results}
\item{insight is provided for component types; software organizations can make informed decisions about how to approach certain problematic components }
\ei 

\begin{table}
\small\renewcommand\arraystretch{1.1}%
\begin{center}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Project} & \textbf{Recall} & \textbf{Prob. False Alarm (Pf)} & \textbf{Precision}\\
&   \begin{tabular}{r r r c c c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
& & & 0\% & 50\% & 100\%  
\end{tabular} 
  & \begin{tabular}{r r r c c c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
& & & 0\% & 50\% & 100\%  
\end{tabular}   
 & \begin{tabular}{r r r c c c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
& & & 0\% & 50\% & 100\%  
\end{tabular} \\
\hline
CM1 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&All&   \boxplot{0}{33}{84}{91}{3}  \\
2&Over&  \boxplot{0}{33}{79}{88}{3}   \\ 
2&Under& \boxplot{0}{33}{68}{87}{3}    \\ 
3&Dense& \boxplot{0}{0}{60}{100}{3}    \\ 
\end{tabular}
 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{0}{0}{25}{3}  \\
2&Over&    \boxplot{0}{11}{20}{67}{3}   \\ 
2&Under&   \boxplot{0}{13}{28}{67}{3}    \\ 
2&All&     \boxplot{0}{8}{15}{67}{3}    \\ 
\end{tabular} & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&All&   \boxplot{0}{25}{75}{93}{3}  \\
1&Over&  \boxplot{0}{20}{75}{92}{3}   \\ 
1&Under& \boxplot{0}{17}{67}{93}{3}    \\ 
1&Dense& \boxplot{0}{0}{50}{100}{3}    \\ 
\end{tabular} \\
\hline
KC1 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{46}{82}{93}{3}  \\
1&Under&  \boxplot{0}{42}{61}{89}{3}   \\ 
1&All& \boxplot{0}{36}{60}{91}{3}    \\ 
1&Over& \boxplot{0}{42}{57}{90}{3}    \\ 
\end{tabular}
 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&All&     \boxplot{0}{8}{12}{64}{3}  \\
1&Over&    \boxplot{0}{10}{17}{57}{3}   \\ 
1&Under&   \boxplot{0}{9}{13}{64}{3}    \\ 
1&Dense&   \boxplot{0}{7}{17}{53}{3}    \\ 
\end{tabular} & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{33}{83}{90}{3}  \\
1&All&  \boxplot{0}{43}{75}{88}{3}   \\ 
1&Under& \boxplot{0}{42}{59}{90}{3}    \\ 
1&Over& \boxplot{0}{43}{59}{89}{3}    \\ 
\end{tabular}\\
\hline
MC1 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Over&   \boxplot{0}{60}{91}{94}{3}  \\
1&Under&  \boxplot{0}{70}{89}{93}{3}   \\ 
2&All& \boxplot{0}{34}{53}{94}{3}    \\ 
3&Dense& \boxplot{0}{33}{75}{90}{3}    \\ 
\end{tabular}
 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Over&   \boxplot{0}{6}{9}{36}{3}  \\
1&Under&  \boxplot{0}{7}{11}{30}{3}   \\ 
2&All& \boxplot{0}{6}{7}{66}{3}    \\ 
3&Dense& \boxplot{0}{6}{11}{66}{3}    \\ 
\end{tabular} & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{12}{21}{98}{3}  \\
1&All&  \boxplot{0}{10}{16}{99}{3}   \\ 
1&Under& \boxplot{0}{6}{16}{100}{3}    \\ 
1&Over& \boxplot{0}{6}{13}{100}{3}    \\ 
\end{tabular} \\
\hline
PC1 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{71}{85}{91}{3}  \\
2&Over&  \boxplot{0}{27}{86}{93}{3}   \\ 
2&All& \boxplot{0}{29}{75}{94}{3}    \\ 
3&Under& \boxplot{0}{30}{75}{91}{3}    \\ 
\end{tabular}
 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{9}{14}{29}{3}  \\
2&Over&  \boxplot{0}{7}{12}{73}{3}   \\ 
2&All& \boxplot{0}{6}{11}{71}{3}    \\ 
3&Under& \boxplot{0}{9}{19}{67}{3}    \\ 
\end{tabular} & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{71}{90}{91}{3}  \\
2&All&  \boxplot{0}{27}{89}{95}{3}   \\ 
2&Under& \boxplot{0}{20}{50}{95}{3}    \\ 
3&Over& \boxplot{0}{25}{58}{94}{3}    \\ 
\end{tabular} \\
\hline
PC3 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Under&   \boxplot{0}{50}{73}{83}{3}  \\
1&Over&  \boxplot{0}{29}{67}{89}{3}   \\ 
2&All& \boxplot{0}{40}{67}{87}{3}    \\ 
2&Dense& \boxplot{0}{42}{64}{86}{3}    \\ 
\end{tabular}
 & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{0}{32}{50}{3}  \\
2&Over&  \boxplot{0}{10}{30}{71}{3}   \\ 
2&Under& \boxplot{0}{16}{27}{50}{3}    \\ 
3&All& \boxplot{0}{13}{31}{60}{3}    \\ 
\end{tabular} & \begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
1&Dense&   \boxplot{0}{18}{67}{98}{3}  \\
1&Under&  \boxplot{0}{21}{43}{95}{3}   \\ 
1&All& \boxplot{0}{14}{38}{96}{3}    \\ 
1&Over& \boxplot{0}{12}{25}{96}{3}    \\ 
\end{tabular} \\
\hline\end{tabular}
\end{center}
\caption{Result statistics per data set. The numeric value next to each treatment represents its Mann-Whitney rank, to the right of each treatment lies the quartile chart for each. Each metric is either sorted by ranking, or in the case of a tie, descending $pd$ and $prec$ or ascending $pf$.}\label{plus-minus}
\end{table}

\begin{figure}
\small
\begin{center}
\begin{tabular}{|c|r|c|c|c|c|}\hline
data & performance & all        & dense      & over     & under\\
set & measure      & components & components & sampling & sampling\\\hline
CM1&	precision&	0&	0& 0& 0\\
	&	recall&			+&	-& -& -\\
	&	pf&			-&	+& -& -\\\hline
KC1&	precision&	0&	0& 0& 0\\
	&	recall&			0&	0& 0& 0\\
	&	pf&			0&	0& 0& 0\\\hline
MC1&	precision&	0&	0&  0&  0\\
	&	recall&			-&	-& 0& 0\\
	&	pf&			-& -& 0& 0\\\hline
PC1&	precision&	-&	+& -& -\\
	&	recall&			-&	+& -& -\\
	&	pf&			-&	+& -& -\\\hline
PC3&	precision&	0&	0&  0&  0\\
	&	recall&			-&	-& 0& 0\\
	&	pf&			-&	+& -& -\\\hline\hline
summary& +&         1& 5& 0& 0\\
       &0 &         6& 6& 9& 9\\
       &- &        8&  4& 6& 6\\\hline         
\end{tabular}
\end{center}
\caption[Dense vs. All vs. Sampling - all metrics per data
set.]{Each treatment is assigned one of $\{+,0,-\}$ depending
on if it $won,tied,lost$ in the statistical rankings of Table~1
(based on a Mann-Whitney test at 95\% confidence). 
Note that $dense$ won five times more often than $all$, 
and lost the least amount of times compared to all other treatments ($4/15$ vs. $8/15$ and $6/15$). 
}\label{fig:summz}
\end{figure}

\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&Train on Dense Components& 31& 69& 91&  \boxplot{0}{31}{69}{91}{3}  \\
2&Train on All Components& 35& 71& 93&  \boxplot{0}{35}{71}{93}{3}    \\ \hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption[Dense Component vs. All Component learning - $pd$ - All Data]{$PD$ values for learning on dense components compared to learning on all components across all data sets, sorted by statistical ranking via a Mann-Whitney test at 95\% confidence.}\label{fig:pds-all}
\end{figure}

\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&Train on Dense Components& 0& 15& 52&  \boxplot{0}{0}{15}{52}{3}  \\
1&Train on All Components& 0& 26& 65&  \boxplot{0}{7}{26}{65}{3}    \\ \hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption[Dense Component vs. All Component learning - $pf$ - All Data]{$PF$ values for learning on dense components compared to learning on all components across all data sets, sorted by statistical ranking via a Mann-Whitney test at 95\% confidence.}\label{fig:pfs-all}
\end{figure}

\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{precision}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&Train on All Components& 20& 78& 95&  \boxplot{0}{20}{78}{95}{3}  \\
1&Train on Dense Components& 12& 75& 96&  \boxplot{0}{12}{75}{96}{3}    \\ \hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption[Dense Component vs. All Component learning - $precision$ - All Data]{$Precision$ values for learning on dense components compared to learning on all components across all data sets, sorted by statistical ranking via a Mann-Whitney test at 95\% confidence.}\label{fig:prec-all}
\end{figure}
