\section{Methodology}
\subsection{Data}
\input{section-III-new}
%\begin{figure*}[!b]
\begin{table}[ht]
\caption{Descriptions of ten software projects used in the paper.
The rows labeled ``NASA'' come from NASA aerospace projects while the rows
labeled ``SOFTLAB'' come  from a Turkish software company writing
applications for domestic appliances.}
\begin{tabular}{l|lll}
\scriptsize
&     &       &   \\
source
&project & language & description \\ \hline
NASA&pc1  & C++ &Flight software for earth orbiting satellite\\
NASA&kc1  & C++ &Storage management for ground data\\
NASA&kc2  & C++ &Storage management for ground data\\
NASA&cm1  & C++ &Spacecraft instrument\\
NASA&kc3  & JAVA &Storage management for ground data\\
NASA&mw1  & C++ &A zero gravity experiment related to combustion\\
SOFTLAB& ar4 & C &Embedded controller for white-goods\\
SOFTLAB& ar3& C &Embedded controller for white-goods\\
NASA&mc2  & C++ &Video guidance system \\
SOFTLAB & ar5 & C &Embedded controller for white-goods\\
\end{tabular}
\label{tbl:data1}
\end{table}

\begin{table}[ht]
\caption{Summary of data from ten software projects of \tbl{data1}, sorted in order of number of functional units.}
\begin{tabular}{l|lrrrrr@{.}l}
\scriptsize
&     &     (\# modules)         &          &  & & &\\
source &project  & examples & features  & loc &   \multicolumn{3}{r}{\%defective}\\\hline
NASA&pc1  & 1,109    & 21       &25,924&&    6&94\\
NASA&kc1  & 845      & 21       &42,965&&    15&45\\
NASA&kc2  & 522      & 21        &19,259&&   20&49\\
NASA&cm1  & 498      & 21        &14,763&&   9&83\\
NASA&kc3  & 458      & 39      &7749&&     9&38\\
NASA&mw1  & 403      & 37      &8341&&     7&69\\
SOFTLAB& ar4 & 107 &  29       &9196&     & 18 & 69\\
SOFTLAB& ar3& 63 &    29       &5624&   & 12 & 70\\
NASA&mc2  & 61      & 39         &6134& & 32&29\\
SOFTLAB & ar5 & 36 &  29       &2732&  & 22 & 22\\
\cline{5-4}
     \multicolumn{4}{c}{~}     &  4,102 
\end{tabular}
\label{tbl:data2}
\end{table}

%\end{figure*} 
%The experiments of this paper use the data of \tbl{data}, which are
%downloaded from the PROMISE repository\footnote{\url{http://promisedata.org/repository}}.
%We do not use defect data from open source software (OSS). 
%Defect reports in OSS can only be mapped to components or files at best. Since we study a finer granularity level, i.e. module: smallest unit of functionality, data from OSS are not appropriate for our work. 
%The static code features of our data are  shown in \tbl{attr}.
%These features divide into lines of code features, Halstead features, and
%McCabe features. These features are explained in \fig{halstead} and \fig{mccabe}.
%We use static code features simply because our industrial data sources include only these. There are
%other defect prediction studies using churn metrics, however either those data are not publicly available or from OSS~\cite{zimmermann-tse-2005,zimmermann-evobook-2008,dallmeier-ase-2007,DBLP:conf/icse/WaliaCN08,DBLP:conf/icse/NagappanB05}. Churn metrics are usually collected at the file level, which does not fit to our research (i.e. we analyze a finer granularity level). These studies usually focus on predicting the number of defects at the file\ component level, whereas we treat defect prediction as a classification problem rather than a regression problem. 
%\begin{center}
%\begin{table}
%\caption{Features used in this study. The Halstead features are explained in \fig{halstead}
%and the Mccabe features are explained in \fig{mccabe}.}\label{fig:attr}
%\scriptsize
%\begin{tabular}{l@{~}|l@{~}|l@{~}|c@{~}p{3.9cm}|}\cline{2-5}
%%\begin{tabular}{|l|l|lcl|}\hline
%%\multicolumn{3}{|c}{grouping}&\multicolumn{2}{|c|}{measure}\\\hline
%		&\multicolumn{2}{l}{m = Mccabe }       &$v(g)$  &cyclomatic\_complexity  \\
%        &        \multicolumn{2}{l}{~}                                 & $iv(G)$&design\_complexity\\
%        &         \multicolumn{2}{l}{~}                                & $ev(G)$&essential\_complexity\\\cline{2-5}
%        &locs&{loc}                         &   &loc\_total (one line = one count\\\cline{3-5}
%	&	&loc(other)                      &   &loc\_blank           \\
%        &       &                                &   &loc\_code\_and\_comment\\
%        &       &                                &   &loc\_comments         \\
%        &        &                                &   &loc\_executable\\
%        &       &                                &    &number\_of\_lines (opening to closing brackets)\\\cline{2-5}
%	&{Halstead} & h& $N_1$ &num\_operators\\
%        &       &                                 &$N_2$ &num\_operands \\
%        &       &                                 &$\mu_1$ &num\_unique\_operators\\
%        &       &                                 &$\mu_2$ &num\_unique\_operands\\\cline{3-5}
%        &       &  { H}&$N$  &length: $N= N_1 + N_2$\\
%        &       &                                 & $V$  &volume: $V=N * log_{2}\mu$\\
%        &       &                                 & $L$  &level: $L=V^*/V$ where\newline $V^*=(2+\mu{_2}^{*})log_{2}(2+\mu{_2}^{*})$\\
%        &       &                                 & $D$  &difficulty: $D=1/L$\\
%        &       &                                 &$I$   &content: $I=\hat{L}*V$ where\newline $\hat{L}=\frac{2}{\mu_{1}} * \frac{\mu_{2}}{N_{2}}$ \\
%        &       &                                 & $E$  &effort: $E=V/\hat{L}$\\
%        &       &                                 & $B$  &error\_est\\
%        &       &                                 & $T$  &prog\_time: $T=E/18$ seconds\\\cline{2-5}
%\end{tabular}
%\end{table}
%\end{center}
\begin{table}
\begin{center}
\caption{Static code features available in \tbl{data2} projects.}\label{tbl:attr1}
\scriptsize
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|}\hline
\#&Feature &pc1 &kc1 &kc2 &cm1 &kc3 &mw1 & ar4& ar3&mc2&ar5\\\hline \hline
1&$branch count $ &X &X &X &X &X &X &X & X&X & X \\\hline 
2&$code and comment loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
3&$comment loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
4&$cyclomatic complexity $&X &X &X &X &X &X &X & X&X & X \\\hline 
5&$design complexity $ &X &X &X &X &X &X &X & X&X & X \\\hline 
6&$halstead difficulty $ &X &X &X &X &X &X &X & X&X & X \\\hline 
7&$halstead effort $ &X &X &X &X &X &X &X & X&X & X \\\hline 
8&$halstead error $ &X &X &X &X &X &X &X & X&X & X \\\hline 
9&$halstead length $ &X &X &X &X &X &X &X & X&X & X \\\hline 
10&$halstead time $ &X &X &X &X &X &X &X & X&X & X \\\hline 
11&$halstead volume $ &X &X &X &X &X &X &X & X&X & X \\\hline 
12&$total operands $ &X &X &X &X &X &X &X & X&X & X \\\hline 
13&$total operators $ &X &X &X &X &X &X &X & X&X & X \\\hline 
14&$unique operands $ &X &X &X &X &X &X &X & X&X & X \\\hline 
15&$unique operators $ &X &X &X &X &X &X &X & X&X & X \\\hline 
16&$executable loc$ &X &X &X &X &X &X &X & X&X & X \\\hline 
17&$total loc$  &X &X &X &X &X &X &X & X&X & X \\\hline 
18&$halstead content $ &X &X &X &X & X &X & & &X& \\\hline
19&$essential complexity$ &X &X &X &X & X &X & & &X&  \\\hline 
20&$halstead vocabulary $ & X& X& X&X  &  & &X & X&  & X \\\hline
21&$blank loc$  &X &X &X &X &X &X &X & X&X & X  \\\hline
22&$call pairs $ & & & & &X &X &X & X& X& X  \\\hline
 23&$condition count $ & & & & &X &X &X & X& X& X  \\\hline
 24&$cyclomatic density $ & & & & &X &X &X & X& X& X  \\\hline
 25&$decision count $ & & & & &X &X &X & X& X& X  \\\hline
 26&$decision density $ & & & & &X &X &X & X& X& X  \\\hline
27&$halstead level $ & & & & &X &X &X & X& X& X  \\\hline
28&$multiple condition count $ & & & & &X &X &X & X& X& X  \\\hline
29&$design density $ & & & & &X &X &X & X& X& X \\\hline
30&$norm cyclomatic comp. $ & & & & &X &X &X & X& X& X \\\hline 
31&$formal parameters $ & & & & &X &X &X & X& X& X  \\\hline
32&$modified condition count $ & & & & & X &X & & &X&   \\\hline 
33&$maintenance severity$ & & & & & X &X & & &X&   \\\hline
 34&$edge count $ & & & & & X &X & & &X&   \\\hline
 35&$node count $ & & & & & X &X & & &X&   \\\hline
 36&$essential density $ & & & & & X &X & & &X&   \\\hline
 37&$global data complexity$ & & & & & X & & & &X&   \\\hline
  38&$global data density $ & & & & & X & & & &X&   \\\hline 
 39&$percent comment $ & & & & & X &X & & &X&  \\\hline 
  40&$number of lines $ & & & & & X &X & & &X&   \\\hline \hline
&Total &21 &21 &21 &21 &39 &37 &29 & 29&39 & 29  \\\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}
\begin{center}
\caption{Static code features shared by NASA and SOFTLAB projects.}\label{tbl:attr2}
\scriptsize
\begin{tabular}{|c|c|c|c|}\hline
\#&Feature &NASA shared&All Shared \\\hline \hline
1&$branch count $ &X&X\\\hline 
2&$code and comment loc$ &X&X \\\hline 
3&$comment loc$ &X&X\\\hline 
4&$cyclomatic complexity $&X&X\\\hline 
5&$design complexity $ &X&X\\\hline 
6&$halstead difficulty $ &X&X\\\hline 
7&$halstead effort $ &X&X\\\hline 
8&$halstead error $ &X&X\\\hline 
9&$halstead length $ &X&X\\\hline 
10&$halstead time $ &X&X\\\hline 
11&$halstead volume $ &X&X\\\hline 
12&$total operands $ &X&X\\\hline 
13&$total operators $ &X&X\\\hline 
14&$unique operands $ &X&X\\\hline 
15&$unique operators $ &X&X\\\hline 
16&$executable loc$ &X&X\\\hline 
17&$total loc$  &X&X\\\hline 
18&$halstead content $ &X&\\\hline
19&$essential complexity$ &X&  \\\hline 
\hline
&Total &19&17 \\\hline
\end{tabular}
\end{center}
\end{table}

Our project data are taken from software developed
in
different geographical locations across North America (NASA) and Turkey (SOFTLAB). Therefore, the static code features that are available for each project vary. While \tbl{attr1} shows the features available for each project, \tbl{attr2} shows the common features for both sources.
%Within a system, the sub-systems
%shared some common code base but did not pass personnel or code
%between sub-systems. 
While NASA and SOFTLAB are one single {\em source} of data, there are several projects within each source.
For example, NASA 
is really an umbrella organization  used to co-ordinate and fund
a large and diverse set of projects:
\bi
\item
The NASA data was
collected from across the United States over a 
a period of five years from numerous
NASA contractors working at different geographical centers.  
\item
These projects
represent a wide
array of projects, including satellite instrumentation, ground control
systems and partial flight control modules (i.e. Attitude Control).  
\item
The
data sets also represent a wide range of code reuse: some of the projects
are 100\% new, and some are modifications to previously deployed code. 
\ei

That is, even if we explore just the NASA data sets, we can still
examine issues of cross- vs within- company data use.
Nevertheless, using our connections with the Turkish software industry, we collected
new data sets in the format of \tbl{attr1} from a Turkish white- goods manufacturer. 
The SOFTLAB datasets (\{ar3, ar4, ar5\}) in \tbl{data1},
are the 
controller software for:
\bi
\item
A washing machine;
\item
 A dishwasher;
\item And a
refrigerator. 
\ei

In summary, 7 datasets are from NASA projects developed at different sites by different teams, hence we treat each of them as if they were from 7 different companies. Remaining 3 datasets are from a Turkish company collected from software for domestic home appliances. Therefore we use 10 projects from 8 different companies. 

\subsection{Naive Bayes Classifier}
Each project in \tbl{data2} contains information from many
modules;  the smallest unit of functionality\footnote{
In other languages, modules may be called ``function'' or
``method''.}.  
To learn defect predictors,  the project data are
augmented with 
one dependent variable holding  boolean
values for
``defects detected''.
The data mining task is to find combinations of static code features that predict
for the boolean value of the dependent variable.

In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem yields better performance than rule based methods (i.e. decision trees, oneR), for the 
\tbl{data2} NASA data\footnote{SOFTLAB data were not available at that time.}~\cite{me07b}.
An extensive study by Lessmann et.al also shows that Naive Bayes performs equivalently well with 15 other methods~\cite{lessmann09}. Therefore, we find Naive Bayes as a viable choice as a classifier to use in our analysis. 

A N{aive} Bayes classifier~\cite{duda76}
is
based on Bayes' Theorem. 
Informally, the
theorem says $next=old*new$ i.e. what we'll believe {\em next} comes
from how {\em new} evidence effects {\em old} beliefs.  More formally:
\begin{equation}\label{eq:nb}
P(H|E)=\frac{P(H)}{P(E)}\prod_iP(E_i|H)
\end{equation}
i.e. given fragments of evidence $E_i$ and a prior probability for
a class $P(H)$, the theorem lets us calculate a  posterior  probability
$P(H|E)$. 
For example, in our data sets, there are two hypotheses: modules are either
defective or not:
\mbox{$H\in \left\{ defective,nonDefective \right\}$}.
Also, if a particular module has $numberofSymbols=27$ and
$LOC=40$ and was previously classified
as ``defective'' then
\[
\begin{array}{lcl}
E_1&:& numberOfSymbols=27\\
E_2& :& LOC=40\\
H&:&  defective
\end{array}
\]

When building defect predictors,
the
prior probability of each class (``defective'' or ``defect-free'')
is calculated, given the defect content of each 
module.
So, if a data set has 100 modules and 25 of them are faulty, then:
\[P(defective)=0.25\]
When testing new data, a module is assigned to the class
with the higher posterior probability, calculated from \eq{nb}.


N{aive} Bayes classifiers are called ``naive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(only the frequency
counts are required for each feature),
 it is possible
that
correlated events are missed by this ``naive'' approach.
Domingos and Pazzani show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, N{aive} Bayes classifiers perform as well
as other seemingly more sophisticated schemes\footnote{ Details of this issue are out of the scope of this paper. For more, please see Table~1 in~\cite{domingos97optimality}}.

% BEGIN DETAILED BAYES
\eq{nb} offers a simple method for handling missing values. Generating a
posterior probability means of tuning a prior probability
to new evidence. If that evidence is missing,
then no tuning is needed. In this case \eq{nb} sets $P(E_i|H)=1$ which, in effect,
makes no change to $P(H)$.

When estimating the prior probability of hypothesis $H$, 
it is common practice~\cite{yang02} to  use 
 an {\em M-estimate} as follows.
Given 
that the total number of hypothesis is $C$,
the total number of training instances is $I$, and  $N(H)$ is the frequency the hypothesis $H$ within
$I$, then
\begin{equation}\label{eq:b}
P(H) = \frac{N(H) + m}{I + m\cdot C}
\end{equation}
Here $m$ is a small non-zero constant (often, $m=1$). Three special cases of \eq{b} are:
\bi
\item
For high frequency hypothesis 
in
large training sets,
$N(H)$ and $I$ 
 are much larger than 
$m$ and $m\cdot C$, so  \eq{b} simplifies
to 
$P(H)= \frac{N(H)}{I}$, as one might expect. 
\item
For low frequency classes
in large training sets,
$N(H)$ is small, $I$ is large,
and 
the prior probability for a rare class is never less than $\frac{1}{I}$;
i.e. the inverse of the number
of instances. If this were not true, rare classes would never appear in predictions.  
\item
For very small data sets, $I$ is small and $N(H)$ is even smaller.
In this case,
\eq{b} approaches the inverse of the number of classes; i.e. $\frac{1}{C}$.
This is a useful approximation when learning from very small data sets 
when all the data relating to a certain class has not yet been seen.
\ei
The prior probability calculated in \eq{b} is a useful lower bound for $P(E_i|H)$.
If some value $v$ is seen $N(f=v|H)$ times in feature $f$'s observations for 
hypothesis $H$, then\begin{equation}\label{eq:laplace}
P(E_i|H) = \frac{N(f=v|H) +l\cdot P(H)}{N(H) + l} 
\end{equation} 
Here, $l$ is the {\em L-estimate} and is set to a 
small constant (Yang \&Webb~\cite{yang02} recommend $l=2$).
Two special cases of are:
\bi
\item
A common situation is when there are many examples of 
an  hypothesis 
and numerous observations have been made
for a particular value. In that situation, 
$N(H)$ and $N(f=v|H)$ are large and
\eq{laplace}
approaches \mbox{$\frac{N(f=v|H)}{N(H)}$}, as one might expect. 
\item
In the case of very
little evidence 
for a rare hypothesis,
\mbox{$N(f=v|H)$} and $N(H)$ are small and \eq{laplace}
approaches $\frac{l\cdot P(H)}{l}$; i.e. the default frequency of an observation in  a hypothesis
is a fraction of
the probability of that hypothesis. This is a useful approximation when very
little data is available. 
\ei

% END DETAILED BAYES
For
numeric features it is common practice for N{aive} Bayes classifiers
to use the Gaussian probability
density function~\cite{witten05}:
\begin{equation}\label{eq:normal}
g(x) =
\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\end{equation} where
$\{\mu,\sigma\}$ are the feature's \{mean,standard
deviation\}, respectively.
To be precise, the probability of a
continuous feature having exactly the  value $x$ is zero,
but the probability that it lies within a small region, say $x \pm
\epsilon/2$, is $\epsilon \times g(x)$.  Since $\epsilon$ is a
constant that weighs across all possibilities, it cancels out and
needs not be computed. 

All the static code features of  \tbl{attr1} are numeric and 
are highly skewed. Therefore, we replace all numeric values with a "log-filter", i.e. $N$ with $ln(N)$.
This 
spreads out skewed
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
This "spreading'' can significantly
improve the effectiveness of data mining, since the distribution of log-filtered feature values fits better
to the normal distribution assumption ~\cite{me07b}.

%Simple  Bayes classifiers are often called ``naive'' since they assume
%independence of each feature. While this assumption
% simplifies the implementation  
%(frequency
%counts are required only for each feature),
% it is possible
%that
%correlated events are missed by this ``naive'' approach.
%Potentially, this is a significant problem for our kinds of data sets where
%the static code measures are highly correlated (e.g. the number of
%symbols in a module increases linearly with module lines of code). 

%In a previous work, we have investigated variants of Bayesian models considering this issue and observed that the independence assumption of naive Bayes is safe for the Nasa datasets~\cite{turhan07qsic}. In another work, we have also performed extensive feature subset selection (FSS) experiments, where similar prediction performances were obtained using small subsets of features selected by Information Gain~\cite{me07b}. However, we did not observe an increase in the prediction performances when we used whole available data. 
%Furthermore, 
%Domingos and Pazzani show theoretically
%that the independence assumption is a problem in a vanishingly small percent
%of cases~\cite{domingos97optimality}. This explains the repeated
%empirical result that, on average, seemingly naive Bayes classifiers perform as well
%as other seemingly more sophisticated schemes (e.g. see Table~1 in~\cite{domingos97optimality}).

%The Domingos and Pazzani result
%also explains our prior experiments where naive Bayes did not perform
%worse than other learners that continually re-sample the data for dependent
%instances (e.g. decision-tree learners that recurse
%on each ``split'' of the data~\cite{quinlan92}).
\subsection{Performance Evaluation}
Data mining effectiveness was measured using $pd$, $pf$ and $balance$~\cite{me07b,me07e}.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% detection ($pd=1$).

Other measures such as $accuracy$  and $precision$
were not used since, as shown in \tbl{data2}, the percent of defective examples
in our projects was usually very small (median value around 13\%).
Accuracy and precision
are poor indicators
of performance for data where the target class (i.e. defective) is so rare
(for more on this issue, see~\cite{me07b,me07e}). 

The results were visualized using
{\em
quartile charts}.  To generate these charts, the performance
deltas for some treatment
are sorted to isolate the median and the lower and upper quartile of numbers.
For example:

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In our quartile charts,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the 50\% percentile value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
The Mann-Whitney U test~\cite{mann47} was used to test
for statistical difference between results.
This non-parametric test replaces performance measure values (i.e. pd, pf, bal) with their
rank inside the population of all sorted values. 
Such non-parametric tests are recommended in data mining since many of the performance
distributions are non-Gaussian~\cite{demsar06}. 


