\section{Experiment \#1: WC-vs-CC}

\subsection{Design}
Our first WC-vs-CC experiments took the 12 example tables of \fig{data}
and built test sets from
10\% of the rows, selected at random.  Defect
predictors were then learned from:
\bi
\item Treatment 1 (CC): all rows from the other 11 tables.
\item Treatment 2 (WC):  just the other 90\% rows of this table;
\ei
The \fig{data} data
comes from
12
NASA systems.  These systems were developed
 in
different geographical locations across North America.
Within a system, the sub-systems
shared some a common code base but did not pass personnel or code
between sub-systems. 

The external validity of generalizing from NASA examples
like \fig{data}
has been discussed elsewhere~\cite{me07d}. In summary, 
NASA makes extensive use of
contractors who are contractually obliged (ISO-9O01) to demonstrate
their understanding and usage of current industrial best
practices. These contractors service many other industries;
for example, Rockwell-Collins builds systems for many government and
commercial organizations.  For these reasons, other noted researchers
such as Basili, Zelkowitz, et al.~\cite{basili02} have argued that
conclusions from NASA data are relevant to the general software
engineering industry.

Most of the \fig{data} tables comes from systems written in ``C/C++''
but at
least of one of the  system was written in JAVA.
For cross-company data, an industrial practitioner may not have access to 
detailed meta-knowledge (e.g. whether it was developed in ``C'' or JAVA).
They may only be aware
that data, from an unknown
source, is available for download from a certain url.
To replicate that scenario,  we will make no
use of our meta-knowledge about \fig{data}. As we shall see, 
a clear stable effect will occur
across all the tables, regardless of (say) the implementation language.

In order to control for   {\em order effects} (where the 
learned theory is unduly effected by the order of the examples)
the entire procedure was repeated 10 times
where, each time, the order of the rows in the table was randomized.
In all, we ran 1200 experiments to compare WC-vs-CC:
\[\left(\mbox{10*10\% test sets}\right) * \left(\mbox{10 randomized orderings}\right) * \left(\mbox{12 tables}\right)\]

All the numeric distributions in the \fig{data}
data
are exponential. A ``log-filter'' replaces all numerics $N$ with $log(N)$.
This 
spreads out exponential
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
This ``spreading'' can significantly
improve the effectiveness of data mining~\cite{me07d}.

In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem work best for the \fig{data} data~\cite{me07d}.
This theorem 
offers a relationship between  fragments of evidence $E_i$,
a prior probability for
a class $P(H)$, and a posteriori  probability
$P(H|E)$: 
\[
P(H|E)=P(H)/ P(E)\prod_iP(E_i|H)
\]
When building defect detectors,
the
posterior probability of each class (``defective'' or ``defect-free'')
is calculated, given the features
extracted from a 
module.
The module is assigned to the possibility
with the highest probability.  
For
numeric attributes, 
a feature's mean $\mu$ and standard deviation $\sigma$ is used in
a
Gaussian  probability
function~\cite{witten05}:
\[f(x) =
1/(\sqrt{2\pi}\sigma)e^{-\frac{(x-\mu)^2}{2\sigma^2}}\] 
Data mining effectiveness was measured using $pd$, $pf$ and $balance$.
Let $\{A,B,C,D\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor.
From these:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
Note that balance a combination function combining $pd$ and $pf$. It returns the 
distance to the ideal position of no false alarms and complete recall $(pf=0,pd=1)$,
normalized such that it runs 0..1. In other work we have added utility weights within
the equation to show that
(e.g.) $pd$ is ``x'' times more important than $pf$. Here, we show results with an even
balance (and future work could repeat this study for non-even balances).

All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% defection ($pd=1$).

Other commonly used measures such as $accuracy$  and $precision$
were not used since, as shown in \fig{data}, the percent of defective examples
in our tables was usually very small (median value around 8\%).
Accuracy and precision are
are poor indicators
of performance for data were the target class is so rare
(for more on this issue, see~\cite{me07d,me07e}). 

The WC and CC results were visualized using
{\em
quartile charts}.  To generate these charts, the performance
deltas for some treatment
are sorted to isolate the median and the lower and upper quartile of numbers.
For example:

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In our quartile charts,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the zero value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}

The Mann-Whitney U test~\cite{mann47} was used to test
for statistical difference between treatments.
This is a non-parametric test that replaces all (e.g.) $pd$ values with their
rank inside the population of all observed and sorted $pd$ values. The U tests then checks
if the median rank of the WC treatments
 were equal, less then, or greater than the CC treatments.
Such non-parametric tests are recommended in data mining since many of the performance
distributions are non-Gaussian~\cite{demsar06}. 

We report one deviation from our prior procedure. The tables of data 
come from different sources and, hence,
have different features. For this study, all the tables were pruned such
that they only contained features that appear in all the twelve tables (see appendix).

\subsection{Results from Experiment \#1}

\fig{ccwsnum} shows the $\{pd,pf\}$ quartile charts for CC vs WC data.
The trend is very clear: CC data dramatically increases {\em both}
the probability of detection and the probability of false alarms. The $pd$
results are particularly striking:
\bi
\item 25\% of the $pd$ values are at 100\%.
\item 50\% of the $pd$ values are above 90\%
\item 75\% of the $pd$ values are at or above 80\%;
\item And all the $pd$ values are at or over 50\%.
\ei
By way of comparison, recall from the above that
than our previous result had a  average $pd$ of $71\%$~\cite{me07d}.

\begin{figure}
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       50&  80&  91&  100& 100& \boxplot{50}{30}{91}{100}{0}\\
 & WC&       17&  67&  80&   88& 100& \boxplot{17}{50}{80}{88}{12}\\\hline
 pf & CC&        4&  26&  52&   73& 100& \boxplot{4}{22}{52}{73}{27}\\
 & WC&        0&  17&  26&   33&  73& \boxplot{0}{17}{26}{33}{40}\\
\hline\end{tabular}
}
\end{center}
\caption{Experiment \#1 results.
Numeric results on left; quartile charts on right. ``Q1'' and ``Q3'' denote
the 25\% and 75\% percentile points (respectively). }\label{fig:ccwsnum}
\end{figure}
\begin{figure}[!b]
\begin{center}
{\footnotesize
\begin{tabular}{|r|c|c|p{2cm}|l|}\hline
%pd(inter) vs  & pf(intra) vs & &  \\
&$pd$    & $pf$ & &  \\
group & $WC \rightarrow CC$    &  $WC \rightarrow CC$& tables & $|tables|$ \\\hline\hline
a& increased               &        increased & CM1 KC1 KC2 MC2 MW1 PC1 PC3 PC4 & 8\\\hline
b& same                    & same             & KC3& 1\\\hline
c& same                    &  increased            & MC1 PC2 &2\\\hline
d& decreased      		 & decreased        &     PC5 &1\\\hline
\end{tabular}}
\end{center}
\caption{U test results (95\% confidence): moving from WC to CC}\label{fig:ccwc}
\end{figure}


To the best of our knowledge, \fig{ccwsnum} are the largest $pd$ values ever obtained
from this data.
However, these very high $pd$ values come at some considerable cost. Note in \fig{ccwsnum}
that the median false alarm
rate has changed from 26\% (with WC) to 52\% (with CC) and the maximum $pf$ rate
now reaches 100\%.

Once a {\em general result} is defined (e.g. CC dramatically
increases both $pf$ and $pd$), it is good practice to look for specific exceptions
to that pattern.
\fig{ccwsnum} shows
results 
from all results from all tables.
\fig{ccwc} shows a summary of results when the U test (95\% confidence)
was applied to
test results from each of the 12 tables,  in isolation 
In the majority case ($\frac{8}{12}$, see group $a$), the general pattern repeats where
using CC instead of WC increases both $pd$ and $pf$ (see group $a$). 
In one case (see group $b$), there was no difference in the results of the different
treatments. In another case, two tables of data (see group $c$) saw no change to
$pd$ but the false alarm grew worse.
In only one case was there a clear contradiction to the majority
case. In group $d$, which only contains the $pc5$ table, using cross-company data
made the $pf$ better (decreased it) at the
cost of also decreasing the $pd$.
In summary, the {\em general result} 
holds in the majority of cases and is only clearly contradicted in the $pc5$ table.

We explain these increases in $pd,pf$ with the following speculation.
Using  a large training set (e.g. eleven of the tables in \fig{data}) informs
not only all the causes of errors, but
also of numerous irrelevancies (e.g. applying statistics gathered from JAVA
programs to ``C'' programs).  Hence, large training sets increase the probability
of detection (since there are more known sources of errors) as well
as the probability of false alarms (since there are more extraneous factors
introduced to the analysis).

As to the contradictory $pc5$ result, 
recall from \fig{data} that this table is the far and away the largest data set used
in this study. $Pc5$ contains $\frac{17,186}{39,158}=43\%$ of the modules, in a single
data set.
Cross-company effects may be invisible when relatively small number of examples
are added to a data set. Hence, it is not
surprising that effects seen in the other, much smaller,
tables of data do not hold in such an exceptionally large table of data.

\subsection{Discussion of Experiment \#1}
When practitioners use defect detectors with high false alarm rates,
they must allocate a large portion of their debugging budget to 
the unfruitful exploration of erroneous alarms.
Hence:
\bi
\item
 For most software applications, very high  $pf$ rates like the WC results
of \fig{ccwsnum} make the detectors impractical to use.
\item
Therefore, we can only recommend cross-company learning 
for mission critical software
were the extra costs associated with high false alarm rates  
are compensated by the associated increase in software assurance.
\ei
When budgetary considerations restrict the exploration of numerous false
alarms,  projects seek detectors with lower $pf$ values and adequate $pd$
values. Our next experiment addresses this concern.


