\documentclass[twocolumn,journal,compsoc]{IEEEtran}

\usepackage{pifont}
\usepackage{times}
\usepackage{alltt}
\usepackage{graphicx}
\usepackage{url}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\makeatletter
  \AddToShipoutPicture{%
    \setlength{\@tempdimb}{.5\paperwidth}%
    \setlength{\@tempdimc}{.5\paperheight}%
    \setlength{\unitlength}{1pt}%
      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
\makeatother

%\documentclass[onecolumn,draftcls,journal,compsoc]{IEEEtran}
%\documentclass[technote]{IEEEtran}
%\documentclass[twocolumn,technote,9pt,compsoc]{IEEEtran}
\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		%\put(0,0){\line(0,1){8}}
		%\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage[nocompress]{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
%\usepackage[nomarkers]{endfloat}
%\let\MYoriglatexcaption\caption
%\renewcommand{\caption}[2][\relax]{\MYoriglatexcaption[#2]{#2}}
\begin{document}

\title{Cross- vs Within- Company Defect Prediction }


\author{Tim~Menzies,~\IEEEmembership{Member,~IEEE,}
Burak~Turhan,~\IEEEmembership{Student~Member,~IEEE,}\\
        Ayse~Bener,~\IEEEmembership{Member,~IEEE,} 
		and Justin~Distefano
\thanks{Dr. Menzies and  Mr. Distefano
are with the Lane Department of Computer Science and Electrical Engineering, 
West Virginia
University.
Mr. Distefano is also Chief Programmer at Integrated Software Metrics.
Emails: \protect\url{tim@menzies.us}
and \protect\url{jdistefano@ismwv.com}
}
\thanks{Mr. Turhan and Dr. Bener are with the 
Department of Computer Engineering, Bogazici University, Turkey.
Emails:
\protect\url{turhanb@boun.edu.tr},
\protect\url{bener@boun.edu.tr}.
}\thanks{The research described in this paper
was supported by
Bogazici University research fund      under grant number BAP-06HA104
and at
West Virginia  University
under grants with NASA's Software Assurance Research Program.
Reference herein to any specific commercial product, process, or service
by trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States Government.
}\thanks{See 
See \protect\url{http://menzies.us/pdf/07interra.pdf} for an earlier draft of this paper.
Manuscript received October 1, 2007; revised XXX,}} 

\markboth{Journal of ???,~Vol.~6, No.~1, January~2007}%
{Menzies \MakeLowercase{\textit{et al.}}: Cross- vs Within-Company Defect Estimation Studies}

%We found that the cross-company model provided poor 
%predictions for the single-company projects and much worse 
%predictions than the single-company model.
\maketitle

\begin{abstract}
Cross-company (CC) data are often the  preferred option when
collecting within-company (WC) data is too expensive or time-consuming.
We show here that CC data can also lead to very high probabilities
of detecting faulty modules (median value of 12 data sets: 91\%).

Nevertheless, we recommend {\em against} CC data since
such data yield very large false alarm rates (median value: 52\%).
WC data, on the other hand, halve the false
alarm rate while decreasing the CC prediction rates by only $\approx10\%$.
We also found that very small amounts of WC data (often, just 100 modules)
are adequate for building defect predictors.
That is, data collection for WC data should rarely be an arduous task.
Hence, for defect prediction,
we  strongly recommend using within-company data and not cross-company data.

KEYWORDS: learning; metrics (product metrics); cross-company; within-company
\end{abstract}

%------------------------------------------------------------------------- 

~\\

\begin{tabular}{|p{.90\linewidth}|}\hline
{\em This is a second draft of a TSE submission. For comments on the changes
between this draft and the last, see the appendix,}\\\hline
\end{tabular}

\input{usa/introduction}
\input{usa/related}
\input{usa/background}

\section{Experiment \#1: WC-vs-CC}
\subsection{Design}
Our first WC-vs-CC experiments repeated the following procedure
for all  12 example tables of \fig{data}.
For each table,
test sets were built from
10\% of the rows, selected at random.  Defect
predictors were then learned from:
\bi
\item Treatment 1 (CC): all rows from the other 11 tables.
\item Treatment 2 (WC):  just the other 90\% rows of this table;
\ei
Most of the \fig{data} tables come from systems written in ``C/C++''
but at
least one of the systems was written in JAVA.
For cross-company data, an industrial practitioner may not have access to 
detailed meta-knowledge (e.g. whether it was developed in ``C'' or JAVA).
They may only be aware
that data, from an unknown
source, are available for download from a certain url.
To replicate that scenario,  we will make no
use of our meta-knowledge about \fig{data}. As we shall see, 
a clear stable effect will occur
across all the tables, regardless of (say) the implementation language.

In order to control for   {\em order effects} (where the 
learned theory is unduly affected by the order of the examples)
our procedure was repeated 20 times,
randomizing the order of the rows in the table each time.
In all, we ran 480 experiments to compare WC-vs-CC:
\[\left(\mbox{2 treatments}\right) * \left(\mbox{20 randomized orderings}\right) * \left(\mbox{12 tables}\right)\]

All the numeric distributions in the \fig{data}
data
are exponential. A ``log-filter'' replaces all numerics $N$ with $log(N)$.
This 
spreads out exponential
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
This ``spreading'' can significantly
improve the effectiveness of data mining~\cite{me07b}.

\input{usa/bayes}
Data mining effectiveness was measured using $pd$, $pf$ and $balance$.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% detection ($pd=1$).

Other measures such as $accuracy$  and $precision$
were not used since, as shown in \fig{data}, the percent of defective examples
in our tables was usually very small (median value around 8\%).
Accuracy and precision
are poor indicators
of performance for data where the target class is so rare
(for more on this issue, see~\cite{me07b,me07e}). 

The WC and CC results were visualized using
{\em
quartile charts}.  To generate these charts, the performance
deltas for some treatment
are sorted to isolate the median and the lower and upper quartile of numbers.
For example:

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In our quartile charts,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the 50\% percentile value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
The Mann-Whitney U test~\cite{mann47} was used to test
for statistical difference between treatments.
This non-parametric test replaces (e.g.) $pd$ values with their
rank inside the population of all sorted $pd$ values. 
Such non-parametric tests are recommended in data mining since many of the performance
distributions are non-Gaussian~\cite{demsar06}. 

We report one deviation from our prior procedure. The tables of data 
come from different sources and, hence,
have different features. For this study, all the tables were pruned such
that they only contained features that appear in all the twelve tables.


\subsection{Results from Experiment \#1}

\fig{ccwsnum} shows the $\{pd,pf\}$ quartile charts for CC vs WC data.
and�\fig{ccwsnum} shows the $\{pd,pf\}$ quartile charts for CC vs WC data.
The trend is very clear: CC data dramatically increases {\em both}
the probability of detection and the probability of false alarms. The $pd$
results are particularly striking. For cross-company data:
\bi
\item 25\% of the $pd$ values are at 100\%.
\item 50\% of the $pd$ values are above 90\%
\item 75\% of the $pd$ values are at or above 80\%;
\item And all the $pd$ values are at or over 50\%.
\ei
\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       50&  80&  91&  100& 100& \boxplot{50}{30}{91}{100}{0}\\
    & WC&       17&  67&  80&   88& 100& \boxplot{17}{50}{80}{88}{12}\\\hline
 pf & CC&        4&  26&  52&   73& 100& \boxplot{4}{22}{52}{73}{27}\\
    & WC&        0&  17&  26&   33&  73& \boxplot{0}{17}{26}{33}{40}\\
\hline\end{tabular}
}
\end{center}
\caption{Experiment \#1 results for NASA tables.
Numeric results on left; quartile charts on right. ``Q1'' and ``Q3'' denote
the 25\% and 75\% percentile points (respectively). 
The upper quartile of the first row is  not visible  since
it runs from 100\% to 100\%; i.e. it has zero length.}\label{fig:ccwsnum}
\end{figure}



\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{|r|c|c|p{2cm}|l|}\hline
%pd(inter) vs  & pf(intra) vs & &  \\
&$pd$    & $pf$ & &  \\
group & $WC \rightarrow CC$    &  $WC \rightarrow CC$& tables & $|tables|$ \\\hline\hline
a& increased               &        increased & CM1 KC1 KC2 MC2 MW1 PC1 PC3 PC4 & 8\\\hline
b& same                    & same             & KC3& 1\\\hline
c& same                    &  increased            & MC1 PC2 &2\\\hline
d& decreased      		 & decreased        &     PC5 &1\\\hline
\end{tabular}}
\end{center}
\caption{Summary of
U-test results (95\% confidence): moving from WC to CC.
For full results, see~\fig{ccwsprojs}. }\label{fig:ccwc}
\end{figure}
By way of comparison, recall from the above that
our previous result had a  average $pd$ of $71\%$~\cite{me07b}.

To the best of our knowledge, \fig{ccwsnum} are the largest $pd$ values ever reported
from these data.
However, these very high $pd$ values come at some considerable cost. Note in \fig{ccwsnum}
that the median false alarm
rate has changed from 26\% (with WC) to 52\% (with CC) and the maximum $pf$ rate
now reaches 100\%.




We explain these increases in $pd,pf$ with the following speculation.
Using  a large training set (e.g. eleven of the tables in \fig{data}) informs
not only all the causes of errors, but
also of numerous irrelevancies (e.g. applying statistics gathered from JAVA
programs to ``C'' programs).  Hence, large training sets increase the probability
of detection (since there are more known sources of errors) as well
as the probability of false alarms (since there are more extraneous factors
introduced to the analysis).

\subsection{Sanity Checks on Experiment \#1}
This section explores 
threats to the external validity
of the Experiment\#1 conclusions. It can be skipped at first reading
of this paper.

Once a {\em general result} is defined (e.g. CC dramatically
increases both $pf$ and $pd$), it is good practice to check for 
specific exceptions
to that pattern. 
\fig{ccwsprojs} shows the $\{pd,pf\}$ quartile charts for 
Experiment \#1 for each NASA table and
\fig{ccwc} shows a summary of results when U tests
were applied to
test results from each of the 12 tables,  {\em
in isolation}:
\bi
\item 
Usually  ($\frac{8}{12}$), the general pattern still holds
(see group $a$). 
\item
In one case (see group $b$), there was no difference in the results of the different
treatments. 
\item
In another case, two tables of data (see group $c$) saw no change to
$pd$ but the false alarm grew worse.
\item
In only one case was there a clear contradiction to the majority
case. In group $d$, containing $pc5$, using cross-company data
decreased $pf$ at the
cost of also decreasing $pd$.
\ei
Overall, the {\em general result} 
holds in the majority of cases and is only clearly contradicted in the $pc5$ table.

$Pc5$'s anomalous behavior prompted the following investigation.
Observe in \fig{data} that the two largest data
sets ($pc5$ and $mc1$) contain more modules than the remaining
10 tables of data.
Therefore, it seemed prudent to repeat our experiment with and without
these two data sets. \fig{sanity1} repeats the cross-company 
parts of experiment \#1 without $pc1$, then without $pc1$ and $mc1$.
Note that the general pattern reported above (of large
$pd$ and $pf$ values) is not altered.
In fact, \fig{sanity1} changes very little from \fig{ccwsnum}.
It turns out that since these larger projects have the lowest
defect rates, they tend to introduce far more non-defective
modules than defective modules. When we remove the largest project
(which contains nearly half of all modules), we reduce this
bias and increase the median $pd$ (from 91\% in \fig{ccwsnum}
to 97\% in \fig{sanity1}).
If we further remove the next largest project ($mc1$), then
$pd$ does not change much. However,
   the 
   first $pf$ quartile jumps from 29\% to 45\%
since now we have removed a very
   large portion of non-defective samples so the model is more frequently saying
   ``defective''. 

In summary, $pc5$ has enough unusual features to let us safely discount
its anomalous results in \fig{ccwc}.

\begin{figure}
{\scriptsize
\begin{center}
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Without largest data set ($pc5$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       33&  83&  97&  100& 100& \boxplot{33}{50}{97}{100}{0}\\
 pf & CC&        14&  29&  54&   73& 98& \boxplot{14}{15}{54}{73}{25}\\
\hline\end{tabular}

~\\~\\

\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Without two largest data set ($pc5 + mc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       33&  83&  97&  100& 100& \boxplot{33}{50}{97}{100}{0}\\
 pf & CC&        12&  45&  55&   76& 98& \boxplot{12}{33}{55}{74}{22}\\
\hline\end{tabular}
\end{center}}
\caption{Variants on the experiment \#1 results.}\label{fig:sanity1}
\end{figure}

\begin{figure*}[!t]
%\begin{figure}[!t]
\begin{center}
\begin{tabular}{cc}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($cm1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       80& 100&  100&  100& 100& \boxplot{80}{20}{100}{100}{0}\\
    & WC&       40&  60&  80&   100& 100& \boxplot{40}{20}{80}{100}{0}\\\hline
 pf & CC&        87&  91&  96&   96& 98& \boxplot{87}{4}{96}{96}{2}\\
   & WC&        24&  27&  33&   38&  47& \boxplot{24}{3}{33}{38}{9}\\
\hline\end{tabular}

%~\\~\\
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 88&  94&  94& 100& \boxplot{82}{6}{94}{94}{6}\\
    & WC&       64&  73&  82&   85& 97& \boxplot{64}{9}{82}{85}{12}\\\hline
 pf & CC&        47&  49&  51&   53& 57& \boxplot{47}{2}{51}{53}{4}\\
   & WC&        27&  34&  36&   38&  40& \boxplot{27}{7}{36}{38}{2}\\
\hline\end{tabular}

%~\\~\\
}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       82& 91&  91&  100& 100& \boxplot{82}{9}{91}{100}{0}\\
    & WC&       55&  73&  82&   91& 100& \boxplot{55}{18}{82}{91}{9}\\\hline
 pf & CC&        57&  62&  64&   74& 81& \boxplot{57}{5}{64}{74}{7}\\
   & WC&        14&  24&  31&   33&  45& \boxplot{14}{10}{31}{33}{12}\\
\hline\end{tabular}

%~\\~\\
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($kc3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       60& 80&  80&  80& 100& \boxplot{60}{20}{80}{80}{20}\\
    & WC&       40&  60&  80&   80& 100& \boxplot{40}{20}{80}{80}{20}\\\hline
 pf & CC&        14&  19&  24&   31& 38& \boxplot{14}{5}{24}{31}{7}\\
   & WC&        10&  17&  21&   26&  36& \boxplot{10}{7}{21}{26}{10}\\
\hline\end{tabular}

%~\\~\\
}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       57& 71&  86&  86& 100& \boxplot{57}{14}{86}{86}{14}\\
    & WC&       57&  71&  86&   86& 100& \boxplot{57}{14}{86}{86}{14}\\\hline
 pf & CC&        17&  18&  19&   20& 21& \boxplot{17}{1}{19}{20}{1}\\
   & WC&        15&  16&  17&   18&  20& \boxplot{15}{1}{17}{18}{2}\\
\hline\end{tabular}

%~\\~\\
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       50& 67&  83&  100& 100& \boxplot{50}{17}{83}{100}{0}\\
    & WC&       17&  33&  67&   67& 83& \boxplot{17}{10}{67}{67}{18}\\\hline
 pf & CC&        55&  64&  73&   73& 100& \boxplot{55}{9}{73}{73}{27}\\
   & WC&        0&  27&  36&   45&  73& \boxplot{0}{27}{36}{45}{28}\\
\hline\end{tabular}

}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($mw1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       75& 75&  100&  100& 100& \boxplot{75}{0}{100}{100}{0}\\
    & WC&       25&  50&  75&   75& 100& \boxplot{25}{25}{75}{75}{25}\\\hline
 pf & CC&        50&  55&  63&   66& 82& \boxplot{50}{5}{63}{66}{16}\\
   & WC&        13&  18&  21&   29&  37& \boxplot{13}{5}{21}{29}{8}\\
\hline\end{tabular}

%~\\~\\
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc1$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       88& 100&  100&  100& 100& \boxplot{88}{12}{100}{100}{0}\\
    & WC&       38&  63&  63&   75& 88& \boxplot{38}{25}{63}{75}{13}\\\hline
 pf & CC&        89&  92&  93&   95& 99& \boxplot{89}{3}{93}{95}{4}\\
   & WC&        17&  25&  27&   30&  34& \boxplot{17}{8}{27}{30}{4}\\
\hline\end{tabular}

%~\\~\\
}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc2$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       67& 67&  100&  100& 100& \boxplot{67}{0}{100}{100}{0}\\
    & WC&       33&  67&  100&   100& 100& \boxplot{33}{34}{100}{100}{0}\\\hline
 pf & CC&        23&  24&  26&   29& 30& \boxplot{23}{1}{26}{29}{1}\\
   & WC&        11&  12&  14&   14&  18& \boxplot{11}{1}{14}{14}{4}\\
\hline\end{tabular}

%~\\~\\
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       69& 81&  88&  88& 100& \boxplot{69}{12}{88}{88}{12}\\
    & WC&       56&  75&  75&   81& 88& \boxplot{56}{19}{75}{81}{7}\\\hline
 pf & CC&        43&  45&  51&   53& 59& \boxplot{43}{2}{51}{53}{6}\\
   & WC&        20&  26&  28&   33&  35& \boxplot{20}{6}{28}{33}{2}\\
\hline\end{tabular}

%~\\~\\
}\\
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc4$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       83& 89&  94&  100& 100& \boxplot{83}{6}{94}{100}{0}\\
    & WC&       67&  78&  89&   94& 100& \boxplot{67}{11}{89}{94}{6}\\\hline
 pf & CC&        40&  45&  48&   51& 55& \boxplot{40}{5}{48}{51}{4}\\
   & WC&        22&  26&  27&   32&  34& \boxplot{22}{4}{27}{32}{2}\\
\hline\end{tabular}
}&
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($pc5$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
pd & CC&       69& 73&  75&  77& 83& \boxplot{69}{4}{75}{77}{6}\\
    & WC&       81&  85&  88&   90& 96& \boxplot{81}{4}{88}{90}{6}\\\hline
 pf & CC&        4&  5&  5&   5& 6& \boxplot{4}{1}{5}{5}{1}\\
   & WC&        11&  12&  12&   12&  14& \boxplot{11}{1}{12}{12}{2}\\
\hline\end{tabular}
}
\end{tabular}
\end{center}
\caption{Project-wise Experiment \#1 results for NASA tables.}\label{fig:ccwsprojs}
\end{figure*}
%\end{figure}



\subsection{Discussion of Experiment \#1}
When practitioners use defect predictors with high false alarm rates (e.g.the 52\% reported above),
they must allocate a large portion of their debugging budget to 
the unfruitful exploration of erroneous alarms.

In our industrial work, we have sometimes seen 
several situations
where detectors with high false alarms  are useful:\begin{itemize}\item{\em
When the cost of missing the target is prohibitively expensive.} In
mission critical or security  applications, the goal of
100\% detection
may be demanded in all situations, regardless of the  cost of chasing false alarms.\item
{\em When only a small  fraction
the data is returned.}
Hayes, Dekhtyar,
\& Sundaram call this fraction $selectivity$ and offer an extensive
discussion of the merits of this measure~\cite{hayes06}.  \item {\em
When there is little or no cost in checking false alarms.}  For
example,  a detector we have found useful in industrial settings
is to check modules  where \[\frac{lines\; of\; comments}{lines\;
of\; code} > 0.25\]This detector triggers on complex functions that
programmers comment extensively, instead of splitting up into
smaller, more maintainable, functions.
This detector suffers from high false alarms-
it often triggers on well-written functions with detailed comments.
On the other hand,
it may be fast
and simple for a human agent to inspect  the identified modules and
discern which
ones were well-written and which were over-commented to disguise
being
badly coded. \end{itemize}

Having said that, we observe that most sites do {\em not} accept
false alarm rates as high as 52\%. Therefore, the conditions under which the
benefits of CC data (high probabilities of detection) despite their high
costs (high false alarm rates) are quite rare.


In summary,
 for most software applications, very high  $pf$ rates like the CC results
of \fig{ccwsnum} make the predictors impractical to use.

\section{Experiment \#2: Incremental WC }
\begin{figure*}
%\begin{figure}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{log/incrementalcm1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalkc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalmw1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalpc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc4.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc5.pdf}
\end{tabular}
\end{center}
\caption{Results from experiment \#2. Training set size grows in units of 100 examples,
 moving left to right
over the x-axis. The MC2 results only appear at the maximum x-value since MC2 
has less than 200 examples.
}\label{fig:inc}
%\end{figure}
\end{figure*}

In the introduction, the case was made that CC data are attractive to
organizations if it avoids a time-consuming or expensive local
data collection program.  In this section,  we remove that case for CC data.
It will be shown that adequate defect predictors can be learned from very
small samples of WC data.

\subsection{Design}
A curious aspect of the above results is that defect predictors
were learned using only a handful of defective modules.
For example, consider a 90\%/10\% train/test split on
$pc1$ with 1,109 modules, only 6.94\% of which are defective.
On average, the training set will only contain \mbox{$1109 * 0.9 * 6.94/100=69$}
defective modules. Despite this, $pc1$ yields an adequate median $\{pd,pf\}$ results
of $\{88,34.5\}$\%. 

Experiment \#2 was therefore designed to check how {\em little} data are required
to learn defect predictors. Experiment \#2 was essentially the same
as the last experiment, but
without treatment \#1 (the
cross-company study). Instead,
experiment \#2
took the 12 example tables of \fig{data}
and 
learned predictors using:
\bi
\item Treatment 3 (reduced WC):  a randomly selected
subset of up to 90\% of the rows.
\ei
After randomizing the order of the rows, training sets
were built using just the first 100,200,300,\ldots rows in the tables.
After training, the learned theory was applied to the remaining rows
not used in training.

Experiment \#1 only used the features found in all tables of data.
For this experiment, we imposed no such restrictions and used whatever features
were available in each data set.
\subsection{Results from Experiment \#2}
Recall that
\eq{bal} defined ``balance'' to be a combination of $\{pd,pf\}$
that decreases if $pd$ decreases or $pf$ increases.
As shown in \fig{inc}, there was very little change in balanced 
performance  after learning from 100,200,300,... examples.
Indeed, there is some evidence that
learning from larger training sets had detrimental effects: the more
training data, the larger the variance in the performance of the learned predictor.
Observe how, in $pc2$, as the training set size increases (moving right along the x-axis)
the dots showing the balance performance start spreading out.
A similar, but smaller, {\em spread effect} can be seen in $kc2$ and $mc1$.

The Mann-Whitney U test was applied to check the visual trends seen in \fig{inc}.
For each table, all results from training sets of size 100,200,300\ldots were
compared to all other results from the same table. The issue was ``how much data are enough?''
 i.e.
what is the $min$imum
training set size that never lost to other training set of a larger size. Usually, that 
$min$ value 
was quite small:

\bi
\item In seven tables $\{cm1,kc2,kc3,mw1,pc3,pc4\}$, $min=100$;
\item In  \{$kc1,pc1$\}, $min=\{200,300\}$ instances, respectively.
\ei
In other tables of data, $min$ was much larger. In $\{pc2,mc1,pc5\}$ the 
$min$ values were found at $\{4900, 8300, 11000\}$, respectively.  However, much smaller
training set sizes performed nearly as well as these larger training sets:
\bi
\item
In $pc5$, predictors learned from 300 examples only lost to other sizes 
twice in 169 trials;
\item
In $mc1$, predictors learned from 400 examples only lost to other sizes
once out of 92 trials);
\item In $pc2$, predictors learned from 800 examples only
lost to other size twice out of 53 trials.
\ei

We explain the experiment \#2 results as follows.
These experiments used simplistic static code features such as lines of code, number of
unique symbols in the module, etc.  
Such simplistic static code features are hardly a complete characterization of
the internals of a function. Fenton offers an insightful example where
{\em the same} functionality is achieved using {\em different}
programming language constructs resulting in {\em different} static
measurements for that module~\cite{fenton97}.  
We would characterize such static code features as having 
{\em limited information content}. 
Limited content
is soon exhausted by repeated sampling.
Hence, such simple features reveal all they can reveal after
a small sample

\begin{figure}
\begin{center} 
\includegraphics[width=2.4in]{plateaus/plateaudots.pdf}
\includegraphics[width=2.4in]{plateaus/plateaudotsC.pdf} 
\end{center}
\caption{Y-axis shows plateau point after learning from data sets
that have up to $X$ examples (from~\cite{orrego04}).
The top plot shows results from using
Naive Bayes (nbk) or a decision tree learner (j48)~\cite{quinlan92}
to predict for discrete classes. Bottom plot shows results from
using linear regression (lsr) or model trees (m5)~\cite{quinlan92b}
to learn predictors for continuous classes. In this study, data
sets were drawn from the UC Irvine data repository~\cite{Blake+Merz:1998}.}\label{fig:plateau} 
\end{figure}

\subsection{Sanity Checks on Experiment \#2}
This section checks for precedents on the Experiment \#2 results
and can be skipped at first reading of this paper.

There is also some evidence that the  results of Experiment 2 (that
performance improvements stop after a few hundred examples) have
been seen previously in the data mining literature (caveat: to the
best of our knowledge, this is first report of this effect in the
defect prediction literature): \bi \item In their discussion on how
to best handle numeric features, Langley and John offers plots of
the accuracy of Naive Bayes classifiers after learning on 10,20,40,..200
examples. In those plots, there is little change in performance
after 100 instances~\cite{john95}.  \item Orrego~\cite{orrego04}
applied four data miners (including Naive Bayes) to  20 data sets
to find the {\em plateau point}: i.e. the point after which there
was little net change in the performance of the data miner.  To
find the plateau point, Oreggo used t-tests to compare the results
of learning from $Y$ or $Y+\Delta$ examples. If, in  a 10-way
cross-validation, there was no statistical difference between $Y$
and $Y+\Delta$, the plateau point was set to $Y$.  As shown in
\fig{plateau}, many of those plateaus where found at $Y\le100$ and
most were found at $Y\le200$. Note that these plateau sizes are
consistent with the results of Experiment 2.  \ei 

\subsection{Discussion of Experiment \#2}
In the majority case, predictors learned from as little as one hundred
examples perform as well as predictors learned from many more examples.
This suggests that the effort associated with learning defect predictors
from within-company data may not be overly large.
For example, \fig{estimate} estimates that the effort required to build and test
100 modules may be as little as 2.4 to 3.7 months.

\begin{figure}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
100  modules may take as little as two to four months to construct. This estimate
was generated as follows:
\bi
\item
In the $cm1$ data base, the median module size is 17 lines. 100 randomly
selected modules would 
therefore use 1700 LOC.
\item
To generate an effort estimate for these modules,
we used the on-line COCOMO~\cite{boehm00a} effort estimator (\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/expert_cocomo2000.html}). Estimates were generated
assuming 1700 LOC and
the required reliability varying from very low to very high.
\item
The resulting estimates ranged from
between 2.4 and 3.7 person months to build and test those modules.
\ei\\\hline
\end{tabular}
\caption{An estimate of the effort required to build and test 100 modules.}\label{fig:estimate}
\end{figure}

\input{turkey/external}
%\input{exp3}

%The 
%The results of the above procedure fall into the four groups of \fig{ccwc}.
%In the usual case ($\frac{8}{12}$),
%moving from within-company (WC) to cross-company (CC) data significantly
%increased {\em both}
%probabilities of detection $pd$ and false alarm $pf$.
%The size of $pd$ increase was quite remarkable. \fig{ccwcnums}
%
%The following chart shows that across our 12 data sets,
%the top quarter of the $pd$s in the inter-experiment were all in the maximum range of $pd=100\%$.
%As predicted by \fig{typical}, the price of this dramatic increase in $pd$ is an near doubling
%of the median $pf$: 
%
%~\\
%
%The increase in individual 
%data sets was even larger. For example, in $KC2$, the median $pf$ rose from 73 to 91\% while
%the median $pf$ rose from 31\% to 64\%:
%
%%\section{Group D- eight examples such as ``KC2''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  82 &  91 &  91 & 100 & 100 & \boxplot{82}{9}{91}{100}{0}\\
%pd / intra &  55 &  73 &  73 &  82 &  91 & \boxplot{55}{18}{73}{82}{9}\\
%pf / inter &  57 &  62 &  64 &  74 &  81 & \boxplot{57}{5}{64}{74}{7}\\
%pf / intra &  14 &  24 &  31 &  33 &  45 & \boxplot{14}{10}{31}{33}{12}\\
%\cline{2-7}\end{tabular}
%}
%
%~\\
%
%In one data set,  a completely opposite effect was observed to the main effect.
%In ``PC5'', the inter-experiment made the the $pd$ worse
%and the $pf$ better:
% 
%
%
%
%
%%\section{Group A- one example: ``PC5''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  69 &  73 &  75 &  77 &  83 & \boxplot{69}{4}{75}{77}{6}\\
%pd / intra &  81 &  85 &  88 &  90 &  96 & \boxplot{81}{4}{88}{90}{6}\\
%pf / inter &   4 &   5 &   5 &   5 &   6 & \boxplot{4}{1}{5}{5}{1}\\
%pf / intra &  11 &  12 &  12 &  12 &  14 & \boxplot{11}{1}{12}{12}{2}\\
%\cline{2-7}
%\end{tabular}
%}
%
%~\\
%
%XXX nothing in the data.

%\section{Open Issues}
%\noindent
%Future work in this area should:
%\bi
%\item
%Benchmark 
%automatic data mining.
%against 
%{\em manual case-based reasoning} methods 
%\item
%Check the stability of our conclusions between {\em random} and
%{\em natural} orderings of project data.:1
%\ei
%\noindent
%The rest of this section expands on these two points.
%
%\subsection{How complex is defect detection?}
%
%Assuming that the above results are externally valid, then it is possible
%that, for five years,
%we have needlessly over-elaborated a very simple problem. 
%Since 2002,  the authors have explored a diverse and complex range
%of data mining methods to learn defect 
%predictors
%(e.g.
%\cite{me02e,me03a,me04c,me07b,me07e,jiang07,gay07}).
%If a few dozen examples
%are enough for defect prediction, then 
%perhaps automatic data miners 
%may not the most
%an inappropriate technology for defect prediction.
%
%For small sample sizes,
%it is possible that 
%a human-in-the-loop case-based reasoning system could perform as well, or better,
%for finding combinations of features that predict for problematic code.
%For example, such an environment could ask
%humans to inspect a random sample of modules and comment on any ``bad smells''
%(to use Martin Fowler's terminology\footnote{For a list of ``bad smells'' see~\cite{fowler99} or
%)\url{http://wiki.java.net/bin/view/People/SmellsToRefactorings}.}).
%
%\subsection{Natural vs Random Orderings}
%
%One threat to external validity of our conclusion that ``a few dozen examples are enough''
% is that 
%it relies on very small randomly selected
%samples. In practice, such {\em random ordering} may not reflect the 
%{\em natural ordering} of 
%module creation and inspection in real-world projects.  
%The random sampling used above in Experiments \#1,2,3 select modules across
%all these different module types. 
%While our results hold for a few dozen modules selected at random across the system,
%they may not hold  
%for a particular order of module creation and inspection seen in real-world projects.
%
%For example, in a real project,
%developers may first generate modules relating to version control,
%then configuration management, then database accessors, then some high-level business 
%reports, and then (finally) the GUI. It is possible that the pattern and frequency
%of defects
%is very different in these different kinds of modules- in which case a few dozen
%examples of defective and non-defective modules may not be sufficient.
%Future research should check this possibility.
%
\section{Conclusion}
The value of cross-vs-within-company data (CC vs WC)
for effort estimation is an open question. Three of the papers
reviewed in the {\em Releated Work} section comment that it can be hard to use
CC data 
due to different data collection practices at different organizations~\cite{mendes07,kitch07,shepperd07}.

Our hypothesis was that the uncertainties seen in studies
of CC- vs WC-data may be due to ambiguity in the definitions of the effort
estimation features. Static code features, on the other hand, can be
collected in a uniform, rapid and automatic manner.
For example, in this study, we have found a clear and unambiguous conclusion amongst
our static code features:
\bi
\item CC-data dramatically increase the probability of detecting defective modules;
\item But CC-data also dramatically increase the false alarm rate. 
\ei
These CC results
are  disappointing result- one of the aspirations of software engineering
is the creation of libraries of background reference information from which
we can learn general principles for optimizing future applications.
While this may indeed be achievable in other aspects of software engineering,
for the specific task of detect prediction, our results show that such a goal
is highly problematic.



Another clear and unambiguous conclusion from static code features is that, for
the purposes of defect prediction,
the need for CC data may be less than previously believed.
Kitchenham et.al. list many reasons to use CC-data including these two points:

\bi
\item
 The time required to collect enough data on past projects from
   within a company may be prohibitive.
\item
Collecting within-company data may take so long
that 
technologies change and older
projects do not represent current practice.
\ei

Our incremental WC results suggest that, in the case of defect prediction,
these two points may not be compelling arguments. 
In most of our experiments, as little as 100 modules may be enough to learn
adequate defect predictors. 
When so few examples are enough,
it is possible that
projects can learn local defect predictors that are relevant to their
current technology in just a few months.


\bibliography{refs,usa/refs,turkey/refs}

%XXXX conclusion. Halstead and McCabe very odd 1970s. sureyin the 21st cenyru we can do better

\bibliographystyle{IEEEtran}


\newpage
\appendix
\section{Replies to Reviewer Comments}
\subsection{Replies to the Editor Comments}

Both reviewers suggest that the second experiment is more relevant,
possibly because the first experiment produced a negative result
indicating that cross-company (CC) prediction is not significantly
better than a random sample. Such negative result, while
interesting, is not particularly surprising given the lack of publications
involving CC defect predictions.

{\bf \bi
\item[1] This was a rhetorical error on our part. We completely agree with this
   comment.  What we should have said in the paper was that the conditions 
   under which CC prediction is useful is so extreme that we just should not 
   do it. Therefore the discussion section of section VI.d is now much more
   negative.
\ei
}
Even more significantly, both reviewers question the CC nature of the
work. The basic reason CC models
tend to fail is that companies work in different domains, have
different process, and define/measure defects and other aspects of
their product and process in different ways. 

{\bf \bi
\item[2] Excellent point- with your permission, we would use the above
   ideas in the new introduction (paragraph 2, section I).
\ei}

 Given that projects
came from NASA, it is not clear to what extent the domain, process,
and measurement varied among the projects throwing doubt on the
premise that the experiment was performed on CC data.  

{\bf \bi
\item[3] Because of your reviews, we searched for more data. We found some-
   three new data sets, in fact. This new data comes not from NASA 
   aerospace applications but from a Turkish company that writes 
   software for domestic home appliances. Nervously, we repeated our
   experiments on this data and we are happy to report that the
   the WC vs CC patterns we found in in American NASA rockets
   also hold for Turkish washing machines (this experiment is reported
in a new section VI of this paper). While this is not 
   definitive proof of the universal external validity of our
   results, we find it a very compelling result suggesting that the
   experiments of this paper apply to a very wide range of software
   from around the world.
\ei}

The other major concern was the lack of comparison to a very rich
body of research in defect prediction area. Some of the more
effective prediction techniques rely on project history\footnote{
 T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy. Predicting fault
incidence using software change history. IEEE Transactions on
Software Engineering, 26(2), 2000.
}.
Clearly, projects with nontrivial history are less likely to
benefit from CC models, while contractor relationships in NASA case
may not be obviously subject to historic models without having
several projects for a contractor. Therefore, the lack of value of
CC techniques in a case where they are arguably the most likely to
succeed is a concern. Lack of comparisons to a context where
the defect prediction does appear to add value is disappointing.

{\bf \bi
\item[4] We added extensive notes on other work (see section II.B)- including
   a note on paper Graves, Karr, Marron \& Siy- in section IIb.
\item[5] And we have added notes on your above paragraph to the conclusion,
and the end of
    section IV. See also, point [10] (below).
\ei}
 
\subsection{Replies to Reviewer 1}

 MAJOR ISSUES:
I'm not convinced of the usefulness of the results of either
experiment. Given that most projects have more than 90\% defect-free
modules, a sizeable number of false positives will need to be tested.
From experiment, the pf median is at 52% for CC which means over
45\% of modules with no defects (assuming 90% defect-free) will need
to be tested. This seems to be no better than picking a module at
random. The assertion that this may be applicable for mission-critical
systems is doubtful because organizations building mission-critical
systems are: (a) very likely following different and highly specialized
development processes, (b) developing more complex systems, and (c)
not likely going to base their QA decisions on data from untrusted
sources. It is also likely that they will retest everything anyway.
Perhaps it might help if the datasets were stratified so that
"mission-critical" projects are separated from those that aren't.
If CC only contained the mission-critical projects, and the results
are still similar, this may at least dispel objections (a) and (b).

{\bf \bi
\item[6] The reviewer is kind to offer a solution to problems (a) and (b) but
   we prefer a more radical fix.  See comment [1] above.
\ei	}

For Experiment \#2, eye-balling Figure 5, the average balance across
all projects appears to be around 0.7. Based on the balance equation,
the pd would be less than 60\% if pf is less than 10\% and less than
65\% if pf is less than 25\%. pf goes over 40\% as pd approaches 100\%.
To me, these results do not seem like useful predictors. How does
this compare to published results using other techniques?

{\bf \bi
\item[7] That baseline data has been added to section II. In short,
    we are in the ball park of other machine learning methods and better
    than current industrial practices (but see comment [19] below).
	Also, and the following text has been added to the paper, in
	our discussions with our new data source, they commented that operate
	in a highly competitive market segment where profit margins are
	very tight. Therefore reducing the cost of the product even by 1
	cent makes a huge difference both in market share and profits. Their
	applications are embedded applications where the software component
	has taken precedence over the hardware in the last decade. Hence
	their problem is a software engineering problem. Confirming the
	world statistics in software development their main challange is
	the testing phase. They would like to have predictors to tell them
	where the defects are before the testing starts so that they could
	efficiently use their scarce resources. Moreover they also need to
	answer the question of how much to test? and what is the trade-off
	of detecting more at the cost of testing more?. Therefore our
	research and its results are quite relevant for them. As a matter
	of fact we have started using our predictor in their development
	environment and got excellent results. Now they are in the process
	of embedding defect prediction into their software development
	process.
\ei}

The result of experiment \#2 where 100 data points in the training
set seem to be sufficient most of the time is very interesting.
However, for this to be a practical guideline, external validity
should be addressed. In addition to the cited references [17,21],
does this apply to other datasets as well? How about defect data
from open source projects? Alternately, is there a theory to explain
this?

{\bf \bi
\item[8] We are very happy to report that this result holds for data
     that is {\em radically} different to NASA aerospace applications. See
	 comment [3] (above) and the new external validity section VI.
\ei}

There is no section discussing previous related work. As a study
 on predicting defect-prone modules, the authors miss a large body
of existing work on defect prediction, especially prediction based
 on history data.

{\bf \bi
\item[9] We agree. We have added extensive notes on related work (see Section IIb and 
comment [4] above).
\ei}

Does not address some of the common questions with cross-company
data. For example: - Is the development process homogeneous? Do
they have similar QA processes? Are the data collection procedures
similar?  - Is there a uniform definition of "defect"? If there
is, does it include inspection defects? Is it only post-release
defects? Are these defects based on problem reports?  - And if
there were non-uniform processes or definitions, what steps were
taken to standardize the metrics?  These should be discussed as
part of threats to validity.

{\bf \bi
\item[10] One challenge with data mining on real world data sets is
     that much of the contextual knowledge is unavailable. All we
     can usually get is the source code. Other important issues
     relating to software process, measurement control, etc may not
     be available. It is certainly true that our predictors
     would be better if we added this information to our data sets.
     However, the absence of that information from our data is
     not a fatal flaw in this approach.
     We note that even without that data, we can generate useful
     predictors (and note that the new section IIb and section IVd
     now has extensive notes on the meaning of "useful").  
\ei}

Other comments:

Missing keywords.

{\bf \bi
\item[11] Added
\ei}

Page 2 (also in Abstract): no explanation was given why the authors
decided to do defect detection rather than effort estimation. After
discussing at length about effort estimation in the opening paragraphs
of Section I, suddenly, the authors presented their work on defect
detection. Perhaps the discussion on effort estimation can be
relegated to a "related works" section. Alternately, discuss how
static code-based predictors can be used to estimate effort.

{\bf \bi
\item[12]
This was a rhetorical error on our part and it is corrected
     in this draft. Firstly, the abstract has been shortened.
     Secondly, as stated in the introduction, elsewhere we have
     explored CC data for effort estimation so this paper turns
     its attention to other areas (defect prediction). Thirdly,
     most of our notes on effort estimation have been isolated in a
     clearly labeled separate section (See IIA).
\ei}

Page 4, paragraph 3, 1st bullet: there is no reference associated with Raffo.

{\bf \bi
\item[13] Added
\ei}

Page 5, many of the features listed in the Appendix (Figure 8) are highly correlated. 
 Discuss how this affects the independence assumptions of Bayesian classification.

{\bf \bi
\item[14] This is very very interesting point and we added notes in the paper to the
     fascinating 1995 results of Pazzani and Domingos (see column II, page 5). 
     They compared that the 
     independence assumption of naive  bayes assumption to a theoretical "better
     bayes" classifier that knew when to delete correlated attributes. "Better bayes"
     turns out to be actually better in only a vanishingly small number of cases. This
	 is a truly impressive paper and we highly recommend it to our graduate students
 	 as a shining example of just how good a data mining paper can be\footnote{
	P. Domingos and M. J. Pazzani, "On the optimality of the simple
	Bayesian classifier under zero-one loss," Machine Learning, vol. 29,
	no. 2-3, pp. 103-130, 1997. [Online]. Available: 
    citeseer.ist.psu.edu/domingos97optimality.html}.
\ei	}

Page 7, Figure 2: in addition to the summary in Fig. 2, it may be helpful to list 
 the average pd and pf values (for both CC and WC) for each project.

{\bf \bi
\item[15] added
\ei}

Typo:
Page 6, Section III.B, sentence after the bullet list: there are two "than"s.

{\bf \bi
\item[16] fixed
\ei}


\subsection{Replies to Reviewer 2}

I would like to see an extension of the research to projects outside
the NASA realm.  Despite the fact that the data presented are from
different NASA organizations, this reader still has the feeling
that the projects are at least related, or similar types of
applications, and perhaps the development teams all share a particular
"NASA-type" development mindset.
I don't require this extension to accept the paper, but it would
undoubtedly make the research more valuable and more widely applicable,
and would certainly strengthen the paper.

{\bf \bi
\item[17] We found new data. See comment [3] above
\ei}

The Appendix is unnecessary, and I suggest moving the table of
static code features into the body of the paper, at page 4, around
line 12.
The text there says "tables of examples are formed(like those in
Fig 1)".  But Fig 1 is only a summary of the systems, while the
table referred to is actually in the Appendix.
The reader would like to know right then what the features are,
without having to look for the Appendix.

{\bf \bi
\item[18] Done
\ei}

I don't believe it's meaningful to compare the defect predictors
in this paper to the detection results by Raffo or the IEEE Metrics
panel (page 4, ll. 21-42) Those results concern defect finding by
inspections, and by their nature they detect not just potential
faulty modules, but they also provide specific debugging information,
i.e, they find the specific flaws in the code.

{\bf \bi
\item[19] 
     We agree with the reviewer that inspections can find errors AND offer advice
     on how to fix them. Our static code detectors, on the other hand can only
     point to problems, not explain them or offer methods to fix them 
	 (actually, there are other testing regimes that are also silent on
     how to fix errors; e.g.  a unit test can report that the output of a 
     program is not what was
     expected, but it can't explain why the differences exist nor where
     to fix them). But here are the two issues that need to be separated:
\bi
\item
      there is the cost of running a oracle that can indicate problematic code areas
	 \item there is the cost of fixing those problems.
\ei
     We {\em can} compare our results with raffo/metrics02 on issue (a). But,
	 as the reviewer correctly observes, issue (b) remains.  
     Footnote \#7 (p4) explores this issue. Our answer is incomplete in
     that it requires further data collection. Given the currently available
     data, all we can report on at this time is issue (a).
\ei}

The discussion in the middle of page 5 (Bayes theorem) needs to be
expanded and better explained, and related to the paper's datasets.
In particular, explain the equation on line 35, and explain what E and E-sub-i represent for your data.  

{\bf \bi
\item[20] Done
\ei}

It would also be helpful to mention the specific non-Bayesian methods
that you investigated in [6] and found did not perform as well.

{\bf \bi
\item[21] Added
\ei}

SYNTACTIC ISSUES:
Specific places to make the text clearer, or remove typos.  Words in single quotes '  ' should be inserted. 
Words in angles should be removed. 
(The line numbers refer to the numbers printed in the Manuscript-central printout, not to the actual 
 count of lines on each page. )

{\bf \bi
\item[22] done
\ei}
\end{document}
 
\section{Accessing Data Mining Tools and Data}

The  goals of this paper were achieved due to the large number of tools and data sets readily
available for free download from the Internet.
Anyone with access to Google can easily find and download
numerous robust data mining toolkits supports hundreds of
combinations of learners, data pre-processors, feature subset
selectors, etc\footnote{
e.g. the R project\footnote{\url{http://www.r-project.org/}},
Orange\footnote{\url{http://www.ailab.si/orange/}},
RapidMiner\footnote{\url{http://rapid-i.com/content/view/10/32/lang,en/}}
and the WEKA~\cite{witten05}.}
Also, there is more data to process. 
For example, the PROMISE repository of SE
data\footnote{\url{http://promisedata.org/repository}} has only
been active for 3 years but already contains 44 data sets. That repository
is growing rapidly (on average, 37\% more data sets each year).
This  growing supply of on-line data sources is due to several recent
developments. 
Organizational inertia means that
once procedures are
in place to:
\bi
\item Build and maintain an artifacts repository;
\item
 Build a sanitizer that
can ``scrub'' data of corporate identifies prior to release;
\item
Resolve any remaining legal issues;
\ei
then once one data set is generated,
many more will follow.  (one for each new product and for each
new release of each product). This has the added benefit that data
collected from multiple projects may have the same same schema.
For example:
\bi
\item The open
source community has proven to be a rich source of information.
PROMISE holds six ECLIPSE data sets (file and package- level metrics
for ECLIPSE 2.0, 2.1, 3.0) as well as data from FreeBSD, Xfree,
Xorg, and other projects.  
\item NASA's Metrics Data Program is now defunct
but that program generated twelve of the data sets used in this
study.
\ei
The increasing  availability of cross-company data 
makes the Kitchenham et.al. research question a very pressing question.

One interesting phenomenon   observed in the PROMISE repository is the 
is the {\em flood} effect: by the time an organization 
The Kitchenham et.al. study was a 
{\em systematic review} that drew its conclusions from a wide-ranging literature search.
Such systematic reviews are becoming more frequent and are an example of the {\em evidence-based
software engineering} movement.


Software managers can use predictor of defective modules to 
focus the testing on parts of the system that seem defect-prone.
These potential defect-prone trouble spots can then
be examined in more detail by, say, model checking, intensive testing, etc.
Previously we have explored the use of data miners to learn such predictors from static
code features. We found that a large class of predictors proposed in numerous
publications were out-performed by Bayes classifiers that combined partial evidence
from multiple features~\cite{me07d}. 

Kitchenham et.al. would characterize that previous work as a {\em within-company} (WC) study
where the data from one organization is applied to new examples within the same organization.
%XXX much more regularity in software engineering that previously suspected

Therefore, companies are tempted to forge WC studies, preferring instead to use some
cross-company (CC) study where data from elsewhere is applied locally.
The value of cross- vs within-company studies for effort estimation has been the subject
of much recent work.
In the special case of effort prediction from static code features, some of those quality controls can be automated.
In the following cross- vs within- study, features of code 
modules were extracted automatically using the same tool set (the McCabe HiQ tool
 that is
widely used in industry.
We find that

\section{Data}

\begin{figure}
\begin{center}
\begin{tabular}{l|rrr}
data & examples & features  & defective\\\hline
pc5  & 17,186   & 38          & 3.0\%\\
mc1  & 9,466    & 38          & 0.71\%\\
pc2  & 5,589    & 36          & 0.41\% \\
pc3  & 1,563    & 37          & 10.23\%\\
pc4  & 1,458    & 37          & 12.2\%\\
pc1  & 1,109    & 21          & 6.94\%\\
kc1  & 845      & 21          & 15.45\%\\
kc2  & 522      & 21          & 20.49\%\\
cm1  & 498      & 21          & 9.83\%\\
kc3  & 458      & 39          & 9.38\%\\
mw1  & 403      & 37          & 7.69\%\\
mc2  & 161      & 39          & 32.29\%
\end{tabular}
\end{center}
\caption{Twelve data sets, sorted in order of \#examples.}\label{fig:data}
\end{figure} 
\subsection{Predictor Mathematics}

Let $\{A,B,C,D\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary predictor. 
Certain standard measures can be computed from
$A,B,C,D$: 
\[
\begin{array}{ll}
pd=recall=&\frac{D}{B+D}\\
pf=&\frac{C}{A+C}\\
prec=precision=&\frac{D}{D+C}\\
acc=accuracy=&\frac{A+D}{A+B+C+D}\\
selectivity=&\frac{C+D}{A+B+C+D}\\
neg/pos=&\frac{A+C}{B+D}
\end{array}
\]
The last measure ($neg/pos$) is quite interesting and
can be used within the above equations as follows:
\begin{equation}\label{eq:one}
prec = \frac{D}{D+C} = \frac{1}{1+ \frac{C}{D}} = \frac{1}{1 + neg/pos \cdot pf/recall}
\end{equation}

Ideally, our predictors have high precision {\em and} high recall {\em and } low false alarm
rates. This ideal situation is very hard to achieve. In practice, predictors must trade-off one goal
for the other.

\subsection{Trade-offs with Recall (pd) and False Alarm Rates (pd)}
Predictor performance can be  assessed via {\em
receiver-operator} (ROC) curves.
Formally,  a defect {\em predictor}   hunts for a {\em signal} that a
software module is defect prone. Signal detection
theory~\cite{heeger98} offers 
ROC curves as an analysis method for assessing different
predictors.
%% ROC curves are widely used in various fields including
%% assessing different clinical computing systems~\cite{adlassnig89} and
%% assessing different machine learning methods~\cite{provost98case}.
%% The central intuition of ROC curves is that different detectors can
%% be assessed via how often they correctly or incorrectly respond to
%% the a signal.
A typical ROC curve is shown in \fig{typical}. The y-axis shows
probability of detection ($pd$) and the x-axis shows probability of false alarms ($pf$).
By definition,
the ROC
curve must pass through the points $pf=pd=0$ and $pf=pd=1$
(a predictor that
never triggers never makes false alarms; a predictor that always
triggers always generates false alarms).
Three interesting trajectories connect these points:
\be
\item
 A straight line from
(0,0) to (1,1) is of little interest since it offers {\em no
information}: i.e.  the probability of
a  predictor firing is the same as it being silent.
\item
Another trajectory is the {\em negative curve} that bends away
from the ideal point. Elsewhere~\cite{me02e}, we have found that
if predictors negate their tests, the negative curve will transpose into a {\em
preferred
curve}.
\item
The point ($pf=0$, $pd=1$) is the ideal position (a.k.a. ``sweet spot'')
on a ROC curve. This is
where we recognize all errors and never make mistakes. {\em Preferred curves} bend
up towards this ideal point.
\ee

In the ideal case, a predictor has a high probability of detecting a
genuine fault ($pd$) and a very low probability of false alarm
($pf$). This ideal case is very rare. The only way to achieve high probabilities
of detection is to trigger the predictor more often. This, in turn, incurs the cost
of more false alarms.

In practice, engineers {\em balance}
between $pf$ and $pd$. To operationalize this notion of {\em
balance}, we define $bal$ to be the Euclidean distance from the sweet
spot $pf=0,pd=1$ to a pair of $<pf,pd>$. For convenience, we
(a)~normalize $bal$ by the maximum possible distance across the ROC square
( $\sqrt{2}$); (b)~subtract this from 1; and (c)~express it as a percentage; i.e.
\begin{equation}\label{eq:bal}
balance= bal             = 1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{equation}
Hence, better and {\em higher} balances fall  {\em closer} to the desired  sweet spot  of $pf=0,pd=1$.
\subsection{Trade-offs with Precision}

Note that \eq{one} can be re-expressed as follows:
\begin{equation}\label{two}
pf = \frac{pos}{neg}\cdot\frac{ (1 -  prec) }{ prec}\cdot recall
\end{equation}
That is, in Equation~\ref{two}, when $recall$ is fixed then
false alarm rate becomes controlled by precision and a 
fixed constant determined by the data set being examined; i.e.
when ($\alpha=neg/pos$) and \mbox{$recall=1$}
then:
\begin{equation}\label{three}
pf = \alpha\cdot \frac{1-prec}{prec}
\end{equation}
From Equation~\ref{three}, it is clear that it is problematic to demand predictors
with high recall {\em and} high precision {\em and} low false alarm rates.
For example, for any targeted recall value,
{\em increasing} 
precision requires
{\em decreasing}
false alarm rates; e.g. for \mbox{$prec\in\{0.5,0.70,0.9,0.95\}$}, $pf$ becomes $\{1,0.43, 0.11, 0.005\}$,
respectively. The effect is particularly marked for data sets
with large $neg/pos$ ratios.

Many software engineering data sets have extremely
large
$neg/pos$ ratios.
%Elsewhere~\cite{me03a,me03q}\footnote{See also the PREDICTIVE toolkit~  \url{http://www.ismwv.com/}},
%we have conducted extensive studies
%on methods for  generating and selecting detectors appropriate to different business contexts.
%The Zhangs' equation enables a more nuanced  selection process.
\begin{figure}[!b]
\begin{center}
\includegraphics[width=2.7in]{locplatonic.pdf}
\end{center}
\caption{Regions of a typical ROC curve.}\label{fig:typical}
\end{figure}
For example:
\bi
\item In our previous DMP study~\cite{me07b}, the data sets studied
had $neg/pos$ ratios
of
1.04, 
7.33,
9,
10.11,
13.29,
15.67, and
249.
\item
  Hayes, Dekhtyar and Sundaram \cite{hayes06} use text mining to                                                                     
find pairs of connected requirements in a corpus of 220 requirements and                                                                     
235 design elements (the same CM-1 dataset used in 
\cite{me07b}).
The total number of possible links in the dataset is $220\cdot 235 =                                                                        
51,700$, while the the ground truth RTM contains 361 links, for the                                                                         
$neg/pos$ ratio of $51,700/361 = 143.2$.   
\item
For an extreme example, Google reports that
over $10^{9}$ web pages contain the phrase ``software''
but only one them is the home page of this journal. Hence, $neg/pos$ for
web searching is at least $10^9$. 
\ei
%For example, consider the  the mission critical situation requiring $recall=1$.
%Equation~\ref{two}
%shows that when $prec=0.25$ and
%$neg=500$ and $pos=100$, achieving the goal of $recall=1$ requires
%a detector with  $pf=0.6$.
%As precision increases, detectors become harder and harder to
%design. For example, continuing the security example,
%at $recall=1$ and $neg/pos=5$,
%increasing the precision 
%0.25 to
%0.5 to 0.75 requires detectors with
%false alarm rates of 0.6 to  0.2 to 0.067 (respectively).
%As discussed below, building high precision detectors
%can be expensive.
%Depending on the available research budget, companies
%might elect to use lower precision detectors that are cheaper to build.
%
%experiments - some minor changes in formulas may affect the exact numbers,
%but not the ball parks), our pf for CM-1 dataset, without filtering,
%feeback, or any other tricks, is probably around 60-65% (we retrieve
%around 33,000 links outright, out of 51700 possible). Only with filtering
%(either by absolute value, or by relative rank) can we get to pf of 0.1 or
%better with decent recall.
%MGF report probability of detection and false alarm rates of
%of \{pd=75\%, pf=25\%\}, respectively. MGF argue that these are high enough to be useful.
%However,  Zhang\&Zhang derive an expression that can compute precision from the data found
%in MGF. Those computed  precisions suggest that the MGF results are  quite useless:
%\[
%\{min,median,max\}=\{2\% 20\%, 70\%\}
%\]
%
%Here, we show that Zhang\&Zhang
%do not take their conclusions far enough. 
%A large number of current industrial problems can be characterized by ``small targets'' and
%``some false alarms''  (quoted terms are defined below).
%Using the expression derived by Zhang\&Zhang, we show that all such domains are prone to
%the problem of low precision.
%However, unlike Zhang\&Zhang, we will argue that low precisions are acceptable in some
%of those domains.
%Also, unlike Zhang\&Zhang, we also will offer a specific research direction 
%on how to improve the MGF results. Interestingly, that research direction
%{\em requires} accepting low precision detectors, at least initially.

Figure~\ref{pprnp}
graphs \eq{one} for the  DMP $neg/pos$ ratios.
\begin{figure}
\begin{center}
\includegraphics[width=3in]{y_0.pdf}
\end{center}
\caption{The relationship between
$pd, prec, recall$ and $neg/pos$. 
}\label{pprnp}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=3in]{y_05.pdf}
\end{center}
\caption{
Figure~\ref{pprnp},
cropped to the region
where $prec>0.5$.
}\label{pprnp1}
\end{figure}
Figure~\ref{pprnp1} does the same, 
but is restricted to zones of higher precision:
only the surface for
$0.5\le prec \le 1$ is shown. 
That shadow of the  surface on the bottom plane
shows that this zone of high precision, high recall, and large
$neg/pos$.
As $neg/pos$ increases, 
high recall\&precision
is only possible when $pf$ becomes vanishingly small.
For example, in the ranges 
\mbox{$0.65 \le prec,recall \le 0.8$}, 
Equation~\ref{two} reports that $pf$ falls into the following ranges:
\begin{itemize}
\item  $0.023 \le pf \le 0.062$  for $neg/pos = 7$;
\item  $0.0108 \le pf \le 0.0287$ for $neg/pos = 15$;
\item  $0.007 \le pf \le  0.0017$ for $neg/pos = 250$;
\end{itemize}
Predictors learned in the domain of
software engineering
rarely yield
high precision predictors (see Figure~\ref{precs}).
Using the above  equations, the reasons for this are very clear:
\bi
\item
Those predictors all try to maximize recall; 
\item
Figure~\ref{pprnp} shows that such 
predictors can only achieve high precision in the rare case of very low $pf$.
\ei

Not only do our   equation explain the Figure~\ref{precs} results,
they also inform the 
 the instability of precision {\em and} the stability of $pf$ and $pd$ (recall) 
seen 
in previous studies~\cite{me04c}"
\bi
\item
Note how, in Figure~\ref{pprnp},
that at very small $pf$ values,
tiny changes in $pf$ can lead to very large changes in $prec$ 
(sudden jumps from zero to one).
\item
The other measures in
Figure~\ref{pprnp}, on the other hand, change far more smoothly and slowly.
\ei
That is, the mathematics of these predictors
explains numerous prior results such as those shown in Figure~\ref{precs} 
(i.e. \cite{me03q,me07b,huan06,hayes06,antoniol02,antoniol05,marcus03}).
\begin{figure}[!b]
\small
\begin{center}
\begin{tabular}{|p{0.95\linewidth}|}\hline
\bi
\item
The DMP paper achieved recalls over 70\% and minimum false alarm
rates of 15\%.  Using our equation and data from DMP, it
can be shown that the DMP precisions were quite low:
$\{min,median,max\}=\{2\% 20\%, 70\%\}$ (the last number came from
a data set with $neg/pos\approx 1$).
\item
Huang et.al.~\cite{huan06} won ``best paper'' at the 2006 IEEE
Requirements Engineering conference with predictors exhibiting
$prec\approx 0.25$.
\item
Without extensive feedback from human experts, the classifiers used
by Hayes, Dekhtyar, and Sundaram exhibited $pf>0.6$ (hence, very
low precisions)~\cite{hayes06}.
\item
Other such as Antoniol et.al.~\cite{antoniol02,antoniol05} and
Marcus et.al.~\cite{marcus03} researchers into software traceability
report that high recall is achievable only with low precision
predictors.  Antoniol has also been exploring bug traces in Mozilla
components and has found the same high-precision, low-recall
trade-off.
\item
After much experiments with linear regression, model tree learners,
Bayes classifiers, decision tree learners, singleton rule learners,
and some home-brew learners~\cite{me03q}, the general trend is very
clear. For those learners, executing on the DMP datasets, $pf \le
pd - 0.5$; That is, for those learners and those data sets, obtaining
high recall figures of $pd>0.6$ implies $pf>0.1$ and, consequently,
low precision.
\ei  
\\\hline \end{tabular} \end{center} 
\caption{Some
low precisions seen in the SE literature.}\label{precs} 
\end{figure}

\subsubsection{When Low Precision is Useful}

Achieving both high precision\& recall \& low false alarm rates can be problematic.
Optimizing for one often compromises the other (especially for data sets with large $neg/pos$ ratios).
Fortunately, there are may industrial
situations 
where low precision and high recall
predictors are useful.
For example, one of us (DiStefano) has used
our low-precision predictors to review flight code
developed at
the NASA Glenn Research Center (Ohio, USA).
When the results of these predictors were presented to the lead flight
engineer, he confirmed that the identified sections (which did not
have any recorded defects) had been problematic to maintain, and
contained several bugs which had not yet been entered into the
defect system. 


For another example,
from
outside the field of SE, 
a user of commercial
web search engine like Google can quickly flick through (say) three
pages of results before finding a page of interest.  Google has so
many return customers since even with precisions of (say) $\frac{1}{30}$,
the effort involved in looking at a page is so low that users don't
mind examining 29 false alarms.  

More generally, there are several situations where low precision
predictors are useful:
\begin{itemize}
\item
{\em When the cost of missing the target is prohibitively expensive.}
In mission critical or security  applications, the goal of
100\% recall may be
demanded in all situations, regardless of the  precision.
\item {\em When only a small  fraction
the data
is returned.}
Hayes, Dekhtyar, \& Sundaram call this fraction
$selectivity$ and offer an extensive discussion of the merits of
this measure~\cite{hayes06}.  \item {\em When there is little or no cost in checking
false alarms.}  For example,  a predictor we have found useful in industrial settings is to
check modules  where \[\frac{lines\; of\; comments}{lines\; of\; code} > 0.25\]
This detector triggers on complex functions that 
programmers comment extensively, instead of splitting up into smaller, more maintainable,
functions.
This detector is imprecise- it often triggers on well-written
functions with detailed comments. 
However, based on commercial experience, we assert
that it is fast and simple for
a human agent to inspect  the identified modules and discern which                                                                    
ones were well-written and which were over-commented to disguise being                                                                      
badly coded.
We use this detector to find code that should be rewritten prior to release.
\end{itemize} 

\section{Data}

This section discusses the data used in this study.
Each module of each data
sets describes the features of that module,
plus the number of defects known for that module.
This data comes from
eight
sub-systems taken from four systems.  These systems were developed
 in
different geographical locations across North America.
Within a system, the sub-systems
shared some a common code base but did not pass personnel or code
between sub-systems. 

Each data set was pre-processed by removing the module identifier feature
(which  is different for each row). Also, the
$error\_count$ column was converted into a boolean feature called {\em defective?}  as follows:
$defective? = (error\_count \ge 1)$.
Finally, the $error\_density$ column was removed (since it can be
derived from line counts and $error\_count$).
The pre-processed data
sets had 38 features, plus one target feature
({\em defective?})
shown in \fig{attr} and include Halstead, McCabe, lines of code, and other miscellaneous
features.
% XXX need introduction words saying contribution is definition of baseline experiment
At the end of \fig{attr} are a set of {\em misc} features that are less well-defined than lines of code features
or the Halstead and McCabe features. 
The meaning of these features is 
   poorly documented in the MDP database. Indeed,
   they seem to be values generated from some unknown tool set that
   was available at the time of uploading the data into the MDP.
Since there are difficulties in reproducing these features at other sites, an argument could be made
for removing them from this study.
A counter-argument is that if static code features are as weak as suggested by Shepherd, Ince, Fenton and Pfleeger
then we should use all possible features in order to make maximum use of the available information.
This study took a middle ground: all these features were passed to the learners and they determined
which ones had most information.

%Two different categories of re-sampling methods commonly used to overcome the 
%problem of imbalanced data.  The first increases the frequency of the minority class in 
%the training set (Oversampling) and the other decreases the frequency of the majority 
%class in the training set (Undersampling) (Weiss and Provost 2001; Drummond and 
%Holte 2003).
%Drummond, C. and R. C. Holte (2003). C4.5, class imbalance, and cost sensitivity: why 
%under-sampling beats over-sampling. Workshop on Learning from Imbalanced 
%Datasets II. 

%In this paper, we used cost curves to explore the in- 
%teraction of over and under-sampling on the decision 
%tree learner C4.5. Surprisingly, over-sampling is in- 
%effective, at least when the default settings are used, 
%often producing little or no change in performance. 
%Under-sampling does produce a reasonable sensitivity 
%to changes in misclassification costs and class distri- 
%bution. But representing these changes internally by 
%down-weighting gives the best performance overall. 

%~\cite{yun04}

An interesting repeated pattern in our data sets are {\em exponential distributions}
in the numeric features.
Elsewhere, we have conducted limited experiments suggesting that a
{\em logarithmic filter} on all numeric values might improve predictor
performance~\cite{me02e}.  
Such a filter replaces all numerics $n$ with their logarithm.
$ln(n)$.
The
log-filtered values are now more evenly spread across the y-range, making it easier
to reason about them. 
To test the value of log-filtering, all the data was passed through one of two filters:
Hence, all our numerics were pre-processed as follows:
\bi
\item {\em logNums}; i.e. logarithmic filtering where all numbers $n$ where replaced
with $log(n)$.
\item
To avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$.
\ei

\section{Non-parametric Analysis of Results}

\subsection{Quartile Charts of Performance Deltas}


The experiment described below explored numerous combinations of
filters, features, and learners all within a
$(M=20)*(N=10)$-way cross-evaluation 
study. Hence,
this experiment 
generated nearly thousands of measuring
$pd,pf,bal$
values from different {\em methods} applied to the
same $test$ data.

The performance
results for each method were sorted and displayed as {\em
quartile charts}.  To generate these charts, the performance
deltas for some $method$ 
were sorted to isolate the median and the lower and upper quartile of numbers.

\[
{\footnotesize
\overbrace{4,7,15}^{q1},20,31,\underbrace{40}_{median},52, 64, \overbrace{70,81,90}^{q4}}
\]
In a quartile chart,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
a vertical bar  is added to
mark the 50\%  value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
We prefer quartile charts of performance deltas to other summarization
methods for M*N studies. Firstly, they offer a very succinct summary
of a large number of experiments. For example, in the next few pages
we will display thousands of results in very little area.
Secondly, they
are a {\em non-parametric} display; i.e. they make no assumptions
about the underling distribution.  Standard practice in data mining is
to compare the mean performance of different methods using
t-test~\cite{witten05}. T-tests are a {\em parametric method} that
assume that the underling population distribution is a Gaussian.
Recent results suggest that there are many statistical issues left to
explore regarding how to best to apply those t-tests for summarizing
M*N-way studies~\cite{brouck03,demsar06}.  
\subsection{Comparison Statistics}

The problem of comparatively assessing $L$ learners run on multiple sub-samples of $D$ data sets
has been extensively studied in the data mining community. 
T-tests that assume Gaussian distributions are strongly deprecated.
For example, Demsar~\cite{demsar06}
 argues that non-Gaussian populations are common enough to require a methodological
change in data mining. 

After reviewing a wide range of
comparisons methods\footnote{Paired t-tests with and without the use of geometric means of the relative ratios;
binomial tests with the Bonferroni correction;
paired t-tests; ANOVA; Wilcoxon; Friedman},
Demsar advocates the use of the 1945 Wilcoxon~\cite{wilcoxon45} 
signed-rank test that compares the ranks for the positive and negative differences (ignoring the signs). 
Writing five years early,
Kitchenham et.al.~\cite{kitc01} comment that the Wilcoxon test has its limitations.
Demsar's report  offers the same conclusion, 
noting that the Wilcoxon test requires that the sample sizes are the same.
To fix this problem, Demsar augments Wilcoxon with the Friedman test.

One test not studied by Demsar is Mann and Whitney's 1947
modification~\cite{mann47} to Wilcoxon rank-sum test (proposed along with his signed-rank test).
We prefer this test since:
\bi
\item
The Mann-Whitney U-test does not require that the sample sizes are the same.
So, in a single U-test, learner $L_1$ can be compared to all its rivals. 
\item
The U-test does not require any post-processing
(such as the Friedman test) to conclude if the median rank of one population (say, the $L_1$ results)
is greater than, equal to, or less than the median rank of another (say, the $L_2,L_3,..,L_x$ results).
\ei
The U-test uses {\em ranks}, not precise numeric values.
For example, if treatment $A$ generates $N_1=5$ values \{5,7,2,0,4\} and treatment $B$ generates $N_2=6$ values \{4,8,2,3,6,7\}, 
then these sort as follows:
\begin{center}
{\small
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c}
Samples& A & A & B & B & A & B & A & B & A & B & B\\
Values & 0 & 2 & 2 & 3 & 4 & 4 & 5 & 6 & 7 & 7 & 8\\
\end{tabular}}\end{center}
On ranking, averages are used when values are the same:
\begin{center}
{\small
\begin{tabular}{c|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c}
Samples& A&  A  &B&  B&  A & B  &A&  B&  A&  B & B\\
Values&  0  &2  &2&  3&  4 & 4  &5&  6&  7&  7 & 8\\
Ranks&   1 &2.5&2.5 &4& 5.5&5.5 &7&  8& 9.5&9.5& 11\\
\end{tabular}}\end{center}
Note that, when ranked in this manner,
the largest value (8) gets the same rank
even if it was
ten to a hundred times larger.
That is, such rank tests
are less susceptible to large outliers.

\fig{mw} shows the U-test for 
$A$ and $B$.
The test concludes that these treatments have the same
median ranked values (at the 95\% significance level). 
As defined in \fig{mw}, this test  counts
the  $wins$, $ties$, and $losses$ for $A$ and $B$
(where $A$ and $B$ are single or groups of methods).
Since we seek methods that can be rejected, the value of interest to us is
how often methods {\em lose}.
\begin{figure}
\begin{tabular}{|p{.95\linewidth}|}\hline
\footnotesize
The sum and median of A's ranks is
\[\begin{array}{ccl}
sum_A &=& 1 + 2.5 + 5.5 + 7 + 9.5 = 25.5\\
median_A& =&5.5
\end{array}\]
and the sum and median of B's ranks is
\[\begin{array}{ccl}
sum_B&=& 2.5 + 4 + 5.5 + 8 + 9.5 + 11 = 40.5\\
median_B&=& 6.75
\end{array}\]
The $U$ statistic is then calculated 
where \mbox{$U_x=sum_x-(N_1(N_2+1))/2$}:
\[\begin{array}{c}
U_A = 25.5 - 5*6/2 = 10.5\\
U_B = 40.5 - 6*7/2 = 19.5
\end{array}\]
These can be converted to a Z-curve using:
\[
\begin{array}{rclcl}
\mu&= &(N_1N_2)/2&=&516.4\\
\sigma & = & \sqrt{\frac{N_1N_2(N_1 + N_2 + 1)}{12}}&=&5.477\\
Z_A & = & (U_A - \mu)/\sigma&=&-0.82\\
Z_B & = & (U_B - \mu)/\sigma&=&0.82\\
\end{array}
\]
(Note that $Z_A$ and $Z_B$ have the same absolute value.
In all case, these will be the same, with opposite signs.)

If $abs(Z) < 1.96$ then the samples $A$ and $B$ have the same median rankings (at
the 95\% significance level). In this case, we add one to both $ties_A$ \& $ties_B$.
Otherwise, their median values can be
compared, using some domain knowledge. In this work, {\em lower} values
are better since we are comparing relative errors. Hence:
\bi
\item
If $median_A < median_B$ add 1 to both $wins_A$ \& $losses_B$.
\item
Else if $median_A > median_B$ add 1 to both $losses_A$ \& $wins_B$.
\item
Else, add 1 to both $ties_A$ \& $ties_B$. 
\ei
\\\hline
\end{tabular}
\caption{An example of the Mann-Whitney U test.}\label{fig:mw}
\end{figure}



\section{Incremental Studies}
\fig{inc}
shows the inter-experimental results as the size 
of the training set increased in size. Note two main visual patterns:
\bi
\item Increasing the training set size does little or nothing to the median $balance$ values.
\item In some data sets, increasing the training set size increased the variability (spread)
in the $balance$ values.
\ei
Mann-Whitney tests were applied to test the first visual pattern. 
Detectors learned from just $N$ instances do as well
as detectors learned from any other number of instances.

\bi
\item
N=100 for \{CM1,KC2,KC3,MC2,MW1,PC3,PC4\}
\item
N=200 for  \{KC1\}
\item
N=300 for \{PC1\}
\item
N=4,900 for \{PC2\} (but N=800 only lost 2 out of 53 trials)
\item
N=8,300 for \{MC1\} (but N=400 only lost once out of 92 trials)
\item
N=11,000 for \{PC5\} (but N=300 only lost twice out of 169 trials)
\ei

%\section{Group B- one example: ``KC3''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  60 &  80 &  80 &  80 & 100 & \boxplot{60}{20}{80}{80}{20}\\
%pd / intra &  40 &  60 &  80 &  80 & 100 & \boxplot{40}{20}{80}{80}{20}\\
%pf / inter &  14 &  19 &  24 &  31 &  38 & \boxplot{14}{5}{24}{31}{7}\\
%pf / intra &  10 &  17 &  21 &  26 &  36 & \boxplot{10}{7}{21}{26}{10}\\
%\cline{2-7}\end{tabular}
%}
%
%\section{Group C- two examples such as ``MC1''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  57 &  71 &  86 &  86 & 100 & \boxplot{57}{14}{86}{86}{14}\\
%pd / intra &  57 &  71 &  86 &  86 & 100 & \boxplot{57}{14}{86}{86}{14}\\
%pf / inter &  17 &  18 &  19 &  20 &  21 & \boxplot{17}{1}{19}{20}{1}\\
%pf / intra &  15 &  16 &  17 &  18 &  20 & \boxplot{15}{1}{17}{18}{2}\\
%\cline{2-7}\end{tabular}
%}
%


\section{Under-sampling Studies}
\fig{under} shows the results of an under-sampling study where $N\in\{25,50,75,...\}$
defective modules were selected along with an equal $N$ number of defect-free modules.
Note the same visual pattern as before: increasing data does not necessarily improve $balance$.

Mann-Whitney tests were applied to test this visual pattern. 
Detectors learned from just $N$ instances do as well
as detectors learned from any other number of instances:

\bi
\item
N=25 for \{CM1,KC2,KC3,MC1,MC2,MW1,PC1,PC2\}
\item
N=75 for \{PC3\} (and N=25 losses three times out of 11).
\item
N=200 for \{PC4\} (but N=25 losses once out of 13 trials)
\item
N=575 for \{KC1\} (for N=25 only lost once out of 25 trials)
\item
N=1025 for \{PC5\} (but at N=25, the lost lose only once our of 25 trials)
\ei


%------------------------------------------------------------------------- 

\end{document}

