%\documentclass[onecolumn,12pt,draftcls,journal,compsoc]{IEEEtran}
\documentclass[twocolumn,journal,compsoc]{IEEEtran}
\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		%\put(0,0){\line(0,1){8}}
		%\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\makeatletter
  \AddToShipoutPicture{%
    \setlength{\@tempdimb}{.5\paperwidth}%
    \setlength{\@tempdimc}{.5\paperheight}%
    \setlength{\unitlength}{1pt}%
      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{5cm}{5cm}\selectfont{Draft v1}}}}}
\makeatother

\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage[nocompress]{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}

\begin{document}

\title{Cross- vs Within-Company\\Defect Prediction Studies}

\author{Tim~Menzies,~\IEEEmembership{Member,~IEEE,}
Burak Turhan,
		Justin~Distefano,
        and~Ayes Bener\thanks{Dr. Menzies and Mr. Distefano are with the Lane Department of Computer Science and Electrical Engineering, West Virginia
University.
Mr. Distefano is also Chief Programmer and Integrated Software Metrics.
Emails: \protect\url{tim@menzies.us},
\protect\url{jdistefano@ismwv.com}.
}
\thanks{Mr. Turhan and Dr. Bener are with the Department of Computer Engineering, Bogazici University, Turkey.
Emails:
\protect\url{turhanb boun.edu.tr},
\protect\url{bener@boun.edu.tr}.
}\thanks{The research described in this paper
was supported by
Bogazici University research fund      under grant number BAP-06HA104
and at
West Virginia  University
under grants with NASA's Software Assurance Research Program.
Reference herein to any specific commercial product, process, or service
by trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States Government.
}\thanks{See 
See \protect\url{http://menzies.us/pdf/07interra.pdf} for an earlier draft of this paper.
Manuscript received October 1, 2007; revised XXX,}} 

\markboth{Journal of ???,~Vol.~6, No.~1, January~2007}%
{Menzies \MakeLowercase{\textit{et al.}}: Cross- vs Within-Company Defect Estimation Studies}

%We found that the cross-company model provided poor 
%predictions for the single-company projects and much worse 
%predictions than the single-company model.
\maketitle

\begin{abstract}\newline
\underline{OBJECTIVE}: This paper tests under what circumstances can individual organizations
rely on cross-company (CC) or within-company (WC)-based defect prediction.

\underline{METHOD}: 
Using a dozen data sets,
detectors predictors were trained from
(a)~eleven sets, then tested on the twelfth; or (b)~trained
and tested on the same data set; or (c)~trained and tested
on a decreasing
amount of within-company data.

\underline{RESULTS}: 
Using within-company data, adequate defect predictors can be learned from
as few as two dozen faulty modules.
However, in $\frac{8}{12}$ data sets,
large amounts of cross-company data significantly increased {\em both} the median
 of 
$pd$ detection  and $pf$ false alarms:
 $pd/pf = 80/26\%$  (WC) and $91/53\%$  (CC).  

\underline{CONCLUSION}:
Companies lacking local data can  use cross-company data to predict detects. 
Given enough cross-company data, such predictors will have large false alarm rates
but a much larger probability
of detection.
Cross-company learning is hence indicated for mission critical software
were the extra costs associated with high false alarm rates  
are compensated by the associated increase in software assurance.
For other classes of software, false alarm rates can be decreased using a small amount of local data.
In our experiments, the use of
local data halved the false
alarm rate while only decreasing $pd$ by $\approx10\%$.
\end{abstract}

%------------------------------------------------------------------------- 

\section{Introduction}
Part of good research is knowing what questions to ask, and when to ask them.
Kitchenham et.al.~\cite{kitch07} pose an important question: when should companies 
learn from their own data, and when should they use someone else's?

This is a very timely question.
In the age of open source development, web-accessible XML data, mash-ups, and
reusable
libraries of SE data, it is quite possible that organizations can access more
data on software engineering
{\em outside} their company than {\em within}.
It is tempting to use such {\em cross-company} data since, as
Kitchenham et.al. observe:
\bi
\item
The time required to collect enough data on past projects from within a  company
may be prohibitive.
\item
By the time enough data is collected with within a company,
technologies may have changed so older
projects may not represent current practice.
\ei

The value of using such external data is an open
question.  Mendes et.al.~\cite{mendes07} found within-company data
performed much better than cross-company data for predicting
estimation effort of web-based projects.  They only recommend using
cross-company data in the special case when that ``data is obtained
using rigorous quality control procedures''.  Kitchenham
et.al.~\cite{kitch07} conclude that the value of CC vs WC data is
unclear: \begin{quote} {\em 
some organisations would benefit from using models derived from
cross-company benchmarking databases but others would not.}~\cite{kitch07}.
\end{quote}

The goal of this paper was to check if existing on-line tools and
data can extend the  Kitchenham et.al. conclusions.
This approach differs from other work that:
\bi
\item
Performs extensive data collection especially for their task such as  Mendes et.al.
\item 
 Performs
extensive and systematic literature reviews such as  Kitchenham et.al.
\ei

The study described below was completed in three weeks. 
Our effort estimation results were not informative but,
when we turned instead to defect prediction,
stable and novel conclusions
were found. These conclusions did not require time consuming data collection or literature reviews. 

Our conclusions are therefore specific and general:
\bi
\item
Specifically,
companies lacking local data can  use cross-company data to predict detects. 
Given enough cross-company data, such predictors will have large false alarm rates
but a much larger probability
of detection.
\item
More generally,  the availability of on-line data means we can speed-up
some kinds of
software engineering research. Once a clear 
theoretical problem has been stated (e.g. the value of cross- vs within-company
data) then  we can quickly make insightful conclusions (caveat: only for those problems
with on-line data).
\ei 

\section{Accessing Data Mining Tools and Data}

The  goals of this paper were achieved due to the large number of tools and data sets readily
available for free download from the Internet.
Anyone with access to Google can easily find and download
numerous robust data mining toolkits supports hundreds of
combinations of learners, data pre-processors, feature subset
selectors, etc\footnote{
e.g. the R project\footnote{\url{http://www.r-project.org/}},
Orange\footnote{\url{http://www.ailab.si/orange/}},
RapidMiner\footnote{\url{http://rapid-i.com/content/view/10/32/lang,en/}}
and the WEKA~\cite{witten05}.}
Also, there is more data to process. 
For example, the PROMISE repository of SE
data\footnote{\url{http://promisedata.org/repository}} has only
been active for 3 years but already contains 44 data sets. That repository
is growing rapidly (on average, 37\% more data sets each year).
This  growing supply of on-line data sources is due to several recent
developments. 
Organizational inertia means that
once procedures are
in place to:
\bi
\item Build and maintain an artifacts repository;
\item
 Build a sanitizer that
can ``scrub'' data of corporate identifies prior to release;
\item
Resolve any remaining legal issues;
\ei
then once one data set is generated,
many more will follow.  (one for each new product and for each
new release of each product). This has the added benefit that data
collected from multiple projects may have the same same schema.
For example:
\bi
\item The open
source community has proven to be a rich source of information.
PROMISE holds six ECLIPSE data sets (file and package- level metrics
for ECLIPSE 2.0, 2.1, 3.0) as well as data from FreeBSD, Xfree,
Xorg, and other projects.  
\item NASA's Metrics Data Program is now defunct
but that program generated twelve of the data sets used in this
study.
\ei
The increasing  availability of cross-company data 
makes the Kitchenham et.al. research question a very pressing quesion.

One interesting phenomonen observed in the PROMISE repository is the 
is the {\em flood} effect: by the time an organization 
The Kitchenham et.al. study was a 
{\em systematic review} that drew its conclusions from a wide-ranging literature search.
Such systematic reviews are becoming more frequent and are an example of the {\em evidence-based
software engineering} movement.


Software managers can use predictor of defective modules to 
focus the testing on parts of the system that seem defect-prone.
These potential defect-prone trouble spots can then
be examined in more detail by, say, model checking, intensive testing, etc.
Previously we have explored the use of data miners to learn such predictors from static
code attributes. We found that a large class of predictors proposed in numerous
publications were out-performed by Bayes classifiers that combined partial evidence
from multiple attributes~\cite{me07d}. 

Kitchenham et.al. would characterize that previous work as a {\em within-company} (WC) study
where the data from one organization is applied to new examples within the same organization.
%XXX much more regularity in software engineering that previously suspected

Therefore, companies are tempted to eskew WC studies, preferring instead to use some
cross-company (CC) study where data from elsewehre is applied locally.
The value of cross- vs within-company studies for effort estimation has been the subject
of much recent work.
In the special case of effort prediction from static code features, some of those quality controls can be automated.
In the following cross- vs within- study, features of code 
modules were extracted automatically using the same tool set (the McCabe HiQ tool\footnote{justin?}) that is
widely used in industry\footnote{jsutin- can we say that predict is an alternate data collector that HiQ}.
We find that

\section{Data}

\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{l|lll}
data & examples & features  & defective\\\hline
pc5  & 17,186   & 38          & 3.0\%\\
mc1  & 9,466    & 38          & 0.71\%\\
pc2  & 5,589    & 36          & 0.41\% \\
pc3  & 1,563    & 37          & 10.23\%\\
pc4  & 1,458    & 37          & 12.2\%\\
pc1  & 1,109    & 21          & 6.94\%\\
kc1  & 845      & 21          & 15.45\%\\
kc2  & 522      & 21          & 20.49\%\\
cm1  & 498      & 21          & 9.83\%\\
kc3  & 458      & 39          & 9.38\%\\
mw1  & 403      & 37          & 7.69\%\\
mc2  & 161      & 39          & 32.29\%
\end{tabular}
\end{center}
\caption{Twelve data sets, sorted in order of \#examples.}
\end{figure} 
\subsection{Detector Mathematics}

Let $\{A,B,C,D\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary detector. 
Certain standard measures can be computed from
$A,B,C,D$: 
\[
\begin{array}{ll}
pd=recall=&\frac{D}{B+D}\\
pf=&\frac{C}{A+C}\\
prec=precision=&\frac{D}{D+C}\\
acc=accuracy=&\frac{A+D}{A+B+C+D}\\
selectivity=&\frac{C+D}{A+B+C+D}\\
neg/pos=&\frac{A+C}{B+D}
\end{array}
\]
The last measure ($neg/pos$) is quite interesting and
can be used within the above equations as follows:
\begin{equation}\label{eq:one}
prec = \frac{D}{D+C} = \frac{1}{1+ \frac{C}{D}} = \frac{1}{1 + neg/pos \cdot pf/recall}
\end{equation}

Ideally, our detectors have high precision {\em and} high recall {\em and } low false alarm
rates. This ideal situation is very hard to achieve. In practice, detectors must trade-off one goal
for the other.

\subsection{Trade-offs with Recall (pd) and False Alarm Rates (pd)}
Detector performance can be  assessed via {\em
receiver-operator} (ROC) curves.
Formally,  a defect {\em predictor}   hunts for a {\em signal} that a
software module is defect prone. Signal detection
theory~\cite{heeger98} offers 
ROC curves as an analysis method for assessing different
predictors.
%% ROC curves are widely used in various fields including
%% assessing different clinical computing systems~\cite{adlassnig89} and
%% assessing different machine learning methods~\cite{provost98case}.
%% The central intuition of ROC curves is that different detectors can
%% be assessed via how often they correctly or incorrectly respond to
%% the a signal.
A typical ROC curve is shown in \fig{typical}. The y-axis shows
probability of detection ($pd$) and the x-axis shows probability of false alarms ($pf$).
By definition,
the ROC
curve must pass through the points $pf=pd=0$ and $pf=pd=1$
(a predictor that
never triggers never makes false alarms; a predictor that always
triggers always generates false alarms).
Three interesting trajectories connect these points:
\be
\item
 A straight line from
(0,0) to (1,1) is of little interest since it offers {\em no
information}: i.e.  the probability of
a  predictor firing is the same as it being silent.
\item
Another trajectory is the {\em negative curve} that bends away
from the ideal point. Elsewhere~\cite{me02e}, we have found that
if predictors negate their tests, the negative curve will transpose into a {\em
preferred
curve}.
\item
The point ($pf=0$, $pd=1$) is the ideal position (a.k.a. ``sweet spot'')
on a ROC curve. This is
where we recognize all errors and never make mistakes. {\em Preferred curves} bend
up towards this ideal point.
\ee

In the ideal case, a predictor has a high probability of detecting a
genuine fault ($pd$) and a very low probability of false alarm
($pf$). This ideal case is very rare. The only way to achieve high probabilities
of detection is to trigger the predictor more often. This, in turn, incurs the cost
of more false alarms.

In practice, engineers {\em balance}
between $pf$ and $pd$. To operationalize this notion of {\em
balance}, we define $bal$ to be the Euclidean distance from the sweet
spot $pf=0,pd=1$ to a pair of $<pf,pd>$. For convenience, we
(a)~normalize $bal$ by the maximum possible distance across the ROC square
( $\sqrt{2}$); (b)~subtract this from 1; and (c)~express it as a percentage; i.e.
\begin{equation}\label{eq:bal}
balance= bal             = 1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{equation}
Hence, better and {\em higher} balances fall  {\em closer} to the desired  sweet spot  of $pf=0,pd=1$.
\subsection{Trade-offs with Precision}

Note that \eq{one} can be re-expressed as follows:
\begin{equation}\label{two}
pf = \frac{pos}{neg}\cdot\frac{ (1 -  prec) }{ prec}\cdot recall
\end{equation}
That is, in Equation~\ref{two}, when $recall$ is fixed then
false alarm rate becomes controlled by precision and a 
fixed constant determined by the data set being examined; i.e.
when ($\alpha=neg/pos$) and \mbox{$recall=1$}
then:
\begin{equation}\label{three}
pf = \alpha\cdot \frac{1-prec}{prec}
\end{equation}
From Equation~\ref{three}, it is clear that it is porblematic to demand detectors
with high recall {\em and} high precision {\em and} low false alarm rates.
For example, for any targeted recall value,
{\em increasing} 
precision requires
{\em decreasing}
false alarm rates; e.g. for \mbox{$prec\in\{0.5,0.70,0.9,0.95\}$}, $pf$ becomes $\{1,0.43, 0.11, 0.005\}$,
respectively. The effect is particularly marked for data sets
with large $neg/pos$ ratios.

Many software engineering data sets have extremely
large
$neg/pos$ ratios.
%Elsewhere~\cite{me03a,me03q}\footnote{See also the PREDICTIVE toolkit~  \url{http://www.ismwv.com/}},
%we have conducted extensive studies
%on methods for  generating and selecting detectors appropriate to different business contexts.
%The Zhangs' equation enables a more nuanced  selection process.
\begin{figure}[!b]
\begin{center}
\includegraphics[width=2.7in]{locplatonic.pdf}
\end{center}
\caption{Regions of a typical ROC curve.}\label{fig:typical}
\end{figure}
For example:
\bi
\item In our previous DMP study~\cite{me07b}, the data sets studied
had $neg/pos$ ratios
of
1.04, 
7.33,
9,
10.11,
13.29,
15.67, and
249.
\item
  Hayes, Dekhtyar and Sundaram \cite{hayes06} use text mining to                                                                     
find pairs of connected requirements in a corpus of 220 requirements and                                                                     
235 design elements (the same CM-1 dataset used in 
\cite{me07b}).
The total number of possible links in the dataset is $220\cdot 235 =                                                                        
51,700$, while the the ground truth RTM contains 361 links, for the                                                                         
$neg/pos$ ratio of $51,700/361 = 143.2$.   
\item
For an extreme example, Google reports that
over $10^{9}$ web pages contain the phrase ``software''
but only one them is the home page of this journal. Hence, $neg/pos$ for
web searching is at least $10^9$. 
\ei
%For example, consider the  the mission critical situation requiring $recall=1$.
%Equation~\ref{two}
%shows that when $prec=0.25$ and
%$neg=500$ and $pos=100$, achieving the goal of $recall=1$ requires
%a detector with  $pf=0.6$.
%As precision increases, detectors become harder and harder to
%design. For example, continuing the security example,
%at $recall=1$ and $neg/pos=5$,
%increasing the precision 
%0.25 to
%0.5 to 0.75 requires detectors with
%false alarm rates of 0.6 to  0.2 to 0.067 (respectively).
%As discussed below, building high precision detectors
%can be expensive.
%Depending on the available research budget, companies
%might elect to use lower precision detectors that are cheaper to build.
%
%experiments - some minor changes in formulas may affect the exact numbers,
%but not the ballparks), our pf for CM-1 dataset, without filtering,
%feeback, or any other tricks, is probably around 60-65% (we retrieve
%around 33,000 links outright, out of 51700 possible). Only with filtering
%(either by absolute value, or by relative rank) can we get to pf of 0.1 or
%better with decent recall.
%MGF report probability of detection and false alarm rates of
%of \{pd=75\%, pf=25\%\}, respectively. MGF argue that these are high enough to be useful.
%However,  Zhang\&Zhang derive an expression that can compute precision from the data found
%in MGF. Those computed  precisions suggest that the MGF results are  quite useless:
%\[
%\{min,median,max\}=\{2\% 20\%, 70\%\}
%\]
%
%Here, we show that Zhang\&Zhang
%do not take their conclusions far enough. 
%A large number of current industrial problems can be characterized by ``small targets'' and
%``some false alarms''  (quoted terms are defined below).
%Using the expression derived by Zhang\&Zhang, we show that all such domains are prone to
%the problem of low precision.
%However, unlike Zhang\&Zhang, we will argue that low precisions are acceptable in some
%of those domains.
%Also, unlike Zhang\&Zhang, we also will offer a specific research direction 
%on how to improve the MGF results. Interestingly, that research direction
%{\em requires} accepting low precision detectors, at least initially.

Figure~\ref{pprnp}
graphs \eq{one} for the  DMP $neg/pos$ ratios.
\begin{figure}
\begin{center}
\includegraphics[width=3in]{y_0.pdf}
\end{center}
\caption{The relationship between
$pd, prec, recall$ and $neg/pos$. 
}\label{pprnp}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=3in]{y_05.pdf}
\end{center}
\caption{
Figure~\ref{pprnp},
cropped to the region
where $prec>0.5$.
}\label{pprnp1}
\end{figure}
Figure~\ref{pprnp1} does the same, 
but is restricted to zones of higher precision:
only the surface for
$0.5\le prec \le 1$ is shown. 
That shadow of the  surface on the bottom plane
shows that this zone of high precision, high recall, and large
$neg/pos$.
As $neg/pos$ increases, 
high recall\&precision
is only possible when $pf$ becomes vanishingly small.
For example, in the ranges 
\mbox{$0.65 \le prec,recall \le 0.8$}, 
Equation~\ref{two} reports that $pf$ falls into the following ranges:
\begin{itemize}
\item  $0.023 \le pf \le 0.062$  for $neg/pos = 7$;
\item  $0.0108 \le pf \le 0.0287$ for $neg/pos = 15$;
\item  $0.007 \le pf \le  0.0017$ for $neg/pos = 250$;
\end{itemize}
Detectors learned in the domain of
software engineering
rarely yield
high precision detectors (see Figure~\ref{precs}).
Using the above  equations, the reasons for this are very clear:
\bi
\item
Those detectors all try to maximize recall; 
\item
Figure~\ref{pprnp} shows that such 
detectors can only achieve high precision in the rare case of very low $pf$.
\ei

Not only do our   equation explain the Figure~\ref{precs} results,
they also inform the 
 the instability of precision {\em and} the stability of $pf$ and $pd$ (recall) 
seen 
in previous studies~\cite{me04c}"
\bi
\item
Note how, in Figure~\ref{pprnp},
that at very small $pf$ values,
tiny changes in $pf$ can lead to very large changes in $prec$ 
(sudden jumps from zero to one).
\item
The other measures in
Figure~\ref{pprnp}, on the other hand, change far more smoothly and slowly.
\ei
That is, the mathematics of these detectors
explains numerous prior results such as those shown in Figure~\ref{precs} 
(i.e. \cite{me03q,me07b,huan06,hayes06,antoniol02,antoniol05,marcus03}).
\begin{figure}[!b]
\small
\begin{center}
\begin{tabular}{|p{0.95\linewidth}|}\hline
\bi
\item
The DMP paper achieved recalls over 70\% and minimum false alarm
rates of 15\%.  Using our equation and data from DMP, it
can be shown that the DMP precisions were quite low:
$\{min,median,max\}=\{2\% 20\%, 70\%\}$ (the last number came from
a data set with $neg/pos\approx 1$).
\item
Huang et.al.~\cite{huan06} won ``best paper'' at the 2006 IEEE
Requirements Engineering conference with detectors exhibiting
$prec\approx 0.25$.
\item
Without extensive feedback from human experts, the classifiers used
by Hayes, Dekhtyar, and Sundaram exhibited $pf>0.6$ (hence, very
low precisions)~\cite{hayes06}.
\item
Other such as Antoniol et.al.~\cite{antoniol02,antoniol05} and
Marcus et.al.~\cite{marcus03} researchers into software tracability
report that high recall is achievable only with low precision
detectors.  Antoniol has also been exploring bug traces in Mozilla
components and has found the same high-precision, low-recall
trade-off.
\item
After much experiments with linear regression, model tree learners,
Bayes classifiers, decision tree learners, singleton rule learners,
and some home-brew learners~\cite{me03q}, the general trend is very
clear. For those learners, executing on the DMP datasets, $pf \le
pd - 0.5$; That is, for those learners and those data sets, obtaining
high recall figures of $pd>0.6$ implies $pf>0.1$ and, consequently,
low precision.
\ei  
\\\hline \end{tabular} \end{center} 
\caption{Some
low precisions seen in the SE literature.}\label{precs} 
\end{figure}

\subsubsection{When Low Precision is Useful}

Achieving both high precision\& recall \& low false alarm rates can be problematic.
Optimizing for one often compromises the other (especially for data sets with large $neg/pos$ ratios).
Fortunately, there are may industrial
situations 
where low precision and high recall
detectors are useful.
For example, one of us (DiStefano) has used
our low-precision detectors to review flight code
developed at
the NASA Glenn Research Center (Ohio, USA).
When the results of these detectors were presented to the lead flight
engineer, he confirmed that the identified sections (which did not
have any recorded defects) had been problematic to maintain, and
contained several bugs which had not yet been entered into the
defect system. 


For another example,
from
outside the field of SE, 
a user of commercial
web search engine like Google can quickly flick through (say) three
pages of results before finding a page of interest.  Google has so
many return customers since even with precisions of (say) $\frac{1}{30}$,
the effort involved in looking at a page is so low that users don't
mind examining 29 false alarms.  

More generally, there are several situations where low precision
detectors are useful:
\begin{itemize}
\item
{\em When the cost of missing the target is prohibitively expensive.}
In mission critical or security  applications, the goal of
100\% recall may be
demanded in all situations, regardless of the  precision.
\item {\em When only a small  fraction
the data
is returned.}
Hayes, Dekhytar, \& Sundaram call this fraction
$selectivity$ and offer an extensive discussion of the merits of
this measure~\cite{hayes06}.  \item {\em When there is little or no cost in checking
false alarms.}  For example,  a detector we have found useful in industrial settings is to
check modules  where \[\frac{lines\; of\; comments}{lines\; of\; code} > 0.25\]
This detector triggers on complex functions that 
programmers comment extensively, instead of splitting up into smaller, more maintainable,
functions.
This detector is imprecise- it often triggers on well-written
functions with detailed comments. 
However, based on commercial experience, we assert
that it is fast and simple for
a human agent to inspect  the identified modules and discern which                                                                    
ones were well-written and which were over-commented to disguise being                                                                      
badly coded.
We use this detector to find code that should be rewritten prior to release.
\end{itemize} 

\section{Data}

This section discusses the data used in this study.
Each module of each data
sets describes the attributes of that module,
plus the number of defects known for that module.
This data comes from
eight
sub-systems taken from four systems.  These systems were developed
 in
different geographical locations across North America.
Within a system, the sub-systems
shared some a common code base but did not pass personnel or code
between sub-systems. 

Each data set was pre-processed by removing the module identifier attribute
(which  is different for each row). Also, the
$error\_count$ column was converted into a boolean attribute called {\em defective?}  as follows:
$defective? = (error\_count \ge 1)$.
Finally, the $error\_density$ column was removed (since it can be
derived from line counts and $error\_count$).
The pre-processed data
sets had 38 attributes, plus one target attribute
({\em defective?})
shown in \fig{attr} and include Halstead, McCabe, lines of code, and other miscellaneous
attributes.

\begin{figure}[!b]
\begin{center}
{\footnotesize
\begin{tabular}{l@{~}|l@{~}|l@{~}|c@{~}p{3.9cm}|}\cline{2-5}
%\begin{tabular}{|l|l|lcl|}\hline
%\multicolumn{3}{|c}{grouping}&\multicolumn{2}{|c|}{measure}\\\hline
		&\multicolumn{2}{l}{m = Mccabe }       &$v(g)$  &cyclomatic\_complexity  \\
        &        \multicolumn{2}{l}{~}                                 & $iv(G)$&design\_complexity\\
        &         \multicolumn{2}{l}{~}                                & $ev(G)$&essential\_complexity\\\cline{2-5}
        &locs&{loc}                         &   &loc\_total (one line = one count\\\cline{3-5}
	&	&loc(other)                      &   &loc\_blank           \\
        &       &                                &   &loc\_code\_and\_comment\\
        &       &                                &   &loc\_comments         \\
        &        &                                &   &loc\_executable\\
        &       &                                &    &number\_of\_lines (opening to closing brackets)\\\cline{2-5}
	&{Halstead} & h& $N_1$ &num\_operators\\
        &       &                                 &$N_2$ &num\_operands \\
        &       &                                 &$\mu_1$ &num\_unique\_operators\\
        &       &                                 &$\mu_2$ &num\_unique\_operands\\\cline{3-5}
        &       &  { H}&$N$  &length: $N= N_1 + N_2$\\
        &       &                                 & $V$  &volume: $V=N * log_{2}\mu$\\
        &       &                                 & $L$  &level: $L=V^*/V$ where\newline $V^*=(2+\mu{_2}^{*})log_{2}(2+\mu{_2}^{*})$\\
        &       &                                 & $D$  &difficulty: $D=1/L$\\
        &       &                                 &$I$   &content: $I=\hat{L}*V$ where\newline $\hat{L}=\frac{2}{\mu_{1}} * \frac{\mu_{2}}{N_{2}}$ \\
        &       &                                 & $E$  &effort: $E=V/\hat{L}$\\
        &       &                                 & $B$  &error\_est\\
        &       &                                 & $T$  &prog\_time: $T=E/18$ seconds\\\cline{2-5}
        &       \multicolumn{2}{l}{misc = Miscellaneous}                                  &&branch\_count\\
        &       \multicolumn{2}{l}{~}                                  &&call\_pairs\\
        &       \multicolumn{2}{l}{~}                                  &&condition\_count\\
        &       \multicolumn{2}{l}{~}                                  &&decision\_count\\
        &       \multicolumn{2}{l}{~}                                  &&decision\_density\\
        &       \multicolumn{2}{l}{~}                                  &&design\_density\\
        &       \multicolumn{2}{l}{~}                                  &&edge\_count\\
        &       \multicolumn{2}{l}{~}                                  &&global\_data\_complexity\\
        &       \multicolumn{2}{l}{~}                                  &&global\_data\_density\\
        &       \multicolumn{2}{l}{~}                                  &&maintenance\_severity\\
        &       \multicolumn{2}{l}{~}                                  &&modified\_condition\_count\\
        &       \multicolumn{2}{l}{~}                                  &&multiple\_condition\_count\\
        &       \multicolumn{2}{l}{~}                                  &&node\_count\\
        &       \multicolumn{2}{l}{~}                                  &&normalized\_cyclomatic\_complexity\\
        &       \multicolumn{2}{l}{~}                                  &&parameter\_count\\
        &       \multicolumn{2}{l}{~}                                  &&pathological\_complexity\\
        &       \multicolumn{2}{l}{~}                                  &&percent\_comments\\\cline{2-5}
\end{tabular}}
\end{center}
\caption{Attributes used in this study. }\label{fig:attr}
\end{figure}
% XXX need introduction words saying contribution is definition of baseline experiment

The Halstead attributes were derived by Maurice Halstead in 1977. He
argued that modules
that are hard to read are more likely to be fault
prone~\cite{halstead77}. Halstead estimates reading complexity by
counting the number of operators and operands in a module: see the
$h$ attributes of \fig{attr}.  These three raw $h$  Halstead
attributes were 
then used to compute the $H$: the eight derived Halstead attributes
using the equations shown in
\fig{attr}.  In between the raw and derived Halstead attributes are
certain intermediaries (which do not appear in the MDP data sets);
 \bi
\item
$\mu= \mu_1 + \mu_2$;
\item
minimum operator count: $\mu_1^*=2$;
\item
$\mu_2^*$  is the minimum operand count and equals the number of module parameters.
\ei

%XXXX conclusion. Halstead and McCabe very odd 1970s. sureyin the 21st cenyru we can do better

An alternative to the Halstead attributes are the complexity attributes proposed by Thomas McCabe in 1976.
Unlike Halstead,
McCabe
 argued that the complexity of
pathways {\em between} module symbols are more insightful than just a count of the symbols~\cite{mccabe76}. 
The first three lines of \fig{attr}
shows McCabe three
main attributes for this pathway complexity. These are defined as follows.
A
module is said to have a {\em flow graph}; i.e. a directed graph where each node corresponds to a
program statement, and each arc indicates the flow of control from
one statement to another.
The {\em cyclomatic complexity} of a module is   $v(G) = e - n +
2$ where $G$ is a program's flow graph, $e$ is the number of arcs in
the flow graph, and $n$ is the number of nodes in the
flow graph~\cite{fenton96}.
The {\em essential complexity}, ($ev(G)$) or a module is  the extent to which a
flow graph can be ``reduced'' by decomposing all the subflowgraphs
of $G$ that are {\em D-structured primes}
(also sometimes referred to as ``proper one-entry
one-exit subflowgraphs''~\cite{fenton96}). $ev(G) = v(G) - m$ where $m$ is the number of subflowgraphs of
$G$ that are D-structured primes~\cite{fenton96}.
Finally, the
{\em design complexity}   ($iv(G)$) of a module is the cyclomatic complexity of a
module's reduced flow graph.

At the end of \fig{attr} are a set of {\em misc} attributes that are less well-defined than lines of code attributes
or the Halstead and McCabe attributes. 
The meaning of these attributes is 
   poorly documented in the MDP database. Indeed,
   they seem to be values generated from some unknown tool set that
   was available at the time of uploading the data into the MDP.
Since there are difficulties in reproducing these attributes at other sites, an argument could be made
for removing them from this study.
A counter-argument is that if static code attributes are as weak as suggested by Shepherd, Ince, Fenton and Pfleeger
then we should use all possible attributes in order to make maximum use of the available information.
This study took a middle ground: all these attributes were passed to the learners and they determined
which ones had most information.

%Two different categories of re-sampling methods commonly used to overcome the 
%problem of imbalanced data.  The first increases the frequency of the minority class in 
%the training set (Oversampling) and the other decreases the frequency of the majority 
%class in the training set (Undersampling) (Weiss and Provost 2001; Drummond and 
%Holte 2003).
%Drummond, C. and R. C. Holte (2003). C4.5, class imbalance, and cost sensitivity: why 
%under-sampling beats over-sampling. Workshop on Learning from Imbalanced 
%Datasets II. 

%In this paper, we used cost curves to explore the in- 
%teraction of over and under-sampling on the decision 
%tree learner C4.5. Surprisingly, over-sampling is in- 
%effective, at least when the default settings are used, 
%often producing little or no change in performance. 
%Under-sampling does produce a reasonable sensitivity 
%to changes in misclassification costs and class distri- 
%bution. But representing these changes internally by 
%down-weighting gives the best performance overall. 

%~\cite{yun04}

An interesting repeated pattern in our data sets are {\em exponential distributions}
in the numeric attributes.
Elsewhere, we have conducted limited experiments suggesting that a
{\em logarithmic filter} on all numeric values might improve predictor
performance~\cite{me02e}.  
Such a filter replaces all numerics $n$ with their logarithm.
$ln(n)$.
The
log-filtered values are now more evenly spread across the y-range, making it easier
to reason about them. 
To test the value of log-filtering, all the data was passed through one of two filters:
Hence, all our numerics were pre-processed as follows:
\bi
\item {\em logNums}; i.e. logarithmic filtering where all numbers $n$ where replaced
with $log(n)$.
\item
To avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$.
\ei

\section{Non-parametric Analysis of Results}

\subsection{Quartile Charts of Performance Deltas}


The experiment described below explored numerous combinations of
filters, attributes, and learners all within a
$(M=20)*(N=10)$-way cross-evaluation 
study. Hence,
this experiment 
generated nearly thousands of measuring
$pd,pf,bal$
values from different {\em methods} applied to the
same $test$ data.

The performance
results for each method were sorted and displayed as {\em
quartile charts}.  To generate these charts, the performance
deltas for some $method$ 
were sorted to isolate the median and the lower and upper quartile of numbers.

\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\underbrace{40}_{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]

In a quartile chart,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the zero value.
The above numbers
would therefore be drawn as follows:


{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
We prefer quartile charts of performance deltas to other summarization
methods for M*N studies. Firstly, they offer a very succinct summary
of a large number of experiments. For example, in the next few pages
we will display thousands of results in very little area.
Secondly, they
are a {\em non-parametric} display; i.e. they make no assumptions
about the underling distribution.  Standard practice in data mining is
to compare the mean performance of different methods using
t-test~\cite{witten05}. T-tests are a {\em parametric method} that
assume that the underling population distribution is a Gaussian.
Recent results suggest that there are many statistical issues left to
explore regarding how to best to apply those t-tests for summarizing
M*N-way studies~\cite{brouck03,demsar06}.  
\subsection{Comparison Statistics}

The problem of comparatively assessing $L$ learners run on multiple sub-samples of $D$ data sets
has been extensively studied in the data mining community. 
T-tests that assume Gaussian distributions are strongly deprecated.
For example, Demsar~\cite{demsar06}
 argues that non-Gaussian populations are common enough to require a methodological
change in data mining. 

After reviewing a wide range of
comparisons methods\footnote{Paired t-tests with and without the use of geometric means of the relative ratios;
binomial tests with the Bonferroni correction;
paired t-tests; ANOVA; Wilcoxon; Friedman},
Demsar advocates the use of the 1945 Wilcoxon~\cite{wilcoxon45} 
signed-rank test that compares the ranks for the positive and negative differences (ignoring the signs). 
Writing five years early,
Kitchenham et.al.~\cite{kitc01} comment that the Wilcoxon test has its limitations.
Demsar's report  offers the same conclusion, 
noting that the Wilcoxon test requires that the sample sizes are the same.
To fix this problem, Demsar augments Wilcoxon with the Friedman test.

One test not studied by Demsar is Mann and Whitney's 1947
modification~\cite{mann47} to Wilcoxon rank-sum test (proposed along with his signed-rank test).
We prefer this test since:
\bi
\item
The Mann-Whitney U-test does not require that the sample sizes are the same.
So, in a single U-test, learner $L_1$ can be compared to all its rivals. 
\item
The U-test does not require any post-processing
(such as the Friedman test) to conclude if the median rank of one population (say, the $L_1$ results)
is greater than, equal to, or less than the median rank of another (say, the $L_2,L_3,..,L_x$ results).
\ei
The U-test uses {\em ranks}, not precise numeric values.
For example, if treatment $A$ generates $N_1=5$ values \{5,7,2,0,4\} and treatment $B$ generates $N_2=6$ values \{4,8,2,3,6,7\}, 
then these sort as follows:
\begin{center}
{\small
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c}
Samples& A & A & B & B & A & B & A & B & A & B & B\\
Values & 0 & 2 & 2 & 3 & 4 & 4 & 5 & 6 & 7 & 7 & 8\\
\end{tabular}}\end{center}
On ranking, averages are used when values are the same:
\begin{center}
{\small
\begin{tabular}{c|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c}
Samples& A&  A  &B&  B&  A & B  &A&  B&  A&  B & B\\
Values&  0  &2  &2&  3&  4 & 4  &5&  6&  7&  7 & 8\\
Ranks&   1 &2.5&2.5 &4& 5.5&5.5 &7&  8& 9.5&9.5& 11\\
\end{tabular}}\end{center}
Note that, when ranked in this manner,
the largest value (8) gets the same rank
even if it was
ten to a hundred times larger.
That is, such rank tests
are less susceptible to large outliers.

\fig{mw} shows the U-test for 
$A$ and $B$.
The test concludes that these treatments have the same
median ranked values (at the 95\% significance level). 
As defined in \fig{mw}, this test  counts
the  $wins$, $ties$, and $losses$ for $A$ and $B$
(where $A$ and $B$ are single or groups of methods).
Since we seek methods that can be rejected, the value of interest to us is
how often methods {\em lose}.
\begin{figure}
\begin{tabular}{|p{.95\linewidth}|}\hline
\footnotesize
The sum and median of A's ranks is
\[\begin{array}{ccl}
sum_A &=& 1 + 2.5 + 5.5 + 7 + 9.5 = 25.5\\
median_A& =&5.5
\end{array}\]
and the sum and median of B's ranks is
\[\begin{array}{ccl}
sum_B&=& 2.5 + 4 + 5.5 + 8 + 9.5 + 11 = 40.5\\
median_B&=& 6.75
\end{array}\]
The $U$ statistic is then calculated 
where \mbox{$U_x=sum_x-(N_1(N_2+1))/2$}:
\[\begin{array}{c}
U_A = 25.5 - 5*6/2 = 10.5\\
U_B = 40.5 - 6*7/2 = 19.5
\end{array}\]
These can be converted to a Z-curve using:
\[
\begin{array}{rclcl}
\mu&= &(N_1N_2)/2&=&516.4\\
\sigma & = & \sqrt{\frac{N_1N_2(N_1 + N_2 + 1)}{12}}&=&5.477\\
Z_A & = & (U_A - \mu)/\sigma&=&-0.82\\
Z_B & = & (U_B - \mu)/\sigma&=&0.82\\
\end{array}
\]
(Note that $Z_A$ and $Z_B$ have the same absolute value.
In all case, these will be the same, with opposite signs.)

If $abs(Z) < 1.96$ then the samples $A$ and $B$ have the same median rankings (at
the 95\% significance level). In this case, we add one to both $ties_A$ \& $ties_B$.
Otherwise, their median values can be
compared, using some domain knowledge. In this work, {\em lower} values
are better since we are comparing relative errors. Hence:
\bi
\item
If $median_A < median_B$ add 1 to both $wins_A$ \& $losses_B$.
\item
Else if $median_A > median_B$ add 1 to both $losses_A$ \& $wins_B$.
\item
Else, add 1 to both $ties_A$ \& $ties_B$. 
\ei
\\\hline
\end{tabular}
\caption{An example of the Mann-Whitney U test.}\label{fig:mw}
\end{figure}



\section{Intra-vs-Inter Results}
In this first round of experiments, we explored using data from local or imported sources.
In each case, some data set was declared ``local'' and another was declared ``import'' .
A training set was drawn from ``import'' and applied to a  test set created from 10\%
drawn from ``local'' (using stratified sampling). 

In the ``intra'' study, the ``local''
data set was the same as ``import''. That is, in these intra-experiments, 
detectors were learned and tested from the same data set.

In the ``inter'' study, the ``local''
data set was different to ``import''. To be precise, ``import'' was all data from all data set
except the ``local'' data.
That is, in these inter-experiments, 
detectors were learned from all avilable data, except data from the test ``local'' set.

Measured in terms of $pd$ and $pf$, the main effect seen in 8 test sets was that importing
lots of data (in the inter-experiment) increased {\em both} $pd$ and $pf$ (see group ``D'', below).
In three other data sets (see groups ``B'' and ``C'') the  changes were either statistically
insignificant (measured using a Mann-Whitney U-test) or the false alarm increased in the intra-experiment. 

{\scriptsize
\begin{tabular}{ll|l|p{1.5cm}|rl}
%pd(inter) vs  & pf(intra) vs & &  \\
group &pd(inter)  & pf(inter)& data sets & n \\\cline{1-4}
a& decreased      		 & decreased        &     PC5 &1& \rule{2.5mm}{2mm}\\
b& same                    & same             & KC3& 1&\rule{2.5mm}{2mm}\\ 
c& same                    &  increased            & MC1 PC2 &2&\rule{5mm}{2mm}\\
d& increased               &        increased & CM1 KC1 KC2 MC2 MW1 PC1 PC3 PC4 & 8&\rule{20mm}{2mm}
\end{tabular}}

The size of $pd$ increase was quite remarkable. The following chart shows that across our 12 data sets,
the top quarter of the $pd$s in the inter-experiemtnt were all in the maximum range of $pd=100\%$.
As predicted by \fig{typical}, the price of this dramatic increase in $pd$ is an near doubling
of the median $pf$: 

{\scriptsize
\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
 & min & q1 & median & q3 & max  \\\cline{2-7}
 pd / inter&       50&  80&  91&  100& 100& \boxplot{50}{30}{91}{100}{0}\\
 pd / intra&       17&  67&  80&   88& 100& \boxplot{17}{50}{80}{88}{12}\\
 pf / inter&        4&  26&  52&   73& 100& \boxplot{4}{22}{52}{73}{27}\\
 pf / intra&        0&  17&  26&   33&  73& \boxplot{0}{17}{26}{33}{40}\\
\cline{2-7}\end{tabular}
}

~\\

The increase in individual 
data sets was even larger. For example, in $KC2$, the median $pf$ rose from 73 to 91\% while
the median $pf$ rose from 31\% to 64\%:

%\section{Group D- eight examples such as ``KC2''}
{\scriptsize
\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
 & min & q1 & median & q3 & max  \\\cline{2-7}
pd / inter &  82 &  91 &  91 & 100 & 100 & \boxplot{82}{9}{91}{100}{0}\\
pd / intra &  55 &  73 &  73 &  82 &  91 & \boxplot{55}{18}{73}{82}{9}\\
pf / inter &  57 &  62 &  64 &  74 &  81 & \boxplot{57}{5}{64}{74}{7}\\
pf / intra &  14 &  24 &  31 &  33 &  45 & \boxplot{14}{10}{31}{33}{12}\\
\cline{2-7}\end{tabular}
}

~\\

In one data set,  a compeltely opposite effect was observed to the main effect.
In ``PC5'', the inter-experiment made the the $pd$ worse
and the $pf$ better:
 




%\section{Group A- one example: ``PC5''}
{\scriptsize
\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
 & min & q1 & median & q3 & max  \\\cline{2-7}
pd / inter &  69 &  73 &  75 &  77 &  83 & \boxplot{69}{4}{75}{77}{6}\\
pd / intra &  81 &  85 &  88 &  90 &  96 & \boxplot{81}{4}{88}{90}{6}\\
pf / inter &   4 &   5 &   5 &   5 &   6 & \boxplot{4}{1}{5}{5}{1}\\
pf / intra &  11 &  12 &  12 &  12 &  14 & \boxplot{11}{1}{12}{12}{2}\\
\cline{2-7}
\end{tabular}
}

~\\

\section{Incremental Studies}
\fig{inc}
shows the inter-experimental results as the size 
of the traiing set increased in size. Note two main visual patterns:
\bi
\item Increasing the training set size does little or nothing to the median $balance$ values.
\item In some data sets, increasing the training set size increased the variability (spread)
in the $balance$ values.
\ei
Mann-Whitney tests were applied to test the first visual pattern. 
Detectors learned from just $N$ instances do as well
as detectors learned from any other number of instances.

\bi
\item
N=100 for \{CM1,KC2,KC3,MC2,MW1,PC3,PC4\}
\item
N=200 for  \{KC1\}
\item
N=300 for \{PC1\}
\item
N=4,900 for \{PC2\} (but N=800 only lost 2 out of 53 trials)
\item
N=8,300 for \{MC1\} (but N=400 only lost once out of 92 trials)
\item
N=11,000 for \{PC5\} (but N=300 only lost twice out of 169 trials)
\ei

%\section{Group B- one example: ``KC3''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  60 &  80 &  80 &  80 & 100 & \boxplot{60}{20}{80}{80}{20}\\
%pd / intra &  40 &  60 &  80 &  80 & 100 & \boxplot{40}{20}{80}{80}{20}\\
%pf / inter &  14 &  19 &  24 &  31 &  38 & \boxplot{14}{5}{24}{31}{7}\\
%pf / intra &  10 &  17 &  21 &  26 &  36 & \boxplot{10}{7}{21}{26}{10}\\
%\cline{2-7}\end{tabular}
%}
%
%\section{Group C- two examples such as ``MC1''}
%{\scriptsize
%\begin{tabular}{r|r@{~}r@{~}r@{~}r@{~}r@{~}|l|}\cline{2-6}
% & min & q1 & median & q3 & max  \\\cline{2-7}
%pd / inter &  57 &  71 &  86 &  86 & 100 & \boxplot{57}{14}{86}{86}{14}\\
%pd / intra &  57 &  71 &  86 &  86 & 100 & \boxplot{57}{14}{86}{86}{14}\\
%pf / inter &  17 &  18 &  19 &  20 &  21 & \boxplot{17}{1}{19}{20}{1}\\
%pf / intra &  15 &  16 &  17 &  18 &  20 & \boxplot{15}{1}{17}{18}{2}\\
%\cline{2-7}\end{tabular}
%}
%

\begin{figure*}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{log/incrementalcm1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalkc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalmw1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalpc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc4.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc5.pdf}
\end{tabular}
\end{center}
\caption{Incremental Results}\label{fig:inc}
\end{figure*}


\begin{figure*}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{log/undercm1.pdf}&
\includegraphics[angle=90,width=2in]{log/underkc1.pdf}&
\includegraphics[angle=90,width=2in]{log/underkc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/underkc3.pdf}&
\includegraphics[angle=90,width=2in]{log/undermc1.pdf}&
\includegraphics[angle=90,width=2in]{log/undermc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/undermw1.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc1.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/underpc3.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc4.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc5.pdf}
\end{tabular}
\end{center}
\caption{Under-sampleing results}\label{fig:under}
\end{figure*}
\section{Under-sampling Studies}
\fig{under} shows the results of an under-sampling stduy where $N\in\{25,50,75,...\}$
defective modules were selected along with an equal $N$ number of defect-free modules.
Note the same visual pattern as before: increasing data does not necessarily improve $balance$.

Mann-Whitney tests were applied to test this visual pattern. 
Detectors learned from just $N$ instances do as well
as detectors learned from any other number of instances:

\bi
\item
N=25 for \{CM1,KC2,KC3,MC1,MC2,MW1,PC1,PC2\}
\item
N=75 for \{PC3\} (and N=25 losses three times out of 11).
\item
N=200 for \{PC4\} (but N=25 losses once out of 13 trials)
\item
N=575 for \{KC1\} (for N=25 only lost once out of 25 trials)
\item
N=1025 for \{PC5\} (but at N=25, the ltest lose only once our of 25 trials)
\ei


\bibliographystyle{IEEEtran}
\bibliography{refs}

%------------------------------------------------------------------------- 

\end{document}

