%\documentclass[onecolumn,draftcls,journal,compsoc]{IEEEtran}
%\documentclass[technote]{IEEEtran}
\documentclass{sig-alternate}
\usepackage{graphicx}
\usepackage{wrapfig,url,times}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		\put(0,0){\line(0,1){8}}
		\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{5cm}{5cm}\selectfont{Draft v5}}}}}
%\makeatother
%
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\title{Limits of Learning  Defect Predictors}

%\author{Tim~Menzies,~\IEEEmembership{Member,~IEEE,}
%Burak Turhan,
%		Gregory~Gay,
%       and~Ayse Bener\thanks{Dr. Menzies and Mr. Gay are with the Lane Department of Computer Science and Electrical Engineering, West Virginia
%University.
%Emails: \protect\url{tim@menzies.us},
%\protect\url{jdistefano@ismwv.com}.
%}
%\thanks{Mr. Turhan and Dr. Bener are with the Department of Computer Engineering, Bogazici University, Turkey.
%Emails:
%\protect\url{turhanb boun.edu.tr},
%\protect\url{bener@boun.edu.tr}.
%}\thanks{The research described in this paper
%was supported by
%Bogazici University research fund      under grant number BAP-06HA104
%and at
%West Virginia  University
%under grants with NASA's Software Assurance Research Program.
%Reference herein to any specific commercial product, process, or service
%by trademark, manufacturer, or otherwise, does not constitute or imply
%its endorsement by the United States Government.
%}\thanks{See 
%See \protect\url{http://menzies.us/pdf/overunder.pdf} for an earlier draft of this paper.
%Manuscript received October 1, 2007; revised XXX,}} 

\begin{document}

\conferenceinfo{ICSE PROMISE Workshop}{2008 Leipzig, Germany}
%\CopyrightYear{2008} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.

\title{
Implications of Ceiling Effects in Defect Predictors
%\titlenote{Submitted to the PROMISE 2008 workshop on repeatable SE experiments.
%%A draft version of this paper may be downloaded fro
%\url{http://menzies.us/pdf/08keys.pdf}
%}
}

\numberofauthors{2}
%
\author{
%% 1st. author
\alignauthor
Tim Menzies$^a$,Burak Turhan$^b$, \\Bojan Cukic$^a$\\ ~\\
       \affaddr{$^a$ Dept of CS\& EE}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, WV, USA}\\
       \email{tim@menzies.us}
       \email{greg@4colorrebellion.com}\\
	\email{yue, cukic@csee.wvu.edu} 
%%% 3rd. author
\alignauthor
Gregory Gay$^a$, Ayese Bener$^b$\\Yue Jiang$^a$ \\~\\
\affaddr{$^b$ Dept. of Computer Engineering}\\ 
\affaddr{Bogazici University, Turkey}\\ 
\email{turhanb@boun.edu.tr}\\
\email{bener@boun.edu.tr} 
}
%
\date{10 January 2008}

\maketitle
\begin{abstract}
There are many methods that
input static code features
and output a predictor for faulty code modules. 
The current generation of detectors are demonstrably useful
and better than current industrial best practice,
Nevertheless,
we observe
that these methods have hit a {\em performance ceiling};
i.e., some inherent upper bound on the amount of information offered
by, say, static code features when identifying modules which contain
faults.  
This paper documents that ceiling effect, and its implications
for research into data mining for defect predictors. 
\end{abstract}



\section{Introduction}

The current state of the art in learning defect predictors
is curiously static. As shown below,
better results have not been forthcoming,
despite the application of supposedly better
data miners.

If better algorithms are not useful, perhaps it is time
to better understand the training data.
Accordingly, instead of:
\bi
\item Keeping the 
{\em training data} constant and 
changing 
the {\em data miners}\ldots
\item This paper
keeps the {\em data miners} constant and 
 changes the {\em training 
data}.
\ei
Specifically, we study how {\em throwing training data away} 
effects the performance of our defect predictors.

The results are quite surprising.
 In experiments with
under/over-sampling
and incremental cross-validation,
we found that:
\bd
\item[Observation:]
Much of the training data can be discarded without
losing effectiveness in defect prediction.
\ed
This leads to the following notion:
\bd\item[Hypothesis:]
Static code features have {\em
limited
information content}.
\ed
This, in turn, leads to three predictions:
\bd
\item[Prediction1:]
The information from static code features can 
quickly and completely discovered by even simple learners.
\item[Prediction2:]
More sophisticated algorithms will discover no
further information.
\item[Prediction3:] Further progress in learning defect predictors
will come not from better algorithms, but from improving the information content of the training data.
\ed
The final discussion section of this paper discusses methods
for increasing the information in our training data.
It turns out that {\bf Prediction1} can be used to greatly
simplify the collection of more insightful training data.


\section{Background}
For some time now, we have applied data miners to 
build defect predictors from static code measures~\cite{%
chen05,%
dekhtyar04,%
jiang07,%
me00p,%
me02e,%
me02g,%
me03a,%
me03k,%
me03q,%
me04c,%
me04g,%
me07b,%
me07e
}.
\begin{figure}[!b]
\scriptsize
\begin{center}
\begin{tabular}{l|lrrr@{.}l}
     &          & (\# modules)         &          &  &\\
data & language & examples & features  & \multicolumn{2}{c}{\%defective}\\\hline
pc5  & C++ & 17,186   & 38          & 3&0\\
mc1  & C++ & 9,466    & 38          & 0&71\\
pc2  & C++ & 5,589    & 36          & 0&41 \\
pc3  & C++ & 1,563    & 37          & 10&23\\
pc4  & C & 1,458    & 37          & 12&2\\
pc1  & C++ & 1,109    & 21          & 6&94\\
kc1  & C++ & 845      & 21          & 15&45\\
kc2  & C++ & 522      & 21          & 20&49\\
cm1  & C++ & 498      & 21          & 9&83\\
kc3  & JAVA & 458      & 39          & 9&38\\
mw1  & C++ & 403      & 37          & 7&69\\
mc2  & C++ & 61      & 39          & 32&29\\\cline{3-3}
     \multicolumn{2}{c}{~}     &  39,158 

\end{tabular}
\end{center}
\caption{Twelve tables of data, sorted in order of number of examples.
All these tables are in the PROMISE
repository. }\label{fig:data}
\end{figure} 
To learn such predictors,
tables of historical
examples (like  those in \fig{data})
are formed where one column has a boolean value
for ``faults detected'' and the other columns describe software
features such as (i)~lines of code, (ii)~number of unique
symbols~\cite{halstead77}, or (iii)~max. number of possible
execution pathways~\cite{mccabe76}. 

Each row in the table holds data
from one ``module''; i.e. the  unit of functionality. Depending on
the language, modules may be called ``functions'', ``methods'',
``procedures'' or ``files''. 

The data mining task is to find
combinations of  features that predict for the value in the defects
column.  Once such combinations are found, managers can use them to
determine where to best focus their QA effort. Better yet, if they
have already focused their QA effort on the most critical portions
of the system, the detectors can ``nudge'' them towards areas that
need the most attention.

In theory, such defect predictors are not useful.
Fenton offers
an insightful example where {\em the same} functionality is achieved
using {\em different} programming language constructs resulting in
{\em different} static measurements for that module~\cite{fenton97}.

In practice, they are quite effective, at least for the NASA code we have
studied~\cite{me07e,me07b,me04g,me04c,me03q,me03k,me02l,me02g,me02e,jiang07}.
In those data sets,
our fault prediction models find defect
predictors~\cite{me07b} with a probability of detection ($pd$) and
probability of false alarm ($pf$) of $mean(pd,pf)=(71\%,25\%)$.
These values are much higher then known industrial averages
for manual defect detection~\cite{fagan86, shu02}.

In 
January 2007, we published a study~\cite{me07b} that
defined a
repeatable experiment in learning defect  predictors. The intent of
that work was to offer a benchmark in defect prediction that other
researchers could repeat/ improve/ refute. That experiment used: \bi
\item Public domain data sets (from the the PROMISE
repository;
\item
Open source data mining
tools (the WEKA toolkit~\cite{witten05});
\item
Randomly sorting 
training data rows (stops
order effects);
\item
10-way cross-validation (to test on data {\em not} used in training);
\item
Learning via multiple types of machine learning algorithms (rule
learners, decision tree learners, Bayes classifiers);
\item
Assessment via multiple criteria such as probability of detection ($pd$),
probability of false alarm ($pf$), and $balance$ that 
combines $\{pd,pf\}$ ($balance$ is defined in \fig{performance});
\item
Statistical hypothesis tests over the assessment criteria;
\item
Novel visualization methods for the results;
\item
Feature subset selection to find the most important subset of the static code
features;
\ei
\begin{figure}[!b]
\begin{center}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms and 100\% detection.

Other measures such as $accuracy$  and $precision$
were not used since, as shown in \fig{data}, the percent of defective examples
in our tables was usually very small (median value around 8\%).
Accuracy and precision
are poor indicators
of performance for data were the target class is so rare
(for more on this issue, see~\cite{me07b,me07e}). \\\hline
\end{tabular}
\end{center}
\caption{Performance measures.}\label{fig:measures}
\end{figure}

Surprisingly, very simple Bayes classifiers (with a simple
pre-processor for the numerics)
out-performed the other studied methods.
Since that study, we have tried to find  better data mining
algorithms for defect prediction. To date, we have failed. Our
recent (as yet, unpublished) experiments have found no additional
statistically significant improvement from the application of the
following data mining methods: logistic regression; average
one-dependence estimators~\cite{webb05}; under- or
over-sampling~\cite{drummond03} random forests~\cite{breiman01},
RIPPER~\cite{cohen95r}, J48~\cite{quinlan92}, OneR~\cite{holte93}
and bagging~\cite{brieman96}. Only boosting~\cite{FreSch97} on
discretized data offers a statistically better result than a Bayes
classifier. However, we cannot recommend boosting: the median
improvement is quite negligible and  boosting is orders of
magnitudes slower than a simple Bayes classifier.

Other researchers have also failed to improve our results. Due to
our connection with PROMISE,
we are aware of studies
by other researchers (currently under review) that tried 
other
data mining methods. Those studies
investigated the statistical difference of the results from two
dozen
learners on the same datasets.
The simple Bayesian method discussed above ties in
first place along with 15 other methods.


%We now we ask for a halt to that line of research.
%Better defect predictors will arise, we will argue,
%not from better algorithms.
%We
%Exponential cost increase quickly exhausts finite QA resources.
%Hence, blind spots can't be avoided and must be managed. Standard
%practice is to apply the best available assessment methods on the
%sections of the program that the best available domain knowledge
%declares is the most critical. However, this focus on
%certain sections can blind us to defects in other areas which,
%through interactions, may cause similarly critical failures.
%Therefore, the standard practice should be augmented with a {\em
%lightweight sampling policy} that (a)~explores the rest of the
%software and (b)~raises an alert on parts of the software that
%appear problematic.  This sampling approach is incomplete by
%definition. Nevertheless, it is the only option when 
%resource limits block complete assessment.
%
%
%One such lightweight sampling policy is the
%use of {\em fault predictors}. To learn such prediction models,
%either from projects previously developed in the same environment or
%from a continually expanding base of current project's artifacts,
%tables of examples are formed where one column has a boolean value
%for ``faults detected'' and the other columns describe software
%features such as (i)~lines of code, (ii)~number of unique
%symbols~\cite{halstead77}, or (iii)~max. number of possible
%execution pathways~\cite{mccabe76}. Each row in the table holds data
%from one ``module''; i.e. the  unit of functionality. Depending on
%the language, modules may be called ``functions'', ``methods'',
%``procedures'' or ``files''. The data mining task is to find
%combinations of  features that predict for the value in the defects
%column.  Once such combinations are found, managers can use them to
%determine where to best focus their QA effort. Better yet, if they
%have already focused their QA effort on the most critical portions
%of the system, the detectors can ``nudge'' them towards areas that
%are also in need of quality improvement.
%
%
%For many years we relied upon straightforward application of data mining algorithms
% to learn quality predictors
%from the artifacts  generated by a software project; see  
%\cite{%
%chen05,%
%come96,%
%dekhtyar04,%
%fea02a,%
%hameph95,%
%hame93,%
%hame94,%
%jiang07,%
%lum06,%
%me00e,%
%me00p,%
%me00u,%
%me02c,%
%me02f,%
%me02g,%
%me02k,%
%me02l,%
%me02n,%
%me03c,%
%me03h,%
%me03i,%
%me03j,%
%me03k,%
%me03n,%
%me03q,%
%me03r,%
%me04a,%
%me04c,%
%me04g,%
%me04h,%
%me05a,%
%me05c,%
%me05d,%
%me06a,%
%me06d,%
%me06e,%
%me06f,%
%me07b,%
%me07e,%
%me07f,%
%Ramakrishnan1995,%
%RamMe1996%
%}.
%
%
%Standard verification and validation (V\&V) textbooks
%\cite{rakitin01} advise using static code complexity attributes to
%decide which modules are worthy of manual inspections. For several
%years, the PIs have worked on-site at the NASA Independent software
%Verification and Validation facility where large government software
%contractors won't review software modules {\em unless} tools like
%the McCabe static source code analyzer predicts that they exhibit
%high code complexity measures.
%
%Nevertheless, static code attributes are hardly a complete
%characterization of the internals of a program module. Fenton offers
%an insightful example where {\em the same} functionality is achieved
%using {\em different} programming language constructs resulting in
%{\em different} static measurements for that module~\cite{fenton97}.
%Fenton uses this example to argue the uselessness of static code
%attributes for fault prediction.
%
%An {\em alternative interpretation} of Fenton's example is that
%static attributes can never be a certain indicator of the presence
%of a  fault.  Nevertheless, they are useful as probabilistic
%statements that the frequency of faults tends to increase in code
%modules that trigger the predictor.
%
%Shepperd~\&~Ince and Fenton~\&~Pfleeger might reject the {\em
%alternative interpretation}. They present empirical evidence  that
%the McCabe static attributes offer nothing more than uninformative
%attributes that are mostly the derivatives of the lines of code.
%Fenton~\&~Pfleeger note that the main McCabe's attribute (cyclomatic
%complexity, or $v(g)$) is highly correlated with lines of
%code~\cite{fenton97}. Also, Shepperd~\&~Ince remark that ``for a
%large class of software it (cyclomatic complexity) is no more than a
%proxy for, and in many cases outperformed by, lines of
%code''~\cite{shepperd94}.
%
%If Shepperd \& Ince and Fenton \& Pfleeger are right, then
%%\bi\item
%%Learning from just lines of code should do as well as anything else;
%%\item
%the performance of  predictors learned by  data mining static code
%features should be poor. However, this is not true, at least for the
%NASA code we have
%studied~\cite{me07e,me07b,me04g,me04c,me03q,me03k,me02l,me02g,me02e,jiang07}.
%Using NASA data, our fault prediction models find defect
%predictors~\cite{me07b} with a probability of detection ($pd$) and
%probability of false alarm ($pf$) of $mean(pd,pf)=(71\%,25\%)$.
%These values can be best understood via a comparison against known
%industrial baselines \cite{fagan86, shu02}.
%
%The research work referenced above has produced useful results.
%However, there is much room for improvement; e.g. probabilities of
%detection reported in recent repeatable studies, should not only be
%achievable on a regular basis but improved.
%
%It has proved difficult to achieve those improvements. In our
%January 2007 TSE study~\cite{me07b} we set out to define a
%repeatable experiment in learning defect  predictors. The intent of
%that work was to offer a benchmark in defect prediction that other
%researchers could repeat/ improve/ refute. That experiment included:
%\bi
%\item
%Public domain data sets (from the the PROMISE
%repository\footnote{\url{http://promisedata.org/repository}});
%\item
%Open source data mining
%tools (the WEKA toolkit~\cite{witten05});
%\item
%Repeated  randomization of the  order of training data (to avoid
%order effects);
%\item
%10-way cross-validation (to assess the results via data {\em not} used in training);
%\item
%Learning via multiple types of machine learning algorithms (rule
%learners, decision tree learners, Bayes classifiers);
%\item
%Assessment via multiple criteria such as probability of detection ($pd$),
%probability of false alarm ($pf$), and $balance$ that combines $\{pd,pf\}$ ;
%\item
%Statistical hypothesis tests over the assessment criteria;
%\item
%Novel visualization methods for the results;
%\item
%Feature subset selection to find the most important subset of the static code
%features;
%\ei
%
%Since that study, we have tried to find  better data mining
%algorithms for defect prediction. To date, we have failed. Our
%recent (as yet, unpublished) experiments have found no additional
%statistically significant improvement from the application of the
%following data mining methods: logistic regression; average
%one-dependence estimators~\cite{webb05}; under- or
%over-sampling~\cite{drummond03} random forests~\cite{breiman01},
%RIPPER~\cite{cohen95r}, J48~\cite{quinlan92}, OneR~\cite{holte93}
%and bagging~\cite{brieman96}. Only boosting~\cite{FreSch97} on
%discretized data offers a statistically better result than a Bayes
%classifier. However, we cannot recommend boosting: the median
%improvement is quite negligible and  boosting is orders of
%magnitudes slower than a simple Bayes classifier.
%
%Other researchers have also failed to improve our results. For the
%past four years, we participated in the PROMISE workshop on
%repeatable software engineering experiments. The rule of that
%workshop requires that if  a paper offers an empirical conclusion,
%then it must also offer the data used to reach the conclusions. That
%data is stored on-line in the PROMISE repository, which we
%administer and maintain for the PROMISE community. Our work on that
%web site, plus our interaction with the PROMISE community, gives us
%a unique insight into the empirical analysis as well as access to,
%as yet, unpublished results. Consequently, we are aware of studies
%by other researchers (currently under review) that tried 25 other
%data mining methods in fault prediction experiments. Those studies
%investigated the statistical difference of the results from 25
%learners on the same datasets using the Wilcoxon signed ranked
%test~\cite{wilcoxon45}. The Bayesian method discussed above ties in
%first place along with 15 other methods.
%

How can we explain
all these failed attempts to improve fault
prediction models in repeatable experiments using same (PROMISE)
data sets? 
One lesson from the above is that exploring better
algorithms may not be productive. Hence, we explored the data that
the algorithms were processing.

Elsewhere~\cite{me07i} we 
checked how {\em little}
information was required to learn a defect predictor. 
 Defect predictors were learned from
\mbox{$N\in\{100, 200, 300,...\}$} instances  then 
$Test$ed on another 100
instances. For each $N$, 10 experiments were performed where
training was conducted on $Train=90\%*N$ instances. For all
experiments, the $N,Train,Test$ instances were selected at random.

\begin{figure}[!t]
\begin{center}
\includegraphics[angle=90,width=2in]{incrementalPC2.pdf}
\includegraphics[angle=90,width=2in]{incrementalKC1.pdf}
\includegraphics[angle=90,width=2in]{incrementalPC5.pdf}
\includegraphics[angle=90,width=2in]{incrementalMC1.pdf}
\end{center}
\caption{Experiments with training set size vs $balance$.
}\label{fig:tone}
\end{figure}
This study  was conducted on the
twelve data sets of\fig{data}. Space does not permit
showing all the results but a representative sample
are shown in 
\fig{tone}~\cite{me07i}. In that
figure, the X-axis is the size of training set and the  Y-axis is
the $balance$ measure defined in \fig{measures}.
%All results come from a hold-out set of 100 instances.
Note that the performance does not change much regardless of whether
the model is inferred from 100 instances or from up to several
thousand instances. In fact, learning from too many training
examples was actually detrimental (witness the widening variance as
the training set increases). A Mann Whitney U test~\cite{mann47}
(95\% confidence) confirms the visual pattern apparent in
\fig{tone}: static code features used as the basis for predicting
module's fault content reveal all that they can reveal after as
little as 100 instances.

The above results motivated new experiments, reported below.

\section{Experiments}
\noindent
The goal of these experiments was two-fold:
\be
\item Can we confirm the  effects 
of \fig{tone}; i.e. that ignore large amounts of the training
data is not harmful?
\item
Can we exploit that effect to build better defect predictors?
\ee
Goal \#1 was achieved and we have
a promising lead  on Goal \#2.

\subsection{Experiment \#1: Over- \& Under- Sampling}

The \fig{tone} experiment randomly discarded training data.
Perhaps a 
more controlled sub-sampling methods
would not damage the information content of the data.
If so then, contrary to \fig{tone}, increasing the sample size
will improve performance.

Over- and under-sampling~\cite{drummond03,yun04} are examples
of better controlled sub-sampling.
Both methods have the potential
to improve the results of a learner in data sets with highly
imbalances class frequencies. For example,
in \fig{data},
 the frequency of the target class is very low
(median value  lies between
6.94\% and 7.69\%).

To sub-sample, a target class if selected; in this study,
we selected made defective modules the target class.
The sampling program runs in two passes through the data. In pass 1,
a table of frequency counts is built. If the desired
goal is reached in fewer instances than the other classes, a second
pass is taken through the data. This is where the two sampling
techniques differ:
\bi
\item In the case of {\em under-sampling}, instances are
selected until the combined number of instances with other classes
is equal to the number of instances with the desired goal. This
results in a much smaller dataset, but the desired goal is no
longer buried inside a larger set of other classes.
\item
{\em Over-sampling} follows a similar procedure but,  instead of
limiting the number of instances with other classes, the sampling
program instead add new instances with your desired goal. Ultimately,
you have a much larger dataset. We have used a fairly simple sampling
method. For both sampling techniques, half of the final dataset
will consist of instances with the desired goal.  
\ei


%However, while a promising technique, there is contradictory evidence
%for its value.  Yiu reports that the appropriate re-sampling technique
%and appropriate classifier is dataset dependent~\cite{yun04}.  In
%limited studies with one learner and a few data sets, Drummond \&
%Holte~\cite{drummond03} found that oversampling had little value.
%They offer some evidence for the value of under-sampling but concluded
%that other methods can do better.  One issue is that while re-sampling
%yields training set contains $neg/pos=1$, the test set still has
%the original distributions.  In any case, this area is ripe for
%further exploration.  
%

%\subsection{Related Work}

%Holte and Drummond \cite{drummond03} are responsible for some of the best research in the area of under and over-sampling. Their 2003 paper explored a very similar situation. They took a look at the effects of sampling with the C4.5 tree learner. Rather than using balance, they used their own “cost curve” measurement \cite{drummond00a}. They found that under-sampling produced excellent results,  it was sensitive to changes in misclassification costs and class distribution. On the other hand, over-sampling produced little to no change in the results.  Over-sampling could be made cost-sensitive if pruning and early stopping parameters were set in proportion to the amount of oversampling to be done. This is, however, computational costly and completely unnecessary as under-sampling removes and need for pruning at no additional cost. Under-sampling does have some problems of its own. For instance, under-sampling introduces non-determinism into the previously deterministic learning process. This variance could make a difference in some research. They ultimately conclude that C4.5 with under-sampling is a good baseline for cost sensitivity research. 

%Drummond and Holte are hardly the only ones to explore this space. C4.5 combined with a sampling policy has become something of a baseline in cost-sensitivity research \cite{domingos99, pazzani94}. Kate McCarthy, Bibi Zabar and Gary Weiss, in their 2005 paper \cite{mccarthy05}, examined whether under-sampling, over-sampling, or cost-sensitive learning yielded the best results. Their goal was to show that cost-sensitive learning beat sampling methods on the C5.0 learner \cite{rulequest} when trying to find total cost (Total Cost = (False Negatives × CFN) + (False Positives × CFP) \cite{zadrozny04}). Their research found no clear winner. When analyzing smaller datasets, they found that over-sampling beat both under-sampling and cost-sensitive learning on all but one dataset. On larger sets, cost-sensitive learning narrowly beat over-sampling. They explain this difference by hypothesizing that cost-sensitive learning performs better with a large amount of training data. 

%\input{timm/challenges.tex}
 
Over- and under-sampling experiments were conducted on the \fig{data}
data using  10*10-way cross-validation\footnote{Ten times, randomize the order of the instances
in the data. Each time, divide data into $i$ bins. For each bin
$j\in1..i$.
Let $test$ be bin $j$ and let $train$ be the remaining bins.
Learn on $train$ then
evaluate on $test$. }.
This studied used a simple Bayes classifier 
(since it was 
useful in our prior experiment~\cite{me07b}) 
plus a C4.5-like decision tree learner,
J4.8~\cite{witten05} (since that was 
used in prior under- and over- sampling experiments~\cite{drummond03}).

The $balance$
results (defined in \fig{measures}
for each $treatment $ (data miner, plus sampling policy)
were sorted and displayed as {\em
quartile charts} of \fig{bands}.  
To generate these charts, the $balance$ results
for some treatment are sorted and labeled as follows:
\[
\footnotesize
\overbrace{\underbrace{4}_{min},7,15}^{q1},20,31,\underbrace{40}_{median},52, 64, \overbrace{70,81,\underbrace{90}_{max}}^{q4}
\]
In a quartile chart,
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
a vertical bar  is added to
mark the 50\%  value.
The above numbers
would therefore be drawn as follows:

{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}
We prefer quartile charts of performance deltas to other summarization
methods for M*N studies. They offer a very succinct summary
of a large number of experiments. 


\begin{figure}[!t]
\scriptsize
\begin{center}
\begin{tabular}{c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}c}
\multicolumn{1}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{2-6}
&min& & median & &max\\\cline{2-6}
treatment&0& 25\%& 50\% & 75\%&100\%\\\cline{1-6}
NB/ none & 21.9& 67.7& 74.6& 81.9&100.0& \boxplot{21.9}{45.8}{74.6}{81.9}{18.1}\\ 
NB/ under&19.9& 67.1& 74.1& 81.6&100.0&  \boxplot{19.9}{47.2}{74.1}{81.6}{18.4}  \\
J48/ under& 21.6& 64.8& 73.6& 82.6&100.0 & \boxplot{21.6}{43.2}{73.6}{82.6}{17.4}\\ 
NB/ over  &17.5& 42.0& 62.5& 72.2&100.0  & \boxplot{17.5}{24.5}{62.5}{72.2}{27.8} \\
J48/ over & 0.0& 29.3& 45.6& 56.2&100.0&  \boxplot{0}{29.3}{45.6}{56.2}{43.8}\\
J48/ none & 0.0& 29.3& 42.3& 54.5&100.0& \boxplot{0}{29.3}{42.3}{54.5}{45.5}\\\cline{2-6}
\end{tabular}
\end{center}
\caption{Over- \& under- \& no sampling results.
Sorted descending by median $balance$ results.
Visually, there is clear loser (J48/none) and
a three-way  tie for the winning method (NB/none, NB/under, J48/under).
These visual impression are confirmed by the statistical
tests  of \fig{stats}.}\label{fig:bands}
\end{figure}


\begin{figure}\small
\begin{center} 
\begin{tabular}{r|r|rrr}
learner / sampler& win - loss & win & loss & ties\\\hline
NB / none &3& 3 & 0& 2\\
NB / under&3& 3 & 0 & 2\\
J48 / under&3&3&0&2\\
NB / over &-1& 2 & 3 & 0\\
J48 / over & -3&1&4&0\\
J48 / none &-5& 0&5&0
\end{tabular}
\end{center}
\caption{Statistical tests on the \fig{bands} results.
Mann-Whitney~\cite{mann47} compared the median ranks of all the $balance$
values seen in \fig{bands}. Two treatments tied if their median ranks were
statistically insignificantly different (95\% confidence). Otherwise,
medians were compared numerically to assign ``win'' or ``loss''.
Table sorted descending  on total number of  $win-loss$ for
each treatment against the other five.
}
\label{fig:stats} \end{figure}
\begin{figure*}[!t]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2.1in]{log/underCM1.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underKC1.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underKC2.pdf}\\
\includegraphics[angle=90,width=2.1in]{log/underKC3.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underMC1.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underMC2.pdf}\\
\includegraphics[angle=90,width=2.1in]{log/underMW1.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underPC1.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underPC2.pdf}\\
\includegraphics[angle=90,width=2.1in]{log/underPC3.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underPC4.pdf}&
\includegraphics[angle=90,width=2.1in]{log/underPC5.pdf}
\end{tabular}
\end{center}
\caption{Micro-sampling results}\label{fig:under}
\end{figure*}

\newpage
\noindent
\fig{bands} is consistent with some prior results:
\bi
\item
The simple Naive Bayes we recommended previously~\cite{me07b}
 performed as well as anything else.
In a result consistent with the limited information hypothesis discussed in the introduction,
seemingly cleverer learning schemes did not out-perform simple Bayesian classifiers.
\item
Just like the \fig{tone} results,
throwing away data (i.e. under-sampling) 
does not degrade the performance of the learner. 
In fact, in the case of J48, throwing away data
 improved the median $balance$
performance from around 40\% to over 70\%.
\item
Under-sampling beat over-sampling for both J48 and Naive Bayes.
This result is
consistent with Drummond \& Holte's sub-sampling experiments~\cite{drummond03}.
\ei
However, \fig{bands} has some new results which, as far as we are aware, have not been reported
elsewhere:
\bi
\item
Observe how $NB/none$ is one of the topped-ranked methods. That is,
sub-sampling  decision tree learning does not out-perform Naive Bayes.
\item 
$NB/none$ ties with $NB/under$.
That is, while
sub-sampling offers no improvement over un-sampled Bayesian learning, 
under-sampling does not harm classifier  performance,
\ei


This last point is most significant.
It means effective detectors can be learned
from a very small sample of the available data- an issue we explore below.
\newpage\subsection{Experiment \#2: Micro-sampling}

There exist numerous incremental case-based reasoning tools~\cite{kol91,kol93,compton05}
that ask humans to audit a stochastic sample
of real-world cases. Insights gained from those sessions are 
automatically
generalized and applied to another random sample, Experts then review the classifications made
on the new sample, and offer further refinements.

When is case-based reasoning preferable to automatic data mining? 
There are many answers to this question, only one of which is explored in this paper.
Case-based reasoning methods require humans to examine and comment on
specific cases.
This is impractical if learning adequate 
theories requires examining a very large number of cases.

The results of experiment \#1 suggest that, for static code measures,
it is not necessary to manually inspect thousands of cases. In fact, just a few hundred may suffice.
Consider
the 9,466 modules of MC1. This data set has a defective
module rate of  0.71\% ; i.e.  9.466*0.71/100=67 modules.
When under-sampling, a data set of 134 modules is created (all the
defective, plus 67 others, selected at random).  

In order to determine the lower-limit on the number of cases that require manual
inspection, we performed
another experiment. Another under-sampling policy was employed, which we call {\em micro-sampling}.
 Given $N$ defective modules
in a data set, \[M \in \{25,50,75,.\} \le N\] 
defective modules were selected
at random. Another $M$ non-defective modules were selected, at random. The combined $2M$ data set was
then passed to a 10*10-way cross validation. 

Formally, under-sampling is a special case of micro-sampling where $M=N$. However, micro-sampling explores
training sets of size up to N, while standard under-sampling just explores once data set of
size $2N$.

For this study we used Naive Bayes since,
in all the above work, it seems to be doing as well as anything else.
\fig{under} shows the results of an under-sampling study where $M\in\{25,50,75,...\}$
defective modules were selected at random, along with an equal $M$ number of defect-free modules.
Note the same visual pattern as before: increasing data does not necessarily improve $balance$.

Mann-Whitney tests were applied to test this visual pattern. 
Detectors learned from small $M$ instances do as well
as detectors learned from any other number of instances.
\bi
\item
For eight data sets,
\{CM1,KC2, KC3,MC1, MC2, MW1, PC1, PC2\},
micro-sampling at $M=25$ did just as well as anything larger sample size.
\item
For one data sets,
\{PC3\}, best results were seen at $M=75$. However , in many cases $\frac{8}{11}$,
$M=25$ did just as well as anything else.
\item
For three data sets \{PC4,KC1,PC5\} best results were seen at $M=\{200,575,1025\}$, respectively. 
However, for these three data sets, in all by one case
$M=25$ did as well as any larger value.
\ei

In summary, case-based reasoning is a viable option for building defect predictors from static code
features. The total number of cases that a human would have to inspect can be allow as 50 randomly
selected modules (25 defective
and 25 non-defective).
%
%\section{How much data is required?}
%\begin{figure*}[!t]
%\begin{center}
%\begin{tabular}{ccc}
%\includegraphics[angle=90,width=2in]{log/incrementalCM1.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalKC1.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalKC2.pdf}\\
%\includegraphics[angle=90,width=2in]{log/incrementalKC3.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalMC1.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalMC2.pdf}\\
%\includegraphics[angle=90,width=2in]{log/incrementalMW1.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalPC1.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalPC2.pdf}\\
%\includegraphics[angle=90,width=2in]{log/incrementalPC3.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalPC4.pdf}&
%\includegraphics[angle=90,width=2in]{log/incrementalPC5.pdf}
%\end{tabular}
%\end{center}
%\caption{Results from experiment \#2. Training set size grows in units of 100 examples,
% moving left to right
%over the x-axis. The MC2 results only appear at the maximum x-value since MC2 
%has less than 200 examples.
%}\label{fig:inc}
%\end{figure*}
%
%
%
%\subsection{Design}
%A curious aspect of the above results is that defect predictors
%were learned using only a handful of defective modules.
%For example, consider a 90\%/10\% train/test split on
%$pc1$ with 1,109 modules, only 6.94\% of which are defective.
%On average, the test set will only contain \mbox{$1109 * 0.9 * 6.94/100=69$}
%defective modules. Despite this, $pc1$ yields an adequate median $\{pd,pf\}$ results
%of $\{88,34.5\}$\%. 
%
%Experiment \#2 was therefore designed to check how {\em little} data is required
%to learn defect predictors. Experiment \#2 was essentially the same
%as the last experiment, but
%without treatment \#1 (the
%cross-company study). Instead,
%experiment \#2
%took the 12 example tables
%and 
%learned predictors using:
%\bi
%\item Treatment 3 (reduced WC):  a randomly selected
%subset of the 90\% rows of this table;
%\ei
%Specially, after randomizing the order of the rows, training sets
%were built using just the first 100,200,300,\ldots rows in the tables.
%After training, the learned theory was applied to  100 rows
%not used in training (selected at random).
%
%Experiment \#1 only used the features found in all tables of data.
%For this experiment, we imposed no such restrictions and used whatever features
%were available in each data set.
%\subsection{Results from Experiment \#2}
%Recall that "balance'' is defined to be a combination of $\{pd,pf\}$
%that decreases if $pd$ decreases or $pf$ increases.
%As shown in \fig{inc}, there was very little change in balanced 
%performance  after learning from 100,200,300,... examples.
%Indeed, there is some evidence that
%learning from larger training sets had detrimental effects: the more
%training data, the larger the variance in the performance of the learned predictor.
%Observe how, in $pc2$, as the training set size increases (moving right along the x-axis)
%the dots showing the on balance performance start spreading out.
%A similar, but smaller, {\em spread effect} can be see in $kc2$ and $mc1$.
%
%The Mann-Whitney U test was applied to check the visual trends seen in \fig{inc}.
%For each table, all results from training sets of size 100,200,300\ldots were
%compared to all other results from the same table. The issue was ``how much data is enough?''
% i.e.
%what is the $min$imum
%training set size that never lost to other training set of a larger size. Usually, that 
%$min$ value 
%was quite small:
%
%\bi
%\item In seven tables $\{cm1,kc2,kc3,mw1,pc3,pc4\}$, $min=100$;
%\item In  \{$kc1,pc1$\}, $min=\{200,300\}$ instances, respectively.
%\ei
%
%In other tables of data, $min$ was much larger. In $\{pc2,mc1,pc5\}$ the 
%$min$ values were found at $\{4900, 8300, 11000\}$, respectively.  However, much smaller
%training set sizes performed nearly as well as these larger training sets:
%\bi
%\item
%In $pc5$, predictors learned from 300 examples only lost to other sizes 
%twice in 169 trials;
%\item
%In $mc1$, predictors learned from 400 examples only lost to other sizes
%once out of 92 trials);
%\item In $pc2$, predictors learned from 800 examples only
%lost to other size twice out of 53 trials.
%\ei
%
%We explain the experiment \#2 results as follows.
%These experiments used simplistic static code features such as lines of code, number of
%unique symbols in the module, etc.  
%Such simplistic static code features are hardly a complete characterization of
%the internals of a function. Fenton offers an insightful example where
%{\em the same} functionality is achieved using {\em different}
%programming language constructs resulting in {\em different} static
%measurements for that module~\cite{fenton97}.  
%We would characterize such static code features as having 
%{\em limited information content}. 
%Limited content
%is soon exhausted by repeated sampling.
%Hence, such simple features reveal all they can reveal after
%a small sample
%
%\begin{figure}
%\begin{center} \includegraphics[width=2.4in]{plateaus/plateaudots.pdf}
%\includegraphics[width=2.4in]{plateaus/plateaudotsC.pdf} \end{center}
%\caption{Y-axis shows plateau point after learning from data sets
%that have up to $X$ examples (from~\cite{orrego04}).
%The top plot shows results from using
%Naive Bayes (nbk) or a decision tree learner (j48)~\cite{quinlan92}
%to predict for discrete classes. Bottom plot shows results from
%using linear regression (lsr) or model trees (m5)~\cite{quinlan92b}
%to learn predictors for continuous classes. In this study, data
%sets were drawn from the UC Irvine data repository~\cite{Blake+Merz:1998}.
%}\label{fig:plateau} \end{figure}
%
%\subsection{Sanity Checks on Experiment \#2}
%This section checks for precedents on the Experiment \#2 results
%and can be skipped at first reading of this paper.
%
%There is also some evidence that the  results of Experiment 2 (that
%performance improvements stop after a few hundred examples) has
%been seen previously in the data mining literature (caveat: to the
%best of our knowledge, this is first report of this effect in the
%defect prediction literature): \bi \item In their discussion on how
%to best handle numeric features, Langley and John offers plots of
%the accuracy of Naive Bayes classifiers after learning on 10,20,40,..200
%examples. In those plots, there is little change in performance
%after 100 instances~\cite{john95}.  \item Orrego~\cite{orrego04}
%applied four data miners (including Naive Bayes) to  20 data sets
%to find the {\em plateau point}: i.e. the point after which there
%was little net change in the performance of the data miner.  To
%find the plateau point, Oreggo used t-tests to compare the results
%of learning from $Y$ or $Y+\Delta$ examples. If, in  a 10-way
%cross-validation, there was no statistical difference between $Y$
%and $Y+\Delta$, the plateau point was set to $Y$.  As shown in
%\fig{plateau}, many of those plateaus where found at $Y\le100$ and
%most were found at $Y\le200$. Note that these plateau sizes are
%consistent with the results of Experiment 2.  \ei
%
\section{Discussion}

This paper advocates for a paradigm change for research into learning defect
predictors from static code measures. Despite much effort, our current generation
of data miners have struck  ceiling effect where supposedly more sophisticated
data miners are not improving the results seen with older and simpler classifiers
such as Naive Bayes. 

Rather than explore more algorithms, here we have explored
the data being processed by the algorithms. Two plateau effects were reported:
\bi
\item After applying under- or over- or no sub-sampling, the efficacy of the
data miners did not improve after 100 examples.
\item After applying micro-sampling, the efficacy of the data miners usally
did not improve after 50 examples.
\ei
When the required data sets are so small, it is valid to ask if automatic methods
can be augmented with human-in-the-loop methods. Elsewhere, we have explored
combining static code measures with other measures that, serendipitously,
a particular domain may
contain. For example, at ISSRE 2007, we reported experiments where 
static code measures were combined with results from an ultra-lightweight text miner.
The results are spectacular, and 
shown in \fig{ttwo}. 
\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.3in]{pc1r.png}
\includegraphics[width=2.3in]{cm1r.png}
\end{center}
\caption{$\{pd,pf\}$ curves seen when using
code and/or requirements features.
Results from
CM1 and PC1 after a 10-way cross validation.
% when modules were considered defective if $\frac{M}{N}$ of the 10-ways
%classified a module as ``defective''. The curves in those plots
%result from varying $M$ from one to ten.
The ideal spot on these ROC curves is top left; i.e. no false alarms
and perfect detection ($\{pd,pf\}=\{1,0\}$). The dashed lines on
those plots show $\{pd,pf\}$ results when fault prediction models
used features mined from requirements text,
or features mined from static code measures (in isolation).
The solid lines
show the results of models which used these two kinds of features in
combination. Note the remarkable improvement when learning was applied
to both sets of features. 
From~\cite{jiang07}.}\label{fig:ttwo}
\end{figure}

The point of \fig{ttwo} is not ``always augment static code features
with requirements features''. This is impractical advice since not all domains
allow data mining specialists access to source code and the requirements that
produced them (e.g. very few open source projects have detailed textual requirements
documents).

Rather, the lesson of \fig{ttwo} is that it can be very useful
to let experts 
access and combine features from whatever sources are locally available. Such
an ``explore whatever'' environment is not an automatic black box data miner.
Rather, it is a human-in-the-loop case-based reasoning (CBR) environment where
humans reflect on the specifics of particular cases, connect to different data sources,
and (sometimes) run automatic data miners combinations or subsets of a variety
of types of features. 


\section{Conclusions}
The development of fault prediction
models has been a very active research area.  The reason for such
a significant attention to automated quality predictors lays in
their practical importance.  Current models are useful, as they
allow software project managers to better guide the allocation
usually meager quality assurance resources to artifacts which need
them the most.  

Recent results now indicate that
this
current research paradigm, which relied on relatively straightforward
application of machine learning tools, has reached its
limits.  Building software quality predictors via data mining is
essentially an inductive generalization over past experience.
According to Mitchell's classic model of data mining~\cite{mitchell97}, any
inductive generalization explores a version space of possible
theories. All data miners hit a performance ceiling effect when
they cannot find additional information that better relates software
measure with fault occurrence.

To build better quality predictors that break through ceiling
effects, we must introduce more topology into the search space.
Standard machine learning algorithms lack the business knowledge
which characterizes software projects. To add that business knowledge,
we propose 
human-in-the-loop CBR tools.
We expect that this approach will allow software managers to safely
focus on the application of quality assurance techniques of choice,
confident that automated quality predictors will raise alerts about
the artifacts where quality issues actually exist.



Three results make us advocate this kind of case-based reasoning environment:

\bi
\item The environment would impractical
if  the human operator must examine a very large number of cases. The micro-sampling
results suggest that this might not be the case. Indeed, as few as 50 randomly
selected instances might suffice.
\item The environment would be very fast to run. If 50 instances are enough,
then training should be virtually instantaneous.  A human user could explore
a very large number of features or combinations of features, all the while
getting very rapid feedback.
\item The environment, or some other new direction, is required. As argued
above, our current generation  of AI algorithms  have hit a ceiling effect.
We doubt that further progress will be made using better algorithms. Instead,
we would advocate methods (like the CBR tool briefly described above),
that increase the information content of the training
data. 
\ei



\bibliographystyle{IEEEtran}
\bibliography{refs}

\end{document}

To implement {\em test two}, we linked code modules to the
requirements that prompted their development. Features were
extracted from the requirements using a lightweight linear-time text
parser. Those features included {\em weak words} such as ``TBD'' and
``but not limited to''; {\em imperatives} such as ``must'',
``should''; and {\em options} such as ``can'', and ``may''. Training
instances for the fault prediction models were then created using
just static code features, just requirements features, or both
(using a database inner join operation to connect requirements with
modules in which they have been implemented).

The results are shown in \fig{ttwo}. This figure  plots $pd$ vs $pf$
seen in the two data sets after a 10-way cross validation.
% when modules were considered defective if $\frac{M}{N}$ of the 10-ways
%classified a module as ``defective''. The curves in those plots
%result from varying $M$ from one to ten.
The ideal spot on these ROC curves is top left; i.e. no false alarms
and perfect detection ($\{pd,pf\}=\{1,0\}$). The dashed lines on
those plots show $\{pd,pf\}$ results when fault prediction models
used requirements or code features in isolation. The solid lines
show the results of models which used these two kinds of features in
combination. Note the remarkable improvement: learning from {\em
multiple perspectives} such as code {\em and} requirements measures
in conjunction, lead to dramatically improved $pd$ and $pf$.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.3in]{pc1r.png}
\includegraphics[width=2.3in]{cm1r.png}
\end{center}
\caption{$\{pd,pf\}$ curves seen when using
code and/or requirements features.
From~\cite{jiang07}.}\label{fig:ttwo}
\end{figure}
The knowledge of the values of multiple feature sets from different
sources along the project's development life cycle leads to
conclusion that the resulting models likely represent
domain-specific, if not project-specific solutions. \fig{ttwo}
strongly suggest in favor of for using multiple feature sets from
different software artifacts.  While an exciting result, it has some
drawbacks. Different projects use different tools and build software
using different processes. Hence, different projects have access to
different sets of features which may be useful for predicting where
faults hide; e.g. an agile process may have no access to the
detailed requirement documents used to build model in \fig{ttwo}.

Other experiments  also suggest that the {\em best} set of features
are domain specific. {\em Feature subset selection} (FSS) checks if
feature subsets
performs as well as  supersets.
In all the data sets studied
in~\cite{me07b}, 38 static code features could be pruned down to
three {\em without} reducing the efficacy of the defect predictor.
Significantly,
in each data sets, FSS selected
{\em different features}. That is, FSS found no universally {\em
best} features for defect prediction.  An analogous result has been
been reported by Shepperd and Ince~\cite{shepperd94}. They reviewed
18 publications in which no {\em best} predictive feature was found.
In those publications, an equal number of studies reported that 
cyclomatic complexity is the same, is better, or is worse
than LOC in predicting defects.

\fig{sorted} explains the FSS results and the results of Shepperd
and Ince. It also cautions that the best set of features will change
from domain to domain. This figure plots the {\em information gain}
associated with 38 static code features.  This {\em infogain}
measure is used in decision tree learners~\cite{quinlan92} to select
the feature that best divides up the examples\footnote{Infogain
compares the number of bits required to encode the distribution of
defective/non-defective classes before and after dividing the
examples amongst the range of feature F.}. The first 16 features
have very similar information content. In fact, given the standard
deviations shown in the error bars of that figure, the 16th-ranked
feature is statistically insignificantly different to the top-ranked
features. Hence, minor changes in the training sample
 change what features are found to be the {\em best} software  quality predictors.


\begin{figure}{r}{3.5in}
\begin{center}
\includegraphics[width=3.5in]{infogain.pdf}
\end{center}
\caption{Mean and standard deviation (t-bars)
of InfoGain in 10*90\% random samples of defect data.
}\label{fig:sorted}
\end{figure}
Based on the above, in contrast with the current trend in the
literature, we recommend against trying supposedly better AI methods
to learn defect predictors. We make this recommendation
for two reasons.

Firstly, different  data miners have failed to improve the state of the art
in learning defect predictors. The early plateau effect of
\fig{tone} suggests that static code features yield all they can
yield after as little as a few hundred examples. This information
can be found by relatively simple learners such as Naive Bayes. That
is, the added value of more sophisticated AI data miners must be
questioned.

Secondly, there is a limited generality in the predictors learned by standard
AI data miners. \fig{ttwo} showed that combining data sources can
yield fault predictors that out-perform predictors learned from
either source in isolation. Different domains and different projects
offer different data sources and, consequently, the {\em best} set
of features to use in a quality prediction will be domain-dependent.
Further evidence for the domain dependent nature of quality
predictors comes from \fig{sorted}. Given a set of features, many of
them compete to be the {\em ``best"} one and minor variations in the
sampling methods can yield different selection of {\em best}
predictors. 


