\documentclass[letterpaper]{article}
\usepackage{aaai}
\usepackage{url}
\usepackage{times}
\usepackage{helvet}
\usepackage{courier}
\usepackage{graphicx}
\usepackage{stfloats}
\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}
\newcommand{\stress}[1]{\sffamily{#1}}

\title{Lessons Learned(ing) from the PROMISE Repository of SE 
Data\thanks{An earlier draft of this paper can be downloaded from
\protect\url{http://menzies.us/pdf/07promise.pdf}}\thanks{The authors gratefuuly acknowledge the 
contribution of Jelber Sayyad Shirabad  for his work
on the first two years of the PROMISE experiment.}
}
\author{
Tim Menzies \and Justin DiStefano  \\ Lane Dept. Com. Sci. \& Elec. Eng.\\ West Virginia University \\ Morgantown, WVU, USA  
\\ \url{tim@menzies.us}\\\url{jdistefano@ismv.com} \And
Gary Boetticher \\ Science \& Computer Eng. \\ Uni. of Houston Clear Lake, \\ Houston, Texas, USA \\ \url{boetticher@cl.uh.edu} \And
Tom Ostrand \\ AT\&T Labs \\ Florham Park,\\ NJ, USA \\ \url{ostrand@research.att.com} 
}
\pubnote{\emph{Submitted to 
AAAI-07 Workshop: Evaluation Methods for Machine Learning II
}} \begin{document}
\maketitle
\section{Introduction}
Each year since 2005,
 the \underline{Pr}edictive \underline{M}odels \underline{i}n \underline{S}oftware \underline{E}ngineering (PROMISE) 
workshop on repeatable software engineering (SE) experiments
asks researchers
to ``put up or shut up''; i.e. publish conclusions {\em and} the data used to generate those
conclusions\cite{me07c}.

Eventually, all the concerns and sampling biases of the UCI machine
learning repository~\cite{Blake+Merz:1998} will apply to the PROMISE
data.  For example, an excessive focus on empirical results can
stunt the development of innovative ideas that are, as yet,
pre-experimental.


However, as we see it,
the field of software engineering is in {\em no danger} of an excess of empiricism. 
Much has been {\em written} about empirical SE~\cite{basili88,fenton96} 
but
little has been {\em concluded} (exception: good progress has been made
on effort estimation models~\cite{jorgensen05,boetticher03}). 

One test of PROMISE is to ask ``what generalities can be learned from it''?
Such generality has eluded prior research:
\bi
\item
In
18 publications discussing defect prediction from static code measures,
an equal number of studies reporting that the
one attribute (the McCabe cyclomatic complexity) is the same; is better; or is worse than
lines of code in predicting defects~\cite{shepperd94}.  
\item
In fifteen studies that compare model-based to expert-based software effort
estimation. Five of those studies found in
favor of expert-based methods;  five  found no difference; 
and five found in favor of model-based estimation~\cite{jorg04}.
\ei
These examples suggest that data mining in software engineering suffers from
a dire {\em sampling bias} problem; i.e. that we don't have enough information to make
any clear conclusions.
In this note, we offer some good news and bad news about the stability problem. Domains and 
measures are identified that are more/less prone to unstable conclusions.

\section{Motivation}\label{sec:nodata}
Much that is considered ``best practice'' in software engineering
has little
evidentiary basis.
For example, in a recent review of papers relating to software verification and validation, we found 
nearly 100 proposed tasks for software V\&V. However,
when we sought empirical evidence for the relative cost-benefits of those tasks, the available evidence
was meager indeed.
Many publications 
base their 
conclusions on a handful of projects
(e.g.~\cite{glass97,dabney07,me03q,me07b});
Others use a Delphi-style analysis
 where the authority of the conclusions
rests on the authority of the author (e.g. ~\cite{boehm01,shu02,madachy97}).
Alternatively, they may
base their conclusions on extensive data collection
but, for reasons of corporate confidentiality, can not
make that data available for public scrutiny (e.g. ~\cite{jiang02,ropp00,raffo04})
(and it
is hard to audit conclusions  based on a Delphi-style analysis or
hidden data).
Several of the these papers (e.g.~\cite{jiang02,ropp00}) offer general advice about how to avoid problems in software developers.
The subjective nature of this advice makes it difficult to consistently
deploy
them 
over a large organization.
Only a few
offer public domain versions of their data and models
(e.g. ~\cite{me06d,me07b,abe06,takagi05}). 
Worst of all, even when taken in combination, these publications cover just a small
fraction of the  V\&V tasks listed in the literature~\cite{wallace89,1012,costello05}.


This lack of empirical evidence for much of the current software engineering enterprise motivates
us to continue the PROMISE experiment.

\begin{figure} {\footnotesize {\em
\begin{center}
\begin{tabular}{r|c@{~}l}
year & \multicolumn{2}{l}{number of data sets}\\\hline
2005 & 18 & \rule{18mm}{2mm}\\  
2006  & 24 & \rule{24mm}{2mm}\\
2007 &  35 & \rule{37mm}{2mm}\\
\multicolumn{3}{c}{~}\\
\multicolumn{3}{c}{~}\\
domain & \multicolumn{2}{l}{number of data sets}\\\hline
defect prediction &  17 & \rule{17mm}{2mm} \\
  effort estimation &    9 & \rule{9mm}{2mm} \\    
misc & 6 & \rule{3mm}{2mm}\\
 text mining &    3 & \rule{1mm}{2mm}\\\cline{1-2}
total & 35
\end{tabular}

\end{center}}}
\caption{Growth of repository (top) and current contents (bottom).} \label{fig:data}
\end{figure}

\section{About the Data}
It is surprising that after only 3 years of operation, PROMISE has collected
the data shown in \fig{data}.
Consider the
track record of the software engineering industry: in 2001,
the Standish group described 
a \$250 billion dollar American software industry where 31\% 
of projects were canceled and 53\% of projects incurred costs 
exceeding 189\% of the original estimate~\cite{standish01}.
Few companies have obtained the high CMM process maturity
required to
review their own short-comings for the purposes of
self-improvement manner~\cite{ahern01,gibson06}.
Most  
corporations hide their data since they are 
reluctant to expose their own development record 
to public scrutiny. 

PROMISE has benefited from several recent developments.
The nature of academic SE is changing as witnessed by the
increased impact ratings of the Journal of Empirical Software Engineering.
More SE researchers are moving to data mining, in part due
to the 
easy access to robust and diverse tools such as the WEKA~\cite{witten05}.

Also, 
there is more data to process.
The open source community has proven to be rich sources of information.
PROMISE holds six ECLIPSE data sets (file and package-level metrics
for ECLIPSE 2.0, 2.1, 3.0) as well as data from FreeBSD, Xfree,
Xorg, and other projects.

NASA is another source of data.
For several years, NASA ran its own requirements
and code repository\footnote{\protect\url{http://mdp.ivv.nasa.gov}}. That
repository is now inactive and some of its data has been copied to PROMISE.
In all, $\frac{15}{35}$ of the current PROMISE
data sets were provided by NASA.

One area that has become particularly active at this time is text mining:
\bi
\item
For example, best paper at the 2006 IEEE requirements engineering conferencing
was awarded to a requirements text mining study~\cite{huan06}. 
\item
Many of the text mining
practitioners exhibit an exemplary experimental technique. 
For example, a recent study checked
the relative merits of
different text mining tools for finding links between requirements documents.
That analysis included a comparison
of human expert and state-of-the-art commercial tools and a new research prototype~\cite{hayes06}. 
\ei

PROMISE has also benefited from the corporate culture of software development.
Consider a company that has organized a metrics
program, an artifacts repository,  built a sanitizer that can
``scrub'' data of corporate identifies prior to release, and resolved
any remaining legal issues. The organizational inertia of that
company means that once {\em one} data set survives that marathon,
then {\em many more} will follow (one for each new product and for
each new release of each product).  
This has the added benefit that PROMISE now contains {\em multiple}
data sets with the {\em same} schema taken from  {\em multiple} domains. Hence,
 we can check the stability
of data mining conclusions.

\begin{figure}[!b]\includegraphics[width=3.2in]{nasaMean.pdf}\caption{
The top plot shows the
number of projects in 27 subsets  of our three data sources.The
middle and bottom plots show the standard deviation and mean in
performance error.  Data subsets are sorted by the error's standard
deviation. Effort models were learned via either standard LC (a standard effort estimation method~\protect\cite{boehm81}) or our 
COSEEKMO workbench that applies multiple learners, plus feature subset selection, to the 
data~\protect\cite{me06d}. }\label{fig:one}\end{figure}

\section{Effort Estimation Instability }
Much of the PROMISE data relates to software effort estimation $\left(\frac{9}{35}\right)$
or defect prediction
from static code measures $\left(\frac{17}{35}\right)$. 
These are two very different problems. The relationship
of {\em project} to generated software {\em modules} is one to many.
With defect prediction, a single project may write thousands of modules and thus create 
thousands of instances. In effort estimation, one project may contribute just a single
instance to an effort database. 
Therefore, while a defect data set may contain thousands of instances,
an effort estimation data set may contain only a few dozen, at best.

Consequently, we would predict that there is much less stability in effort estimation than defect prediction.
\begin{figure*}
    \begin{center}
\begin{minipage}{4.5in}
    \includegraphics[width=4.5in]{lmeansv.pdf}
\end{minipage}~~\begin{minipage}{2.5in}
\footnotesize
{\bf Definitions:}
Let $\{A,B,C,D\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary detector. 
\[\begin{array}{rcl}
pd=recall&=&\frac{D}{B+D}\\
pf&=&\frac{C}{A+C}\\
prec=precision&=&\frac{D}{D+C}\\
acc=accuracy&=&\frac{A+D}{A+B+C+D}\\
\frac{neg}{pos}&=&\frac{A+C}{B+D}\\
\end{array}\]
Also, $effort$ is the \% of the code selected by the detector
that a human must now inspect.
\end{minipage}
\end{center}
    \caption{Mean $\mu$ and standard deviation $\sigma$
    of changes in defect detector statistics between
a baseline (tested on the training set) and another data set
(tested on different data).
A zero value in this figure indicates
that the detector worked the same
on training and  test data.
    Dots
    denote mean ($\mu$) values.
    Whiskers extend from $\mu + \sigma$ to $\mu-\sigma$.
From~\protect\cite{me03q}.
}
    \label{fig:lmeansvs}
    \end{figure*}
This turns out to be the case. 
\fig{one} illustrates {\em instability} in learned software
{\em effort estimators}.
Shown in that figure are the
data set size, mean, and performance standard deviation from 27 data sets across
30 trials where ten records were selected at random for testing.
The size of the training sets was small (ranging from 20 to 161)
and
the standard deviations were
overwhelmingly larger than the means.
Elsewhere, we have explored numerous data mining methods
for  learning software effort estimates including
feature subset selection
over model trees, linear regression,
Boehm's local calibration method,etc~\cite{boehm81,quinlan92b,me06d}.
To date, we have found no stability in
what is the ``best'' methods. 
The reason for this instability is very simple:
the data sets are so small (20 to 121 instances; 16 attributes)
and hence very ``quirky'' and highly variable. 

Note that data mining methods
can help the effort instability problem, but only to a limited degree.
the dashed line of \fig{one}
shows the results after applying a Kohavi\&Johns-style WRAPPER~\cite{kohavi97}. 
Note that the deviation
is reduced, but still remains relatively large compared to the mean.
  
 
\section{Defect Detection  Stability }
Effort estimation data sets are typically smaller than defect detection
data sets.
Hence, while the former may be unstable, the latter may be very stable.

For example,
\fig{lmeansvs} illustrates {\em stability} in defect detection.
The task here was learning a predictor for software module defects from static code measures.
The data sets contained hundreds to thousands of instances and the target class (modules with
known defects) comprised 2\% to 20\% of the data.
In this study,
different learners (e.g. M5'~\cite{quinlan92b}, J48~\cite{quinlan92}, ROCKY~\cite{me03a}, LSR=linear regression)
were
{\em trained} on one of five data sets;
{\em self-applied} on the same training set (to generate a baseline
performance measure);
then {\em tested}
on each of the other four data sets.
The results  show that on
most performance measures (accuracy, probability of detection- also known as recall;
probability of false alarms)
the detector performance were remarkably stable.
The exception was precision, which exhibited very large standard deviations.
 We will return
to this observation, in  the next section.

(One digression before proceeding:
the stability of \fig{lmeansvs} seems to contradict the study mentioned above regarding the
instability in the performance of the McCabe static code measures.
In this regard, \fig{sorted} is informative.
That figure shows
features in a PROMISE defect prediction data set used in \fig{lmeansvs}.
Note how the highest ranked
features (those on the left-hand-side) offer very similar
information.  That is, there are no clear winners so minor changes in
the training sample (the 90\% sub-sampling used in subsetting or a
cross-validation study) can result in the selection of very different
``best'' single methods.) 
\begin{figure}
\begin{center}\includegraphics[width=3.5in]{infogain.pdf}\end{center}\caption{InfoGain
for KC3 attributes. Calculated by WEKA's InfoGain operator~\cite{witten05}.
Lines show means
and t-bars show standard deviations after 10 trials on 90\% of the
training data (randomly selected).}\label{fig:sorted}\end{figure}
 
\section{Precision Instability}

\fig{lmeansvs} reports that defect predictors are stable on most measures, and very unstable
with respect to precision.
It turns out that precision is an inherently unstable measure, at least for 
software engineering data sets with a large $\frac{neg}{pos}$.
Recall that the fraction $\frac{neg}{pos}$ reflects the relative size of the target classes $pos$. 
When then there are very few $pos$ instances in the training data,
then
$\frac{neg}{pos}$
is large, 

Large $\frac{neg}{pos}$ ratios are very common in software engineering:
\bi
\item
For example, in the PROMISE data sets for defect detection, the number of instances
with defective modules is very small. Assuming that the goal of the learning
is to find those defective modules, then  
those $\frac{neg}{pos}$ ratios
range from 
1.04, 
7.33,
9,
10.11,
13.29,
15.67, to
249.
\item
  Hayes, Dekhtyar and Sundaram use text mining to                                                                     
find pairs of connected requirements in a corpus of 220 requirements and                                                                     
235 design elements.
The total number of possible links in the dataset is $220\cdot 235 =                                                                        
51,700$, while the the ground truth RTM contains 361 links, for the                                                                         
$\frac{neg}{pos}$ ratio of $51,700/361 = 143.2$~\cite{hayes06}.   
\ei
Despite much advice to the contrary  (e.g. ~\cite{provost98case}), it is still common to
report classifier performance in SE in  terms of accuracy  and text mining performance in terms
of the f-measure which is the harmonic mean of precision and 
recall\footnote{$\frac{2\cdot prec \cdot recall}{ prec + recall}$}; e.g. \cite{nagappan05,hayes06,huan06}.
In the case of large $\frac{neg}{pos}$ ratios, accuracy is clearly an incomplete report of
classifier performance\footnote{When $\frac{neg}{pos}=99$, 99\% accuracy can be achieved without finding
a single example of the target class.}. 
Also, as shown below, precision measures are very unstable for data sets with large $\frac{neg}{pos}$ ratios.

\begin{figure}
\begin{center}
\includegraphics[width=3.5in]{y_0.pdf}
\end{center}
\caption{ $Pd, prec, recall$ and $\frac{neg}{pos}$. 
}\label{pprnp}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=3.5in]{y_05.pdf}
\end{center}
\caption{
Figure~\ref{pprnp},
cropped to the region
where $prec>0.5$.
}\label{pprnp1}
\end{figure}To demonstrate when precision is unstable, we 
re-arrange standard definitions of 
\fig{lmeansvs} (right hand side)
to derive:
\begin{equation}\label{one}
prec = \frac{D}{D+C} = \frac{1}{1+ \frac{C}{D}} = \frac{1}{1 + neg/pos \cdot pf/recall}
\end{equation}
Figure~\ref{pprnp}
graphs Equation~\ref{one} for increasingly large $\frac{neg}{pos}$ ratios.
Figure~\ref{pprnp1} does the same, 
only for zones of higher precision:
only the surface for
$0.5\le prec \le 1$ is shown. 
The shadow of the  surface on the bottom plane
shows this zone of high precision.
As $\frac{neg}{pos}$ increases, 
high recall\&precision
is only possible when $pf$ becomes vanishingly small.
This explains a repeated result in the SE literature: there
are no known reports of high precision and high recall detectors.
Typical results include:
\bi
\item
Huang et.al. won ``best paper'' at the 2006 IEEE
Requirements Engineering conference with detectors exhibiting
$prec\approx 0.25$~\cite{huan06}.
\item
Without extensive feedback from human experts, the classifiers used
by Hayes, Dekhtyar, and Sundaram exhibited $pf>0.6$ (hence, very
low precisions)~\cite{hayes06}.
\item
Defect predictors built from the PROMISE repository achieves
achieves recall over 70\%, minimum false alarm
rates of 15\%, yet  very low precisions:
$\{min,median,max\}=\{2\% 20\%, 70\%\}$ (the last number came from
a data set with $neg/pos\approx 1$)~\cite{me07b}.
\item
Researchers into
the auto-generation of traceability links 
report that high recall is achievable only with low precision
detectors~\cite{antoniol02,antoniol05,marcus03}.
\ei  

\noindent
More importantly, these results also  explain 
 the instability of precision {\em and} the stability of $pf$ and $pd$ (recall) 
seen 
in 
Figure~\ref{fig:lmeansvs}:
\bi
\item
Note how, in Figure~\ref{pprnp},
that at very small $pf$ values,
tiny changes in $pf$ can lead to very large changes in $prec$ 
(sudden jumps from zero to one).
\item
The other measures in
Figure~\ref{pprnp}, on the other hand, change far more smoothly and slowly.
\ei

\section{Discussion}
There is insufficient evidence
to
support current claims about supposed ``best practice'' in the field of software engineering.
In the area of V\&V, for example, we discussed above how few of the (nearly) 100 known
V\&V tasks had any 
evidentiary basis. 

To prove the promise of PROMISE, we need to find general conclusions within the repository
data.  In this short note we have reviewed some of the progress to date:
\bi
\item
There are many examples of unstable conclusions in software
engineering, possibly due to corporate cultures that discourage data sharing.
\item
Initially, we had some pessimism that we could collect enough data to make
general conclusions. 
This pessimism turned out to be unfounded.
NASA and the open source community have been particularly supportive. Text mining
is emerging as a field with a strong empirical tradition that will offer many
 new source of data.
\item
The ratio of effort estimates to generated software modules is one to many. 
Hence:
\bi
\item
Defect databases (reporting module statistics)
tend to be 
larger than effort estimation databases.
\item
Effort estimators exhibit fewer regularities than defect estimators.
\ei
\item 
Theoretically and empirically, it is very difficult to build detectors with both high precision {\em and} high recall
for data sets with
large $\frac{neg}{pos}$ values,
\item
Another problem with large $\frac{neg}{pos}$ data sets is that
they make precision measurements highly unstable.
Precision is hence
deprecated for that class of data.  
\ei
Time will tell if PROMISE can collect enough
data to extend and improve this list of conclusions.

\bibliographystyle{aaai}
\bibliography{refs}
\end{document}
