\documentclass{sig-alternate}
\usepackage{times}
\usepackage{cite}
\usepackage{algorithm}
\usepackage{bar}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{SEI}{Data Quality Workshop}
\CopyrightYear{2010} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{X-XXXXX-XX-X/XX/XX}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

%\title{An Empirical Study on the Influence of Evaluation Criteria used in Software Cost Estimation  }
\title{Selecting Quality Data}

%\numberofauthors{4}
%\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
%\alignauthor
%Author1
% 2nd. author
%\alignauthor
%Author2
% 3rd. author
%\alignauthor 
%Author3
%\and  % use '\and' if you need 'another row' of author names
% 4th. author
%\alignauthor 
%Author4

%


\numberofauthors{1} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor  Tim Menzies\\
            \affaddr{CS \& EE, WVU, Morgantown, USA}\\
       \email{tim@menzies.us}\\
	August 26, 2010
}



\maketitle
\begin{abstract}
Data quality planning
can waste much time and resources if it inappropriately elaborates
or collects data that is not significant for decision making.
In the experiments shown here, we see that often most data is insignificant.
Therefore, we propose iterative quality planning where version $i+1$ of
the data quality plan is informed by {\em significance studies} on samples
of data collected using plan $i$.
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\section{Introduction}
%What data quality plans are appropriate for the five levels of the capability maturity model:
%
%\be
%\item {\em Level 1 - Initial (chaotic):} undocumented data quality processes that are 
%ad hoc, uncontrolled, and may be in a constant state of change.
%\item
%{\em Level 2 - Repeatable:}
%At this level, some repeatable data quality plain is in place which ensure that
%the data quality process is maintained, even during periods of organizational stress.
%\item
%{\em Level 3 - Defined:}
%It is characteristic of processes at this level that there are sets of defined and documented standard processes established and subject to some degree of improvement over time. These standard processes are in place (i.e., they are the AS-IS processes) and used to establish consistency of process performance across the organization.
%\item
%{\em 
%Level 4 - Managed:}
%It is characteristic of processes at this level that, using process metrics, management can effectively control the AS-IS process (e.g., for software development ). In particular, management can identify ways to adjust and adapt the process to particular projects without measurable losses of quality or deviations from specifications. Process Capability is established from this level.
%\item{\em
%Level 5 - Optimizing:}
%It is a characteristic of processes at this level that the focus is on continually improving process performance through both incremental and innovative technological changes/improvements.
%\ee
%
Organizational data quality plans include 
{\em definitional} plans, {\em collection} plans, and {\em significance}
studies:
\bi
\item There are many {\em definitional} quality plans that range
from  lightweight to 
ultra-heavyweight.
For an example of the former, 
see short list of the 17 key process areas of COCOMO-II that define the COCOMO {\em process maturity} feature\footnote{\url{http://sunset.usc.edu/research/COCOMOII/expert_cocomo/drivers.html\#PMAT}}.
 As an example of the latter, see the 7,193 pages of the Microsoft's preferred XML standard
for its Office products\footnote{\url{http://en.wikipedia.org/wiki/Office_Open_XML}}.
\item
Once the data is defined then {\em collection} quality plans describe how
an organize collects
and stores its data. 
\item Finally, using the stored data, {\em significance studies} describe how
the stored data can be  used to provide feedback from quality improvements within
an organization. Later in this paper, we offer two examples of such significance studies.
\ei
Ideally,
the definitional and collection plans are active before the significance study is conducted.
This ideal case may be hard to achieve, particularly in the context
of large government software projects developed by networks of contractors and sub-contractors
and sub-sub-contractors etc. At least in the NASA experience, it is hard
to define and collect data in a consistent manner across such a network. 
If
difficult
to
obtain project data,
especially
across
multiple
organizations.
This
is due to the business sensitivity associated with the data as well as 
differences in how the metrics are defined, collected, archived and interpreted:
\bi
\item 
In many cases the required data has not been archived at all. For 
example, after two years we were only able to add 7 records to our 
NASA wide software cost metrics repository. 
\item
Managers
are reluctant to release all their data, just in case it shows something
undesirable about their organization (e.g. a higher defect rate that other contractors).
\ei
Is it possible to conduct significance studies where
\bi
\item
The data definitions are unknown or ambiguous;
\item
Or the data collection is ad hoc and sporadic?
\ei
According to the experience of the
PROMISE conference on repeatable experiments in software engineering\footnote{\url{http://promisedata.org/2010}}, the answer is ``yes''.
PROMISE is a research  community that applies significance studies data to offer feedback on a software
project. In the usual case, PROMISE researchers use data sets that were collected elsewhere by
other research or industrial teams who are now uncontactable. Hence, all PROMISE can
access are the data files.
For this workshop, the key point of PROMISE is that this data 
has low quality definitions and/or collection.

A typical significance study at PROMISE is some kind of hold-out
study where a model
is learned from some of the data, then tested on the rest. In the worst case, the data is
of such low quality that no effective model can be learned and the test performance is very
poor.
However, there are many examples at PROMISE of results that are much better than this worst case.
Significant lessons learned at PROMISE
include how to use project data to:
\bi
\item {\em Predict} software quality features such as number of defects~\cite{w08}
or development time~\cite{boetticher05}.
\item
{\em Change}  a project in order 
to improve some quality feature~\cite{menzies09}.
\ei
The lesson of PROMISE is that while high quality data is best, it is possible to offer
effective feedback to organizations using low quality
data. The key are {\em selection operators} that prune away superfluous and noisy data
to reveal the significant information underneath. The rest of this paper offers
case studies on this selection process. Note that, in all cases,
{\em throwing away} parts of the data lead to significant improvement in the model.

The conclusion of this paper  is that while definition and collection plans for data
quality are important, {\em the right level of data quality may be lower than currently
suspected}. Selection operators can repair low-quality data, at least to the point where
significance
business decisions can be made about the project. Hence, when writing quality plans
for data
definition and collection,
we recommend {\em auditing} those plans with an exploratory data significance study.
Elaborations on the
data definitions and data collection become superfluous when those elaborations {\em no
longer improve the recommendations coming from  the significance studies}.

\section{Background: Less is More}
Collection data from organizations is expensive and time-consuming. It hence seems strange
to propose throwing away such a valuable resource. Yet there are many reasons to do exactly that:


%%%%%%%%%%%%

\bd
\item[Noise:] Data collection was somehow flawed for a particular feature with the
result that one or more features are {\em noisy} (i.e. 
contains spurious signals not associated with
variations to projects). 
Learning a cost estimation model is easier when
the learner does not have to struggle with fitting the model
to confusing noisy data.
\item[Correlated features:] 
The reverse problem of noise is associated features.  If multiple features
are tightly correlated, then 
using all of them
will diminish the likelihood that either feature attains significance.
A repeated result in data mining is that removing some of the correlated
features increases the effectiveness of the learned model
(the
reasons for this are subtle and vary according to which particular
learner is being used~\cite{hall03}). 
\item[Under-sampling:] The number of possible influences on a project
is quite large and, usually, historical data sets on projects for a 
particular company are quite small. This motivates the removal
of features, even if those features are theoretically useful.
\ed
\fig{dels} shows that under-sampling
is not just a theoretical concern.
In the {\em nasa60} data set\footnote{From the PROMISE data archive \url{http://promisedata.org/data}.}
 most
 projects were rated as having
 a {\em high} complexity.  Therefore, this data set would not support
conclusions about the interaction of (say) extra high complex
projects with other features. A learner would be wise to
remove this feature (and a cost modeling analyst would be wise
to suggest to their NASA clients that they 
refine the local definition of ``complexity'').

\begin{figure}[!h]
\begin{barenv}
\setyaxis{0}{50}{10}
%\setxaxis{2}{6}{1}
 \bar{2}{8}[\small low] % -5
 \bar{5}{8}[\small nominal] %-4
 \bar{50}{8}[\small high] % -3
 \bar{2}{8}[\small v.high] % -2
 \bar{1}{8}[\small x.high]  % -1
\end{barenv}
\caption{Distribution of software complexity {\em cplx}
within the {\em nasa60}  data set. 
}
\label{fig:dels}
\end{figure}

Whatever the reason (noise, correlation, under-sampling),
a repeated result in the data mining community is that simpler
models with equivalent or higher performance can be built
by {\em feature subset selection} (FSS) algorithms that 
intelligently remove of useless features. 

\subsection{Feature Selection With WRAPPER}

Kohavi and John's
WRAPPER~\cite{kohavi97} 
builds models using an increasing number
of features; i.e. $N=1,2,3,4,...$.
For each $N$, there is a current {\em selected} set (initially,
the empty set)
 and a set of remaining features (those that have not
been selected). For each remaining feature, a new
bigger set is tried containing the current set, plus one of the remaining
features. The set that yields the best learned model
becomes the current set, $N$ becomes $N+1$  
and the process continues.
WRAPPER stops when either the remaining set is empty or
there has been no significant improvement in the learned model
for the last five additions (in which case, those last five
additions are deleted).

WRAPPER is thorough and, according to Hall and Holmes,
often yields the best results~\cite{hall03}.
In theory, is quite slow.
It is simple to see why: there are a large
number of possible subsets to explore.
A naive search through all possible 
subsets of the 15 COCOMO-I features
would have to explore  32768 subsets.
However, for project data sets with only a few hundred records, of less,
it is quite a useful technique.

\subsection{Feature Selection With SIGNIFICANCE}
For larger projects, it is useful to have an alternative to the exponential-time
search of the WRAPPER. 

Given two classes $C_1$ and $C_2$ of
frequency $N_1 = |C_1|$ and $N_2 = |C_2|$, if some feature range occurs with frequency $F_1,F_2$ in
$C_1,C_2$ (respectively), then after Mozina et al.~\cite{mozina04}, the $significance$ of
that range in selecting for $C_1$ is
\begin{equation}\label{eq:fenton}
significance = log\left(\frac{ F_1/N_1}{F_2/N_2}
\right)
\end{equation} 
and if $significance=0$, then that range is {\em insignificant}
since it occurs with equal frequency in $C_1$ than $C_2$.

Note that \eq{fenton}, unlike the WRAPPER, takes linear time to compute.  

\section{Case Study \#1: Using WRAPPER}
In our first case study, the WRAPPER was applied to some
COCOMO effort estimation data (from Boehm's 1981 COCOMO text~\cite{boehm81},
from the COCOMO-II data, 
and  some more recent NASA data)
using the experimental method of \fig{exper1}.
\begin{figure}[!b]
\begin{tabular}{|p{0.95\linewidth}|}\hline
Ten
             times (each time using a randomly selected 90\% of the
             rows),    the value of a column was then set to the number
             of times the WRAPPER selected that column.
Once the WRAPPER ordered the columns, we randomized the order of the
rows and starting column pruning.  

To ensure statistical validity,
randomization (followed by column pruning) was repeated 30 times.  
For each repeat, at each stage of the pruning, the lowest value column
was removed.  The rows in the remaining columns were then divided into
training and test sets (each time using a randomly selected 67\% of
the rows for the training set). A cost model was learned (using linear
regression) from the training set and then assessed, using PRED, on
the test set.

Once the 30 repeats were completed, a best model was selected by
looking at the mean and standard deviation of model performance at
each pruning step.  The best model was the one that t-tests confirmed
out-performed all the other states of the column pruning. The mean
value of that model over the 30 repeats was then reported.\\\hline
\end{tabular}
\caption{The experiment of Case Study 1.}\label{fig:exper1}
\end{figure}
The results were measured using
PRED(30), which is calculated from 
the relative error, or RE (the relative
size of the difference between the actual and estimated value):
\[
RE = \frac{estimate - actual}{actual}
\]
PRED(N) reports the average percentage of estimates that were within
N\% of the actual values. If a data set has $R$ rows, then:
\[
PRED(N) = \frac{100}{R} \sum_i^R\left\{
\begin{array}{l}
1\;if\;MRE_i\;{\le}\;\frac{N}{100}\\
0\;otherwise
\end{array}\right.
\]
For example, a PRED(30)=50\% means that half the estimates are within
30\% of the actual. 
\begin{figure*}[!t]
{\footnotesize
\begin{center}
\includegraphics[width=4in]{pruned.pdf}
\end{center}}
\caption{Number of columns.}\label{fig:prunep}
\end{figure*}
\begin{figure*}[!t]
{\footnotesize
\begin{center}
%Results from Schofield~\cite{shepperd97}&~&Results from this paper\\\cline{1-1}\cline{3-3}
\includegraphics[width=1.33in]{colp.pdf}\includegraphics[width=1.33in]{lightRowColp.pdf}\includegraphics[width=1.33in]{heavyRowColp.pdf}
\end{center}}
\caption{Effects on PRED of
different pruning methods. 
For space reasons,
only the PRED(30) results are shown. A longer version of this paper, in preparation,
includes PRED(20) and PRED(25) results.
}\label{fig:rowp}
\end{figure*}

The results of WRAPPER's column pruning are shown in \fig{prunep} and \fig{rowp}. 
In those figures, the datasets are sorted on the x-axis in terms of their size
(biggest on the left, smallest on the right).

\fig{prunep} shows how much data was thrown away by the WRAPPER.
{\em Before} pruning (shown in red), our data sets had 22 columns for the COCOMO-II
data sets ($cii0$ and $cii4$) and 15 columns for the other COCOMO-I data sets.
{\em After} pruning (shown in green), there are often much fewer columns.
 The blue
line shows the percentage of the $before$ columns removed by pruning.
For example, very little was pruned from $cii$ while most of the columns were
pruned from $p03$.  On average, over 65\% of the columns were
pruned. Sometimes, the pruning was quite heavy with over 80\% of the
columns pruned away.

\fig{rowp} shows the benefits of the pruning. The red line shows the PRED(30) results
{\em before} pruning and the green line shows the PRED(30) results {\em after} pruning.
The improvement produced by pruning is the gap between the red and the green lines.
Two features of this figure are worthy of comment:
\be
\item
Pruning {\em always} improved estimation effectiveness, sometimes dramatically.
For data sets with under 20 examples, the $after$ results
are better by a factor of 2 to 4 than $before$.
\item
With one exception ($c03$), the general trend across the graphs
is clear: as data set size {\em shrinks}, the improvement {\em
increases}. That is, pruning is most important when dealing with small data sets.
\ee

In summary, in all our experiments, it was {\em always} useful to ignore a portion 
of the available data.

\begin{figure*}[!t]
{\footnotesize
\begin{center}
\includegraphics[width=6in]{nomofss.png}
\end{center}}
\caption{Feature selection on Fenton's data.}\label{fig:nomo}
\end{figure*}

\section{Case Study \#2: Using SIGNIFICANCE}

The benefits of feature selection have been observed in many other domains.
Norman Fenton~\cite{fenton07a}
builds Bayes networks representing expert intuitions about connections
of factors that degrade software quality.  We applied the SIGNIFICANCE simple feature selector
to determine which nodes in these networks were most influential on creating the worst
defects. 

His data was scored by $\frac{defects}{KLOC}$, then divided into five divisions.
We calling the last division, with the most defects {\em worst} and the other divisions {\em rest}.

\fig{nomo} ranks how much features in a Fenton network select for $worst$.
The features are sorted according to the {\em significance} of their ranges (computed from
 \eq{fenton}).
Observe how:
\bi
\item
In terms of selecting for worst defects, only a few ranges are important;
e.g. $log\_KLOC\_new$
has the greatest significance for predicting for worst defects.
\item
Most features have nearly zero significance; i.e.   
{\em most features do not matter}.  
\ei
Some of these insignificant ranges are surprising. For example, in
this data, $programmer\_capability$ falls fourth from the bottom
since all its ranges have near-zero significance. At first glance,
this seems a crazy result- of course programmer capability effects
the defects per KLOC. However,
recalling the discussion around \fig{dels}, feature
selection does not comment on the $absolute$ significance of some feature. Rather, it
comments on feature significance {\em relative to other features}. Of course program complexity
has an impact on effort. However, given the data of \fig{dels}, it is not as important
as other features. 

Similarly, in \fig{nomo}, programmer capability may be significant in some $absolute$ sense.
However, given Fenton's data, it is $relatively$ less important that other features
such as lines of new code.

\section{Discussion}
The lesson of feature selection is that, given the local conditions of a particular project,
most features can be insignificant.
For anyone writing a software data quality plan, 
this result has major implications. It may be a waste of effort to expend great effort 
to precisely define all possible data points, then spend much time collecting data according to
those definitions. Rather, a more effective approach would be:
\be
\item Quickly implement a draft definition/collection plan.
\item As soon as any data is available, apply feature selection.
\item Only elaborate the data items that feature selection reports are
 significant.
\ee 



\bibliographystyle{abbrv}
\bibliography{../../../../trunk/tex/refs,new} 
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%

\end{document}


In order to support this claim,  this paper demonstrates a study of nearly one
hundred different algorithms executed on numerous datasets. For example,
this paper explores The experiment explores the following research questions:

{\bf RQ1:} Is there any stability in the conclusions gained from the currently available
datasets of real project data?

{\bf RQ2:} 
Our reading of the effort estimation
literature is that there is {\em little,little,extensive} focus on the {\em datasets,preprocessors,learners}.
Is this current emphasis correct and sound? 

{\bf RQ3:} Are there datasets to avoid when conducting estimation experiments?

{\bf RQ4:} Can we reasonably to declare a best effort estimator?


