\clearpage
\section*{Comments on Reviewer Feedback}

All the reviewers were correct in guiding us away from the previous focus of the last
version of this paper. In the prior version we looked too hard at conclusion instability
and the benefits of Mann-Whitney. As Reviewer1 said
``in any case using Mann Whitney U is hardly new - it's just the non-parametric analog of the t test''. 
After deemphasizing Mann-Whitney, we found that we make our main point without wasting the reader's
time on an established technique.
 
Reviewer1 did this paper a great service by pointing out the Shepperd and Kadoda (2001, TSE) reference.
It turns out that we were basing our work on the wrong premise. As can be seen in our new
introduction, the Shepperd and Kadoda paper is a much better framing for this work.

Reviewer1 also commented
``The paper isn't very detailed on this but I would guess the 2 data sets are 
similar because they share the same features''. On reflection, this lead to the speculation
that one way to reduce conclusion instability is to make a better selection of mature
 features that a large community has experimented with. In this
draft, we stress the role of the COCOMO features in reducing uncertainty. The prior draft made
no comment on this possibility and we thank Reviewer1 for this insight.

Reviewer1 pointed out several small typographical and semantic errors in the text that we have now fixed.  
For example, keywords have now been added. Also, when
we first introduce the WRAPPER we added the comment suggested by this reviewer: 
\bi
\item
{\em WRAPPER and LOCALW are very thorough search algorithms that explore
subsets of the features, in no set order. In the worst case, this search is an exhaustive
examination of all combinations; i.e this search takes time $O(2^F)$.}
\ei

Reviewer1 asked for more details on our WRAPPER methods 
saying ``there are lots of things you can do with a wrapper so much more info is needed here''.
A full exposition of all these methods would take many pages so we added the following text to the appendix:
\bi
\item {\em
The above description of WRAPPER should be read as a brief
introduction to all the techniques associated with this kind of column pruner. A WRAPPER is powerful tool
and can be extensively and usefully customized (e.g. using a hash-table cache to hold
the frequently seen combinations; alternative search methods to best-first search; etc). We refer the interested
reader to the thorough treatment of the subject found in Miller~\cite{miller02} and Kohavi \& Johns~\cite{kohavi97}.}
\ei

Reviewer1 remarked that the reference 35 might be incorrect. To the best of our knowledge, it is and would accept guidance on this point.

Reviewer2 asked us to review our terminology:
``The authors demonstrate conclusion stability for a portfolio of 154 methods 
(I would prefer the term 'instances of a method' as just changing some parameters in a 
process does not create a new method) when applied to 19 subsets of two sources.''
Hence, we added this footnote to the introduction:
\bi
\item
{\em To be precise, COSEEKMO currently supports 15 learners and 8 row/column pre-processors
which can be applied to two different sets of internal tuning parameters.
In one view, this combination of (15+8*8)*2=158 different
estimation generation methods are not really different ``methods'';
rather it might be more accurate to call them ``instances of methods''.
This paper does not adopt that terminology for the following reason.
To any user of our tools, our menagerie of estimation software
contains 158 {\em oracles} that may yield different effort estimates.
Hence, in the view adopted by this paper, they are 158
competing methods that must be assessed and (hopefully)
pruned to a more management size.}
\ei

Reviewer2 correctly takes issue with our treatment of Gaussians. All that material is deleted in this draft.
Reviewer2 also correctly points out an error in our description of sparse data and the NEAREST method, which
we fixed in this draft.

Reviewer2 comments ``the main limitation of the research to me is the use of just two data sets 
(coming from 1981 and 1993). Even though there are subsets created, this still does not change anything
 on this limitation. This is especially true for the type of conclusions drawn to demonstrate conclusion stability.''.

In reply, we make two points:
\bi
\item
We use 19 data sets from two sources. Each source was a database storing a very diverse set of projects.
For example, while much of our data comes from NASA, NASA is a very diverse enterprise serviced by a very large
number of commercial contractors.
\item Figures 3 and 4 explore the overlap in our 19 data sets. While some overlap exists, for the most part our data falls
into different pools.
\ei
 
Reviewer3 correctly pointed out errors in out treatment of 
MRE and papers by Foss, Myrtviet and Kitchenham.
All that material is deleted from this draft.

Reviewer3 argues that we should not use an unpaired Mann-Whitney to study our results, and that a paired
test would be more appropriate. Paired tests are appropriate when different methods are applied to the same
train/test data. This is not the case when comparing methods that use row/column pruners vs methods that
do not. Hence, we still maintain we should use an unpaired comparison test.

\end{document}
\subsection{Symptoms of Instability}

KFM
caution that,
historically, ranking estimation methods has been done quite poorly. 
Based on an analysis of two (non-COCOMO)
data sets as well as simulations over artificially generated data set,
Foss et al. and Myrtviet et.al.
concluded that numerous commonly used methods such as the mean
MRE\footnote{MRE = magnitude of relative error $=abs(actual-predicted)/actual$.} are
unreliable measures of estimation effectiveness.
Also, the conclusions reached from these standard measures can vary wildly
depending on which subset of the data is being used for testing~\cite{myrtveit05}.

\begin{figure}
\begin{center}
\includegraphics[width=3.5in]{Parametric.pdf}
\end{center}
\caption{Results of 2 different runs of COSEEKMO comparing
two methods using mean MRE values. Points on the
Y-axis show the difference in mean relative error (MRE) between method1 and method2.
Lower values endorse method2 since, when such values occur,
method2 has a lower error than method1.}
\label{fig:parametric}
\end{figure}

\fig{parametric} demonstrates conclusion instability.
It shows two experimental runs. In each run,
30 times, effort estimate models were built for our 19 data sets using
two methods. Each time, an effort model was built from a randomly selected 90\% of the data.
Results are expressed in terms of the difference in mean MRE between the
two subsets; e.g. in Run~\#1, method1 had a much larger mean MRE than method2.

After Run~\#1, the results endorse method2 since that method either (a)~did better (lower errors)
as method1 or (b)~had similar performance to method1.
However, that conclusion is not stable. Observe in Run~\#2 that:
\bi
\item
The improvements of method2 over method1 disappeared 
in subsets 1,2,3,7, and 11.
\item
Worse, in subsets 1,2, and 11 method1 performed
dramatically better than method2.
\ei
The deviations seen in 30 repeats of the above procedure
were quite large: within each data set, the standard deviation
on the MREs were $\{median,max\} = \{150\%,649\%\}$~\cite{me06d}.
Port (personal communication) has proposed a bootstrapping method to determine the true performance distributions
of COSEEKMO's methods. That method would require $10^2$ to $10^3$ re-samples and, given
COSEEKMO's current run-times, it would take $10^2$ to $10^3$ days to terminate.

One troubling result from the \fig{parametric} study is that the number of training
examples
was {\em not} connected to the size of standard deviation. A pre-experimental intuition
was that the smaller the training set, the worse the prediction instability.
On the contrary, we found small and large instability (i.e. MRE standard deviation)
for both small
and large training sets~\cite{me06d}. That is, instability
cannot be tamed by further data collection. Rather, the data must be processed
and analyzed in some better fashion (e.g. U test described below).

These large instabilities explain the contradictory results in the effort estimation
literature.
Jorgensen reviews fifteen studies that compare model-based to expert-based estimation. 
Five of those studies found in
favor of expert-based methods; five found no difference; 
and five found in favor of model-based estimation~\cite{jorg04}.
Such diverse conclusions are to be expected if models exhibit large instabilities in their performance.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.5in]{graphs/re_nasa93_fg_g.pdf}
\includegraphics[width=2.5in]{graphs/re_nasa93_year_1975.pdf}
\includegraphics[width=2.5in]{graphs/re_nasa93_mode_embedded.pdf}
\includegraphics[width=2.5in]{graphs/re_coc81_langftn.pdf}
\end{center}
\caption{Relative errors seen in COSEEKMO's
experiments on four data sets. Thin lines show the actual
values. Thick lines show a Gaussian distribution that uses
mean and standard deviation of the actual values.
From top to bottom, the plots are of:
{\em (top)} NASA ground systems;
NASA software written around 1975;
NASA embedded software (i.e. software developed within tight hardware, software, and operational constraints);  
{\em (bottom)} some FORTRAN-based software systems.
}\label{fig:re}
\end{figure}

\subsection{Diagnosing the Cause}

The thin line of \fig{re} is drawn by sorting the 
relative error\footnote{RE = $(predicted - actual)/actual$} (RE)
seen in four of the subsets studied in \fig{parametric}.
Observe that while most of the actual RE values are nearly zero, an {\em infrequent}
number
(on the right hand side)
are {\em extremely large} (up to 8000 in the second plot). 
Such large {\em spikes} in RE result when
the predicted values are 
much larger than the actual values and result from
(1)~noise in the data or (2)~a training set that learns an overly steep
exponential function for the effort model. 

The size of the spikes in \fig{re} are remarkable. Research papers typically report RE values in the range 
$0\le RE\le3$\cite{me06d}.   
Such values are completely 
dwarfed by errors in the range of 8000 such as those seen in the second plot of \fig{re} (hence, most
of the thin lines in \fig{re} are flat). 
These very large, but
infrequent, outliers explain conclusion instability:
\bi
\item
Large outliers can make mean calculations
highly misleading. A single large outlier can make the mean
value far removed from the median\footnote{Median: 
the value below which 50\% of the values fall.}. 
\item
For data sets with only a small number of outliers (e.g. \fig{re}), the conclusions reached from
different subsets can be very different,
depending on the absence or presence of the infrequent outliers.
\ei

\fig{re} also illustrates how poorly standard methods assess the performance of effort estimation data.
Demsar~\cite{demsar06} offers a definition of {\em standard methods} in data mining.
In his study of four years of proceedings from the {\em International Conference
on Machine Learning}, Demsar
found that the standard
method of comparative assessment 
were t-tests over some form of repeated sub-sampling
such as cross-validation, separate subsets, or randomized re-sampling. 
Such t-tests assume that the distributions being studied are Gaussian and, as shown by the
thick line of \fig{re}, effort estimation results can be highly
non-Gaussian.
These thick lines show a Gaussian cumulative 
distribution function computed from the means and standard deviations of 
the actual RE values (the thin lines).
For example, the Gaussian approximation to the {\em actual values} of
\[\{1.1, 1.3, 1.5, 1.7, 2.1, 2.3, 2.7, 800\}\]
has a mean of 101.6 and a standard deviation of 282.2.
Observe how poorly such Gaussian distributions represent the actual  RE values:
\bi
\item
There exists orders of magnitude differences 
between the {\em actual} plots (the thin lines) and the {\em Gaussian} approximations
(the thick lines). 
\item
The Gaussian goes negative while none of our effort estimation methods assume that
it takes less than no time to build software.
\ei
\subsection{Fixing Instability}

The problem of comparatively assessing $L$ learners run on multiple sub-samples of $D$ data sets
has been extensively studied in the data mining community. 
T-tests that assume Gaussian distributions are strongly deprecated.
For example, Demsar~\cite{demsar06} argues that non-Gaussian populations are 
common enough to require a methodological change in data mining. 

After reviewing a wide range of
comparisons methods\footnote{Paired t-tests with and without the use of geometric means of the relative ratios;
binomial tests with the Bonferroni correction;
paired t-tests; ANOVA; Wilcoxon; Friedman},
Demsar advocates the use of the 1945 Wilcoxon~\cite{wilcoxon45} 
signed-rank test that compares the ranks for the positive and negative differences (ignoring the signs). 
Writing five years earlier,
Kitchenham et.al.~\cite{kitc01} comment that the Wilcoxon test has its limitations.
Demsar's report offers the same conclusion, 
noting that the Wilcoxon test requires that the sample sizes are the same.
To fix this problem, Demsar augments Wilcoxon with the Friedman test.

One test not studied by Demsar is Mann and Whitney's 1947
modification~\cite{mann47} to Wilcoxon rank-sum test (proposed along with his signed-rank test).
We prefer this test since:
\bi
\item
The Mann-Whitney U test does not require that the sample sizes are the same.
So, in a single U test, learner $L_1$ can be compared to all its rivals. 
\item
The U test does not require any post-processing
(such as the Friedman test) to conclude if the median rank of one population (say, the $L_1$ results)
is greater than, equal to, or less than the median rank of another (say, the $L_2,L_3,..,L_x$ results).
\ei

We used Mann-Whitney to compare a set of methods, computing the
$wins$, $ties$, and $losses$ for each.
Since we seek methods that can be rejected, the value of interest to us is
how often methods {\em lose}.
\begin{figure}
\begin{tabular}{|p{.95\linewidth}|}\hline
\footnotesize
The sum and median of A's ranks is
\[\begin{array}{ccl}
sum_A &=& 1 + 2.5 + 5.5 + 7 + 9.5 = 25.5\\
median_A& =&5.5
\end{array}\]
and the sum and median of B's ranks is
\[\begin{array}{ccl}
sum_B&=& 2.5 + 4 + 5.5 + 8 + 9.5 + 11 = 40.5\\
median_B&=& 6.75
\end{array}\]
The $U$ statistic is calculated from 
\mbox{$U_x=sum_x-(N_1(N_2+1))/2$}:
\[\begin{array}{c}
U_A = 25.5 - 5*6/2 = 10.5\\
U_B = 40.5 - 6*7/2 = 19.5
\end{array}\]
These can be converted to a Z-curve using:
\[
\begin{array}{rclcl}
\mu&= &(N_1N_2)/2&=&516.4\\
\sigma & = & \sqrt{\frac{N_1N_2(N_1 + N_2 + 1)}{12}}&=&5.477\\
Z_A & = & (U_A - \mu)/\sigma&=&-0.82\\
Z_B & = & (U_B - \mu)/\sigma&=&0.82\\
\end{array}
\]
(Note that $Z_A$ and $Z_B$ have the same absolute value.
In all case, these will be the same, with opposite signs.)

If $abs(Z) < 1.96$ then the samples $A$ and $B$ have the same median rankings (at
the 95\% significance level). In this case, we add one to both $ties_A$ \& $ties_B$.
Otherwise, their median values can be
compared, using some domain knowledge. In this work, {\em lower} values
are better since we are comparing errors. Hence:
\bi
\item
If $median_A < median_B$ add 1 to both $wins_A$ \& $losses_B$.
\item
Else if $median_A > median_B$ add 1 to both $losses_A$ \& $wins_B$.
\item
Else, add 1 to both $ties_A$ \& $ties_B$. 
\ei
\\\hline
\end{tabular}
\caption{An example of the Mann-Whitney U test.}\label{fig:mw}
\end{figure}

end{document}

