The COSEEKMO premise was that the ``best'' effort estimation
method varies from domain to domain. Rather than select one as
universally ``best'' for all domains, it therefore makes more sense
to try as many as possible. 
Unfortuneately,  COSEEKMO has grown into an overly-cumbersome
system. It takes four days to apply
COSEEKMO's  155 methods 
to our standard suite of 19 data sets.
This is too slow since COSEEKMO
samples just a small percentage of the space of possible effort estimation methods.
We view estimators as data mining historical records.
All data miners have several biases: 
\bi
\item
a {\em language bias} constraining the ontology of their learned model;
\item
a {\em search bias} constraining how the learner navigates its search through a (potentially)
overwhelming number of output models; 
\item
a {\em evaluation bias} that states how the learned model will be evaluated (and
this may be used to offer advise to the search bias):
\item
an
{\em over-fitting avoidance bias} controlling how the learner prunes away possibly
spurious parts of the learned models~\cite{witten05}.  
\ei
For example, the 
\{language,search,evaluation, over-fitting\} biases of four data miners can be
recombined to create up to $4^4=256$
different data miners.

Not only is the number of data miners {\em exponential} on the 
 the number of biases, but there exists a {\em factorial} number of
combinations of those data miners.
Hence, it is hardly surprising that COSEEKMO explores only a portion
of the space of possible effort estimation methods:
\bi
\item
Myrteit et.al.~\cite{myrtveit05} categorize effort estimation methods into a 
tree with seven leaves. The
current version COSEEKMO mostly supports just two of those seven leaves (regression
analysis).
\item
COSEEKMO
does not currently explore clustering,
neural networks, case-based reasoning, bagging, or boosting.
\ei

As data mining tools become more freely available, we predict
that an increasing number of researchers
will apply a larger space of data mining methods to effort estimation. The COSEEKMO
experience is that even a relatively small number of methods (under 200) can become
cumbersome to use.
Hence, while {\em creating} new effort estimation methods is important research,
it is also important to be able to 
recognizing 
ineffective methods and {\em remove} them.

Recogizing ineffective methods is a complex task.
Myrtveit et.al.~\cite{myrtveit05} comment that:
\begin{quote}
{\em ...commonly used measuring 
procedures hinder meaningful comparative studies on 
prediction models.} 
\end{quote}
For example, based on the recommendations of standard texts in the data mining field,
COSEEKMO's rejection
rules made extensive use of standard statistical methods (t-tests) for comparing populations.
These tests assume that the results are drawn from a 
Gaussian distributions. As shown below, COSEEKMO's estimates turned out to be highly
non-Gaussian and included a number of very extreme outliers.  

Myrteveit et.al. also offer studies where the conclusions on which methods was ``best''
depended
on what subset of the available data is used to make conclusions.
For example, COSEEKMO takes a data set and extracts ten test records, selected at random.
An effort estimation model is learned on the remaining data, then applied to the test records.
After many such trials, COSEEKMO sorted  effort evaluation methods by the
mean magnitude of the relative error (MRE)% 
~\footnote{MRE= $\frac{abs(actual - predicted}{actual}$}.
A repeated result was that the  sort order was unstable over
over different randomized trials.

Finally, Myrteveit et.al. show that the decision regarding ``best'' method can
depend on the evaluation bias (which they call the ``accuracy indicator'').
For example, Kitchenham et.al.~\cite{kitc01} offer an insightful example
where the ``best'' method changed depending on whether they were assess using
mean MRE, or
magnitude of the relative error
or PRED(30)~\footnote{The percentage of MREs under than 25\%.}.
Foss et.al.~\cite{foss05} explore numerous evaluation measures
to find that they give contradictory conclusions.
They conclude:
\begin{quote}
it is probably futile to search for the Holy Grail:
a  single, simple-to-use, universal 
goodness-of-fit kind of metric, which can be applied with 
ease to compare a linear regression model with a nonlinear, 
arbitrary function approximator (for example,
an estimation-by-analogy model).
\end{quote}

Foss \& Myrtveit et.al. carefully document problems with
numerous evaluation biases. 
Their conclusions discuss general future research directions 
for possibly creating better evaluation biases.  However, no
specific evaluation biases are proposed.

Our comment here is three-fold. Firstly, an operational
evaluation method is required. Two of us (Hihn and Lum)
are charged with generating effort estimates for 
JPL space missions software. Since such estimates can be generated in multiple ways,
we have a duty of care to {\em not} use methods that are relatively poorer than others.

Secondly, problems with evaluation bias is unavoidable. Changing
the goals changes the product and there if there is no universally
accepted definition of the goals, there will be no universally
acceptance of the learned model.  Hence, we advice researchers to
use the simplest evaluation bias possible since any more elaborate
bias may require elaborate justification that may be disputed by
some member of the audience.  Following the advise of Shepperd
(personnel communitication), this study will use the median RE
($actual - predicted$).  since RE is simpler than the magnitude of
the RE and median measures are not as sensitive as mean measures
to outliers.

Also, in the sequel, we apply the advice of 
Kitchenham et.al. and use
ranked RE values, rather than the
use of their precise values; e.g. the precise values
\{0.1, 0.5, 055, 0.7\} have the rankings \{1,2,3,4\}.
While our sort orderings for precise values were unstable over different randomized
trials, the sort orderings obtained using ranked 
values are remarkably stable.
Also, the non-parametric statistics that process ranked values can avoid the Gaussian
assumptions that confused our original COSEEKMO study (a result that was first
reported for effort estimation in~\cite{kiti01}).  

XXX issue is not what works best always. but what often works. survey lots of data. our
goal is not to declare a winner- indeed no such winner ever exists and we advocate
the use of multiple methods. but there are too many multiples- some disappear. we want to change
the discussion from "what works best" to "under what biases can we find what usually is not best"
. here, we state one bias and argue why we selected that. 

the point being that there never will be one answer. there will be evaluation biases,
sometimes with known problems, sometimes without. further research will always evolve
our notion of ``best bias''. but in the meantime we are duty-bound to the SE industry
to offer some statement on what bias we recommend, why, and show what can be done with that
on multiple data sets.

\bi
\item
When this systematic error was corrected, we found the main result of this paper:
{\em most effort estimation methods perform relatively poorly}.
\item
After pruning away the relatively worst methods,
we found that {\em there is no one {\em best} remaining effort estimation method}. 
That is, different  data sets are best
processed
by different methods.
\ei

This last  finding endorses our
our previous work~\cite{me06d};
i.e. it is useful to have a tool like COSEEKMO for automatically  applying multiple methods
to the same data.

The first and second finding have major implications for effort estimation research.
T-tests are a widely recommended method~\cite{witten05},
the use of which must now be questioned for effort estimation.
(this paper uses non-parametric Mann-Wilcox tests).
Also, COSEEKMO can now be significantly
optimized. By ignoring the relatively poorer perform methods, COSEEKMO now runs in 20 minutes,
not four days.  
This means that COSEEKMO is now a suitable platform for comparatively assessing a wider range of methods.

The rest of this paper details the above results.
An extended set of findings is offered in the conclusion.
The significance of this paper is two-fold. Firstly, it shows that evaluating effort estimation methods is
more complex that previously reported (e.g.in~\cite{me06d}). Secondly, it shows that there is much
work to be done on the comparative evaluation of different
effort estimation methods. For example, this paper reviews
155 effort estimation
methods that the literature suggests {\em might} be useful, but finds that 
only four of them  are demonstrably
useful on our data suite.


\section{The Gaussian Assumption}
To assess rank $M$ estimation methods, COSEEKMO repeats the following procedure $R$ times:
\bi
\item
A test set of $T$ items is selected at random,
from the data.
\item
$M$  estimation methods are tuned on the remaining data. 
\item
$M*T$ {\em predicted} effort estimates are generated for all items in the test suite, using each tuned method. 
\ei
Typical values are $\{R=20,M=155,T=10\}$ in which case,
this process results in $R*M*T=62,000$ predictions. The predicted and actual values generated above can be summarized using the mean magnitude of the relative
error (MRE).
Given
$P$ predictions, then the mean MRE is calculated as follows:
\begin{equation}\label{eq:mmre}
mean MRE=\frac{\sum_i^P\frac{abs(actual_i - predicted_i)}{actual_i}}{ P} 
\end{equation}
Myrtveit et.al ~\cite{myrtveit05}  caution against use mean MRE. Their simulations show
that, when used to assess different methods, its results are highly unstable and that
the conclusions regarding ``which model is best'' depends on the particular sample at hand.
We explain their result as follows: effort estimates are highly non-Gaussians and contain
small numbers of extreme outliers. Such outliers confuse any measure based on mean values
(e.g. 100  homeless people and one millionaire have a mean income of 
$\frac{100*0 + 1,000,000}{11}=90,906$- a number that characterizes
the social conditions of none of the members of this population). 
Instability regarding ``which model is best''
can arise when the outliers change values: the presence of one
very large outlier  can XXX

Shepperd (personal communication) advises uses simpler measures for  effort estimation.
One such measure is the {\em relative error}:
\begin{equation}
RE=\frac{actual - predicted}{actual}
\end{equation}
COSEEKMO compares the performance of
a pair of methods and rejects one according to some statistical tests.
This repeats until there are no methods left to reject. The surviving methods are decleared $best$ and their
estimates are shown to the user.
If it can be shown that the performance deltas are {\em not} insignificant,
then their mean values can be compared with $<$ amnd $>$ to determine
{\em winners} and {\em losers}.

COSEEKMO uses t-tests to check for insignificance.
Given $N$ performance
deltas, between two methods, if the mean
and standard deviation of $N$ performance deltas are $\mu$ and $\sigma$ respectively
then, according to the t-test, the deltas are {\em insignificantly} small if 
\bi
\item
The sample
size is tiny (i.e. small $N$);
\item
Or the mean delta is tiny (i.e. small $\mu$);
\item
Or if  large
standard deviations ($\sigma$) dwarf the means;
i.e.  
\begin{equation}\label{eq:msn}
\left(\frac{\mu}{\sigma / \sqrt{N}} = \frac{\mu \sqrt{N}}{\sigma} \right)< X
\end{equation}
\ei
Here, $X$ comes from a table of standard threshold values. For example,
at $N=10$, a 95\% confidence that the deltas are insignificant comes
from a $X$ threshold of 2.228.

One issue with \eq{msn},
is that $\{X,\mu,\sigma\}$
are calulated assuming Gaussian
distributions. This is a standard assumption, endorsed by many texts
(e.g.~\cite{witten05}). To check this assumption,
we plotted (i)~the
{\em actual}
RE values
seen in $R*M*T$ applications of COSEEKMO;
and (ii)~the actual values approximated as a {\em Guassian}; i.e.
values sampled from a Guassian distribution  with the mean and standard 
deviations of the actual values. 
\fig{re}  shows four such plots.
From top to bottom,
the plots are of:
\bi
\item {\em Top:}
NASA ground systems;
\item NASA software wrtten around 1975;
\item
NASA embedded software; i.e.
software  developed within 
tight hardware, software and operational constraints 
\item {\em Bottom:}
Some FORTRAN-based software systems, collected by Boehm~\cite{boehm81}.
\ei

The presence of large outliers is an argument against using mean performance values.
The poor fit of the actual curves to the Guassian curve is an argument
against characterizing these distributions as Gaussians.
For these two reasons,
in the sequel, we will use non-parametric statistics to compare methods.

\section{Methods}

All COSEEKMO's methods use a $P*F$ table of historical data:
\bi
\item
Each {\em row}
of that table comprises information about one project $p$ including
the actual development effort (in months). 
\item
Each {\em column}
relates to one feature $f$ of a project such as lines of code (KLOC), 
schedule pressure (sced), etc. The features used in this
study come from Boehm's COCOMO81 work as are shown in
\fig{em}.
\ei
The rows of this table is  then divided into a $Train$ and $Test$ set. The different
methods are then applied to the training data to generate a model. This
model is then applied to
each project $p{\in}Test$.


COSEEKMO's 144 methods 
combine:
\bi
\item
 various {\em
learners} such as
standard linear regression, local calibration,
model trees, nearest neighbor algorithms, and
stacking;
\item
and various {\em pre-processors}
such as logging, row pruners, and column pruners.
\ei

Details on these learners and pre-processors are offered below. The
main point to be made here is that COSEEKMO's 144 combinations of
learners and pre-processors sample just  a small space of 
the set of possible
data miners. Repeating our introductory remarks,
before we can expand the space of COSEEKMO's learners,
we need first prune these 144 methods back to a more manageable set.

XXX explains much of ingrett's results

XXXX this paper refers to COCOMO when, more precisely, it really should be saying COCOMO-I.
Boehm et.al. released a major update to  COCOMO-I in 1999~\cite{boehm99}. However, none of
the data on the 161 projects used to learn the COCOMO-II model is publicly available. On the other hand, public domain
project data exists
for 156 COCOMO-81 projects. Hence, we use COCOMO-I rather than COCOMO-II since that allows us to access more data. Also,
it allows others to reproduce/ refute/ improve our results.

\subsection{Pre-processors}

Recall that all the above learners input a table of historical
data with $p$ projects in the rows and $f$ features in the columns.
\footnote{Since COCOMIN's column pruning and LOCOMO's row pruning
combine multiple learning methods, they are two examples of the stacking algorithms discussed
in the introduction.}

,,,XXX contrast via runtimes
Linearization

Row pruning

Column pruning


These learners exectu
These can be combined with row pruning methods, stacking
 
and model trees. 
 
\subsection{Class of Methods}
We use the notation $columnPruner+rowPruner+learner$ to describe the COSEEKMO
effort estimation methods. From COSEEKMO's 160 methods, we comment on 8 broad
classes of methods:

\bi
\item[a.]: {\em none+none+standard LC}. That is, using standard LC
 without any pruning. This is the currently accepted
best practice in the COCOMO community~\cite{boehm00}.
\item[b,]: {\em cocomin+none+lc}: i.e. row pruning, no column 

a,d represent standard practice

\item
best practice callibration method 
\ei

combines the above into 160 ways methods, each of which
contains a row pruner, a column pruner, and a learner.
For the sake of exposition, we group those methods into 8 classes:



To explain the other cost and benefits of nearest neighbor algorithms, we first
discuss the assumptions made by all the above methods:
\bi
\item
LC assumes
that
effort is exponential on KLOC; 
that the data contains certain features such as \fig{em};
and
that these features effect effort according to \fig{effortmults}.
We can hence call LC ``assumption-strong''.
\item
Linear regression and model trees are ``assumption-weak'' since
they make fewer assumptions. While they assume that the data fits 1 (or more)
linear models, they assume nothing about
the names of the  features.
or the presence of background knowledge on how factors
influence effort (e.g. \fig{effortmults}.
\item
Finally, nearest neighbor is ``assumption-less'' since it does
not even assume any parametric form (linear or otherwise) of the learned model.
\ei

(Note that while  nearest neighbors are assumption-less, they are not
``assumption-free''- see the above discussion on Wilson \& Martinez.)

%\section{Non-parametric Tests}
%\begin{figure}
%\small
The Mann-Whitney-Wilcox (MWW)
test checks if the central {\em ranks} of one treamtments's values
are different to the other.
For example,
suppose a treatment $A$ results  in
$N_1=5$ values \{5,7,2,0,4\}
and a treatment $B$ results in $N_2=6$ values \{4,8,2,3,6,7\}.
The null hypothesis in the values $A$ and $B$ do not differ.
If so,  then the raw values within treatments A and B would be 
about the same, on balance, 
so the rankings that derive from them would be evenly mixed. 
To implement the test, all the values are sorted: 
\begin{center}
{\small
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|}\hline
Samples& A & A & B & B & A & B & A & B & A & B & B\\
Values & 0 & 2 & 2 & 3 & 4 & 4 & 5 & 6 & 7 & 7 & 8\\\hline
\end{tabular}}\end{center}

The sorted values are ranked 
(and average ranks are used if two or more values are the same):
\begin{center}
{\small
\begin{tabular}{|c|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c|}\hline
Samples& A&  A  &B&  B&  A & B  &A&  B&  A&  B & B\\
Values&  0  &2  &2&  3&  4 & 4  &5&  6&  7&  7 & 8\\
Ranks&   1 &2.5&2.5 &4& 5.5&5.5 &7&  8& 9.5&9.5& 11\\\hline
\end{tabular}}\end{center}

%\caption{Using Mann-Whitney-Wilcox to generate $win$ and $loss$ scores
%for a set of methods.
%}\label{fig:wl}
%\end{figure}.

In the data mining literature, a widely used performance measure
is $wins-losses$.
In our case, 
this will a value ranging
from -144 to 144.
When repeated over the 19 subsets in our standard data set,
this generates $win - loss$ statistics for each method, for each data set.

\begin{figure*}[!t]
\begin{center}
\begin{minipage}{.4\linewidth}
\vspace{-0pt}
\footnotesize
\begin{tabular}{|r@{ = }l|r@{~}r@{~}r|}\hline
class& notes&		ties	&wins&	losses\\\hline
a&standard lc&2&6&0\\
b&cocomin+none+lc&3&5&0\\
c&cocomin+locomo+lc&7&1&0\\
d&none+manualStrat+lc&3&4&1\\
e&none+locomo+lc&5&1&2\\
f&m5pW+none+m5p&3&2&3\\
g&localW+none+lc&4&1&3\\
h&lsrW+none+lsr&3&1&4\\
i&none+nearest+none&0&0&8\\\hline
\end{tabular}
\end{minipage}
\begin{minipage}{.5\linewidth}
\includegraphics[width=4in]{LossesNasa93.pdf}
\end{minipage}
\end{center}
\caption{NASA93 results.
Left-hand-side shows ties/wins/losses for 9 classes of methods on
all 9 subsets of NASA93,
sorted by losses.
Right-hand-side shows results on individual subsets: for each subset,
160 methods are sorted by their losses.
Note that for this plot,   {\em better} methods score {\em fewer} losses.
The far right-hand0side columns
marks the position of 9 classes
of methods amongst the losses for each subset.
}\label{fig:nasa93}
\end{figure*}


\begin{figure*}[!t]
\begin{center}
\begin{minipage}{.4\linewidth}
\vspace{-0pt}
\footnotesize
\begin{tabular}{|r@{ = }l|r@{~}r@{~}r|}\hline
class& notes&		ties	&wins&	losses\\\hline
a&standard lc&0&8&0\\
e&none+locomo+lc&2&5&1\\
d&none+manualStrat+lc&2&5&1\\
b&cocomin+none+lc&2&4&2\\
c&cocomin+locomo+lc&2&4&2\\
g&localW+none+m5p&0&3&5\\
f&m5pW+none+lc&0&2&6\\
h&lsrW+none+lsr&0&1&7\\
i&none+nearest+none&0&0&8\\\hline
\end{tabular}
\end{minipage}\begin{minipage}{.5\linewidth}
\includegraphics[width=4in]{LossesCoc81.pdf}
\end{minipage}
\end{center}

\caption{COC81 results. Left-hand-side shows ties/wins/losses for 9 classes of methods on
all 7 subsets of Boehm's 1981 COCOMO-81
data set~\cite{boehm81}, sorted by losses.
Right-hand-side has the same format as \fig{nasa93}.
}\label{fig:coc81}
\end{figure*}


\fig{coc81} and \fig{nasa93} shows the patterns of $wins-losses$
for subsets of two public domain COCOMO data sets (described in the appendix).
Eight classes of the 144 methods are marked with various symbols. Often,
those symbols fall very close togehter and so, for the sake of clarity,
the position of the  $win-losses$ are repeated on the right hand side. These classes are labeled a,b,c,d,e,f,g,h (and are descibed
below).
In these two figures, the {\em higher} the $win-losses$, the {\em better}
the method. 

From the following comments are noteworthy.
Firstly, if one class of methods was always better than the others, then
it's symbols would appear appear {\em higher} than the rest.
When 
standard LC (class $a$)
appears highest in \fig{coc81}, other methods often beat it
in other data sets (\fig{nasa93}).
This is an important baseline
result. If it was otherwise, then this whole study would
offer no improvement over Boehm's 1981 effort modeling work.

Secondly, while standard LC and simple row pruning 
(class $a,b$)
appear higher than anything else
in \fig{coc81},
that this pattern does not re-occur in
\fig{nasa93}.
This finding confirms the COSEEKMO assumption; i.e. that it
is best to apply a range of effort
estimation methods to a given data
set.

Thirdly, if  one class of methods 
was always worse that the others, then its symbols would
appear {\em lower} than the rest.
Clearly, simple averaging of nearest neighbors (class $d$)
is 
worse than all others. That is, if possible,
we recommend using the COCOMO domain assumptions
rather than the no-assumptions policy of a standard nearest
neighbor algorithm.

Fourthly, linear-time column pruning (class $c$)
does better than complex
column pruning (class $e,f,h$). 

single models with simple row or col(class perform better than
single intricate models or multiple intricate models.

multiple intricate models (class $f$) do not perform
better than single simpler models. This effect is 

inctricate model 

No method does
so
 top of the always scores highest $wins-losses$ in all data
sets, then 
No method always scores highest $wins-losses$ in all subsets.
This supports the core COSEEKMO assumption that 
One 
class of the methods are those using an ANGEL-style nearest neighbor
algorithm.


