This paper is 
a cautinary tale for data mining researchers.
Five years ago, we began with the following question; can the new
generation of data miners offer better effort estimates than
traditional methods? 
So far, our answer is ``somewhat, but not much''. 
Therefore, like Cohen~\cite{cohen95}, 
we would strongly advise data mining researchers working
on effort estimation to carefully compare their supposedly more sophisticated
approach against a wide range of alternatives.

To justify our conclusion, that many variants of data mining for effort
esimation add little value to that task, we report here a large experiment
with data mining over some COCOMO data.
The COCOMO effort estimation model is
is widely used by, amongst others, NASA and American software companies
working NASA sub-contracts. 
Unlike other effort estimators such as PRICE-S~\cite{park88}, 
SEER-SEM~\cite{jensen83}, or SLIM~\cite{putnam92}, COCOMO is a public domain 
with published data and baseline results~\cite{chulani99}. 
COCOMO-I, defined in 1981~\cite{boehm81}, contains 15 parameters,
plus lines of code and COCOMO-II, defined in 2000~\cite{boehm00a}
contains several more.  Our work focuses on COCOMO-I
since  there exists nearly
100 public domain examples of prior projects in the COCOMO-I 
format\footnote{ \url{http://promisedata.org/repository/data/nasa93/}}
but none in the 
COCOMO-II format\footnote{At the 2005 COCOMO forum, there were  
discussions about relaxing the access restrictions on the
nearly 200 examples used to build the \mbox{COCOMO-II} model.
To date, those discussions have not progressed.}.

After much experimentation, reported below, we found that
of all the 158 variants we explored,
a single linear model (with two simple extensions for column and row pruning),
performs as well or better than anything else.
For example, learners
which assume multiple linear models  (such as model trees~\cite{quinlan92})
or no parametric form at all (such as nearest neighbor
estimation~\cite{shepperd97}) perform relatively poorly.
Also,  elaborate searches did not add value to effort
estimation.  Exponential time   searching  for the best subsets of
the columns, performed no better than near-linear-time column pruning.

We make no claim these results hold over {\em all} learning methods.
There are any number of such methods and we have not
surveyed them all.  New open source data mining toolkits are appearing
with increasing frequency such as the R
project\footnote{\url{http://www.r-project.org/}},
Orange\footnote{\url{http://www.ailab.si/orange/}}, and the
WEKA~\cite{witten05}.  All the learners in all these toolkits can
be {\em stacked} by {\em meta-learning} schemes where the conclusions
of one data miner influences the next.  There exists one stacking
for every ordering of $N$ learners; so five learners can be stacked
$5!=120$ ways and ten learners can be stacked in millions of different
ways.  It is quite possible that another learner or another combination
of learners might refute our result. 

However, until it can be shown  that some alternate approach $X$ does
better than  the 158 variants explored below,
then our result
should be welcomed news to software engineers.
Engineers have a duty to apply accepted best practices to their tasks.
With so 
learning methods readily available, software engineers might be concerned
that they are generating effort estimates using less-than-best methods.
Our results suggest that they need not explore a very wide range of
effort estimation methods.
At least for COCOMO-style data, $\frac{154}{158}=97\%$ of the variants
explored below
add little or nothing to 
a linear model (with simple column and row pruning).

\section{Contextual Notes}
We do not claim that automatic data mining methods are the {\em only} way to do
effort estimation. Figure XXX lists best practices 
Further, we do not claim that automatic data mining methods are the {\em only} way to do
effort estimation. Figure XXX lists best practices 
{E}ffort
estimation
  methods can be divided into {\em model-based} and {\em expert-based} methods.
Model-based methods
use some algorithm to summarize old data and make predictions about new projects.
Expert-based methods use human expertise 
(possibly augmented with process guidelines
or checklists) is used to generate predictions.  

\begin{figure}
{\footnotesize
\begin{center}
\begin{tabular}{|p{3.3in}|}\hline

According to  Jorgensen~\cite{jorg04},  expert-based  best practices
include:
\bi
\item[1.]
evaluate estimation accuracy, but avoid high evaluation pressure;
\item[2.] avoid conflicting estimation goals;
\item[3.] ask the estimators to justify and criticize their estimates;
\item[4.] avoid irrelevant and unreliable estimation information;
\item[5.]  use documented data from previous development tasks;
\item[6.] find estimation experts with relevant domain background;
\item[7.] estimate top-down and bottom-up, independently of each other;
\item[8.] use estimation checklists;
\item[9.] combine estimates from different experts and estimation strategies;
\item[10.] assess the uncertainty of the estimate;
\item[11.] provide feedback on estimation accuracy; and, 
\item[12.] provide estimation training opportunities.
\ei
\\\hline
According to Boehm~\cite{boehm81,boehm00b}; Chulani~\cite{chulani98,chulani99}; 
Kemerer~\cite{kemerer87}; Stutzke~\cite{stutzke05}; 
Shepperd~\cite{shepperd97}; our
own  work~\cite{me04h,chen05,me05c};
and a recent tutorial at the
2006 International Conference of the International Society of Parametric Analysts~\cite{cpp06},
best practices for model-based estimation include at least the following:
\bi
\item[13.] {\em Reuse regression parameters} learned from prior 
projects on new projects;
\item[14.]  {\em Log-transforms} on costing data before performing 
{\em linear regression} to 
learn log-linear effort
models;
 \item[15.] {\em Model-tree} learning to generate models for non-linear relationships;
\item[16.] {\em Stratification}, i.e. given a database of past projects, and a current 
  project to be estimated,   just learn models from those                                 
   records from similar projects;
\item[17.] {\em Local calibration}, i.e. tune a general model to the local data
via     a small number of special                               
   tuning parameters;
\item[18.]
{\em Hold-out} experiments for testing the
learned effort model~\cite{me04h};
\item[19.] Assessing effort model uncertainty via the {\em performance deviations}
seen during the hold-out experiments of item \#17;
\item[20.] {\em Variable subset selection} methods
for minimizing the size of the learned effort model~\cite{miller02,kirsopp02,chen05,me05c}. 
\ei
\\\hline
This separation of model-based and expert-based methods is not a strict division since
some practices fall into both categories:
e.g. \#4 \& \#20 are similar as are \#10 \& \#19. Also, one way to view model-based methods
is that they seek algorithms to make maximal use of \#5. Further,
some research actively tries to combine the two approaches:
\bi
\item[21.] Shepperd's
{\em case-based reasoning} tools~\cite{shepperd97} explore algorithmic methods for
emulating expert analogical reasoning; 
\item[22.] Chulani \& Boehm's {\em Bayesian tuning
method}~\cite{chulani99} for regression models allows an algorithm 
to carefully  combine expert judgment with
the available data;  
\item[23.] This paper will argue for the use of heuristic {\em rejection rules}
to represent expert intuitions on how to rank different effort models.
\ei\\\hline
\end{tabular}
\end{center}}
\caption{Three different categories of effort estimation best practices:
expert-based (at top); model-based (in the middle); methods that combine
expert and model-based (at bottom).  }\label{fig:bests}
\end{figure}



Nor do we claim that our results hold beyond parametric
COCOMO-style effort estimation. Parametric estimation assumes that the effort 
conforms to some pre-determined analytical form (e.g. a single linear
relationship). COCOMO-style estimation
requires that domain information be encoded using a rigid set
of features such as programmer capability and software process
complexity. Other estimation methods allow for arbitrary
domain information and do not assume any parametric form (e.g.
the case-based reasoning~\cite{shepperd07,li07}).
There may well exist some non-parametric non-COCOMO style effort estimation
method that out-performs our 158 variants. 

Our challenge to the empirical software engineering community is to find
some learner, or combination of learners that indeed performs better
than the ones we have examined.
To that end, this paper defines an experimental procedure that could
demonstrate the superioty of some as yet untried learning methods.
We illustrate that experimental procedure
 with a study of 158 variants of different learners.
This study will show that, amongst those 158 variants,
a single linear models (augmented with simple column/row pruning)
performs as well as anything else.


Until such time as this challenged is answered, then our results 


The challenge is to refute the above results 
Firstly, at least for COCOMO-style estimation,


Elsewhere~\cite{baker07}
we have found no improvement using bagging~\cite{brieman96} and boosting~\cite{FreSch97} methods for
COCOMO-style data sets.
In this report, we examine 158 variants on numerous data mining schemes.

, we have found that one of four methods is always better than another 154 methods:
\bi
\item
A single linear model is adequate for the purposes of effort estimation.
All the methods that assume multiple linear models, such as model trees $(f)$,
or no parametric form at all, such as nearest neighbor $(i)$, perform
relatively poorly.
\item
Elaborate searches do not add value to effort estimation.
All the $O(2^F)$ column pruners do worse than near-linear-time column pruning. 
\item
The more intricate methods such as model trees do no better than other methods. 
\ei


Consider the task of software effort estimation in a standard
industrial setting. A talented engineer, is asked to give an estimate
of the cost of the next project. The estimate is required for a
planning meeting, scheduled for tomorrow.

Our
engineer looks into various effort estimation models.
A quick web search shows our engineer that there are any number of
easily downloadable data mining tools that can learn models from
data such as the R project\footnote{\url{http://www.r-project.org/}},
Orange\footnote{\url{http://www.ailab.si/orange/}}, and the
WEKA~\cite{witten05}.  The ready availability of these data mining methods is
somewhat confusing to our engineer since all these tools contain
many different learners.
A recent journal paper~\cite{me06d} proposed trying them a large number of them,
then seeing which worked best on the local data.
After a frantic night of scripting, the engineer found one with best 
mean magnitude of relative error (MRE) $\frac{abs(predicted - actual)}{actual}$.
Using that model, the engineer delivered an estimate to her boss.

The following scenarios describes the context of this paper. It is not
an imaginery scenario; indeed, it is exactly the situation that spawned
this research.
 
these toolkits can be {\em stacked} by {\em meta-learning} schemes
where the conclusions of one data miner influences the next.  There
exists one stacking for every ordering of $N$ learners; so (say)
five learners can be stacked $5!=120$ ways and ten learners can be
stacked in millions of different ways.


 containsIf our engineer works for on an American government sub-contract, they will probablly read the COCOMO literature
the American 
