%XXX get rid of the fig{r1}r efgerences
%XXX add the case for calibration
\documentclass{sig-alternate}
\usepackage{color,url,graphicx,cite,amssymb}
\DeclareGraphicsExtensions{.epsf,.jpg,.eps,.pdf}

 \newenvironment{smallitem}
  {\setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{itemize}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt}
   \setlength{\itemsep}{0pt}}
  {\end{itemize}}

 \newenvironment{smallnum}
  {\setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{enumerate}
   \setlength{\leftmargin}{.2in}  \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt}
   \setlength{\itemsep}{0pt}}
  {\end{enumerate}}

\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallnum}}
\newcommand{\ee}{\end{smallnum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}
\begin{document}

%\conferenceinfo{} {May 15--21, 2005, St. Louis, Missouri, USA.}
\CopyrightYear{2005}
\crdata{}%1-58113-963-2/05/0005}

\title{Validation Methods for  Calibrating Software Effort Models}
\numberofauthors{3}
\author{
\alignauthor{Tim Menzies\\
  \affaddr{Computer Science}
  \affaddr{Portland State University}
  {\small \email{tim@menzies.us}}
}
\alignauthor{Dan Port$^\star$ Zhihao Chen$^\ddag$~\\
  \affaddr{$^\star$Uni. of Hawaii, Computer Science, Manoa;}\\
  \affaddr{$^\ddag$Center for Software Engineering,}\\
  \affaddr{Uni of Southern California}
  {\small \email{dport@hawaii.edu, zhihaoch@cse.usc.edu}}
}
\alignauthor{Jairus Hihn\\Sherry Stukes\\
  \affaddr{Jet  Propulsion Laboratory, Pasadena} 
  {\small \email{jairus.m.hihn@jpl.nasa.gov, sherry.stukes@jpl.nasa.gov}}}
}
\date{\today}
\maketitle
\begin{abstract}
COCONUT calibrates effort estimation models using an exhaustive search
over the space of calibration parameters in a COCOMO~I model.  This
technique is much simpler than other effort estimation method yet
yields PRED levels comparable to those other methods. Also, it does so
with less project data and fewer attributes (no scale
factors). However, a comparison between COCONUT and other methods is
complicated by differences in the experimental methods used for effort
estimation.  A review of those experimental methods concludes that
software effort estimation models should be calibrated to local data
using incremental holdout (not jack knife) studies, combined with
randomization and hypothesis testing, repeated a statistically
significant number of times.
\end{abstract}

\category{D.2.9}{Software Engineering}{Time Estimation}
\category{K.6.3}{Software Management}{Software Process}

\terms{Management, measurement, economics, experimentation}
\keywords{COCOMO, calibration, incremental cross-validation



\section{Introduction}



Software effort models work better when
calibrated with local data~\cite{stukes98,stukes91,stukes99,ferens98,habib98,lum02,shepperd97,boehm00b}. 
However data collection from industry is notoriously slow and
most industrial sites lack the resources to conduct extensive
calibration studies. For example,  few industrial
sites that could repeat a COCOMO~II-style calibration experiment~\cite{boehm00b}:

\bi
\item
An initial regression analysis was conducted on  
83 projects to generated the COCOMO~I
model~\cite{boehm81};
\item
Further data collection found  78 more projects;
\item
A DELPHI panel convened where experts offered their best judgment
on factors controlling software costs; 
\item
A Bayesian calibration
technique integrated the   DELPHI results
with data from the 83+78 projects.  
\ei

To shortcut the development time of an effort estimation model, the
COCOMO team added {\em calibration parameters} 
$a$ and $b$ in their model. They recommended
``having at least\ldots 10 points for calibration both
the multiplicative constant $a$ and the baseline exponent
$b$''.\cite[p175]{boehm81}.
Shepperd and Schofield make a similar conclusion, saying that their
effort estimation models stabilized after seeing 15 projects~\cite{shepperd97}.
However, the evidence for these claims is not conclusive.
Shepperd and Schofield only repeated their stability studies three times
and the variance in their results is rather large.
Also, when the COCOMO team tried two experiments with
calibration from 8 projects, the first one succeeded and the second failed.
They remark:
\begin{quote}
..the local calibration of (parameters) is more beneficial when
there is data on many projects versus just 8~\cite[p180]{boehm00b}.
\end{quote}

This observation was the
starting point for this  paper. If eight projects
is too little data then when is enough data enough?
Is 10 or 15 enough? If ``many
projects'' are required, how many is ``many''?

Clearly, more experimentation is required to understand the
calibration of software effort estimation models.  In defining a
validation experiment for our own COCONUT calibration
tool, we reviewed prior experiments on effort estimation.
Comparing any two of those experiments is difficult since their
experimental methods can be very different.  Hence, we
defined a new procedure that combines the best features of past
research into calibrating effort estimation models.  We commend this
procedure to the effort estimation community since it
controls for certain effects that can complicate calibration.
Further, it can report the minimum point at which enough project data
is enough.
 
\begin{figure*}
{\scriptsize
\begin{center}
\begin{minipage}{.12\linewidth}
\underline{KEY:}

xh= extra high

vh= very high

h=  high

n= nominal

l= low

vl= very low
\end{minipage}~
\begin{minipage}{.88\linewidth}
\begin{tabular}{r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{}l|p{1cm}@{}}
rely&data&cplx&time&stor&virt&turn&acap&aexp&pcap&vexp&lexp&modp&tool&sced&ksloc&&actual effort (months)\\\hline
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&2&.2&8.4\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&3&.5&10.8\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&5&.5&18\\
n&l&h&n&n&l&l&h&vh&h&n&h&n&n&n&6&&24\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&9&.7&25.2\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&7&.7&31.2\\
n&l&h&n&n&l&l&h&vh&n&n&l&n&n&n&11&.3&36\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&8&.2&36\\
h&n&h&n&n&n&n&n&n&n&n&n&n&n&n&6&.5&42\\
n&n&h&n&n&n&n&n&n&n&n&n&n&n&n&8&&42\\
n&l&h&n&n&l&l&h&vh&h&n&h&n&n&n&20&&48\\
n&n&n&n&n&n&n&n&h&h&n&n&n&n&n&10&&48\\
n&l&h&n&n&l&l&h&vh&h&n&h&n&n&n&15&&48\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&10&.4&50\\
n&n&h&n&n&n&n&n&n&n&n&n&n&n&n&13&&60\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&14&&60\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&19&.7&60\\
n&l&h&n&xh&l&l&h&h&n&n&h&n&n&n&32&.5&60\\
n&l&h&n&n&l&l&h&h&h&n&h&n&n&n&31&.5&60\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&12&.8&62\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&15&.4&70\\
n&l&h&n&n&l&l&h&vh&vh&n&h&n&n&n&20&&72\\
h&l&h&xh&xh&l&h&h&h&h&n&h&h&h&n&7&.5&72\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&16&.3&82\\
h&n&n&h&n&n&n&n&h&h&n&n&n&n&n&15&&90\\
n&h&h&vh&n&n&h&h&h&h&n&h&l&l&h&11&.4&98.8\\
vh&n&xh&h&h&l&l&n&h&n&n&n&l&h&n&21&&107\\
n&n&h&h&n&n&n&n&n&n&n&n&n&n&n&16&&114\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&25&.9&117.6\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&24&.6&117.6\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&29&.5&120\\
n&h&h&vh&n&n&h&h&h&h&n&h&l&l&h&19&.3&155\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&32&.6&170\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&35&.5&192\\
h&n&h&n&n&n&n&n&h&h&n&n&n&n&n&38&&210\\
n&l&h&n&n&h&l&h&h&h&l&vl&n&n&n&100&&215\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&48&.5&239\\
n&n&h&n&h&n&n&h&h&n&n&h&h&n&h&47&.5&252\\
n&h&vh&n&n&l&n&h&n&vh&l&n&h&n&l&70&&278\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&66&.6&300\\
n&l&h&n&xh&l&l&h&vh&vh&n&h&n&n&n&150&&324\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&66&.6&352.8\\
n&l&h&n&n&l&l&h&vh&vh&n&h&n&n&n&100&&360\\
n&l&h&n&n&l&l&h&n&n&n&vl&n&n&n&100&&360\\
h&n&h&h&h&l&h&n&h&n&n&n&l&vh&n&50&&370\\
h&h&n&n&n&l&l&n&h&h&n&h&n&n&n&79&&400\\
n&n&n&n&n&l&n&h&vh&vh&l&h&h&n&n&190&&420\\
n&n&h&n&n&n&n&n&n&h&n&h&h&h&n&90&&450\\
h&l&h&n&n&l&l&n&n&n&n&h&h&n&l&115&.8&480\\
n&h&h&vh&n&n&h&h&h&h&n&h&l&l&h&78&&571.4\\
h&n&vh&h&h&l&h&h&n&n&h&h&l&vh&h&101&&750\\
n&vh&h&vh&vh&l&h&vh&h&n&l&h&vh&vh&l&161&.1&815\\
h&h&l&n&n&n&h&h&h&n&n&n&h&n&n&284&.7&973\\
vh&h&h&vh&vh&n&n&vh&vh&vh&n&h&h&h&l&227&&1181\\
n&h&h&vh&h&l&h&h&n&h&l&h&h&n&l&177&.9&1248\\
n&h&l&n&n&h&n&h&h&n&n&n&h&h&n&282&.1&1368\\
h&n&h&h&h&l&h&n&h&n&n&n&l&vh&n&219&&2120\\
l&n&n&n&n&l&l&h&h&vh&n&h&l&l&h&423&&2300\\
h&l&h&n&n&l&l&n&n&h&n&n&h&vl&n&302&&2400\\
h&n&h&h&h&l&h&n&h&n&n&n&l&vh&n&370&&3240\\
\end{tabular}
\end{minipage}
\end{center}}
\caption{NASA effort data used in this study. Available on-line at
the {\em PROMISE} repository of public domain software engineering data sets:
\protect\url{http://promise.site.uottawa.ca/SERepository/datasets/cocomonasa.arff}.}\label{fig:data}
\end{figure*}

The rest of this paper reviews the need for local calibration,
documents the effort models used in our work, discusses prior
experiments in effort estimation, and presents results from our
preferred method. Based on those results, we speculate that the
COCONUT tool is an effective and simpler alternative to traditional
effort estimation tools.  This paper is offered as a baseline result
with the hope that other researchers will try to reproduce and
out-perform our results.

Before beginning, we digress for a comment on the problems
associated with collecting industrial software engineering
data. 
In the future, we hope to apply our methods to data from
many sources. However, for now,
the methods of this paper have only been
applied to the historical
NASA data of \fig{data}. 
There is a good reason for this.
Collecting such new data is
very difficulty. Software projects are notoriously
difficult to control\footnote{Recall the 1995
report
of the Standish group that described a \$250 billion
dollar American software industry where 31\% of projects
were canceled and 53\% of projects incurred costs exceeding
189\% of the original estimate~\cite{standish95}.}.
Corporations are therefore  reluctant to expose their
own software development
record to public scrutiny. 

Given this data shortage, it is important that the available
data is put to best use. Therefore, much of this paper is devoted to a
methodological discussion on {\em how} to best calibrate a cost model.


\section{The Case for Local Calibrations}

Before discussing {\em how} to perform
local calibration, this section discusses
{\em why} such calibrations are useful.

 
The case {\em against} calibration is statistical: if a large
database of examples exists, then the distributions in that large database should
include the particulars of some current project.  Hence, it would seem advantageous
to {\em not} calibrate in order to  make full use of the knowledge in the large database of examples.



The problem with this argument is that it assumes a database large
enough to capture the underlying distributions of the software
development process. \fig{samples} suggests that we do not have such a
``large enough'' database. 

That figure  shows the percentage of times
an attribute was selected  by a $N$=4 hill climbing
search. In hill climbing,
if adding a
attribute to a set of currently selected attributes does not improve
the score (in this, effort estimation accuracy), 
then that set is marked ``stale''. If that set
remains ``stale'' even after $N$ more additions, then the last $N+1$
added attributes are rejected and the hill climber searches
elsewhere.   Otherwise, the attributes in the non-stale set are selected
as being influential on the score.

This hill-climbing search was conducted on
COCOMO~II data divided into pre- and post-1990
projects. Ten experiments were performed with each of the two
divisions. Each experiment performed hill climbing on 
a
randomly selected 90\% of the division data.
\fig{samples} only shows the {\em most often influential  attributes};
 i.e. those that were selected
in five or more of the experiments in {\em either}
 the pre- or post-1990 sample. 

Some of those
 attributes were
selected with similar frequencies in both samples
($pcon$, $site$, $size$). 
However, the remaining influential attributes
{\em had very different influence on effort} in the pre- and post-1990 samples.
Some of the changes we can't explain such as the dramatic
drop in the importance of $pcap$ (programmer capability).
Other chances correspond to
general trends in the computer industry.
For example,
the $team$ cost driver (i.e. team cohesion) 
has become more significant. This is not unexpected, given the 
 boom in  knowledge management, experience management
and organization learning since 1990.  Another interesting change is
that the $stor$ cost driver (i.e. main memory storage constraint) has
 become dramatically less important. After all the development in
hardware and software for the last two decades, main storage is more readily 
available.

Regardless of the reason for the change in the influence of the
attributes, the general lesson of \fig{samples} is that data collected
in one context (e.g. pre-1990) may not be completely relevant to some
other context.  An often repeated result in effort estimation is that
{\em stratification} of the available data into similar projects, can
improve the effectiveness of estimation.  Similar projects have less
variation and so can be easier to calibrate. 
A commonly used measure of predictive accuracy is the ``PRED'' value
defined is \tion{perform}.
Chulani
et.al.~\cite{chulani99} report that stratifying data can improved
their ``PRED'' values anywhere between 5\% to 12\%.   Shepperd and
Schofield
report even more dramatic improvements in ``PRED'' from 
4\% to 44\%~\cite{shepperd97}.



\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{impact.pdf}
\end{center}
\caption{Influence of various COCOMO~II attributes within two
sub-samples of the 2004 database found by a hill climbing ($N$=4) search.
}\label{fig:samples}
\end{figure}





\section{Effort Estimation Models}

Having made the case that 
local calibration is useful,  
this section describes the BASE and COCOMO~I
effort estimation model used in our calibration experiments.


There are many effort estimation models\footnote{See the excellent
review in~\cite[Chapter 2]{clark97}.}  and the one we used here 
was chosen based on the available publishable data.  Our intent
was to define a repeatable effort estimation experiment so that others
may repeat to refute or improve our results.  The COCOMO~II data is
{\em not} publishable since it was collected on condition of
confidentiality with the companies supplying the data.

\fig{data} showed the data used in this study. That data  
comes from 60 NASA projects from different centers
for projects from the 1980s and 1990s. Since it is from NASA,
it is  stratified
to just aerospace applications. 
The data is in  COCOMO I format so it lacks the scale
factors (defined below) used in COCOMO II. 


COCOMO measures effort in calendar months of 152 hours
(and includes development and management hours).  COCOMO assumes that
the effort grows more than linearly on software size.  In our
experiments, we will model this assumption two ways. The BASE model
assumes:

{\footnotesize
\begin{equation}\label{eq:base}
months=a* KSLOC^b.
\end{equation}}
where $a$ and $b$ are domain-specific
parameters and KSLOC is estimated directly or computed from  function
points.

In our experiments, this BASE
equation is compared to
a more  sophisticated model:

{\footnotesize
\begin{equation}\label{eq:one}
months=a*\left(KSLOC^b\right)*\left(\prod_jEM_j\right)
\end{equation}}
%XXXwhat about repeatability (folks repeating other people's work), straw man.
%also, acceptable if >= cocomo1 stratified results
%competent if pred(30) cocomo stratified results
%good if pred(25)> 83%
\eq{one}
is Boehm's  COCOMO~I model~\cite{boehm81}.
$EM_j$ is one of a set of {\em effort
multipliers} shown in \fig{em}.  

In COCOMO~I, the exponent on KSLOC was a single value
ranging from 1.05 to 1.2.  
In COCOMO~II, the exponent $b$ of \eq{one}
was divided into a constant,
plus the sum of five {\em scale factors} which
modeled issues
such as ``have we built this kind
 of system before?''. 
The COCOMO~II effort multipliers are similar but
 COCOMO~II dropped one of the
\fig{em} parameters; renamed some others;
and 
added a few more (for ``required level of reuse'',
 ``multiple-site development'', and ``schedule pressure'').

The numeric values of the effort multipliers are shown in \fig{effortmults}.
These were learnt by Boehm after a
regression analysis of the projects
in the  COCOMO~I data set~\cite{boehm81}.
The multipliers fall into three groups: those that are {\em positively} correlated
to more effort; those that are {\em negatively} correlated to more effort;
and a third group containing just schedule information. In COCOMO~I, $sced$
has a $U$-shaped correlation to effort; i.e. giving programmers either 
{\em too much} or
{\em too little} time to develop a system can be detrimental.

The last column of \fig{effortmults} is $\frac{EM_{max}}{EM_{min}}$
and shows the overall
\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{l|r@{:~}l|}\cline{2-3}
increase&acap&analysts capability\\
these to&pcap&programmers capability\\
decrease&aexp&application experience\\
effort &modp&modern programing practices\\
&tool&use of software tools\\
&vexp&virtual machine experience\\
&lexp&language experience\\\cline{2-3}
\multicolumn{3}{c}{}\\\cline{2-3}
      &sced&schedule constraint\\\cline{2-3}
\multicolumn{3}{c}{}\\\cline{2-3}
decrease&stor&main memory constraint\\
these to&data&data base size\\
decrease&time&time constraint for cpu\\
effort&turn&turnaround time\\
&virt&machine volatility\\
&cplx&process complexity\\
&rely&required software reliability\\\cline{2-3}
\end{tabular}}
\end{center}
\caption{{
COCOMO~I effort multipliers.}}\label{fig:em}
\end{figure}
\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{c|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r|r}
    &very&&&&very&extra&productivity\\
    &low&low&nominal&high&high&high&range\\\hline
acap   &1.46   &1.19   &1.00   &0.86   &0.71   &&2.06\\
pcap   &1.42.  &1.17   &1.00   &0.86   &0.70 &&1.67\\
aexp   &1.29   &1.13   &1.00   &0.91   &0.82   &&1.57\\
modp   &1.24.  &1.10 &1.00 &0.91 &0.82 &&1.34\\
tool   &1.24 &1.10 &1.00 &0.91 &0.83 &&1.49\\
vexp   &1.21 &1.10 &1.00 &0.90 &  &&1.34\\
lexp   &1.14 &1.07 &1.00 &0.95 &  &&1.20\\\hline
sced   &1.23 &1.08 &1.00 &1.04 &1.10 &  &\\\hline
stor   &       &       &1.00   &1.06   &1.21   &1.56&-1.21\\
data   &    & 0.94 &1.00 &1.08 &1.16&&-1.23\\
time   &  &    &1.00   &1.11   &1.30   &1.66&-1.30\\
turn   &       &0.87   &1.00   &1.07   &1.15   &&-1.32\\
virt   &       &0.87   &1.00   &1.15   &1.30   &&-1.49\\
cplx   &0.70 &0.85 &1.00 &1.15 &1.30 &1.65&-1.86\\
rely   &0.75& 0.88& 1.00 & 1.15 & 1.40&&-1.87\\\hline
\end{tabular}}
\end{center}
\caption{{
COCOMO I effort multiplier values.}}\label{fig:effortmults}
\end{figure}
effect of a single effort
multiplier. For example, moving {\em acap}
(analyst experience) from very low to very high will
{\em most decrease} effort while 
moving {\em rely} (required reliability) from
very low to very high will {\em most increase} effort.

There is more to COCOMO that the above description.  The COCOMO~II text~\cite{boehm00b} is over 500 pages long and
offers all the details needed to implement data capture and analysis
of COCOMO in an industrial context. 

%\vfill\eject
\section{Performance Measures}\label{sec:perform}

Our preferred experiment is a combination of the best
techniques seen in the 
effort
estimation calibration experiments of \fig{predmmre}.
Those experiments all take the form:

{\footnotesize
\[\{Repeats,Train,Test\}\]}
That is, $Repeats$ times the available  data 
is divided into a $Train$ set 
and a $Test$ set.
Some tool is applied to the
training set to learn a model or calibrate parameters within a model.
The model or parameters are  then frozen and
applied to the $Test$ set. 

When the $Train$ and $Test$ set do not intersect
this is called a {\em holdout} study. Holdout studies
check the utility of the calibrated parameters using
data not used during calibration. 
If the calibrations are tested on the training set,
this is not a holdout study. Two non-holdout studies
are shown in \fig{predmmre}: see the $|Test|=0$ entries.
An extreme form of holdout
studies are the {\em jack knife} studies
used in rows $\{d{\ldots}l\}$ of \fig{predmmre})
where
{\em every} example is removed
one at a time, training occurs on the remaining data, then testing
is conducted on the single holdout example. 

The results of \fig{predmmre}
are expressed in terms of  MMRE and PRED(N)
(for notes on other performance measures,  see \cite[chapter 5]{witten99}).
MMRE and PRED are computed from the 
 {\em relative error}, or RE, which is the relative size
of the difference
between the actual and estimated value:

{\footnotesize
\[
RE_i = \frac{estimate_i - actual_i}{actual_i}
\]}
Given a data set of of size $D$, a $Train$ing set of 
size $(X=|Train|)\le D$, and a 
$test$ set  of size $T=D-|Train|$, then 
the mean magnitude of the relative error, or MMRE, is the 
percentage of the absolute values of the relative errors,
averaged over the $T$ items in the $Test$ set; i.e.


{\footnotesize
\[\begin{array}{c}
MRE_i=abs(RE_i)\\
MMRE_i=\frac{100}{T}\sum_i^T MRE_i
\end{array}\]}
PRED(N)
reports the average percentage of estimates that were within N\% of
the actual values:

{\footnotesize
\[
PRED(N) = 
\frac{100}{T}
\sum_i^T\left\{
\begin{array}{l}
1\;if\;MRE_i\;{\le}\;\frac{N}{100}\\
0\;otherwise
\end{array}\right. 
\]}


For example,
e.g. PRED(30)=50\% means that half  the estimates
are within  30\% of the actual. 
Shepperd and Schofield comment that: 
\begin{quote}
MMRE is fairly conservative with a bias
against overestimates while Pred(25) will identify those
prediction systems that are generally accurate but occasionally
wildly inaccurate~\cite[p736]{shepperd97}.
\end{quote}





\begin{figure*}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{p{0.5cm}|c|r|r|r|p{2cm}|p{3in}|p{1cm}}
id&Holdout?&Repeats&$|Train|$&$|Test|$&Best Results&Notes&Reference\\\hline
a&y&3&105&10&PRED(25)=75\%& Wittig\&Finnie studied neural networks predicting
software effort 
that yielded an average PRED(25) of 75\%&\cite{wittig97}\\\hline  
b&n&        & &   &MMRE=70\%& Back propagation neural nets &\cite{srinivasan95}\\\hline
c&n&1       &  8 &    0&PRED(20)=100\%& In the COCOMO~II
experiment discussed in the introduction,
a PRED(20)=75\% model
was calibrated to a PRED(20)=100\%.&\noindent\cite{boehm00b} p175-181\\\hline

d$_0$&y&28&27&1&PRED(25)=21\%&The Mermaid data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline
d$_1$&y&?&?&1&PRED(25)=39\%&The Mermaid E data: stratified, estimation by analogy&~\cite{shepperd97}\\\hline

d$_2$&y&?&?&1&PRED(25)=25\%&The Mermaid N data: stratified, estimation by analogy&~\cite{shepperd97}\\\hline

e&y&21&20&1&PRED(25)=23\%&The Real-time1 data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline

f&y&24&23&1&PRED(25)=33\%&The Albrecht data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline
g&y&38&37&1&PRED(25)=39\%&The Finnish data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline

h&y&15&14&1&PRED(25)=40\%&The Kemerer data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline

i$_0$&y&77&76&1&PRED(25)=42\%&The Desharnais data: unstratified, estimation by regression&~\cite{shepperd97}\\\hline
i$_1$&y&?&?&1&PRED(25)=47\%&The Desharnais 1 data: stratified, estimation by analogy&~\cite{shepperd97}\\\hline

i$_2$&y&?&?&1&PRED(25)=48\%&The Desharnais 2 data: stratified, estimation by regression&~\cite{shepperd97}\\\hline

i$_3$&y&?&?&1&PRED(25)=70\%&The Desharnais 3 data: stratified, estimation by analogy&~\cite{shepperd97}\\\hline

j&y&21&20&1&PRED(25)=43\%&The Atkinson data: unstratified, estimation by regression&~\cite{shepperd97}\\\hline

k&y&18&17&1&PRED(25)=44\%&The Telecom 1 data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline

l&y&33&32&1&PRED(25)=51\%&The Telecom 2 data: unstratified, estimation by analogy&~\cite{shepperd97}\\\hline



m&n&1&83&0&PRED(30)=64\%&
Boehm improved the COCOMO-II 1997  PRED(30) results
from 52\% to 64\%, before and after stratification. &\cite{boehm00b}\\\hline
n&y&15&121&40&PRED(30)=69\%&Chulani (of the COCOMO II team)  
conducted  experiments in
calibrates the COCOMO~II 2000 software effort model
using a Delphi panel and a Bayesian calibration method.&\cite{boehm00b}\\\hline
o&y&3 & 67& 10&MMRE=21\%&
Mair et.al. compared the performance of adaptive neural nets (ANN),
case-based reasoning, rule induction, and linear regression.
In that study, the ANN produced the best MMRE's of
21,53,66\% in each of the three runs.&\cite{mair00investigation}\\\hline

p&y&10&99&10&PRED(25)=83.3\%&
Boetticher reported a $10*99/10$ study that yielded
software effort estimates with an average PRED(25)/MMRE across the ten
repeats of of 83.3\% /14.7\% (respectively).& \cite{boetticher03}\\\hline
&&&56&4&PRED(30)=70\%& &\\
&&&52&8&PRED(30)=70\%&&This\\
q&y&30&48&12&PRED(30)=70\%&&study.\\
&&&44&16&PRED(30)=70\%&&\\
&&&\ldots&\ldots&&&\\
\end{tabular}}
\end{center}
\caption{A sample of effort estimation experiments in the 
literature. Missing numbers, marked with ``?'', come
from a lack of information in the source document.}\label{fig:predmmre}
\end{figure*}

The rows labeled $\{d,i\}$  in \fig{predmmre}
illustrate how stratification can improve effort estimation.
In those experiments,
the unstratified data of $\{d_0,i_0\}$ was divided
into related subsets to form $\{d_1,d_2\}$ and $\{i_1,i_2,i_3\}$.
The $d_0/d_1$ comparison shows the least improvement: 21\% to 25\% while
the $\{i_0/i_3\}$ comparison  show the greatest improvement:
42\% to 70\%.


An interesting variant on the standard calibration experiment is the
{\em incremental holdout experiment} used by  (e.g.)  Shepperd and Schofield
which explores ``the dynamic behavior of effort prediction by
simulating the growth of a dataset over
time''~\cite[p741]{shepperd97}.  That is, as the study progresses,
more and more of the data is moved from test to training in
units of size $\delta$.  That is, holdout  experiments
convert a $\{Repeats,Train,Test\}$ experiment into

{\footnotesize\[\{Repeats,Train,Test,\delta\}\]}
For example, if at every step in a holdout study, one example is
moved from $Test$ to $Train$r then such a 
\[\{Repeats,
|Train|=1,|Test|=100,\delta=1\}\]
study could look like this:

{\footnotesize
\begin{verbatim}	
  while( Repeats-- ) 
      randomizeOrder
      train on item  1     test on items 2..100
      train on items 1,2,  test on items 3..100
      train on items 1,2,3 test on items 4..100
      ..
      train on items 1..97 test on item 98,99,100
      train on items 1..98 test on item 99,100
      train on items 1..99 test on item 100
\end{verbatim}}

\section{Experimental Method}

Each of the \fig{predmmre} studies has merit, but their results 
 are hard to compare since they mostly use different data sets
and different experiment methods.  To remedy that situation, we
describe below a single experimental design that combines {\em all}
the best features of the above work. Hopefully, this design will
become a template for future reports.

Like many of the researchers listed in
\fig{predmmre},
we endorse holdout studies.
The alternative to holdout studies
is to assess the learned theory on the data used to generate it.
Such non-holdout studies are useful for finding patterns in historical
data. However, if the goal is to generate models that have some useful
future validity, then the learned theory should be tested on data not
used to build it.  Failing to do so can result in a excessive
over-estimate of the learned model- for example, Srinivasan and Fisher
report an 0.82 correlation between the predictions generated by their
learned decision tree and the actual software development effort seen
in their training set~\cite{srinivasan95}. However, when that data was
applied to data from another project, that correlation fell to under
0.25.

We endorse holdout studies, but not jack knife studies.  Witten and
Frank~\cite{witten99} argue against jack knifing, offering examples
where jack knifing can lead to widely inaccurate results. Witten and
Frank's preferred alternate procedure, which is somewhat of a standard
in the data mining literature, is to use holdout sets of around 10\%
of the available data (when the data set is large) or repeated
experiments with a 33\% holdout (when the data set is small).  In
partial support of Witten and Frank's view, note that the Shepperd and
Schofield experiments (rows $i{\ldots}l$) of \fig{predmmre} and
generally {\em lower} than, say, the COCOMO experiments (rows
$\{c,m,n\}$), or Boetticher's neural net study (row $p$).

With the exception of Shepperd and Schofield,
when researchers conduct holdout studies, they usually use large
training sets and much smaller test sets; e.g.  see the training sets of size
67,99,105,140 and test sets of size 10,10,10,40  used by Mair, Boetticher, Wittig, Chulani in rows
$\{o,p,a,n\}$ respectively.  Hence, we recommend the use of holdout 
studies since these can report how well a system behaves as
the amount of available data is reduced.

Holdout  studies should be repeated more
often than (e.g.) the three runs reported by Shepperd and
Schofield. After three repeats, those researchers reported
convergence in their estimator at between 10 to 15
projects. But their results (Figures 1 and 2,
\cite[p741]{shepperd97}) show a huge variance. We recommend 
repeating holdout studies 30 times in order to gain statistics
on the variance.

When repeating holdout
experiments,
it is good practice to randomize the order of  the input examples
(e.g. as done  by Shepperd and Schofield).
Many algorithms have an {\em order effect} such that their performance
changes dramatically if the inputs are re-ordered
(the classic example is the ``Quick-Sort'' algorithm  that performs  badly
on input that is already sorted in descending order).
Kermer reported order effects in his analysis data
from 15 projects: if training was restricted to 9 particular
projects, the learning was far more successful~\cite{kemerer87}.
Randomization
avoids such order effects.


Some researchers try their calibration method on data sets used by
other researchers
(e.g.~\cite{shepperd97,srinivasan95,mair00investigation}), but most
don't.  
To (partially) reduce this problem, we follow the lead of 
Srinivasan and  Fisher and  place all
our data sets on the web\footnote{See 
\url{http://www.vuse.vanderbilt.edu/~dfisher/tech-reports/raw-TSE-95}.
and \url{http://promise.site.uottawa.ca/SERepository/datasets-page.html}.}
We encourage
other researchers to do the same.


Experimental results should be reported using more than just mean
PRED(N). Chulani et.al. report min,mean,max values seen in their holdout 
studies while Shepperd and Schofield offer performance graphs that
let the reader see the variance in their technique.  We recommend
going further and
reporting the results results using means {\em and} standard
deviations seen over the 30 repeats.

It is good practice to benchmark elaborate or resource intensive techniques
against simpler alternatives.
In his text on empirical AI~\cite{cohen95}, Cohen argues that 
such comparisons are 
important since, sometimes, 
sophistication is superfluous since the simpler method can achieve 
similar results to the more complex method (e.g. as seen in Holte's
famous study {\em
Very Simple Classification Rules Perform Well on
                  Most Commonly Used Datasets}~\cite{holte93}).
We will compare our COCONUT method with a {\em straw man}
alternative (which uses the BASE equation of \eq{base}).
When comparing two techniques,
it is also good
practice to use t-tests to compare means and deviations
of two alternative techniques.

Lastly, we report results in terms of PRED(N), not MMRE.
This is a pragmatic decision- we have found PRED(N) easier to explain
to business users than MMRE. Also, there are more PRED(N) reports
in the literature than MMRE. This is
perhaps due to the influence of the COCOMO researchers who reported
their 1999 landmark study  using PRED(N)~\cite{chulani99}.


\section{COCONUT}

This section describes COCONUT, 
a calibration tool that implements an incremental holdout study
for the BASE model of \eq{base} and the COCOMO~I model of \eq{base}.
COCONUT is short for ``COCOMO, \underline{N}ot \underline{U}nless \underline{T}uned''.

COCONUT was inspired by Cohen's recommendation, discussed above,
that complex methods should be compared to very simple ones.
\fig{predmmre} shows studies where effort estimation has been attempted
using analogy, rule induction, neural nets, Bayesian tuning, regression, etc etc.
To the best of our knowledge, no one has previously attempting
calibration using just an exhaustive search through the range of possible $a$
and $b$ parameters.  Such a search is tractable: just the two
for-loops of lines 5/6 of the {\tt learn} function of \fig{entire}.

\begin{figure}[!t]
{\scriptsize
\begin{verbatim} 
   function learn() { # from 'D' examples
 1.   Repeats=30
 2.   while(Repeats--) {  
 3.     randomizeOrderOfProjects()
 4.     for(x=2;x<=D;x += 3) { 

          ###### Train on 'x' examples
 5.       for(a=2;   a <=5; a += 0.3) {
 6.         for(b=1; b<=1.3;  b += 0.05) {
 7.           sum=test(1,x,a,b,Pred)
 8.           if (sum < least) { least=sum
 9.                              BestA=a
10.                              BestB=b }}}

          ###### Test on 'D-x' examples
11.       failures = test(x+1,N,BestA,BestB,Pred)
12.       print Repeats " " i " " failures}}}

    function test(start,stop,a,b,est,act,i,mmre) {
13.   failures=0
14.   for(i=start;i<=stop;i++) {
15.     est   = a*size(i)^b*em(i)
16.     act   = actual(i)
17.     mmre  = abs((act-est)/act)*100
18.     if (mmre > pred) failures++}
19.   return failures}

     function em(i, j,out) {
20.     if   ( Base ) { return 1 }
21.     else { #return product of effort multipliers }
\end{verbatim}}
\caption{{
The COCONUT tool: pseudo-code.}}\label{fig:entire}
\end{figure}
Given $D$ projects, the {\tt test} function of the
COCONUT tool shown  in \fig{entire} inputs
$a$ and $b$ parameters and computes an effort estimate (see line 15)
for
the projects numbered
$start$ to
$stop$ ($start \le stop \le D$). 
The {\tt learn} function (at lines 5 to 10) calls {\tt test}
for projects 1 to $i$ ($i{\le}D$) to find the $a$ and $b$ parameters that
minimizes the {\tt test} failure count. These best $a$ and $b$ values are
then {\tt test}ed on the projects numbered $i+1$ to $D$ (see lines 11,12). 

COCONUT is a very simple system and could be improved. For example,
\eq{base} and \eq{one} shows that $a$ and $b$ effect $effort$
monotonically and continuously.  Hence, the exhaustive search on lines
5 and 6 of {\tt learn} might be replaced with some form of binary
search. Secondly, COCONUT is currently implemented in an interpreted
C-like language (awk) and a reimplementation in ``C'' would make it
run faster.  Nevertheless, given that this sub-optimal exhaustive
search implemented in an interpreted language takes just a few minutes
to generate our results, we are not motivated to explore
optimizations. Further, the exhaustive nature of the search makes it
hard to argue that some other method might do better than COCONUT.


\section{Incremental Holdout Results}


\fig{nasaresults} shows the data of \fig{data}
being processed by a COCONUT incremental holdout study of the form:
\[\{Repeats=30, |Train|=4,|Test|=56, \delta=4\}|\]
That is, 30 times, the 60 projects of \fig{data} were randomly reordered
and divided into a small training set (with 4 projects)
and a much larger test set (with 56 items).
COCONUT then calibrated COCOMO~I and BASE 
using the training set. 
The best calibrations
found during training for each model were 
then used by each model on the test set. Four items from the
test site were then shifted to the training set and the
training/testing process repeated. This continued until the test
set was exhausted.

\begin{figure}
{\footnotesize
\begin{center}
\includegraphics[width=3in]{pdfs/cocomo81nasa17_ALL_PRED30_results.pdf}\\    
\includegraphics[width=3in]{pdfs/cocomo81nasa17_ALL_PRED20_results.pdf} 
\end{center}}
\caption{ BASE and COCONUT on
the \fig{data}.}\label{fig:nasaresults}
\end{figure}

In \fig{nasaresults},
the COCONUT-calibrating-BASE results are shown as a dashed line and the COCONUT-calibrating-COCOMO~I results
are shown as a solid line. 
Note that the mean PRED(30) values rise higher than the mean PRED(20)
values since PRED(20) is a more stringent test than PRED(30).


The error bars in \fig{nasaresults}
show $\pm 1$ standard deviation around the mean 
for each $\delta$ is the incremental holdout study.
These error bars are larger at low $X$ values
(when the training set is small) and at high $X$ values
(when the test set is small). Calibrating on a few projects,
or testing on a few projects, means that quirks with those
projects can disperse the results. The large variances at lower
$X$ values could explain the contradictory results reported in the
introduction; i.e. when one calibration study on $X$=8 projects
was successful but another failed.

\subsection{Hypothesis Testing}

\begin{figure}
{\footnotesize
\begin{center}
\includegraphics[width=3in]{pdfs/cocomo81nasa17_ALL_PRED30_results_changes.pdf}\\
\includegraphics[width=3in]{pdfs/cocomo81nasa17_ALL_PRED20_results_changes.pdf}
\end{center}}
\caption{Hypothesis testing on the \fig{nasaresults} results.}\label{fig:nasatestresults}
\end{figure}

\subsubsection{Comparing Mean PREDs from Two Models}

\fig{nasatestresults} shows the results of hypothesis testing on the
\fig{nasaresults} results. At the top of each plot is a thick line
that can take one of three values:
``base wins'', ``tie'', and ``coconut wins''.  This line shows the results
of a {\em inter}-method t-test. For each $X$ value, the null hypothesis
is posed that the mean PRED values for COCONUT-calibrating-BASE is the same as for 
COCONUT-calibrating-COCOMO~I.
If a two-sided t-test accepts this hypothesis, then
there is no significant difference between the PRED levels reached via
calibrating the two models. 

At nearly all $X$ values, COCONUT-calibrating-COCOMO~I ``wins'';
i.e. generates mean PRED values that are {\em both}
significantly different  (at the 95\% level) to the COCONUT-calibrating-BASE results {\em and}
which 
are higher than
the BASE PRED mean values. To the best of our knowledge, this is
the first experimental validation of the effort multipliers used
in COCOMO.

\subsubsection{Comparing Changes in Mean PRED From One Model}

At the bottom of each plot in \fig{nasatestresults} are two
step functions, one for COCONUT-tuning-BASE and one for COCONUT-tuning-COCOMO~I. This shows
the result of the {\em intra}-method t-test.
This t-test checks that a method's
mean score at some $X$ value is significantly
different to the last significant change for that method. 
Such a comparison
could reject the hypothesis that two adjacent and different
means are truly different, if the variances of the means were
too large. 

When the mean performance of the studies methods stops
changing significantly, then methods have {\em plateau-ed} and further
data collection for that learner is superfluous.
At the 95\% confidence level, in \fig{nasatestresults}, PRED(20) plateaus at 54%
and PRED(30) plateaus at 72\% after $X$=12 and $X$=20 projects
respectively.
As might be expected, it takes longer to achieve good results
on the more stringent PRED(20) criteria than the weaker PRED(30) criteria.


\section{Discussion}

A PRED(30)=70\%  result on holdout data seems as good as the
state-of-the-art results shown in \fig{predmmre}. 
For example, the COCONUT
result
seems very close to the COCOMO~II results of PRED(30) of 69\% on holdout data
reported by Chulani et.al.~\cite{chulani99}.  Further, COCONUT
achieves this level of performance using {\em far less effort} that the Chulani
et.al. result:
\bi
\item
Chulani et.al.'s improvements on COCOMO-I needed
78 more projects,  a DELPHI
panel of experts, and some Bayesian calibration.
\item
The peak PRED(30) values
in \fig{nasaresults} were generated using the  COCOMO~I effort
multipliers, 40 more projects, no DELPHI panel, no Bayesian calibration,
and  the very simple exhaustive enumeration technique of COCONUT. 
\ei

However,
directly comparing the \fig{nasatestresults}
results with  \fig{predmmre}
is problematic. 
Chulani
et.al. did not  report the standard deviation
in their holdout experiments and without that information,
it is difficult to compare our method to theirs.
To be fair, it is quite correct that they did not report standard
deviation since they repeated their holdouts only 15 times (a number
too small to collect accurate information about standard deviations).
Nevertheless, 
should they ever repeat their study using 20 to 30 repeats,
it would be interesting to view their standard deviation results.


Also, 
the Chulani and COCONUT results come from
different dataset (161 records in COCOMO~II format
vs 60 NASA records in COCOMO~I format) and it would be preferred
to apply the same techniques to same dataset (something we hope
to do in the near future).

\section{Threats to External Validity}

Two threats to our study come from {\em linearity}
and {\em granularity} of the effort models. If the
effort model is non-linear, then there could exist some
region in the space of the calibration parameters
\mbox{$<a,b>$} where the performance of the estimation model
changes radically. Linearity is a problem in any extrapolation-based
method that assumes that the region between two samples changes
smoothly between the two samples.  Granularity is a related concern:
if a model is searched in steps of size $M$, and the model's
behavior can change dramatically is some space $N{\le}M$, then
the explorer can miss important features of the model.

COCONUT's complete search through the range of \mbox{$<a,b>$} values makes no
extrapolation assumptions.  Hence, COCONUT is immune to linearity
problems.  

Theoretically, COCONUT could suffer from the granularity
problem. 
However, \fig{entire} steps through $a$ in steps of 0.3 and $b$ in
steps of 0.05 and these steps are smaller than the differences
we see in the literature for \mbox{$<a,b>$} calibrations for COCOMO~I.
Hence, we are somewhat confident that COCONUT does not have a granularity problem.

Other threats to external validity are covered by 
the incremental holdout study methodology. By randomizing the order
of the input examples, we are reducing the impact of order effects.
By conducting holdout studies, we are reducing the chance of over-fitting.
By only reported significant changes (found via hypothesis testing),
we are reducing the odds that our results arise from some statistical quirk.

Nevertheless,
the results presented in this paper come from a single dataset.
and, clearly, the above analysis needs to be repeated
on other data sets. Currently, we are exploring access
to COCOMO~II data. 


 

\section{Future Work}

Our results suggest an alternative to the 
 Chulani et.al. method for using expert opinion to calibration
COCOMO-like models.

One of the main motivations for the Bayesian analysis
of COCOMO-II was that the regression results from the 
83+78 projects had slopes that contradicted certain expert intuitions.
For example,
 regression of the COCOMO data
concluded that building reusable components {\em decreased}
development costs. Most experts believe that the extra
effort required to generalize a design actually {\em increases}
the cost of building such components. This anomaly was explained
 as follows: the 83+78 projects
  did not contain enough samples of projects that make heavy
use of reuse. To DELPHI panel and the subsequent Bayesian
calibration was used to fill in the gaps in the project data with
expert knowledge. This combination of DELPHI+Bayesian methods
proved successful: COCOMO-II had much higher PRED(N) levels
than COCOMO-I. 

If 20 projects is enough for local calibration, then it might be
possible to simplify  the DELPHI tuning process used by Chulani et.al.
in the COCOMO~II project. 
Descriptions of 20 projects could be generated from, say, 4
experts each asked to describe 
5 projects representing the kind of work conducted by their company. 
Unlike a DELPHI panel,
this group of experts would not need to conduct extensive discussions to
explore their conclusions.  If those descriptions were made in terms of
the COCOMO-I parameters, COCONUT could then tune an effort model to
that expert option using 
those 20 expert-generated examples. We
acknowledge that this proposal is speculative. Nevertheless, if
successful, it could significantly reduce the data collection effort
required to calibrate a local model.
  

\section{Conclusion}

COCONUT conducts an exhaustive search over the space of the $<a,b>$
parameters in a COCOMO~I model.  This technique is much simpler than
other methods used to learn good effort models including neural
nets~\cite{boetticher03,mair00investigation,srinivasan95,wittig97},
linear
regression~\cite{shepperd97,boehm81,boehm00b,mair00investigation},
rule induction~\cite{mair00investigation}, \mbox{DELPHI/Bayesian}
tuning~\cite{chulani99,boehm00b}, and analogical
reasoning~\cite{shepperd97}, Further COCONUT seems to yield  PRED levels
comparable to other methods, and does so with less project data and
fewer attributes (no scale factors).

COCONUT was analyzed here using  
incremental holdout (not jack knife)
studies, combined with randomization and hypothesis
testing, repeated 30 times.
We commend this experimental method to the effort estimation community since it 
 controls for the variance problem in holdout
experiments; and can report the minimum point at which enough project
data is enough.

This paper is offered as a baseline result with the hope
that other researchers will try to reproduce and out-perform our
results.  
Using incremental holdout studies, it would be very
clear if some effort estimation method out-perform COCONUT. Such an alternate
technique would be superior if (a)~it
was simpler to implement or faster to execute than COCONUT; (b)~it
achieves higher predictive accuracies or lower variances than
\fig{nasaresults} or \fig{nasatestresults}; 
and (c)~it do so using 
less project data than COCONUT.
To facilitate the search for a better-than-COCONUT technique,
we  have followed the lead of Srinivasan
and Fisher, and  placed all our data on the web.

We would also encourage the creation of public domain effort
estimation data sets. Naturally,
we would have preferred to have
conduct this study using other data sets such as COCOMO-II.
However, methodologically,  it is
important that our results be reproducible. Therefore, we used the
COCOMO-I data in \fig{data} and look forward to the time when we can
try conduct incremental holdout studies with COCONUT on other data sets

Finally,
the  COCOMO~II rule that 
``10
projects is enough for local calibration'' needs some qualification.
Certainly, \fig{nasatestresults} shows that improvements in mean PRED
start leveling off after 5-10 projects.
However, at 5-10 projects, the 
variance in the mean PRED values is still
decreasing and we 
 we have seen statistically significant PRED improvements 
 up to 20 projects.

 



% \fig{abtunings} show how TUNES changed $a$ and $b$ for the PRED(20) runs on
% the JPL and COCOMO-I data.
% For all the experiments reported here,
% the $a$ values were explored from 2 to 5 in increments of 0.2; i.e. 
% $a\in \{2, 2.2,...,4.8,5.0\}$.
% The $b$ values were explored from 0.9 to 1.2 in increments of 0.02; i.e.
% $b\in\{0.9,0.92,...,1.18,1.2\}$. 
% The whiskers in \fig{tunings}
% are 2 standard deviations wide, centered
% on the mean value.
% As seen before in \fig{tunings}, 
% the variances tend to decrease as more training data is used.
% Note that the
% JPL runs converge to significantly much higher $a$ and $b$ 
%  values that offered by standard COCOMO-I and COCOMO-II
% (those standard values are $\left<a,b\right>=\left<2.94,0.91\right>$
% respectively).
% Hence, JPL would
% be advised not to use un-tuned COCOMO-I for its effort estimations.


% \begin{figure}[!t]
% {\footnotesize
% \begin{center}
% \begin{tabular}{lcc}
% {\bf PRED(30):}&
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED30.pdf}  &    
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED30_changes.pdf}\\
% {\bf PRED(25):}&
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED25.pdf}  &    
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED25_changes.pdf}\\
% {\bf PRED(20):}&
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED20.pdf}  &    
% \includegraphics[width=1.4in]{pdfs/cocomo81nasa17_60_PRED20_changes.pdf}\\
% \end{tabular}
% \end{center}}
% \caption{$30*6/54-6$ experiments with the
% LSR and M5'.
% }\label{fig:baseline}
% \end{figure}





% \begin{figure*}[!t]
% {\scriptsize
% \begin{center}
% \begin{tabular}{p{1.1cm}|cc|cc|}\cline{2-5}
% &\multicolumn{2}{|c|}{PRED(30)} &
% \multicolumn{2}{|c|}{PRED(20)}\\\cline{2-5}
% NASA&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasa17_ALL_PRED30.pdf}&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasa17_ALL_PRED30_changes.pdf}&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasa17_ALL_PRED20.pdf}&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasa17_ALL_PRED20_changes.pdf}\\\cline{2-5}
% centerX&
% \includegraphics[width=1.2in]{pdfs/nasacenter3_ALL_PRED30.pdf}&              
% \includegraphics[width=1.2in]{pdfs/nasacenter3_ALL_PRED30_changes.pdf}&
% \includegraphics[width=1.2in]{pdfs/nasacenter3_ALL_PRED20.pdf}&              
% \includegraphics[width=1.2in]{pdfs/nasacenter3_ALL_PRED20_changes.pdf}\\\cline{2-5}
% projectX&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject2_ALL_PRED30.pdf}&         
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject2_ALL_PRED30_changes.pdf}&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject2_ALL_PRED20.pdf}&         
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject2_ALL_PRED20_changes.pdf}\\\cline{2-5}
% \end{tabular}
% \end{center}}
% \end{figure*}






% \begin{figure*}[!t]
% {\scriptsize
% \begin{center}
% \begin{tabular}{p{1.1cm}|cc|p{1.1cm}|cc|}\cline{2-3}\cline{5-6}

% centerY&
% \includegraphics[width=1.2in]{pdfs/nasacenter2_ALL_PRED30.pdf}&              
% \includegraphics[width=1.2in]{pdfs/nasacenter2_ALL_PRED30_changes.pdf}\\\cline{2-3}\cline{5-6}


% planning&\includegraphics[width=1.2in]{pdfs/missionPlanning_ALL_PRED30.pdf}& 
% \includegraphics[width=1.2in]{pdfs/missionPlanning_ALL_PRED30_changes.pdf}&
% avionics&
% \includegraphics[width=1.2in]{pdfs/avionics_ALL_PRED30.pdf}&         
% \includegraphics[width=1.2in]{pdfs/avionics_ALL_PRED30_changes.pdf}\\\cline{2-3}\cline{5-6}

% projectY&
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject4_ALL_PRED30.pdf}&         
% \includegraphics[width=1.2in]{pdfs/cocomo81nasaproject4_ALL_PRED30_changes.pdf}\\\cline{2-3}\cline{5-6}
% \end{tabular}
% \end{center}}
% \end{figure*}




%\begin{figure}
%\includegraphics[width=2.2in]{loc.pdf}
%\caption{{LOC,effort for JPL \& COCOMO81}}\label{fig:loc}
%\end{figure}



% \section{Future Work}

% do this with COCOMOII!

% Scrivansa and fisher warn about the 
% {\em concept drift}  problem. seek minimal number of projects
% to guard against concept drift.

% The above work is based on data from one organization (NASA). 
% We are currently seeking research partners who can offer data
% from other organizations for a COCONUT sensitivityxs study,

% If the offered data was in the COCOMO~II format, then
% that would let us build and validate  COCONUT~II; i.e.
% a parameter tuning tool for COCOMO-II (this would require very
% little changes to the internals of COCONUT~I). 

% Also, an obvious possibility for improving COCONUT would be to add
% some sort of robust regression facility to the test module that
% reported outliers. Statistics could be gathered
% in the train phase that summarize the range of variables seen
% during training. During test phase, if an example seems to be
% an outliers from the data seen during training, then COCONUT
% could still provide an estimate, but could flag that estimate
% with the caution that it has had little experience with this
% kind of project.




\section*{Acknowledgments}
This research was conducted at Portland State
University  under NASA
contract NCC2-0979 and NCC5-685.  The work was sponsored by the NASA
Office of Safety and Mission Assurance under the Software Assurance
Research Program led by the NASA IV\&V Facility.  Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States Government.


\bibliographystyle{plain}
\bibliography{../../refs}

\end{document}




Cohen advised that more sophisticated methods should be base-lined
against much simpler ones. This study tried to compare a very
simple calibration methods (COCONUT's exhaustive exploration of the
space of the $<a,b>$ parameters in a COCOMO~I model) with 
other effort estimation techniques. The resulting PRED
results {\em seem} as good as previous
state-of-the-art results. However,
issues with experimental
methods prevented a direct comparison. 

The results obtained are at least suggestive that sophisticated
methods such as analogy, rule induction, neural nets, Bayesian tuning,
regression might be over-kill for the calibration problem. Further
work is required to assess this possibility.

To facilitate such an assessment, much of this paper has been devoted
to a discussion of experimental methods for effort estimations.  

other researchers

This paper has introduced {\em incremental holdout studies}, a method of assessing
a calibration method using progressively less training data. Sensitivity studies
are important for finding the minimal amount of data required before
a new model can be adequately calibrated. In the case of the COCONUT tool
on the NASA data of \fig{data}, it was found that at the PRED(20) and PRED(30)
levels, the predictive accuracy:
\bi
\item Did not rise much after 12 projects;
\item
 And did not rise significantly 
at all after 24 projects. 
\ei


Based on a sensitivity study of 60 NASA records in a COCOMO~I format
using the COCONUT tool,
we conclude that 
there is much more to calibration than statements like ``10
projects are enough'':
As shown above in \fig{r1}, {\em different PRED
levels} can be reached, depending on how much data is used for
calibration. For example, our mean PRED(30) 
started leveling at $X$=6 projects
but our PRED(20) results did not start leveling till $X$=12 projects.

However, it is very important {\em not} to assess
effort estimation experiments using just mean PRED levels.
It is important to {\em study the mean and the standard
deviation of the predictive accuracy} as well.
If this is not done, then the results from a calibration study can be
misrepresented. For example,
in the case of \fig{r1},
we might have inappropriately reported PRED(20) figures that 
were must higher than what could be justified by our results.


It is hoped that this paper will become a baseline measurement against
which other calibration results will be compared.  Such comparisons are
vital to improve the field of effort estimation. Such comparisons
should be based on the mean {\em and standard deviation} of the
techniques being compared.  

