\section{Software Estimation Research}
Case-based software estimation such as {\em Case-Based Reasoning} (CBR) is a widely explored area in software engineering research~\cite{shepperd97,Mendes2003,Kirsopp2002,Walkerden1999}. Based on collective experience, when a manager sees an estimate, his/her immediate question is ``how can I change that?''. While the effort estimation literature describes many estimation methods (both model-based and case-based
~\cite{pendharkar05,Li2009,Lipowezky1998,Walkerden1999,Kirsopp2002,Mendes2003,jorgensen05,shepperd07,shepperd97})
in order to address manager's immediate concern, {\W}2 focuses on how to {\em change} estimates.

{\W}2 explores multiple goals such as reducing development effort {\em and} defects {\em and} the total calendar
time to deliver the software. Instead, most other work in this area explores a single goal.
For example,
Pendharkar et al.~\cite{pendharkar05} demonstrate the utility of Bayes networks in software effort
estimation while Fenton and Neil explore Bayes nets and software defect prediction~\cite{fenton99},
%Unlike this paper,
neither of these teams links defect models to effort models. 
In addition, as mentioned above, these work focus much
more on {\em prediction}, rather than on the subsequent problem of learning how to {\em change} those
predictions.

\section{Search-Based Software Engineering (SBSE)}
Multi-goal optimization in Search-Based Software Engineering (SBSE) is well explored in the field~\cite{harman04}.
SBSE 
employs optimization techniques
from
operations research and meta-heuristic search (for example in simulated annealing and genetic algorithms)
in an attempt to hunt for near-optimal
solutions.
Harman~\cite{harman04} distinguishes AI search-based methods from those seen in
standard numeric optimizations.
Such optimizers offer settings to all controllables.
This may result in needlessly complex recommendations since a repeated empirical
observation is that many model inputs are contaminated or correlated in similar ways 
to model outputs~\cite{hall03}. Such contaminated or correlated variables can be pruned  to
generate simpler solutions that are easier and quicker to understand.
For continuous variables, there
are many work on feature selection~\cite{miller02} and
techniques like principal component analysis~\cite{Dil84} to reduce
the number of dimensions reported by a data analysis.  Some
studies report that discrete AI methods perform better at
reducing the size of the reported theory~\cite{hall03}.

The SBSE approach can and has been applied successfully to many
software engineering domains such as requirements engineering~\cite{jalali08},
 but more commonly used in software testing~\cite{andrews07}.
Harman's work provides the inspiration to this study in an attempt to experiment simulated annealing for our model-based methods~\cite{me07f}
(which performed worse than {\W}2). 



\section{Model: Benefits}
High-level abstraction models represent and transmit common software patterns observed in
multiple specific situations~\cite{gof95}.
At a keynote
address at PROSIM'05 Walt Scacchi noted that merely writing a model
can clarify  local business processes~\cite{pfahl05}. 
Executable software process models can be used for many purposes
including but not limited to reducing the inspection effort 
at different stages of the software life cycle~\cite{me02f}.
Even if a model lacks a sophisticated execution engine, it can still be used
for what-if queries that are insightful to different business processes
(e.g. see Boehm et al.'s what-if studies in Chapter Three of~\cite{boehm00b}). 

Models can combine and summarize {\em both} expert insights {\em and} local data. Fenton~\cite{fenton07}
builds
the general structure of his  Bayes nets via workshops of business
knowledge.  The details of these structures are then tuned via local
data. Elsewhere, Boehm reports the advantages of combining local data
with model structures initialized via expert knowledge~\cite{chulani99}.
 
Another subtle advantage of models is data sharing. Schulz reports that
organizations that are reluctant
to share specific data, may be willing to share models (if those models
do not reveal
details from particular business sites)~\cite{schulz10}.

Finally, models let us extrapolate from past examples to new examples. A trend that is sampled by
$N$ historical examples can be extended to offer predictions for new examples that have not been seen previously.


\section{Model: Drawbacks}\label{sect:draw}
Extrapolation, while sometimes useful, may over fit the data. If that occurs, then a model may offer unsupported recommendations. For example, as shown in the results section, model-based methods were ineffective since, sometimes,
they proposed conclusions that applied to {\em none} of the test data. 

\input{cocomo.tex}

Another drawback with model-based tools is that they only accept data that conforms to the ontology of the model
(i.e. use the input values of the model).
If local data does not conform to that  ontology, then the tool cannot be applied.
For example, \fig{data} shows the data sets used in this study. Model-based methods
can only process the two data sets that conform to the COCOMO ontology of \fig{cocattrs}. On the other hand,
the {\W}2 case-based method can process all of them.

%For example, prior to {\W}, all our experiments on software development effort were based on the
%restrictive ontology of COCOMO an aspect noted by other researchers such as Shepperd~\cite{shepperd07}:
%\begin{quote}
%{\em A limiting factor of this work \cite{Menzies2006} is that it 
%requires data ... to conform to the COCOMO 
%data model. }
%\end{quote}
%Yet another drawback with model-based methods is that their conclusions are only 
%as good as the underlying model.
%Models can be validated via domain data but
%if the available data is not
%relevant to some new project, then the calibrated  model
%may be inappropriate for the new project.
%For example, Jorgensen \& Shepperd caution that
%the data used to derive COCOMO may not apply to all corporations~\cite{jorgensen05}.
%
Models need to be learned from data and
collecting that data can be difficult 
due to the business sensitivity associated with the data as 
well as differences in how the metrics are determined, collected and 
archived. In many cases the required data is not archived at 
all. In our experience, for example, after two years \texttt{promisedata.org} was only able to add 7 records 
to its NASA-wide software cost metrics repository~\cite{me07f}. Alternatively, open-source code repositories are a rich source of {\em product} information, but usually lack {\em process details} such as the descriptions of the applications experience of the developers.
%For example,
%the experiments of this paper were conducted
%on seven projects of \fig{data}, each with less
%than, on average, just a few dozen examples or less.

Other researchers also have noted similar problems with collecting process data.
%The effort estimation datasets used in
%Mendes et al.~\cite{Mendes2003}, Auer et al.~\cite{Auer2006},
%Baker~\cite{baker07},  and Li et al.~\cite{Li2009} have
%median number of examples 13,15,33,52 (respectively).
Lowry~\cite{lowry10} discusses the complexities involved in calibrating his
software failure models.
Those models require 
parameters that are clearly antiquated.
For example, he mentions a
commercial model-based cost estimation tool that requires a parameter 
that rates
``the time it takes for a software 
development environment to respond to a keyboard input''. When software was written on  
remote time-shared computers, this was an important factor. However,  today 
it is irrelevant  but it is kept in 
the model for backwards compatibility and because it was 
measured in the software projects on which the model was 
calibrated. 

Baker~\cite{baker07} discusses another serious concerns with model calibration: {\em tuning instability}.
Software construction is a very human-intensive process, therefore the data collected
from that process
is as varied as the humans building the code.
Consider the following simplified COCOMO~\cite{boehm00b} model: 
\begin{equation}\footnotesize\label{eq:simpcoc}
effort = a \cdot LOC^{b +  pmat}\cdot acap  
\end{equation}
The equation presents COCOMO's core assumption 
that software development effort is exponential on software size.
In this equation,  $a$ and $b$ control the linear and exponential
inferences (respectively) on model estimates; 
while
$pmat$ (process maturity) and $acap$ (analyst capability) are project choices
articulated by managers. 
\eq{simpcoc} contains only two features ($acap,pmat$) and a full COCOMO model contains a set of project descriptors as shown in \fig{cocattrs}.

%\begin{figure}
%\begin{center}
%\includegraphics[width=2.2in]{figs/nasa93_300.pdf}
%\end{center}
%\caption{Learned $(a,b)$ values via local calibration on 300*90\% samples
%of the NASA93ii COCOMO data set.}\label{fig:ab}
%\end{figure}
%

Baker~\cite{baker07} learned values of $(a,b)$ for a full COCOMO model
using Boehm's local calibration method~\cite{boehm81}
from 300 random samples of 90\% of the available project data. 
The ranges varied widely:
\begin{equation}\label{eq:ab}
\left( 3.2 \le a \le 9.4 \right) \wedge \left(0.8 \le b \le 1.12\right)
\end{equation}
Such large variation in model tunings not only violates standard gradient descent methods,
but it also obscure any
 benefits observed within a particular  project change.
Suppose a proposed technology  doubles productivity, 
but $a$ changed from 9.0 to 4.5, any improvement would be obscured by the {\em tuning instability}.



\begin{figure}[!t]
\begin{center}
\includegraphics[width=5.0in]{figs/ranges.pdf}
\caption{COCOMO 1 effort multipliers, and the sorted coefficients found by linear regression from
twenty 66\% sub-samples (selected at random) from the NASA93 PROMISE data set; from~\cite{me05a}.
}
\label{fig:coc1}
\end{center}
\end{figure}

The plot in \fig{coc1} shows the $b$ values learned from twenty 66\% samples (selected at random) of the NASA93 data set from the PROMISE repository. Prior to learning, training data was {\em linearized}
in the manner recommended by Boehm ($x$ was changed to $log(x)$; for details, see~\cite{me05a}).
During learning, a greedy back-select removed attributes with no impact on the estimates: hence, some of the attributes have less than 20 results. After learning, the coefficients were unlinearized.

While some of the coefficients are stable (e.g. the white circles of $loc$ remains stable around 1.1), the coefficients of other attributes are highly unstable:
\bi
\item The $(max - min)$ range of some of the coefficients is very large; e.g. the upside down black triangles of $stor$ ranges from $-2 \le b \le 8$.
\item Consequently, nine of the coefficients in \fig{coc1} jump from negative to positive.
\ei
We have seen instability in other datasets, including the COC81 data used by Boehm to derive the general form of \eq{simpcoc}~\cite{me05a}. This is an troubling observation.

In summary, model-based methods can suffer from:
\bi
\item Inappropriate extrapolations;
\item Ontology restrictions;
\item Untamed variance inside the models
\ei
Hence the need explore alternative methods.