\section{Case Studies}

Recall that the goal of our study is to analyze
simulations of process estimation using COCOMO and COQUALMO
models on typical NASA projects. Our purpose is to evaluate the relative merits
of adopting software reuse compared to other project improvement strategies.
The comparison is based on effort, quality, and
schedule measured in person-months, defects per KSLOC, and months,
respectively. In this study we explore the perspective of the Project Manager in the
context of NASA software development.

\fig{cases} partially describes the two NASA case studies
we explore in terms of the COCOMO and COQUALMO input parameters.
Both case studies reflect typical ranges seen at NASA's Jet Propulsion
Laboratory~\cite{me08c}.

\begin{figure}[!t]
\renewcommand{\baselinestretch}{0.95}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&rely&3&5&tool&2\\
JPL &data&2&3&sced&3\\
flight&cplx&3&6&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&7&418&&
\end{tabular}~~~~~~~~~~~~~\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&rely&1&4&tool&2\\
JPL &data&2&3&sced&3\\
ground&cplx&1&4&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&11&392&&\\
\end{tabular}
\end{center}
\caption{Two case studies.
Numeric values 
$\{1,2,3,4,5,6\}$ map to $\{very low, low, nominal,
high, very high, extra high\}$.}\label{fig:cases}
\end{figure}

Inside our model,
project choices typically range from
1 to 5 where ``3'' is the nominal
value that offers no change to the default estimate. 
Some of the project choices in \fig{cases}
are known precisely (see all the choices with single {\em values}).
But  many of the features in \fig{cases} do not
have precise values
(see all the features that {\em range} from some {\em low} to {\em high}
value).  

We evaluate the effects of the project improvement strategies 
on the case studies above using STAR: a Monte Carlo engine 
augmented by a simulated annealer.

STAR runs as follows.
First,  a project $P$ is specified as a set of min/max ranges to the input variables of
STAR's models:
\bi
\item
 If  a variable is known to be exactly $x$, then
then \mbox{$min=max=x$}. 
\item
Else, if a variable's exact value is not known but the range of possible values
is known, then min/max is set to the smallest and largest value in that range 
of possibilities.
\item
Else, if  a variable's value  is completely unknown  then min/min is set to the full
range of that variable in \fig{cparams}. 
\ei
Second, STAR's simulated annealer\footnote{Simulated annealers randomly
alter part of the some $current$ solution. If this $new$ solution scores better
than the current solution, then $current=new$. Else, at some probability
determined by a temperature variable, the simulated annealer may jump to a sub-optimal
$new$
solution.
Initially the temperature is ``hot'' so the annealer jumps all over the solution
space. Later, the temperature ``cools'' and the annealer reverts to   a
simple hill climbing search that only jumps to new better solutions.
For more details, see ~\cite{kirkpatrick83}.}
seeks constraints on $P$ that most improve the model's score.
A  particular subset of $P'\subseteq P$ is scored by using $P'$
as inputs to the COCOMO and COQUALMO. When those models run, variables are selected
at random from the min/max range of possible tunings $T$ and  project
options $P$. 
In practice,
the majority  of the  variables in $P'$ can be removed without effecting the score;
i.e. our models exhibit a {\em keys effect} where a small number of variables control
the rest~\cite{me07d}. Finding that minimal set of variables is very useful for management
since it reveals the {\em least} they need to change in order to {\em most} improve the 
outcome.
Hence, simulated annealing, STAR takes a third step.

In this third step, a
Bayesian
sensitivity analysis finds the  smallest subset of $P'$ 
that most effects the output. The scores seen during simulated annealing are sorted
into the (10,90)\% (best,rest) results. Members of $P'$ are then ranked by their Bayesian
probability
of appearing in $best$. 

After ranking members of $P'$,
STAR then imposes  the top $i$-th 
ranked items of $P'$ on the model inputs, then running the models 100 times.
This continues until the scores seen using $i+1$ items is not statistically
different to those seen using $i$ (t-tests, 95\% confidence).
STAR returns items $1..i$ of $P'$ as the {\em least} set of project decisions that
{\em most} reduce effort, defects, and development time.
We call these returned items the {\em policy}. 

Note that STAR constraints the project options $P$ but never the tuning options $T$.
That is, the {\em policy} generated by STAR contains 
parts of the project options $P$ that most improve the score, despite variations
in the tunings $T$.
This approach has the advantage that it can reuse
COCOMO models without requiring local tuning data.

Previously~\cite{me08c} we have shown that this approach, that does not use local tuning,
generates estimates very similar
to those generated after using local tuning via the ``LC'' method proposed by Boehm
and in widespread use in the COCOMO community~\cite{boehm81}.  We have explained
this effect as follows. Uncertainty in the project options $P$ and the tuning
options $T$ contribute to uncertainty in the estimates generated by
STAR's models. However, at least for the COCOMO and COQUALMO models
used by STAR, the  uncertainty created by $P$ dominates that of $T$. Hence,
any uncertainty in the output can be tamed by constraining $P$ and not $T$.


The reader may wonder why we use an stochastic method like STAR:
would not a simpler method suffice?
For example, 
Many of the relationships inside COCOMO
model are linear and a 
 simple
linear extrapolation across the space of possibilities could 
assess the relative effectiveness of different changes.
In reply, we note that:
\bi
\item 
Even after tuning
the gradient of the relationships 
may not be known with certainty.
For example, in the COCOMO effort model, predictions are affected linearly and exponentially
by two types of input parameters; the new project data and the historical dataset. In COCOMO this results in
the coefficients, $a$ and $b$, which define the relationship between size and effort.
Baker~\cite{baker07} tuned these
$a,b$ values using data from NASA systems. After  thirty 90\% random samples of that data,
the $a,b$ ranges were surprisingly large:
$\left( 2.2 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right)$. Baker's results
forced a rethinking of much our prior work in this area. Instead of exploring better learners
for local calibration, now we use tools like STAR to search models for conclusions that persist across
the space of possible  calibrations.
\item
Simplistic linear extrapolation may be inappropriate when
optimizing for effort {\em and} time {\em and} defects, there
may be contradictory effects.  For example, we have results where
reducing effort leads to a dramatic increase in defects~\cite{me09a}.
Hence, optimizing our models is not a simple matter of moving fixed
distances over some linear effect: there are also some trade-offs
to be considered (e.g. using a tool that considers combinations of
effects, like STAR).
\ei
