\section{Case Studies}

Recall that the goal of our study is to analyze
simulations of process estimation using COCOMO and COQUALMO
models on typical NASA projects. Our purpose is to evaluating the relative merits
of adopting a software reuse compared to other project improvement strategies.
The comparison is based on effort, quality, and
schedule measured in man-months, defects per KSLOC, and months,
respectively. In this study we explore the perspective of the Project Manager in the
context of NASA software development.

\fig{cases} partially describes the two NASA case studies
we explore in terms of the COCOMO and COQUALMO input parameters.
Both case studies reflect typical ranges seen at NASA's Jet Propulsion
Laboratory~\cite{me08c}

\begin{figure}[!t]
\renewcommand{\baselinestretch}{0.95}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&rely&3&5&tool&2\\
JPL &data&2&3&sced&3\\
flight&cplx&3&6&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&7&418&&
\end{tabular}~~~~~~~~~~~~~\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&rely&1&4&tool&2\\
JPL &data&2&3&sced&3\\
ground&cplx&1&4&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&11&392&&\\
\end{tabular}
\end{center}
\caption{Two case studies.
Numeric values 
$\{1,2,3,4,5,6\}$ map to $\{very low, low, nominal,
high, very high, extra high\}$.}\label{fig:cases}
\end{figure}

Inside our model,
project choices typically range from
1 to 5 where ``3'' is the nominal
value that offers no change to the default estimate. 
Some of the project choices in \fig{cases}
are known precisely (see all the choices with single {\em values}).
But  many of the features in \fig{cases} do not
have precise values
(see all the features that {\em range} from some {\em low} to {\em high}
value).  

%Sometimes the ranges of choices
%are very narrow (e.g., the process maturity of JPL ground software is between 2
%and 3), and  sometimes the ranges are very broad.
%\fig{cases} does not mention all the  features listed in \fig{emsf2}
%inputs.  For example, our defect predictor has inputs for
%use of {\em automated analysis}, {\em peer
%reviews}, and {\em execution-based testing tools}. During STAR's search,
%for all project choices
%not mentioned in \fig{cases}, values are picked at random from the full
%range of \fig{emsf2}.


We evaluate the effects of the project improvement strategies 
on the case studies above using STAR: a Monte Carlo engine 
augmented by a simulated annealing search engine.
STAR runs in six phases.
In terms of standard machine 
learning theory, step 1 generates a training set; steps 2,3,4 do some generalization;
and step 5 tests the learned theory on data not seen during training.

\begin{description}
\item[1. {\em SAMPLE:}]
To sample the ranges from the models,
STAR runs a simulated annealer  $K_1=10,000$ times. 
\item[2. {\em DISCRETIZE:}]
The data seen in the $K_1$ samples is then discretized into $D=10$ bins, each spanning 10\%
of the range min to max.
\item[3. {\em CLASSIFY:}]
The ranges are divided
into those of the BEST\% {\em best} or {\em rest}.
\item[4.
{\em RANK:}]
The ranges are then ranked in increasing order by their probability of appearing 
in the $best$ outcomes * the support of the range;
\item[5.
{\em PRUNE:}]
STAR then runs $K_2$ experiments with the models where the top ranked ranges
$1..X$ ranges are pre-set and the
remaining ranges can be selected at random.
\item[6.
{\em REPORT:}]
STAR returns the $1..X$ ranges that  produce the least effort, defects, and development time.
\end{description}

The reader may wonder why we use an stochastic method like STAR:
would not a simpler method suffice?
For example, 
Many of the relationships inside COCOMO
model are linear and a 
 simple
linear extrapolation across the space of possibilities could 
assess the relative effectiveness of different changes.
In reply, we note that:
\bi
\item 
Even after tuning
the gradient of the relationships 
may not be known with certainty.
For example, the COCOMO effort model predictions are affected
linearly and exponentially by two features $a,b$.
Baker~\cite{baker07} tuned these
$a,b$ values using data from NASA systems. After  thirty 90\% random samples of that data,
the $a,b$ ranges were surprisingly large:
$\left( 2.2 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right)$. Baker's results
forced a rethinking of much our prior work in this area. Instead of exploring better learners
for local calibration, now we use tools like STAR to search models for conclusions that persist across
the space of possible  calibrations.
\item
Simplistic linear extrapolation may be inappropriate when
when optimizing for effort {\em and} time {\em and} defects, there
may be contradictory effects.  For example, we have results where
where reducing effort leads to a dramatic increase in defects~\cite{me09a}.
Hence, optimizing our models is not a simple matter of moving fixed
distances over some linear effect: there are also some trade-offs
to be considered (e.g. using a tool that considers combinations of
effects, like STAR).
\ei
