% AI search to SEESAW
% dump para 1,2 of section 5
%
%  $Description: Author guidelines and sample document in LaTeX 2.09$ 
%
%  $Author: ienne $
%  $Date: 1995/09/15 15:20:59 $
%  $Revision: 1.4 $
%

\documentclass[times,10pt,twocolumn]{article} 
\usepackage{IEEE_CS_LATEX/latex8}
\usepackage{times,url}
\usepackage{alltt}
\usepackage{graphicx}
\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\tion}[1]{\S\ref{tion:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


%\documentstyle[times,art10,twocolumn,latex8]{article}

%------------------------------------------------------------------------- 
% take the % away on next line to produce the final camera-ready version 
\pagestyle{empty}

%------------------------------------------------------------------------- 
\begin{document}

\title{
How to Avoid Drastic Software
Process Change (using Stochastic Stability)
}
\author{Tim Menzies\thanks{
Part of the research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.
}%\thanks{Submitted to ICSE`09, \today }\thanks{Download at
%\protect\url{http://menzies.us/pdf/08drastic.pdf}.}
,  Steve Williams,\\
Oussama El-Rawas\\
CSEE, WVU, USA\\
tim@menzies.us \\swilli12@mix.wvu.edu\\
oelrawas@mix.wvu.edu
\and
Barry Boehm\\
       Computer Science Dept.\\
 Uni. S. California, USA\\
       boehm@sunset.usc.edu
\and
Jairus Hihn\\
       Jet Propulsion Laboratory, \\California Inst. of Technology,\\
       California, USA\\
       jairus.hihn@jpl.nasa.gov
}


\maketitle
\thispagestyle{empty}

\begin{abstract}
Before performing {\em drastic changes} to a project,  it is  worthwhile to
thoroughly explore the available options within the current structure of a
project.  An alternative to drastic change are {\em internal changes} that
adjust current options within a software project.   In this paper, we  show  that the
effects of numerous internal changes can out-weigh the effects of  drastic
changes.  That is,  the benefits of drastic change can often be achieved
without disrupting a project.

The key to our technique is SEESAW, a novel stochastic stability tool that
(a)~considers a very large set of minor changes using stochastic sampling; and
(b)~carefully selects the right combination of effective minor changes.

Our results show, using SEESAW, project managers have more project improvement options than
they currently realize.  This result should be welcome news to  managers
struggling to maintain control and continuity over  their project in the face
of multiple demands for drastic change.
\end{abstract}



%------------------------------------------------------------------------- 
\Section{Introduction}

Software process control is complicated by uncertainty.
Exactly how good are our analysts? How large is the final system? Do we 
understand our tools? These are some of the questions that most
managers
cannot answer precisely.

For example, if asked ``what is the required reliability
on this project?'' a manager
may reply ``well, the code has to run but this is not  a safety critical system''.
In terms of, say, the COCOMO~\cite{boehm00}
five-point reliability scale,
the manager's answer describes a  {\em range
of options} ``low'' to ``high''. Some of options are {\em controllable}; e.g.
the manager may offer the customer an early delivery of the software if the customer considers
{\em not} building a ``high'' reliability system. 

This paper compares the effects of  (a)~{\em internal changes} to a project
that adjust the controllables within their current known ranges and 
(b)~{\em drastic changes} that require
moving a project outside of current bounds. For example, for projects with  a range
of reliability options ``low'' to ``high'',
an {\em internal change} would be setting reliability to a value within
``low'' to ``high''. On the other hand, a {\em drastic change} would be to suddenly
demand that a project becomes a 
``very high'' reliability project.
\fig{dchange} lists the disruptive changes of nine drastic
changes.  For example,
``improve personnel'' (by firing staff and hiring new 
programmers) may lead to a 
major union
dispute. Other drastic changes like  ``reduce functionality'' can  restrict  the functionality
of the
next release. This, in turn, could mean less revenue if the reduced functionality software
has  
less market appeal.

 
%For example, your manager has returned from ICSE`09
%excited about the next new thing in software engineering.
%``We need to drastically alter our projects,'' she says, ``in order to support (say) automatic
%formal methods, agile process,  etc.''. 
%This change
%may entail switching from Windows to LINUX (since there are more 
%research prototypes for automated formal methods running on LINUX);
%stopping production
%while engineers retrain;  or, worse, firing the engineers and making new hires. 
%Alternatively,
%your manager knows that a budget cut is imminent and 
%wants to discuss drastic reduction in project scope.
%This change may
%alter projected revenue streams 
%(since
%the next release will have less features and hence less market
%appeal). Changing revenue projections can be very detrimental to
%the rate
%of investor capital received by a software development company.
%For a list of other drastic changes, see
%\fig{dchange}.
%
\begin{figure}[!b]
\footnotesize
\begin{tabular}{c@{~}p{.4\linewidth}p{.5\linewidth}}
&{\bf Drastic change} & {\bf Possible undesirable impact}\\\hline
1&Improve personnel & Firing and re-hiring personnel leading to wide-spread union unrest.\\\hline
2&Improve tools, techniques, or  
development platform & Changing operating systems, IDEs, coding languages\\\hline
3&Improve precedentness / \newline development flexibility &
Changing the goals
of the project and the development method.\\\hline
4&Increase architectural \newline analysis /  risk resolution &
Far more elaborate early life cycle analysis.\\\hline
5&Relax schedule  & Delivering the system later.\\\hline
6&Improve process maturity & May be expensive in the short term.\\\hline
7&Reduce 
functionality & Delivering less than expected.\\\hline
8&Improve the team & Requires effort on team building.\\\hline

9&Reduce quality 
& Less user approval, smaller market.
\end{tabular}
\caption{Nine drastic changes from~\cite{boehm99}.}\label{fig:dchange}
\end{figure}
\newcommand{\boxplot}[5]{
\begin{picture}(100, 7)
\put(#1, 2){\line(0, 1){4}}
\put(#1, 4){\line(1, 0){#2}}
\put(#3, 4){\circle*{3}}
\put(#3, 4){\line(1, 0){#4}}
\put(#5, 2){\line(0, 1){4}}
\put(50, 3){\line(0, 1){4}}
\end{picture}
}
SEESAW is a tool that seeks alternatives to drastic change.
SEESAW stochastically explores
combinations of internal changes.
When controlling mission-critical software, such stochastic methods are not desirable. However,
when studying the uncertainty associated with unknowns about software processes, stochastic algorithms
like SEESAW are very useful.
If run multiple times,
stochastic sampling can return multiple options so users can study the range of possible
solutions.
That is,
SEESAW
offers both a solution and a comment on the {\em stability} of that solution.


%We contrast the results of {\em drastic change} and  {\em AI search} with the current project structure
%as follows.
%After an AI search within the bounds of the current project,
%the resulting recommendations can be implemented by the manager running that project.
%For example, the client may be happy with reliability levels from moderate to high
%and the project manager is free to adjust the intended reliability to moderate.
%On the other hand, {\em drastic change} 
%requires the involvement of senior management since it
%falls outside of the current definition of a project.
%For example, a drastic change might require firing the entire current development
%team.
%
\begin{figure}[!t]
\noindent
{\scriptsize\begin{center}
\begin{tabular}{ rc @{} c @{} c @{}}
        & & Defects / KLOC\\
 Change & & (normalized 0..100, min..max)\\
\hline
 {\bf SEESAW}&  & \boxplot{2.2727273}{2.2727273}{4.5454545}{2.2727273}{6.818182} &  \\
 Improve pcap &  & \boxplot{11.363636}{2.2727273}{13.636364}{2.2727273}{15.909091} &  \\
 Improve tool/tech/plat &  & \boxplot{20.90909}{2.7272727}{23.636364}{5.0}{28.636364} &  \\
 Reduce functionality &  & \boxplot{26.363636}{3.6363637}{30.0}{3.1818182}{33.18182} &  \\
 Improve pmat &  & \boxplot{41.81818}{6.3636365}{48.18182}{16.363636}{64.545456} &  \\
 Improve prec/flex &  & \boxplot{47.272728}{7.2727275}{54.545456}{16.363636}{70.90909} & \\
 Improve team &  & \boxplot{49.545456}{7.7272725}{57.272728}{12.727273}{70.0} &  \\
 Relax schedule &  & \boxplot{57.727272}{5.909091}{63.636364}{10.0}{73.63636} &  \\
 Arch/risk resolution &  & \boxplot{52.272728}{13.181818}{65.454544}{10.454545}{75.90909} & \\
 Do nothing &  & \boxplot{81.818184}{4.090909}{85.90909}{2.7272727}{88.63636} &  \\
 Reduce quality &  & \boxplot{92.72727}{2.7272727}{95.454544}{2.2727273}{97.72727} & \\ \hline
 &  & 50\% & 
\end{tabular}
\end{center}}
\caption{
Defect predictions from changing
a project (25 to 75\% percentile range from 100 simulations;  median values shown as a dot).
}\label{fig:results}
\end{figure}
This paper checks the stability of predictions about drastic change
or internal changes learned by SEESAW's stochastic sampling.
For the drastic changes of \fig{dchange} and four NASA case studies,
our
main result is that:
\begin{quote}
{\em Combinations of internal changes
usually perform as well or better than
drastic changes}.
\end{quote}
That is, using internal changes,
the benefits of drastic change can be achieved without
project disruption.
%This result should be welcome news to  managers
%struggling to maintain control and continuity over  their project in the face
%of multiple demands for drastic change.
%
%XXX related work. bayes etc.
%
For example, \fig{results} compares defect/KLOC
predictions from SEESAW and
the drastic changes of \fig{dchange}.
SEESAW's recommendations, comprising only
internal changes,
resulted
in the smallest defect predictions (see line 1).

%More formally, we say that we if a model exhibits three properties, then
%we can control it without local tuning data.
%A model converts inputs to outputs using tuned internal parameters; i.e.,
%\[Out=model(In,Tunings)\]
%The output mean
%   $\mu$ and variance $\sigma$ comes from
%the mean and variance of $In$   ($\mu_{I},\sigma_{I}$) and
%the space of possible $Tunings$ ($\mu_T,\sigma_T$).
%Model tuning takes historical data to change $\mu_T$ and reduce $\sigma_T$
%and, clearly, untuned models have large $\sigma_T$ and hence large output $\sigma$.
%If the goal is to reduce model output (property1)
%and if model
%output variance is reduced by the mean
%(i.e.,  property2: $\mu \propto \sigma$)
%then (a)~minimizing 
%mean output values has the desirable side-effect of also reducing the variance; and
%(b)~the variance added by 
%sampling across $\{\mu_T,\sigma_T\}$ can be selecting 
%appropriate input values $selected \subseteq I$. 
%Also, 
%for models where the effects on the output of $I$ are greater than $T$ (property3), then 
%the output $\mu$ can be minimized by using the right input selections.
%
%The
%empirical results presented below (and in~\cite{me08c,me07f}) are consistent with the hypothesis
%that commonly used software process models (specifically, the USC COCOMO suite of models)
%have these three properties; i.e., they can be controlled by AI search within the space of current project options.

\input{cocomo}



The rest of this paper explains how \fig{results} was generated.
After some preliminaries,  we discuss the options
available within the current structure of four sample projects.
\fig{dchange}'s drastic changes  will then be implemented as 
operators on those projects.
Next we describe SEESAW and the models it operates on:
the COCOMO effort and time estimator~\cite[p29-57]{boehm00b}; 
and the COQUALMO defect predictor~\cite[p254-268]{boehm00b}.
In the results that follow we show that, for our case studies,
SEESAW's search through the space
of internal changes usually
out-performs the 
drastic changes of \fig{dchange}.

The contribution of this paper is a demonstration that 
managers have more options than they may
realize.
Stochastic stability is an  insightful 
method
for discovering useful project reconfigurations that improve a project while avoiding requiring drastic and
disruptive project change.   


\Section{Preliminaries }

The reader may wonder why we use a stochastic method like SEESAW
to explore project options. Would not a simpler method suffice?
For example, in the case of {\em linear models} that have been {\em
precisely  tuned} using local data, it is a simple
matter to check if a combination of internal changes does better than drastic change.
Many of the relationships inside COCOMO
model are linear.
For such models,  ``what-if'' queries require just 
a simple
{\em linear extrapolation} to
assess the relative effectiveness of, say, reducing functionality versus
some combination of internal changes.

Unfortunately, not all tunings are {\em precise}. Sometimes, even after tuning,
the gradient of the relationships 
may not be known with certainty.
For example, the COCOMO effort model predictions are affected
linearly and exponentially by two features $a,b$.
Baker~\cite{baker07} tuned these
$a,b$ values using data from NASA systems. After  thirty 90\% random samples of that data,
the $a,b$ ranges were surprisingly large:
\begin{equation}\label{eq:ab}
\left( 2.2 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right) 
\end{equation} 
Elsewhere we have been partially successful in
reducing the range of \eq{ab}
with feature subset selection (FSS)~\cite{chen05,me06d} or more
data collection. 
FSS reduces but does not eliminate the $a,b$ variance. Also, further
data collection is possible, but only at great organizational expense. 
This is due to  data not being collected or
the business sensitivity associated
   with the data as well as differences in how the metrics are defined,
   collected and archived. For example, after two years we were only able to
add 7 records to a NASA-wide software cost metrics repository~\cite{me07f}.
Having failed to generate precise tunings, we developed 
SEESAW to discover what stable conclusions we could find within the space of possible tunings.

Another drawback with simplistic linear extrapolation is  that, when
optimizing for effort {\em and} time {\em and} defects, there may
be contradictory effects.  For example, we show below
one result
where effort {\em reduced}, but defects
{\em increased dramatically}. Hence, optimizing our models is not
a simple matter of moving fixed distances over some linear effect:
there are also some trade-offs to be considered (e.g. using a tool that considers combinations of effects,
like SEESAW).
\begin{figure}[!t]
\renewcommand{\baselinestretch}{0.95}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&prec&1&2&data&3\\
OSP:&flex&2&5&pvol&2\\
Orbital&resl&1&3&rely&5\\
space&team&2&3&pcap&3\\
plane&pmat&1&4&plex&3\\
&stor&3&5&site&3\\
&ruse&2&4&&\\
&docu&2&4&&\\
&acap&2&3&&\\
&pcon&2&3&&\\
&apex&2&3&&\\
&ltex&2&4&&\\
&tool&2&3&&\\
&sced&1&3&&\\
&cplx&5&6\\
&KSLOC&75&125\\\hline
&prec&3&5&flex&3\\
OSP2 &pmat&4&5&resl&4\\
&docu&3&4&team&3\\
&ltex&2&5&time&3\\
&sced&2&4&stor&3\\
&KSLOC&75&125&data&4\\
&&&&pvol&3\\
&&&&ruse&4\\
&&&&rely&5\\
&&&&acap&4\\
&&&&pcap&3\\
&&&&pcon&3\\
&&&&apex&4\\
&&&&plex&4\\
&&&&tool&5\\
&&&&cplx&4\\
&&&&site&6\\\hline
&rely&3&5&tool&2\\
JPL &data&2&3&sced&3\\
flight&cplx&3&6&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&7&418&&\\\hline
&rely&1&4&tool&2\\
JPL &data&2&3&sced&3\\
ground&cplx&1&4&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&11&392&&\\
\end{tabular}
\end{center}
\caption{Four case studies.
Numeric values 
$\{1,2,3,4,5,6\}$ map to $\{very low, low, nominal,
high, very high, extra high\}$.}\label{fig:cases}
\end{figure}
\Section{Options Within Projects}
SEESAW ranks project changes using the COCOMO and COQUALMO effort, time,
and defect predictors. The terminology of those models is reviewed
in \fig{cparems}.

Using that terminology, \fig{cases} summarizes four NASA case studies:
\bi
\item OSP is the GNC (guidance, navigation, and control)
component of NASA's 1990s {\em Orbital Space Plane}; 
\item OSP2 is a later version of OSP;
\item Flight and ground systems reflect typical ranges seen at NASA's Jet Propulsion Laboratory. 
\ei
Some of the features in \fig{cases}
are known precisely (see all the features with single {\em values}).
But  many of the features in \fig{cases} do not
have precise values
(see all the features that {\em range} from some {\em low} to {\em high}
value).  Sometimes the ranges
are very narrow (e.g., the process maturity of JPL ground software is between 2
and 3), and  sometimes the ranges are very broad.
\fig{cases} does not mention all the  features listed in \fig{cparems}
inputs.  For example, our defect predictor has inputs for
use of {\em automated analysis}, {\em peer
reviews}, and {\em execution-based testing tools}. For all inputs
not mentioned in \fig{cases}, values are picked at random from the full
range of \fig{cparems}.

\Section{Imposing Drastic Changes}
Using \fig{cases}, we can now clearly distinguish
between
{\em drastic} and   {\em internal} changes.
Drastic changes means
reorganizing a project {\em outside} of the ranges
shown in \fig{cases}, while internal changes 
reorganizes  a project {\em within}
those ranges.
Internal changes merely move the project into some subset of the current project assumptions.
Drastic changes, on the other hand, can completely alter those assumptions.
Take as an example the ``improve process maturity'' drastic change from \fig{dchange}.
According to \fig{cases}, JPL ground systems have process maturities in the range 2 to 3
(low to nominal).
Demanding very high
CMM levels for these JPL ground systems would likely
take years,
require the commitment  of very senior JPL management, and necessitate
extensive
retraining of local staff and/or hiring of new staff.

\begin{figure}[!t]
\footnotesize
\begin{tabular}{c@{~}p{.5\linewidth}p{.4\linewidth}}
&{\bf Drastic change} 
& {\bf Effects on \fig{cases}} \\\hline

1&Improve personnel 
& 
acap = 5; pcap = 5;
                    pcon = 5 \newline 
                    apex  = 5 ;
                    plex = 5 ;
                    ltex = 5 \\\hline
2&Improve tools, techniques, or  development platform 
& 
    time = 3; 
                    stor = 3\newline
                    pvol = 2;
                    tool = 5\newline
                    site = 6\\\hline
3&Improve precedentness / development flexibility 
&
prec = 5;
flex = 5
\\\hline
4&Increase architectural analysis /  risk resolution 
&resl = 5
\\\hline
5&Relax schedule  
& 
    sced = 5      
\\\hline

6&Improve process maturity 
& pmat = 5
\\\hline

7&Reduce \newline
functionality 
& 
 data = 2; 
                    kloc  * 0.5
\\\hline
8&Improve the team 
&
team = 5 
\\\hline

9&Reduce quality 
& 
  rely = 1 ;
                    docu = 1\newline
                    time = 3 ;
                    cplx = 1
\end{tabular}
\caption{Implementing drastic changes.
}\label{fig:how}
\end{figure}

\fig{how} defines the values we imposed on each case study as part
of each drastic change. 
Most of the values in \fig{how} are self-explanatory with two exceptions.
Firstly, the 
$kloc*0.5$ in ``reduce functionality'' means that,  when imposing this drastic change,
we only implement half the system.
Secondly, most of the features fall in the range one to five.
However, some have minimum values of 2 or higher (e.g., $pvol$ in ``improve tools/tech/dev''),
and some have maximum values of 6 (e.g., $site$ in ``improve tools/tech/dev'').
This explains why some of the drastic changes result in values other than one or five.

To impose a drastic change on a case study,
if that change refers to feature $X$ (in the right-hand column of
\fig{how}),
then we
first
(a)~removed $X$ from the values and ranges of the case study (if it was present); then
(b)~added the changes of \fig{how} as fixed values for that case study.

\Section{Finding Alternatives to Drastic Change}
SEESAW searches within the ranges of \fig{cases} to find constraints
that most
reduce development effort, development time, and defects.
\input{seesaw}
\input{model}
\input{results2.tex}
\Section{Results}

Our experiments compared predictions
after either
(a)~applying
the drastic changes of \fig{dchange};
or (b)~constraining some internal options selected
by 
SEESAW trying to minimize a linear combination of normalized defects, effort, and time;
or (c)~doing nothing at all and just running a Monte Carlo simulation
across the ranges of \fig{cases}.
Each experiment comprised 100 runs of \fig{MWF}. Each time a set
of project options were scored, $m$ values were selected at random
from 
Equations \ref{eq:mem},\ref{eq:linesf},\ref{eq:coq} and \ref{eq:linedefrem}.

In 
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect},
the rows are sorted by median scores.
The ``Do nothing'' row  comes from  Monte Carlo simulations of the current ranges, without any changes.
The {\em ranks} shown in the left-hand column are computed from a Mann-Whitney 
test at 95\% confidence 
test (this test was chosen since
(a) due to the random nature of
Monte Carlo simulations, the inputs to each run are not paired;
and (b) ranked tests make no, possibly inappropriate,
 assumption about normality of the results).
Two rows have the same rank 
if there is 
no statistical difference in their distributions. 

Each row shows
results from 100 
{\tt run}s (see \fig{MWF}): 
\bi
\item All results are normalized to run 0..100, min..max.
\item Each row shows the 25\% to 75\% quartile range of the normalized
scores 
collected during the simulation.
\item The median result is shown as a black dot. 
\ei
All the performance scores (effort, time, defects) get {\em better} when
the observed scores get {\em smaller}. 
Note that, usually, SEESAW obtains the better scores.

Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} show results from 12 experiments:
\[ 
\{effort, time, defects\}*\{flight, ground, OSP, OSP2\}\]
In $\frac{12}{12}$ (i.e., all) experiments, the median score from
SEESAW is better than at least half of the other changes.
Better yet, in $\frac{6}{12}$ (i.e., half) experiments, SEESAW
offers the best predictions,
and in two more experiments (the defect results for OSP and OPS2), 
SEESAW ties the top rank.
This result is the justification 
for our claim in the introduction that 
in  a majority of experiments ($\frac{8}{12}$), SEESAW's
search through the space of internal changes to current project options
does as well or better than
drastic change.

\input{freq.tex}
Another result of interest is that the range of SEESAW results is typically
smaller than most other changes
in
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} (observe how the 75\% - 25\% band
in SEESAW's results are usually smaller than all the others).
That is, of all the changes proposed here, the most stable were those generated by the stochastic
search. This paradoxical result
is simple to explain: SEESAW does not just {\em sample} a space of options; it also hunts for
options that {\em reduces} the median score within those options.
For models where the variance is proportional to the median, then reducing the median score
also reduces the variance.

Another stable feature of SEESAW are the actual recommendations.
An issue with stochastic sampling is that the output recommendations can
vary dramatically from one run to another.
Such variance can occur when a stochastic methods staggers between
multiple solutions of near equal value.
\fig{freq} shows the recommendations made by 20 runs of SEESAW for the flight and ground projects.
Note that the majority of the recommendations occur in all 20 runs and
represent the most important
actions that a project manager could make.
The recommendations that occur at low frequencies are
still valid, if used in conjunction with the other recommendations of that particular run.

The worst results from SEESAW
are see in
\fig{effort} where the median effort found by SEESAW
is near the 
middle of all changes. 
This result is hardly surprising.
This result comes from OSP2  and this case study
is
the most constrained case explored by this
paper (observer in \fig{cases} that,  for the most part,
the features all have fixed values). SEESAW works by exploring subsets of the available options.
When that option space is limited, the benefits of SEESAW are also limited.
That is, we cannot find
successful options unless we are first given enough  options to explore.
Previously~\cite{me07g} we have expressed this as
``if you fix everything, there is nothing left to fix''.

In other cases where drastic changes wins,
those changes
are often unacceptable.  For example,
in the effort and time results, SEESAW is beaten by ``reduce
quality'' in $\frac{4}{12}$ experiments.  The benefits of these
effort and time reductions should be weighed against the
implications on defects. \fig{defect} shows that ``reduce quality''
results in the largest number of defects seen in any change. While
there may exist projects that permit delivering code with large
numbers of defects, we believe that usually this is unacceptable.

Another drastic change that defeats SEESAW is ``improve pcap'' (personnel capabilities).
Recalling \fig{how}, we see that improving personnel capability requires changing
two analyst features, three personnel features, as well as addressing some issues relating
to language and tool experience. Of all the drastic changes studied in this paper, this
change is the most complex. It might also be the hardest to implement since it requires
dramatic change to the current personnel working on the project (perhaps even firing old
staff in order to hire new personnel).
If an organization can handle such large-scale change, then in some
cases (e.g., 
in $\frac{4}{12}$ experiments from 
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect})
the ``improve pcap'' option might indeed
outperform SEESAW.
However, managers might want to avoid such complex organizational change, preferring
instead to fine tune their project within its current structure using just internal changes.

\Section{Discussion}
Our thesis is that there exist some process models (e.g. COCOMO/COQUALMO)
that can still be used to rank proposed process changes, even without sufficient data to achieve
precise local tunings. This is not
to say that local tuning should not be attempted. Indeed, our strong view is that the preferred option is to use
precisely tuned models. However, in domains where such tunings
are unavailable, we recommend SEESAW.

We show that for COCOMO/COQUALMO, prediction uncertainty can be mitigated 
by selecting the right
project options {\em regardless} of uncertainty in the tunings.
This is {\em not} to say that prediction uncertainty of {\em all}
process models can be mitigated in the same manner. However, the observation that COCOMO/COQUALMO models
can be controlled in this manner is
one more reason to prefer these models.

Nor do we  claim that 
stochastic stability {\em always} beats {\em all} drastic change for all projects.
Indeed, we offer  examples where
extremely drastic change {\em defeats} the internal changes proposed by SEESAW.
However, in those cases, those changes were so drastic 
(reduce functionality, or make major change the personnel) that many managers may
prefer SEESAW's slightly-less-then-best recommendations.

\Section{Related Work}
Much of the related work on uncertainty in software engineering uses a Bayesian analysis.
For example,
Pendharkar et.al.~\cite{pendharkar05} demonstrate the utility of Bayes networks in effort
estimation while Fenton and Neil explore Bayes nets and defect prediction~\cite{fenton99}
(but unlike this paper,
neither of these teams links defect models to effort models).
We elect to take a non-Bayesian approach since most of the industrial and government
contractors we work with use parametric models like
COCOMO. 

The process simulation community (e.g. Raffo~\cite{raffo05c}) studies
models far more elaborate than COCOMO or COQUALMO. While such models
offer more detailed insight into an organization, the effort required
to tune them is non-trivial. For example, Raffo spent two years
tuning one such model to one particular site~\cite{raffo95}.

Other related work is the search-based SE approach advocated by Harmon~\cite{harman04}. 
Search-Based Software Engineering (SBSE) uses optimization techniques
from
operations research and meta-heuristic search (e.g. simulated annealing and genetic algorithms)
to hunt for near
optimal solutions to complex and over-constrained
software engineering problems. 
The SBSE approach can and has been applied to many
problems in software engineering (e.g. requirements engineering~\cite{jalali08})
 but most often in the field of software testing~\cite{andrews07}.
Harmon's writing inspired us try simulated annealing to search the what-ifs in untuned COCOMO
models~\cite{me07f}. However, we found that SEESAW ran much faster and produced results with far less
variance than simulated annealing.

\Section{Future Work}
The results of this paper assume that all the options found by SEESAW are {\em controllable};
i.e. a manager can implement all of SEESAW's recommendations. Future work should repeat this
analysis assuming that only certain subsets of the internal options are controllable.

Also,
this paper only explores the drastic changes of \fig{dchange} (these where defined in one
of our prior publications~\cite{boehm99}). There may be other drastic changes that defeat
the recommendations of SEESAW. Future work should explore a wider range of drastic changes.

%As to our own related work,
%for several decades, one of us (Boehm) has worked on a variety of software process models including
%the  COCOMO effort and time predictor~\cite[p29-57]{boehm00b}; 
%and
%the COQUALMO defect predictor~\cite[p254-268]{boehm00b}.
%One advantage of these models is that,
%unlike other models such as PRICE-S~\cite{park88}, 
%SLIM~\cite{putnam92}, or SEER-SEM~\cite{jensen83},
%our models are
%fully described in the literature. Also, at least
%for the effort model, there exist  baseline results~\cite{chulani99}. 
%\fig{cparems} lists the features of those  models.
%
%%XXX related work
%Most of the COCOMO literature
%concerns 
% {\em manual} use of these models for decision making or
%{\em automatic model tuning}.
%An alternative approach is the use of
%simulated annealing~\cite{kirkpatrick83} to explore {\em automatic} decision making
%{\em without local tuning}.
%In the COCOMO community, our work is somewhat unique. A standard approach in that community
%is to perform Monte Carlo analysis across the input space, sort the output predictions, then
%report the X\% percentile band as ``the'' prediction (usually 50\% to 75\%)~\cite{me08f}. 
%The work in
%this paper is different in that SEESAW does not just {\em
%report} the prediction variance, it tries
%to {\em reduce} it.
%
%
%At ASE`07~\cite{me07f}
%we used COCOMO and COQUALMO and
%simulated annealing to
%explore a very focused question: when is it
%useful to use automated software engineering tools? 
%While that question is of great interest to the automated software engineering
%community, here we explore a question 
%that is relevant to a much broader audience: what are the relative merits
%of combinations of internal changes versus drastic project changes?
%
%
%In his text {\em Empirical Methods in 
%Artificial Intelligence}, Cohen~\cite{cohen95}
%advises baselining a sophisticated
%better
%methods against simpler alternatives. 
%A simpler alternative to SEESAW is 
%the manual application of known best
%practices in improving  a project. 
%Before we endorse SEESAW or any other  algorithm,
%it is appropriate to first test
%if these engines do better than accepted best practices.
%Previously~\cite{boehm99}, we have reviewed  nine 
%such practices that can be manually applied---see the list   in \fig{dchange}.
%This paper performs Cohen's baseline test 
%by comparing the effects of applying accepted best practices versus the
%results of applying changes proposed by SEESAW.
%
%
\Section{Conclusion}
Drastic change is highly disruptive to a project.
For example, your manager has returned from ICSE`09
excited about the next new thing in software engineering.
``We need to drastically alter our projects,'' she says, ``in order to support (say) automatic
formal methods, agile process,  etc.''. 
This change
may entail switching from Windows to LINUX (since there are more 
research prototypes for automated formal methods running on LINUX);
stopping production
while engineers retrain;  or, worse, firing the engineers and making new hires. 

An alternative to drastic change are combinations of internal changes within the current
bounds of a project. Given precise local tunings to linear software process models 
(e.g. the COCOMO effort predictor and the COQUALMO defect predictors), then simple
linear extrapolation may suffice for exploring different process options.
However, when the tunings are not precise, or competing goals have to be achieved, 
simple linear extrapolation is insufficient.

This paper has explored the hard case of
(a)~ranking different process changes using (b)~models with competing influences
that (c)~have not been precisely tuned using local data.
In this case, the core task is to find {\em stability} within a very large space of possibilities.
Given a project with some variance in its process options and a model with some variance
in its tunings, then the model prediction
uncertainty  is some combination of the uncertainty
 with a project's
options and uncertainty with the model's tunings.
Local data can be used to reduce 
the uncertainty in the
tunings. However, as shown in \eq{ab}, even after tuning it is possible tuning uncertainty remains.

Based on research dating back to 1981~\cite{boehm81}, we assert that the
space of {\em
possible tunings} for COCOMO/COQUALMO is well defined
(see Equations \ref{eq:mem},\ref{eq:linesf},\ref{eq:coq} and \ref{eq:linedefrem}).
We show here that
at least for the these models,
%$U_3$ dominates $U_2$ since:
%\begin{quote}
%{\em The prediction uncertainty $U_1$ associated with untuned models can be mitigated via the
%right choice of project options.
%}
%\end{quote}
%This result does not mean local tuning 
%is worthless. Ideally, process options are ranked via precise local tunings. However, in domains with
%to insufficient data for precise tunings, it is still possible to rank
%process options. 
%
it is possible to explore
a very large range of ``what-ifs'' (the space of all possible tunings and internal project changes
)  
to find project option settings that improve model predictions, despite uncertainty in the tunings.

%XXX controllable assumption

Specifically, in studies with 4 NASA case studies and 8 drastic
changes, we found that, usually, SEESAW's stochastic
sampling found internal options that worked as well or
out-performed drastic change. Essential to the above is {\em stochastic stability}. Stochastic search enables the rapid
sampling of a very wide range of options. More importantly, when studying the space of
unknowns associated with a particular project (e.g. \fig{cases}),
stochastic sampling
can return multiple results after multiple runs. These results can be studied to check
the
{\em stability} of the solutions within the space of project options.
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} 
show that
the SEESAW stochastic stability tool typically generated the smallest range in
performance predictions.  That is, of all the changes studied here,
SEESAW's recommendations were typically the most stable.

Our results suggest that project managers
may have more project improvement options than they currently realize.
Hence, before performing drastic
changes it can be worthwhile to check
for stochastically stable effects amongst the internal changes, if only to understand
better 
the available 
options within the current structure of  a project.

\section*{Acknolwedgements}
Thanks to 
Phillip Green,
Topi Haapio and Andi Marcus and 
for their feedback on earlier drafts.
% local tuning still useful

\bibliographystyle{IEEE_CS_Latex/latex8}
\bibliography{refs}

\end{document}
\underline{{\bf KEY:}}\\
The quartile charts, shown on left,
plot the defects predictions after either
AI search with  AI search (line one);
or  applying various drastic changes; or 
after 
doing nothing (second last line).
\bi
\item
Results are sorted on 
median results from 100 Monte Carlo simulations.
\item
The ``Baseline'' row (second from the bottom) shows the defects predicted from the projects,
without any additional constraints.
\item
Median results are  shown as black circles. These
medians are surrounded  left and right
by
black lines showing bottom/top quartiles (respectively).
\item
The ranks shown in the left-hand-column are computed from a Mann-Whitney 95\% confidence test:
two rows have the same rank 
if there is 
no statistical difference in their distributions. 
For example, the
four rows have the same rank since their distributions are essentially the same.
\ei\\\hline
\end{tabular}
\end{minipage}
\caption{AI search (see line 1) reduces defects
far more than 
drastic process changes.
As might be expected, ``reducing quality'' 
produces more defects than
``Baseline'' (see last last).

